Data processing method and device, equipment, storage medium and program product
Patent Information
- Application Number
- CN202280094762.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-23
- Publication Date
- 2024-11-19
AI Technical Summary
Existing technology is difficult to effectively process and apply a large amount of first-principles data, resulting in low data retrieval efficiency and inability to meet users' lightweight application needs.
Through progressive disassembly and extraction of data layer by layer, structured process data and key data are obtained, and their associated data identifiers are stored in non-relational databases and search engine databases, and the data are processed based on the characteristics of first principles data. Storage and retrieval to achieve cross-database data association retrieval.
It improves the data retrieval response speed and efficiency, meets the user's diverse and lightweight application needs for first-principles data, and ensures the best performance and data security of the database system.
Smart Images

Figure CN118984990A_ABST
Abstract
Description
Data processing method, device, equipment, storage medium and program product Technical Field
[0001] The present application belongs to the field of data processing technology, and in particular relates to a data processing method, apparatus, device, storage medium and program product. Background Art
[0002] Based on the principles of interaction between atomic nuclei and electrons and their basic laws of motion, and applying the principles of quantum mechanics, algorithms that directly solve the Schrödinger equation based on specific requirements and after some approximations are conventionally referred to as first principles. First principles data can be obtained through calculations based on first principles.
[0003] First principles data typically generates a large amount of raw data in a relatively single use case. How to organize this massive amount of raw data to ensure effective application is a technical challenge that needs to be addressed.
[0004] Summary of the Invention
[0005] The embodiments of the present application provide a data processing method, apparatus, device, storage medium and program product to solve the technical problem in the prior art of how to sort out first-principles data with a large amount of original data to ensure effective data application.
[0006] A first aspect of an embodiment of the present application provides a data processing method, including:
[0007] Based on the first-principles raw data processing, structured first-principles process data is obtained;
[0008] Information extraction is performed on the first principle process data to obtain key data corresponding to the first principle original data; wherein the key data and the first principle process data have associated data identifiers.
[0009] In the technical solution of the embodiment of the present application, data is progressively decomposed and extracted layer by layer based on the original data to obtain process data and key data, and data identifiers that associate key data with process data. Data decomposition is performed in combination with the data characteristics of the first principles data to achieve effective data combing of the first principles data with a large amount of original data, thereby ensuring the data application effect.
[0010] In some embodiments, before the first principles raw data is processed to obtain structured first principles process data, the method further includes:
[0011] Obtaining an output file obtained based on a first-principles calculation, wherein the output file includes the first-principles raw data;
[0012] Detecting whether the output file meets a preset verification condition;
[0013] The step of processing the raw data based on the first principles to obtain structured first principles process data is performed when the output file meets a preset verification condition;
[0014] The preset verification conditions include: the presence of a complete end judgment symbol in the output file, and / or the data rationality of the first principle original data.
[0015] This processing process makes a computational integrity and rationality judgment on the output file containing the first-principles raw data, ensuring that the calculated first-principles raw data is legal and valid, providing the correct premise for subsequent data processing.
[0016] In some embodiments, the first-principles-based raw data processing to obtain structured first-principles process data includes:
[0017] According to the set fields, content extraction is performed on the raw data of the first principle to obtain a structured field data matrix;
[0018] The field data matrix is determined as structured first-principles process data.
[0019] In this process, setting fields are introduced to extract the first-principles raw data to obtain a field data matrix, so that the extracted first-principles process data has structured characteristics.
[0020] In some embodiments, the first-principles-based raw data processing to obtain structured first-principles process data includes:
[0021] According to the set fields, content extraction is performed on the raw data of the first principle to obtain a structured field data matrix;
[0022] Based on the field data matrix, according to the physical and chemical performance calculation formula, a structured physical and chemical performance data matrix is calculated;
[0023] The field data matrix and the physicochemical property data matrix are determined as structured first-principles process data.
[0024] This process further introduces the physicochemical performance calculation formula on the basis of introducing the set fields, realizes the extraction of the first-principles original data to obtain the field data matrix and the calculation of the physicochemical performance data matrix based on the field data matrix, so that the extracted first-principles process data has multi-level structured characteristics.
[0025] In some embodiments, extracting information from the first principle process data to obtain key data corresponding to the first principle raw data includes:
[0026] First data for retrieval is extracted from the first principle process data, and the first data is determined as the key data corresponding to the first principle original data.
[0027] In this way, the first data for retrieval is extracted from the process data as key data, which enables hierarchical sorting of the first principle data with a huge amount of original data, ensuring the effectiveness of data retrieval applications.
[0028] In some embodiments, extracting information from the first principle process data to obtain key data corresponding to the first principle raw data includes:
[0029] Extracting first data for retrieval from the first principle process data, and parsing the first principle process data according to a set calculation rule to obtain second data for retrieval;
[0030] The first data and the second data are determined as the key data corresponding to the first principle original data.
[0031] In this way, the first data for retrieval is extracted from the process data and the second data for retrieval is further parsed based on the first data. Both of them are used as key data to achieve hierarchical sorting of the first principle data with a huge amount of original data, ensuring the application effect of data retrieval.
[0032] In some embodiments, before the first-principles raw data processing is performed to obtain structured first-principles process data, the method further includes:
[0033] The first principle original data is compressed and stored in the object storage gateway.
[0034] The original data is stored in an object in a compressed manner, thereby ensuring the efficiency of original data storage and extraction.
[0035] In some embodiments, the method further comprises:
[0036] The first principle process data is stored in a non-relational database, and the key data is stored in a search engine database; wherein the search engine database stores the data identifier.
[0037] In this process, different data storage tools are set up, and the data obtained from each layer of processing is stored in the corresponding tools respectively. The data characteristics of the first principles data are combined instead of using a general data processing method for data storage, and the associated data identifiers between the disassembled data are recorded. This design ensures that the data storage structure is more in line with the data characteristics of the first principles data itself, improves the database system's retrieval response speed for the first principles data, meets the user's demand for lightweight application of the first principles data, and can combine the data characteristics to maximize the performance of the database.
[0038] In some embodiments, after storing the key data in the search engine database, the method further includes:
[0039] Backing up the key data in the search engine database to obtain backup data;
[0040] The backup data is stored in the non-relational database.
[0041] Based on the data backup function of the search engine database, the storage space and resource usage in the search engine database are reduced, and the data security between the data storage hierarchical structures is improved.
[0042] In some embodiments, after storing the key data in the search engine database, the method further includes:
[0043] In response to the data retrieval instruction, data retrieval is performed based on the search engine database.
[0044] Here, a data retrieval function is set in the search engine database where the key data is located to meet the lightweight retrieval application needs of the user side and ensure the optimal performance of the database.
[0045] In some embodiments, the step of performing data retrieval based on the search engine database in response to the data retrieval instruction includes:
[0046] In response to a data retrieval instruction, extracting first target data from the key data based on the search engine database;
[0047] Based on the first target data, a data retrieval result is output.
[0048] Here, on the basis of data processing, process data and key data are extracted by progressive decomposition layer by layer, and the data obtained by each layer are stored in the corresponding database respectively, the search engine database is further used to search for key data to meet the lightweight retrieval application needs of the user end.
[0049] In some embodiments, the step of performing data retrieval based on the search engine database in response to the data retrieval instruction includes:
[0050] In response to the data retrieval instruction, extracting second target data associated with the target data identifier from the first principle process data based on the search engine database;
[0051] Based on the second target data, a data retrieval result is output.
[0052] With the help of data identification that associates key data with process data, first-principles process data stored in non-relational databases are retrieved based on the search engine database, realizing cross-database data association retrieval and meeting the user-side's diverse lightweight retrieval application needs for first-principles data.
[0053] In some embodiments, the step of performing data retrieval based on the search engine database in response to the data retrieval instruction further includes:
[0054] In response to a data retrieval instruction, extracting third target data including the target data identifier from the key data based on the search engine database;
[0055] Outputting a data retrieval result based on the second target data includes outputting a data retrieval result based on the second target data and the third target data.
[0056] While the search engine database is retrieving the key data stored in itself, it uses data identifiers that associate key data with process data to retrieve the first principles process data stored in non-relational databases, realizing cross-database data association retrieval and meeting the user's further diversified lightweight retrieval application needs for first principles data.
[0057] In some embodiments, outputting a data retrieval result based on the second target data and the third target data includes:
[0058] generating a first data retrieval result based on the third target data, and outputting the first data retrieval result to a first display page;
[0059] generating a second data retrieval result corresponding to the second target data according to the structural features of the second target data;
[0060] The first data retrieval result and the second data retrieval result are output to the second display page.
[0061] By providing different search result display pages, process data and key data are displayed as data search results, providing users with different data search results, and meeting users' hierarchical search application needs for first principles data.
[0062] In some embodiments, outputting a data retrieval result based on the second target data includes:
[0063] generating a second data retrieval result corresponding to the second target data according to the structural features of the second target data;
[0064] The second data retrieval result is output to a second display page. Through the retrieval result display page, the process data is displayed as the data retrieval result, meeting the user's hierarchical retrieval application requirements for first principles data.
[0065] In some embodiments, generating a second data retrieval result corresponding to the second target data according to the structural features of the second target data includes:
[0066] Get the drawing instructions triggered by the selected interactive drawing option;
[0067] In response to the drawing instruction, extracting drawing data from the second target data according to the structural features;
[0068] A data presentation image is generated as the second data retrieval result based on the drawing data.
[0069] Provide users with interactive drawing options to meet their customized search and display needs.
[0070] In some embodiments, before performing data retrieval based on the search engine database in response to the data retrieval instruction, the method further includes:
[0071] Obtain data retrieval information based on first principles input;
[0072] determining the data retrieval instruction based on the data retrieval information;
[0073] In which, the data retrieval instruction includes: a first data retrieval instruction for performing data retrieval on the key data based on the data retrieval information, a second data retrieval instruction for performing data retrieval on the first principle process data based on the data retrieval information, or a third data retrieval instruction for performing data retrieval on the key data and the first principle process data based on the data retrieval information.
[0074] Based on the data retrieval information input by the user based on the first principles, different data retrieval instructions can be determined to effectively carry out targeted data retrieval processing based on the user input information, thereby meeting the user's diversified retrieval application needs for first principles data.
[0075] In some embodiments, determining the data retrieval instruction based on the data retrieval information includes:
[0076] The structural information, property information and calculation parameters included in the data retrieval information are encapsulated in a retrieval statement to obtain a data retrieval instruction carrying the encapsulated retrieval statement.
[0077] In this way, the data retrieval instruction corresponding to the data retrieval information input by the user can be effectively determined, thereby improving the accuracy of data retrieval.
[0078] In some embodiments, after storing the first principle process data in a non-relational database and storing the key data in a search engine database, the method further includes:
[0079] Detecting data consistency between the non-relational database and the search engine database according to set rules;
[0080] When it is determined that there is data inconsistency between the non-relational database and the search engine database, the process returns to the step of processing the original data based on the first principles to obtain structured first principles process data.
[0081] By detecting the data consistency between the non-relational database and the search engine database, the problem of data inconsistency between the non-relational database and the search engine database can be overcome by re-executing the data parsing and data storage operations.
[0082] In some embodiments, after storing the first principle process data in a non-relational database and storing the key data in a search engine database, the method further includes:
[0083] Get data adjustment instructions;
[0084] In response to the data adjustment instruction, after determining that the data adjustment is approved, adjust the data storage logic based on the data adjustment instruction;
[0085] Based on the adjusted data storage logic, return to the step of performing the first-principles-based raw data processing to obtain structured first-principles process data.
[0086] According to the adjustment instructions triggered by the user, the data warehousing logic is adjusted to re-execute the data parsing and data warehousing operations based on the adjusted data warehousing logic, so that the data between the non-relational database and the search engine database are consistent and the data warehousing is adjustable.
[0087] In some embodiments, the first principle raw data is stored in an object storage gateway; and the method further comprises:
[0088] Extracting target original data from the first principle original data based on the object storage gateway;
[0089] Based on the target original data, a data retrieval result is output.
[0090] Further realize the function of retrieving original data in the database system to ensure the comprehensiveness of the retrieval function.
[0091] A second aspect of an embodiment of the present application provides a data processing device, including:
[0092] A data processing module is used to process the raw data based on the first principles to obtain structured first principles process data;
[0093] The information extraction module is used to extract information from the first principle process data to obtain key data corresponding to the first principle original data; wherein the key data and the first principle process data have associated data identifiers.
[0094] A third aspect of an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.
[0095] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.
[0096] The fifth aspect of the present application provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.
[0097] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, which can be implemented in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0098] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference numerals are used throughout the drawings to represent the same components. In the drawings:
[0099] FIG1 is a flow chart of a data processing method according to some embodiments of the present application;
[0100] FIG2 is a schematic diagram of the hierarchical relationship of data analysis in some embodiments of the present application;
[0101] FIG3 is a data processing flow chart of some embodiments of the present application;
[0102] FIG4 is a data processing flow chart of some embodiments of the present application;
[0103] FIG5 is a data processing flow chart of some embodiments of the present application;
[0104] FIG6 is a data processing flow chart of some embodiments of the present application;
[0105] FIG7 is a schematic diagram of a data storage structure according to some embodiments of the present application;
[0106] FIG8 is a flowchart of data backup in some embodiments of the present application;
[0107] FIG9 is a flow chart of a data retrieval process according to some embodiments of the present application;
[0108] FIG10 is a flow diagram of data processing in some embodiments of the present application;
[0109] FIG11 is a flow chart of a data retrieval process according to some embodiments of the present application;
[0110] FIG12 is a flow chart of a data retrieval process according to some embodiments of the present application;
[0111] FIG13 is a flowchart of outputting data retrieval results according to some embodiments of the present application;
[0112] FIG14 is a schematic diagram showing the display of the data retrieval results of this application on a display page;
[0113] FIG15 is a flow chart of generating data retrieval results according to some embodiments of the present application;
[0114] FIG16 is a flowchart of a process for determining a data retrieval instruction according to some embodiments of the present application;
[0115] FIG17 is a flow chart of a data retrieval process according to some embodiments of the present application;
[0116] FIG18 is a structural diagram of a data processing device according to some embodiments of the present application;
[0117] FIG19 is a structural diagram of a computer device according to some embodiments of the present application. DETAILED DESCRIPTION
[0118] The following embodiments of the technical solution of the present application will be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application and are therefore only examples and are not intended to limit the scope of protection of the present application.
[0119] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.
[0120] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood to indicate or imply relative importance or implicitly specify the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is more than two, unless otherwise clearly and specifically defined.
[0121] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0122] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.
[0123] First-principles data typically involves a large amount of raw data in a relatively single use case. In specific applications, based on the principles of interaction between atomic nuclei and electrons and their basic laws of motion, and applying quantum mechanics, algorithms that directly solve the Schrödinger equation after some approximations are often referred to as first-principles methods. First-principles data can be obtained through first-principles calculations.
[0124] Faced with first-principles data with huge amounts of original data, effective data sorting is required to ensure effective data application.
[0125] The data objects targeted in the embodiments of this application are first principles data. The characteristics of first principles data include: 1. The amount of raw data is huge, but the usage scenario is single. This raw data is only used to provide basic backup and storage, as well as data viewing and downloading; 2. The data obtained by parsing the raw data is structured data with a large amount of data. This data can be used for two-dimensional and three-dimensional rendering and then graphically displayed to users; 3. The information that can be retrieved corresponding to the raw data is structural information, property information, etc. The data volume is small, but there are a large number of combined query scenarios.
[0126] Therefore, the inventors came up with the idea of proposing a data analysis and sorting method for first principles data, based on the data analysis level and the inherent data characteristics of the first principles data.
[0127] With reference to FIG1 , in some embodiments, a data processing method includes:
[0128] Step 101, processing the raw data based on the first principles to obtain structured first principles process data;
[0129] First principles raw data is first principles data obtained after data calculation and processing based on input data using first principles. This first principles raw data is unstructured data.
[0130] The first principles process data is structured data obtained by parsing the above first principles raw data.
[0131] This step processes unstructured raw data into structured process data.
[0132] In a specific implementation, data extraction can be performed on the first-principles raw data according to a set dimension to obtain structured first-principles process data corresponding to the set dimension; alternatively, data can be filled into the first-principles raw data at a set position in the data structure matrix using a set data structure matrix to obtain structured first-principles process data. This is merely an example and is not intended to be limiting.
[0133] Step 102: extract information from the first principles process data to obtain key data corresponding to the first principles original data.
[0134] Among them, the key data and the first principle process data have associated data identifiers.
[0135] The data identifier indicates the association relationship between the key data and the first principles process data. Through the data identifier, the connectivity between the key data and the first principles process data is established.
[0136] Optionally, the data identifier may be a data identification code or data feature description information, which is merely an example and not intended to be limiting.
[0137] The process of extracting information from first principles process data can be to directly extract key data from the first principles process data, or to perform data analysis based on the first principles process data and then extract the analysis results to obtain key data. This is merely an example and is not intended to be limiting.
[0138] This step parses and generates a relatively small amount of key data from the structured process data.
[0139] In some embodiments, as shown in FIG2 , data decomposition and extraction are performed on the basis of the original data (i.e., the first principles original data) to obtain process data (i.e., the first principles process data), and key data are further extracted based on the process data. In combination with the data characteristics of the first principles data having a large amount of data, process data and key data are obtained through layer-by-layer progressive decomposition and extraction of the data, as well as data identifiers that associate the key data with the process data, thereby realizing effective data combing of the first principles data with a large amount of original data.
[0140] Among them, raw data refers to the first principles data directly obtained after computing and processing based on input data by high-performance computers according to the first principles of scientific computing. In one example, the raw data specifically refers to unstructured data files obtained by standard software based on input files and computing cluster operations according to first principles. The content of data files varies greatly depending on the stored content. Important data files can exceed 500,000 lines, and some larger data files (such as charge density result files) can contain more than 1 million lines (file storage space can reach more than 10GB).
[0141] Process data refers to structured data derived from raw data analysis. Process data contains as much useful information as possible from the raw data. The large amount of data can be used for 2D or 3D rendering and graphical presentation to users. In data retrieval applications, this process data can be used to implement customized plotting capabilities.
[0142] Key data refers to the frequently used data after analyzing process data. It is usually used for retrieval and quick display. It is characterized by high importance and small data volume.
[0143] In the technical solution of the embodiment of the present application, data is progressively decomposed and extracted layer by layer based on the original data to obtain process data and key data, and data identifiers that associate key data with process data. Data decomposition is performed in combination with the data characteristics of the first principles data to achieve effective data combing of the first principles data with a large amount of original data, thereby ensuring the data application effect.
[0144] Furthermore, in one example, before the above step 101 processes the raw data based on the first principles to obtain structured first principles process data, the following steps are further included:
[0145] Obtain the output file obtained based on the first principles calculation and check whether the output file meets the preset verification conditions.
[0146] The output file contains the first-principles raw data.
[0147] Among them, when the output file meets the preset verification conditions, step 101 is executed to process the original data based on the first principles to obtain structured first principles process data, and other steps including step 102 after step 101 are executed continuously.
[0148] That is, the step of obtaining structured first-principles process data based on first-principles raw data processing is performed when the output file meets the preset verification condition.
[0149] The above-mentioned preset verification conditions include: the presence of a complete end judgment symbol in the output file, and / or the first-principles original data has data rationality.
[0150] That is, it is necessary to determine whether there is a complete end judgment symbol in the output file containing the first-principles raw data, or to determine whether the first-principles raw data in the output file containing the first-principles raw data has data rationality, or to determine whether there is a complete end judgment symbol in the output file containing the first-principles raw data and the first-principles raw data therein has data rationality.
[0151] In a specific application example, the original file generated by a supercomputing cluster for first-principles scientific calculations is an output file containing the original first-principles data. Output files such as VASPRUN.XML, OUTCAR, and log files are used to calculate file integrity based on these output files. This typically involves determining whether the file generation process completed normally. This determination is made by determining whether the output file contains complete end-of-line descriptors. For example, this involves determining whether the last line of the file contains "," the second-last line contains "Voluntary context switches:," and the last line contains "Normal termination of Gaussian."
[0152] You can also judge the rationality of the data in the file. For example, when judging the rationality of structural relaxation optimization based on business needs, you can determine whether the words "reached required accuracy" exist within the last approximately 1000 lines of the output file OUTCAR.
[0153] The data parsing process is triggered after determining that the output file meets the above preset verification conditions.
[0154] The above processing process ensures the legality and validity of the calculated first-principles raw data by performing computational integrity and rationality judgment on the output file containing the first-principles raw data, providing the correct premise for subsequent data processing.
[0155] In some embodiments, as shown in FIG3 , step 101 processes the raw data based on first principles to obtain structured first principles process data, including:
[0156] Step 301: extract the content of the first-principles original data according to the set fields to obtain a structured field data matrix;
[0157] Step 302: Determine the field data matrix as structured first-principles process data.
[0158] The set fields are, for example, frequency fields related to optical properties, imaginary fields of dielectric functions, etc. In the process of processing the first-principles raw data to obtain structured first-principles process data, data extraction can be performed directly from the first-principles raw data, that is, the content contained in the raw data is directly extracted to form a structured field data matrix, which is used as the first-principles process data.
[0159] In this process, setting fields are introduced to extract the first-principles raw data to obtain a field data matrix, so that the extracted first-principles process data has structured characteristics.
[0160] In other embodiments, in addition to directly extracting the content contained in the original data to form a structured field data matrix, the content contained in the original data can be further parsed to generate a structured physical and chemical performance data matrix, which can be used as first-principles process data.
[0161] As shown in FIG4 , step 101 processes the raw data based on the first principles to obtain structured first principles process data, including:
[0162] Step 401: extract the content of the first-principles original data according to the set fields to obtain a structured field data matrix;
[0163] Step 402: Based on the field data matrix, a structured physical and chemical performance data matrix is calculated according to the physical and chemical performance calculation formula;
[0164] Step 403: Determine the field data matrix and the physical and chemical performance data matrix as structured first-principles process data.
[0165] In this process, based on the field data matrix extracted from the raw data, the physical and chemical performance calculation formula is introduced to perform data calculations, resulting in a physical and chemical performance data matrix. This matrix, together with the field data matrix, serves as structured first-principles process data. The resulting structured process data consists of two parts: one part is the content originally contained in the raw data, and the other part is the data content generated after parsing the content contained in the raw data.
[0166] In the processing process of the above two examples, in the specific implementation, the first-principles raw data can be recorded in the output file OUTCAR generated by the first-principles calculation software after processing the input data. In the process of obtaining structured first-principles process data based on the first-principles raw data processing, taking the data extraction of optical properties in the raw data as an example, it is first necessary to accurately locate the imaginary part field of the frequency-related dielectric function in OUTCAR and extract 7 columns * about 1000 to 10000 rows of large matrix data. This data is the imaginary part matrix of the dielectric function. The imaginary part matrix of the dielectric function can be used as a field data matrix to form structured first-principles process data.
[0167] Alternatively, based on the imaginary part matrix of the dielectric function, other optical property-related physicochemical data, such as the refractive index, extinction coefficient, and absorption coefficient matrix, can be further calculated using physicochemical formulas to form a physicochemical performance data matrix. The resulting process data consists of two parts: {a. the directly extracted imaginary part matrix of the dielectric function, and b. the parsed matrix of other physicochemical data}, resulting in structured first-principles process data consisting of a field data matrix and a physicochemical performance data matrix.
[0168] This process further introduces the physicochemical performance calculation formula on the basis of introducing the set fields, realizes the extraction of the first-principles original data to obtain the field data matrix and the calculation of the physicochemical performance data matrix based on the field data matrix, so that the extracted first-principles process data has multi-level structured characteristics.
[0169] In some embodiments, as shown in FIG5 , step 102 extracts information from the first principles process data to obtain key data corresponding to the first principles raw data, including:
[0170] Step 501: extracting first data for retrieval from first-principles process data;
[0171] Step 502: Determine the first data as key data corresponding to the first principles original data.
[0172] This process, combined with the usage characteristics of the first principles data, further extracts data based on the extracted first principles process data to obtain the first data for retrieval.
[0173] Similarly, in the process of extracting information from first principles process data and obtaining key data corresponding to the first principles original data, data extraction can be performed directly from the first principles process data, that is, the first data for retrieval can be directly extracted from the content contained in the process data and used as the key data.
[0174] In this way, the first data for retrieval is extracted from the process data and used as key data to achieve hierarchical sorting of the first principle data with a huge amount of original data, ensuring the effectiveness of data retrieval applications.
[0175] In other embodiments, in addition to directly extracting the content contained in the process data to obtain first data for retrieval, the content contained in the process data may be further parsed to generate second data for retrieval, which may serve as key data.
[0176] As shown in FIG6 , step 102 extracts information from the first principles process data to obtain key data corresponding to the first principles original data, including:
[0177] Step 601: extracting first data for retrieval from first principles process data, and parsing the first principles process data according to a set calculation rule to obtain second data for retrieval;
[0178] Step 602: Determine the first data and the second data as key data corresponding to the first principles original data.
[0179] In this process, the key data generated includes two parts: one part is the content contained in the process data, and the other part is the data content generated after parsing the content contained in the process data.
[0180] In the specific implementation of the processing processes in the above two examples, for example: in the process of extracting key information of optical properties in process data, the Fermi level value, energy value and other data can be directly extracted from the structured optical property data in the process data, and used as the first data for retrieval, forming key data including {Fermi level value and energy value}.
[0181] Alternatively, based on the structured optical property data, the absorption coefficient can be obtained by calculating the combination of the solar spectrum and the absorption coefficient matrix, solving the integral area, and the band gap value can be obtained based on the part of the element value in the imaginary matrix of the dielectric function that changes from 0 to other values, or the band gap value can be obtained based on the distance between the bottom of the conduction band and the top of the valence band in the energy band calculation, and finally the key data including {absorption coefficient, band gap value, Fermi level value and energy value} are obtained.
[0182] In this way, the first data for retrieval is extracted from the process data and the second data for retrieval is further parsed based on the first data. Both of them are used as key data to achieve hierarchical sorting of the first principle data with a huge amount of original data, ensuring the application effect of data retrieval.
[0183] The above examples provide an implementation process for sorting and analyzing data. Furthermore, based on the above examples, after sorting and analyzing the data, it is also necessary to effectively store and retrieve the data.
[0184] With the development of digital information, data storage and retrieval have become increasingly common and important. Existing data storage methods store data objects in designated database tables, connect the database to a user interface, and when data retrieval is needed, keywords or combinations of keywords are used to perform combined queries across NoSQL and SQL databases, improving retrieval efficiency.
[0185] In order to improve the response speed of the database and enhance the data retrieval efficiency, the prior art has also proposed a method of pre-processing the retrieval input information and internally connecting to the database for data retrieval query based on the pre-processed retrieval information, that is, a retrieval processing method that adopts pre-screening followed by internal connection. Specifically, it is necessary to pre-analyze the query language and then use the converted query mode to retrieve data from the database, thereby reducing the amount of data returned by NoSQL conversion and the number of table connection operations, improving the algorithm execution efficiency, thereby reducing the utilization rate of SQL storage space, and ultimately improving data response efficiency and data query efficiency.
[0186] However, the inventors have discovered that for some special data processing applications, especially those with huge amounts of data, the existing data storage structures and data retrieval technologies still cannot meet the requirements in terms of retrieval efficiency.
[0187] Due to the original nature of first-principles data, in actual use, faced with huge data volumes and users' demand for lightweight applications of first-principles data, existing database systems need to traverse and match key data for all data when facing data query requirements. When applied to the retrieval processing of first-principles data, data positioning and extraction efficiency will be low, the overall response speed of the database system will be low, and it will be difficult to meet the user-side demand for lightweight applications of first-principles data.
[0188] Therefore, the inventors came up with the idea of proposing a data storage structure for first principles data from the data storage level, combining the data characteristics of the first principles data itself.
[0189] As shown in Figure 7, in one embodiment, the original data of the first principles data is stored in the object storage gateway, process data is obtained based on the original data, the process data is stored in a non-relational database, and key data is further extracted based on the process data and stored in a search engine database.
[0190] Among them, object storage gateways, non-relational databases and non-relational databases all have data storage functions.
[0191] In this data storage level, by setting different data storage tools, different levels of data processed on the basis of original data are stored separately. The data analysis and storage are implemented by combining the data characteristics of the first principles data instead of adopting the general data processing method. The data storage structure is improved at the data storage level so as to be able to give full play to the optimal performance of the database by combining the data characteristics.
[0192] Non-relational databases (NoSQL) differ from relational databases in that they do not guarantee the ACID (Atomicity, Consistency, Isolation, Durability) properties of relational data. They are suitable for storing structured process data. Examples include CouchDB, MongoDB, HBase, and Redis.
[0193] Among them, the object storage gateway implements object storage of data.
[0194] Object storage is a computer data storage architecture that manages data as objects, in contrast to other storage architectures such as file systems, which manage data as a hierarchy of files, and block storage, which manages data as blocks within sectors. Each object typically consists of the data itself, a variable amount of metadata, and a globally unique identifier. An example of an object storage gateway is Amazon S3.
[0195] Search engine databases are data processing tools that combine data storage and search engine functions. In one example, a search engine database is a type of non-relational database specialized for data content search. They use indexes to group similar features within the data and improve search capabilities. They are optimized to handle very long, semi-structured, or unstructured data. They typically offer specialized methods such as full-text search, complex search expressions, and search result ranking. Examples include Solr, Elasticsearch, and Splunk.
[0196] The above process performs a step-by-step progressive decomposition and extraction of data based on the original data to obtain process data and key data for retrieval. At the same time, different data processing tools are set to store the data obtained from each layer of processing in the corresponding tools. The data characteristics of the first principles data are combined instead of using a general data processing method to decompose and store the data. In combination with the data characteristics of the first principles data itself, the data storage structure is improved at the data storage level to be able to give full play to the optimal performance of the database in combination with the data characteristics.
[0197] Furthermore, the above data storage structure specifically corresponds to some data processing methods to realize data storage and data retrieval applications.
[0198] According to some embodiments of the present application, based on the data storage structure shown in Figure 7, before step 101, based on the first principles raw data processing to obtain structured first principles process data, it can also include: compressing the first principles raw data and storing it in an object storage gateway.
[0199] During the specific implementation process, the output file obtained based on the first-principles calculation can be obtained first, and after detecting that the output file meets the preset verification conditions, that is, after the output file passes the calculation integrity and rationality judgment, all files under the path of the output file containing the first-principles original data are compressed and then stored in the object storage gateway.
[0200] This process enables the storage of original data in the object storage gateway and stores the original data in an object-based manner in a compressed manner, thereby ensuring the efficiency of original data storage and extraction.
[0201] Differently, in some examples, the data processing method further includes:
[0202] The first principle process data is stored in a non-relational database, and the key data is stored in a search engine database; wherein the search engine database stores data identifiers that associate the key data with the first principle process data.
[0203] In one example, in the above processing process, storing the first principles process data in a non-relational database can occur after step 101, which processes the original data based on the first principles to obtain structured first principles process data, without step 102 being a necessary prerequisite.
[0204] Alternatively, in one example, storing the first principles process data in a non-relational database in this step can occur after extracting information from the first principles process data in step 102 to obtain key data corresponding to the first principles original data, and can be implemented together with the step of storing the key data in a search engine database.
[0205] Among them, key data can be stored in the search engine database in the form of full parameters to ensure that the user end can provide a large number of concurrent fast retrievals on this basis.
[0206] The search engine database stores data identifiers for associating key data with first principles process data. In one example, a group of key data with an associated relationship is identical to or corresponds to the data identifiers corresponding to the first principles process data.
[0207] In this process, different data storage tools are set up, and the data obtained from each layer of processing are stored in the corresponding tools respectively. The data characteristics of the first principles data are combined instead of using a general data processing method for data storage, and the associated data identifiers between the disassembled data are recorded to ensure that in the subsequent data retrieval processing process, it can respond to different retrieval requirements and carry out corresponding data positioning and extraction from the data storage structure for the data obtained from different levels of processing, so as to meet the user-side lightweight application requirements of the first principles data to different degrees, so as to be able to combine the data characteristics to give full play to the optimal performance of the database.
[0208] According to some embodiments of the present application, optionally, based on the data storage structure shown in Figure 7, key data can also be backed up. In conjunction with Figure 8, after storing the key data in the search engine database, the following is also included:
[0209] Step 801, backing up key data in the search engine database to obtain backup data;
[0210] Step 802: Store the backup data in a non-relational database.
[0211] Here, the data stored in one of the databases in the data storage structure shown in FIG. 7 is backed up in another database.
[0212] The above processing process stores the backup data of key data in the search engine database in a non-relational database outside of itself, reducing the storage space and other hardware resources occupied in the search engine database, ensuring data persistence and integrity, and improving data security and stability between data storage hierarchies.
[0213] Furthermore, in the embodiments of the present application, after proposing a data storage structure for first principles data from the data storage level and combining the data characteristics of the first principles data itself, on this basis, some examples are also provided in the embodiments of the present application to illustrate the processing process of the data retrieval application.
[0214] Specifically, the search engine database has both data storage and data retrieval functions.
[0215] According to some embodiments of the present application, after storing the key data in the search engine database, the method further includes:
[0216] In response to the data retrieval instruction, data retrieval is performed based on the search engine database.
[0217] Specifically, the search engine database can perform data retrieval on key data and / or first principle process data based on data retrieval instructions.
[0218] This implementation method selects a search engine database where the key data is located that has a data retrieval function for key data and / or process data, and uses the efficient retrieval capability of the search engine database to carry out data retrieval and associated queries for key data and first-principles process data, thereby meeting the user end's lightweight retrieval application needs at different levels and ensuring the optimal performance of the database.
[0219] According to some embodiments of the present application, as shown in FIG9 and FIG10 , the above-mentioned response to the data retrieval instruction and the implementation of data retrieval based on the search engine database include:
[0220] Step 901, in response to a data retrieval instruction, extracting first target data from key data based on a search engine database;
[0221] Step 902: output data retrieval results based on the first target data.
[0222] After receiving a data retrieval instruction triggered by a user, a response is made to the data retrieval instruction, and the retrieval keyword indicated in the data retrieval instruction can be passed to the search engine database. The search function provided in the search engine database is used to extract the first target data that matches the retrieval keyword from the key data stored therein.
[0223] When outputting the data retrieval result, the first target data may be directly output as the data retrieval result, or the first target data may be organized into a set data display format and output as the data retrieval result.
[0224] This process corresponds to the process of searching key data by the search engine database.
[0225] Here, based on the data processing of extracting process data and key data layer by layer and storing the data obtained from each layer in the corresponding database, we further use the search engine database to search for the key data stored in itself, improve the search response speed and search feedback efficiency, and meet the lightweight search application needs of the user end.
[0226] According to some embodiments of the present application, as shown in FIG10 and FIG11 , the aforementioned response to the data retrieval instruction and the implementation of data retrieval based on the search engine database include:
[0227] Step 1101 , in response to a data retrieval instruction, extracting second target data associated with a target data identifier from first principle process data based on a search engine database;
[0228] Step 1102: output data retrieval results based on the second target data.
[0229] After receiving a data retrieval instruction triggered by a user, a response is made to the data retrieval instruction, and the retrieval keywords indicated in the data retrieval instruction can be passed to the search engine database. The search function of the search engine database is utilized, and with the help of data identifiers that associate key data with process data, the first principles process data stored in the non-relational database is retrieved based on the search engine database, thereby realizing cross-database data association retrieval.
[0230] When outputting the data retrieval result, the second target data may be directly output as the data retrieval result, or the second target data may be organized into a set data display format and output as the data retrieval result.
[0231] This process, with the help of data identification that associates key data with process data, can quickly locate process data stored in non-relational databases, achieve efficient multi-database data association, improve the overall response efficiency of the database system, and meet the diverse retrieval application needs of the user end.
[0232] According to some embodiments of the present application, as shown in FIG10 and FIG12 , the aforementioned response to the data retrieval instruction and the implementation of the data retrieval based on the search engine database may further include, in addition to step 1101:
[0233] Step 1103, in response to the data retrieval instruction, extracting third target data including the target data identifier from the key data based on the search engine database;
[0234] Then, correspondingly, step 1102 outputs the data retrieval result based on the second target data as follows:
[0235] Step 1104: output data retrieval results based on the second target data and the third target data.
[0236] After receiving a data retrieval instruction triggered by a user, a response is made to the data retrieval instruction, and the retrieval keyword indicated in the data retrieval instruction can be passed to the search engine database. The search function of the search engine database is used to extract the third target data containing the target data identifier from the key data stored therein, and then the target data identifier corresponding to the third target data is used to realize cross-database data association retrieval of the first principle process data stored in the non-relational database, thereby retrieving the second target data.
[0237] Correspondingly, when outputting the data search results, the data search results are output based on the second target data and the third target data. Specifically, the second target data and the third target data can be directly output as the data search results, or the second target data and the third target data can be organized into a set data display format and output as the data search results.
[0238] The above processing process can realize data association retrieval between databases based on the data identification that associates key data with first principles process data according to the user's retrieval needs, and provide users with multi-level data retrieval results based on the storage hierarchy of the database system, thereby meeting the user's diverse lightweight application needs for first principles data.
[0239] According to some embodiments of the present application, in combination with FIG. 10 and FIG. 13 , step 1104 outputs a data search result based on the second target data and the third target data, including:
[0240] Step 1301: Generate a first data retrieval result based on the third target data, and output the first data retrieval result to a first display page.
[0241] Here, the third target data can be directly used as the first data retrieval result, or the third target data can be used as the first data retrieval result after data processing in a set format. The set format can be but is not limited to a data construction format or a data display arrangement format.
[0242] Step 1302: Generate a second data retrieval result corresponding to the second target data according to the structural features of the second target data.
[0243] The second target data with structural features can be directly used as the second data search result. Alternatively, content of the second target data can be extracted according to the structural features of the second target data, and the extracted data content can be integrated to generate the second data search result.
[0244] The structural features include, for example, data content structure features, field meanings in the data structure, and other characteristic information. The integration includes, for example, natural text integration of the data or content deduplication integration of the data. This is merely an example and is not intended to be limiting.
[0245] Step 1303: Output the first data retrieval result and the second data retrieval result to the second display page.
[0246] The display of the first data retrieval result on the first display page and the display of the first data retrieval result and the second data retrieval result on the second display page may be synchronous or asynchronous.
[0247] In the above steps, as shown in FIG14 , the first data retrieval result is output to the first display page, and the first data retrieval result and the second data retrieval result are output to the second display page. The first display page can be a quick preview page, and the second display page can be a detail display page. The content retrieved from the key data is displayed as the display content of the quick response page on the quick preview page, and the content retrieved from the key data and the content retrieved from the process data are displayed as detailed information on the detail display page, thereby achieving differentiated page display of different content.
[0248] The first display page and the second display page may be pages arranged side by side, or the second display page may be a page displayed after jumping from the first display page.
[0249] The above processing process provides different search result display pages, displays process data and key data as data retrieval results on differentiated pages, provides users with different data retrieval results, meets users' hierarchical retrieval application needs for first principles data, and improves user retrieval experience through hierarchical data display response speed.
[0250] According to other embodiments of the present application, step 1102 outputs a data retrieval result based on the second target data, including:
[0251] According to the structural features of the second target data, a second data retrieval result corresponding to the second target data is generated; and the second data retrieval result is output to the second display page.
[0252] This process corresponds to step 1101, which responds to the data retrieval instruction and extracts the second target data associated with the target data identifier from the first principle process data based on the search engine database, so as to realize the separate display of the second target data extracted from the process data in the set display page.
[0253] In an application process, when it is necessary to display only the detailed information of the search result, the second data search result corresponding to the second target data can be selected to be displayed as detailed information separately on the detailed display page (ie, the second display page).
[0254] In this way, by displaying the process data as data retrieval results on a separate page, the detailed display of the retrieval results can be achieved, meeting the user's demand for displaying diversified retrieval results.
[0255] In some embodiments, as shown in FIG15 , generating a second data retrieval result corresponding to the second target data according to the structural features of the second target data includes:
[0256] Step 1501: Obtain a drawing instruction triggered by a selected interactive drawing option.
[0257] Here, the interactive drawing options include, for example, setting options for drawing data value ranges, drawing dimension setting options, drawing display size options, drawing data combination options, drawing data parsing options, and the like.
[0258] Step 1502 : In response to the drawing instruction, extract drawing data from the second target data according to the structural features.
[0259] The drawing instruction indicates drawing parameters corresponding to the selected interactive drawing option. Based on the drawing parameters, matching drawing data is extracted from the process data (here, the second target data) with structured features for drawing rendering.
[0260] Step 1503: Generate a data display image based on the drawing data as a second data retrieval result.
[0261] In the above process, interactive drawing options are provided to the user, so that the user can use these options to realize customized drawing of the target process data, improve the display effect of the search results, and meet the user's customized search display needs.
[0262] According to some embodiments of the present application, as shown in FIG16 , before responding to the data retrieval instruction and performing data retrieval based on the search engine database, the method further includes:
[0263] Step 1601, obtaining data retrieval information based on first principles input;
[0264] Step 1602: Determine a data retrieval instruction based on the data retrieval information.
[0265] The data retrieval instruction includes a first data retrieval instruction for performing data retrieval on key data based on the data retrieval information. The first data retrieval instruction corresponds to the processing of the aforementioned steps 901 and 902.
[0266] The data retrieval instruction includes a second data retrieval instruction for performing data retrieval on the first principle process data based on the data retrieval information. The first data retrieval instruction corresponds to the processing of the aforementioned steps 1101 and 1102.
[0267] Alternatively, the data retrieval instruction includes a third data retrieval instruction for performing data retrieval on the key data and the first principle process data based on the data retrieval information. The third data retrieval instruction corresponds to the processing of the aforementioned steps 1103, 1101 and 1104.
[0268] Among them, data retrieval information is, for example, structural information of the first principles data to be retrieved (such as chemical formula, element information, dimensional information, atomic number, crystal system, space group number, density, volume, etc.), property information (such as band gap information, absorption coefficient, energy information, energy barrier information, etc.), calculation parameters (such as truncation radius, magnetic information, functional type, projection type, etc.), basic information (author, calculation time, import time, calculation identification number, structure identification number, calculation time, number of cores used, etc.) or a combination of the above information.
[0269] Based on the data retrieval information, a retrieval statement can be constructed to obtain a data retrieval instruction corresponding to the retrieval statement.
[0270] This process can determine different data retrieval instructions based on the data retrieval information input by the user based on the first principles, so as to effectively carry out targeted data retrieval processing based on the user input information and meet the user's diversified retrieval application needs for first principles data.
[0271] Furthermore, according to some embodiments of the present application, step 1602, determining a data retrieval instruction based on the data retrieval information, specifically includes:
[0272] The structural information, property information and calculation parameters contained in the data retrieval information are encapsulated into a retrieval statement to obtain a data retrieval instruction carrying the encapsulated retrieval statement.
[0273] In specific applications, after receiving the data retrieval information entered by the user on the user-side interface, the retrieval statement can be encapsulated based on the content contained in the data retrieval information. For example, the retrieval conditions or retrieval keywords formed by structural information, property information and calculation parameters are embedded in a specific position in the SQL retrieval statement, and then the statement is encapsulated to obtain a data retrieval instruction carrying the encapsulated retrieval statement.
[0274] The encapsulated search statement may include not only structural information, property information, and calculation parameters, but also other search information. Users may input the information based on their needs, and there is no special limitation on this.
[0275] In this way, the data retrieval instruction corresponding to the data retrieval information input by the user can be effectively determined, thereby improving the accuracy of data retrieval.
[0276] After the data retrieval instruction is generated, the search engine database can subsequently perform rapid retrieval positioning based on the retrieval statement carried in the data retrieval instruction and provide retrieval result information.
[0277] Furthermore, in some embodiments of the present application, data operation and maintenance processing can be further performed on the database.
[0278] In one example, optionally, after storing the first principles process data in a non-relational database and storing the key data in a search engine database, the method further includes:
[0279] The data consistency between the non-relational database and the search engine database is detected according to the set rules; when it is determined that there is data inconsistency between the non-relational database and the search engine database, the step of returning to execute the first principles-based original data processing to obtain structured first principles process data.
[0280] The setting rule may be a set time interval or a set period, wherein the set time interval may be a fixed time interval or a time interval with increasing or decreasing length, and may be specifically set according to actual needs.
[0281] During the implementation process, it is necessary to re-trigger the data parsing and warehousing processing flow when it is determined that there is a data inconsistency between the non-relational database and the search engine database. Specifically, the execution is restarted from step 101, and step 102 is continued to implement data content extraction and analysis, and then the first principle process data is stored in the non-relational database, and the key data is stored in the search engine database. The data warehousing processing flow.
[0282] By detecting the data consistency between the non-relational database and the search engine database, the problem of data inconsistency between the non-relational database and the search engine database can be overcome by re-executing the data parsing and data storage operations.
[0283] In another example, optionally, after storing the first principle process data in a non-relational database and storing the key data in a search engine database, the method further includes:
[0284] Obtain data adjustment instructions; respond to the data adjustment instructions, and after confirming that the data adjustment is approved, adjust the data storage logic based on the data adjustment instructions; based on the adjusted data storage logic, return to execute the first principles-based original data processing to obtain structured first principles process data.
[0285] Here, the user may input logic adjustment control information on the client interface, and data adjustment instructions may be generated accordingly based on the logic adjustment control information.
[0286] After detecting the corresponding data adjustment instruction, the data adjustment application corresponding to the data adjustment instruction needs to be approved. After the approval is passed, the data warehousing logic can be adjusted based on the logical adjustment control information indicated in the data adjustment instruction, and based on the adjusted data warehousing logic, the data parsing and warehousing processing flow can be re-triggered. Specifically, based on the adjusted data warehousing logic, the execution starts again from step 101, and then step 102 is executed to realize the content extraction and analysis of the data, and then the first principle process data is stored in the non-relational database, and the key data is stored in the search engine database. The data warehousing processing flow.
[0287] During this process, the data entry logic is adjusted according to the adjustment instructions triggered by the user, and the data parsing and data entry operations are re-executed based on the adjusted data entry logic, so that the data between the non-relational database and the search engine database are consistent and the data entry is adjustable.
[0288] In some embodiments, as shown in FIG17 , when the first principles raw data is stored in the object storage gateway, the data processing method further includes:
[0289] Step 1701: extracting target raw data from first-principles raw data based on the object storage gateway;
[0290] Step 1702: output data retrieval results based on the target original data.
[0291] Here, the function of directly extracting and retrieving the original data is provided. When the user needs to browse the first-principles original data, he can directly retrieve the matching target data from the object storage gateway for display by entering the search command in the user interface.
[0292] The above processing process provides the function of searching the original data, ensures the comprehensiveness of the search function, and meets the user's multi-faceted data retrieval needs for first principles data.
[0293] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.
[0294] Based on the same inventive concept, the embodiments of the present application further provide a data processing device. The data processing device provided in the embodiments of the present application can implement each process of the embodiments of the above-mentioned data processing method and can achieve the same technical effects. Therefore, the specific limitations in one or more data processing device embodiments provided below can refer to the limitations of the data processing method above. To avoid repetition, they are not repeated here.
[0295] In one embodiment, as shown in FIG18 , a data processing device 18 is provided, including: a data processing module 1801 and an information extraction module 1802 .
[0296] The data processing module 1801 is used to process the raw data based on the first principles to obtain structured first principles process data;
[0297] The information extraction module 1802 is configured to extract information from the first principle process data to obtain key data corresponding to the first principle original data; wherein the key data and the first principle process data have associated data identifiers.
[0298] In some embodiments, the apparatus further comprises:
[0299] Verification module, used for:
[0300] Obtaining an output file obtained based on a first-principles calculation, wherein the output file includes the first-principles raw data;
[0301] Detecting whether the output file meets a preset verification condition;
[0302] The step of processing the raw data based on the first principles to obtain structured first principles process data is performed when the output file meets a preset verification condition;
[0303] The preset verification conditions include: the presence of a complete end judgment symbol in the output file, and / or the data rationality of the first principle original data.
[0304] In some embodiments, the verification module is specifically configured to:
[0305] According to the set fields, content extraction is performed on the raw data of the first principle to obtain a structured field data matrix;
[0306] The field data matrix is determined as structured first-principles process data.
[0307] In some embodiments, the data processing module 1801 is specifically configured to:
[0308] According to the set fields, content extraction is performed on the raw data of the first principle to obtain a structured field data matrix;
[0309] Based on the field data matrix, according to the physical and chemical performance calculation formula, a structured physical and chemical performance data matrix is calculated;
[0310] The field data matrix and the physicochemical property data matrix are determined as structured first-principles process data.
[0311] In some embodiments, the information extraction module 1802 is specifically configured to:
[0312] First data for retrieval is extracted from the first principle process data, and the first data is determined as the key data corresponding to the first principle original data.
[0313] In some embodiments, the information extraction module 1802 is specifically configured to:
[0314] Extracting first data for retrieval from the first principle process data, and parsing the first principle process data according to a set calculation rule to obtain second data for retrieval;
[0315] The first data and the second data are determined as the key data corresponding to the first principle original data.
[0316] In some embodiments, the apparatus further comprises:
[0317] The storage module is used to compress and store the original data of the first principle in an object storage gateway.
[0318] In some embodiments, the storage module is further configured to:
[0319] The first principle process data is stored in a non-relational database, and the key data is stored in a search engine database; wherein the search engine database stores the data identifier.
[0320] In some embodiments, the apparatus further comprises:
[0321] Backup module for:
[0322] Backing up the key data in the search engine database to obtain backup data;
[0323] The backup data is stored in the non-relational database.
[0324] In some embodiments, the apparatus further comprises:
[0325] The retrieval module is used to respond to the data retrieval instruction and perform data retrieval based on the search engine database.
[0326] In some embodiments, the retrieval module is specifically configured to:
[0327] In response to a data retrieval instruction, extracting first target data from the key data based on the search engine database;
[0328] Based on the first target data, a data retrieval result is output.
[0329] In some embodiments, the retrieval module is specifically configured to:
[0330] In response to the data retrieval instruction, extracting second target data associated with the target data identifier from the first principle process data based on the search engine database;
[0331] Based on the second target data, a data retrieval result is output.
[0332] In some embodiments, the retrieval module is further configured to:
[0333] In response to a data retrieval instruction, extracting third target data including the target data identifier from the key data based on the search engine database;
[0334] Outputting a data retrieval result based on the second target data includes outputting a data retrieval result based on the second target data and the third target data.
[0335] In some embodiments, the retrieval module is specifically configured to:
[0336] generating a first data retrieval result based on the third target data, and outputting the first data retrieval result to a first display page;
[0337] generating a second data retrieval result corresponding to the second target data according to the structural features of the second target data;
[0338] The first data retrieval result and the second data retrieval result are output to the second display page.
[0339] In some embodiments, the retrieval module is specifically configured to:
[0340] generating a second data retrieval result corresponding to the second target data according to the structural features of the second target data;
[0341] Output the second data retrieval result to the second display page. In some embodiments, the retrieval module is more specifically configured to:
[0342] Get the drawing instructions triggered by the selected interactive drawing option;
[0343] In response to the drawing instruction, extracting drawing data from the second target data according to the structural features;
[0344] A data presentation image is generated as the second data retrieval result based on the drawing data.
[0345] In some embodiments, the apparatus further comprises:
[0346] An instruction determination module is used to:
[0347] Obtain data retrieval information based on first principles input;
[0348] determining the data retrieval instruction based on the data retrieval information;
[0349] In which, the data retrieval instruction includes: a first data retrieval instruction for performing data retrieval on the key data based on the data retrieval information, a second data retrieval instruction for performing data retrieval on the first principle process data based on the data retrieval information, or a third data retrieval instruction for performing data retrieval on the key data and the first principle process data based on the data retrieval information.
[0350] In some embodiments, the instruction determination module is specifically configured to:
[0351] The structural information, property information and calculation parameters included in the data retrieval information are encapsulated in a retrieval statement to obtain a data retrieval instruction carrying the encapsulated retrieval statement.
[0352] In some embodiments, the apparatus further comprises:
[0353] The first data updating module is configured to:
[0354] Detecting data consistency between the non-relational database and the search engine database according to set rules;
[0355] When it is determined that there is data inconsistency between the non-relational database and the search engine database, the process returns to the step of processing the original data based on first principles to obtain structured first principles process data.
[0356] In some embodiments, the apparatus further comprises:
[0357] The second data updating module is used to:
[0358] Get data adjustment instructions;
[0359] In response to the data adjustment instruction, after determining that the data adjustment is approved, adjust the data storage logic based on the data adjustment instruction;
[0360] Based on the adjusted data storage logic, return to the step of performing the first-principles-based raw data processing to obtain structured first-principles process data.
[0361] In some embodiments, the first principle raw data is stored in an object storage gateway; the retrieval module is further specifically configured to:
[0362] Extracting target original data from the first principle original data based on the object storage gateway;
[0363] Based on the target original data, a data retrieval result is output.
[0364] Each module in the above-mentioned data processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.
[0365] In one embodiment, as shown in FIG19 , a computer device is provided. The computer device 19 of this embodiment includes: at least one processor 1900 (only one is shown in FIG19 ), a memory 1901, and a computer program 1902 stored in the memory 1901 and executable on the at least one processor 1900. When the processor 1900 executes the computer program 1902, the steps of any of the above-described method embodiments are implemented.
[0366] The computer device 19 can be a computing device such as a desktop computer, laptop, PDA, or cloud server. The computer device 19 can include, but is not limited to, a processor 1900 and a memory 1901. Those skilled in the art will appreciate that FIG19 is merely an example of a computer device 19 and does not limit the computer device 19. The computer device 19 can include more or fewer components than shown, or a combination of certain components or different components. For example, the computer device can also include input and output devices, network access devices, buses, and the like.
[0367] The processor 1900 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0368] The memory 1901 may be an internal storage unit of the computer device 19, such as a hard disk or memory of the computer device 19. The memory 1901 may also be an external storage device of the computer device 19, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device 19. Furthermore, the memory 1901 may include both an internal storage unit of the computer device 19 and an external storage device. The memory 1901 is used to store the computer program and other programs and data required by the computer device. The memory 1901 may also be used to temporarily store data that has been output or is about to be output.
[0369] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0370] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.
[0371] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0372] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of the apparatus or unit, which can be electrical, mechanical or other forms.
[0373] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.
[0374] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0375] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0376] The present application implements all or part of the processes in the above-mentioned embodiment methods, and may also be implemented through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0377] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.
Claims
1. A data processing method, characterized in that: include: Based on the first-principles raw data processing, structured first-principles process data is obtained; Information extraction is performed on the first principle process data to obtain key data corresponding to the first principle original data; wherein the key data and the first principle process data have associated data identifiers.
2. The method according to claim 1, characterized in that Before the first-principles-based raw data processing is performed to obtain structured first-principles process data, the method further includes: Obtaining an output file obtained based on a first-principles calculation, wherein the output file includes the first-principles raw data; Detecting whether the output file meets a preset verification condition; The step of processing the raw data based on the first principles to obtain structured first principles process data is performed when the output file meets a preset verification condition; The preset verification conditions include: the presence of a complete end judgment symbol in the output file, and / or the data rationality of the first principle original data.
3. The method according to claim 1, characterized in that The first-principles-based raw data processing obtains structured first-principles process data, including: According to the set fields, content extraction is performed on the raw data of the first principle to obtain a structured field data matrix; The field data matrix is determined as structured first-principles process data.
4. The method according to claim 1, wherein The first-principles-based raw data processing obtains structured first-principles process data, including: According to the set fields, content extraction is performed on the raw data of the first principle to obtain a structured field data matrix; Based on the field data matrix, according to the physical and chemical performance calculation formula, a structured physical and chemical performance data matrix is calculated; The field data matrix and the physicochemical property data matrix are determined as structured first-principles process data.
5. The method according to claim 1, wherein The information extraction of the first principle process data to obtain key data corresponding to the first principle original data includes: First data for retrieval is extracted from the first principle process data, and the first data is determined as the key data corresponding to the first principle original data.
6. The method according to claim 1, characterized in that The information extraction of the first principle process data to obtain key data corresponding to the first principle original data includes: Extracting first data for retrieval from the first principle process data, and parsing the first principle process data according to a set calculation rule to obtain second data for retrieval; The first data and the second data are determined as the key data corresponding to the first principle original data.
7. The method according to any one of claims 1 to 6, characterized in that Before the first-principles-based raw data processing is performed to obtain structured first-principles process data, the method further includes: The first principle original data is compressed and stored in the object storage gateway.
8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The first principle process data is stored in a non-relational database, and the key data is stored in a search engine database; wherein the search engine database stores the data identifier.
9. The method according to claim 8, characterized in that After storing the key data in the search engine database, the method further includes: Backing up the key data in the search engine database to obtain backup data; The backup data is stored in the non-relational database.
10. The method according to claim 8, characterized in that After storing the key data in the search engine database, the method further includes: In response to the data retrieval instruction, data retrieval is performed based on the search engine database.
11. The method according to claim 10, characterized in that The responding data retrieval instruction and performing data retrieval based on the search engine database include: In response to a data retrieval instruction, extracting first target data from the key data based on the search engine database; Based on the first target data, a data retrieval result is output.
12. The method according to claim 10, characterized in that The responding data retrieval instruction and performing data retrieval based on the search engine database include: In response to the data retrieval instruction, extracting second target data associated with the target data identifier from the first principle process data based on the search engine database; Based on the second target data, a data retrieval result is output.
13. The method according to claim 12, characterized in that The responding data retrieval instruction and performing data retrieval based on the search engine database further includes: In response to a data retrieval instruction, extracting third target data including the target data identifier from the key data based on the search engine database; Outputting a data retrieval result based on the second target data includes outputting a data retrieval result based on the second target data and the third target data.
14. The method according to claim 13, characterized in that The outputting of data retrieval results based on the second target data and the third target data includes: generating a first data retrieval result based on the third target data, and outputting the first data retrieval result to a first display page; generating a second data retrieval result corresponding to the second target data according to the structural features of the second target data; The first data retrieval result and the second data retrieval result are output to the second display page.
15. The method according to claim 12, characterized in that The outputting of the data retrieval result based on the second target data includes: generating a second data retrieval result corresponding to the second target data according to the structural features of the second target data; The second data retrieval result is output to a second display page.
16. The method according to claim 14 or 15, characterized in that Generating a second data retrieval result corresponding to the second target data according to the structural features of the second target data includes: Get the drawing instructions triggered by the selected interactive drawing option; In response to the drawing instruction, extracting drawing data from the second target data according to the structural features; A data presentation image is generated as the second data retrieval result based on the drawing data.
17. The method according to claim 10, wherein The responding data retrieval instruction, before performing data retrieval based on the search engine database, further includes: Obtain data retrieval information based on first principles input; determining the data retrieval instruction based on the data retrieval information; In which, the data retrieval instruction includes: a first data retrieval instruction for performing data retrieval on the key data based on the data retrieval information, a second data retrieval instruction for performing data retrieval on the first principle process data based on the data retrieval information, or a third data retrieval instruction for performing data retrieval on the key data and the first principle process data based on the data retrieval information.
18. The method according to claim 17, characterized in that The determining the data retrieval instruction based on the data retrieval information includes: The structural information, property information and calculation parameters included in the data retrieval information are encapsulated in a retrieval statement to obtain a data retrieval instruction carrying the encapsulated retrieval statement.
19. The method according to claim 8, characterized in that After storing the first principle process data in a non-relational database and storing the key data in a search engine database, the method further includes: Detecting data consistency between the non-relational database and the search engine database according to set rules; When it is determined that there is data inconsistency between the non-relational database and the search engine database, the process returns to the step of processing the original data based on the first principles to obtain structured first principles process data.
20. The method according to claim 8, characterized in that After storing the first principle process data in a non-relational database and storing the key data in a search engine database, the method further includes: Get data adjustment instructions; In response to the data adjustment instruction, after determining that the data adjustment is approved, adjust the data storage logic based on the data adjustment instruction; Based on the adjusted data storage logic, return to the step of performing the first-principles-based raw data processing to obtain structured first-principles process data.
21. The method according to any one of claims 1 to 6, characterized in that The first principle raw data is stored in the object storage gateway; the method further includes: Extracting target original data from the first principle original data based on the object storage gateway; Based on the target original data, a data retrieval result is output.
22. A data processing device, characterized in that: include: A data processing module is used to process the raw data based on the first principles to obtain structured first principles process data; The information extraction module is used to extract information from the first principle process data to obtain key data corresponding to the first principle original data; wherein the key data and the first principle process data have associated data identifiers.
23. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 21 are implemented.
24. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 21 are implemented.
25. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 21 are implemented.
Citation Information
Patent Citations
Method and system applied to high-throughput integration calculation and data management of material gene engineering
CN106372329A
Automatic collecting and processing method and system for high-throughput material calculation data
CN107273670A
Method for obtaining material calculation data
CN110334055A
Data processing method and device
CN113626436A
Methods and apparatus to compensate first principle-based simulation models
US20110010138A1
