Data fidelity evaluation method and system

Abstract

The invention discloses a data fidelity evaluation method and system, and relates to the crossing field of material science and machine learning, and the method comprises the following steps: obtaining a multi-fidelity data set, and employing the data size contained in the data set with the minimum data size in the multi-fidelity data set; constructing and improving a prediction model, and performing uncertainty quantization on the data set by using the improved prediction model to obtain a prediction result; and sorting and analyzing the prediction results to complete data fidelity evaluation. According to the method, the effect of predicting the uncertainty can be achieved only by modifying the loss function of the prediction model, meanwhile, modification of the loss function is very flexible and convenient, a user can change the loss function according to own requirements, and the application possibility and convenience of the method are also improved.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary field of materials science and machine learning, and more specifically to a data fidelity evaluation method and system. Background Art

[0002] At present, machine learning has long been integrated into the field of materials, and machine learning models have made good progress in predicting material properties. Matbench, which is specifically used to benchmark machine learning models in the field of materials, has emerged. It displays and introduces excellent machine learning models for predicting material properties from all over the world, such as MEGNet and CGCNN. For data-driven machine learning models, data is undoubtedly important. In the field of materials, there are a wide variety of data sets with different quantities and chemical accuracies, namely multi-fidelity data. How to use these multi-fidelity data directly affects the final prediction accuracy of the model.

[0003] Unfortunately, even though multi-fidelity data is widely available in the field of materials, machine learning models for predicting material properties are rarely used in multi-fidelity learning methods, and the evaluation and ranking of multi-fidelity data sets are currently only performed on the basis of empirical theory or the data set itself, that is, the experimental data set is usually regarded as the highest fidelity data set, and the data set calculated by density functional theory (DFT) is regarded as the low fidelity, and the fidelity ranking of different DFT data sets is also obtained by comparing with the experimental data set (the more similar the distribution of the DFT data set is to the experimental data set, the higher the fidelity). This ranking method may have certain problems, that is, there is no quantitative analysis. For the above ranking method, when comparing the fidelity of data sets calculated by different DFT calculations, all available data sets are often used. The amount of existing data for different DFT data sets is also very different, and the final presentation of its fidelity ranking is often related to the "quality" and "quantity" of the data.

[0004] Uncertainty Quantification (UQ) has long been used to solve various real-world problems in science and engineering, including fields such as computer vision, image processing, medical image analysis, and natural language processing. For the prediction of material properties, it is useful to provide uncertainty quantification (UQ), which not only helps to evaluate the reliability of model prediction results, but also provides more information support for decision makers. In the process of material research and development and application, understanding the uncertainty of prediction results can help researchers better weigh risks and costs. However, there are currently few studies focusing on uncertainty in the field of materials. For example, the existing technologies use integrated neural networks, dropout neural networks, Bayesian neural networks, error / residual prediction networks, and Gaussian process regressors with neural network means (GP NN-μ )) and Convolution-Fed Gaussian Process (CFGP).) to predict uncertainty. The results show that the best effect is achieved by providing features for Gaussian process regression through convolutional neural networks. In the prior art, the graph neural network material property prediction models MEGNet and CGCNN were used to evaluate and compare four UQ methods: guided ensemble method, inductive conformance method, Delta metric method and evidential reasoning method. The results show that the more popular ensemble methods are not always the best choice for UQ in material property prediction. Although there have been a lot of studies on uncertainty quantification in science and engineering, it has not been widely discussed in materials science and related research on its application in multi-fidelity ranking has not yet appeared.

[0005] Therefore, how to provide a quantitative analysis multi-fidelity ranking evaluation method based on the uncertainty of the prediction results is an urgent problem that technicians in this field need to solve. Summary of the invention

[0006] In view of this, the traditional machine learning model for predicting properties in the material field cannot provide corresponding uncertainty for the predicted value. Considering the importance of uncertainty in the process of material property prediction and combining the previous research foundation on UQ in material property prediction, the present invention provides two different and easy-to-migrate and expand methods for machine learning models in the material field to obtain uncertainty, and evaluates the data fidelity on this basis. The invention items are as follows:

[0007] (1) Directly predict uncertainty by modifying the loss function of the prediction model.

[0008] (2) After extracting descriptors from the prediction model, GPR is used to train and predict uncertainty.

[0009] (3) Evaluate data fidelity using the two uncertainty quantification methods mentioned above.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] A data fidelity evaluation method comprises the following steps:

[0012] Acquire a multi-fidelity data set, and use the data amount contained in the data set with the smallest data amount among the multi-fidelity data sets;

[0013] Build and improve the prediction model, and use the improved prediction model to quantify the uncertainty of the data set to obtain the prediction results;

[0014] The prediction results are sorted and analyzed to complete the data fidelity evaluation.

[0015] Preferably, the specific process of obtaining the prediction result includes:

[0016] Modify the output layer dimension of the prediction model to n, and then train the model;

[0017] Modify the loss function in the prediction model code;

[0018] Call the prediction model to make predictions and output the prediction results.

[0019] Preferably, an activation function is added at the end of the output layer of the prediction model to ensure that the prediction uncertainty u is greater than 0.

[0020] Preferably, the process of sorting and analyzing the prediction results to complete the data fidelity evaluation includes:

[0021] The prediction interval coverage included in the forecast results is used to evaluate the fidelity of the data.

[0022] When the prediction interval coverage is less than or equal to 0, it means that the true value corresponding to the i-th data is included in the prediction range, and the fidelity evaluation result is reliable; otherwise, the fidelity evaluation result is unreliable;

[0023] After completing the reliability judgment of the prediction interval coverage rate, the prediction interval coverage rate is sorted to obtain a sorting result.

[0024] Preferably, the process of obtaining the uncertainty quantification result further includes:

[0025] After extracting descriptors from the prediction model, the Gaussian regression process is used to train and predict uncertainty, including:

[0026] Step 1: Build and train a prediction model;

[0027] Step 2: Extract the multi-fidelity data set, and obtain the descriptors of the prediction model corresponding to all data by directly predicting the descriptors at the corresponding positions of the prediction interval coverage rate of the uncertainty through the prediction model;

[0028] Step 3: The extracted descriptors are trained by Gaussian process regression (GPR);

[0029] Step 4: The original data to be predicted is first passed through the model trained in step 1 to obtain the corresponding descriptor, and then the obtained descriptor is input into the Gaussian process regression model trained in step 3, and the uncertainty is predicted by the Gaussian process regression model.

[0030] Preferably, the system of the data fidelity evaluation method comprises the following modules:

[0031] Acquisition module: used to acquire a multi-fidelity data set, using the data volume contained in the data set with the smallest data volume among the multi-fidelity data sets;

[0032] Model building module: used to build and improve prediction models;

[0033] Optimization module: using the improved prediction model to quantify uncertainty of the data set to obtain uncertainty quantification results;

[0034] Evaluation module: used to sort and analyze the uncertainty quantification results and complete data fidelity evaluation.

[0035] It can be known from the above technical solutions that, compared with the prior art, the present invention discloses a data fidelity evaluation method and system. The present invention adopts a more widely accepted and more frequently used prediction model to predict uncertainty. The effect of predicting uncertainty can be achieved only by modifying the loss function of the prediction model itself. At the same time, the modification of the loss function is very flexible and convenient, and the user can change it according to their own needs, which also improves the possibility and convenience of the application of the invention. In addition, with respect to the method of predicting uncertainty by extracting descriptors and combining them with Gaussian processes, the present invention also provides a combination of descriptors extracted by the prediction model and Gaussian processes, and gives the detailed descriptor extraction position of the prediction model used, which makes up for the vacancy of the combination of the prediction model and the Gaussian process in predicting uncertainty. Finally, combined with the uncertainty quantification method in the field of materials mentioned above, the present invention provides a distinctive multi-fidelity data evaluation method from the aspect of evaluating data fidelity through uncertainty quantification. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0037] Figure 1 A flow chart of the fidelity data evaluation method provided by the present invention;

[0038] Figure 2 A schematic diagram of the structure of a bandgap data set provided by an embodiment of the present invention;

[0039] Figure 3 A distribution diagram of the prediction interval coverage of uncertainty directly predicted by MEGNet for the PBE dataset provided in the embodiment of the present invention;

[0040] Figure 4 A distribution diagram of the prediction interval coverage rate of the HSE dataset provided by the embodiment of the present invention directly predicting uncertainty through MEGNet;

[0041] Figure 5 A distribution diagram of the prediction interval coverage of uncertainty of the GLLB dataset provided by the embodiment of the present invention directly predicted by MEGNet;

[0042] Figure 6 A distribution diagram of the prediction interval coverage rate of the SCAN dataset provided by the embodiment of the present invention directly predicting uncertainty through MEGNet;

[0043] Figure 7 A distribution diagram of the prediction interval coverage rate of the PBE dataset provided in the embodiment of the present invention directly predicting uncertainty through CGCNN;

[0044] Figure 8 A distribution diagram of the prediction interval coverage rate of the HSE dataset provided by the embodiment of the present invention directly predicting uncertainty through CGCNN;

[0045] Fig. 9 A distribution diagram of the prediction interval coverage rate of the GLLB dataset provided by the embodiment of the present invention directly predicting uncertainty through CGCNN;

[0046] Fig.10 A distribution diagram of the prediction interval coverage rate of the SCAN dataset provided in the embodiment of the present invention directly predicting uncertainty through CGCNN;

[0047] Fig.11 A schematic diagram of obtaining MEGNet and CGCNN descriptors provided in an embodiment of the present invention;

[0048] Fig.12 A schematic diagram of using GPR to train descriptors and predict uncertainty provided by an embodiment of the present invention;

[0049] Fig.13 A distribution diagram of the results obtained after the MEGNet provided in the embodiment of the present invention extracts the PBE descriptor and then trains it with GPR;

[0050] Fig.14 A distribution diagram of the results obtained after the MEGNet provided in the embodiment of the present invention extracts the HSE descriptor and then trains it with GPR;

[0051] Fig.15 A distribution diagram of the results obtained after GPR training after extracting GLLB descriptors from MEGNet provided in an embodiment of the present invention;

[0052] Fig.16 A distribution diagram of the results obtained after the MEGNet provided in the embodiment of the present invention extracts the SCAN descriptor and then trains it with GPR;

[0053] Fig.17 A distribution diagram of the results obtained after the CGCNN provided in the embodiment of the present invention extracts the PBE descriptor and undergoes GPR training;

[0054] Fig.18 A distribution diagram of the results obtained after the CGCNN provided in the embodiment of the present invention extracts the HSE descriptor and undergoes GPR training;

[0055] Fig.19 A distribution diagram of the results obtained after GPR training after extracting GLLB descriptors from CGCNN provided in an embodiment of the present invention;

[0056] Fig. 20 The result distribution diagram obtained after the CGCNN provided in the embodiment of the present invention extracts the SCAN descriptor and then undergoes GPR training. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] At present, the evaluation method for data fidelity is only performed on the data itself (for example, directly comparing the distribution of low-fidelity data and high-fidelity data for evaluation; the closer the distribution of the data is to the designed highest-fidelity data, the higher its fidelity). It does not involve the use of models to evaluate data fidelity. Unlike previous data fidelity evaluation methods, the present invention provides a method for evaluating data fidelity by quantifying the uncertainty of a prediction model. On this basis, two methods are used (modifying the loss function and extracting descriptors through Gaussian process regression mentioned below) to enable the model to predict uncertainty, and finally obtain uncertainty prediction results corresponding to multi-fidelity data sets. The prediction results include prediction interval coverage. By sorting the values ​​of these prediction interval coverage, the sorting results of data fidelity under different methods can be obtained.

[0059] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] The embodiment of the present invention discloses a data fidelity evaluation method, comprising the following steps: obtaining a multi-fidelity data set, wherein the multi-fidelity data set is the amount of data contained in a data set with the smallest amount of data among the multi-fidelity data sets;

[0061] Multi-fidelity datasets usually include the following types of data: 1) High-fidelity data: These data usually come from high-precision experimental measurements or high-fidelity computational models, and can provide accurate information, but the cost and time to obtain the data per unit time are high; 2) Low-fidelity data: These data usually come from simplified models, fast simulations, or rough measurements of experimental data, and can be obtained quickly, but have low accuracy, and the cost and time to obtain the data per unit time are low; 3) Medium-fidelity data: These data are between high-fidelity and low-fidelity, and the cost and time to obtain the data per unit time are medium.

[0062] Multi-fidelity datasets are very common in engineering fields, such as aerospace, mechanical engineering, materials, etc. Taking the aerospace field as an example, when designing an aircraft, high-fidelity data obtained by a computational fluid dynamics (CFD) model can be used to perform detailed aerodynamic analysis, and low-fidelity data obtained by an empirical model can be combined for rapid evaluation to optimize the design process and reduce development time. The present invention is applied to the field of materials science. There are a large number of multi-fidelity datasets in the field of materials science. For example, different results are obtained by calculating the properties of the same material (such as band gap values) using density functional theory (DFT) with different chemical accuracies, which creates multi-fidelity datasets in the field of materials. In addition, the field of materials can also obtain material properties through experiments, and the datasets obtained in this way are often considered to have higher fidelity (relative to DFT). The multi-fidelity dataset used in the present invention is a band gap dataset obtained by different density functional theories (including four different DFTs: PBE, HSE, GLLB and SCAN), and an experimental dataset (Exp) obtained from the ICSD database is used as a test set for evaluation.

[0063] Build and improve the prediction model, and use the improved prediction model to quantify the uncertainty of the data set to obtain the prediction results;

[0064] The prediction results are sorted and analyzed to complete the data fidelity evaluation. The specific process is as follows: Figure 1 shown.

[0065] Specifically, the prediction model used in the embodiment of the present invention is a material graph neural network model, more specifically, it can be MEGNet or CGCNN. The data set used should be a material structure data set, for example, the corresponding structure data set in matbench can be used. In theory, as long as MEGNet and CGCNN can be trained and predicted normally, the method of the embodiment of the present invention can be used. The following takes the data set used in the prior art as an example to introduce the implementation method. The experimental data set Exp is also used as the data set with the highest fidelity. Then, the fidelity of the four data sets PBE, HSE, GLLB and SCAN needs to be evaluated. The data set distribution is as follows: Figure 2 As shown in the figure, they are PBE (52348 samples), HSE (6030 samples), GLLB (2290 samples) and SCAN (472 samples). The experimental data set (Exp) in ICSD (Inorganic Crystal Structure Database) has a total of 2703 samples.

[0066] Since data fidelity needs to be evaluated quantitatively, the data volume used for evaluation is the data volume contained in the dataset with the smallest data volume in the current multi-fidelity datasets, that is, the number of samples 472 is used as the data volume of each dataset; for datasets with a total data volume greater than this value, random grouping and averaging are used as the final result; for example, the PBE dataset has 52,348 samples, which is more than 110 times the number of samples of the smallest dataset SCAN (472). At this time, 10 datasets with 472 samples in each group are randomly sampled from the PBE dataset for training, and finally the average of the results of each group of the 10 datasets is taken as the result of the PBE dataset; the same is true for the HSE and GLLB datasets; since the SCAN dataset is the smallest dataset, it does not need random sampling and can be trained directly; the Exp dataset is only used to evaluate the final test effect.

[0067] Furthermore, the specific process of obtaining the prediction results includes:

[0068] Modify the output layer dimension of the prediction model to n, the specific value can be 2, and then train the model;

[0069] Modify the loss function in the prediction model code, the formula is as follows:

[0070] Loss=(P true -P) 2 +(|P true -P|-u) 2

[0071] Among them, P true represents the true value, P and u represent the predicted value and its corresponding uncertainty, respectively, and |·| represents the absolute value operation;

[0072] Call the prediction model to make predictions and output the prediction results.

[0073] Furthermore, a softplus activation function is added at the end of the output layer of the prediction model to ensure that the prediction uncertainty u is greater than 0.

[0074] Furthermore, the process of sorting and analyzing the prediction results and completing the data fidelity evaluation includes:

[0075] The prediction interval coverage included in the prediction results is used to evaluate the fidelity of the data. The formula is as follows:

[0076] E i =(T i -U i )·(T i -L i )

[0077] Among them, E irepresents the prediction interval coverage of the i-th data, T i Indicates the true value corresponding to the i-th data, U i and L i Respectively represent the upper and lower bounds of the predicted value of the i-th data;

[0078] When the prediction interval coverage E i When it is less than or equal to 0, it means that the true value corresponding to the i-th data is included in the prediction range, and the fidelity evaluation result is reliable; otherwise, the fidelity evaluation result is unreliable;

[0079] After completing the reliability judgment of the prediction interval coverage rate, the prediction interval coverage rate is sorted to obtain a sorting result.

[0080] like Figure 3-6 As shown, the prediction interval coverage results of multi-fidelity datasets (PBE, HSE, GLLB, SCAN) are obtained by MEGNet by modifying the loss function to obtain uncertainty:

[0081] like Figure 3 As shown in Figure 2, the PBE dataset directly predicts the prediction interval coverage of uncertainty through MEGNet; 49.70% of the prediction interval coverage is less than or equal to 0, and 50.30% is greater than 0.

[0082] like Figure 4 As shown in the figure, the HSE dataset directly predicts the prediction interval coverage of uncertainty through MEGNet; 38.39% of the prediction interval coverage is less than or equal to 0, and 61.61% is greater than 0.

[0083] like Figure 5 As shown in Figure 2, the GLLB dataset directly predicts the uncertainty prediction interval coverage through MEGNet; 7.99% of the prediction interval coverage is less than or equal to 0, and 92.01% is greater than 0.

[0084] like Figure 6 As shown in the figure, the SCAN dataset directly predicts the prediction interval coverage of uncertainty through MEGNet; 12.94% of the prediction interval coverage is less than or equal to 0, and 87.06% is greater than 0.

[0085] From the uncertainty directly predicted by MENGet, it can be found that the final sorting order of the prediction interval coverage from large to small is: PBE>HSE>GLLB>SCAN.

[0086] like Figure 7-10As shown, the prediction results of the prediction interval coverage after the uncertainty is obtained by CGCNN by modifying the loss function for multi-fidelity datasets (PBE, HSE, GLLB, SCAN):

[0087] like Figure 7 As shown in the figure, the PBE dataset directly predicts the prediction interval coverage of uncertainty through CGCNN; 55.40% of the prediction interval coverage is less than or equal to 0, and 44.60% is greater than 0.

[0088] like Figure 8 As shown in the figure, the HSE dataset directly predicts the prediction interval coverage of uncertainty through CGCNN; 52.81% of the prediction interval coverage is less than or equal to 0, and 47.19% is greater than 0.

[0089] like Fig. 9 As shown in the figure, the GLLB dataset directly predicts the prediction interval coverage of uncertainty through CGCNN; 57.77% of the prediction interval coverage is less than or equal to 0, and 42.23% is greater than 0.

[0090] like Fig.10 As shown in the figure, the SCAN dataset directly predicts the prediction interval coverage of uncertainty through CGCNN; 50.52% of the prediction interval coverage is less than or equal to 0, and 49.48% is greater than 0.

[0091] From the uncertainty directly predicted by CGCNN, it can be found that the final sorting order of the prediction interval coverage from large to small is: GLLB>PBE>HSE>SCAN.

[0092] Furthermore, the process of obtaining uncertainty quantification results also includes:

[0093] After extracting descriptors from the prediction model, the Gaussian regression process is used to train and predict uncertainty, including:

[0094] Step 1: Build and train a prediction model, where the prediction model is a material graph neural network model, more specifically, MEGNet or CGCNN.

[0095] Step 2: Extract the multi-fidelity data set, and obtain the descriptors of the prediction model corresponding to all data by directly predicting the descriptors at the corresponding positions of the prediction interval coverage rate of the uncertainty through the prediction model;

[0096] Step 3: The extracted descriptors are trained by Gaussian process regression (GPR);

[0097] Step 4: The original data to be predicted is first passed through the model trained in step 1 to obtain the corresponding descriptor, and then the obtained descriptor is input into the Gaussian process regression model trained in step 3, and the uncertainty is predicted by the Gaussian process regression model.

[0098] Specifically, Figure 11-12 As shown, there are schematic diagrams of obtaining MEGNet and CGCNN descriptors and a schematic diagram of using GPR to train descriptors and predict uncertainty.

[0099] like Fig.11 As shown in the figure, the location where the descriptor is obtained is marked by the red dotted box in the figure; (a) shows the location where the MEGNet model obtains the descriptor, which is after the node feature (E), edge feature (V) and global state feature (u) are concatenated; (b) shows the location where the CGCNN model obtains the descriptor, which is after the node feature (E) and edge feature (V) pass through the L2 fully connected layer.

[0100] like Fig.12 As shown in the figure, different data sets are trained with MEGNet and CGCNN models to obtain corresponding descriptors, and then GPR is used to train and predict on the test set to obtain the predicted values ​​and corresponding uncertainties under different models for different data sets.

[0101] like Figure 13-16 As shown, the uncertainty prediction interval coverage results of multi-fidelity datasets (PBE, HSE, GLLB, SCAN) are obtained by extracting descriptors from MEGNet through GPR:

[0102] like Fig.13 As shown in Figure 1, the results obtained after MEGNet extracted the PBE descriptor and then trained with GPR; 87.28% of the prediction interval coverage is less than or equal to 0, and 12.72% is greater than 0.

[0103] like Fig.14 As shown in Figure 1, the results obtained after MEGNet extracted the HSE descriptor and then trained with GPR; 89.79% of the prediction interval coverage is less than or equal to 0, and 10.21% is greater than 0.

[0104] like Fig.15 As shown in Figure 1, the results obtained after MEGNet extracted the GLLB descriptor and then trained with GPR; 79.22% of the prediction interval coverage is less than or equal to 0, and 20.78% is greater than 0.

[0105] like Fig.16 As shown in Figure 1, the results obtained after MEGNet extracted the SCAN descriptor and then trained with GPR; 88.54% of the prediction interval coverage is less than or equal to 0, and 11.46% is greater than 0.

[0106] In summary, after sorting the prediction interval coverage obtained by GPR after extracting descriptors through MEGNet, the results of prediction interval coverage from large to small are: HSE>SCAN>PBE>GLLB.

[0107] like Figure 17-20 As shown in the figure, after extracting descriptors from CGCNN for multi-fidelity datasets (PBE, HSE, GLLB, SCAN), the uncertainty prediction interval coverage results are obtained through GPR:

[0108] like Fig.17 As shown in the figure, CGCNN extracts the PBE descriptor and obtains the result after GPR training; 90.53% of the prediction interval coverage is less than or equal to 0, and 9.47% is greater than 0.

[0109] like Fig.18 As shown in the figure, CGCNN extracts the HSE descriptor and obtains the result after GPR training; 92.31% of the prediction interval coverage is less than or equal to 0, and 7.69% is greater than 0.

[0110] like Fig.19 As shown in the figure, CGCNN extracts the GLLB descriptor and obtains the result after GPR training; 86.54% of the prediction interval coverage is less than or equal to 0, and 13.46% is greater than 0.

[0111] like Fig. 20 As shown in the figure, CGCNN extracts the SCAN descriptor and obtains the result after GPR training; 86.46% of the prediction interval coverage is less than or equal to 0, and 13.54% is greater than 0.

[0112] In summary, after the prediction interval coverage obtained by GPR after extracting the descriptor through CGCNN is sorted, the results of the prediction interval coverage from large to small are: HSE>PBE>GLLB>SCAN.

[0113] The method for evaluating data fidelity is obtained by sorting the prediction interval coverage of each data set. The higher the prediction interval coverage of the data set, the higher its fidelity. For example, in the results of the present invention, the prediction interval coverage of the four fidelity data sets PBE, HSE, GLLB, and SCAN obtained by GPR after extracting descriptors through MEGNet are 87.28%, 89.79%, 79.22%, and 88.54%, respectively. Then the order of data fidelity sorting from large to small is: HSE>SCAN>PBE>GLLB. In this way, the final evaluation result can be obtained, that is, the data set HSE has the highest fidelity among the four data sets, GLLB has the lowest fidelity, and the fidelity of the SCAN and PBE data sets is in the middle. The sorting result will vary depending on the model used.

[0114] The relationship between Gaussian process prediction uncertainty and graph neural network is:

[0115] Both graph neural networks and Gaussian process regression are designed to predict uncertainty. Then, the uncertainty plus or minus the predicted value can get the interval of the prediction result. Through this prediction interval, the percentage of the true value falling within the prediction interval (i.e., the prediction interval coverage) can be obtained. Ultimately, the size of this prediction interval coverage will determine the fidelity of the data set.

[0116] The original prediction model does not have the ability to directly predict uncertainty, so the embodiment of the present invention modifies it (corresponding to the method of modifying the loss) to enable it to acquire this ability; the Gaussian process itself has the ability to predict uncertainty, but it cannot directly process material data sets (input is structural type data), so the embodiment of the present invention also extracts descriptors from the prediction model (the descriptors can be used as input for Gaussian process regression, corresponding to another method mentioned in the embodiment of the present invention) to train the Gaussian process regression and enable it to also predict the uncertainty of the material data set.

[0117] In summary, the present invention improves the prediction model, and uses the improved prediction model to predict uncertainty. The effect of predicting uncertainty can be achieved by simply modifying the loss function of the prediction model itself. At the same time, the modification of the loss function is very flexible and convenient, and the user can change it according to their own needs, which also improves the possibility and convenience of the application of the invention. In addition, with respect to the method of predicting uncertainty by extracting descriptors and combining them with Gaussian processes, the present invention also provides a combination of descriptors extracted by the prediction model and Gaussian processes, and gives a detailed descriptor extraction position of the prediction model used, which fills the gap in the prediction of uncertainty by combining the prediction model with the Gaussian process. Finally, combined with the uncertainty quantification method in the field of materials mentioned above, the present invention provides a distinctive multi-fidelity data evaluation method from the aspect of uncertainty quantification evaluation of data fidelity.

[0118] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0119] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data fidelity evaluation method, characterized in that: The following steps are involved: Acquire a multi-fidelity data set, and use the data amount contained in the data set with the smallest data amount among the multi-fidelity data sets; Build and improve the prediction model, and use the improved prediction model to quantify the uncertainty of the data set to obtain the prediction results; The prediction results are sorted and analyzed to complete the data fidelity evaluation.

2. A data fidelity evaluation method according to claim 1, characterized in that: The specific process of obtaining the prediction results includes: Modify the output layer dimension of the prediction model to n, and then train the model; Modify the loss function in the prediction model code; Call the prediction model to make predictions and output the prediction results.

3. A data fidelity evaluation method according to claim 2, characterized in that: Add an activation function at the end of the prediction model output layer to ensure that the prediction uncertainty u is greater than 0.

4. A data fidelity evaluation method according to claim 1, characterized in that: The process of sorting and analyzing the prediction results and completing the data fidelity evaluation includes: The prediction interval coverage included in the forecast results is used to evaluate the fidelity of the data; When the prediction interval coverage is less than or equal to 0, it means that the true value corresponding to the i-th data is included in the prediction range, and the fidelity evaluation result is reliable; otherwise, the fidelity evaluation result is unreliable; After completing the reliability judgment of the prediction interval coverage rate, the prediction interval coverage rate is sorted to obtain a sorting result.

5. A data fidelity evaluation method according to claim 1, characterized in that: The process of obtaining uncertainty quantification results also includes: After extracting descriptors from the prediction model, the Gaussian regression process is used to train and predict uncertainty.

6. A data fidelity evaluation method according to claim 5, characterized in that: After extracting descriptors from the prediction model, the Gaussian regression process is used to train and predict uncertainty, including: Step 1: Build and train a prediction model; Step 2: Extract the multi-fidelity data set, and obtain the descriptors of the prediction model corresponding to all data by directly predicting the descriptors at the corresponding positions of the prediction interval coverage rate of the uncertainty through the prediction model; Step 3: The extracted descriptors are trained by Gaussian process regression (GPR); Step 4: The original data to be predicted is first passed through the model trained in step 1 to obtain the corresponding descriptor, and then the obtained descriptor is input into the Gaussian process regression model trained in step 3, and the uncertainty is predicted by the Gaussian process regression model.

7. A system using the data fidelity evaluation method according to any one of claims 1 to 6, characterized in that: Contains the following modules: Acquisition module: used to acquire a multi-fidelity data set, using the data volume contained in the data set with the smallest data volume among the multi-fidelity data sets; Model building module: used to build and improve prediction models; Optimization module: using the improved prediction model to quantify uncertainty of the data set to obtain uncertainty quantification results; Evaluation module: used to sort and analyze the uncertainty quantification results and complete data fidelity evaluation.

8. A computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the data fidelity evaluation method according to any one of claims 1 to 6.

Images