Method for quantifying uncertainty of satellite temperature data

Abstract

The invention discloses an uncertainty quantification method for satellite temperature data, and the method comprises the steps: training a plurality of deep learning models which are the same in structure and different in parameter initialization, so as to obtain prediction results of the plurality of models for the same input data; according to the prediction results of the multiple models, a calibration data set is constructed, and the calibration data set comprises an experience quantile and a prediction quantile; training a calibration model according to the calibration data set, and mapping a predicted quantile in the calibration data set to an experience quantile; inputting the initial quantile level value into a calibration model to obtain a calibrated quantile level value, and obtaining a prediction vector corresponding to the calibrated quantile level according to a prediction value of the deep learning model; and evaluating the calibrated uncertainty interval quality according to the predicted vector corresponding to the quantile level after calibration and the predicted vector corresponding to the initial quantile level. The method is based on a model integration method and is combined with the characteristics of a quantile calibration method to form a high-quality uncertainty calibration method.

Description

Technical Field

[0001] The present invention relates to the technical field of satellite engineering, and particularly relates to a method for quantifying the uncertainty of satellite temperature data. Background Art

[0002] In practical engineering, there are a large number of historical satellite temperature data. When analyzing these data, deep learning models are often used. However, due to factors such as the structure design and parameter initialization of the deep learning model, model uncertainty will be caused, resulting in uncertainty in the final satellite temperature prediction. How to effectively quantify and calibrate model uncertainty to improve the quality of uncertainty quantification is an urgent problem to be solved at present.

[0003] Traditional model uncertainty quantification methods usually include model ensemble, Bayesian neural network, etc. Model ensemble generally requires training multiple identical models with different initial parameters, and finally averaging all predicted values to obtain the final prediction result. This method is simple and easy to operate, applicable to any deep neural network model, and has been widely used. The Bayesian neural network method needs to calculate the posterior distribution of model parameters, but it is difficult to give an analytical distribution and usually requires an approximate method, which has a relatively low calculation efficiency and a complex calculation process. The above methods all consider the influence of model uncertainty on the predicted value and effectively improve the prediction accuracy.

[0004] The quality of uncertainty quantification refers to the goodness or badness of uncertainty quantification, that is, the quantified uncertainty interval should contain the real data with a certain probability as much as possible. To effectively improve the quality of uncertainty quantification, uncertainty calibration is a commonly used method and has been widely studied at present. Uncertainty calibration usually includes methods such as variance calibration, quantile calibration, and parameter scaling. Variance calibration and parameter scaling methods often require the model to directly output the mean and variance, so as to correct the variance. These methods have certain limitations on the structure of the model. The quantile calibration method is applicable to Bayesian neural networks and improves the quality of uncertainty quantification by correcting the quantiles predicted by the model.

[0005] In the existing methods, although the model ensemble method can effectively quantify model uncertainty, it does not consider the quality of uncertainty quantification. And although the quantile calibration method can effectively calibrate uncertainty and improve the uncertainty quality in Bayesian neural networks, it has not been applied to deep neural network models. Summary of the Invention

[0006] To solve some or all of the above technical problems existing in the prior art, the present invention provides a method for quantifying the uncertainty of satellite temperature data.

[0007] The technical solution of the present invention is as follows:

[0008] A method for quantifying the uncertainty of satellite temperature data is provided, and the method includes:

[0009] Training multiple deep learning models with the same structure but different parameter initializations to obtain the prediction results of multiple models for the same input data;

[0010] Constructing a calibration data set according to the prediction results of the multiple models, where the calibration data set includes empirical quantiles and predicted quantiles;

[0011] Training a calibration model according to the calibration data set to map the predicted quantiles in the calibration data set to empirical quantiles;

[0012] Inputting the initial quantile level value into the calibration model to obtain the calibrated quantile level value, and obtaining the prediction vector corresponding to the calibrated quantile level according to the prediction value of the deep learning model;

[0013] Evaluating the quality of the calibrated uncertainty interval according to the prediction vector corresponding to the calibrated quantile level and the prediction vector corresponding to the initial quantile level.

[0014] In an embodiment of the present invention, the deep learning model is a multi-layer perceptron network.

[0015] In an embodiment of the present invention, the deep learning model is a long short-term memory network.

[0016] In an embodiment of the present invention, after constructing multiple initial deep learning models, an appropriate batch size, learning rate, number of epochs, and optimization algorithm are selected, and the model parameters are continuously updated during the training process to obtain the trained deep learning model.

[0017] In an embodiment of the present invention, the training of the calibration model according to the calibration data set specifically includes:

[0018] Using the isotonic regression method to train the calibration data set, thereby constructing a monotonically increasing function as the calibration model.

[0019] In an embodiment of the present invention, the predicted quantile is calculated based on the comparison between the model prediction value and the true value, and the formula is as follows:

[0020]

[0021] In the formula, is the predicted quantile of the test sample x s , {(x s , y s )|s = 1,..., m} is the test set data, is the predicted value obtained by the test sample x s through multiple deep learning models, and N is the number of predicted values.

[0022] In one embodiment of the present invention, the empirical quantile is calculated based on the ranking of the predicted quantiles of all test samples, and the formula is as follows:

[0023]

[0024] In the formula, is the empirical quantile of the test sample x s , is the predicted quantile of all test samples, and m is the number of all test samples.

[0025] In one embodiment of the present invention, according to the predicted vector corresponding to the calibrated quantile level and the predicted vector corresponding to the initial quantile level, the interval coverage rate and the interval average width are calculated, and the calibration effect is evaluated according to the interval coverage rate and the interval average width. The interval coverage rate is used to measure the coverage ability of the prediction interval, and the interval average width is used to evaluate the compactness of the prediction interval.

[0026] In one embodiment of the present invention, the closer the interval coverage rate is to the confidence level and the smaller the interval average width is, the more reliable and accurate the prediction interval is.

[0027] The main advantages of the technical solution of the present invention are as follows:

[0028] The method for quantifying the uncertainty of satellite temperature data of the present invention combines the characteristics of the quantile calibration method and forms a high-quality method for calibrating the uncertainty of the satellite temperature prediction twin calculation model based on the model integration method, making up for the deficiencies of the existing uncertainty calibration methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0030] Figure 1 It is a flowchart of the method for quantifying the uncertainty of satellite temperature data according to an embodiment of the present invention;

[0031] Figure 2 It is an overall architecture diagram of the method for quantifying the uncertainty of satellite temperature data according to an embodiment of the present invention;

[0032] Figure 3 It is a flowchart of the basic model training and prediction process in the method for quantifying the uncertainty of satellite temperature data according to an embodiment of the present invention;

[0033] Figure 4 Flowchart of the training and calibration process of the calibration model in the method for quantifying the uncertainty of satellite temperature data according to an embodiment of the present invention. Detailed implementation manners

[0034] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0035] The technical solutions provided by the embodiments of the present invention will be described in detail below with reference to the drawings.

[0036] An embodiment of the present invention provides a method for quantifying the uncertainty of satellite temperature data. As shown in the attached Figure 1 figures, it includes:

[0037] S1. Train multiple deep learning models with the same structure but different parameter initializations to obtain the prediction results of multiple models for the same input data.

[0038] The method proposed by the present invention can be applied to any deep neural network. Therefore, the constructed network model can be determined according to actual needs, such as a multi-layer perceptron model, a long short-term memory network, etc. To construct multiple models with the same structure, we use the same network structure, but the parameter initialization values of each model are different. For example, we can use different random number seeds to initialize the weights and biases of each model. Next, we divide the existing satellite temperature data into a training data set and a test data set. The training data set is used to train the model, while the test data set is used to evaluate the model performance. Each model is trained using the training data set, and the model parameters are continuously updated through an optimization algorithm to enable it to better predict satellite temperature data. After training, the test data set is input into each trained model to obtain the prediction results of each model for the test data. For simplicity of expression, the deep neural network model trained in this step for temperature prediction is hereinafter referred to as the basic model.

[0039] S2. Construct a calibration data set according to the prediction results of multiple models. The calibration data set includes empirical quantiles and predicted quantiles.

[0040] For each test sample, we have the prediction values of multiple models for it. For example, for the test sample x s , we have N model prediction values . The prediction value It can be regarded as a sample following an unknown distribution, and based on these samples, model calibration can be further achieved.

[0041] According to the set of predicted values, we can obtain the empirical quantile and the predicted quantile. The predicted quantile represents the value at a specific percentile position among all model predicted values. The empirical quantile represents the proportion of samples whose predicted quantiles of the target test samples are the same as or less than this value. Combine the predicted quantile and the empirical quantile of each test sample to construct a calibration dataset.

[0042] S3. Train a calibration model according to the calibration dataset, and map the predicted quantiles in the calibration dataset to the empirical quantiles.

[0043] After obtaining the calibration dataset, train the calibration model. The calibration model can map the predicted quantiles in the calibration dataset to the empirical quantiles. That is, after obtaining the calibration model, for any given quantile level τ, after inputting it into the calibration model, the calibrated quantile level can be obtained. Thus, the predicted values at the corresponding quantile levels are calibrated.

[0044] S4. Input the initial quantile level value into the calibration model to obtain the calibrated quantile level value, and obtain the prediction vector corresponding to the calibrated quantile level according to the predicted values of the deep learning model.

[0045] To visually compare the calibrated and uncalibrated prediction results, we can obtain the prediction vectors for a given series of quantile levels, and the prediction vectors corresponding to the quantile levels after passing through the calibration model for the given quantile levels.

[0046] Specifically, we can specify an initial quantile level value, such as 0.95, which represents that we hope the prediction result contains the 95% confidence interval of the true value. Input the initial quantile level value into the calibration model to obtain the calibrated quantile level value. According to the calibrated quantile level value and the predicted values of each model, we can calculate the predicted values corresponding to the calibrated quantile level value for each model. Combine the predicted values corresponding to the calibrated quantile level value for all models to obtain the calibrated prediction vector.

[0047] S5. Evaluate the quality of the calibrated uncertainty interval according to the prediction vector corresponding to the calibrated quantile level and the prediction vector corresponding to the initial quantile level.

[0048] We can calculate the interval coverage rate and the interval average width of the calibrated prediction vector, and compare them with the interval coverage rate and the interval average width of the initial prediction vector. If the interval coverage rate of the calibrated prediction vector is closer to the target confidence level and the interval average width is smaller, it indicates that the calibration effect is good.

[0049] In summary, the uncertainty quantification method for satellite temperature data provided by the embodiments of the present invention is based on the model integration method and combines the characteristics of the quantile calibration method to form a high-quality uncertainty calibration method for the satellite temperature prediction twin calculation model, making up for the deficiencies of existing uncertainty calibration methods.

[0050] The following details each step and the related principles in the uncertainty quantification method for satellite temperature data provided by the embodiments of the present invention.

[0051] The overall architecture of the uncertainty quantification method for satellite temperature data proposed in the embodiments of the present invention is as Figure 2 shown. It mainly includes three parts. First is the basic model training and prediction part, where multiple identical models are constructed with different initial parameters, and the existing temperature data is used for training and prediction. Second is the post-training calibration part, where a calibration dataset is constructed based on the prediction results and an isotonic regression method is used to train the calibration model. Finally is the uncertainty quality assessment part, which compares the uncertainty quantification results before and after calibration.

[0052] I. Basic Model Training and Prediction

[0053] The basic model in the embodiments of the present invention is used to complete satellite temperature prediction. The basic model can be any deep neural network model that can complete the satellite temperature prediction task. Therefore, the constructed network model can be determined according to actual needs, such as a multi-layer perceptron model (MLP), a long short-term memory network (LSTM), etc. First, the existing satellite temperature data is divided into a training dataset and a test dataset. The training dataset is defined as {(x i , y i ) | i = 1,..., n}, and the test dataset is defined as {(x s , y s ) | s = 1,..., m}. The basic model training process is similar to traditional deep learning methods. After constructing the model structure, appropriate parameters such as batch size, learning rate, number of epochs, and optimization algorithm are selected, and the model parameters are continuously updated during the training process to finally obtain the optimal parameter values. Based on the model integration method, the present invention needs to train N identical models, but the initial parameters of each model are randomly generated. After all models are trained, the test set is input into each trained model to obtain the predicted values of the N models. For each test sample x s , its predicted value is When N is large enough, the predicted value can be regarded as a sample obeying an unknown distribution, and model calibration can be further achieved based on these samples. The flowchart of the basic model training and prediction process is as Figure 3 shown.

[0054] II. Post-Training Calibration

[0055] After obtaining the predicted values from all models, a calibration dataset is constructed based on the predicted values. Based on the quantile calibration method, the calibration dataset generally consists of two parts: the empirical quantile and the predicted quantile. Taking the test sample x s as an example, its corresponding predicted value is The predicted quantile is calculated as follows:

[0056]

[0057] is the predicted quantile corresponding to the target quantile level q of the test sample x s , is the predicted value of the j-th model for the predicted sample x s , and N is the number of base models or predicted values. is the indicator function:

[0058]

[0059] The purpose of this formula is to calculate the proportion of the predicted value distribution of the given test sample x s that is below the quantile level q. The calculated is the predicted quantile of the quantile q, indicating the proportion of the predicted values in the N models that are less than or equal to

[0060] The empirical quantile needs to be calculated based on the predicted values of all test samples, that is, calculate the predicted quantiles of other test samples and compare them with the predicted quantile s of the test sample x . The calculation formula is as follows:

[0061]

[0062] In the formula, is the empirical quantile of the test sample x s , is the predicted quantile of all test samples, and m is the number of all test samples.

[0063] is the empirical quantile calculated based on , reflecting the relative position in the overall predicted quantile distribution of the test samples. If is large (close to 1), it means that the predicted values of this sample are mostly above the quantile q, and the empirical quantile will also be large.

[0064] Therefore, the constructed calibration dataset is The closer the empirical quantile and the predicted quantile are, the higher the quality of uncertainty.

[0065] If the model prediction is completely accurate, then and should show a linear relationship (theoretically a diagonal relationship). If the data set is sorted, it is expected that the empirical quantile and the predicted quantile approximately show a linear relationship, that is However, in actual situations, the two often do not match exactly and show a disordered fluctuation relationship. Therefore, in order to improve the quality of uncertainty, the isotonic regression method is used to train the calibration data set, and by adjusting the predicted quantile to make it closer to the empirical quantile thus constructing a monotonically increasing function. The isotonic regression method belongs to a type of regression algorithm, and its form is similar to the conventional algorithm. Those skilled in the art can refer to relevant published literature for the specific algorithm, and it will not be elaborated in the embodiments of the present invention. After obtaining the calibration model, for any given quantile level τ, after inputting it into the calibration model, the calibrated quantile level can be obtained, Figure 4 thus calibrating the predicted value corresponding to the quantile level. The complete flow chart of post-training calibration is as

[0066] III. Uncertainty Quality Assessment

[0067] According to the calibration model, the calibrated quantile corresponding to any quantile level can be obtained. To visually compare the calibrated and uncalibrated prediction results, it can be calculated in the following two cases:

[0068] (1) Uncalibrated result: Given a series of quantile level values τ 1 , τ 2 ,...., τ T , directly according to the predicted value of each test set sample According to the distribution of the predicted values, the predicted vector corresponding to each quantile level is obtained

[0069]

[0070] (2) Calibrated result: Given a series of quantile level values τ 1 , τ 2 ,...., τ T , after inputting it into the calibration model, the calibrated quantile level value is obtained. According to the predicted value of each test set sample Similarly, according to the distribution of the predicted values, the predicted vector corresponding to each calibrated quantile level is obtained

[0071]

[0072] According to the relevant definitions of Prediction Interval Coverage Probability (PICP) and Mean Prediction Interval Width (MPIW), the predicted values at the corresponding quantile levels need to be further calculated to obtain the prediction interval, that is, given the quantile level value τ 1 ', τ 2 ',...., τ T '(by default, τ 1 ' ≥ τ 1 , τ 2 ' ≥ τ 2 ,...., τ T ' ≥ τ T ), repeating the above two cases, we get and Therefore, the interval coverage rate and the mean interval width are calculated as follows:

[0073]

[0074]

[0075]

[0076] Through the above formula, the PICP and MPIW corresponding to the confidence level of τ 1 ' - τ 1 , τ 2 ' - τ 2 ,...., τ T ' - τ T can be obtained, where m represents the number of test samples, represents the lower bound of the confidence interval, represents the upper bound of the confidence interval, and y s is the true value of the prediction sample. PICP represents the proportion of the actual value y s falling within the prediction interval among the test samples.

[0077] The ideal PICP should be close to the specified confidence level, such as 95% or 99%. If the actual PICP is less than the confidence level, it means that the prediction interval is not sufficient to contain the true value, which may lead to low credibility; if the PICP is much higher than the confidence level, it means that the interval may be too conservative.

[0078] MPIW represents the average of the prediction interval widths of all test samples. The narrower the width, the higher the accuracy of the prediction result, but it may also reduce the probability of the true value falling into the interval (affecting PICP). On the premise of ensuring a sufficiently high PICP, the smaller the MPIW, the better, which reflects the compactness of the prediction interval.

[0079] PICP and MPIW are two core metrics for the quality of uncertainty quantification: PICP measures the coverage ability (accuracy) of the interval. MPIW measures the compactness (precision) of the interval. Ideally, PICP is close to the target confidence level and MPIW is small enough, indicating that the prediction interval is both reliable and precise.

[0080] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. In addition, in this article, "front", "rear", "left", "right", "upper" and "lower" are referenced based on the placement state shown in the drawings.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for quantifying uncertainty in satellite temperature data, characterized in that: include: Train multiple deep learning models with the same structure but different parameter initialization to obtain prediction results of multiple models for the same input data; Constructing a calibration data set according to the prediction results of the multiple models, wherein the calibration data set includes empirical quantiles and predicted quantiles; Training a calibration model based on a calibration data set, mapping predicted quantiles in the calibration data set to empirical quantiles; Inputting the initial quantile level value into the calibration model to obtain the calibrated quantile level value, and obtaining the prediction vector corresponding to the calibrated quantile level according to the prediction value of the deep learning model; The quality of the calibrated uncertainty intervals is assessed based on the prediction vectors corresponding to the calibrated quantile levels and the prediction vectors corresponding to the initial quantile levels.

2. The uncertainty quantification method of satellite temperature data according to claim 1, characterized in that: The deep learning model is a multi-layer perceptron network.

3. The uncertainty quantification method of satellite temperature data according to claim 1, characterized in that: The deep learning model is a long short-term memory network.

4. The uncertainty quantification method of satellite temperature data according to claim 1, characterized in that: After building multiple initial deep learning models, select the appropriate batch size, learning rate, iteration cycle, and optimization algorithm, and continuously update the model parameters during the training process to obtain a trained deep learning model.

5. The uncertainty quantification method of satellite temperature data according to claim 1, characterized in that: The training of the calibration model according to the calibration data set specifically includes: The calibration data set is trained using the rank-preserving regression method to construct a monotonically increasing function as the calibration model.

6. The uncertainty quantification method of satellite temperature data according to claim 1, characterized in that: The prediction quantile is calculated based on the comparison between the model prediction value and the true value, and the formula is as follows: In the formula, For the test sample x s The predicted quantile of {(x s ,y s )|s=1,...,m} is the test set data, For the test sample x s The predicted values ​​obtained by multiple deep learning models, N is the number of predicted values.

7. The uncertainty quantification method of satellite temperature data according to claim 6, characterized in that: The empirical quantile is calculated based on the ranking of the predicted quantiles of all test samples, and the formula is as follows: In the formula, For the test sample x s The empirical quantile of is the predicted quantile of all test samples, and m is the number of all test samples.

8. The uncertainty quantification method of satellite temperature data according to claim 7, characterized in that: According to the prediction vector corresponding to the calibrated quantile level and the prediction vector corresponding to the initial quantile level, the interval coverage and the average interval width are calculated, and the calibration effect is evaluated based on the interval coverage and the average interval width. The interval coverage is used to measure the coverage ability of the prediction interval, and the average interval width is used to evaluate the compactness of the prediction interval.

9. The uncertainty quantification method of satellite temperature data according to claim 8, characterized in that: The closer the interval coverage is to the confidence level, the smaller the average width of the interval is, indicating that the prediction interval is both reliable and accurate.

Images