Online filling method for missing values ​​of Antarctic seeing

Abstract

An embodiment of the present application provides a method for online filling of missing values ​​of Antarctic seeing, the method comprising: obtaining historical seeing data and meteorological historical data of an observation site; using the Spearman rank correlation coefficient method to screen out target meteorological data with high correlation with seeing; using the meteorological historical data as input and the seeing historical data as output, establishing training sample data of the observation site, and using a standard scoring method to standardize the training sample data; based on the training sample data, selecting a rational quadratic kernel to calculate the covariance between sample input variables; using a Gaussian process regression method to establish a regression model between seeing and meteorological data; when seeing monitoring is interrupted, applying the regression model to fill in missing values ​​for the target site in real time; using the latest seeing and meteorological data to update the regression model on a weekly basis to avoid a decrease in model accuracy.

Description

Technical Field

[0001] The present application relates to the fields of astronomy and data analysis, and in particular to a method for online filling of missing values ​​of Antarctic seeing. Background Art

[0002] The Dome A region in Antarctica has the characteristics of continuous darkness, good free-atmosphere seeing, and a low atmospheric boundary layer, which are very suitable for observations by optical infrared and radio telescopes. Currently, many astronomical observation equipment has been installed and operated in the region. Among all the site parameters, the excellent seeing conditions in the Antarctic region are the most important factor that attracts astronomers.

[0003] Seeing is an important parameter used to characterize the degree of influence of atmospheric turbulence on the resolution of the telescope. It has a decisive influence on high-precision imaging and observation of dark sources. Therefore, in order to make full use of the excellent seeing conditions in the Antarctic region, it is necessary to monitor the seeing in real time and adjust the telescope's observation plan according to the monitoring results to avoid missing the best observation window and improve the scientific output of the telescope. Seeing is usually measured by a differential image motion monitor (DIMM), which consists of an optical device, a rack that can be pointed and tracked, and a control computer. In order to complete the seeing observation, it is necessary to continuously track and observe the designated stars.

[0004] However, in actual operation, the extreme environment of Antarctica will seriously affect the operation of DIMM equipment. The summer temperature in Dome A is -25℃ to -40℃, and the winter temperature reaches -55℃ to -80℃. The low temperature environment increases the load torque of the DIMM rack transmission system, and in extreme cases it may even get stuck, resulting in interruption of seeing observation. On the other hand, low temperature and temperature fluctuations can cause frost on the mirror surface of DIMM's optical equipment, which will reduce the light transmittance and eventually cause the monitored seeing value to be lost.

[0005] In order to make full use of the observation time of astronomical equipment and improve scientific output, missing values ​​of seeing must be filled. In the prior art, commonly used missing value filling methods mainly include: (1) adjacent filling method: fill the current missing value based on the data of the nearest time point before / after the missing value; (2) moving average method: take the value in a time window before the missing value occurs, calculate its average value or / median to fill the missing value; (3) interpolation method: fill the missing value by satisfying a certain fitting relationship through the neighboring points of the missing value. However, the above methods are only suitable for filling a small amount of data. When there are a large number of continuous missing values ​​in the data, the use of the above filling methods will cause large errors. In addition, the above methods are all offline filling and cannot fill the missing values ​​online in real time.

[0006] Based on the above technical requirements, a method for filling missing values ​​of seeing in real time and online is urgently needed. The present invention uses meteorological monitoring data installed near astronomical observation equipment to fill missing values ​​of seeing in real time. Mainly compared with seeing monitoring, meteorological data monitoring has no moving parts and optical equipment, so the equipment reliability is high and the data is continuous. Summary of the invention

[0007] The embodiment of the present application provides a method for online filling of missing values ​​of Antarctic seeing, so as to solve the problem of missing seeing values ​​of the seeing monitoring device DIMM caused by the extreme environment of Antarctica in the related art.

[0008] In one embodiment of the present application, a method for online filling of missing values ​​of Antarctic seeing is proposed, the method comprising:

[0009] Obtain historical seeing data and meteorological data of the observation site;

[0010] Analyze the historical seeing data and the historical meteorological data, and use the Spearman rank correlation coefficient method to select target meteorological data with high correlation with the seeing;

[0011] Taking the target meteorological data as input and the historical seeing data as output, establishing training sample data of the observation site, and using the z-score standard scoring method to standardize the training sample data;

[0012] Based on the training sample data, selecting a rational quadratic kernel to calculate the covariance between input variables of the training sample data;

[0013] Establishing the negative logarithm of the conditional probability of the training sample data as the likelihood function, taking the minimum value of the likelihood function as the objective function, and using the quasi-Newton method as the optimization algorithm to obtain a Gaussian process regression model between seeing and meteorological data;

[0014] Apply the regression model to fill missing values ​​of real-time seeing at the target site, including:

[0015] Determine whether the target seeing can be collected normally. When the target seeing data value collected at a certain moment is 0, it is considered that the target seeing collection is interrupted. At this time, the target meteorological data at the current moment is obtained, and after standardization, it is input into the regression model to calculate the predicted value;

[0016] The predicted values ​​output by the regression model are de-standardized to obtain fill values ​​for missing values ​​of the target seeing.

[0017] In one embodiment, the regression model is further updated according to the seasonality and periodicity of the meteorological data, wherein the update uses the hyperparameters in the current regression model as initialization values, uses the new seeing data and new meteorological data collected at the observation station in the current week as new sample data, and repeats the following steps:

[0018] Taking the new meteorological data as input and the new seeing data as output, establishing new training sample data of the observation site, and using the z-score standard scoring method to perform standardization processing on the new training sample data;

[0019] Based on the new training sample data, a rational quadratic kernel is selected to calculate the covariance between sample input variables;

[0020] The negative logarithm of the conditional probability of the new training sample data is established as the likelihood function, the minimum value of the likelihood function is taken as the objective function, and the optimization algorithm is the quasi-Newton method to obtain a trained new regression model.

[0021] In one embodiment, the Spearman rank correlation coefficient method p s The formula is:

[0022]

[0023] Where d is the difference in rank between the two columns and n is the length of each column.

[0024] In one embodiment, the z-score standard scoring method calculation formula is:

[0025]

[0026] In the formula, x and are the data before and after standardization, u and σ are the mean and variance of the samples, respectively.

[0027] In one embodiment, the rational quadratic kernel calculation formula is:

[0028]

[0029] In the formula, For x i and x j The Euclidean distance of f ,α,l] T .

[0030] In one embodiment, the calculation formula of the likelihood function is:

[0031]

[0032] In the formula,

[0033] In one embodiment, the Gaussian process regression model predicts The calculation formula is as follows:

[0034]

[0035] In the formula, K and K * are all shorthand forms of the covariance matrix, where K * =K(x * ,X) is the training data input X and the current input x * The covariance matrix of , K = K (X, X) is the autocovariance matrix of the training data input X.

[0036] In one embodiment, the denormalization formula is:

[0037]

[0038] In the formula, y pred This is the value used to fill in the missing seeing value.

[0039] In one embodiment, the meteorological historical data includes temperature, temperature difference, wind speed, wind difference, wind direction, air pressure and relative humidity.

[0040] Through the embodiments of the present application, meteorological data can be used to fill in missing values ​​of seeing online, thereby improving the accuracy of seeing filling, and the model has the ability to update online: historical meteorological and seeing data of the observation site are obtained, and meteorological data with high correlation with the seeing data are screened out for model training based on the Spearman rank correlation coefficient method, so as to avoid data redundancy that may cause model overfitting and affect prediction accuracy; an online filling method for missing values ​​of Antarctic seeing based on a Gaussian process regression model is established to realize real-time filling of missing seeing; the latest meteorological data is used to update the model on a weekly basis to avoid a decrease in model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 is a schematic diagram of the overall process according to an embodiment of the present application;

[0043] Figure 2 is a flow chart of a method for online filling of missing seeing values ​​according to an embodiment of the present application;

[0044] Figure 3is a graph of seeing data changes according to an embodiment of the present application;

[0045] Figure 4 is a wind speed data variation diagram according to an embodiment of the present application;

[0046] Figure 5 is a comparison chart of test results according to an embodiment of the present application;

[0047] Figure 6 4 is a diagram for filling missing seeing values ​​according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] Specifically, Figure 1 As shown, a method for online filling of missing values ​​of Antarctic seeing is proposed, and the method includes:

[0050] S202, obtaining historical seeing data and historical meteorological data of the observation site;

[0051] S204, analyzing the historical seeing data and the historical meteorological data, and using the Spearman rank correlation coefficient method to select target meteorological data with a high correlation with the seeing;

[0052] S206, using the target meteorological data as input and the historical seeing data as output, establishing training sample data of the observation site, and using the z-score standard scoring method to perform standardization processing on the training sample data;

[0053] S208, based on the training sample data, selecting a rational quadratic kernel to calculate the covariance between the training sample input variables;

[0054] S210, establishing a negative logarithm of the conditional probability of the training sample data as a likelihood function, taking the minimum value of the likelihood function as the objective function, and using the quasi-Newton method as the optimization algorithm to obtain a Gaussian process regression model between seeing and meteorological data;

[0055] S212, applying the regression model to the target site to perform real-time missing value filling; judging whether the target seeing can be collected normally, when the target seeing data value collected at a certain moment is 0, it is considered that the target seeing collection is interrupted at this time, and the target meteorological data at the current moment is obtained, and after standardization, it is input into the regression model to calculate the predicted value;

[0056] S214, performing a denormalization process on the predicted value output by the regression model to obtain a filling value for the missing value of the target seeing.

[0057] In one embodiment, the method further includes:

[0058] S1, updating the regression model according to the seasonality and periodicity of the meteorological data, the update uses the hyperparameters in the current regression model as initialization values, and uses the new seeing data and new meteorological data collected at the observation station in the current week as new sample data, and repeats the following steps:

[0059] S1.1 takes the new meteorological historical data as input and the new seeing data as output, establishes new training sample data of the observation site, and uses the z-score standardization method to standardize the new training sample data;

[0060] S1.2 Based on the new training sample, a rational quadratic kernel is selected to calculate the covariance between sample input variables;

[0061] S1.3 Establish the negative logarithm of the conditional probability of the new training sample data as the likelihood function, take the minimum value of the likelihood function as the objective function, and use the quasi-Newton method as the optimization algorithm to obtain a trained new regression model.

[0062] In one embodiment, the above method further includes: the Spearman rank correlation coefficient method ρ s The formula is:

[0063]

[0064] Where d is the difference in rank between the two columns and n is the length of each column.

[0065] In one embodiment, the z-score standard scoring method calculation formula is:

[0066]

[0067] In the formula, x and are the data before and after standardization, u and σ are the mean and variance of the samples, respectively.

[0068] In one embodiment, the rational quadratic kernel calculation formula is:

[0069]

[0070] In the formula, For x i and x j The Euclidean distance of f ,α,l] T .

[0071] In one embodiment, the calculation formula of the likelihood function is:

[0072]

[0073] In the formula,

[0074] It should be noted that after the above steps, the partial derivative of the obtained likelihood function with respect to the hyperparameters is obtained:

[0075]

[0076] In the formula, tr() means finding the trace of a matrix.

[0077] Finally, the quasi-Newton optimization algorithm is used to find the minimum point of the likelihood function in the above formula, and the optimal hyperparameters of the hyperparameters are calculated. The calculation process of the quasi-Newton optimization algorithm is as follows:

[0078] 1) Given an initial value θ (0) , accuracy threshold ε, and let H0=I, k=0.

[0079] 2) Determine the search direction

[0080] 3) Order s k =λ k d k ,θ k+1 :=θ k +s k .

[0081] 4) If || g k+1 ||<ε, the algorithm ends.

[0082] 5) Calculate y k =g k+1 -g k .

[0083] 6) Calculate H k+1 :

[0084] 7) Let k:=k+1, go to step 2).

[0085] In one embodiment, the trained Gaussian process regression model is used for real-time monitoring of the site: first, it is determined whether the seeing can be collected normally. When the seeing data value collected at a certain moment is 0, it is considered that the seeing collection is interrupted at this time; then the meteorological data at the current moment is obtained, and the meteorological data is standardized into x * , x * Input into the trained Gaussian process regression model, and the predicted value of the Gaussian process regression model can be calculated by the following formula:

[0086] The Gaussian process regression model predicts the value The calculation formula is as follows:

[0087]

[0088] In the formula, K and K * are all shorthand forms of the covariance matrix, where K * =K(x * ,X) is the training data input X and the current input x * The covariance matrix of , K = K (X, X) is the autocovariance matrix of the training data input X.

[0089] In one embodiment, the denormalization formula is:

[0090]

[0091] In the formula, y pred This is the value used to fill in the missing seeing value.

[0092] like Figures 2 to 6 As shown, the following specific example is used to explain:

[0093] The method includes three parts: Gaussian process regression model training, online filling of missing seeing values ​​and online model updating. The first part is to analyze the correlation between historical meteorological data and seeing data of a certain observation station, and select meteorological data with high correlation for Gaussian process regression model training. The second part is to judge whether the seeing real-time collection system is interrupted. If the collection is interrupted at a certain moment, the meteorological data collected at the current moment is standardized and used as input parameters to input into the trained Gaussian process regression model to obtain the predicted value. The predicted value is de-standardized and used as the filling value of the missing value of the seeing time series. At the same time, in order to ensure the prediction accuracy of the Gaussian process regression model, the Gaussian process regression model is updated online every week using the new data collected in the current week.

[0094] In order to verify the effectiveness and rationality of the Gaussian process regression model for seeing filling, the meteorological data and seeing data of the Antarctic DomeA observation point were used for analysis and verification. The specific steps are as follows:

[0095] (1) Obtain historical values ​​of meteorological data and seeing data

[0096] The meteorological data and seeing data used in this embodiment are respectively derived from the multi-layer automatic weather station and DIMM instrument installed in Dome A in Antarctica. By integrating the meteorological data and seeing data collected in 2019, a total of nearly one month of meteorological data and seeing data synchronization data are obtained, from January 23 to March 7, 2019, with 7336 data samples. The multi-layer automatic weather station consists of a 15m high tower, with 10 temperature sensors installed from -1m (i.e. below the ice surface) to 14m high, 7 wind speed / direction sensors installed from 2m to 14m high, and a pressure sensor and a relative humidity sensor installed at 2m high. Figure 3 is the seeing change diagram, Figure 4 The wind speed data at a height of 2 m are shown, with the missing values ​​of the seeing data marked with a black dotted box.

[0097] (2) Calculation of correlation between meteorological data and seeing data

[0098] The correlation coefficients between different meteorological parameters and seeing were calculated by Spearman rank correlation to eliminate irrelevant or redundant input features. The calculation results are shown in Table 1, where ρ represents the correlation coefficient between a certain meteorological parameter and seeing, T h , W h 、Wd h They represent the temperature, wind speed and wind direction at an altitude of hm, respectively, and Ap and Rh represent the air pressure and humidity at an altitude of 2m, respectively. Finally, the four most relevant meteorological parameters are selected as input for the training of the Gaussian process regression model, namely, the temperature at 14m (T 14 ), the temperature difference between 10m and 0m (T 10 -T0), wind speed at 2m (W2) and wind difference between 4m and 2m (W4-W2).

[0099] Table 1 Correlation between meteorological data and seeing

[0100]

[0101] (3) Training and testing of missing seeing value filling model

[0102] During training, only the data set without missing values ​​of seeing is used for training. After the input and output data are standardized by the z-score method, the rational quadratic kernel is selected as the kernel function for Gaussian process regression model training. The optimal hyperparameters of the kernel function calculated by the quasi-Newton method are: θ = [σ f ,α,l] T =[2.47,0.01,0.85] T .

[0103] The accuracy of the prediction results of the test set is expressed by the root mean square error (RMSE) and the coefficient of determination (R 2 ) is used to evaluate. RMSE is a measure of the deviation between the predicted data and the measured data. The smaller the value, the higher the accuracy of the prediction network. 2 Indicates the degree of fit of the regression model, R 2 The closer the value is to 1 or 100%, the better the model fit is. 2 The formula is as follows.

[0104]

[0105]

[0106] Where: y' i and i are the estimated value and the measured value, respectively. is the average value of the actual sample. The test results are expressed as RMSE and R 2 To evaluate the performance of the model on the test set. Figure 5 This is the test result of seeing. It can be seen from the figure that RMSE is 0.20arcsec, R 2 The value is 0.91, indicating that the regression accuracy of the model is high, which verifies the effectiveness of the present invention. In order to further verify the effectiveness of the present invention, it is compared with the commonly used adjacent filling method. When the adjacent filling method is used to fill the missing values ​​of seeing, the RMSE is 0.26 arcsec, and R 2 The value is 84%. Compared with this method, the prediction accuracy of the method proposed in the present invention is improved by 23%.

[0107] (4) Online missing value filling of seeing

[0108] The Gaussian process regression model trained based on historical values ​​is used to fill in the missing values ​​of the seeing at the observation site online. When the seeing data value collected at a certain moment is 0, it is considered that the seeing collection is interrupted at this time. The meteorological data with high correlation with the seeing at the current moment is obtained and input into the Gaussian process regression model to obtain the model's predicted value. The predicted value is de-normalized and used as the filling value of the missing seeing value. The effect of filling the missing seeing value is shown in the figure below. Figure 6 As shown, in order to clearly demonstrate the filling effect, the figure shows the filling of some missing seeing values.

[0109] (5) Online update of Gaussian process regression model

[0110] Since meteorological data are seasonal and cyclical, the newly collected meteorological data have a strong correlation with the data in the most recent period. In order to ensure the accuracy of the Gaussian process regression model, every week, the hyperparameter θ of the current Gaussian process regression model is used as the initial value, and the meteorological data collected in the current week and the corresponding seeing data are used to train the Gaussian process regression model to update the Gaussian process regression model online.

[0111] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for online filling of missing values ​​of Antarctic seeing, characterized in that: The method comprises: Obtain historical seeing data and meteorological data of the observation site; Analyze the historical seeing data and the historical meteorological data, and use the Spearman rank correlation coefficient method to select target meteorological data with high correlation with the seeing; Taking the target meteorological data as input and the historical seeing data as output, establishing training sample data of the observation site, and using the z-score standard scoring method to standardize the training sample data; Based on the training sample data, selecting a rational quadratic kernel to calculate the covariance between input variables of the training sample data; The negative logarithm of the conditional probability of the training sample data is used as the likelihood function, the minimum value of the likelihood function is used as the objective function, and the optimization algorithm is the quasi-Newton method to obtain the Gaussian process regression model between seeing and meteorological data. Apply the regression model to fill in missing values ​​of real-time seeing at the target site, including: Determine whether the target seeing can be collected normally. When the target seeing data value collected at a certain moment is 0, it is considered that the target seeing collection is interrupted. At this time, the target meteorological data at the current moment is obtained, and after standardization, it is input into the regression model to calculate the predicted value; The predicted values ​​output by the regression model are de-standardized to obtain fill values ​​for missing values ​​of the target seeing.

2. The method according to claim 1, characterized in that The update also includes updating the regression model according to the seasonality and periodicity of the meteorological data, wherein the update uses the hyperparameters in the current regression model as initialization values, uses the new seeing data and new meteorological data collected at the observation station in the current week as new sample data, and repeats the following steps: Taking the new meteorological data as input and the new seeing data as output, establishing new training sample data of the observation site, and using the z-score standard scoring method to perform standardization processing on the new training sample data; Based on the new training sample data, a rational quadratic kernel is selected to calculate the covariance between sample input variables; The negative logarithm of the conditional probability of the new training sample data is used as the likelihood function, the minimum value of the likelihood function is used as the objective function, and the optimization algorithm is the quasi-Newton method to obtain a trained new regression model.

3. The method according to claim 1, characterized in that: The Spearman rank formula is: Where d is the difference in rank between two columns, n is the length of each column, and ρ s is the Spearman rank correlation coefficient.

4. The method according to claim 1, characterized in that: The z-score standard scoring method calculation formula is: In the formula, x and are the data before and after standardization, u and σ are the mean and variance of the samples, respectively.

5. The method according to claim 1, characterized in that: The rational quadratic kernel calculation formula is: In the formula, For x i and x j The Euclidean distance of f ,α,l] T .

6. The method according to claim 1, characterized in that: The calculation formula of the likelihood function is: In the formula, 7. The method according to claim 1, characterized in that: The Gaussian process regression model predicts the value The calculation formula is as follows: In the formula, K and K * are all shorthand forms of the covariance matrix, where K * =K(x * ,X) is the training data input X and the current input x * The covariance matrix of , K = K (X, X) is the autocovariance matrix of the training data input X.

8. The method according to claim 7, characterized in that: The denormalization formula is: In the formula, y pred That is, it is the filling value of the missing seeing value, and u and σ are the mean and variance of the samples respectively.

9. The method according to claim 1, characterized in that: The meteorological historical data include temperature, temperature difference, wind speed, wind difference, wind direction, air pressure and relative humidity.

Images