Reflected radiation data interpolation method, system and application

Abstract

The invention discloses a reflected radiation data interpolation method, system and application, and relates to the meteorological field, the reflected radiation data interpolation method comprises the following steps: obtaining observed radiation data and meteorological data; carrying out device difference correction on the incident short wave radiation data; calculating a clearness index; eliminating abnormal values of the reflected short-wave radiation data, the incident short-wave radiation data after the difference correction and the meteorological data; carrying out interpolation on the missing incident short wave radiation data; interpolating the missing reflection short wave radiation data; and systematically evaluating the interpolation result of the reflected short-wave radiation data. The invention provides a reflected radiation data processing method with an abnormal value elimination mechanism, a self-adaptive interpolation capability and a system evaluation function, which can systematically solve the problems of abnormity and missing in observation data, improve the accuracy of an interpolation result, data integrity and credibility, and improve the reliability of the observation data. And a solid data basis is provided for subsequent applications such as carbon-water flux calculation, ecological model simulation and remote sensing inversion.

Description

Technical Field

[0001] This invention relates to the field of meteorology, and in particular to a method, system and application for interpolating reflected radiation data. Background Technology

[0002] In studies of energy and mass exchange in ecosystems, radiation data (including incident and reflected radiation) serves as a crucial foundation for estimating surface energy budget and vegetation photosynthetic efficiency, and has wide-ranging applications in ecological monitoring, weather forecasting, and carbon flux models. With the widespread adoption of flux observation systems (such as eddy covariance techniques), obtaining high temporal resolution radiation and meteorological data has become possible. However, limitations such as instrument malfunctions, meteorological interference, human error, or data transmission issues often result in a large number of missing and outlier values ​​in the observed data.

[0003] In existing technologies, methods such as linear interpolation, mean substitution, regression models, or machine learning algorithms are commonly used to fill in missing radiation data. However, these methods often fail to guarantee the physical rationality and continuity of the interpolation results when faced with strong seasonal variations and nonlinear dynamic characteristics of radiation data. This is especially true in short-term, high-change scenarios (such as cloudy skies turning sunny), which can easily lead to interpolation errors or distorted results.

[0004] Furthermore, some methods fail to fully integrate the coupling relationship between meteorological factors (such as temperature, relative humidity, VPD, etc.) and radiation characteristics, thus limiting the adaptability of interpolation models to environmental changes. Meanwhile, for the processing of reflected radiation data, most studies rely heavily on static empirical ratios or fixed regression coefficients, lacking an adaptive adjustment mechanism based on sliding windows, making it difficult to assess data uncertainty while ensuring interpolation accuracy.

[0005] Therefore, there is an urgent need for a reflectance and radiation data processing method with outlier removal mechanism, adaptive interpolation capability, and system evaluation function, which can systematically solve the problems of anomalies and missing data in observation data, improve data integrity and reliability, and provide a solid data foundation for subsequent applications such as carbon and water flux calculation, ecological model simulation and remote sensing inversion. Summary of the Invention

[0006] To address the above problems, this invention provides a method, system, and application for interpolating reflected radiation data, aiming to systematically solve the problems of anomalies and missing data in observation data and improve data integrity and reliability.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for interpolating reflected radiation data includes the following steps: S1. Acquire the incident shortwave radiation data SRin and reflected shortwave radiation data SRout, meteorological data, and local latitude and longitude information observed within a specified time period; wherein, the incident shortwave radiation data SRin and reflected shortwave radiation data SRout observed within the specified time period include multiple radiation data arranged in order of timestamps, with each radiation data corresponding to the acquisition timestamp; the meteorological data includes air temperature Tair or soil temperature Tsoil, saturated vapor pressure difference VPD, or relative humidity RH every half hour; S2. Perform instrument error correction on the incident shortwave radiation data SRin to obtain the instrument error corrected incident shortwave radiation data SRin_cal; S3. Calculate the clear sky index Kt based on the incident shortwave radiation data SRin_cal after instrument error correction, latitude and longitude information and day number; S4. Perform outlier removal processing on the reflected shortwave radiation data SRout, the incident shortwave radiation data SRin_cal after instrument error correction, and the meteorological data to obtain radiation data and meteorological data after outlier removal. S5. Based on the incident shortwave radiation data SRin_cal after outlier removal and meteorological data, the missing incident shortwave radiation data is interpolated to obtain the incident shortwave radiation data interpolation result SRin_fill. S6. Based on the reflected shortwave radiation data SRout after outlier removal, the incident shortwave radiation data interpolation result SRin_fill, and the clear sky index Kt, the missing reflected shortwave radiation data is interpolated to obtain the reflected shortwave radiation data interpolation result SRout_fill. S7. Perform a systematic evaluation on the interpolation result SRout_fill of the reflected shortwave radiation data.

[0008] Furthermore, step S4 specifically includes: S41. Data in the incident shortwave radiation data SRin_cal that do not meet the preset threshold after the rejector difference correction are removed to obtain the initial incident shortwave radiation data; the preset threshold includes: incident shortwave radiation value ≥1200 W / m² or ≤0 W / m². S42. Remove data from the reflected shortwave radiation data SRout that do not meet the preset threshold to obtain the initial reflected shortwave radiation data; the preset threshold includes: reflected shortwave radiation value ≤ 0 W / m², or reflected shortwave radiation value ≥ the incident shortwave radiation value SRin_cal after instrument error correction for the corresponding timestamp; S43. Remove meteorological data that does not meet the preset thresholds to obtain initial meteorological data; the preset thresholds include: air temperature ≤ -70℃, soil temperature ≥ 30℃ or ≤ -15℃, relative humidity ≥ 100% or ≤ 0%, and saturated water vapor pressure difference ≤ 0 hPa. S44. Use a sliding window to perform secondary outlier removal on the initial incident shortwave radiation data, initial reflected shortwave radiation data, and initial meteorological data. The window size is 7 days, the moving step is 1 day, and the n-times standard deviation method is used within the window to obtain the flux data after outlier removal. The value of n is dynamically adjusted according to the data type and season.

[0009] Furthermore, in the n-times standard deviation method: When processing initial incident shortwave radiation data and meteorological data, n=3; When processing initial reflected shortwave radiation data during the growing season, n=3; When processing initial reflected shortwave radiation data outside the growing season, n=4.

[0010] Furthermore, step S5 specifically includes: S51. Align the incident shortwave radiation data SRin_cal after removing outliers with the meteorological data according to the timestamp to generate an incident shortwave radiation data interpolation input file. S52. Using the sMDSGapFill method provided by the "REddyProc" package in the programming software R, the missing incident radiation data is interpolated using the meteorological data in the incident data interpolation input file, generating an incident shortwave radiation data interpolation output file. S53. Organize the incident shortwave radiation data interpolation output file and extract the incident shortwave radiation data interpolation result SRin_fill.

[0011] Furthermore, step S6 specifically includes: S61. Align the incident shortwave radiation data SRin_cal and the reflected shortwave radiation data SRout after removing outliers according to the timestamp to generate a reflected shortwave radiation data interpolation input file. S62. Interpolating the reflected shortwave radiation data using an adaptive window estimation method, including: S621. Calculate the albedo α for each timestamp. i =SRout / SRin_cal; S622. Using a 3-day initial window size, filter data within the window whose difference between the clear sky index Kt and missing data is less than 0.1, and calculate α within the window that meets the filtering requirements. i The mean α s Standard deviation sd; S623, using α s α s±sd is multiplied by the corresponding SRin_cal of the missing value to generate the imputation result SRout_fill and confidence interval. The confidence interval includes SRout_Usd and SRout_Dsd, where SRout_Usd and SRout_Dsd are the upper and lower limits of the confidence interval with the standard deviation sd as the reference index, respectively. S624. If there is no data in the window that meets the filtering requirements, expand the difference range in the filtering requirements by ±0.1 until valid data is found. S625. If there is no valid data in the window, the window size will be increased by 1 day before or after the interpolation is successful. The maximum window size is 61 days. S626. Mark the window size SRout_gap for the imputation data. For existing data, the value of SRout_gap is 0. For missing values, the value of SRout_gap is the window size corresponding to the imputation. S627. Organize the above calculation results to obtain the reflection data interpolation results.

[0012] Furthermore, step S7 specifically includes: S71. Determine data availability based on confidence intervals of interpolation results; S72. Determine the quality level of the reflection data interpolation result based on the window size SRout_gap of the marked interpolation data. The smaller the window size SRout_gap of the marked interpolation data, the higher the quality, and vice versa.

[0013] A reflected radiation data interpolation system, comprising: The data acquisition module is used to acquire incident shortwave radiation data SRin, reflected shortwave radiation data SRout, and meteorological data; The data processing module is used to execute steps S2-S6 of any of the methods described above; An evaluation module is used to perform step S7 of any of the methods described above.

[0014] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0015] A computing device includes a processor and a memory, the memory storing a computer program, the processor executing the program to implement the method described in any of the preceding claims.

[0016] Compared with the prior art, the technological advancements achieved by this invention are as follows: This invention provides a reflection radiation data processing method with an outlier removal mechanism, adaptive interpolation capability, and system evaluation function. It can systematically solve the problems of anomalies and missing data in observation data, improve the accuracy of interpolation results, and thus improve data integrity and reliability, providing a solid data foundation for subsequent applications such as carbon flux calculation, ecological model simulation, and remote sensing inversion. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0018] In the attached diagram: Figure 1 This is a flowchart of the data processing method of the present invention; Figure 2 This is a framework diagram of the data processing method of the present invention; Figure 3 This is a comparison chart of the original data and the missing data interpolation results in an embodiment of the present invention. Detailed Implementation

[0019] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0020] Example 1: A method for interpolating reflected radiation data, the data processing flowchart is as follows. Figure 1 As shown, the data processing method framework is as follows: Figure 2 As shown, it includes the following steps: S1. Obtain observed radiation and meteorological data.

[0021] Radiation data, related meteorological data, and latitude and longitude information for 2020 were obtained from flux tower No. 1 at the Qingyuan Forest Ecosystem National Field Scientific Observation and Research Station in Liaoning Province (hereinafter referred to as Qingyuan Forest Station). The radiation data includes: shortwave incident radiation data SRin and shortwave reflected radiation data SRout every half hour at Qingyuan Forest Station No. 1 in 2020. The meteorological data includes: air temperature Tair or soil temperature Tsoil and saturated vapor pressure difference VPD or relative humidity RH every half hour at Qingyuan Forest Station No. 1 in 2020. The radiation data and meteorological data have the same timestamp, corresponding to the data acquisition time.

[0022] S2. Perform instrument error correction on the shortwave incident radiation data SRin to obtain the instrument error corrected shortwave incident radiation data SRin_cal corresponding to each timestamp in 2020. The correction coefficient used for the instrument error correction is:

[0023] S3. Calculate the clear sky index Kt based on the incident shortwave radiation data SRin_cal after instrument error correction, latitude and longitude information, and day number; S4. Perform outlier removal processing on the shortwave reflected radiation data SRout, the shortwave incident radiation data SRin_cal after instrument error correction, and the meteorological data to obtain outlier-removed radiation data and meteorological data corresponding to each timestamp in 2020.

[0024] In this embodiment, S4 specifically includes: S41. Remove the shortwave incident radiation data SRin_cal after instrument error correction from the radiation data in 2020 that meets any outlier condition to obtain the initial shortwave incident radiation data.

[0025] The outlier conditions include: The shortwave incident radiation value after instrument error correction in the radiation data is greater than 1200 W / m. 2 ; The shortwave incident radiation value after instrument error correction in the radiation data is less than 0 W / m. 2 .

[0026] S42. Remove the reflected radiation data SRout data that meet any outlier condition from the radiation data in 2020 to obtain the initial shortwave reflected radiation data.

[0027] The outlier conditions include: The shortwave reflected radiation value in the radiation data is less than 0 W / m 2 ; When the timestamps are the same, the shortwave reflected radiation value in the radiation data is less than the shortwave incident radiation after the instrument error correction.

[0028] S43. Remove meteorological data that meet any outlier condition from the meteorological data of 2020 to obtain the initial meteorological data.

[0029] The outlier conditions include: The air temperature value in the meteorological data is less than -70℃; The soil temperature value in the meteorological data is greater than 30℃; The soil temperature value in the meteorological data is less than -15℃; The relative humidity value in the meteorological data is greater than 100%; The relative humidity value in the meteorological data is less than 0%; The saturated vapor pressure difference in meteorological data is less than 0 hPa; S44. Use a sliding window to remove outliers from the initial shortwave incident radiation, shortwave reflected radiation, and meteorological data. Use the n-times standard deviation method within the window, with a window size of 7 days and a moving step of 1 day to obtain the flux data after outlier removal.

[0030] The value of n is determined based on the geographical location and climatic conditions of Qingyuan Forest Station, including: When processing initial incident shortwave radiation and meteorological data, n = 3; When dealing with the initial reflected shortwave radiation during the growing season, n = 3; When dealing with initial reflected shortwave radiation during the non-growing season, n = 4.

[0031] S5. For the incident radiation result SRin_cal after outlier removal, the missing incident radiation data is interpolated using the meteorological data after outlier removal to obtain the incident radiation data interpolation result.

[0032] In this embodiment, S4 specifically includes: S51. After removing outliers from S4, the shortwave incident data SRin_cal and meteorological data are combined according to the principle of the same timestamp to obtain the incident data interpolation input file.

[0033] S52. Use the "REddyProc" package in the programming software R-4.4.2 to process the above-mentioned incident data interpolation input file. Use the sMDSGapFill method to interpolate the meteorological data in the shortwave incident data interpolation input file. At the same time, use the meteorological data to interpolate the missing shortwave incident radiation data to obtain the shortwave incident data interpolation output file.

[0034] S53. Organize the shortwave incident data interpolation output file and extract the shortwave incident data interpolation result SRin_fill.

[0035] S6. Based on the shortwave reflection data results after outlier removal, the missing shortwave reflection radiation data is interpolated using the shortwave incident radiation data after outlier removal to obtain the shortwave reflection radiation data interpolation results.

[0036] In this embodiment, S6 specifically includes: S61. The shortwave incident data SRin_cal and shortwave reflection data SRout, after removing outliers obtained in S4, are combined according to the principle of the same timestamp to obtain the reflection data interpolation input file.

[0037] S62. Using an adaptive window estimation method, interpolate the reflection data, wherein S62 specifically includes: S621. Calculate the ratio of SRout to SRin_cal for each timestamp in 2020, i.e., the albedo α. i .

[0038] S622. The initial window size is 3 days (including missing data, a total of 144 data points). For each missing value of SRout retrieved in the window, and filtering out data within the window where the clear sky index Kt differs from the missing value by 0.1, calculate all α values ​​within the window that meet the filtering criteria. i The mean α s Standard deviation sd.

[0039] S623, respectively using α s α s ±sd is multiplied by the corresponding SRin_cal of the missing value to generate the imputation result SRout_fill and confidence interval. The confidence interval includes SRout_Usd and SRout_Dsd, where SRout_Usd and SRout_Dsd are the upper and lower limits of the confidence interval with the standard deviation sd as the reference index, respectively.

[0040] S624. If there is no data that meets the filtering requirements in the initial window, the filtering requirements are changed to the difference between the clear sky index Kt and the missing data being less than 0.2 (that is, the difference range is expanded to ±0.1). If there is still no data that meets the filtering requirements, the difference range is expanded to ±0.1 until data that meets the filtering requirements is found.

[0041] S625. If all SRout values ​​in the initial window are missing, the window size is increased to 5 days (that is, increased by one day before and after). If all SRout values ​​are still missing, the window size is increased by one day before and after until the imputation is successful. The maximum window size is 61 days.

[0042] S626. Add SRout_gap. For existing data, the value of SRout_gap is 0. For missing values, the value of SRout_gap is the window size corresponding to the imputation.

[0043] S627. Organize the above calculation results to obtain the reflection data interpolation results.

[0044] For example, the original data and the missing data imputation results are compared. Figure 3 As shown.

[0045] S7. Perform a system evaluation on the reflection data interpolation results.

[0046] In this embodiment, S7 specifically includes: S71, SRout_Usd, and SRout_Dsd are the upper and lower limits of the confidence interval with the standard deviation sd as the reference index, respectively. The confidence interval is used to judge the usability of the reflection data interpolation results.

[0047] S72. Determine the quality level of the reflection data interpolation result based on SRout_gap. The smaller the SRout_gap, the higher the quality, and vice versa.

[0048] Example 2: A method for interpolating reflected radiation data, comprising: The data acquisition section is used to obtain half-hourly radiation data and meteorological data observed in 2020 by flux tower No. 1 at Qingyuan Forest Station; In the data processing section, outlier removal and gap interpolation were performed on the shortwave reflected radiation data of flux tower No. 1 at Qingyuan Forest Station in 2020 using the methods described in steps S2-S6 above. In the system evaluation section, the shortwave reflected radiation data of flux tower No. 1 at Qingyuan Forest Station in 2020 were systematically evaluated using the method described in step S7 above.

[0049] Example 3: A computer-readable storage medium storing one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the method of Example 1.

[0050] Example 4: A computing device, comprising: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method of Embodiment 1.

[0051] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for interpolating reflected radiation data, characterized in that, Includes the following steps: S1. Acquire the incident shortwave radiation data SRin and reflected shortwave radiation data SRout, meteorological data, and local latitude and longitude information observed within a specified time period; wherein, the incident shortwave radiation data SRin and reflected shortwave radiation data SRout observed within the specified time period include multiple radiation data arranged in order of timestamps, with each radiation data corresponding to the acquisition timestamp; the meteorological data includes air temperature Tair or soil temperature Tsoil, saturated vapor pressure difference VPD, or relative humidity RH every half hour; S2. Perform instrument error correction on the incident shortwave radiation data SRin to obtain the instrument error corrected incident shortwave radiation data SRin_cal; S3. Calculate the clear sky index Kt based on the incident shortwave radiation data SRin_cal after instrument error correction, latitude and longitude information and day number; S4. Perform outlier removal processing on the reflected shortwave radiation data SRout, the incident shortwave radiation data SRin_cal after instrument error correction, and the meteorological data to obtain radiation data and meteorological data after outlier removal. S5. Based on the incident shortwave radiation data SRin_cal after outlier removal and meteorological data, the missing incident shortwave radiation data is interpolated to obtain the incident shortwave radiation data interpolation result SRin_fill. S6. Based on the reflected shortwave radiation data SRout after outlier removal, the incident shortwave radiation data interpolation result SRin_fill, and the clear sky index Kt, the missing reflected shortwave radiation data is interpolated to obtain the reflected shortwave radiation data interpolation result SRout_fill. S7. Perform a systematic evaluation on the interpolation result SRout_fill of the reflected shortwave radiation data.

2. The method according to claim 1, characterized in that, Step S4 specifically includes: S41. Data in the incident shortwave radiation data SRin_cal that do not meet the preset threshold after the rejector difference correction are removed to obtain the initial incident shortwave radiation data; the preset threshold includes: incident shortwave radiation value ≥1200 W / m² or ≤0 W / m². S42. Remove data from the reflected shortwave radiation data SRout that do not meet the preset threshold to obtain the initial reflected shortwave radiation data; the preset threshold includes: reflected shortwave radiation value ≤ 0 W / m², or reflected shortwave radiation value ≥ the incident shortwave radiation value SRin_cal after instrument error correction for the corresponding timestamp; S43. Remove meteorological data that does not meet the preset thresholds to obtain initial meteorological data; the preset thresholds include: air temperature ≤ -70℃, soil temperature ≥ 30℃ or ≤ -15℃, relative humidity ≥ 100% or ≤ 0%, and saturated water vapor pressure difference ≤ 0hPa. S44. Use a sliding window to perform secondary outlier removal on the initial incident shortwave radiation data, initial reflected shortwave radiation data, and initial meteorological data. The window size is 7 days, the moving step is 1 day, and the n-times standard deviation method is used within the window to obtain the flux data after outlier removal. The value of n is dynamically adjusted according to the data type and season.

3. The method according to claim 2, characterized in that, In the n-times standard deviation method: When processing initial incident shortwave radiation data and meteorological data, n=3; When processing initial reflected shortwave radiation data during the growing season, n=3; When processing initial reflected shortwave radiation data outside the growing season, n=4.

4. The method according to claim 1, characterized in that, Step S5 specifically includes: S51. Align the incident shortwave radiation data SRin_cal after removing outliers with the meteorological data according to the timestamp to generate an incident shortwave radiation data interpolation input file. S52. Using the sMDSGapFill method provided by the "REddyProc" package in the programming software R, the missing incident radiation data is interpolated using the meteorological data in the incident data interpolation input file, generating an incident shortwave radiation data interpolation output file. S53. Organize the incident shortwave radiation data interpolation output file and extract the incident shortwave radiation data interpolation result SRin_fill.

5. The method according to claim 1, characterized in that, Step S6 specifically includes: S61. Align the incident shortwave radiation data SRin_cal and the reflected shortwave radiation data SRout after removing outliers according to the timestamp to generate a reflected shortwave radiation data interpolation input file. S62. Interpolating the reflected shortwave radiation data using an adaptive window estimation method, including: S621. Calculate the albedo α for each timestamp. i =SRout / SRin_cal; S622. Using a 3-day initial window size, filter data within the window whose difference between the clear sky index Kt and missing data is less than 0.1, and calculate α within the window that meets the filtering requirements. i The mean α s Standard deviation sd; S623, using α s α s ±sd is multiplied by the corresponding SRin_cal of the missing value to generate the imputation result SRout_fill and confidence interval. The confidence interval includes SRout_Usd and SRout_Dsd, where SRout_Usd and SRout_Dsd are the upper and lower limits of the confidence interval with the standard deviation sd as the reference index, respectively. S624. If there is no data in the window that meets the filtering requirements, expand the difference range in the filtering requirements by ±0.1 until valid data is found. S625. If there is no valid data in the window, the window size will be increased by 1 day before or after the interpolation is successful. The maximum window size is 61 days. S626. Mark the window size SRout_gap for the imputation data. For existing data, the value of SRout_gap is 0. For missing values, the value of SRout_gap is the window size corresponding to the imputation. S627. Organize the above calculation results to obtain the reflection data interpolation results.

6. The method according to claim 1, characterized in that, Step S7 specifically includes: S71. Determine data availability based on confidence intervals of interpolation results; S72. Determine the quality level of the reflection data interpolation result based on the window size SRout_gap of the marked interpolation data. The smaller the window size SRout_gap of the marked interpolation data, the higher the quality, and vice versa.

7. A reflected radiation data interpolation system, characterized in that, include: The data acquisition module is used to acquire incident shortwave radiation data SRin, reflected shortwave radiation data SRout, and meteorological data; The data processing module is configured to perform steps S2-S6 of the method according to any one of claims 1-6; An evaluation module is used to perform step S7 of the method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the method of any one of claims 1-6.

9. A computing device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the program to implement the method of any one of claims 1-6.

Images