A CO2 flux data interpolation method integrating outlier detection and LightGBM

Abstract

The present invention relates to the technical field of environmental monitoring, and discloses a CO2 flux data interpolation method integrating outlier detection and LightGBM, comprising: acquiring carbon dioxide flux data and meteorological factor data of a study area; performing data segmentation and correction on the carbon dioxide flux data to obtain half-hourly carbon dioxide flux data and friction wind speed values; using a single-class support vector machine based on anomaly detection to perform outlier detection on the half-hourly carbon dioxide flux data to obtain half-hourly carbon dioxide flux data with outliers removed, and removing untrustworthy values ​​through a friction wind speed threshold to obtain half-hourly carbon dioxide flux data with untrustworthy values ​​removed; inputting the half-hourly carbon dioxide flux data with untrustworthy values ​​removed and the meteorological factor data into a LightGBM gap filling model for training to obtain a trained LightGBM gap filling model and interpolated carbon dioxide flux data; the method improves the calculation or interpolation accuracy of carbon dioxide flux data.

Description

Technical Field

[0001] The present invention relates to the technical field of environmental monitoring, and in particular to a CO2 flux data interpolation method integrating outlier detection and LightGBM. Background Art

[0002] Net ecosystem carbon dioxide flux refers to the change in ecosystem carbon accumulation caused by plant photosynthesis, carbon storage in the canopy air, and carbon emissions consumed by biological and abiotic respiration in the ecosystem. It mainly reflects the carbon balance of the ecosystem from the perspective of flux observation, and determines the "sink-source" or "source-sink" transformation process of the surface ecosystem. With the increasing severity of global climate change, carbon dioxide (CO2), as the main component of greenhouse gases, has been increasing in concentration in the atmosphere, leading to global temperature rise and frequent extreme climate. In order to cope with climate change, how to accurately measure and evaluate the carbon dioxide flux in the ecosystem has become an important scientific research topic. Especially in alpine areas, these areas play an increasingly important role in the global carbon cycle due to their unique climate and geographical characteristics. At present, there are many methods and models for the measurement and calculation of carbon dioxide, but the research on the unique ecological characteristics of alpine areas is still relatively weak. Most existing methods are mainly concentrated in temperate or tropical regions, resulting in insufficient accuracy in the measurement of CO2 flux in alpine areas, which affects the comprehensive understanding of the global carbon cycle.

[0003] Although some studies have explored the carbon dioxide exchange in alpine regions, there are still some shortcomings. The climatic conditions in alpine regions are extremely harsh, and equipment layout and data collection face huge challenges, so it is difficult to obtain long-term and stable monitoring data. In addition, during the flux observation process, due to factors such as instrument failure (sensor lines, wiring), extreme weather conditions, power supply, and instrument operation and maintenance timeliness, outliers and data missing will occur. According to statistics, in the observations of the European Flux Network and the American Flux Network, the carbon dioxide flux data has a high missing rate, which brings difficulties to the application of flux tower data. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a CO2 flux data interpolation method integrating outlier detection and LightGBM, which is used to solve the problem of low accuracy of carbon dioxide flux calculation results caused by uncertainty and missing data acquisition in the prior art.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A CO2 flux data interpolation method integrating outlier detection and LightGBM includes the following steps:

[0007] S1. Obtain carbon dioxide flux data and meteorological factor data in the study area;

[0008] S2. Performing data segmentation and correction on the carbon dioxide flux data to obtain half-hourly carbon dioxide flux data and friction wind speed values ​​of the half-hourly carbon dioxide flux data;

[0009] S3, using a single-class support vector machine based on anomaly detection to detect outliers on the half-hourly carbon dioxide flux data, and obtaining the half-hourly carbon dioxide flux data with outliers removed;

[0010] S4. According to the friction wind speed value of the half-hour carbon dioxide flux data, the half-hour carbon dioxide flux data with abnormal values ​​removed is subjected to untrustworthy value removal through the friction wind speed threshold to obtain the half-hour carbon dioxide flux data with untrustworthy values ​​removed;

[0011] S5. Input the half-hourly carbon dioxide flux data and meteorological factor data after removing unreliable values ​​into the LightGBM gap filling model for training to obtain the trained LightGBM gap filling model and interpolated carbon dioxide flux data.

[0012] Furthermore, step S1 specifically includes:

[0013] S11. Set the sampling frequency and sampling time period, and use the eddy covariance system to collect carbon dioxide flux data in high-altitude cold areas;

[0014] S12, using micro-meteorological observation equipment to collect micro-meteorological data in the high-cold area, and using a three-spline interpolation method to interpolate the micro-meteorological data to obtain interpolated micro-meteorological data;

[0015] Among them, micrometeorological data include four-component radiation, photosynthetically active radiation, soil temperature, humidity, air pressure, two-dimensional ultrasonic wind speed, and air temperature and humidity;

[0016] S13. Unify the time resolution of the interpolated micrometeorological data with the time resolution of the carbon dioxide flux data to obtain meteorological factor data.

[0017] Furthermore, step S3 specifically includes:

[0018] S31, setting relevant parameters of a single-class support vector machine based on anomaly detection, and inputting sample data to train the single-class support vector machine based on anomaly detection, to obtain a trained single-class support vector machine based on anomaly detection;

[0019] Among them, the relevant parameters include the upper limit of the training error score and the lower limit of the support vector score, the kernel type, and the kernel function bandwidth;

[0020] S32, inputting the half-hour carbon dioxide flux data into the trained single-class support vector machine based on anomaly detection to perform outlier detection, obtaining outlier labels and removing the half-hour carbon dioxide flux data with outlier labels, to obtain the half-hour carbon dioxide flux data with outliers removed.

[0021] Furthermore, step S4 specifically includes:

[0022] S41, dividing the half-hour carbon dioxide flux data excluding abnormal values ​​into day and night, dividing the data greater than or equal to the set photosynthetic radiation threshold value in the half-hour carbon dioxide flux data excluding abnormal values ​​into daytime period data, and dividing the data less than the set photosynthetic radiation threshold value in the half-hour carbon dioxide flux data excluding abnormal values ​​into nighttime period data;

[0023] S42, using the quantile method to divide the night-time data into a number of night-time data subsets of equal size;

[0024] The amount of data in each nighttime data subset is equal;

[0025] S43, dividing each nighttime data subset of equal size into a number of data intervals of equal size;

[0026] S44, calculating the average half-hourly carbon dioxide flux data value of each data interval of equal size;

[0027] S45, calculating the average value of the half-hourly carbon dioxide flux data of the last 10 data intervals of each data interval of equal size;

[0028] S46. According to the friction wind speed value of the half-hour carbon dioxide flux data, obtain the friction wind speed value of each data interval of equal size and calculate the average value to obtain the average friction wind speed value of each data interval of equal size;

[0029] S47, sequentially determine whether the average half-hour carbon dioxide flux data value of each data interval of equal size is higher than x times the average value of the half-hour carbon dioxide flux data of the last 10 intervals of each data interval of equal size; if so, use the average friction wind speed value of the current data interval as the friction wind speed threshold; otherwise, continue to determine until the next data interval is found to meet the condition, and use the average friction wind speed value of the data interval as the friction wind speed threshold;

[0030] S48. Mark all night-time data that are less than the friction wind speed threshold as invalid night-time data, and obtain half-hourly carbon dioxide flux data with unreliable values ​​removed.

[0031] Furthermore, step S5 specifically includes:

[0032] S51, using the meteorological factor data as the original data, and using the half-hourly carbon dioxide flux data with unreliable values ​​removed as the label data, to obtain a labeled original data set, and dividing the labeled original data set into a training set and a test set in proportion;

[0033] S52. Set the basic parameters of the LightGBM gap filling model;

[0034] Among them, the basic parameters include evaluation indicators, loss function type, number of leaf nodes, and boosting algorithm type parameters;

[0035] S53, inputting the training set into the LightGBM gap filling model for training, and using the grid search method to perform grid search on the LightGBM gap filling model to obtain the best hyperparameter group of the LightGBM gap filling model, and using the best hyperparameter group to retrain the LightGBM gap filling model to obtain a trained LightGBM gap filling model;

[0036] S54. Input the test set into the trained LightGBM gap filling model to interpolate missing values ​​and obtain interpolated half-hourly carbon dioxide flux data.

[0037] The present invention has the following beneficial effects:

[0038] 1. The present invention proposes a CO2 flux data interpolation method that integrates outlier detection and LightGBM, introduces a single-class support vector machine for anomaly detection to remove outliers from half-hourly carbon dioxide flux data, thereby enhancing the effectiveness of data processing, thereby avoiding calculation errors caused by outliers and improving the reliability of carbon dioxide exchange data;

[0039] 2. The friction wind speed threshold calculation is introduced to improve the data accuracy of carbon dioxide flux data by eliminating unreliable values, thereby improving the recognition accuracy of the model;

[0040] 3. The integration of outlier detection and LightGBM data interpolation improves the applicability and accuracy of the model, while improving the multidimensionality and comprehensiveness of the carbon dioxide exchange calculation, and greatly improving the accuracy of the carbon dioxide calculation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of a CO2 flux data interpolation method that integrates outlier detection and LightGBM proposed in the present invention;

[0042] Figure 2 A schematic diagram of the results of outlier detection using a single-class support vector machine based on anomaly detection;

[0043] Figure 3 Schematic diagram of the training curve of the LightGBM gap filling model without outlier detection;

[0044] Figure 4 Schematic diagram of the training curve of the LightGBM gap filling model after outlier detection;

[0045] Figure 5 Schematic diagram of interpolation accuracy of the LightGBM gap filling model without outlier detection;

[0046] Figure 6 Schematic diagram of the interpolation accuracy of the LightGBM gap filling model after outlier detection. DETAILED DESCRIPTION

[0047] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0048] like Figure 1 As shown, a CO2 flux data interpolation method integrating outlier detection and LightGBM includes the following steps S1-S5:

[0049] S1. Obtain carbon dioxide flux data and meteorological factor data in the study area.

[0050] In this embodiment, the purpose of obtaining the carbon dioxide flux data and meteorological factor data of the study area is to perform data processing in subsequent steps through the collected carbon dioxide flux data, that is, to use the measured carbon dioxide flux data as label data to train the LightGBM gap filling model, and obtain a trained LightGBM gap filling model. When the meteorological factor data is input into the model, the missing values ​​of carbon dioxide can be interpolated to ensure the accuracy of the calculated results, thereby evaluating the influence of meteorological factors on carbon dioxide exchange flux. In addition, in this embodiment, the study area is a high-cold area.

[0051] Specifically, step S1 includes S11-S13:

[0052] S11. Set the sampling frequency and sampling time period, and use the eddy covariance system to collect carbon dioxide flux data in high-altitude cold areas.

[0053] In this example, the eddy covariance system consists of a three-dimensional sonic anemometer and a CO2 / H2O gas analyzer. The data collector has a sampling frequency of 10 Hz and collects and stores the 30-min average value of the net ecosystem CO2 exchange (NEE) flux data, i.e., the carbon dioxide flux data.

[0054] S12. Collect micrometeorological data in high-cold areas using micrometeorological observation equipment, and use the three-spline interpolation method to interpolate the micrometeorological data to obtain interpolated micrometeorological data; wherein the micrometeorological data include four-component radiation, photosynthetically active radiation, soil temperature, humidity, air pressure, two-dimensional ultrasonic wind speed, and air temperature and humidity.

[0055] In this embodiment, an automatic weather station (micrometeorological observation equipment) is installed near the eddy covariance system for synchronous measurement of meteorological environmental factors, mainly including four-component radiation, photosynthetically active radiation, soil temperature, humidity, air pressure, two-dimensional ultrasonic wind speed, and air temperature and humidity, i.e., micrometeorological data; wherein, four-component radiation and photosynthetically active radiation are monitored by the radiation observation system; soil temperature and humidity are monitored by the soil temperature and humidity observation system. In addition, since the collected micrometeorological data may have some missing values, the three-spline interpolation method is used to interpolate them to obtain accurate micrometeorological data.

[0056] S13. Unify the time resolution of the interpolated micrometeorological data with the time resolution of the carbon dioxide flux data to obtain meteorological factor data.

[0057] In this embodiment, since the time resolution of the collected micrometeorological data is 10 minutes, after the missing values ​​in the micrometeorological data are interpolated using the cubic spline interpolation method, the time resolution of the interpolated micrometeorological data needs to be converted to 30 minutes to be consistent with the time resolution of the carbon dioxide flux data, thereby facilitating data processing in subsequent steps.

[0058] S2. Perform data segmentation and correction on the carbon dioxide flux data to obtain half-hourly carbon dioxide flux data and friction wind speed values ​​of the half-hourly carbon dioxide flux data.

[0059] In this embodiment, the collected carbon dioxide flux data is preprocessed to ensure the accuracy and integrity of the data. Since the obtained eddy flux system observation data, i.e., carbon dioxide flux data, cannot be directly analyzed and used, it needs to be converted and segmented, which is mainly divided into two steps. The first step is to use Loggernet software to segment and convert the data by half an hour, and the second step is to use Eddypro software to correct and analyze the segmented and converted data; in addition, when Eddypro performs data processing, all paths are English data, and there cannot be Chinese characters and other symbols. The specific data processing process is as follows: (1) Raw data cutting and conversion; first, use Loggernet software to segment the carbon dioxide flux data into half-hour segments, and convert the segmented data format into TOB1 or TOA5; (2) EddyPro software data processing; EddyPro project files are mainly divided into projectinfo and matedate file editor, and the matedate file editor file-related parameters are divided into three parts: Station, instruments, and Raw File Description; which mainly include site equipment description, site data description, project file basic settings, eddypro data processing settings, and Variables settings; after completing the above-mentioned related parameter settings, start the data processing run, and finally obtain the processed 30-minute flux data primary product; among them, the processed 30-minute flux data primary product includes half-hour carbon dioxide flux data and its corresponding friction wind speed value.

[0060] S3. Use a single-class support vector machine based on anomaly detection to detect outliers on the half-hourly carbon dioxide flux data to obtain the half-hourly carbon dioxide flux data with outliers removed.

[0061] In this embodiment, the latest machine learning anomaly detection method - one-class support vector machine (One-ClassSVM) is used to determine outliers, so as to improve the accuracy and robustness of outlier detection. One-class support vector machine can be used for unsupervised learning. It learns the decision function for anomaly detection, and its main function is to classify new data into normal values ​​similar to the training set or dissimilar outliers. The goal of the one-class support vector machine is to find a hyperplane that can surround normal data, that is, by establishing a boundary so that most training samples are surrounded by this boundary, and abnormal samples will be excluded, thereby improving the accuracy of half-hour carbon dioxide flux data.

[0062] Specifically, step S3 specifically includes S31-S32:

[0063] S31. Set relevant parameters of a single-class support vector machine based on anomaly detection, and input sample data to train the single-class support vector machine based on anomaly detection to obtain a trained single-class support vector machine based on anomaly detection; wherein the relevant parameters include the upper limit of the training error score and the lower limit of the support vector score, the kernel type, and the kernel function bandwidth.

[0064] In this embodiment, the relevant parameter settings are specifically as follows: nu = "outliers_fraction", kernel = "rbf", gamma = "scale", where the parameter outliers_fraction is the upper limit of the training error fraction and the lower limit of the support vector fraction, and must be between 0 and 1. It represents the proportion of outliers in the data set. Since a smaller outliers_fraction value means that the model will be more conservative for abnormal samples, that is, the boundary is defined more strictly, and vice versa, more points will be allowed to be regarded as normal, thereby forming a looser boundary, it is set to 0.05 in this embodiment so that Used to help the model determine the sensitivity of the boundary; Kernel is the type of kernel used, which is set to rbf in this embodiment, which enables the single-class support vector machine to use a nonlinear function to project the hyperspace to a higher dimension and try to find an optimal hyperplane (boundary) to surround most of the normal samples, where the boundary (or decision boundary) refers to a set of boundary lines or hyperplanes that separate normal samples from abnormal samples, that is, the goal of this model is to find a way to separate normal data so as to classify unseen data as normal or abnormal; gamma determines the bandwidth of the kernel function, which is set to scale in this embodiment and can be set to auto if necessary, that is, automatic setting is selected.

[0065] At the same time, a set of sample data is used to train the One-Class SVM model. During the training, the model is used to learn the distribution characteristics of the samples, and a boundary is determined to distinguish normal samples from potential abnormal samples. Then, the test set is predicted based on the trained model to obtain the outlier label, that is, the predicted outlier. Because, during the prediction process, the One-Class SVM will generate a label for each test sample, and the meaning of the label is: 1: indicates that the sample is considered a normal sample; -1: indicates that the sample is considered an outlier. Then filter out the samples marked as -1 (outliers), and only keep the samples marked as 1 (normal values); the final data set is the CO2 flux data set after outliers are removed, and these data can better represent the normal state.

[0066] S32, inputting the half-hour carbon dioxide flux data into the trained single-class support vector machine based on anomaly detection to perform outlier detection, obtaining outlier labels and removing the half-hour carbon dioxide flux data with outlier labels, to obtain the half-hour carbon dioxide flux data with outliers removed.

[0067] In this embodiment, Figure 2 The results of outlier detection based on a single-class support vector machine are shown. Figure 2 It can be seen that 87 outliers were eliminated, thus improving the data accuracy; in addition, the data selected were the half-hourly carbon dioxide flux data of Qilian Station in 2022.

[0068] S4. According to the friction wind speed value of the half-hour carbon dioxide flux data, the half-hour carbon dioxide flux data with abnormal values ​​removed is subjected to unreliable value removal through a friction wind speed threshold to obtain the half-hour carbon dioxide flux data with unreliable values ​​removed.

[0069] In this embodiment, since the size of the friction wind speed indicates the degree of turbulent exchange in the atmosphere, the smaller the friction wind speed, the less sufficient the turbulent exchange. At this time, the carbon flux value measured by the eddy correlation technology is often lower than the actual value, and cannot reflect the actual situation of the carbon exchange amount of the ecosystem. This situation with a smaller friction wind speed usually occurs under a stable stratification state at night; in addition, there are many reasons for underestimating the carbon flux at night, which is still a controversial issue. The quality control room of the nighttime carbon flux observation data is usually determined according to the distribution trend of the turbulent carbon flux with the friction wind speed. The threshold of the effective friction wind speed is determined, and related studies point out that if the carbon flux at a lower friction wind speed threshold is eliminated, there is a better simulation relationship between the nighttime carbon flux and the environmental factor. Therefore, the moving point method (MPT) or the breakpoint detection method (CPT) can be used to obtain the quantile results of the friction wind speed thresholds in each season, and then the half-hour carbon dioxide flux data is filtered out according to the threshold, so as to obtain high-precision half-hour carbon dioxide flux data that excludes unreliable values.

[0070] Specifically, step S4 specifically includes S41-S48:

[0071] S41. Divide the half-hour carbon dioxide flux data after removing abnormal values ​​into day and night, and divide the data greater than or equal to the set photosynthetic radiation threshold in the half-hour carbon dioxide flux data after removing abnormal values ​​into daytime period data, and divide the data less than the set photosynthetic radiation threshold in the half-hour carbon dioxide flux data after removing abnormal values ​​into nighttime period data.

[0072] In this embodiment, the half-hour carbon dioxide flux data with outliers removed is divided into daytime data and nighttime data, that is, the photosynthetic radiation value of the half-hour carbon dioxide flux data with outliers removed is greater than or equal to 10Wm -2 The data is marked as daytime data (ID = 1), and the photosynthetic radiation value of the half-hour carbon dioxide flux data with an abnormal value less than 10Wm -2 The data is marked as the night time period number (ID=0).

[0073] S42. Use the quantile method to divide the night-time data into several night-time data subsets of equal size.

[0074] The amount of data in each nighttime data subset is equal.

[0075] In this embodiment, all night time period data are extracted and quantile division is performed. The extracted night time period data are divided into 6 equal-sized night data subsets according to the size of the data values ​​using the quantile method, thereby ensuring that the amount of data in each night data subset is equal.

[0076] S43, dividing each nighttime data subset of equal size into a number of data intervals of equal size.

[0077] In this embodiment, each nighttime data subset of equal size is divided into data intervals, and for each nighttime data subset, 20 data intervals of equal size are obtained.

[0078] S44. Calculate the average half-hourly carbon dioxide flux data value for each data interval of equal size.

[0079] S45. Calculate the average value of the half-hourly carbon dioxide flux data of the last 10 data intervals of each data interval of equal size.

[0080] S46. According to the friction wind speed value of the half-hour carbon dioxide flux data, the friction wind speed value of each data interval of equal size is obtained and the average value is calculated to obtain the average friction wind speed value of each data interval of equal size.

[0081] S47. Determine in turn whether the average half-hour carbon dioxide flux data value of each data interval of equal size is higher than x times the average value of the half-hour carbon dioxide flux data of the next 10 intervals of each data interval of equal size. If so, use the average friction wind speed value of the current data interval as the friction wind speed threshold. Otherwise, continue to judge until the next data interval is found to meet the condition, and use the average friction wind speed value of the data interval as the friction wind speed threshold.

[0082] In this embodiment, the value of x is 0.95.

[0083] S48. Mark all night-time data that are less than the friction wind speed threshold as invalid night-time data, and obtain half-hourly carbon dioxide flux data with unreliable values ​​removed.

[0084] S5. Input the half-hourly carbon dioxide flux data and meteorological factor data after removing unreliable values ​​into the LightGBM gap filling model for training to obtain the trained LightGBM gap filling model and interpolated carbon dioxide flux data.

[0085] In this embodiment, the LightGBM gap filling model is trained to interpolate the missing or discarded carbon dioxide flux data in the meteorological factor data.

[0086] Specifically, step S5 includes S51-S54:

[0087] S51. The meteorological factor data is used as the original data, and the half-hourly carbon dioxide flux data with unreliable values ​​removed is used as the label data to obtain a labeled original data set, and the labeled original data set is divided into a training set and a test set in proportion.

[0088] In this embodiment, the characteristic data (meteorological factor data) of the input model is normalized to make its mean value 0 and variance 1, and then the input model is trained. The purpose of this processing is to improve the convergence speed during the model training process.

[0089] S52. Set the basic parameters of the LightGBM gap filling model; the basic parameters include evaluation indicators, loss function type, number of leaf nodes, and boosting algorithm type parameters.

[0090] In this embodiment, the basic parameters are set as follows: 'boosting_type': 'gbdt'; 'objective': 'regression', i.e., the loss function type selects the regression model; 'metric': 'rmse', i.e., the evaluation index adopts the root mean square error; 'num_leaves': 30, i.e., the number of leaf nodes is set to 30; 'boosting_type', i.e., the boosting algorithm type parameter is the gradient boosting decision tree (gbdt).

[0091] S53. Input the training set into the LightGBM gap filling model for training. At the same time, use the grid search method to perform grid search on the LightGBM gap filling model to obtain the optimal hyperparameter group of the LightGBM gap filling model, and use the optimal hyperparameter group to retrain the LightGBM gap filling model to obtain a trained LightGBM gap filling model.

[0092] In this embodiment, grid search is used to optimize model parameters. First, a parameter grid is defined, which includes parameters such as 'num_leaves', 'max_depth, 'n_estimators', 'learning_rate', 'feature_fraction', 'bagging_fraction', and 'bagging_freq' ('num_leaves': number of leaves, 'max_depth: maximum depth of the tree, 'n_estimators': number of estimators, 'learning_rate': learning rate, 'feature_fraction': feature ratio (used to randomly select the feature ratio used by each tree during training), 'bagging_freq', and 'feature_fraction'. ging_fraction': sample ratio (used to randomly select the sample ratio used by each tree during training), 'bagging_freq': sample sampling frequency (sampling is performed every how many iterations)) Then create a GridSearchCV object, where the parameter cv is set to 5, that is, 5-fold crossover, and perform grid search, and finally get the best hyperparameter combination: num_leaves=50, max_depth=50, n_estimators=100, learning_rate=0.01, feature_fraction=0.9, bagging_fraction=0.9, bagging_freq=7.

[0093] S54. Input the test set into the trained LightGBM gap filling model to interpolate missing values ​​and obtain interpolated half-hourly carbon dioxide flux data.

[0094] In this example, the evaluation indicators MAE and RMSE are used to evaluate the interpolation performance of the trained LightGBM gap filling model for missing values ​​or missing values ​​of meteorological factor data, and R 2 The indicator evaluates the degree of fit between the actual measured CO2 flux data (i.e., the half-hourly CO2 flux data with unreliable values ​​removed) and the interpolated half-hourly CO2 flux data. The calculation formulas are as follows:

[0095]

[0096] Among them, R 2 represents the degree of fit, MAE represents the mean absolute error, RMSE represents the root mean square error, n represents the number of data, x j ,y j They represent the actual measured CO2 flux data and the interpolated half-hourly CO2 flux data, that is, the measured value and predicted value of CO2 exchange, respectively. They respectively represent the means of the measured and predicted values ​​of carbon dioxide exchange; in addition, the mean absolute error reflects the absolute size of the error in the estimated carbon dioxide exchange; the root mean square error is used to measure the degree to which the estimated carbon dioxide exchange deviates from the measured value. The smaller the value, the better the model performance; the degree of fit reflects the proportion of the influence of the input meteorological factors on the carbon dioxide exchange. The closer the value is to 1, the greater the percentage change in carbon dioxide exchange caused by the meteorological factors input to the model.

[0097] In addition, visualization tools (such as Matplotlib and Seaborn in Python) are used to draw scatter plots of predicted values ​​and actual values ​​to intuitively evaluate the model effect. At the same time, residual analysis of the model is performed to check the residual distribution and verify the assumptions of the model, thereby verifying the effectiveness of the CO2 flux data interpolation method that integrates outlier detection and LightGBM proposed in the present invention, as follows:

[0098] like Figure 3-Figure 4 As shown, Figure 3 shows the training curve of the LightGBM gap filling model without outlier detection, Figure 4 The training curve of the LightGBM gap filling model after outlier detection is shown. Figure 3 and Figure 4 It can be seen that the training curve and test curve accuracy of the LightGBM gap filling model after outlier detection are higher than those of the LightGBM gap filling model without outlier detection.

[0099] like Figure 5-Figure 6 As shown, Figure 5 The interpolation accuracy of the LightGBM gap filling model without outlier detection is shown. Figure 6 The interpolation accuracy of the LightGBM gap filling model after outlier detection is shown. Figure 5-Figure 6 It can be seen that the interpolation accuracy of the LightGBM gap filling model after outlier detection is r 2 is 0.71, which is higher than the LightGBM gap filling model without outlier detection; 2 is the coefficient of determination, which indicates the variability of the dependent variable y explained by the model, R 2 The value range is from 0 to 1; RMSE is the root mean square error, which is used to measure the degree to which the estimated value of carbon dioxide exchange deviates from the measured value. The smaller the value, the better the model performance; MAE is the average error, which reflects the absolute size of the error in the estimated value of carbon dioxide exchange; P<0.001 is the significance level, indicating that according to the statistical test results, the influence of the independent variable x on the dependent variable y is significant.

[0100] In summary, the CO2 flux data interpolation method proposed in the present invention, which integrates outlier detection and LightGBM, enhances the multidimensionality and comprehensiveness of carbon dioxide flux calculation by combining meteorological factor data with carbon dioxide flux data (vorticity observation data); at the same time, by comprehensively considering meteorological factors, it can better reflect the carbon dioxide exchange characteristics of alpine ecosystems under different climatic conditions, thereby improving the applicability and accuracy of the model; in addition, the present invention introduces the latest machine learning anomaly detection method-one-class support vector machine (One-Class SVM) for outlier judgment, which can effectively detect and eliminate outliers in observed data, enhance the effectiveness of data processing, thereby avoiding calculation errors caused by outliers and improving the reliability of carbon dioxide exchange data; compared with traditional methods, the present invention can capture the complex nonlinear relationship of data, so that the interpolation results are closer to the true value, and significantly improve the credibility of interpolation. Most importantly, the present invention can adapt to CO2 flux data under different ecosystems and different environmental conditions, has wide applicability and flexibility, and the method has mature deep learning tools and frameworks, making the interpolation method easy to implement and promote, suitable for scientific researchers and ecological managers. By optimizing the algorithm and strict data quality control process, this method makes the calculation results more reliable and transparent, laying the foundation for data sharing in related fields in the future, and promoting cooperation and communication in ecological and climate-related research. With the intensification of global climate change, the changes in the ecosystem in the alpine regions are becoming increasingly significant. The CO2 flux data interpolation method proposed in this invention will provide important scientific data for the research, prediction and formulation of response strategies for climate change, and promote further research in related fields.

[0101] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0102] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A CO2 flux data interpolation method integrating outlier detection and LightGBM, characterized in that: The following steps are involved: S1. Obtain carbon dioxide flux data and meteorological factor data in the study area; S2. Performing data segmentation and correction on the carbon dioxide flux data to obtain half-hourly carbon dioxide flux data and friction wind speed values ​​of the half-hourly carbon dioxide flux data; S3, using a single-class support vector machine based on anomaly detection to detect outliers on the half-hourly carbon dioxide flux data, and obtaining the half-hourly carbon dioxide flux data with outliers removed; S4. According to the friction wind speed value of the half-hour carbon dioxide flux data, the half-hour carbon dioxide flux data with abnormal values ​​removed is subjected to untrustworthy value removal through the friction wind speed threshold to obtain the half-hour carbon dioxide flux data with untrustworthy values ​​removed; S5. Input the half-hourly carbon dioxide flux data and meteorological factor data after removing unreliable values ​​into the LightGBM gap filling model for training to obtain the trained LightGBM gap filling model and interpolated carbon dioxide flux data.

2. The CO2 flux data interpolation method integrating outlier detection and LightGBM according to claim 1, characterized in that: Step S1 specifically includes: S11. Set the sampling frequency and sampling time period, and use the eddy covariance system to collect carbon dioxide flux data in high-altitude cold areas; S12, using micro-meteorological observation equipment to collect micro-meteorological data in the high-cold area, and using a three-spline interpolation method to interpolate the micro-meteorological data to obtain interpolated micro-meteorological data; Among them, micrometeorological data include four-component radiation, photosynthetically active radiation, soil temperature, humidity, air pressure, two-dimensional ultrasonic wind speed, and air temperature and humidity; S13. Unify the time resolution of the interpolated micrometeorological data with the time resolution of the carbon dioxide flux data to obtain meteorological factor data.

3. The CO2 flux data interpolation method integrating outlier detection and LightGBM according to claim 2, characterized in that: Step S3 specifically includes: S31, setting relevant parameters of a single-class support vector machine based on anomaly detection, and inputting sample data to train the single-class support vector machine based on anomaly detection, to obtain a trained single-class support vector machine based on anomaly detection; Among them, the relevant parameters include the upper limit of the training error score and the lower limit of the support vector score, the kernel type, and the kernel function bandwidth; S32, inputting the half-hour carbon dioxide flux data into the trained single-class support vector machine based on anomaly detection to perform outlier detection, obtaining outlier labels and removing the half-hour carbon dioxide flux data with outlier labels, to obtain the half-hour carbon dioxide flux data with outliers removed.

4. The CO2 flux data interpolation method integrating outlier detection and LightGBM according to claim 3 is characterized in that: Step S4 specifically includes: S41, dividing the half-hour carbon dioxide flux data excluding abnormal values ​​into day and night, dividing the data greater than or equal to the set photosynthetic radiation threshold value in the half-hour carbon dioxide flux data excluding abnormal values ​​into daytime period data, and dividing the data less than the set photosynthetic radiation threshold value in the half-hour carbon dioxide flux data excluding abnormal values ​​into nighttime period data; S42, using the quantile method to divide the night-time data into a number of night-time data subsets of equal size; The amount of data in each nighttime data subset is equal; S43, dividing each nighttime data subset of equal size into a number of data intervals of equal size; S44, calculating the average half-hourly carbon dioxide flux data value of each data interval of equal size; S45, calculating the average value of the half-hourly carbon dioxide flux data of the last 10 data intervals of each data interval of equal size; S46. According to the friction wind speed value of the half-hour carbon dioxide flux data, obtain the friction wind speed value of each data interval of equal size and calculate the average value to obtain the average friction wind speed value of each data interval of equal size; S47, sequentially determine whether the average half-hour carbon dioxide flux data value of each data interval of equal size is higher than x times the average value of the half-hour carbon dioxide flux data of the last 10 intervals of each data interval of equal size; if so, use the average friction wind speed value of the current data interval as the friction wind speed threshold; otherwise, continue to determine until the next data interval is found to meet the condition, and use the average friction wind speed value of the data interval as the friction wind speed threshold; S48. Mark all night-time data that are less than the friction wind speed threshold as invalid night-time data, and obtain half-hourly carbon dioxide flux data with unreliable values ​​removed.

5. The CO2 flux data interpolation method integrating outlier detection and LightGBM according to claim 4 is characterized in that: Step S5 specifically includes: S51, using the meteorological factor data as the original data, and using the half-hourly carbon dioxide flux data with unreliable values ​​removed as the label data, to obtain a labeled original data set, and dividing the labeled original data set into a training set and a test set in proportion; S52. Set the basic parameters of the LightGBM gap filling model; Among them, the basic parameters include evaluation indicators, loss function type, number of leaf nodes, and boosting algorithm type parameters; S53, inputting the training set into the LightGBM gap filling model for training, and using the grid search method to perform grid search on the LightGBM gap filling model to obtain the best hyperparameter group of the LightGBM gap filling model, and using the best hyperparameter group to retrain the LightGBM gap filling model to obtain a trained LightGBM gap filling model; S54. Input the test set into the trained LightGBM gap filling model to interpolate missing values ​​and obtain interpolated half-hourly carbon dioxide flux data.

Images