An ozone precursor prediction method and system based on three-layer stacked ensemble learning

Abstract

The application provides an ozone precursor prediction method and system based on three-layer stacked ensemble learning, which comprises the following steps: using the first layer of the three-layer stacked ensemble learning model to perform feature screening and strengthening; using the second layer of the three-layer stacked ensemble learning model to perform XGBoost regression, ElasticNet regression and Ridge regression according to the principle of base learner; and using the third layer of the three-layer stacked ensemble learning model to perform meta-learner random forest regression. The application solves the technical problem of insufficient prediction accuracy of the prior art under the conditions of feature collinearity, small sample and nonlinear mixed relationship.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of machine learning and environmental gas detection, specifically to a method and system for predicting ozone precursors based on three-layer stacked ensemble learning. Background Technology

[0002] Currently, the problem of secondary pollutants is becoming increasingly prominent. Among them, ozone (O3) pollution has become a global environmental challenge. Unlike primary pollutants, ozone is not directly emitted, but rather formed in the atmosphere through complex photochemical reactions of precursors such as nitrogen oxides (NOx) and volatile organic compounds (VOCs). Under certain specific conditions, these photochemical reactions can intensify, leading to a rapid increase in ozone concentration. Ozone formation involves various VOCs. However, these VOCs have significantly different reactivity and volatility, resulting in vastly different contributions to ozone formation. Some VOCs, such as aromatic hydrocarbons and alkenes, are key precursors and play a dominant role in ozone formation, while the contributions of other VOCs are minimal. Therefore, accurately predicting the ozone formation mechanism, especially identifying key VOC precursors, is a prerequisite for effectively controlling ozone pollution. Ozone pollution can be controlled by controlling these key precursors.

[0003] Machine learning is an effective method for identifying ozone precursors, but existing machine learning methods face the following technical bottlenecks in ozone precursor prediction:

[0004] 1. Severe Multicollinearity Among Features: Volatile Organic Compounds (VOCs) typically originate from common pollution sources. The concentrations of various VOCs emitted from the same source change synchronously, and meteorological conditions also affect all features simultaneously, leading to high correlations among features. Traditional linear models, such as ordinary least squares, estimate variance inflation and unstable coefficients under these conditions. While Lasso can perform feature selection, it tends to randomly select one of the highly correlated features, easily overlooking key variables.

[0005] 2. Limited sample size and susceptibility to overfitting: Environmental monitoring data acquisition is costly, often requiring high-precision instruments to obtain high-quality datasets, resulting in a typically limited sample size. Multicollinearity further reduces the effective sample size, making a single model prone to overfitting on the training set and exhibiting severely insufficient generalization ability.

[0006] 3. Coexistence of linear and nonlinear relationships: There are both linear main effects and nonlinear interactions between VOCs and ozone, such as the synergistic effect of multiple VOCs. A single type of model is difficult to capture these two mixed characteristics simultaneously.

[0007] In existing technical solutions, linear models struggle to handle nonlinear relationships and are unstable in estimation under multicollinearity; single nonlinear models such as support vector machines and neural networks are prone to overfitting, exhibiting large model biases and complex parameter tuning; simple ensemble methods such as voting or averaging fail to fully utilize the complementary information between base learners; regularization methods such as Lasso and Ridge only perform feature processing or parameter shrinkage, failing to comprehensively address complex prediction problems. Therefore, there is an urgent need for a machine learning method that can comprehensively handle feature collinearity, small-sample overfitting, and mixed relationship modeling.

[0008] The existing invention patent application document CN117877599A, entitled "Method for Constructing a Coupled Box Model of Volatile Organic Compound (VOC) Generation and Particulate Matter Growth," describes a method that includes: first, compiling single gas-phase oxidation models of VOC precursors in polluted air into a single system and simultaneously simulating them under real atmospheric conditions to construct an organic gas-phase oxidation model; then, coupling the organic gas-phase oxidation model with a particulate matter kinetic condensation and growth model to construct a coupled box model of organic matter generation and particulate matter growth; finally, using atmospheric reactants and reaction conditions as input data, running the coupled box model and verifying its performance, and then analyzing the contribution mechanism of organic matter across the entire volatile range to particulate matter growth through model mechanism analysis. This existing technology employs a mechanism-driven model, but relies on assumed chemical reaction pathways and other physical or chemical rules, requiring a large amount of environmental monitoring data to predict future data, resulting in unsatisfactory performance in source tracing.

[0009] The existing invention patent application document CN121068843A, entitled "Method, System, Computer Equipment and Medium for Identifying Atmospheric Pollution Sources in Dust Situations," describes a method that includes: identifying the types of volatile organic compounds (VOCs) and fine particulate matter (PM2.5) in the atmosphere of a dusty scene to be identified within a target area, and measuring the concentrations of VOCs and PM2.5; calculating the proportion of mineral dust in the PM2.5 types and concentrations to obtain mineral dust quantification results, and determining the dust source contribution weight based on the increase in active metals in the mineral dust quantification results; and performing positive matrix factorization prediction on the VOC types, VOC concentrations, PM2.5 types, and PM2.5 concentrations based on the dust source contribution weights to determine the pollution source contribution ratio of the atmosphere of the dusty scene to be identified. This existing scheme uses a classic source apportionment method, which cannot solve the multicollinearity problem commonly found in VOCs datasets, and cannot output ozone concentrations at future times.

[0010] The existing invention patent application document CN119962762A, entitled "A Multi-Model Combination Air Quality Prediction Method and System Combining Dynamic Feature Screening," describes a method that includes: preprocessing historical meteorological and air quality data of a target area to generate historical meteorological and air quality data for target stations; screening multiple meteorological factors to determine multiple target meteorological factors; establishing and training a first prediction model and a second prediction model; training the first prediction model using a time-weighted loss function; and using a random forest model as the input to the second prediction model, which includes the output of the first prediction model. Based on the multiple target meteorological factors, it extracts meteorological time-series data for target stations from future meteorological data of the target area. Finally, it predicts future air quality data for target stations using the first and second prediction models based on the meteorological time-series data. This existing approach is a sequential hierarchical feature learning method that uses a complex GRU network. Such deep learning models are extremely prone to overfitting under small sample conditions.

[0011] In summary, existing technologies suffer from insufficient prediction accuracy under conditions of feature collinearity, small sample size, and nonlinear mixed relationships. Summary of the Invention

[0012] The technical problem to be solved by this invention is: how to solve the problem of insufficient prediction accuracy of existing technologies under conditions of feature collinearity, small sample size and nonlinear mixed relationships.

[0013] This invention solves the above-mentioned technical problems by employing the following technical solution: A method for predicting ozone precursors based on three-layer stacked ensemble learning, comprising: S1. Collect the environmental monitoring dataset. In the first layer of the three-layer stacked ensemble learning model, perform lasso regression on the environmental monitoring dataset to complete feature selection and obtain the filtered feature matrix. Add time-lag features and perform feature enhancement operations to obtain the enhanced feature matrix. S2. Based on the enhanced feature matrix and the target variable, train no less than two heterogeneous base learners in parallel. Each heterogeneous base learner independently learns the mapping relationship of the target variable and generates predicted values. The predicted values ​​are combined to obtain the meta-feature matrix and obtain the meta-features. Among them, the heterogeneous base learners include: XGBoost (eXtreme Gradient Boosting) base learner, ElasticNet (elastic network) base learner and Ridge (ridge-stabilized structure) base learner. S3. In the third layer of the three-layer stacked ensemble learning model, a meta-learner random forest regression is performed. Based on the meta-features, the random forest is trained to learn the optimal weighted combination of parameters to obtain an applicable prediction model and predict the ozone precursors.

[0014] This invention uses a three-layer stacked ensemble learning model for ozone precursor prediction. This model can identify key ozone precursors from volatile organic compounds (VOCs) under conditions of high dimensionality, high multicollinearity, and small sample size. Compared to existing technologies, this invention reduces bias: the different perspectives of multiple base learners reduce the systematic bias of a single model; it also reduces variance: by ensemble averaging the prediction fluctuations of each model, variance is reduced; and it enhances robustness: anomalous predictions from a single type of base learner do not dominate the final result, improving the robustness of the prediction algorithm. This invention not only solves the multicollinearity problem but also outputs accurate ozone concentrations for future timeframes.

[0015] In a more specific technical solution, during the Lasso regression of S1, feature selection is performed using an objective function through L1 regularization. During L1 regularization, a sharp-angled geometric structure is introduced through the L1 norm penalty term, solved by the coordinate descent method, and an automatic feature selection mechanism is executed. The soft threshold function is used to set the coefficients smaller than the regularization parameter λ to zero and to shrink the coefficients larger than the regularization parameter λ by an equal amount. Regularization parameter selection is performed, and the regularization parameter λ is determined through K-fold cross-validation. The intensity of feature selection is controlled based on the regularization parameter λ and the mean square error on the k-th fold validation set.

[0016] Through in-depth analysis, this invention uses Lasso regression as one of the base learners in the first layer for feature selection, thus solving the feature selection problem at different levels.

[0017] The three-layer stacked ensemble learning model used in this invention differs from existing technologies in its combination hierarchy and purpose. This invention employs a parallel ensemble framework, with multiple base learners independently learning the same problem. It uses a more robust tree model, which is more suitable for processing small sample data commonly found in the field of environmental monitoring.

[0018] In a more specific technical solution, in S1, during the process of adding time lag features and performing feature enhancement operations, the p' features in the selected feature matrix that have undergone lasso regression are enhanced to obtain enhanced features, and then processed to obtain the enhanced feature matrix.

[0019] Ozone formation is not an immediate result of precursor emissions, but rather the result of a photochemical cycle of NOx and VOCs over several hours. This invention adds a time lag feature after the transport delay from the emission source to the receiver region, which also involves the accumulation and transformation of intermediate products. Therefore, adding a time lag feature can more accurately reflect the true causal relationship between precursors and ozone.

[0020] This invention is based on feature enhancement operations and uses a purely data-driven approach for ensemble learning, which does not require assumptions about complex chemical mechanisms and is more suitable for prediction tasks.

[0021] In a more specific technical solution, XGBoost regression is performed in the XGBoost base learner; and a gradient boosting framework for the XGBoost base learner is set up. The XGBoost base learner uses a forward addition model to minimize the objective function by progressively adding decision trees. For the objective function after minimization, calculate the first derivative, second derivative, and tree complexity. Then, perform a second-order Taylor approximation based on the regularization terms of the first derivative, second derivative, and tree complexity to obtain the optimization objective in round t. Set the tree splitting criteria for the XGBoost base learner; Divide the candidate split sample set into the left subtree I L and right subtree I R Based on the tree splitting criterion, the penalty coefficient for the number of leaf nodes, and the L2 penalty for leaf weights, the splitting reward is defined; a greedy strategy is used to search for features and splitting points, and the split that maximizes the splitting reward is selected. The XGBoost base learner is configured with a regularization mechanism. The XGBoost base learner employs a three-layer regularization strategy, which regularizes the expression tree structure based on the learning rate and the objective function after minimization.

[0022] In a more specific technical solution, ElasticNet regression operations are performed in the ElasticNet base learner; Hybrid regularization is performed, where the regularization parameter α controls the weights of the L1 and L2 regularization terms, thus completing the ElasticNet fusion of L1 and L2 regularization operations. The coordinate descent method is used to solve the problem. The variables other than the j-th coefficient are fixed. The L2 term introduces an additional shrinkage effect by changing the second parameter of the soft threshold function. The unregularized optimal value of the variables other than the j-th coefficient is fixed, and the update operation of the j-th coefficient is completed.

[0023] Compared to existing technologies that simply use Lasso, ElasticNet's advantage lies in its group effect. When multiple features are highly correlated, Lasso tends to randomly select one, while ElasticNet tends to select all correlated features simultaneously. This process is specifically implemented through the stickiness function applied by the L2 term, making the coefficients of similar features similar.

[0024] In a more specific technical solution, in S2, the Ridge base learner obtains the closed-form solution of the Ridge regression based on the enhanced feature matrix, the minimized objective function, and the regularization parameters, thus completing the Ridge regression operation.

[0025] In a more specific technical solution, in S2, for at least two collinearity cases, when determining feature correlation, the low signal-to-noise ratio direction in the feature space is compressed during the Ridge regression process to complete the stable estimation operation; a bias-variance tradeoff operation is performed in the Ridge base learner.

[0026] The heterogeneity of the base learners used in this invention ensures learning from different perspectives, enabling meta-learners to obtain complementary advantages through weighted combination.

[0027] Compared to the original features, meta-features represent the optimal representations of each base learner, containing the information extracted by each base learner. Meta-learners operate within this high-level feature space, enabling them to learn the optimal weighted combination, which outperforms the simple averaging methods used in existing techniques.

[0028] In a more specific technical solution, S3 implements an integration mechanism, in which the random forest constructs T independent decision trees by aggregating Bagging and feature randomness through self-service aggregation. Implement a variance reduction mechanism, where the prediction variance of a single decision tree is σ. 2 Based on the prediction variance and the correlation coefficient between trees, the integrated prediction variance as T→∞ is calculated.

[0029] In a more specific technical solution, S3 determines feature importance. Random forests reduce parameters by calculating impurity and measure feature importance.

[0030] In a more specific technical solution, an ozone precursor prediction system based on three-layer stacked ensemble learning includes: The feature selection and enhancement module is used to collect environmental monitoring datasets. In the first layer of the three-layer stacked ensemble learning model, lasso regression is performed on the environmental monitoring datasets to complete feature selection and obtain the selected feature matrix. Time lag features are added to perform feature enhancement operations and obtain the enhanced feature matrix. The base learner regression module is used to train at least two heterogeneous base learners in parallel based on the enhanced feature matrix and the target variable. Each heterogeneous base learner independently learns the mapping relationship of the target variable and generates predicted values. The predicted values ​​are combined to obtain the meta-feature matrix and obtain the meta-features. The heterogeneous base learners include XGBoost base learner, ElasticNet base learner and Ridge base learner. The base learner regression module is connected to the feature selection and enhancement module. The meta-learner random forest regression module is used to perform meta-learner random forest regression in the third layer of the three-layer stacked ensemble learning model. Based on meta-features, it trains the random forest to learn the optimal weighted combination of parameters to obtain an applicable prediction model and predict ozone precursors. The meta-learner random forest regression module is connected to the base learner regression module.

[0031] The present invention has the following advantages over the prior art: This invention uses a three-layer stacked ensemble learning model for ozone precursor prediction. This model can identify key ozone precursors from volatile organic compounds (VOCs) under conditions of high dimensionality, high multicollinearity, and small sample size. Compared to existing technologies, this invention reduces bias: the different perspectives of multiple base learners reduce the systematic bias of a single model; it also reduces variance: by ensemble averaging the prediction fluctuations of each model, variance is reduced; and it enhances robustness: anomalous predictions from a single type of base learner do not dominate the final result, improving the robustness of the prediction algorithm. This invention not only solves the multicollinearity problem but also outputs accurate ozone concentrations for future timeframes.

[0032] Through in-depth analysis, this invention uses Lasso regression as one of the base learners in the first layer for feature selection, thus solving the feature selection problem at different levels.

[0033] The three-layer stacked ensemble learning model used in this invention differs from existing technologies in its combination hierarchy and purpose. This invention employs a parallel ensemble framework, with multiple base learners independently learning the same problem. It uses a more robust tree model, which is more suitable for processing small sample data commonly found in the field of environmental monitoring.

[0034] Ozone formation is not an immediate result of precursor emissions, but rather the result of a photochemical cycle of NOx and VOCs over several hours. This invention adds a time lag feature after the transport delay from the emission source to the receiver region, which also involves the accumulation and transformation of intermediate products. Therefore, adding a time lag feature can more accurately reflect the true causal relationship between precursors and ozone.

[0035] This invention is based on feature enhancement operations and uses a purely data-driven approach for ensemble learning, which does not require assumptions about complex chemical mechanisms and is more suitable for prediction tasks.

[0036] Compared to existing technologies that simply use Lasso, ElasticNet's advantage lies in its group effect. When multiple features are highly correlated, Lasso tends to randomly select one, while ElasticNet tends to select all correlated features simultaneously. This process is specifically implemented through the stickiness function applied by the L2 term, making the coefficients of similar features similar.

[0037] The heterogeneity of the base learners used in this invention ensures learning from different perspectives, enabling meta-learners to obtain complementary advantages through weighted combination.

[0038] Compared to the original features, meta-features represent the optimal representations of each base learner, containing the information extracted by each base learner. Meta-learners operate within this high-level feature space, enabling them to learn the optimal weighted combination, which outperforms the simple averaging methods used in existing techniques.

[0039] This invention solves the technical problem of insufficient prediction accuracy in the prior art under conditions of feature collinearity, small sample size, and nonlinear mixed relationships. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the basic steps of an ozone precursor prediction method based on three-layer stacked ensemble learning according to Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of data stream processing for an ozone precursor prediction method based on three-layer stacked ensemble learning according to Embodiment 1 of the present invention. Figure 3 This is a schematic diagram of the framework for parallel training of three heterogeneous base learners in the second layer of Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the weighted fusion mechanism of the random forest meta-learner in the third layer of Embodiment 1 of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Example 1 like Figure 1 and Figure 2 As shown, the present invention provides a method for predicting ozone precursors based on three-layer stacked ensemble learning, which includes the following basic steps: S1. Collect the environmental monitoring dataset. In the first layer of the three-layer stacked ensemble learning model, perform lasso regression on the environmental monitoring dataset to complete feature selection and obtain the filtered feature matrix. Add time-lag features and perform feature enhancement operations to obtain the enhanced feature matrix. In this embodiment, Lasso regression is used to filter variables in the environmental monitoring dataset. Specifically, because the emission sources emit multiple pollutants simultaneously, the independent variables in the environmental monitoring dataset are strongly correlated, increasing the difficulty for the model to understand which features truly drive the predictions and affecting model training. To reduce the impact of the correlation between independent variables on model training, the first layer of the three-layer stacked ensemble learning model designed in this embodiment uses Lasso regression to filter variables in the original environmental monitoring dataset.

[0043] Input data includes: the original feature matrix X raw ∈R N×p , where N is the number of samples, p is the original feature dimension, R represents a real number, and raw represents the original data; The processing includes: identifying p′ selective features from p original features, satisfying: p′ <p; The output data includes: the filtered feature matrix X′∈R N×p’ To build a simpler, more efficient, and interpretable three-layer stacked ensemble learning model.

[0044] In Lasso regression, feature selection is performed using the objective function through L1 regularization. In the basic principle of Lasso in this embodiment, Lasso achieves feature selection through L1 regularization, and the objective function is:

[0045] in, Describe the objective function. n Indicates the number of features. y Let X represent the output data, X represent the feature matrix, |||| represents the L2 norm, and || represents the L1 norm. Represents the regression coefficient vector. This represents the regularization parameter.

[0046] During L1 regularization, a sharp-angled geometric structure is introduced through the L1 norm penalty term, solved by the coordinate descent method, and an automatic feature selection mechanism is executed. The soft threshold function is used to set the coefficients smaller than the regularization parameter λ to zero and to shrink the coefficients larger than the regularization parameter λ by an equal amount. In this embodiment, compared to the traditional standard least squares method, the introduction of the L1 norm penalty term has a sharp-angled geometry. When the regularization parameter λ is sufficiently large, the coefficients of unimportant features are compressed to zero, thereby achieving an automatic feature selection mechanism. This property is particularly evident when solved using the coordinate descent method:

[0047] Among them, the soft threshold function For less than λ The coefficient is set to zero for parameters greater than the regularization parameter. λ The coefficients are shrunk by an equal amount. This represents the sign-determination function. Indicates the first j The new regression coefficients for each feature Indicates the first j eigenvectors X j transpose, y Represents the target variable vector. Indicates except the first j The feature matrix outside of the features, Indicates except the first j The regression coefficient vector outside of the coefficients.

[0048] Regularization parameter selection is performed by... K Cross-validation determines regularization parameters λ According to the regularization parameter λ The mean squared error on the k-th fold validation set controls the intensity of feature selection.

[0049] In this embodiment, regularization parameters are selected; specifically, the regularization parameters... λ Control the intensity of feature selection. (By...) K Cross-validation determines regularization parameters λ :

[0050] Among them, MSE k (λ) is the first k The mean squared error on the validation set, arg min represents the value of the independent variable that minimizes the objective function. This represents the optimal regularization parameter that minimizes the average cross-validation error. We ultimately use this parameter as the training parameter for the model.

[0051] An excessively large regularization parameter λ can lead to the accidental deletion of important features, resulting in high bias. Conversely, an excessively small regularization parameter λ cannot effectively filter out noisy features, leading to high variance in the model.

[0052] During the feature enhancement process, which involves adding time lag features, the p' features from the selected feature matrix that have undergone lasso regression are enhanced to obtain enhanced features, which are then processed to obtain the enhanced feature matrix.

[0053] Input data includes: a filter feature matrix X′∈R N×p’ , where N is the number of samples and p' is the number of variables obtained by lasso regression screening; The processing includes: processing the lasso regression results Each feature is selected, and feature enhancement is performed to obtain... One enhanced feature; Output data includes: enhanced feature matrix To construct a more efficient, interpretable, high-performing, and stable three-layer stacked ensemble learning model.

[0054] S2. Based on the enhanced feature matrix and the target variable, train no less than two heterogeneous base learners in parallel. Each heterogeneous base learner independently learns the mapping relationship of the target variable and generates predicted values. The predicted values ​​are combined to obtain the meta-feature matrix and obtain the meta-features. Among them, the heterogeneous base learners include: XGBoost (eXtreme Gradient Boosting) base learner, ElasticNet (elastic network) base learner and Ridge (ridge-stabilized structure) base learner. The input data includes: the enhanced feature matrix output from the first layer. and the target variable y∈R N ; like Figure 3 As shown, in this layer, three heterogeneous base learners are trained in parallel, and each heterogeneous base learner independently learns the mapping relationship of the target variable.

[0055] The process includes simultaneously training three heterogeneous base learners: XGBoost, ElasticNet, and Ridge. The output data includes: predictions generated by three heterogeneous base learners. , , ∈R N Combined into a feature matrix X meta : X meta =[ , , ]∈R N×3 in, This represents the prediction value of the XGBoost base learner. This represents the prediction value of the ElasticNet base learner. This represents the prediction value of the Ridge base learner; In this embodiment, XGBoost regression is performed in the XGBoost base learner; and the gradient boosting framework of the XGBoost base learner is set. The XGBoost base learner uses a forward addition model to minimize the objective function by progressively adding decision trees. In the gradient boosting framework of this embodiment, XGBoost employs a forward addition model, minimizing the objective function by progressively adding decision trees:

[0056] in, Indicates the first i The predicted value for each sample, Indicates the first k The first decision tree i The predicted value for each sample, x i Indicates the first i The feature vector of each sample.

[0057] For the objective function after minimization, calculate the first derivative, second derivative, and tree complexity. Then, perform a second-order Taylor approximation based on the complexity regularization terms of the first derivative, second derivative, and tree, and obtain the... t The optimization goal of the wheel; No. t Optimization goal of the wheel We obtain the following through the second-order Taylor approximation:

[0058] in, and The first i First and second derivatives of heterogeneous basis learners Ω(f t ) For the first t The complexity regularization term for trees, Indicates the first i The prediction values ​​of a heterogeneous base learner Indicates the preceding t-1 The cumulative predicted value of the trees, Indicates the first t tree to the first i The predicted value for each sample, f t Indicates the first t A decision tree, Ω The regularization term represents the complexity.

[0059] Compared to traditional GBDT which only uses the first derivative, this invention uses the second derivative information to enable the XGBoost base learner to perform Newton's method updates more accurately, thus accelerating the convergence speed.

[0060] Set the tree splitting criteria for the XGBoost base learner; Divide the candidate split sample set into the left subtree I L and right subtree I RBased on the tree splitting criterion, the penalty coefficient for the number of leaf nodes, and the L2 penalty for leaf weights, the splitting reward is defined; a greedy strategy is used to search for features and splitting points, and the split that maximizes the splitting reward is selected.

[0061] In the tree splitting criterion of this embodiment, the candidate split divides the sample set into the left subtree I. L and right subtree I R The splitting payoff is defined as:

[0062] in, γ The penalty coefficient for the number of leaf nodes. λ The L2 penalty is applied to the leaf weights. A greedy strategy is used to search all features and split points, selecting the split that maximizes the gain. G L This represents the sum of the first derivatives of the left subtree. G R This represents the sum of the first derivatives of the right subtree. H L This represents the sum of the second derivatives of the left subtree. H R This represents the sum of the second derivatives of the right subtree.

[0063] The XGBoost base learner is configured with a regularization mechanism. The XGBoost base learner employs a three-layer regularization strategy, which regularizes the expression tree structure based on the learning rate and the objective function after minimization.

[0064] In the regularization mechanism of this embodiment, the XGBoost base learner adopts a three-layer regularization strategy: Tree structure regularization:

[0065] in, T Indicates the number of decision trees. Indicates the preceding t-1 The cumulative predicted value of the trees, w j Indicates the first j The weights of the leaf nodes; Shrinkage: ,in η∈(0,1) The learning rate; Subsampling: Each training round uses only a subset of samples and features, introducing randomness; These mechanisms effectively prevent overfitting and improve generalization ability.

[0066] Perform ElasticNet regression operations within the ElasticNet base learner; Perform hybrid regularization, where regularization parameters are used. α Control the weights of the L1 and L2 regularization terms to complete the ElasticNet fusion L1 and L2 regularization operation; In this embodiment of ElasticNet regression, hybrid regularization is performed, with ElasticNet combining L1 and L2 regularization:

[0067] Wherein, regularization parameters α∈[0,1] Control the weights of L1 and L2. When α=0 For Ridge, α=1 For Lasso, The objective loss function is defined as the function that minimizes the objective loss function. β Represents the regression coefficient vector ， express L1 of β norm , express β of L2 The square of the norm.

[0068] When x1≈x2, the ElasticNet solution satisfies β1≈β2, while Lasso can give combinations of β1≠0, β2=0, or the opposite.

[0069] The coordinate descent method is used to solve the problem, with the first position fixed. j The variables other than the coefficients include the L2 term, which introduces an additional contraction effect by changing the second parameter of the soft threshold function, while the first coefficient is fixed. j Find the unregularized optimal value of the variables other than the nth coefficient, and complete the process for finding the nth coefficient. j Update operations for each coefficient; In the solution algorithm of this embodiment, the coordinate descent method is used to solve the problem, while fixing the first... j When the variable other than the coefficient is used for the first... j Each coefficient is updated:

[0070] Where is the unregularized optimal value with other coefficients fixed. The L2 term introduces an additional contraction effect by changing the second parameter of the soft threshold function.

[0071] In the Ridge base learner, the closed-form solution of the Ridge regression is obtained based on the enhanced feature matrix, the minimized objective function, and the regularization parameters, thus completing the Ridge regression operation.

[0072] In the regression process of the Ridge base learner in this embodiment, a closed-form solution is calculated; specifically, the closed-form solution of the Ridge regression is:

[0073] By sending to X T Adding nλI to the diagonal of X ensures that the matrix is ​​always invertible and well-defined, yielding stable solutions even in the case of multicollinearity. n represents the number of samples, λ represents the regularization parameter, and I represents the identity matrix.

[0074] For at least two collinearity cases, when determining feature correlation, the low signal-to-noise ratio directions in the feature space are compressed during the Ridge regression process to complete the stable estimation operation; a bias-variance tradeoff operation is performed in the Ridge base learner. In this embodiment, to address multicollinearity, when features are highly correlated, the variance estimated by OLS (Ordinary Least Squares) will be very large, and the condition number is: κ(X) T X)→∞. κ represents the ratio of the largest eigenvalue to the smallest eigenvalue of the matrix within the parentheses following it; Ridge regression compresses low signal-to-noise ratio directions in the feature space to stabilize the estimation.

[0075] From the perspective of spectral decomposition, let eigenvalues Then Ridge estimates for:

[0076] The coefficients in the eigenvalue direction are significantly compressed, thereby reducing variance. Among these, X T X Represents the characteristic covariance matrix. Represents the largest eigenvalue. Represents the smallest eigenvalue. Indicates the first j 1 eigenvalue, Indicates the first j 1 eigenvector.

[0077] In this embodiment, a bias-variance tradeoff is performed; specifically, Ridge achieves a significant reduction in variance by increasing bias. As λ increases, bias monotonically increases while variance monotonically decreases, with the optimal λ value being where the sum of the two is minimized. Ridge often outperforms OLS in the case of multicollinearity, even though OLS is unbiased.

[0078] S3. In the third layer of the three-layer stacked ensemble learning model, a meta-learner random forest regression is performed. Based on the meta-features, the random forest is trained to learn the optimal weighted combination of parameters to obtain an applicable prediction model and predict the ozone precursor prediction results. In this embodiment, the input data includes: meta-features X generated by the second layer of the three-layer stacked ensemble learning model. meta ∈R N×3 ; During the processing, the random forest is trained to learn the optimal weighted combination method; Output data includes: final prediction results of ozone precursors. ∈R N .

[0079] The ensemble mechanism is implemented, where the random forest constructs T independent decision trees by aggregating Bagging and feature randomness through bootstrapping:

[0080] Among them, each tree h t Independent growth on randomly sampled subsets of data and random subsets of features. This represents the prediction result of the random forest.

[0081] Implement a variance reduction mechanism, where the prediction variance of a single decision tree is . σ 2 Calculate based on prediction variance and inter-tree correlation coefficient. T→∞ Time-integrated prediction variance:

[0082] Where ρ is the correlation coefficient between trees. Due to the correlation between trees, it is impossible to eliminate... ρσ 2 Items, but compared to a single tree σ 2 ,pass T The average of decision trees can significantly reduce overall variance, especially when ρ It works better when it is smaller. This represents the variance of the integrated predictions.

[0083] In this embodiment, the outputs of the three heterogeneous base learners in the second layer are used as meta-features, which are more diverse than a single decision tree and further reduce the variance of the ensemble prediction.

[0084] To determine feature importance, random forests reduce parameters by calculating impurity and measure feature importance.

[0085] In this embodiment, the random forest calculates the value of impurity reduction. Measuring the importance of features :

[0086] in, splits on X j Indicates in featuresX j All splits on the base learner; this provides a quantitative assessment of the importance of the meta-learner to each output of the base learner. By observing the importance coefficients of the three heterogeneous base learners, it is possible to understand which base learner's prediction contributes the most to the final result.

[0087] like Figure 4 As shown, in this embodiment, the three-layer stacked ensemble learning model adopts a collaborative mechanism of stacked ensemble, and the base learners are diverse, with each of the three base learners having its own characteristics; specifically: XGBoost is nonparametric and can capture information about complex nonlinearities and feature interactions. ElasticNet has specific parameterization features, strong interpretability, and is relatively robust to multicollinearity application scenarios; Ridge is simple and linear, computationally efficient, and performs stably in scenarios with strong feature correlation. This heterogeneity ensures learning from different perspectives, enabling meta-learners to gain complementary advantages through weighted combinations.

[0088] The advantages of this invention employing a two-layer learning approach: Information flow from the first layer to the second layer: Original feature X raw Lasso Filtering Feature X′ after filtering and enhancement Three base learners trained in parallel Meta-feature X meta ; Meta-feature characteristics generated in the second layer: X meta =[ , , ]∈R N×3 ; Information flow from the second to the third layer: Meta-feature X meta Random Forest Training Learning optimal weights Ozone precursor prediction results ; This invention improves the model's generalization performance, and the advantages of stacked integration are: Bias reduction: The different perspectives of multiple heterogeneous base learners reduce the systematic bias of a single model; Variance reduction: predicting volatility by integrating and averaging learners from heterogeneous bases; Robustness enhancement: Abnormal predictions from a heterogeneous base learner will not dominate the final result.

[0089] In an ideal scenario, the learners from each heterogeneous base are unbiased and have low correlation, and the mean square error of the stack is... for:

[0090] Here, "stack" represents stacking. e k Indicates the first k The prediction error of each base learner e l This represents the prediction error of the l-th base learner. This represents the covariance between errors. Let represent the mean square error of the k-th base learner.

[0091] in, The average of the base learner error, The variance reduction term contributing to negative covariance. It exhibits low correlation, indicating that the higher the diversity among base learners, the more pronounced the variance reduction effect.

[0092] Example 2 This embodiment elucidates the role of volatile organic compounds (VOCs) in ozone formation precursors in the Beijing area. The purpose of this invention is to construct a three-layer stacked ensemble learning model—feature engineering, base model, and meta-model—that accurately predicts key VOC precursors for ozone formation. This model can reveal pollution source mechanisms, identify which VOCs are key contributors to ozone formation, and help understand the environmental impact of different pollution sources. It can also support pollution control, providing a scientific basis for targeted pollution source control, enabling environmental protection departments to prioritize the control of key precursor emissions, rationally allocate limited environmental resources, and improve governance efficiency. Furthermore, it improves forecasting models, supports the improvement of air quality forecasting models, and enhances the accuracy and early warning capabilities of ozone pollution forecasts.

[0093] In this embodiment, the data selected came from data collected by the Science Island Branch of the University of Science and Technology of China. Ozone concentration and the concentrations of various volatile organic compounds were collected using high-precision instruments in a chemical industrial park. Furthermore, the pollutants measured by the instruments varied on different days, and data was collected every 50 minutes, resulting in a large dataset spanning nearly two years. This high-quality and large-volume dataset provided excellent validation for the present invention.

[0094] In the implementation of this embodiment, a machine learning model is built, and the building operation includes: Install Python 3.8 and the libraries required for this invention; The Stacking framework is built with Lasso regression as the first layer for variable selection, XGBoost, ElasticNet, and Ridge as the base learners as the second layer, and random forest as the meta learner as the third layer.

[0095] Except for Random Forest, where the number of decision trees is manually set to 100, the hyperparameters of other machine learning models are determined by grid search.

[0096] Write the interface for input and output data to prepare for subsequent model training. This deep learning model is denoted as ML (Machine Learning).

[0097] In the implementation of this embodiment, the original dataset is also analyzed using an ML model, wherein: Data alignment was performed. Specifically, since the volatile organic compounds measured in this dataset differed due to different dates, the dataset was first aligned in this embodiment. The set of measured substances with the highest frequency was selected as the independent variable. After data alignment, the data and its size were (5987, 39), where the last column was the concentration of the target variable ozone, and the first 38 were the independent variables, i.e., the names of volatile organic compounds.

[0098] Data cleaning and standardization are performed. Specifically, some sensitive machine learning models are highly sensitive to outliers, such as isolated and missing values, which can affect model training. After outlier processing, the dataset size X∈R is determined. 2926×39 Because the dimensions and values ​​of different independent variables vary significantly, they cannot be directly applied to machine learning. The dataset should be standardized to ensure that the characteristics of the data itself do not affect machine learning predictions. This example uses the StandardScaler data standardization method to standardize the original features in the dataset, i.e., Z-score standardization. The standardization formula is shown below:

[0099] in, Represents the standardized eigenvalues. Represents the original feature values. This represents the average value of the column containing the data. This represents the standard deviation of the column containing the data. It transforms the values ​​of each feature for all variables into a distribution with a mean of 0 and a standard deviation of 1.

[0100] Feature selection and enhancement are performed; specifically, Lasso regression is used to select variables from the original feature matrix, and the optimal regularization parameter is determined through 5-fold cross-validation. λThis minimizes the mean squared error of cross-validation. After screening, 14 important features are retained, and the coefficients of some weakly correlated features are compressed to zero.

[0101] Based on the screening features, and according to the time lag characteristics of photochemical reactions, five time lag features are constructed for each VOC screening feature, namely the time lag features at the first 1 time point, the second time point, the third time point, the fifth time point, and the seventh time point, forming an enhanced feature matrix X′′∈R. 2919×84 .

[0102] Parallel training of the base learners is performed. Specifically, the aforementioned enhanced feature matrix X′′ is used as input, and the dataset is divided into training and test sets in an 8:2 ratio. Within the training set, 5-fold cross-validation is used for hyperparameter selection, and three heterogeneous base learners—XGBoost, ElasticNet, and Ridge—are trained separately. After independently fitting on the training set, each of the three heterogeneous base learners outputs its predicted values ​​for the training samples, forming the meta-feature matrix X. meta .

[0103] Perform meta-learner fusion operation; specifically, fuse the meta-feature matrix X. meta As input, a random forest regression is trained as the meta-learner. The random forest is set to have 100 trees, with a maximum of 3 features per tree, corresponding to three heterogeneous base learners, and a minimum of 5 leaf samples. After training, the random forest learns the optimal weighted combination of the predictions from the three heterogeneous base learners, and outputs the ozone precursor prediction results.

[0104] In this embodiment, the results are evaluated. In addition to the designed three-layer stacked ensemble learning model, the present invention also sets up a traditional method model as a control group. The design significance of each control model is shown in the table below:

[0105] The goodness-of-fit coefficient R of the traditional method 2 While it was difficult to overcome the 0.8 threshold, the three-layer stacked ensemble learning approach designed in this application ultimately achieved superior results in R. 2 The mean squared error (RMSE) was 0.9403, and the model's performance on the test set was basically consistent with that on the training set, with no obvious overfitting, indicating that the model has good generalization ability. The RMSE (Root Mean Squared Error) was 0.0276, the MAE (Mean Absolute Error) was 0.0214, and the MSE (Mean Squared Error) was 0.0008.

[0106] In summary, this invention uses a three-layer stacked ensemble learning model for ozone precursor prediction. This model can identify key ozone precursors from volatile organic compounds under conditions of high dimensionality, high multicollinearity, and small sample size. Compared to existing technologies, this invention reduces bias: the different perspectives of multiple base learners reduce the systematic bias of a single model; it also reduces variance: by ensemble averaging the prediction fluctuations of each model, variance is reduced; and robustness is enhanced: anomalous predictions from a single type of base learner do not dominate the final result, improving the robustness of the prediction algorithm. This invention not only solves the multicollinearity problem but also outputs accurate ozone concentrations for future timeframes.

[0107] Through in-depth analysis, this invention uses Lasso regression as one of the base learners in the first layer for feature selection, thus solving the feature selection problem at different levels.

[0108] The three-layer stacked ensemble learning model used in this invention differs from existing technologies in its combination hierarchy and purpose. This invention employs a parallel ensemble framework, with multiple base learners independently learning the same problem. It uses a more robust tree model, which is more suitable for processing small sample data commonly found in the field of environmental monitoring.

[0109] Ozone formation is not an immediate result of precursor emissions, but rather the result of a photochemical cycle of NOx and VOCs over several hours. This invention adds a time lag feature after the transport delay from the emission source to the receiver region, which also involves the accumulation and transformation of intermediate products. Therefore, adding a time lag feature can more accurately reflect the true causal relationship between precursors and ozone.

[0110] This invention is based on feature enhancement operations and uses a purely data-driven approach for ensemble learning, which does not require assumptions about complex chemical mechanisms and is more suitable for prediction tasks.

[0111] Compared to existing technologies that simply use Lasso, ElasticNet's advantage lies in its group effect. When multiple features are highly correlated, Lasso tends to randomly select one, while ElasticNet tends to select all correlated features simultaneously. This process is specifically implemented through the stickiness function applied by the L2 term, making the coefficients of similar features similar.

[0112] The heterogeneity of the base learners used in this invention ensures learning from different perspectives, enabling meta-learners to obtain complementary advantages through weighted combination.

[0113] Compared to the original features, meta-features represent the optimal representations of each base learner, containing the information extracted by each base learner. Meta-learners operate within this high-level feature space, enabling them to learn the optimal weighted combination, which outperforms the simple averaging methods used in existing techniques.

[0114] This invention solves the technical problem of insufficient prediction accuracy in the prior art under conditions of feature collinearity, small sample size, and nonlinear mixed relationships.

[0115] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting ozone precursors based on three-layer stacked ensemble learning, characterized in that, The method includes: S1. Collect the environmental monitoring dataset. In the first layer of the three-layer stacked ensemble learning model, perform lasso regression on the environmental monitoring dataset to complete feature selection and obtain the filtered feature matrix. Add time lag features and perform feature enhancement operations to obtain the enhanced feature matrix. In Lasso regression, feature selection is performed using the objective function through L1 regularization. In the L1 regularization process, a sharp-angled geometric structure is introduced through the L1 norm penalty term, solved by the coordinate descent method, and an automatic feature selection mechanism is executed. The soft threshold function is used to set the coefficients smaller than the regularization parameter λ to zero and to shrink the coefficients larger than the regularization parameter λ by an equal amount. Regularization parameter selection is performed, and the regularization parameter λ is determined through K-fold cross-validation. The intensity of feature selection is controlled based on the regularization parameter λ and the mean square error on the k-th fold validation set. During the feature enhancement operation, the time lag feature is added; the p' features in the selected feature matrix that have undergone lasso regression are enhanced to obtain enhanced features, and the enhanced feature matrix is ​​obtained through further processing. S2. Based on the enhanced feature matrix and the target variable, train no less than two heterogeneous base learners in parallel. Each heterogeneous base learner independently learns the mapping relationship of the target variable and generates predicted values. The predicted values ​​are combined to obtain the meta-feature matrix and obtain the meta-features. The heterogeneous base learners include: XGBoost base learner, ElasticNet base learner and Ridge base learner. S3. In the third layer of the three-layer stacked integrated learning model, a meta-learner random forest regression is performed. Based on the meta-features, the random forest learner is trained to obtain the optimal weighted combination of learning parameters to obtain an applicable prediction model and predict the ozone precursor prediction results. An ensemble mechanism is implemented, in which the random forest constructs T independent decision trees by aggregating Bagging and feature randomness through a bootstrap aggregation. Implement a variance reduction mechanism, where the prediction variance of a single decision tree is σ. 2 Based on the prediction variance and the correlation coefficient between trees, the integrated prediction variance as T→∞ is calculated.

2. The ozone precursor prediction method based on three-layer stacked ensemble learning according to claim 1, characterized in that, In step S2, XGBoost regression is performed in the XGBoost base learner; and the gradient boosting framework of the XGBoost base learner is set. The XGBoost base learner adopts a forward addition model, which minimizes the objective function by progressively adding decision trees; For the objective function after minimization, the first derivative, second derivative, and tree complexity are obtained. A second-order Taylor approximation is performed based on the complexity regularization terms of the first derivative, second derivative, and tree to obtain the optimization objective in the t-th round. Set the tree splitting criteria for the XGBoost base learner; Divide the candidate split sample set into the left subtree I L and right subtree I R Based on the tree splitting criterion, the penalty coefficient for the number of leaf nodes, and the L2 penalty for leaf weights, the splitting reward is defined; a greedy strategy is used to perform feature and splitting point search operations to select the split that maximizes the splitting reward. The XGBoost base learner is configured with a regularization mechanism, wherein the XGBoost base learner employs a three-layer regularization strategy, which regularizes the expression tree structure based on the learning rate and the objective function after minimization.

3. The ozone precursor prediction method based on three-layer stacked ensemble learning according to claim 1, characterized in that, In the ElasticNet base learner, perform ElasticNet regression operations; Hybrid regularization is performed, where the regularization parameter α controls the weights of the L1 and L2 regularization terms, thus completing the ElasticNet fusion of L1 and L2 regularization operations. The coordinate descent method is used to solve the problem. The variables other than the j-th coefficient are fixed. The L2 term introduces an additional shrinkage effect by changing the second parameter of the soft threshold function. The unregularized optimal value of the variables other than the j-th coefficient is fixed, and the update operation of the j-th coefficient is completed.

4. The ozone precursor prediction method based on three-layer stacked ensemble learning according to claim 1, characterized in that, In the Ridge base learner, the closed-form solution of the Ridge regression is obtained based on the enhanced feature matrix, the minimized objective function, and the regularization parameter, thus completing the Ridge regression operation.

5. The ozone precursor prediction method based on three-layer stacked ensemble learning according to claim 1, characterized in that, For at least two collinearity scenarios, when determining feature correlation, the low signal-to-noise ratio directions in the feature space are compressed during Ridge regression to complete a stable estimation operation; a bias-variance tradeoff operation is performed on the Ridge base learner.

6. The ozone precursor prediction method based on three-layer stacked ensemble learning according to claim 1, characterized in that, In step S3, feature importance is determined, wherein the random forest measures the feature importance by calculating impurity reduction parameters.

7. An ozone precursor prediction system based on three-layer stacked ensemble learning, used to execute the ozone precursor prediction method based on three-layer stacked ensemble learning as described in any one of claims 1 to 6, characterized in that, The system includes: The feature selection and enhancement module is used to collect environmental monitoring datasets. In the first layer of the three-layer stacked ensemble learning model, lasso regression is performed on the environmental monitoring datasets to complete feature selection and obtain the selected feature matrix. Time lag features are added to perform feature enhancement operations to obtain the enhanced feature matrix. The base learner regression module is used to train at least two heterogeneous base learners in parallel based on the enhanced feature matrix and the target variable. Each heterogeneous base learner independently learns the mapping relationship of the target variable, generates predicted values, and combines the predicted values ​​to obtain the meta-feature matrix and acquire meta-features. The heterogeneous base learners include XGBoost base learners, ElasticNet base learners, and Ridge base learners. The base learner regression module is connected to the feature selection and enhancement module. The meta-learner random forest regression module is used to perform meta-learner random forest regression in the third layer of the three-layer stacked ensemble learning model. Based on the meta-features, it trains the random forest to learn the optimal weighted combination of parameters to obtain an applicable prediction model and predict the ozone precursor prediction results. The meta-learner random forest regression module is connected to the base learner regression module.

Images