Carbon emission prediction and threshold identification method based on space blocking random forest

Abstract

The invention discloses a carbon emission prediction and threshold identification method based on a space blocking random forest, and the method comprises the steps: collecting carbon emission and multi-source influence factor values of the carbon emission with a county region as an analysis unit, and carrying out the preprocessing of the carbon emission and the multi-source influence factor values, and building a data set; all county areas of the data set are divided into a plurality of connected blocks according to spatial proximity, then all connected blocks are divided into K sets, the county areas of each set are similar in number, and data samples form a data subset; respectively taking each data subset as a verification set and all other data subsets as training sets, training K random forest models by adopting a cross validation method, and training a carbon emission prediction model after performance evaluation is qualified; calculating an SHAP value matrix of a corresponding test set by using each trained random forest model, fitting an SHAP curve of each influence factor, and extracting a threshold value; and using the carbon emission prediction model to predict the carbon emission of the target county. According to the method, the prediction stability, generalization ability and SHAP interpretation consistency of the carbon emission prediction model can be improved.

Description

Technical Field

[0001] This invention belongs to the field of environmental science and geographic information system technology, specifically relating to a method for carbon emission prediction and threshold identification based on spatially blocked random forest. Background Technology

[0002] Existing studies extensively employ statistical regression methods such as the Kaya identity and the STIRPAT model to analyze carbon emission drivers from the perspectives of macroeconomic structure and energy intensity. However, these studies often remain at the provincial or higher administrative level, making it difficult to answer the question of "what should be adjusted within the county and where should it be adjusted."

[0003] With the development of remote sensing and geographic information technology, spatial data such as nighttime light, land cover, and topography are widely used to construct spatial proxies for CO2 emissions; machine learning methods (random forests, gradient boosting trees, support vector machines, multilayer perceptrons, etc.) are also beginning to be used to model complex nonlinear relationships, improving prediction accuracy to some extent. However, at the county scale, especially when geared towards planning decisions, current methods still have the following shortcomings:

[0004] ① Insufficient handling of the combination of land structure variables. County-level land use structure is usually given in the form of "the proportion of land area of ​​each type", which is a typical combination of data (each component is non-negative and the sum is 1). Many studies directly treat these proportions as ordinary independent variables and include them in regression or machine learning models, ignoring the simplex constraint. This can easily lead to misleading correlations and unstable importance rankings, affecting the interpretability of the model.

[0005] ② Spatial correlation is not fully considered, leading to a "leakage" problem in validation. Many machine learning applications still use random partitioning of training and test sets for cross-validation, failing to consider the spatial autocorrelation of county-level emissions. Adjacent regions often have similar terrain, industries, and infrastructure. Random partitioning will group spatially similar samples into different partitions, resulting in overly optimistic accuracy assessments and making it difficult to guarantee the model's generalization ability to unknown areas.

[0006] ③ Insufficient interpretability, making it difficult to extract actionable thresholds and "operational windows". Existing machine learning research mostly stays at the level of "ranking the importance of variables", lacking systematic analysis of nonlinear response curves, and is unable to give quantitative conclusions such as "in which range of changes in a certain land structure indicator will lead to drastic changes in emissions, or in which range it will basically saturate", resulting in relatively crude policy implications. Summary of the Invention

[0007] This invention provides a carbon emission prediction and threshold identification method based on spatially blocked random forest, which can improve the prediction stability of carbon emission prediction models and the consistency of SHAP interpretation.

[0008] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0009] A method for carbon emission prediction and threshold identification based on spatially blocked random forest includes:

[0010] Using counties as the unit of analysis, we collected carbon emissions and their multi-source influencing factors, preprocessed the collected data, and constructed a dataset.

[0011] The dataset is divided into several connected components based on spatial proximity. Then, all connected components are divided into K sets, with a similar number of counties in each set. The data samples of all counties in each set constitute a data subset.

[0012] K random forest models were trained using cross-validation with each data subset as the validation set and all other data subsets as the training set. After the performance evaluation was qualified, a carbon emission prediction model was trained based on the dataset.

[0013] The SHAP value matrix of the corresponding test set is calculated using the K random forest models obtained from training, the SHAP dependency curves of each influencing factor are fitted, and the segmented thresholds are extracted.

[0014] Use carbon emission prediction models to predict carbon emissions in the target county.

[0015] Furthermore, the multi-source influencing factors include land use structure, as well as nighttime light distribution, topographic distribution, road density, population size, and / or GDP.

[0016] Furthermore, the land use structure, whose numerical value refers to the share of various land use and functional structures, includes the following preprocessing steps:

[0017] First, the values ​​of land use structure and other influencing factors are uniformly projected onto an equal-area coordinate system;

[0018] Then, the shares of each land use and functional structure obtained by projecting onto the equal-area coordinate system are replaced with smaller values, and then CLR transformation is performed to obtain the representation of each share in Euclidean space:

[0019] ; ; In the formula, and The first The first county The numerical representation of the share of land use and functional structure after small value substitution and CLR transformation. Indicates the number of types of land use and functional structure. Indicates the first A share vector composed of the share of all land use and functional structure in each county. Indicates the first Share vector of each county The geometric mean of each component is used as the centering term in the CLR transform.

[0020] Furthermore, the preprocessing of nighttime light distribution and terrain distribution includes:

[0021] First, the values ​​of nighttime light distribution / terrain distribution and other influencing factors are projected onto an equal-area coordinate system;

[0022] Then, the nighttime light distribution / terrain distribution of different counties obtained by projecting onto the equal area coordinate system is resampled on a unified grid, and the sum or mean is calculated as the nighttime light / terrain data of each county.

[0023] Furthermore, preprocessing of various types of collected data also includes performing log(1+x) transformation and standardization on all non-combinatorial variables.

[0024] Furthermore, using clustering or regular grid methods, all counties are divided into several connected blocks based on spatial proximity.

[0025] Furthermore, using each influencing factor as the independent variable and the SHAP value as the dependent variable, the SHAP dependency curves of each influencing factor are fitted based on the influencing factor values ​​in the test set and the corresponding calculated SHAP values.

[0026] Furthermore, the SHAP dependency curve was fitted using a locally weighted regression method.

[0027] Furthermore, the method for extracting the segmented thresholds of influencing factors is as follows:

[0028] First, the first j Factors x j The SHAP dependency curve is represented as Calculate the SHAP dependency curve first derivative ;

[0029] Then calculate the absolute value of the derivative. The threshold value at which it is distributed at a preset high quantile is denoted as δ. high The threshold values ​​distributed at the preset low quantiles are denoted as δ. low , satisfying 0 < δ low <δ high If there exists that satisfy Furthermore, if the value of an influencing factor undergoes a sign change, that value is recorded as an inflection point.

[0030] Then satisfy ≥δ high The range of values ​​for influencing factors is denoted as the sensitive zone of the influencing factors; [the following is a list of parameters] that satisfy [the following is a list of parameters]: ≤δ low The range of influencing factors is denoted as the plateau region; the range of values ​​for δ is defined as the plateau region. low < <δ high The range of values ​​for influencing factors is denoted as the shoulder area of ​​the influencing factors, that is, the range from the sensitive area to the plateau area.

[0031] Finally, the influencing factor values ​​corresponding to the boundary points between the sensitive area and the shoulder area, and between the shoulder area and the plateau area, are used as the segmented threshold outputs.

[0032] Furthermore, after extracting the segmented thresholds, the method also includes: comparing the values ​​of each influencing factor in the target county with the corresponding segmented thresholds to determine the segment in which the values ​​of each influencing factor in the carbon emissions of the target county are located.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. By combining data and performing CLR transformation, share variables such as land structure and functional structure can be modeled in Euclidean space, thereby improving the predictive stability of carbon emission prediction models and the consistency of SHAP interpretation.

[0035] 2. All counties in the study area are divided into connected components according to spatial proximity to construct spatially blocked cross-validation folds. This spatially blocked random forest can effectively control the evaluation bias caused by spatial autocorrelation, making the carbon emission prediction model trained in subsequent steps more reliable when it is extended across regions.

[0036] 3. The SHAP value is calculated based on the random forest model obtained by K-fold cross-validation, and the SHAP dependence curve of carbon emission influencing factors is calculated. Then, the threshold is identified by the derivative to construct different segments, realizing the transformation from "black box prediction" to "clear threshold / operation window", providing a refined quantitative basis for land use structure adjustment and infrastructure layout.

[0037] 4. Social benefits: It provides direct basis for provincial, municipal and county-level management departments to formulate differentiated emission reduction paths and optimize industrial and land use structures; it helps to improve the scientific and refined level of "dual carbon" decision-making and reduce the issuance of "one-size-fits-all" indicators; through a unified data pipeline and model framework, it is conducive to horizontal comparison and experience promotion between different regions. Attached Figure Description

[0038] Figure 1 This is a flowchart of the carbon emission prediction and threshold identification method described in the embodiments of this application. Detailed Implementation

[0039] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.

[0040] This embodiment provides a carbon emission prediction and threshold identification method based on spatially blocked random forest, referencing... Figure 1 As shown, it includes:

[0041] Step 1: Using counties as the unit of analysis, collect carbon emissions and their multi-source influencing factors, preprocess the collected data, and construct a dataset.

[0042] The multi-source influencing factors mentioned in this embodiment include land use structure, nighttime light distribution, topography, road density, population size, and GDP.

[0043] The land use structure, as defined, refers to the proportion of various land uses and functional structures (including industry, commercial and service, urban residential, other towns, rural settlements, cultivated land, forest land, water bodies, grassland, etc.), given in the form of "proportion of land area for each type." This is typical composite data, satisfying the geometric constraints of non-negativity and a sum of 1 (simplex space). Ignoring the simplex constraint and not processing the data as composite data will lead to unstable correlation and importance results. To address this issue, this embodiment preprocesses the land use structure as follows:

[0044] First, the values ​​of land use structure and other influencing factors are uniformly projected onto an equal-area coordinate system;

[0045] Then, the share of each land use and functional structure obtained by projecting onto the equal-area coordinate system is replaced with a smaller value (e.g., replacing 0 with 10). -6 Then, a centralized logarithmic ratio (CLR) transformation is performed to obtain the representation of each share in Euclidean space:

[0046] ; ; In the formula, and The first The first county The numerical representation of the share of land use and functional structure after small value substitution and CLR transformation. Indicates the number of types of land use and functional structure. Indicates the first A share vector constructed from the shares of all land use and functional structure in each county. Indicates the first Share vector of each county The geometric mean of each component is used as the centering term in the CLR transform.

[0047] CLR transformation is performed on share combination data of various land use and functional structures. In other embodiments, equivalent methods such as isometric log-ratio (ILR) can be used instead of CLR, as long as the geometric constraints on the combination space are maintained.

[0048] The preprocessing of the nighttime light distribution and terrain distribution in this embodiment includes:

[0049] First, the values ​​of nighttime light distribution / terrain distribution and other influencing factors are projected onto an equal-area coordinate system;

[0050] Then, the nighttime light distribution / terrain distribution of different counties obtained by projecting onto the equal area coordinate system is resampled on a uniform grid (e.g., 500m), and the sum or mean is calculated as the nighttime light / terrain data of each county.

[0051] Furthermore, this embodiment also performs analysis on road density, population size, GDP, and preprocessed nighttime light / terrain data. Transformation and standardization (subtract the mean / divide the standard deviation).

[0052] Finally, the preprocessed multi-source influencing factors and carbon emissions are combined to construct a dataset for subsequent steps of model training, testing, and SHAP value calculation.

[0053] Step 2: Divide all counties in the dataset into several connected components based on spatial proximity, and then divide all connected components into K sets. The number of counties in each set is similar, and the data samples of all counties in each set constitute a data subset.

[0054] Specifically, this embodiment can divide the study area into several spatially connected blocks based on the county's spatial location, using clustering or regular grid methods. This step implements a spatially blocked data subset partitioning method. Other partitioning strategies can also be used, such as hierarchical clustering, k-means clustering, or manually specifying region groups, as long as the principle of "leaving out whole blocks" is maintained during cross-validation.

[0055] Step 3: Using each data subset as the validation set and all other data subsets as the training set, train K random forest models using cross-validation. After the performance evaluation is satisfactory, train a carbon emission prediction model based on the dataset.

[0056] Step 2 divides all counties in the study area into connected components based on spatial proximity to construct spatially blocked cross-validation folds. This ensures that data samples with spatial connectivity are in the same data subset, enabling model validation to be completed in Step 3. This spatially blocked random forest can effectively control the evaluation bias caused by spatial autocorrelation, making the carbon emission prediction model trained in subsequent steps more reliable when extended across regions.

[0057] Since the dataset can be divided into K subsets, cross-validation can be used to train K different random forest models. Then, each trained random forest model is used to make predictions on the corresponding test set, and performance metrics such as root mean square error (RMSE), mean absolute error (MAE), and conventional coefficient (R²) are calculated. The performance metrics of all K random forest models are then summarized. If they meet the performance requirements for carbon emission prediction in this invention, a new random forest model is retrained based on the dataset as the carbon emission prediction model.

[0058] Step 4: Calculate the SHAP value matrix of the corresponding test set using the K random forest models obtained from training, fit the SHAP dependency curve of each influencing factor and extract the segmented threshold.

[0059] For each trained random forest model, the SHAP value matrix of the test set (i.e., the subset of data that was not used to train the random forest model) on the random forest model is calculated. Each influencing factor for each test sample corresponds to one SHAP value. Therefore, for each data sample in all test sets, each influencing factor value corresponds to one SHAP value. At this point, for each influencing factor, the influencing factor can be used as the independent variable and the SHAP value as the dependent variable. Based on the influencing factor values ​​in the test set and the corresponding calculated SHAP values, the SHAP dependency curve of each influencing factor can be fitted.

[0060] Specifically, the SHAP dependency curve can be fitted using locally weighted regression or other smoothing methods, denoted as the fitting function being a function. .

[0061] The method for extracting the segmented thresholds for each influencing factor is as follows: First, calculate the SHAP dependency curve. first derivative The derivative is approximated by numerical differencing at several equidistant or quantile points, such as forward differencing or central differencing; then the absolute value of the derivative is calculated. The threshold value at which it is distributed at a preset high quantile is denoted as δ. high The threshold values ​​distributed at the preset low quantiles are denoted as δ. low , satisfying 0 < δ low <δ high ; and will satisfy Furthermore, the value of the influencing factor that undergoes a sign change is denoted as the inflection point (if it exists). Further, the condition that satisfies... ≥δ high The range of values ​​for influencing factors is denoted as the sensitive region of the influencing factor. This means that a steeper slope on the smoothed SHAP dependency curve and even small changes in the influencing factor value can lead to significant SHAP fluctuations, thus having a greater impact on carbon emissions. ≤δ low The range of values ​​for influencing factors is denoted as the plateau region, meaning that the influencing factors have a limited (saturated or leveled) impact on carbon emission intensity within this range; δ low < <δ high The range of influencing factors is denoted as the shoulder region of the influencing factors, i.e., the range where the sensitive region transitions to the plateau region. The influencing factor values ​​corresponding to the boundary points of the sensitive region / shoulder region and the shoulder region / plateau region are used as segmented thresholds for output, denoted as follows: and (If multiple discontinuous intervals occur, the boundary point adjacent to the platform area can be selected as the threshold.)

[0062] Specifically, since the values ​​of various factors influencing carbon emissions are preprocessed scaled data before being input into the model, it is necessary to adjust the extracted thresholds: , The preprocessing process is reversed and transformed back to the original physical scale, such as the land use and functional structure share value of 0~1, and the road density km / km².

[0063] The sensitive areas, shoulder areas, and platform areas obtained based on the extraction thresholds can be used to guide counties in planning and controlling carbon emission influencing factors at the policy level, or to further determine the current range of the influencing factor value in the target county, so as to guide the improvement of control.

[0064] Step 5: Use a carbon emission prediction model to predict the carbon emissions of the target county.

[0065] When using a carbon emission prediction model to predict the carbon emissions of a target county, it is necessary to collect the values ​​of multiple influencing factors of carbon emissions in the target county according to step 1 and perform the same preprocessing. Then, the preprocessed values ​​of multiple influencing factors are input into the carbon emission prediction model to predict and output the carbon emissions of the target county.

[0066] Step 6: Compare the values ​​of each influencing factor in the target county with the corresponding segment thresholds to determine the segment in which the values ​​of each influencing factor of carbon emissions in the target county are located.

[0067] Specifically, the collected values ​​of each influencing factor in the target county can be compared with the threshold scale after inverse transformation in step 4 to determine whether the values ​​of each influencing factor of carbon emissions in the target county are in its sensitive area, shoulder area or plateau area.

[0068] This invention can be implemented using any programming language such as Python, R, or Java, or integrated into GIS / remote sensing software (such as ArcGIS Pro, QGIS, or GEE). Furthermore, parameters such as raster resolution and fold number K can be adjusted according to the characteristics of the study area without affecting the carbon prediction and segmented threshold extraction capabilities of this invention.

[0069] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.

Claims

1. A method for carbon emission prediction and threshold identification based on spatially blocked random forest, characterized in that, include: Using counties as the unit of analysis, we collected carbon emissions and their multi-source influencing factors, preprocessed the collected data, and constructed a dataset. The dataset is divided into several connected components based on spatial proximity. Then, all connected components are divided into K sets, with a similar number of counties in each set. The data samples of all counties in each set constitute a data subset. K random forest models were trained using cross-validation with each data subset as the validation set and all other data subsets as the training set. After the performance evaluation was qualified, a carbon emission prediction model was trained based on the dataset. The SHAP value matrix of the corresponding test set is calculated using the K random forest models obtained from training, the SHAP dependency curves of each influencing factor are fitted, and the segmented thresholds are extracted. Use carbon emission prediction models to predict carbon emissions in the target county.

2. The carbon emission prediction and threshold identification method based on spatially blocked random forest according to claim 1, characterized in that, The multi-source influencing factors include land use structure, as well as nighttime light distribution, topographic distribution, road density, population size and / or GDP.

3. The carbon emission prediction and threshold identification method based on spatially blocked random forest according to claim 2, characterized in that, The land use structure refers to the proportion of various land use and functional structures, and its preprocessing includes: First, the values ​​of land use structure and other influencing factors are uniformly projected onto an equal-area coordinate system; Then, the shares of each land use and functional structure obtained by projecting onto the equal-area coordinate system are replaced with smaller values, and then CLR transformation is performed to obtain the representation of each share in Euclidean space: ; ； In the formula, and The first The first county The numerical representation of the share of land use and functional structure after small value substitution and CLR transformation. Indicates the number of types of land use and functional structure. Indicates the first A share vector composed of the share of all land use and functional structure in each county. Indicates the first Share vector of each county The geometric mean of each component is used as the centering term in the CLR transform.

4. The carbon emission prediction and threshold identification method based on spatially blocked random forest according to claim 2, characterized in that, Preprocessing of nighttime light distribution and terrain distribution includes: First, the values ​​of nighttime light distribution / terrain distribution and other influencing factors are projected onto an equal-area coordinate system; Then, the nighttime light distribution / terrain distribution of different counties obtained by projecting onto the equal area coordinate system is resampled on a unified grid, and the sum or mean is calculated as the nighttime light / terrain data of each county.

5. The carbon emission prediction and threshold identification method based on spatially blocked random forest according to claim 2, characterized in that, Preprocessing of various types of collected data also includes processing all non-combinatorial variables. conduct Transformation and standardization.

6. The carbon emission prediction and threshold identification method based on spatially blocked random forest according to claim 1, characterized in that, Using clustering or regular grid methods, all counties are divided into several connected blocks based on spatial proximity.

7. The carbon emission prediction and threshold identification method based on spatially blocked random forest according to claim 1, characterized in that, Using each influencing factor as the independent variable and the SHAP value as the dependent variable, the SHAP dependency curves of each influencing factor are fitted based on the influencing factor values ​​in the test set and the corresponding calculated SHAP values.

8. The carbon emission prediction and threshold identification method based on spatially blocked random forest according to claim 7, characterized in that, The SHAP dependency curve was fitted using a locally weighted regression method.

9. The carbon emission prediction and threshold identification method based on spatially blocked random forest according to claim 1, characterized in that, The method for extracting segmented thresholds for influencing factors is as follows: First, the first j Factors x j The SHAP dependency curve is represented as Calculate the SHAP dependency curve first derivative ; Then calculate the absolute value of the derivative. The threshold value at which it is distributed at a preset high quantile is denoted as δ. high The threshold values ​​distributed at the preset low quantiles are denoted as δ. low , satisfying 0 < δ low <δ high If there exists that satisfy Furthermore, if the value of an influencing factor undergoes a sign change, that value is recorded as an inflection point. Then satisfy ≥δ high The range of values ​​for influencing factors is denoted as the sensitive zone of the influencing factors; [the following is a list of parameters] that satisfy [the following is a list of parameters]: ≤δ low The range of influencing factors is denoted as the plateau region; the range of values ​​for δ is defined as the plateau region. low < <δ high The range of values ​​for influencing factors is denoted as the shoulder area of ​​the influencing factors, that is, the range from the sensitive area to the plateau area. Finally, the influencing factor values ​​corresponding to the boundary points between the sensitive area and the shoulder area, and between the shoulder area and the plateau area, are used as the segmented threshold outputs.

10. The carbon emission prediction and threshold identification method based on spatially blocked random forest according to claim 1, characterized in that, After extracting the segmented thresholds, the process also includes: comparing the values ​​of each influencing factor in the target county with the corresponding segmented thresholds to determine the segment in which the values ​​of each influencing factor in the carbon emissions of the target county are located.

Images