Design and identification method of high-precision landfill site remote sensing index for municipal solid waste

Abstract

The application discloses a high-precision landfill remote sensing index design and identification method for municipal solid waste, and comprises data set construction, preprocessing of the constructed data, training of a TabPFN model through the preprocessed data, determination of n band features ranking at the top of the influence on the model prediction performance after the model training is completed, generation of a landfill remote sensing index threshold value through mathematical combination based on the n band features, verification of the accuracy of the landfill remote sensing index threshold value in landfill identification, and determination that a target region is a landfill when a calculated landfill remote sensing index of the target region is greater than or equal to the threshold value, and determination that the target region is not a landfill when the calculated landfill remote sensing index is less than the threshold value. The application improves the accuracy of landfill detection by designing a remote sensing index specially for landfill identification, and is suitable for municipal solid waste monitoring, environmental protection and other fields, and can effectively support solid waste extraction tasks in complex urban environments.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of remote sensing technology and machine learning, and particularly relates to a high-precision landfill remote sensing index design method for municipal solid waste, and a method for identifying municipal solid waste landfill areas through the landfill remote sensing index. BACKGROUND

[0002] Landfill identification is a key step in environmental protection and waste management, directly related to the control of soil and groundwater pollution and the maintenance of ecological balance. Traditional methods such as on-site inspection, although providing direct observation, are time-consuming and costly, especially in large areas, making it difficult to achieve efficient coverage. Studies have shown that remote sensing technology, with its wide coverage and regular observation capabilities, has become an economical and efficient alternative, widely used in landfill detection and monitoring.

[0003] According to recent research, remote sensing technology has made significant progress in landfill detection, mainly including the following methods:

[0004] Multispectral imagery: By analyzing visible and near-infrared bands, land cover changes and vegetation health can be detected. For example, the Normalized Difference Vegetation Index (NDVI) is often used to identify disturbed areas, but its effectiveness in landfill detection is limited, as the spectral characteristics of landfills are similar to bare soil. Thermal infrared imagery: Utilizes the higher surface temperature generated by organic matter decomposition in landfills for detection, particularly suitable for monitoring potential emission points of landfill gas. Synthetic Aperture Radar (SAR): By analyzing echo changes, surface geometric changes caused by waste dumping can be detected, suitable for observing cloud-covered areas. Hyperspectral sensors: Provide more detailed spectral information, theoretically improving classification accuracy, but due to data processing complexity and high cost, practical application is less common.

[0005] In recent years, machine learning techniques have been introduced into remote sensing data analysis to enable automated detection of landfills. In particular, deep learning models such as Convolutional Neural Networks (CNN) have been used to classify scenes from aerial and satellite images. For example, studies have shown that CNN can automatically identify illegal landfills by training labeled data, but face the following challenges:

[0006] High false positive rate: The spectral characteristics of landfills are similar to other land covers (such as construction sites or bare soil), leading to high false positive rates. Data requirements: A large number of accurately labeled data sets are needed for training, but existing public data sets are scarce, limiting the development and validation of models. Limited generalization ability: The performance of the model varies greatly in different regions and seasons, making it difficult to achieve uniform application worldwide.

[0007] While existing remote sensing indices such as NDVI are primarily used for vegetation health assessment, they perform poorly in landfill detection as they cannot effectively distinguish landfills from other non-vegetation covered areas. Other indices such as the Normalized Difference Built-up Index (NDBI) can be used to detect urbanized areas, but they also lack specific design for landfills. Studies have shown that there is currently a lack of indices optimized specifically for landfill identification, resulting in limited detection accuracy.

[0008] Meanwhile, remote sensing data often faces data preprocessing challenges in practical applications, including missing value handling, outlier detection, and high-dimensional data dimensionality reduction. Studies have shown that these problems directly affect model performance, but existing methods have not formed a standardized process. For example, the multicollinearity problem of hyperspectral data can lead to model overfitting, which needs to be solved by dimensionality reduction techniques such as Principal Component Analysis (PCA).

[0009] In summary, existing methods have significant limitations in landfill identification, especially in distinguishing true positives and false positives. Studies emphasize the need to develop more accurate detection methods, combining multi-source data and advanced analysis techniques to improve the generalization ability and practical application effect of the model. This provides a technical background and market demand for the design of new landfill remote sensing indices. SUMMARY

[0010] The purpose of the present application is to provide a high-precision landfill remote sensing index design method for urban solid waste, and a method for identifying urban solid waste landfill areas through the landfill remote sensing index, to solve the problems of high misjudgment rate and insufficient generalization ability in urban solid waste landfill identification.

[0011] To achieve the above purpose, the technical solution of the present application is:

[0012] The high-precision landfill remote sensing index design method for urban solid waste, characterized in that the method comprises:

[0013] Step 1: Data set construction, using a computer vision model to preliminarily identify solid waste landfills in the target area, generating preliminary identification results. If the machine vision model determines that the target area is a solid waste landfill, it is identified as positive. True positives and false positives are distinguished by manual labeling or field investigation to construct a training data set. The data set includes remote sensing data of the target area and the labeling results corresponding to true positives or false positives of the remote sensing data.

[0014] Step 2: Data preprocessing, filling missing values, detecting outliers and reducing dimensions for the data collected in step one;

[0015] Step 3: Model training and optimization, a TabPFN model is trained based on the pre-processed data in step two, and the TPE algorithm of the Hyperopt library is used for model hyperparameter optimization to determine the optimal model parameters, and the model training is completed;

[0016] Step 4: After completing the model training in step three, the Permutation Importance method is used to measure the impact of each feature on the model prediction performance, and by randomly shuffling the feature values, the importance of the spectral bands in the remote sensing data is quantified, and the top n band features that affect the model prediction performance are selected;

[0017] Step 5: Landfill remote sensing index design, based on the top n bands selected in step four, remote sensing indices are generated through mathematical combination;

[0018] Step 6: Landfill remote sensing index verification, test the generalization ability of the landfill remote sensing index on independent data sets, compare the performance with existing landfill remote sensing indices, and verify the accuracy and robustness of the landfill remote sensing index in landfill identification.

[0019] Further, the computer vision model is BCA-NET, which identifies potential solid waste landfill areas from remote sensing images of target areas.

[0020] Further, the data processing includes first filling missing values using K-nearest neighbor method or hot backup filling method, then performing anomaly detection using Isolation Forest method, and then performing dimensionality reduction on the data, the dimensionality reduction processing uses Principal Component Analysis (PCA) method, and the first 10-20 dimensions with cumulative contribution rate of 95% are retained.

[0021] Further, the TPE algorithm of the Hyperopt library is used for hyperparameter optimization, including 50 trials, and the performance indicators are evaluated on a 20% validation set.

[0022] Further, the top n band features that affect the model prediction performance include using SHAP to calculate the contribution of each feature to the model prediction based on Shapley values.

[0023] Further, the landfill remote sensing index calculation method is:

[0024] ;

[0025] where the spectral data of band A, band B and band C are obtained and processed by Google Earth Engine, is a weight coefficient.

[0026] A method for identifying municipal solid waste landfills by using a high-precision landfill remote sensing index for municipal solid waste:

[0027] Spectrum information of the target area is collected, n band features ranked first in landfill identification are extracted from the spectrum information of the target area, and a landfill remote sensing index of the target area is calculated according to a landfill remote sensing index calculation formula according to the n band features; when the calculated landfill remote sensing index is greater than or equal to the landfill remote sensing index threshold, the target area is a landfill; when the calculated landfill remote sensing index is less than the landfill remote sensing index threshold, the target area is not a landfill.

[0028] The present application is applicable to the fields of municipal solid waste monitoring, environmental protection and land use planning, can effectively support the extraction of solid waste in complex urban environment, and provide data support for related policy making, and the beneficial effects are described in detail as follows:

[0029] 1. Significantly improved identification accuracy and specificity

[0030] The present application designs a remote sensing index specially for landfill identification, uses TabPFN model and feature importance analysis in the field of landfill detection, and significantly improves the accuracy of landfill detection. The existing method (such as NDVI) is mainly used for vegetation health evaluation, and it is difficult to distinguish landfills from other non-vegetation covered areas (such as bare soil or construction sites), resulting in a high false positive rate. At the same time, the threshold is optimized through ROC curve and Youden index, reducing the misjudgment and improving the true positive rate. Research shows that the AUC value on the independent data set is significantly higher than that of NDVI.

[0031] 2. Enhanced generalization ability and robustness

[0032] The present application is verified on independent data sets, which cover different regions (such as cities, rural areas) and different time periods (such as dry season, wet season), ensuring the universality of the method. The existing method often has insufficient generalization ability due to the limitation of training data, especially in cross-regional applications. The verification results show that the performance of the method of the present application is stable under different environmental conditions, solving the problem of seasonal and regional differences of the prior art, and being suitable for landfill monitoring in a global range.

[0033] 3. Efficient data preprocessing and model training

[0034] Remote sensing data often faces the challenges of missing values, outliers and high dimensionality, and the present application solves these problems through systematic data preprocessing, improving the efficiency of model training. Data preprocessing significantly improves data quality and optimizes model training efficiency. Research shows that the convergence speed of the model after dimensionality reduction is improved by more than 30%.

[0035] 4. Enhanced model interpretability and transparency

[0036] The application analyzes feature importance by Permutation Importance, SHAP and LIME methods, providing interpretable basis for landfill identification. Existing methods are often "black box" models, which are difficult to understand the model decision-making process, limiting practical application.

[0037] 5. Innovative landfill remote sensing index design and comparative advantage

[0038] The landfill remote sensing index designed by the application is superior to existing general indexes (such as NDVI) by using entropy weight method and TOPSIS method to comprehensively sort bands. Comparative experiments show that the AUC value of the new index on independent data set is higher than that of NDVI, especially in the ability to distinguish under complex land cover, and the research supports its application potential in landfill identification.

[0039] 6. Practical application value and environmental benefits

[0040] The method of the application is suitable for environmental monitoring, urban planning and solid waste management fields, and can quickly identify illegal landfills to reduce environmental pollution risks. Existing methods are inefficient and difficult to meet large-scale monitoring needs. Research shows that the method of the application can significantly reduce monitoring costs and improve regulatory efficiency, especially in resource-limited areas. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The flowchart of the landfill remote sensing index design method for urban solid waste of the application is shown. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the application more clear and understandable, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described here are part of the embodiments of the application, not all embodiments, and are not intended to limit the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the application.

[0043] The application designs a new type of landfill remote sensing index, combines spectral information, computer vision model and field verification data, uses data preprocessing, model optimization and feature importance analysis technology, realizes efficient distinction of true positive and false positive in landfill target detection, and improves the accuracy and robustness of landfill identification. The application is suitable for environmental monitoring, urban planning and solid waste management fields.

[0044] The application discloses a high-precision landfill remote sensing index design method for urban solid waste, and comprises the following steps:

[0045] Step 1: data set construction

[0046] The application first performs preliminary landfill identification on the target area through a published computer vision model such as BCA-NET to generate a preliminary identification result. For example, the target area is preliminarily identified through the computer vision model BCA-NET, if the identification result is "yes", it is judged that the target area is a solid waste landfill. If the identification result is "no", it is judged that the target area is not a solid waste landfill.

[0047] BCA-NET is a model specially designed for landfill detection, which can extract potential solid waste landfill areas from the remote sensing image of the target area. Then, based on human eye recognition or field investigation, the preliminary results are labeled as true positive and false positive to construct a data set containing true positive and false positive samples. This step ensures the accuracy of the data and solves the problem of insufficient labeled data in the prior art. For example, if the identification result of the target area by BCA-NET is "yes", and based on human eye recognition or field investigation, the target area is indeed a solid waste landfill, then the identification result of BCA-NET is true positive. If the identification result of the target area by BCA-NET is "yes", but based on human eye recognition or field investigation, the target area is not a solid waste landfill, then the identification result of BCA-NET is false positive.

[0048] The application can judge whether the identification result of the computer vision model such as BCA-NET is correct.

[0049] In the application, the spectral information and related spectral indices of the target area with true positive result and the target area with false positive result are obtained by using the Google Earth Engine (GEE) platform. The solid waste landfill and the non-solid waste landfill correspond to different spectral band characteristics. For example, the living garbage area, the construction waste area and the industrial waste area included in the solid waste landfill and the grassland, the water body and the highway included in the non-solid waste landfill correspond to different spectral band characteristics. GEE provides access to satellite multispectral data such as Landsat and Sentinel, and the extracted features include visible light, near-infrared and short-wave infrared band values, and pre-calculated spectral indices (such as NDVI and NDWI). These data form a multi-modal data set suitable for machine learning training, laying a foundation for subsequent analysis.

[0050] Step 2: data preprocessing

[0051] Since remote sensing data often have missing values due to cloud cover, sensor failure, etc., first, the missing values of each band are coarsely filled to ensure data integrity and provide data for anomaly detection algorithms. Here, the global median of each band or the median grouped by land cover type can be used for fast filling, i.e. coarse filling.

[0052] On the data set after coarse filling, the iForest algorithm is applied for anomaly detection. iForest builds multiple isolation trees through randomization and calculates the average path length of each sample; the path length significantly shorter than the top 5% of normal samples (or the threshold determined through cross-validation) is marked as abnormal and removed.

[0053] After removing the subset of anomalies, a fine filling method is used. The present application uses K-nearest neighbor (KNN) or hot standby filling method technology for fine filling. KNN calculates the distance between sample points and the nearest K neighbors (usually K=5 or 10), and fills in the missing values by weighted average neighbor values; hot standby filling method randomly selects values from similar samples to fill in, maintaining the data distribution characteristics. For spectral data, KNN is particularly suitable for handling continuous band values, ensuring that the filling results are consistent with the characteristics of the ground features.

[0054] Considering the multi-dimensionality of spectral data (such as tens to hundreds of bands), the present application uses principal component analysis (PCA) or multiple correspondence analysis (MCA) for dimension reduction. PCA retains the principal components with the largest variance through linear transformation, usually selecting the top 10-20 dimensions with a cumulative contribution rate of 95%, to reduce computational complexity and reduce multicollinearity. MCA is suitable for data containing categorical variables (such as land cover types), and reduces dimensions by analyzing the correspondence between variables. After dimension reduction, the data dimension is significantly reduced, improving the efficiency of subsequent model training.

[0055] Step 3: Model training and optimization

[0056] k-fold cross-validation is a statistical method for evaluating the performance of machine learning models, using the preprocessed data from step 2, performing 5 or 10 times of model training and validation, and selecting TabPFN as the best model for training. The present application performs k-fold cross-validation, such as 5-fold or 10-fold, on the data set preprocessed in step two, compares the performance of various machine learning baseline models such as random forest and support vector machine, and evaluates their applicability in landfill identification tasks. According to the characteristics of the data, such as high-dimensional spectral data and sample imbalance, TabPFN is selected as the best model for training. TabPFN is a transformer-based table data prediction model that is good at capturing complex relationships between features, and is particularly suitable for classification tasks of landfill spectral data.

[0057] The Tree-structured Parzen Estimator (TPE) of the Hyperopt library guides the selection of hyperparameters by constructing a probability model, and evaluates the performance of model parameters through accuracy or loss on the validation set. Compared with traditional grid search (Grid Search) and random search (Random Search), it more intelligently utilizes historical evaluation results to optimize the search direction. To optimize the performance of TabPFN, the Tree-structured Parzen Estimator (TPE) algorithm of the Hyperopt library is used for hyperparameter tuning. The optimization process includes 50 trials (n_trials=50), and the performance indicators such as accuracy or AUC are evaluated on the 20% validation set (n_validation_size=0.2). The search space covers model configurations (such as the number of integrations) and inference parameters (such as the number of attention heads). If all trials fail, a default model is created to ensure robustness. The random seed control (random_state) in the code ensures that the results are repeatable. The random seed can control the initial state of the random number generator, ensuring that the random sequence generated under the same conditions is reproducible.

[0058] Hyperparameters are tuned by Tree-structured Parzen Estimator (TPE) of the Hyperopt library, and model training is completed.

[0059] Step 4: After completing the model training in step three, the Permutation Importance method is used to measure the impact of each feature on the model prediction performance, and feature importance analysis is performed. By randomly shuffling feature values, the importance of spectral bands in remote sensing data is quantified, and the top n band features that affect the model prediction performance are selected, such as the top 3-5 band features.

[0060] Permutation Importance is a method for evaluating the impact of each feature on the prediction performance of a machine learning model. It is achieved by randomly shuffling the values of a certain feature and observing the change in model performance, thereby measuring the importance of the feature. Usually, a model is first trained with the original data, and its baseline performance on the validation set is recorded. If a feature is important for model prediction, shuffling its values will significantly reduce the model performance, because it destroys the relationship between the feature and the target. If the feature is irrelevant or redundant, the performance change is not significant after shuffling.

[0061] SHAP (SHapley Additive exPlanations) is a model explanation method based on Shapley value in game theory, which is used to quantify the contribution of each feature to the model prediction. It explains the output of the model by calculating the marginal contribution of each feature.

[0062] LIME (Local Interpretable Model-agnostic Explanations) is a method for explaining the prediction results of a single sample of a machine learning model. It constructs a simple and interpretable local model, such as a linear model, near the target sample, and explains the prediction behavior of the original model through the parameters of this simple model.

[0063] In the present invention, after the model training is completed, the Permutation Importance method is used to measure the influence of each feature on the model prediction performance. By randomly shuffling the feature values and observing the performance decline, the importance of the spectral bands is quantified. In addition, SHAP and LIME are used to provide explanatory analysis. SHAP calculates feature contributions based on Shapley values to generate global and local importance. LIME approximates predictions through a local linear model to explain the effect of single sample features. The combination of the three provides a scientific basis for subsequent band selection. In the present invention, the Permutation Importance method is used to measure the influence of each feature on the model prediction performance. In a specific embodiment, the "features" used for Permutation Importance evaluation are not the original pixel values or the bands themselves, but the first ten principal components obtained after principal component analysis (PCA) dimensionality reduction. The first ten principal components are described as follows: First, the original variables consist of twelve Sentinel-2 spectral bands and seven typical spectral indices. The twelve Sentinel-2 spectral bands B1, B2, B3, B4, B5, B6, B7, B8, B8A, B9, B11, and B12 are described in detail as follows. Sentinel-2 is a satellite for Earth observation launched by the European Space Agency (ESA), mainly used for environmental monitoring, agriculture, forestry, land use, etc. Its multi-spectral imager (MSI) carries 12 spectral bands, covering different wavelength ranges from visible light to short-wave infrared.

[0064] Table 1-12 spectral bands of Sentinel-2

[0065]

[0066] Seven typical spectral indices include: normalized vegetation index NDVI, normalized water body index NDWI, normalized snow and ice index NDSI, chlorophyll fluorescence related index TVX, drought stress index DPNI, improved soil adjusted vegetation index MSAVI, soil background index BSI.

[0067] Second point: PCA dimension reduction, the above 19 dimensions, that is, 12 bands plus 7 index spectral features are input into PCA; according to the cumulative variance contribution rate, the first 10 principal components are selected, which together explain >=95% of the total variance; the 10 principal components constitute the input "features" of the subsequent model.

[0068] In the present application, SHAP calculates feature contribution based on Shapley value, and generates global and local importance. Global importance (Global Importance) describes the average contribution or influence of a feature on the entire data set, that is, measures "which features are most valuable in the overall. Local importance (Local Importance) describes the influence of a feature on the prediction result of a single sample, focusing on "details of how a record is interpreted by the model."

[0069] Permutation Importance randomly shuffles the values of a certain feature column for all samples or a specified validation set, re-evaluates the model performance such as accuracy, AUC, mean square error, etc. After repeating multiple times, the average performance drop is obtained, which quantifies the global importance of the feature.

[0070] SHAP calculates the absolute value of Shapley value of each feature on all samples, and obtains the global importance ranking, which also reflects the overall feature contribution. The Shapley value of a single sample is directly output, and it is known that each feature in the sample promotes the prediction value to increase or decrease; local interpretation can be performed on demand for several samples. Since calculating SHAP value for all samples is very time-consuming, generally only a part of samples are selected for calculation, so it is a local and global importance indicator.

[0071] LIME generates a batch of "perturbation" data near a single sample, fits a simple linear model to the relationship between model output and feature change, and uses linear coefficients to explain the local importance of each feature under the sample. In the present application method, lime values of all samples are calculated and averaged, which is also a global and local indicator.

[0072] The feature importance values of three different angles, permutation importance, global SHAP and local SHAP / LIME, have been obtained, and in the application, the optimal ranking of feature importance is automatically given by the entropy weight method combined with the TOPSIS multi-attribute decision process through the multi-index comprehensive evaluation method.

[0073] Step 5: Landfill remote sensing index design

[0074] The remote sensing index designed for the identification of solid waste landfill in the application first uses M index to measure the separability of a specific band or band combination between solid waste landfill and other land cover types (such as forest, water body, bare soil), which directly reflects the distinguishability of the feature. M index refers to Jeffries-Matusita distance, which calculates the statistical distance of the spectral mean and variance of two types of samples of solid waste landfill and other land cover types, and the larger the value, the stronger the distinguishability.

[0075] Based on the three importance indicators - permutation importance, SHAP value and M index, the entropy weight method combined with the TOPSIS method is used for comprehensive ranking in the application. The entropy weight method assigns weights according to the information entropy of the three importance indicators, and the greater the information quantity, the higher the weight.

[0076] TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) is a multi-attribute decision analysis (MADM) method, which is used for comprehensive ranking of multiple alternative schemes, and by calculating the distance of each scheme from the ideal solution (optimal solution) and the negative ideal solution (worst solution), the relative advantages and disadvantages of the scheme are evaluated, and finally the comprehensive closeness is ranked.

[0077] In step three, multiple bands are selected by Permutation Importance method, SHAP, and LIME. TOPSIS generates a comprehensive ranking by calculating the distance of each band to the ideal solution (best) and negative ideal solution (worst), and selects the top-ranked bands (such as the top 2-5 bands). A landfill remote sensing index is generated by mathematical combination, such as difference and ratio. For example, the spectral data of bands A and B with high importance ranking are selected to calculate the landfill remote sensing index. The spectral data of bands A and B are obtained from Google Earth Engine (GEE). In the GEE platform, the spectral data extracted from Sentinel-2 images by points is "surface reflectance", which belongs to a kind of reflected spectral value. Landfill remote sensing index = (band A - band B) / (band A + band B), which highlights the spectral characteristics of the landfill.

[0078] The Youden index is an indicator for evaluating the performance of diagnostic tests. The present application determines the optimal threshold value of the landfill remote sensing index by the Youden index. To determine the optimal threshold value of the landfill remote sensing index, the machine learning dataset is traversed, the range of the landfill remote sensing index value is adjusted, the receiver operating characteristic (ROC) curve is plotted, and the sensitivity (true positive rate) and specificity (1-false positive rate) of each threshold value are calculated. The Youden index = sensitivity + specificity - 1. Based on the Youden index, the threshold value corresponding to the maximum Youden index in the roc curve is selected as the landfill remote sensing index. The threshold value corresponding to the maximum Youden index is used as the critical value to distinguish landfills and non-landfills, and the design of the landfill remote sensing index is completed.

[0079] The design of the landfill remote sensing index specifically includes:

[0080] Step 5.1: M-index separation evaluation

[0081] This step evaluates the discrimination ability of each band or band combination between solid waste landfills and other typical features (such as forests, water bodies, and bare soil) from the perspective of "spectral separability of the category itself". The index used is the M-index, which is based on the ratio of the difference between the means of the two classes to the variance within the class. The specific calculation formula is as follows:

[0082]

[0083] Where μ1 and μ2 are the means of the two classes of samples in a certain band or combination, and σ1 and σ2 are the corresponding standard deviations. The larger the M value, the better the separation of the band in distinguishing the two types of features. Generally, M>1.0 indicates excellent discrimination ability, and M<1.0 indicates poor discrimination effect. This index does not depend on the model and only reflects the natural separability in the original spectral space.

[0084] Step 5.2 Global Model Importance Analysis (Permutation Importance)

[0085] This step quantifies the influence of each feature (i.e., principal component) on the overall prediction performance from the perspective of "model internal mechanism". The method is to disturb the value of a certain feature one by one, record the average degree of decline of the model performance (such as AUC, accuracy, mean square error, etc.) caused by the disturbance, as the importance score of the feature. The higher the score, the greater the contribution of the feature to the model performance, which is one of the global explanation methods.

[0086] Step 5.3 Global and Local Shapley Value Analysis (SHAP)

[0087] This step quantifies the marginal contribution of each feature (principal component) in model prediction from the perspective of game theory. SHAP (SHapley Additive exPlanations) value is based on the Shapley value principle, which considers each feature as a "participant" and each model output as a "cooperative income". The Shapley value of a certain feature x i is defined as follows:

[0088]

[0089] Where: F is the set of all features, S is a subset of features not containing i; f S (x) represents the prediction value of the model for sample x using only the features in subset S; is the Shapley value of feature x i , reflecting the marginal contribution of the feature to the current prediction result.

[0090] In this invention, there are two levels of application: (1) Global SHAP: Calculate the Shapley value of each principal component for all samples, and take the average of the absolute value, which represents the average importance of the principal component to the overall prediction, which is used for global feature sorting;

[0091]

[0092] Where ϕ i (j) represents the Shapley value of the i-th principal component of the j-th sample.

[0093] Local SHAP: Select several typical samples x j , calculate the Shapley value of each principal component , which is used to analyze why the model gives the current prediction result for this sample. Used for anomaly detection.

[0094] Step 5.4 Local linear explanation average value (LIME) of the whole sample.

[0095] LIME (Local Interpretable Model-agnostic Explanations) is a model-independent, local perturbation-based explanation method. The core idea is to generate a series of perturbation samples near each sample x to be explained, and construct a local data set:

[0096]

[0097] wherein represents the i-th perturbation sample generated near the original sample , represents the prediction result of the black box model for the perturbation sample, and m represents the number of generated perturbation samples. Then, taking x as the center, different weights ,

[0098]

[0099] are given to the perturbation samples, and a local weighted linear model is fitted:

[0100]

[0101] wherein β k is the linear contribution of the feature z k to the model output in this local area. Each sample corresponds to a set of β k values, reflecting the local importance of the principal component at this sample. In the present application, in order to enhance stability, the average value of the LIME explanation results of all samples is taken to obtain the average linear contribution of each principal component:

[0102]

[0103] This processing method makes it possible for LIME to participate in subsequent global weight calculation and TOPSIS sorting while retaining its local sensitivity. In summary, the M index measures the natural separability of features from spectral differences; PermutationImportance and global SHAP evaluate feature contributions from the global logic of the model; and local SHAP and LIME provide individual sample-level explanations. The combination of the three takes into account both the separability of the data itself and the interpretability of the model reasoning.

[0104] Step 5.5 Normalization processing by entropy weight method

[0105] ​​To standardize the handling of metrics with different dimensions and units, the M-index, Permutation Importance, global SHAP, and local explanatory values ​​(SHAP or LIME) are normalized. Assuming there are n training samples and m evaluation metrics (where m is set to 4 using the method described above), where... Let m represent the evaluation value of the m-th importance evaluation metric under the n-th training sample, forming the evaluation matrix as follows:

[0106]

[0107] Then, let Z be the matrix of its normalization, and each element in Z... To determine whether there are negative numbers in the Z matrix, if so, another standardization method needs to be used for X.

[0108] Standardizing matrix X once yields A matrix, its standardized formula is:

[0109]

[0110] The standardized matrix is ​​represented as follows:

[0111]

[0112] Subsequently, the entropy weight method was used to allocate the weights of each indicator, and the probability matrix P was calculated, where each element p in P... ij The calculation formula is as follows:

[0113]

[0114] The smaller the entropy value, the greater the difference and the stronger the discrimination ability of the indicator among different principal components, and the higher the weight assigned; the objective weight of each type of indicator is obtained and used for subsequent comprehensive ranking.

[0115] For the j-th indicator, the formula for calculating its information entropy is:

[0116]

[0117] in Indicates the first The sample at the th The normalized weight value for each indicator reflects the relative importance or contribution of the sample to that indicator; n represents the total number of samples, and m represents the total number of indicators. Information entropy is used to measure the first The greater the entropy value, the less information the indicator provides, indicating that the indicator has less variability across samples.

[0118] Definition of information utility value: d j =1—e j Then the greater the information utility value, the more information it corresponds to. By normalizing the information utility value, we can get the entropy weight of each indicator:

[0119]

[0120] Step 5.6 TOPSIS comprehensive ranking

[0121] Based on the weighted and normalized matrix obtained by the entropy weight method, the TOPSIS method is used to rank the principal components. The specific steps include:

[0122] (1) First, determine the positive ideal solution (the maximum value of each indicator) and the negative ideal solution (the minimum value of each indicator), define the maximum value as:

[0123]

[0124] Define the minimum value as:

[0125]

[0126] Calculate the Euclidean distance of each principal component to the positive ideal solution and the negative ideal solution respectively:

[0127]

[0128] The positive ideal solution of the i-th indicator (i.e. the maximum value or optimal value in all schemes of this indicator) is represented by: The negative ideal solution of the i-th indicator (i.e. the minimum value or worst value in all schemes of this indicator) is represented by: The normalized value of the i-th scheme in the i-th indicator is represented by: The weight of the i-th indicator is represented by: m represents the total number of indicators.

[0129] Then the non-normalized score of the i-th (i=1,2,…,n) evaluation object can be calculated: .

[0130] Normalize the score: At this time, the higher the comprehensive score, the closer the principal component is to the optimal scheme, and the higher the ranking.

[0131] Step 5.7 Band combination and landfill remote sensing index construction

[0132] ​​​​The top 2-5 principal components corresponding to the original band or band combination are selected to construct a landfill remote sensing index. However, before this, it is necessary to judge the positive and negative correlation between the selected principal component band and the landfill to assist in the construction of the landfill remote sensing index. In the present application, the Point-Biserial correlation coefficient is used to measure the linear relationship between a dichotomous variable (such as landfill / non-landfill) and a continuous variable (principal component). The calculation relationship is as follows:

[0133]

[0134] wherein, are the mean values of the principal components corresponding to type=1 (landfill) and type=0 (non-landfill), respectively; s is the standard deviation of all samples; n1 and n0 are the sample numbers of type=1 and type=0, respectively; and n is the total sample number.

[0135] According to the correlation coefficient, the landfill remote sensing index can be designed. If there are only three bands, the index can be constructed by the following formula:

[0136]

[0137] wherein band A, band B and band C are the top three bands, and band A is positively correlated with the category, band B is negatively correlated with the category, and band C is negatively correlated with the category. These indexes are designed to further enhance the spectral contrast between the landfill and the background objects, so as to facilitate classification and identification;

[0138] Parameter is a key weight coefficient for balancing the contribution of PC1 and PC2 in the formula. It is obtained through strict statistical analysis of known samples, and the specific steps are as follows: first, a labeled sample set containing known landfill and non-landfill objects is needed. In the sample set, the category of each pixel is labeled (for example, landfill=1 and non-landfill=0). Then, the correlation between the value of each principal component (PC1, PC2, etc.) and the binary label of "whether it is a landfill" is calculated. Since one is a continuous variable (PC value) and the other is a binary classification variable (label), the most suitable statistical indicator is the Point-Biserial Correlation Coefficient. Here, the correlation coefficient Corr(PC, Label) should be: Corr(PC1, Label) is a significant positive value, and Corr(PC2, Label) is a significant negative value;

[0139] Parameter ​The role of PC2 is to quantify the importance of PC2 relative to PC1. This relative importance can be expressed as the ratio of the strength of the correlation between the two. Therefore, The calculation formula is:

[0140] a = |Corr(PC2, Label)| / |Corr(PC1, Label)|;

[0141] That is, Equal to the absolute value of the correlation between PC2 and the label divided by the absolute value of the correlation between PC1 and the label.

[0142] Step 5.8: landfill remote sensing index setting based on the Youden index

[0143] On the labeled training or validation set, traverse multiple thresholds for the constructed remote sensing index, and for each candidate threshold, count the number of true positives (TP) that correctly identify samples as landfills, true negatives (TN) that correctly identify as non-landfills, false positives (FP) that incorrectly identify as landfills, and false negatives (FN) that incorrectly identify as non-landfills. Then, based on the above statistical quantities, calculate the sensitivity TPR (TPR, the proportion of correctly identified positive samples) and the specificity TNR (TNR, the proportion of correctly identified negative samples) under the threshold. The formula is TPR = TP / (TP+FN), and the formula is TNR = TN / (TN+FP). Then, according to the obtained sensitivity and specificity, calculate the Youden index J by the formula J = TPR + TNR- 1. Where J is the Youden index: J = TPR + TNR−1.

[0144] The optimal threshold is the threshold that maximizes the Youden index J. Finally, by comparing the Youden indices corresponding to all candidate thresholds, select the candidate threshold that maximizes the Youden index J, and take the threshold that maximizes the Youden index J as the final classification decision threshold of the landfill remote sensing index, achieving efficient and interpretable solid waste landfill identification.

[0145] Step 6: Landfill remote sensing index verification

[0146] To verify the generalization ability of the landfill remote sensing index, the present application tests on independent data sets, including remote sensing data of different regions (such as cities, rural areas) and different time periods (such as dry seasons, wet seasons). Apply the landfill remote sensing index to calculate the performance indicators of landfill identification, such as accuracy, F1 score, AUC, to evaluate its effect on unseen data.

[0147] Meanwhile, the performance of the new landfill remote sensing index is compared with existing remote sensing indices such as NDVI. NDVI is mainly used for vegetation health assessment, and the calculation formula is (NIR - Red) / (NIR + Red), but it has limited performance in landfill detection. The superiority of the new landfill remote sensing index in distinguishing landfills from other non-vegetation covered areas is proved by comparing the AUC value or false positive rate of the ROC curve.

[0148] The design of the landfill remote sensing index is illustrated below.

[0149] In order to further integrate prior knowledge related to solid waste landfill sites into the spectral screening step, consider a variety of additional spectral content closely related to the identification and judgment of solid waste landfill sites, not just single data analysis on existing band data. On the basis of spectral data, further integrate related spectral indices, such as normalized difference vegetation index (NDVI), which can reflect vegetation coverage, while solid waste landfill sites generally have low vegetation coverage due to the stacking of solid waste; soil adjustment index (NDSI), used to distinguish soil and non-soil areas, the soil characteristics of solid waste landfill sites may be different from the surrounding areas; building index (BI), used to detect buildings, solid waste landfill sites may contain specific buildings or structures (leachate treatment plant, landfill gas collection device); thermal vegetation index (TVX), used to distinguish vegetation and bare soil, the thermal infrared characteristics of solid waste landfill sites may be different from the surrounding areas. More spectral index selection and formula calculation are shown in the following table (Table 2):

[0150] Table 2 Various spectral indices

[0151]

[0152] Finally, adding the original collected 12-band data of Sentinel-2 (excluding the almost zero B10 band) and the above-mentioned 7 related spectral indices, a total of 19 spectral related data values are recorded for a solid waste site. In order to better train the subsequent spectral model and achieve the purpose of distinguishing solid waste landfill sites, the recognition results of YOLOv8 model on remote sensing images are verified on site, and some typical ground object data negative samples are further introduced on the existing site data, including buildings, vegetation, water, wasteland and a series of easily confused site categories obtained in the preliminary investigation, and the sample labels are designed as zero, a total of 519 true landfill and 220 easily confused ground object spectral data.

[0153] Based on the above-mentioned all the spectral data, the primary work in the spectral model layer is to preprocess the spectral bands in the multi-modal database. Since there is no missing data, only abnormal value detection and dimension reduction processing are needed. The dimension reduction processing of the data is carried out by using the method of principal component analysis (PCA). Since remote sensing data usually contains multiple bands, each band may carry redundant or related information, therefore, before model training, principal component analysis needs to be performed on the spectral features to extract key spectral information, reduce data dimension, and improve computational efficiency. PCA projects high-dimensional data into low-dimensional space by finding the main variation direction of the above-mentioned 19 kinds of data, retains the components with the most information, and finally obtains 10 principal component features with contribution rate greater than 95% without linear correlation, which can completely represent the original 19 kinds of spectral bands.

[0154] Table 3 Principal component coefficient matrix (example of the first five principal components)

[0155]

[0156] The method selected for spectral screening is the TabPFN method proposed in Nature. TabPFN adapts a new architecture to the table structure, and each cell in the table is assigned an independent representation, which means that each cell can be processed and focused on individually. Moreover, a bidirectional attention mechanism is introduced to further enhance the model's understanding of table data. On the one hand, by adding a 1D feature attention mechanism, the cells in the same feature column can interact and transfer information, enabling the model to capture the variation patterns and relationships of different samples on the same feature. On the other hand, by adding a 1D sample attention mechanism, the cells in different sample rows can exchange information, thereby identifying the overall differences and similarities between different samples. This bidirectional attention mechanism enables TabPFN to learn from context, ensuring that the model can consistently extract and utilize information regardless of the order of samples and features, thereby improving the model's stability and generalization ability.

[0157] To further explore the applicability of TabPFN model to spectral table data, K-fold cross-validation is used to further compare and select the baseline model. Random forest, XGBoost and CatBoost are selected as experimental controls. These three methods are excellent machine learning methods in table data and are commonly used in table data classification and regression tasks. The ROC (Receiver Operating Characteristic Curve) is selected as the main evaluation index. ROC is an important tool for measuring the performance of binary classification models, especially for imbalanced data sets. It describes the trade-off between false positives and true positives, and can help evaluate model performance at different thresholds. Currently, the site identification task in this project is a binary classification task that requires the use of spectral models to eliminate false positive targets. In addition, accuracy, recall, and F1 score are introduced as auxiliary evaluation indicators to further measure the performance of the model in identifying solid waste sites.

[0158] Table 4 Spectral model comparison

[0159]

[0160] According to the table, TabPFN performs best on all four evaluation indicators, especially in terms of recall rate. Compared with the other three methods, TabPFN maintains a very high recall rate, which means that the TabPFN method can ensure the comprehensiveness and integrity of the solid waste landfill site survey, and the highest accuracy also fully proves the rationality of selecting TabPFN as the spectral model.

[0161] The key to designing a solid waste landfill site is to determine the most important components from the 10 principal components. Based on the different weights and positive and negative correlations of the principal components, a spectral index suitable for the site can be better designed. The determination of principal component weights uses three explainability analysis methods to integrate selection, including SHAP, LIME, and Permutation Importance. Here, M-index is not used for simplicity, as it requires additional spectral data from other features.

[0162] After obtaining the results of the three explainability methods, in order to comprehensively consider the results of each method, entropy weight method and TOPSIS method are used for comprehensive evaluation. Finally, the results of the explainability analysis method of each principal component are as follows: the weights of the three explainability analysis methods obtained by the entropy weight method are SHAP-0.3296, LIME-0.3325, and Permutation Importance-0.3378. Based on the above weights, the corresponding principal component weight values are further obtained using TOPSIS. The greater the weight value of each principal component, the higher the importance of the principal component.

[0163] Based on the importance weight determined in the table, PC1, PC2, PC5 three principal components are selected as the importance components of the solid waste landfill site spectral index design. Considering the positive and negative correlation between different principal components and site categories, according to the data statistics, it is determined that the relationship between PC1 and site is positive correlation, and the relationship between PC2 and PC5 and site is negative correlation. After comparison and selection of multiple index design methods, the solid waste landfill site spectral discrimination index SLDI (Solid-waste Landfill Discrimination Index) is finally determined:

[0164]

[0165] Where PC1, PC2 and PC5 are linear combinations of different wave bands, sgn() is the sign function, and the specific coefficients are shown in Table 3.

[0166] In order to further analyze the rationality of the designed solid waste landfill site spectral discrimination index, the main wave band composition of PC1, PC2 and PC5 is analyzed. It is not difficult to find that in addition to the three important wave bands B2, B3 and B4 in rgb, there are many wave bands closely related to solid waste landfill sites in the three principal components, such as B9 wave band in PC5, which is a short wave infrared wave band, mainly used for imaging soil type, geological characteristics, and extremely sensitive to vegetation and soil moisture changes. From the mechanism of spectrum, it is indeed consistent with the spectral that needs to be focused on for solid waste landfill site identification. The most important component in PC2 is MASVI, which is a modified soil vegetation index. This index is mainly composed of B8 wave band data, which can well reflect the solid waste landfill situation of the site and accurately identify the difference between the landfill area and the surrounding site. Therefore, the final determination of the principal components considered in SLDI index is the combination of important spectral wave bands closely related to solid waste landfill site identification to a certain extent.

[0167] After the formula of the corresponding index is constructed, in order to further explore the threshold of the landfill site remote sensing index corresponding to the solid waste landfill site, the accuracy of the site identified under different threshold recall rates is calculated, and the verification is carried out on the independent data set. It is found that the SLDI of 57 sites calculated is greater than or equal to 2, and these sites are all solid waste landfill sites. It means that when the SLDI value of a site is greater than or equal to 2, the site has a great possibility of being a solid waste landfill site. Even to say that among the 120 sites with SLDI value greater than or equal to 1.4, there are only 6 sites judged as non-solid waste landfill sites, which means that when the SLDI value of a site is greater than or equal to 1.48, it is still nearly 95% of the solid waste landfill site, which is also a considerable probability. Therefore, in general, the spectral discrimination index SLDI of the solid waste landfill site is indeed feasible for the simple identification of the solid waste landfill site. Through the calculation of SLDI value and the spectral index threshold mentioned above, the probability of the site being a solid waste landfill site can be preliminarily given. The greater the SLDI value calculated, the greater the probability of the site being a solid waste landfill site. Similarly, the Youden index can also be used to determine a threshold to comprehensively determine the recall rate and accuracy. The threshold setting has strong subjectivity.

[0168] The present application solves the problems of high misjudgment rate and insufficient generalization ability in landfill site identification in the prior art through a systematic method. Compared with traditional methods (such as NDVI), the newly designed landfill remote sensing index more accurately captures the spectral characteristics of the landfill site; combined with multi-modal data and machine learning optimization, the robustness and practical application value of the model are ensured.

[0169] The present application also discloses a high-precision landfill site identification method for urban solid waste. The target area is determined to be a landfill site by the method, which only needs to calculate the landfill remote sensing index corresponding to the target area. The judgment process is, for example, collecting the spectral information of the target area. Since the band characteristics affecting the identification results have been determined in step three, and the top 2-5 band characteristics affecting the landfill site identification have been determined in step four, in subsequent practical applications, only the 2-5 band characteristics are extracted from the spectral information of the target area, and the landfill remote sensing index of the target area is calculated according to the landfill remote sensing index calculation formula. When the calculated landfill remote sensing index is greater than or equal to the threshold determined in step five, the target area is a landfill site; when the calculated remote sensing index is less than the threshold determined in step five, the target area is not a landfill site.

[0170] The above embodiments are merely exemplary but not intended to limit the present application. Based on the above description, those skilled in the art can make other variations and modifications of the present application without departing from the scope of the present application. Accordingly, the present application intends to include all possible variations and modifications within the scope of the present application.

Claims

1. A high-precision landfill site remote sensing index design method for municipal solid waste, characterized in that The method comprises: Step 1: data set construction, using a computer vision model to preliminarily identify a target area in a solid waste landfill, to generate a preliminary identification result, if the machine vision model judges that the target area is a solid waste landfill, it is identified as positive, and the true positive and false positive samples are distinguished by manual labeling or field investigation to construct a training data set, the training data set comprises remote sensing data of the target area and labeled results corresponding to the true positive or false positive of the remote sensing data; Step 2: data preprocessing, missing value filling, outlier detection and dimensionality reduction processing are performed on the data collected in step 1; Step 3: model training and optimization, a TabPFN model is trained through the data preprocessed in step 2; Step 4: after completing the model training in step 3, the influence of each feature on the prediction performance of the model is measured, the importance of the spectral bands in the remote sensing data is quantified, and the top n band features affecting the prediction performance of the model are selected; The method for measuring the influence of each feature on the prediction performance of the model comprises Permutation Importance global model importance analysis, global and local Shapley value analysis, and local linear explanation average value calculation of the whole sample, the global model importance analysis method is to disturb the value of a feature one by one, record the average degree of decline of the model performance caused by the disturbance, and take it as the importance score of the disturbed feature, the global and local Shapley value analysis is to quantify the marginal contribution of each feature in the model prediction, the local linear explanation average value calculation of the whole sample comprises obtaining the linear contribution of the feature to the model output in the local region, the Permutation Importance value, the global SHAP value and the local explanation value are normalized, the normalized data are sorted in importance by the TOPSIS method, and the top n bands with high importance are obtained; Step 5: landfill remote sensing index design, based on the top n bands selected in step 4, the landfill remote sensing index is calculated by mathematical combination, and the threshold of the remote sensing index is determined.

2. The method for designing high-precision landfill remote sensing index for municipal solid waste according to claim 1, characterized in that: The computer vision model is BCA-NET, which identifies potential solid waste landfill areas from remote sensing images of target areas.

3. The method for designing high-precision landfill remote sensing index for municipal solid waste according to claim 1, characterized in that: The data preprocessing in step 2 comprises using the global median of each band or the median grouped according to the land cover type for rapid filling, on the data set after rapid filling, an isolation forest algorithm is applied for outlier detection, the isolation forest algorithm builds multiple isolation trees by random, calculates the average path length of each sample, and marks and removes the data with a path length shorter than 5% of the normal samples, and then uses a K-nearest neighbor filling method for fine filling, the K-nearest neighbor filling method fills the missing values by calculating the distance between the sample point and the nearest K neighbors and weighting the average neighbor value.

4. The method for designing high-precision landfill remote sensing index for municipal solid waste according to claim 1, characterized in that: The training of the TabPFN model comprises model hyperparameter tuning by a Tree-structured Parzen Estimator of a Hyperopt library, and the Tree-structured Parzen Estimator of the Hyperopt library is used to guide the selection of hyperparameters by constructing a probability model, and the performance of model parameters is evaluated by accuracy or loss on a validation set.

5. The method for designing high-precision landfill remote sensing index for municipal solid waste according to claim 1, characterized in that, The calculation method of the landfill remote sensing index is: selecting n bands with high importance ranking, and generating the landfill remote sensing index through mathematical combination difference or ratio.

6. The method for designing high-precision landfill remote sensing index for municipal solid waste according to claim 1, characterized in that, The calculation method of the landfill remote sensing index is: ； wherein the spectral data of the waveband A, the waveband B and the waveband C are obtained by spectral data of Google Earth Engine, is a weight coefficient.

7. The method for designing high-precision landfill index of municipal solid waste by remote sensing according to claim 1, characterized in that, The method for determining the threshold value of the landfill remote sensing index comprises: traversing multiple candidate threshold values for the constructed remote sensing index on a labeled training or validation set, and for each candidate threshold value, counting the number of true positive examples TP correctly identified as landfills, true negative examples TN correctly identified as non-landfills, false positive examples FP incorrectly identified as landfills, and false negative examples FN incorrectly identified as non-landfills; calculating the sensitivity TPR and the specificity TNR based on the counted numbers, the formula of TPR is TPR = TP / (TP+FN), and the formula of TPR is TPR = TN / (TN+FP); according to the obtained sensitivity and specificity, calculating the Youden index J through the formula J = TPR + TNR - 1; by comparing the Youden indexes corresponding to all candidate threshold values, selecting the candidate threshold value that makes the Youden index J reach the maximum value as the final classification decision threshold value of the landfill remote sensing index.

8. The method for designing high-precision landfill index of municipal solid waste by remote sensing according to claim 7, characterized in that, The verification of the threshold value of the landfill remote sensing index comprises: testing the generalization ability of the landfill remote sensing index on an independent data set, comparing the performance with existing landfill remote sensing indexes, and verifying the accuracy and robustness of the threshold value of the landfill remote sensing index in landfill identification.

9. A method for identifying urban solid waste landfills by the method for designing a high-precision landfill remote sensing index for urban solid waste according to any one of claims 1-8: Collecting spectral information of a target area, extracting n band features ranking first in the influence on landfill identification from the spectral information of the target area, and calculating the landfill remote sensing index of the target area according to the landfill remote sensing index calculation formula according to the n band features; when the calculated remote sensing index is greater than or equal to the threshold value of the landfill remote sensing index, the target area is a landfill; when the calculated remote sensing index is less than the threshold value of the landfill remote sensing index, the target area is not a landfill.

Images