Soil arsenic content spectrum prediction model based on continuous projection algorithm

Abstract

The invention discloses a soil arsenic content spectrum prediction model based on a continuous projection algorithm, and belongs to the technical field of soil arsenic content prediction. A technical route of overlapping peak separation preprocessing, continuous projection algorithm feature screening and multi-source feature fusion to AdaBoost lightweight ensemble learning is adopted, and the technical problems of spectrum overlapping peak interference, complex model parameter adjustment, neglecting of the influence of mining activities and deformation on the arsenic content and the like are solved. According to the method, the accuracy and robustness of soil arsenic content prediction are improved, the model debugging process is simplified, and the method is suitable for soil arsenic pollution accurate monitoring and ecological risk assessment of the grouting and filling mining area.

Description

Technical Field

[0001] This invention belongs to the field of soil arsenic content prediction technology, specifically relating to a spectral prediction model for soil arsenic content based on a continuous projection algorithm. Background Technology

[0002] Grouting and backfilling is a key technology for green mining, effectively controlling surface subsidence. However, leakage of grouting materials and soil structure damage caused by surface deformation during the grouting process can alter the adsorption-desorption balance and spatial distribution of arsenic in the soil, thus affecting the spectral response characteristics of arsenic content. Accurate monitoring of soil arsenic content in such areas is of great significance for mine ecological restoration and environmental risk management.

[0003] Existing spectral prediction techniques for soil arsenic content have the following prominent problems:

[0004] The impact of mining activities and surface deformation is not considered: the existing model relies only on the single correlation between soil spectrum and arsenic content, ignoring the indirect effects of grouting and backfilling mining parameters and surface deformation on arsenic content distribution, resulting in a significant decrease in the accuracy of prediction of mining disturbance areas;

[0005] Significant interference from overlapping spectral peaks: The spectral characteristics of arsenic in soil easily overlap with the spectral peaks of components such as organic matter and clay minerals. Traditional preprocessing methods cannot effectively separate overlapping peaks, leading to distortion in the extraction of sensitive features.

[0006] To address the aforementioned issues, this invention proposes a soil arsenic content spectral prediction model based on the continuous projection algorithm and AdaBoost ensemble learning, aiming to improve prediction accuracy, robustness, and processing efficiency. Summary of the Invention

[0007] The purpose of this invention is to provide a spectral prediction model for soil arsenic content based on a continuous projection algorithm, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a soil arsenic content spectral prediction model based on a continuous projection algorithm, comprising the following steps:

[0009] S1. In the area affected by grouting and filling mining, surface soil samples are collected using a grid-based sampling method according to the mining disturbance gradient. Three duplicate samples are set at each sampling point. Grouting parameters at the mining site are collected in real time using sensors. Then, InSAR technology is used in combination with a total station to collect surface deformation data.

[0010] S2. The arsenic content of soil samples was determined by standard chemical analysis, and the raw spectral data of soil samples were collected by a spectrometer.

[0011] S3. An adaptive iterative reweighted penalized least squares algorithm is used for baseline correction. The number of iterations is set to 20 and the penalty factor is 1000 to eliminate spectral baseline drift. The db4 wavelet basis is used to perform 3-5 layers of wavelet decomposition on the baseline-corrected spectrum, retaining the high-frequency coefficients of the 2nd to 4th layers containing overlapping peak information. Then, the high-frequency noise is removed by the threshold method, and the pure spectrum after the overlapping peaks are separated is reconstructed.

[0012] S4. For each wavelength point of the pure spectrum, calculate the Pearson correlation coefficient with arsenic content, grouting volume, grouting pressure, and settlement, respectively. Calculate the comprehensive correlation coefficient according to the weighting formula, and then use the comprehensive correlation coefficient as a weighting factor to integrate it into the continuous projection algorithm. Set the number of characteristic wavelengths to a range of 15-40, and screen out the set of sensitive characteristic wavelengths that are strongly correlated with arsenic content, grouting parameters, and surface deformation.

[0013] S5. Perform feature fusion with sensitive characteristic wavelength data, grouting and filling mining parameters, and surface deformation data to construct a multi-dimensional input feature set;

[0014] S6. Simplify the parameter tuning process and build a prediction model by combining the AdaBoost algorithm. Then, train the model using a multi-dimensional input feature set to achieve soil arsenic content prediction.

[0015] Preferably, in S1, the sampling frequency of the surface deformation data is consistent with the soil sample collection frequency, with a time difference of ≤24 hours, to ensure the spatiotemporal matching of the data.

[0016] Preferably, in S2, the specific procedures for determining the measured value of arsenic content in soil samples and collecting the original spectral data of soil samples are as follows: After the soil samples are naturally air-dried, impurities are removed, and they are ground through a 100-mesh sieve, the arsenic content is determined by inductively coupled plasma mass spectrometry. The average value of three replicate samples is taken as the measured value. The original spectral data of the soil samples is collected using a visible-near-infrared spectrometer. Each sample is scanned five times, and the average value is taken after removing abnormal spectra.

[0017] Preferably, the spectrometer maintains stable ambient temperature and humidity during the acquisition process, and performs whiteboard calibration before each acquisition.

[0018] Preferably, in S4, the formula for calculating the comprehensive correlation coefficient is R=0.5r1+0.2r2+0.15r3+0.15r4, where r1 is the correlation coefficient between wavelength and arsenic content, r2 is the correlation coefficient with grouting volume, r3 is the correlation coefficient with grouting pressure, and r4 is the correlation coefficient with settlement.

[0019] The weighting formula for the comprehensive correlation coefficient can be dynamically adjusted according to the mining intensity of different mining areas. The adjustment range is as follows: arsenic content weight 0.4-0.6, grouting parameter weight 0.1-0.3, and surface deformation weight 0.1-0.3.

[0020] Preferably, in S5, the process of constructing a multi-dimensional input feature set is as follows: the sensitive feature wavelength data, grouting parameters, and surface deformation data are normalized to the [0, 1] interval using the min-max normalization method, and then spliced ​​together in the order of sensitive spectral features, grouting parameters to surface deformation data to construct a 30-40 dimensional multi-dimensional fusion feature set.

[0021] Preferably, in S6, the simplified parameter tuning process includes: fixing the base learner as a shallow decision tree with a depth of 3-5 layers, limiting the number of base learners to 20-80, and limiting the learning rate to 0.05-0.3, and using an improved particle swarm optimization algorithm to search for parameters, setting the number of particles to 20 and the number of iterations to ≤50, with the minimum root mean square error of the test set as the objective function, and optimizing the two key parameters of the number of base learners and the learning rate;

[0022] The model training process is as follows: the fused feature set in S5 is divided into a training set and a test set in a 7:3 ratio, and input into the lightweight AdaBoost model for training, outputting the predicted soil arsenic content.

[0023] Preferably, the improved particle swarm optimization algorithm introduces a linearly decreasing inertia weight, with an initial inertia weight of 0.9, which decreases to 0.4 at the end of the iteration, in order to balance the global search and local search capabilities.

[0024] Preferably, the model training process employs 5-fold cross-validation, and simultaneously outputs three performance metrics—the coefficient of determination R², the root mean square error, and the mean absolute error—on both the training and test sets to comprehensively evaluate the model performance.

[0025] Preferably, the data input of the prediction model supports batch data import in Excel and CSV formats, and has a built-in data quality verification function that can automatically identify and mark abnormal data.

[0026] Compared with the prior art, the beneficial effects of the present invention are:

[0027] 1. This invention integrates grouting and backfilling mining parameters with surface deformation data into a spectral prediction model, establishing a synergistic correlation between mining, deformation, arsenic content, and spectrum. This solves the problem of prediction distortion in mining-disturbed areas. Furthermore, by extracting key information from spectral features using a continuous projection algorithm and combining mining parameters with dynamic surface deformation data, a high-precision soil arsenic content prediction system is constructed. This not only improves the model's adaptability under complex geological conditions but also optimizes prediction parameters through real-time data feedback. It effectively overcomes the prediction bias problem caused by data gaps in traditional models in mining-disturbed areas, significantly improving the reliability and practicality of the prediction results.

[0028] 2. This invention utilizes a combined algorithm of airPLS and db4 wavelet transform to accurately identify and separate the overlapping portions of arsenic characteristic peaks from those of other components, ensuring clear and accurate extraction of arsenic characteristic information from complex spectral signals. This process not only reduces data noise interference but also significantly improves the accuracy and efficiency of subsequent feature extraction, laying a solid data foundation for building a high-precision soil arsenic content prediction model.

[0029] 3. This invention, by fixing non-critical parameters and focusing on optimizing two core parameters, combined with an improved particle swarm optimization algorithm to shorten the search cycle, not only significantly reduces the complexity of model parameter tuning but also greatly improves tuning efficiency, enabling even non-professionals to complete model debugging in a short time. Simultaneously, the optimized model also shows a significant improvement in prediction accuracy, more accurately reflecting the arsenic content in the soil, providing more reliable technical support for soil pollution monitoring and remediation. Attached Figure Description

[0030] Figure 1 This is a flowchart of the present invention;

[0031] Figure 2 The spectral reflectance of the soil sample in this invention;

[0032] Figure 3 The spectral feature bands selected by the continuous projection algorithm SPA of this invention;

[0033] Figure 4 This is to improve the accuracy of the soil arsenic content prediction model of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Please see Figures 1-4 This invention provides a spectral prediction model for soil arsenic content based on a continuous projection algorithm, comprising the following steps:

[0036] S1. In the area affected by grouting and filling mining, surface soil samples are collected using a grid-based sampling method according to the mining disturbance gradient. Three duplicate samples are set at each sampling point. Grouting parameters at the mining site are collected in real time using sensors. Then, InSAR technology is used in combination with a total station to collect surface deformation data. The sampling frequency of surface deformation data is consistent with the sampling frequency of soil samples, with a time difference of ≤24 hours to ensure the spatiotemporal matching of the data.

[0037] Further, soil sample collection: In the area affected by grouting and backfilling mining, surface soil samples (0-20cm) were collected using a grid-based sampling method according to the mining disturbance gradient (near the mining face, medium distance, and far distance control area). Three replicate samples were set at each sampling point, and a total of 150-200 sample groups were collected.

[0038] Grouting and backfilling mining parameter acquisition: The grouting parameters of each sampling point in the corresponding area are recorded synchronously through sensors (pressure sensors, flow sensors, and proportioning meters, etc.) at the mining site, including grouting time, single-hole grouting volume, peak grouting pressure, cement-water glass ratio, admixture type and dosage.

[0039] Surface deformation data acquisition: InSAR technology (for large-scale monitoring) and total station (for verification in key areas) are used to acquire surface deformation data at sampling points, including cumulative settlement, horizontal displacement, and deformation rate, to ensure the spatiotemporal consistency of deformation data with soil sample collection.

[0040] S2. After the soil samples were naturally air-dried, impurities removed, and ground through a 100-mesh sieve, the arsenic content was determined by inductively coupled plasma mass spectrometry. The average value of three replicate samples was taken as the measured value. The original spectral data of the soil samples were collected using a visible-near-infrared spectrometer. Each sample was scanned five times, and the average value was taken after removing abnormal spectra.

[0041] S3. An adaptive iterative reweighted penalized least squares algorithm is used for baseline correction. The number of iterations is set to 20 and the penalty factor is 1000 to eliminate spectral baseline drift. The db4 wavelet basis is used to perform 3-5 layers of wavelet decomposition on the baseline-corrected spectrum, retaining the high-frequency coefficients of the 2nd to 4th layers containing overlapping peak information. Then, the high-frequency noise is removed by the threshold method, and the pure spectrum after the overlapping peaks are separated is reconstructed.

[0042] Furthermore, a combined algorithm of airPLS baseline correction and db4 wavelet transform decomposition is used to process the original spectrum. The specific process is as follows: The airPLS algorithm is used to eliminate spectral baseline drift. The number of iterations is set to 20 and the penalty factor is set to 1000 to correct the feature distortion caused by the spectral baseline shift. Then, the db4 wavelet basis is used to decompose the baseline-corrected spectrum into 3-5 layers to separate the low-frequency signal (background noise) and the high-frequency signal (feature information). Then, the high-frequency coefficients containing overlapping peak information (layers 2-4) are retained. High-frequency noise is removed by thresholding (default threshold is 0.05). The spectrum is reconstructed to obtain the pure spectrum after the overlapping peaks are separated, thus achieving effective separation of the arsenic characteristic peak from other component peaks.

[0043] S4. For each wavelength point of the pure spectrum, calculate the Pearson correlation coefficient with arsenic content, grouting volume, grouting pressure, and settlement. Calculate the comprehensive correlation coefficient using the weighting formula, and then incorporate this comprehensive correlation coefficient as a weighting factor into the continuous projection algorithm. Set the number of characteristic wavelengths to 15-40, and select the set of sensitive characteristic wavelengths strongly correlated with arsenic content, grouting parameters, and surface deformation. The formula for calculating the comprehensive correlation coefficient is R = 0.5r1 + 0.2r2 + 0.15r3 + 0.15r4, where r1 is the correlation coefficient between wavelength and arsenic content, r2 is the correlation coefficient with grouting volume, r3 is the correlation coefficient with grouting pressure, and r4 is the correlation coefficient with settlement. The weighting formula for the comprehensive correlation coefficient can be dynamically adjusted according to the mining intensity of different mining areas. The adjustment range is: arsenic content weight 0.4-0.6, grouting parameter weight 0.1-0.3, and surface deformation weight 0.1-0.3.

[0044] Furthermore, after obtaining the purified spectrum, a successive projection algorithm is used to extract characteristic wavelengths from the purified spectrum. This algorithm uses vector projection analysis to progressively screen out characteristic wavelength variables with minimal information redundancy and no correlation, thereby effectively compressing the dimensionality of the spectral data, removing redundant information, and retaining characteristic wavelengths highly correlated with soil arsenic content, providing key feature inputs for the subsequent construction of a high-precision prediction model.

[0045] In its implementation, the algorithm first vectorizes the pure spectral data to construct an initial vector space. Then, it randomly selects one vector from the initial vectors as the starting projection vector, calculates the projection value of this vector onto the remaining vectors, and selects the vector with the largest projection value as the next projection vector. This process is repeated until a predetermined number of characteristic wavelength variables are selected. This method ensures that the selected characteristic wavelength variables have the greatest information contribution and the least information overlap in the spectral data, thereby improving the accuracy and stability of the prediction model for soil arsenic content.

[0046] Correlation Calculation: For each wavelength point in the pure spectrum, the Pearson correlation coefficient with arsenic content, grouting volume, grouting pressure, and settlement is calculated to quantify the correlation strength between wavelength and multi-source information. A comprehensive correlation is calculated, with weight allocation reflecting the logic of arsenic content as the core and grouting and deformation as secondary factors. Furthermore, the weights can be dynamically adjusted according to the mining intensity of the mining area (arsenic content weight 0.4-0.6, grouting / deformation weight 0.1-0.3) to adapt to different scenarios.

[0047] SPA algorithm fusion screening: The comprehensive correlation coefficient is incorporated into the continuous projection algorithm (SPA) as a weighting factor, and the number of feature wavelengths is set to be 15-40. Finally, a set of sensitive wavelengths that are strongly correlated with arsenic content, mining parameters and surface deformation are screened out.

[0048] Breaking through the limitations of traditional spectral feature screening that only focuses on the single correlation between arsenic and the spectrum, this method integrates engineering and environmental information such as grouting and deformation, making the screened features more multi-dimensional and interpretable, and providing high-quality and highly targeted input for subsequent soil arsenic content prediction models.

[0049] S5. The sensitive characteristic wavelength data, grouting parameters, and surface deformation data are normalized to the [0, 1] interval using the min-max normalization method, and then spliced ​​together in the order of sensitive spectral features, grouting parameters and surface deformation data to construct a 30-40 dimensional multi-dimensional fusion feature set.

[0050] Furthermore, data standardization employs min-max normalization to uniformly scale sensitive characteristic wavelength data, grouting parameters, and surface deformation data to the [0, 1] interval, eliminating dimensional differences between different data (e.g., grouting volume is in m³, deformation is in cm; direct merging would cause the model to be biased towards a certain type of data due to differences in numerical ranges). The feature splicing order follows the sequence from sensitive spectral features to grouting parameters and then to surface deformation data. This order is designed based on the direct correlation between the data and soil arsenic content. Spectral features directly reflect arsenic content, grouting parameters are mining factors that indirectly affect arsenic distribution, and surface deformation is a result of mining activities. This direct-to-indirect logic helps the model learn correlation patterns more efficiently. Feature set dimensions: Ultimately, a 30-40 dimensional multi-dimensional fusion feature set is constructed, meaning the model will simultaneously learn the distribution patterns of soil arsenic content from three dimensions: spectrum, mining engineering, and topographic changes.

[0051] This multi-source data fusion approach overcomes the shortcomings of traditional models that rely solely on single spectral features and ignore the indirect effects of mining activities and surface deformation, thus significantly improving the accuracy of soil arsenic content prediction.

[0052] S6. Simplify the parameter tuning process and construct a prediction model using the AdaBoost algorithm. Then, train the model using a multi-dimensional input feature set to predict soil arsenic content. The simplified parameter tuning process includes: fixing the base learners to shallow decision trees with a depth of 3-5 layers, limiting the number of base learners to 20-80, and limiting the learning rate to 0.05-0.3. An improved particle swarm optimization algorithm is used for parameter search, setting the number of particles to 20 and the number of iterations to ≤50. The objective function is to minimize the root mean square error of the test set, optimizing the two key parameters: the number of base learners and the learning rate. The model training process is as follows: the fused feature set from S5 is divided into training and test sets in a 7:3 ratio, input into a lightweight AdaBoost model for training, and output the predicted soil arsenic content. The improved particle swarm optimization algorithm introduces a linearly decreasing inertia weight, with an initial inertia weight of 0.9, decreasing to 0.4 at the end of the iteration to balance global and local search capabilities. The model training process employs 5-fold cross-validation, and simultaneously outputs three performance metrics—the coefficient of determination (R²), root mean square error (RMSE), and mean absolute error—on both the training and test sets to comprehensively evaluate model performance.

[0053] Furthermore, the technical logic for simplifying the parameter tuning process is as follows: Parameter tuning is a crucial step in optimizing machine learning models, but the traditional AdaBoost algorithm requires optimizing numerous parameters (base learner type, number, learning rate, regularization coefficient, etc.), resulting in an excessively large parameter search space, leading to long tuning cycles and high barriers to entry. This solution simplifies the process by fixing non-critical parameters, limiting the range of critical parameters, and employing intelligent algorithm search strategies.

[0054] For fixed base learner types, shallow decision trees with a depth of 3-5 layers are chosen as base learners. Shallow decision trees have low complexity, avoiding overfitting while ensuring the diversity of base learners (AdaBoost relies on the "voting" of multiple base learners to improve performance). The key parameter range is limited to only two core parameters: the number of base learners (20-80) and the learning rate (0.05-0.3). The number of base learners determines the ensemble level of the model, and the learning rate determines the contribution weight of each base learner to the final prediction; both are key regulating factors for AdaBoost performance. The intelligent algorithm search uses an improved particle swarm optimization algorithm instead of traditional grid search. The particle swarm optimization algorithm simulates the "swarm intelligence" of birds foraging, rapidly iterating and optimizing in the parameter space, significantly reducing parameter tuning time.

[0055] The principle behind the improved Particle Swarm Optimization (PSO) algorithm: Traditional PSO algorithms suffer from strong global search capabilities in the early stages but slow convergence in later stages. This scheme introduces a linearly decreasing inertia weight for optimization: the initial inertia weight is set to 0.9, at which point the algorithm has strong global search capabilities, exploring potential optimal parameter combinations over a large range. At the end of the iteration, it is decreased to 0.4, at which point the local search capabilities are enhanced, allowing for fine-tuning near previously found optimal solutions. This balances the contradiction between global exploration and local convergence, ensuring that the optimal parameters are found quickly and accurately.

[0056] Implementation details of model training: Model training is the core process of transforming multi-source feature sets into arsenic content prediction capabilities. It is necessary to strictly control data partitioning, training logic and verification methods: The multi-source fusion feature set is divided into a training set (70% of the data, used for model learning patterns) and a test set (30% of the data, used to verify the model's generalization ability) in a 7:3 ratio.

[0057] Model type: Lightweight AdaBoost is used, which is the optimized model obtained through the simplified parameter tuning process described above. Its core is to integrate the prediction results of each base learner into the final output through weighted voting of multiple base learners.

[0058] Validation method: Five-fold cross-validation is used. The training set is further split into five subsets, and four subsets are used for training and one subset for validation, repeated five times, with the average performance value taken. This method avoids performance bias caused by the randomness of data partitioning and provides a more objective assessment of model stability.

[0059] Performance evaluation dimensions: To comprehensively assess the model's accuracy, stability, and generalization ability, the following three metrics should be considered simultaneously: Coefficient of determination R0 2 The linear fit between the model's predicted and actual values ​​is reflected, ranging from 0 to 1; values ​​closer to 1 indicate a better fit. The root mean square error (RMSE) reflects the average deviation between the predicted and actual values, with units consistent with arsenic content (e.g., mg / kg); smaller values ​​indicate higher prediction accuracy. The mean absolute error (MAE) reflects the average absolute deviation between the predicted and actual values. It is also used to measure prediction accuracy, but is less sensitive to outliers than RMSE, and can help assess the model's robustness.

[0060] The spectrometer maintains stable ambient temperature and humidity during the acquisition process, and whiteboard calibration is performed before each acquisition.

[0061] Furthermore, during sampling, areas with flat, debris-free soil surfaces should be selected to ensure that the spectral data accurately reflects the soil arsenic content characteristics. Simultaneously, to minimize external interference, sampling should be conducted outside of periods of strong direct sunlight, ideally on cloudy days or in the early morning or late afternoon when the light is softer. In addition, for soil samples from different plots, detailed information such as their geographical location and soil type should be recorded to allow for a comprehensive consideration of the influence of various factors on soil arsenic content during subsequent analysis.

[0062] The predictive model supports batch data import in Excel and CSV formats, and has a built-in data quality verification function that can automatically identify and mark abnormal data.

[0063] Furthermore, the model employs the Continuous Projection (SPA) algorithm for feature variable selection. This algorithm iteratively calculates and finds the variable combination with the maximum information content and minimum collinearity in the spectral data, effectively reducing data dimensionality and improving model computational efficiency. The selected feature variables are then input into a Support Vector Machine (SVR) to construct the prediction model. The SVR maps the data to a high-dimensional space through a kernel function, achieving high-precision prediction of arsenic content within a framework of minimizing structural risk. A cross-validation strategy is introduced during model training, using five-fold cross-validation to optimize the hyperparameter combination and ensure the model's generalization ability across different datasets.

[0064] The working principle and usage of this invention are as follows: The core working principle, and the basis for the correlation between spectral density and arsenic content, is as follows: Electronic transitions and chemical bond vibrations of arsenic in soil produce characteristic absorption peaks in the visible-near-infrared spectral range. However, the original spectrum suffers from baseline drift, noise interference, and overlapping peaks with other components, causing the signal directly related to arsenic content to be masked. The model uses preprocessing techniques to remove interference and highlight the characteristic spectral signal of arsenic.

[0065] The core role of the Continuous Projection Algorithm (SPA): SPA is a key tool for dimensionality reduction and feature selection in the model. Its principle is based on orthogonal projection and variance maximization. For high-dimensional spectral data (typically containing hundreds of wavelengths), SPA iteratively selects the wavelengths with the largest variance contribution and the strongest independence from the original spectrum through successive projection operations, forming a set of feature wavelengths. The algorithm evaluates the importance of wavelengths by calculating the standardized value of the projection residuals, avoiding redundant information and multicollinearity problems, and reducing the computational complexity of subsequent models. Combined with the correlation constraint between the measured arsenic content and the spectral data, it ensures that the selected feature wavelengths have clear physical meaning (i.e., directly related to the arsenic content).

[0066] Model ensemble optimization principle: In the preprocessing stage, adaptive iterative reweighted penalized least squares correction of baseline drift is used, and overlapping peaks are separated through db4 wavelet transform to solve spectral data quality issues. The prediction model uses the feature wavelengths selected by SPA as input and combines machine learning algorithms (such as AdaBoost and support vector machines) to construct the prediction model. Multi-base learner ensemble or kernel function mapping is used to capture the nonlinear correlation between feature wavelengths and arsenic content, thereby improving prediction accuracy.

[0067] Detailed usage process: Phase 1: Data acquisition and preprocessing

[0068] step Operation content Technical Requirements Purpose Soil sample collection Topsoil samples were collected using a grid-based sampling method, with three replicate samples at each sampling point. Covering areas with different contamination gradients to avoid sample contamination Ensure sample representativeness Sample pretreatment Air dry naturally, remove impurities (stones, plant residues), and grind to pass through a 100-mesh sieve. Sample uniformity ≤ 0.1 mm, moisture content ≤ 5%. Eliminating the influence of differences in physical state on the spectrum Actual arsenic content measurement Arsenic content was determined by ICP-MS, and the average value of three repeated measurements was taken. Measurement accuracy ≤ 0.01 mg / kg Obtain the true labels for model training Spectral data acquisition The samples were scanned using a visible-near-infrared spectrometer (350–2500 nm, resolution ≤3 nm), with each sample scanned 5 times. Scanning environment: constant temperature 20±2℃, constant humidity 50±5%, whiteboard calibration before each scan. Acquire high-quality raw spectral data

[0069] Phase 2: Spectral Data Preprocessing

[0070] Baseline correction: Run the airPLS algorithm, set the number of iterations to 20 and the penalty factor to 1000, to eliminate baseline offset caused by instrument drift.

[0071] Overlapping peak separation: 3-5 layers of decomposition are performed using the db4 wavelet basis, retaining the high-frequency coefficients of the 2nd to 4th layers. Noise is removed by thresholding (threshold 0.05) to reconstruct a pure spectrum.

[0072] Data standardization: The preprocessed spectral data is min-max normalized (scaled to the [0, 1] interval) to eliminate dimensional differences.

[0073] Phase 3: Feature Wavelength Screening Based on SPA

[0074] Parameter settings: Preprocessed spectral data matrix (n_samples×n_wavelengths). n_samples×n_wavelengths is a common matrix representation of spectral data, where n_samples is the number of samples (e.g., spectral data from different substances or measurement points), and n_wavelengths is the number of wavelengths (e.g., the resolution of the spectrometer or the measurement wavelength range). This matrix structure is widely used in spectral analysis, chemometrics, and other fields, such as as input data for algorithms like principal component analysis or multivariate scattering correction. Measured arsenic content vector (n_samples×1). Set the number of target feature wavelengths (typically 15–40, adjusted according to data complexity).

[0075] Algorithm Execution: Initialize the wavelength selection set to empty, and iteratively calculate the projection score (correlation with arsenic content or standardized projection residual value) of each unselected wavelength. In each iteration, select the wavelength with the highest score and add it to the set until the target number of features is reached. Output the filtered feature wavelength index and its corresponding score.

[0076] Results verification: Visualize the score trend of the characteristic wavelengths, remove wavelengths with abnormal scores, and finally determine 20-30 sensitive characteristic wavelengths.

[0077] Phase 4: Model Training and Prediction

[0078] Data partitioning: The characteristic wavelength data and the measured arsenic content values ​​were divided into a training set (modeling) and a test set (validation) in a 7:3 ratio.

[0079] Model building: Select a suitable prediction algorithm (such as lightweight AdaBoost or SVM) and build a model with the feature wavelength as input and arsenic content as output.

[0080] Parameter tuning and optimization: Use particle swarm optimization (PSO) or grid search to optimize model parameters (such as the number of base learners and learning rate in AdaBoost).

[0081] Model validation: Evaluate model performance using a test set. Key metrics include coefficient of determination (R² ≥ 0.85), root mean square error (RMSE ≤ 0.5 mg / kg), and mean absolute error (MAE ≤ 0.3 mg / kg). If performance is unsatisfactory, return to stage 3 to adjust the number of SPA feature wavelengths or optimize preprocessing parameters.

[0082] Phase 5: Practical Application Prediction

[0083] Collect soil samples to be predicted, and repeat the pretreatment and spectral acquisition process. Extract the values ​​of the sample spectrum at the characteristic wavelengths selected by SPA, and input them into the trained prediction model to output the arsenic content prediction results and confidence intervals.

[0084] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to the above embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A spectral prediction model for soil arsenic content based on a continuous projection algorithm, characterized in that, Includes the following steps: S1. In the area affected by grouting and filling mining, surface soil samples are collected using a grid-based sampling method according to the mining disturbance gradient. Three duplicate samples are set at each sampling point. Grouting parameters at the mining site are collected in real time using sensors. Then, InSAR technology is used in combination with a total station to collect surface deformation data. S2. The arsenic content of soil samples was determined by standard chemical analysis, and the raw spectral data of soil samples were collected by a spectrometer. S3. An adaptive iterative reweighted penalized least squares algorithm is used for baseline correction. The number of iterations is set to 20 and the penalty factor is 1000 to eliminate spectral baseline drift. The db4 wavelet basis is used to perform 3-5 layers of wavelet decomposition on the baseline-corrected spectrum, retaining the high-frequency coefficients of the 2nd to 4th layers containing overlapping peak information. Then, the high-frequency noise is removed by the threshold method, and the pure spectrum after the overlapping peaks are separated is reconstructed. S4. For each wavelength point of the pure spectrum, calculate the Pearson correlation coefficient with arsenic content, grouting volume, grouting pressure, and settlement, respectively. Calculate the comprehensive correlation coefficient according to the weighting formula, and then use the comprehensive correlation coefficient as a weighting factor to integrate it into the continuous projection algorithm. Set the number of characteristic wavelengths to a range of 15-40, and screen out the set of sensitive characteristic wavelengths that are strongly correlated with arsenic content, grouting parameters, and surface deformation. S5. Perform feature fusion with sensitive characteristic wavelength data, grouting and filling mining parameters, and surface deformation data to construct a multi-dimensional input feature set; S6. Simplify the parameter tuning process and build a prediction model by combining the AdaBoost algorithm. Then, train the model using a multi-dimensional input feature set to achieve soil arsenic content prediction.

2. The soil arsenic content spectral prediction model based on the continuous projection algorithm according to claim 1, characterized in that, In S1, the sampling frequency of the surface deformation data is consistent with the soil sample collection frequency, with a time difference of ≤24 hours, to ensure the spatiotemporal matching of the data.

3. The soil arsenic content spectral prediction model based on the continuous projection algorithm according to claim 1, characterized in that, In S2, the specific procedures for determining the measured value of arsenic content in soil samples and collecting the original spectral data of soil samples are as follows: After the soil samples are naturally air-dried, impurities are removed, and they are ground through a 100-mesh sieve, the arsenic content is determined by inductively coupled plasma mass spectrometry. The average value of three replicate samples is taken as the measured value. The original spectral data of the soil samples is collected using a visible-near-infrared spectrometer. Each sample is scanned five times, and the average value is taken after removing abnormal spectra.

4. The soil arsenic content spectral prediction model based on the continuous projection algorithm according to claim 1, characterized in that, The spectrometer maintains stable ambient temperature and humidity during the acquisition process, and performs whiteboard calibration before each acquisition.

5. The soil arsenic content spectral prediction model based on the continuous projection algorithm according to claim 1, characterized in that, In S4, the formula for calculating the comprehensive correlation coefficient is R=0.5r1+0.2r2+0.15r3+0.15r4, where r1 is the correlation coefficient between wavelength and arsenic content, r2 is the correlation coefficient with grouting volume, r3 is the correlation coefficient with grouting pressure, and r4 is the correlation coefficient with settlement. The weighting formula for the comprehensive correlation coefficient can be dynamically adjusted according to the mining intensity of different mining areas. The adjustment range is as follows: arsenic content weight 0.4-0.6, grouting parameter weight 0.1-0.3, and surface deformation weight 0.1-0.

3.

6. The soil arsenic content spectral prediction model based on the continuous projection algorithm according to claim 1, characterized in that, In S5, the process of constructing a multi-dimensional input feature set is as follows: The sensitive feature wavelength data, grouting parameters, and surface deformation data are normalized to the [0, 1] interval using the min-max normalization method, and then spliced ​​together in the order of sensitive spectral features, grouting parameters to surface deformation data to construct a 30-40 dimensional multi-dimensional fusion feature set.

7. A soil arsenic content spectral prediction model based on a continuous projection algorithm according to claim 1, characterized in that, In S6, the simplified parameter tuning process includes: fixing the base learner as a shallow decision tree with a depth of 3-5 layers, limiting the number of base learners to 20-80, and limiting the learning rate to 0.05-0.

3. An improved particle swarm optimization algorithm is used to search for parameters, setting the number of particles to 20 and the number of iterations to ≤50. The objective function is to minimize the root mean square error of the test set, and to optimize the two key parameters of the number of base learners and the learning rate. The model training process is as follows: the fused feature set in S5 is divided into a training set and a test set in a 7:3 ratio, and input into the lightweight AdaBoost model for training, outputting the predicted soil arsenic content.

8. A soil arsenic content spectral prediction model based on a continuous projection algorithm according to claim 7, characterized in that, The improved particle swarm optimization algorithm introduces a linearly decreasing inertia weight, with an initial inertia weight of 0.9, which decreases to 0.4 at the end of the iteration, in order to balance the global search and local search capabilities.

9. A soil arsenic content spectral prediction model based on a continuous projection algorithm according to claim 7, characterized in that, The model training process employs 5-fold cross-validation and simultaneously outputs three performance metrics—the coefficient of determination (R²), root mean square error (RMSE), and mean absolute error—on both the training and test sets to comprehensively evaluate model performance.

10. A soil arsenic content spectral prediction model based on a continuous projection algorithm according to claim 1, characterized in that, The prediction model supports batch data import in Excel and CSV formats, and has a built-in data quality verification function that can automatically identify and mark abnormal data.

Images