Hyperspectral characteristic wave band extraction and inversion method

Abstract

The invention discloses a hyperspectral characteristic wave band extraction and inversion method used in the field of remote sensing. The method comprises the steps of ADSPA characteristic wave band extraction and B-XGBoost inversion. The characteristic wave band extraction comprises the steps of data collection and preprocessing to obtain a target variable and hyperspectral image data; and on the basis of the SPA, defining a set of unselected wavebands, randomly selecting an initial iteration waveband, setting a random seed and setting iteration times N to construct an ADSPA, iterating waveband reflectivity data on the ADSPA, and then extracting a characteristic waveband. The inversion model adopts a Bayesian optimization strategy to adjust a plurality of parameters of the model, sets an optimal parameter result as a B-XGBoost model parameter, carries out inversion on a characteristic wave band output by the ADSPA, and outputs R, RMSE and MAE results to evaluate precision. And finally, an SHAP value can be calculated, and the contribution degree of the characteristic wave band screened by the ADSPA to inversion is evaluated again by using the SHAP value.

Description

Technical Field

[0001] This invention is applied to the field of remote sensing monitoring and relates to hyperspectral technology. Specifically, it is a method for extracting and inverting hyperspectral feature bands by combining the Adaptive Continuous Projection Algorithm (ADSPA) and the Extreme Gradient Boosting (XGBoost) model. Background Technology

[0002] Remote sensing technology, especially UAV hyperspectral technology, enables large-scale, high-frequency, and high-precision environmental monitoring. However, hyperspectral data often contains a large amount of redundant information, requiring effective dimensionality reduction methods to improve inversion accuracy. Dimensionality reduction methods typically include feature band extraction and data compression. Feature bands refer to bands that a class of targets is only sensitive to incident light in specific bands. These bands can better reflect the properties and characteristics of the target ground features.

[0003] In the field of remote sensing water quality monitoring, there are many patent applications for water quality monitoring methods using hyperspectral technology. For example, CN110567905B uses the aforementioned training samples to train a DBN dimensionality reduction model and an ElasticNet inversion model. Compared with existing technologies, this method has the advantages of lower requirements for training samples, simpler measurement, and more accurate results. Although DBN can extract deep features, its hierarchical structure may limit its flexibility in feature combination and abstraction. CN117092047A, based on satellite hyperspectral images, transforms satellite hyperspectral remote sensing data to a defined band range; determines a highly correlated two-band dataset with a determination coefficient greater than the determination coefficient threshold; based on the principle that highly correlated two-bands do not appear in the same water quality index's feature remote sensing bands, it traverses and solves all the required comprehensive remote sensing reflectance; uses an artificial neural network method to traverse and establish an inversion model between the target water quality index and the comprehensive remote sensing reflectance; and uses the established inversion model to estimate the water quality index of the target area. The method of this invention can be used to quickly and accurately determine the characteristic remote sensing bands of different water quality indicators, and has more accurate, precise and reliable feature extraction and water quality indicator estimation capabilities. However, this invention only uses linear interpolation for dimensionality reduction, which may not be able to fully extract the nonlinear features in the data, and the artificial neural network model may have overfitting problems, resulting in poor generalization ability of the model. CN119337251A integrated hyperspectral water quality analysis method uses three dimensionality reduction methods for dimensionality reduction processing: principal component analysis, t-distributed random neighborhood embedding, uniform manifold approximation and projection, and performs fusion dimensionality reduction through parameter trade-offs; furthermore, it uses three machine learning algorithms, namely support vector machine, random forest and decision tree, to train and test the hyperspectral water quality inversion model on the dimensionality-reduced spectral data; finally, the coefficient of determination R2 and root mean square error RMSE are selected as the test indicators for the fitting accuracy of the hyperspectral water quality inversion model. Although multiple dimensionality reduction methods are combined, the weight allocation in the fusion dimensionality reduction process may be relatively subjective, lacking an automatic weight optimization method, and lacking optimization of the number of feature bands, which may lead to unstable model performance.

[0004] Continuous Projection (SPA) is a feature selection method that minimizes collinearity in the vector space through forward looping. Utilizing vector projection analysis, it selects feature bands with the lowest redundancy and collinearity, reducing the number of parameters required for modeling and improving modeling efficiency. It does not require calculating the covariance matrix of the entire dataset, which is particularly useful when dealing with large-scale datasets. Furthermore, SPA can progressively adapt to the characteristics of the data, making it potentially more promising than traditional methods in certain situations. The journal (De Almeida Valber Elias, et al., 2018. Vis-NIR spectrometric determination of Brix and sucrose in sugar production samples using kernel partial least squares with interval selection based on the successive projections algorithm, Talanta, 181, 38-43) combines Spectroscopic Spatial Regression (SPA) with kernel-based partial least squares regression (Kernel-PLS). Kernel-PLS can handle nonlinear features in the data, improving the model's fitting ability by mapping the data to a high-dimensional feature space. The SPA algorithm is used for interval selection within Kernel-PLS. This approach retains the advantages of Kernel-PLS in handling nonlinearity while reducing the number of parameters in the model through interval selection, thus improving the model's simplicity and interpretability. Compared to full-spectrum Kernel-PLS, the interval selection continuous projection (iSPA) algorithm-Kernel-PLS model involves fewer parameters. This makes the model simpler, reduces computational complexity, decreases the risk of overfitting, and improves the model's generalization ability. However, SPA also has some limitations under different task scenarios. First, the selection of initial bands is limited, as it relies on manually setting initial bands as the starting point, which affects the final selected band set and model performance. If the initial bands are not optimal, it will lead to suboptimal solutions in the entire band selection process. Second, the number of feature bands set manually is difficult to match with the actual number of feature bands contained in the dataset, resulting in blindness. Third, the lack of a supervision mechanism causes deviations between the feature bands and reality. Therefore, adjusting the traditional SPA algorithm to adapt to the feature band extraction process of hyperspectral data can significantly broaden the application scenarios of hyperspectral remote sensing. To address the shortcomings of the SPA algorithm, this invention innovatively proposes the ADSPA algorithm (Adaptive Continuous Projection Algorithm), which is an improvement on the aforementioned limitations of the SPA algorithm.

[0005] The XGBoost model is a gradient boosting decision tree-based model that enhances the algorithm's generalization and prediction capabilities by incorporating multiple weak learning machines into a robust framework. CN117470776A describes a hyperspectral remote sensing method for chlorophyll a inversion in water quality based on the SSA-XGBoost algorithm, comprising the following steps: acquiring hyperspectral data of the water body and chlorophyll a concentration data of the monitored area; calculating the remote sensing reflectance of the water body based on the hyperspectral data, performing preprocessing, and conducting correlation analysis with the chlorophyll a concentration data; selecting the optimal feature bands or combined bands based on the correlation analysis results; constructing a limit gradient boosting algorithm model based on the optimal feature bands or combined bands; optimizing the model parameters of the limit gradient boosting algorithm using a sparrow search algorithm to obtain the optimal limit gradient boosting algorithm model and verify its accuracy. XGBoost performance largely depends on parameter settings. To achieve higher prediction accuracy and overcome the limitations of manual parameter adjustment, this invention employs a Bayesian optimization method to finely adjust the parameters of the B-XGBoost model, a global optimization strategy based on a probabilistic model. Summary of the Invention

[0006] The technical problem solved by this invention is to provide a method for extracting and inverting hyperspectral feature bands. The method uses ADSPA to extract hyperspectral feature bands, eliminating the limitations of initial band selection in SPA and the arbitrariness of setting the number of feature bands, and establishes a supervision mechanism. The method uses B-XGBoost inversion and utilizes Bayesian parameter optimization to eliminate the limitation of the XGBoost model depending on parameter settings.

[0007] The technical solution adopted in this invention is a method for extracting and inverting hyperspectral characteristic bands, including ADSPA extraction of characteristic bands and B-XGBoost inversion. The steps for ADSPA extraction of characteristic bands include: 1) Data collection and preprocessing: Sample collection and hyperspectral image data collection are carried out simultaneously. The indicators in the samples are monitored as target variables. Hyperspectral image data are obtained using UAVs and hyperspectral sensors. The hyperspectral image data is preprocessed to obtain a hyperspectral reflectance matrix. The preprocessing includes SG smoothing.

[0008] 2) Constructing ADSPA: Define the set Z of unselected bands: ; Select initialization band: Randomly select index k from the set Z. (0) The column is used as the initial band; Set a random seed; the value of the random seed is set in the range of 0-100, and is set to 0 in the first iteration; Set the range of iterations N to: 5 ≤ N ≤ 50; Calculate the orthogonal projection value: For each band index j ∉ Z of the unselected bands in the set Z, calculate the data vector Spec in the j-th column of the hyperspectral matrix according to the following formula j Orthogonal projection in the selected band subspace:

[0009] Calculate the L2 norm: Calculate the L2 norm ∥P Specj ∥2 according to the following formula using the orthogonal projection:

[0010] Select the characteristic band index: Find the maximum norm from the L2 norms calculated in the previous step, and select the characteristic band index k corresponding to this maximum norm from the set Z (n) :

[0011] Add k (n) to the set of characteristic band indices, and remove k from the set Z (n) , replace n with n + 1 for the number of iterations, and set k (n) as the initial band X for the next iteration k （n） [[ID=3,0]].

[0012] Output the final set of characteristic band indices: If n < N, return for continued iteration; if n = N, the iteration ends, and output the final set of characteristic band indices:

[0013] Evaluation of the final set of characteristic band indices W: Divide the set W into 5 equal parts, perform 5-fold cross-validation to train a random forest regressor, and verify the coefficient of determination R² predicted by the random forest regressor each time; The average value of R² evaluates the quality of the characteristic bands [[ID=,]]

[0014] 3) Characteristic band extraction Combine the target variables in step 1) with the hyperspectral band reflectance matrix to form the input matrix of ADSPA, and output the final set of characteristic band indices after calculation in step 2) to complete the extraction of characteristic bands

[0015] Furthermore, the B-XGBoost inversion includes the following steps: 1) B-XGBoost model construction: The XGBoost model is optimized by Bayesian optimization to become a B-XGBoost model, and the key parameters n_estimators, max_depth, and learning_rate are globally searched 2) B-XGBoost inversion: n_estimators, max_depth, and learning_rate are set as B-XGBoost model parameters. The feature band dataset output by ADSPA is read for inversion. The inversion includes model training and parameter optimization, 5-fold cross-validation, target optimization, and outputs R², RMSE, and MAE to evaluate the accuracy.

[0016] Furthermore, after B-XGBoost inversion, SHAP value analysis is performed. The SHAP value is calculated according to the following formula, and the SHAP value evaluates the contribution of the characteristic bands to the inversion:

[0017] The beneficial effects of this invention are as follows: Iterative testing using randomized initial wavebands overcomes the limitations of manually setting initial wavebands and avoids the possibility of local optima or global suboptimal solutions. The introduction of a random forest regressor to evaluate the quality of different combinations of characteristic wavebands eliminates the blindness of manually setting the number of wavebands. The post-supervision mechanism maximizes the matching of various water quality indicators with their characteristic wavebands. The use of a Bayesian parameter-optimized B-XGBoost model for parameter tuning eliminates the limitation of the XGBoost model's dependence on parameter settings. Attached Figure Description

[0018] Figure 1 This is a flowchart of the process of Example 1; Figure 2 This is a distribution map of sampling points in Example 1; Figure 3 This is a concentration graph of various water quality indicators for sample 1 in Example 1; Figure 4 These are photos of the hyperspectral image data acquisition equipment and sampling environment from Example 1; Figure 5 This is the original effect image of hyperspectral reflectance collected in Example 1; Figure 6 yes Figure 5 Image showing the hyperspectral reflectance after SG smoothing; Figure 7 This is a SHAP summary diagram of the water quality indicator NH3-N; Figure 8 This is a graph evaluating the accuracy of the TP inversion model for water quality indicators. Detailed Implementation

[0019] To more clearly illustrate the technical solution of this invention, the following detailed description is provided through the acquisition, characteristic band extraction, and inversion of hyperspectral data of water quality indicators in flowing water bodies. These embodiments are only used to illustrate the invention and do not limit the scope of protection of the invention. For ease of understanding, the following parent-child combinations are explained as follows: SPA (Successive Projection Algorithm): A continuous projection algorithm; one of the methods for extracting hyperspectral feature bands. Hyperspectral images contain a large number of spectral bands, characterized by high data dimensionality and significant information redundancy, which easily leads to problems such as storage pressure, low processing efficiency, and a decrease in classification accuracy as the number of bands increases. Feature band extraction technology reduces data dimensionality, improving analysis efficiency while retaining key information. Hyperspectral feature band extraction is a crucial step in hyperspectral data processing.

[0020] PCA (Principal Component Analysis): One of the hyperspectral dimensionality reduction methods.

[0021] ADSPA (Adaptive Successive Projection Algorithm) is a hyperspectral feature band extraction algorithm. On one hand, it uses the cross-validation method of a random forest regressor to evaluate the impact of different numbers of bands on algorithm performance, ensuring that the selected number of bands provides optimal model performance and avoiding the limitations of subjective selection. On the other hand, by randomly selecting initial bands and conducting multiple experimental analyses, it can explore a wider solution space and has a stronger ability to select better feature combinations. Then, by calculating orthogonal projection and selecting the band corresponding to the maximum projection value, it can more accurately select feature bands, reducing the impact of data redundancy and spectral clutter.

[0022] XGBoost (eXtreme Gradient Boosting): The extreme gradient boosting model, the inversion model used in this invention. Quantitative inversion of hyperspectral remote sensing is the process of converting spectral data into physical and biochemical parameters (such as water quality indicators SS, Turbine, TP, NH3-N, etc.) by analyzing the spectral reflectance characteristics of continuous bands and combining mathematical models or machine learning methods.

[0023] B-XGBoost (Bayes-XGBoost): An XGBoost model optimized using Bayesian methods.

[0024] SHAP (Shapley Additive Explanations): SHAP is widely used in the interpretability analysis of machine learning models, especially when it is necessary to understand the prediction results of complex models. By calculating the SHAP value of each feature, we can intuitively see the specific contribution of each feature to the model output, thereby helping to better understand the model's behavior and decision-making basis.

[0025] SS (Suspended solids): Suspended solids, one of the water quality indicators.

[0026] Turbidity: one of the water quality indicators.

[0027] TP (Total phosphorus): One of the water quality standards.

[0028] NH3-N (Ammonia nitrogen): Ammonia nitrogen, one of the water quality indicators.

[0029] R 2 (R-squared): Coefficient of determination, one of the evaluation indicators for inversion results.

[0030] MAE (Mean absolute error): One of the evaluation metrics for inversion results.

[0031] RMSE (Root mean squared error): One of the evaluation metrics for inversion results.

[0032] -MSE (Negative mean squared error): The negative mean squared error is a rating format that converts the prediction error index (MSE) into a rating format required by the optimization algorithm. The smaller the actual error (MSE), the larger the -MSE value, indicating better parameter performance, which enables the optimization algorithm to automatically find the optimal parameters that minimize the prediction error.

[0033] SG smoothing: A polynomial smoothing algorithm based on the least squares principle, also known as convolutional smoothing. It smooths data by fitting a low-order polynomial within a sliding window while preserving key signal features (such as peaks and inflection points), thus serving as a data denoising tool.

[0034] Python: A computer programming software used in this invention for data processing.

[0035] `n_estimators`: The number of base models (such as decision trees) embedded in XGBoost. In ensemble learning algorithms, this represents the number of base models (such as decision trees). A larger value indicates higher model complexity, but also a corresponding increase in computational cost. A trade-off between performance and efficiency is necessary to avoid overfitting. In this invention, `n_estimators` is selected as an integer from 50 to 500.

[0036] `max_depth`: In XGBoost, the maximum depth of a single decision tree. Limiting the number of branches (i.e., depth) of the tree prevents the model from becoming too complex and improves generalization ability. Excessive depth can lead to overfitting, while insufficient depth may result in underfitting. In this invention, `max_depth` is chosen as an integer from 3 to 10.

[0037] `learning_rate`: The step size in XGBoost, is a core parameter for adjusting the model's convergence speed. A smaller value results in lower computational efficiency, while a larger value leads to faster convergence but lower accuracy. It is often used in conjunction with `n_estimators` for joint optimization. In this invention, `learning_rate` is selected from real numbers ranging from 0.01 to 0.3. Example 1

[0038] The workflow of the water body monitoring method for hyperspectral feature band extraction and inversion in this embodiment is as follows: Figure 1 As shown, the process includes steps such as data collection (top left), ADSPA spectral feature band extraction (bottom left), and XGBoost inversion (right).

[0039] 1. Data Collection and Preprocessing. Water sample collection and hyperspectral image data acquisition are conducted simultaneously. Water quality indicators in the water samples are monitored. Image data of the water body is acquired using a drone and a hyperspectral sensor. The hyperspectral image data is preprocessed to obtain the hyperspectral reflectance matrix. The preprocessing includes SG smoothing.

[0040] In this embodiment, water samples were collected from the Fuyang River in Handan City, Hebei Province. A total of 94 water samples were collected, and the sampling points are distributed as shown in the attached figure. Figure 2 As shown, artificial water body sample collection and hyperspectral image data acquisition were carried out simultaneously. "Simultaneous" means that image data was acquired at the same location while water samples were being collected.

[0041] According to the requirements of the "Technical Specification for Monitoring Surface Water and Wastewater" (HJ / T 91-2002), the water sample collection point is located 0.5 m below the water surface, and two 1000 ml water samples are collected from each sampling point and stored under cold storage. This invention selects turbidity, suspended solids (SS), and NH3 as the parameters for water body sampling. -N (ammonia nitrogen) and TP (total phosphorus) were used as the water quality indicators for testing. Water quality indicator analysis was conducted according to the laboratory analytical methods in the "Surface Water Environmental Quality Standard" (GB 3838-2002). The concentrations of each water quality indicator in the sample are as follows: Figure 3 As shown, the left vertical axis represents the measured concentrations of TP and NH3-N, and the right vertical axis represents the measured concentrations of Turb and SS. The measured concentration data are used as the target variable.

[0042] Figure 4 This embodiment uses hyperspectral image data acquisition equipment and sampling environment photographs. The remote sensing image acquisition utilizes a hyperspectral sensor mounted on a drone. The drone used is a DJI Matrice M300, flying at an altitude of 100 m. The hyperspectral sensor is the iSpecHyper-VM100 from Leisen Optics, with a wavelength range of 400-1000 nm and a total of 300 bands. A 10% calibration reference board is placed to calculate the reflectance of the hyperspectral image. A ground base station provides real-time two-way communication relay, allowing real-time observation of the hyperspectral sensor's acquisition video footage on a computer. The calibration points provide geographical location information for the hyperspectral image.

[0043] After acquiring hyperspectral image data, the computer performs preprocessing, including image cropping, stitching, radiometric correction, and georegistration, to obtain UAV hyperspectral reflectance images. Water reflectance information is then extracted based on sampling points. After removing five outlier samples with significant deviations from the measured concentration, each remaining sample is combined with reflectance data from 300 bands to form an 89×300 preliminary reflectance matrix. This matrix contains a large amount of noise signals, such as... Figure 5 As shown, the initial reflectance matrix is ​​smoothed using an SG filter to improve the smoothness of the spectrum and reduce noise interference, resulting in a smoothed hyperspectral reflectance matrix, as shown. Figure 6 As shown. By Figure 6 It can be seen that the peaks are mainly concentrated around 560~570 nm, 680 nm and 800 nm. After SG smoothing, the noise is significantly reduced, and the trend and characteristic distribution are not affected. However, the difference is reduced and the peak shape is sharper, which highlights the difference to a certain extent.

[0044] This step obtains the target variables and hyperspectral reflectance matrix for water quality standards TP (total phosphorus), NH3-N (ammonia nitrogen), Turbidity, and SS (suspended solids).

[0045] 2. ADSPA Construction and Spectral Feature Band Extraction. This step uses programming software to construct ADSPA and extract feature bands. To fully explore the feature bands of each water quality indicator, this step sets 5 ≤ ​​N ≤ 50 as the search range for the number of feature bands N in ADSPA.

[0046] 1) Constructing ADSPA 1.1 Define the set Z of unselected bands: Set Z is the set of all possible initial hyperspectral bands that have not been selected. Set Z contains all existing feature bands, and subsequent feature band extraction is derived from this set Z. In other words, set Z contains all unselected band indices j, satisfying the following condition:

[0047] Where: k (n) It is the selected band index; j is the index of the unselected band; J is the column number of the hyperspectral matrix, i.e., the total number of bands; N is the total number of currently selected bands.

[0048] Band indexes are identifiers used to label bands, distinguishing different bands; similar to a directory, they serve as an auxiliary structure for quickly locating and accessing data. Before the first iteration begins, the set Z contains all bands used for iterative calculations. Iteration is a process of gradually approaching a problem solution or completing a task by repeatedly performing a series of operations and progressively updating the state. This process ensures that ADSPA avoids repetitive processes when searching for feature bands, searching only within the set Z of unselected bands. This setting avoids the problem of repeated selections caused by the fixed search path in traditional SPA, thus eliminating the limitations of initial band selection.

[0049] 1.2 Setting the Random Seed: The random seed is used to initialize the random number generator. The random number generator produces a random sequence of numbers using Python's built-in algorithm. Different random seeds will generate different random sequences. In set Z, an initial band index k is randomly selected. (0) The algorithm begins iterating from the starting point, then traverses all bands, exploring their characteristic bands. This step introduces the source of randomness in the algorithm. Randomly selecting the starting point is to explore more possibilities and avoid getting trapped in local optima. Setting a random seed ensures the repeatability of the experiment, allowing for subsequent verification of the results. For example, a column can be randomly selected from the TP reflectivity matrix as the initial band X. k （0） and use k (0) This indicates the index of the selected initial band. Initial Band X k （0） It is the kth element in the reflectivity matrix corresponding to TP. (0) The reflectivity of the column. Similarly, NH3-N, Turb, and SS all use this method to extract characteristic bands.

[0050] The initial random seed value is set to 0, and the random seed value ranges from 0 to 100. Fixing the random seed value ensures that the random results generated under that value are the same, thus guaranteeing the reproducibility of the characteristic band results extracted by ADSPA. That is, when ADSPA starts running, it first sets the random seed value, and all subsequent running steps of ADSPA are the results under the condition of that random seed value.

[0051] In the original SPA, the index of the initial iteration band is fixed, and iteration stops when the set number of bands for SPA is met. This setting can lead to local optima or global suboptimal solutions in SPA. ADSPA, by randomly selecting the initial iteration band, can explore the reflectivity matrix more extensively and seek global optima.

[0052] 1.3 Calculating Orthogonal Projection: Orthogonal projection is a key mathematical tool for measuring how much information a band carries that is independent of the selected bands. For each unselected band index j∉Z, calculate the j-th column data vector Spec in its hyperspectral matrix. j Orthogonal projection onto the selected band subspace (an information space composed of spectral data from the selected characteristic bands):

[0053] Among them, Spec j It is the data vector (dimension m×1) in the j-th column of the hyperspectral matrix. n It represents the algorithm's iteration progress, i.e., the current iteration count; Spec n It is a matrix (dimension m×n) composed of the data vectors of the bands selected in the first n iterations. m is the sample size; T represents the transpose operation; Spec n T It is a Spec j The result of the transpose operation.

[0054] 1.4 Calculate the L2 norm: Calculate the L2 norm of the projected vector using orthogonal projection ∥ P Specj ∥2, the L2 norm can quantify the strength of orthogonal projected vectors:

[0055] P Specj (i) is the orthogonal projection vector P Specj The i-th element.

[0056] 1.5. From the calculated L2 norm, calculate the maximum norm (arg max). Only relative comparisons are needed, so remove the square root and calculate |P.Specj (即可), select the band index corresponding to the maximum norm. The L2 norm of the projection vector of this band is the largest, which means it carries the most independent information. This step is the core iterative process of the algorithm. The goal is to find the characteristic band that can provide the most information from the remaining bands, that is, the band with the least information overlap with the selected bands.

[0057]

[0058] From the set Z of currently unselected bands, select the band with the largest norm (i.e., vector length) of the projection vector P Specj and record its index as k (n) . Then add k (n) to the set of selected band indices and remove k from Z (n) to ensure no repeated selection. Increase the iteration count n by 1, i.e., replace n with n + 1, and continue the iterative calculation.至此,完成了当前的迭代计算,将选出的特征波段加入特征波段索引集合,同时将其从待选波段中移除,确保后续不会重复选择. So far, the current iterative calculation is completed. Add the selected characteristic band to the characteristic band index set and remove it from the candidate bands to ensure no repeated selection in the future.

[0059] If the iteration count n < N, that is, the total number of target bands has not been reached, return and continue the iteration. If n = N, the iterative algorithm loop ends, and the final characteristic band index set is output, that is, the set composed of the reflectance data of each characteristic band:

[0060] The set W contains the reflectance matrix of the characteristic bands and the target variable. The reflectance matrix of the characteristic bands is composed of the original spectral reflectance data corresponding to the characteristic bands selected by ADSPA.

[0061] 1.6. Evaluation of the final characteristic band index set W. Use a random forest regressor as a post-supervision mechanism to analyze the quality of the final characteristic band index set W. Divide the set W into 5 equal parts and randomly divide the training set into 5 subsets (folds) of similar sizes. Use 4 of the subsets to train the random forest regressor in turn and use the remaining 1 subset for validation, repeating 5 times to ensure that each subset is used as the validation set once. Calculate the coefficient of determination R² predicted by the random forest regressor for each validation. The final score is the average of the 5 cross-validation R² results. This average R² directly reflects the prediction ability of the characteristic band combination. Evaluate the quality of the characteristic bands through the random forest regressor and 5-fold cross-validation.

[0062] By systematically trying different numbers of bands (N) and different random band iteration starting points, evaluate the final characteristic band index set using the R² metric, and finally select the characteristic band combination with the strongest prediction ability. That is, within the search range of the number of characteristic bands (5 ≤ N ≤ 50), make the random forest regressor reach the highest R 2When the value is given, the corresponding number of characteristic bands N and the set of characteristic bands are obtained, thus yielding the set of characteristic bands with the highest R². The set of characteristic bands and the corresponding reflectance matrix are then output to an Excel file.

[0063] ADSPA makes the following optimizations compared to SPA: 1) It overcomes the limitations of manually setting initial bands by using randomized initial bands for iteration, avoiding the possibility of local optima or global suboptimal solutions. 2) It introduces a random forest regressor to evaluate the quality of different combinations of characteristic bands, optimizing the blindness of manually setting the number of bands. 3) The post-supervision mechanism can maximize the matching of various water quality indicators with their characteristic bands.

[0064] 2) Extraction of spectral feature bands Measured concentration data of TP, NH3-N, Turb, and SS were used as target variables, and combined with the hyperspectral reflectance matrix to form the NH3-N reflectance matrix, TP reflectance matrix, Turb reflectance matrix, and SS reflectance matrix, respectively. ADSPA was then used for iterative calculations to extract characteristic bands from these matrices. Finally, the characteristic bands of TP, NH3-N, Turb, and SS were output as their respective reflectance matrices in Excel files.

[0065] This invention uses measured concentration data as a supervision mechanism, overcoming the limitations of unsupervised traditional SPA, ensuring that the selected feature bands possess both information independence and predictive relevance, and solving the problem of poor adaptability of feature band extraction results to multiple parameters in traditional SPA. For example, the feature bands of TP selected by ADSPA are 495.94 nm, 624.05 nm, and 515.26-528.11 nm, while for NH3-N they are the intervals of 491.64-502.39 nm and 510.97-523.39 nm.

[0066] The characteristic bands selected by the ADSPA algorithm for each water quality indicator in this invention are not entirely the same (TP characteristic bands: 495.94 nm, 624.05 nm, and 515.26-528.11 nm; NH3-N: 491.64-502.39 nm and 510.97-523.39 nm). It can be seen that ADSPA selects different characteristic bands for different water quality indicators, which avoids the generation of local optima and optimizes the blindness of manually setting the number of bands.

[0067] 3. B-XGBoost model construction and inversion: 1) B-XGBoost Model Construction Bayesian optimization is a global optimization strategy based on a probabilistic model. It improves search efficiency by intelligently selecting parameter combinations, overcoming the problem that XGBoost performance is highly dependent on parameter settings and requires manual parameter tuning. This invention uses Bayesian optimization to adjust the n_estimators (number of decision trees), max_depth (maximum tree depth), and learning_rate (learning rate, step size) parameters in the XGBoost model, thus creating the B-XGBoost model.

[0068] The B-XGBoost process includes the following steps: (1) Read the target file: Subsequent parameter tuning will be based on the information in the file, that is, to find a set of tailor-made parameter settings for the file. Different files correspond to different parameter settings. The reflectance matrix files after the extraction of TP, NH3-N, Turb and SS characteristic bands are used as the target files. (2) Define the parameter search range: Determine the model parameters to be optimized and their value range, namely n_estimators (50, 500), max_depth (3, 10), and learning rate (0.01, 0.3). (3) Establish a prediction model: Construct a probabilistic model (Gaussian process model) that can predict parameter performance. (4) Intelligent parameter selection: The expected improvement (EI) strategy is used to select the parameter combination most likely to improve performance in turn. Expected improvement (EI) is a sampling function used for Bayesian optimization. Its core idea is to achieve a balance between exploration and exploitation in order to efficiently find the optimal solution of the objective function. (5) Parameter performance verification: The parameter performance was evaluated using the 5-fold cross-validation method, and 5-fold cross-validation was performed on the reflectance matrix files of TP, NH3-N, Turb and SS respectively.

[0069] (6) Iterative optimization: Repeat steps (2)-(4) until the preset number of evaluations is reached. The number of evaluations is set according to actual needs. For 3-5 dimensional problems, 25-35 evaluations can achieve 95% optimal solution. In this embodiment, it is set to 32, which means that the Bayesian optimization process will perform 32 complete iterations. 32 Bayesian iterations are equivalent to 500 random searches. That is, each iteration will select a new set of parameters through the EI strategy, perform 5-fold cross-validation, evaluate and record parameter performance. (6) Output optimal parameters: Select the best-performing parameter combination from all evaluation results.

[0070] 2) B-XGBoost model inversion Based on the optimal parameter combination output from the B-XGBoost parameter tuning results, the n_estimators, max_depth, and learning_rate of XGBoost are set. Then, the corresponding parameter tuning target file, such as the reflectance matrix file of SS, is read and inverted to finally obtain the R of the water quality index SS. 2 MAE and RMSE. Similarly, the remaining water quality indicators were inverted using B-XGBoost parameter tuning according to the same rule. The inversion results for different water quality indicators are shown in Table 1 below.

[0071] Table 1. Retrieval results of various water quality indicators by ADSPA and B-XGBoost

[0072] The table shows that the R values ​​of TP, NH3-N, Turbine, and SS are... 2 The values ​​were 0.707, 0.580, 0.916, and 0.896, respectively. MAE and RMSE remained in the low range (Turb and SS values ​​were relatively large, but this was due to the data itself, and this situation exists in the industry). The ADSPA algorithm of this invention can filter its characteristic bands according to different targets, greatly reducing the amount of data computation and model complexity, and has high generalization ability. It has a significant advantage in lightweighting the overall inversion model and can effectively, quickly, and efficiently invert water quality index concentrations. Compared with the original SPA-B-XGBoost (results are shown in Table 2), its R... 2The performance of MAE and RMSE is inferior to ADSPA, which is due to the inherent limitations of SPA. The artificially set number of bands and the globally suboptimal solution prevent it from accurately selecting the characteristic bands of water quality indicators, thus failing to achieve high-precision inversion results. Similarly, this invention simultaneously selects the original SPA and Principal Component Analysis (PCA) methods to extract characteristic bands from the dataset. The extracted sensitive bands are used as a new dataset along with the full-spectrum bands for model training and inversion. The unreduced hyperspectral matrix is ​​also used for comparative analysis. The reflectance matrices of the generated characteristic bands are all optimized and inverted using B-XGBoost. The performance of each dimensionality reduction method is evaluated based on the inversion results. Then, the XGBoost model is used for water quality indicator inversion. The parameter search space is defined, including the number of decision trees, the maximum depth of the trees, and the learning rate (n_estimators, max_depth, learning_rate), etc. The Bayesian optimization method of the B-XGBoost model is used for parameter optimization and inversion. The results are shown in Table 2. It can be seen that the full spectrum and PCA perform worse than ADSPA in terms of result evaluation indicators. This indicates that the ADSPA and B-XGBoost systems in this invention can select characteristic bands according to different targets, greatly reducing the amount of data computation and model complexity, and have high generalization ability, showing a significant advantage in lightweighting the overall inversion model. Based on B-XGBoost, water quality concentration can be accurately and efficiently inverted, and the high model accuracy is beneficial for accurately monitoring the water environment quality of urban rivers, reservoirs, and lakes.

[0073] Table 2. Statistics of inversion results for full spectrum, SPA, and PCA Example 2

[0074] To further verify the ability of ADSPA to extract characteristic bands and the quality of the extracted characteristic bands in this invention, this embodiment uses SHAP to provide a new perspective for analyzing the importance of characteristic bands of water quality indicators. By calculating the SHAP value of each feature, the contribution of each characteristic band to the B-XGBoost inversion results can be evaluated. In this embodiment, the results of Example 1 are used in the data acquisition, characteristic band extraction, and inversion parts. That is, based on the research foundation of Example 1, SHAP analysis is performed on the characteristic bands extracted by ADSPA. The main steps include: 1. SHAP value calculation This embodiment utilizes the SHAP analysis method to interpret the prediction results of the B-XGBoost model. The calculation and analysis of SHAP values ​​are performed using Python, and the specific calculation process is as follows:

[0075] In the formula, f is the prediction function of the model (input each row of reflectance data of the sample, output the predicted concentration value). x represents the reflectance data for each row of the input sample; W It is the set of all characteristic bands; N is the number of characteristic bands; S represents the complete set of all characteristic bands (e.g., TP characteristic bands: 495.94 nm, 624.05 nm, and 515.26-528.11 nm, etc.). T is any subset of S (i.e., a set composed of selected bands from S). ∑ T⊑S To sum over all cases where T is a subset of S, the feature set T is a subset of the feature set S. During the calculation of the SHAP value, all possible subsets of S are considered. X S\T The input is the feature set S after removing the feature set T, which means it only contains features from S that do not belong to T.

[0076] 2. Feature Importance Assessment Taking the water quality index NH3-N as an example, SHAP analysis was used to assess the contribution of the characteristic bands of NH3-N to the inversion results. The results are as follows: Figure 7 As shown in the figure, the SHAP plot can be used to interpret the output of the B-XGBoost model. The SHAP value represents the contribution of each feature to the model's prediction result. The horizontal axis is the SHAP value; positive values ​​indicate that the feature value is beneficial to the model's prediction output, and negative values ​​are the opposite. The vertical axis is the distribution of different feature bands. The color axis reflects the magnitude of the feature value, with yellow indicating higher values ​​and blue indicating lower values. The distribution of points shows the relationship between the feature value and the model output, and the color bar on the right can help with understanding. It can be seen that the features at 634.68 nm, 523.83 nm, and 517.40 nm show a positive correlation with the prediction result, that is, the high reflectance (or absorption) intensity of the band has a positive effect on the model's prediction. The wavelength at 513.11 nm shows a negative correlation with the inversion result. Therefore, 634.68 nm, 523.83 nm, and 517.40 nm are the top three most important feature bands for NH3-N, because these three feature bands have the greatest contribution to the model inversion.

[0077] 3. Evaluation of the accuracy of characteristic band removal and inversion To further verify the rationality of the experimental results, this embodiment removed the characteristic bands of each water quality indicator and then performed inversion and SHAP analysis again.

[0078] Taking the water quality index TP as an example, the characteristic bands of TP are removed and marked as TP-remove. Then, it is inverted by B-XGBoost and analyzed by SHAP. The inversion results are also expressed using R. 2 RMSE and MAE were evaluated. Results are as follows: Figure 8 As shown.

[0079] Depend on Figure 8 It can be seen that the R of TP after removing the characteristic bands 2 Although the R² of TP decreased slightly (ΔR²=0.017) from 0.707 to 0.690, its RMSE and MAE increased by 36.8% and 50.0% respectively (from 0.019 and 0.014 to 0.026 and 0.021), indicating that TP's ability to capture subtle spectral changes was impaired. This is because removing feature bands can reduce the model's robustness and its ability to capture subtle information in the data to varying degrees, leading to an increase in RMSE and MAE. Therefore, the above results demonstrate the indispensability of the removed feature bands to the model's interpretability and prove the accuracy and effectiveness of the feature bands extracted by ADSPA.

[0080] In summary, removing all feature bands from TP resulted in varying degrees of decrease in the accuracy of the prediction results for each parameter, significantly impacting the prediction performance of ADSPA-B-XGBoost. Further SHAP analysis revealed a similar decrease in the number and SHAP value of the remaining feature bands, indicating that the residual features contained less information and contributed less to the model. This validates the effectiveness and importance of the feature bands extracted by the ADSPA algorithm and demonstrates that the ADSPA-B-XGBoost-SHAP framework can achieve high-precision water quality index inversion from complex hyperspectral data, representing a novel method for water quality index band extraction.

[0081] The above description is only a preferred embodiment of the present invention in the field of water quality monitoring and is not intended to limit the present invention. The present invention can also be used in the fields of soil monitoring and vegetation monitoring.

Claims

1. A method for extracting and inverting hyperspectral characteristic bands, characterized in that: This includes ADSPA extraction of characteristic bands and B-XGBoost inversion; the steps for ADSPA extraction of characteristic bands include: 1) Data collection and preprocessing: The acquisition of hyperspectral image data by drones and hyperspectral sensors is carried out simultaneously with sample acquisition. The actual monitoring indicators of the samples are used as target variables. The hyperspectral image data is preprocessed to obtain a hyperspectral reflectance matrix. The preprocessing includes SG smoothing. 2) Constructing ADSPA: Define the set Z of unselected bands: ; Select initialization band: Randomly select index k from the set Z. (0) The column is used as the initial band; Set a random seed; the value of the random seed is set in the range of 0-100, and is set to 0 in the first iteration; Set the range of iterations N to: 5 ≤ N ≤ 50; Calculate the orthogonal projection value: For the band index j∉Z of each unselected band in the set Z, calculate the j-th column data vector Spec in the hyperspectral matrix according to the following formula. j Orthogonal projection into the selected band subspace: ; Calculate the L2 norm: Calculate the L2 norm ||P using the orthogonal projection according to the following formula. Specj ∥2: ; Selecting the characteristic band index: Find the maximum norm from the L2 norm calculated in the previous step, and select the characteristic band index k corresponding to the maximum norm from the set Z. (n) : ; k (n) Add to the characteristic band index set, and remove k from the set Z. (n) The number of iterations is replaced by n+1, and k is... (n) Set as the initial band X for the next iteration k （n） ; Output the final set of characteristic band indices: If n < N, return to continue the iteration; if n = N, the iteration ends, and output the final set of characteristic band indices: ; Evaluation of the final feature band index set W: Divide the set W into 5 equal parts, train the random forest regressor with 5 cross-validations, and validate the determination coefficient R² of the random forest regressor prediction each time; evaluate the quality of the feature bands by the average R². 3) Feature band extraction: The target variable from step 1) is combined with the reflectance matrix of each hyperspectral band to form the input matrix of ADSPA. After calculation in step 2), the final set of feature band indexes is output, thus completing the extraction of feature bands.

2. The method for extracting and inverting hyperspectral characteristic bands according to claim 1, characterized in that: The B-XGBoost inversion includes the following steps: 1) B-XGBoost model construction: The XGBoost model is optimized into a B-XGBoost model using Bayesian optimization, and the key parameters n_estimators, max_depth, and learning_rate are searched globally. 2) B-XGBoost Inversion: n_estimators, max_depth, and learning_rate are set as B-XGBoost model parameters. The feature band dataset output by ADSPA is read for inversion. The inversion includes model training and parameter optimization, 5-fold cross-validation, and target optimization. After inversion, R², RMSE, and MAE are output to evaluate the accuracy of the feature bands.

3. The method for extracting and inverting hyperspectral characteristic bands according to claim 2, characterized in that: After the B-XGBoost inversion, SHAP value analysis is performed, and the SHAP value is calculated according to the following formula: 。

Images