Integrated hyperspectral water quality analysis method

Abstract

An integrated hyperspectral water quality analysis method belongs to the field of hyperspectral water quality analysis. First, water quality data and water quality images are collected in the early stage to pre-process the data; secondly, three dimensionality reduction methods are used for dimensionality reduction processing, and fusion dimensionality reduction is performed through parameter trade-off selection; thirdly, a machine learning algorithm is used to train and test the hyperspectral water quality inversion model for the spectral data after dimensionality reduction; finally, the hyperspectral water quality inversion model is selected and optimized. The present invention adopts an innovative fusion strategy in data dimensionality reduction processing, which can achieve better data dimensionality reduction effect, effectively remove noise and redundant information, and provide a more accurate and reliable data foundation; in terms of model training and testing, a variety of machine learning algorithms are used to train and test the spectral data after dimensionality reduction, and the most suitable method is selected according to various indicators for model training and prediction, which can make full use of the advantages of different algorithms and improve the accuracy and reliability of water quality analysis.

Description

Technical Field

[0001] The present invention belongs to the field of hyperspectral water quality analysis, and provides an integrated hyperspectral water quality analysis method, which is an integrated method for monitoring water quality analysis through hyperspectral technology. Background Art

[0002] With the development of society and the improvement of environmental awareness, the demand for water quality monitoring and analysis is becoming increasingly urgent. As an advanced analytical method, hyperspectral technology has shown great application potential in the field of water quality analysis because it can capture rich spectral information in water quality. However, although there are some water quality analysis systems based on hyperspectral technology, these systems still face many challenges in practical applications. Although the existing hyperspectral water quality analysis systems can process and analyze hyperspectral data to a certain extent, their model reasoning capabilities are obviously insufficient. These systems often find it difficult to establish an accurate quantitative relationship model between water quality parameters and spectral data, resulting in the inability to accurately analyze and quantify key components in water quality, such as total nitrogen, total phosphorus, ammonia nitrogen, dissolved oxygen, turbidity, suspended solids, and chemical oxygen demand (COD). This limitation means that the existing systems can only provide preliminary water quality assessment results and cannot meet the actual needs of in-depth and accurate analysis of water quality.

[0003] In addition, the existing systems also have problems of singleness and inaccuracy in data dimensionality reduction processing. Traditional dimensionality reduction methods, such as principal component analysis (PCA), such as the Chinese invention patent of a water quality monitoring system and method based on hyperspectral imaging (CN 116165148A). Although they can reduce the dimension of the data to a certain extent, they often cannot fully retain the important information in the data, resulting in poor performance of the reduced-dimensional data in subsequent model training and prediction. At the same time, the existing system lacks a comprehensive comparison and trade-off selection of different dimensionality reduction methods, and cannot select the most suitable dimensionality reduction method according to the specific data characteristics and analysis requirements. More importantly, the existing system also has limitations in model training and testing. For example, the Chinese invention patent of a water-air collaborative water quality monitoring system and method based on spectral imaging (CN 118130394A). They can usually only use a single machine learning algorithm for model training from beginning to end, and cannot fully utilize the advantages of different algorithms, nor can they select the most suitable algorithm for prediction based on the performance indicators of the model. This limitation limits the accuracy and reliability of the existing system in water quality analysis. Therefore, a new water quality analysis method is urgently needed, which adopts an innovative fusion strategy in data dimension reduction processing. At the same time, the method should also have high flexibility and accuracy in model training and testing, use multiple machine learning algorithms to train and test the reduced spectral data, and select the most suitable method for model training and prediction based on various indicators. Summary of the invention

[0004] In order to achieve efficient and high-precision monitoring of water quality, the present invention designs an integrated water quality analysis method based on a hyperspectral water quality analysis system, wherein the spectral range of the hyperspectral water quality analysis system is 400nm~1100nm. The method adopts an innovative fusion strategy in data dimensionality reduction processing, and combines three methods, namely PCA, t-distributed random neighbor embedding (t-SNE) and uniform manifold approximation and projection (UMAP), to weigh and select parameters to achieve better data dimensionality reduction effect. At the same time, the method is also highly flexible and accurate in the training and testing of hyperspectral water quality inversion models. It can use a variety of machine learning algorithms such as support vector machine (SVM), random forest and decision tree to train and test the reduced spectral data, and select the most suitable method for training and prediction of hyperspectral water quality inversion models according to various indicators.

[0005] To achieve the above purpose, the present invention adopts the following technical solution.

[0006] The integrated hyperspectral water quality analysis method includes the following steps:

[0007] Step 1: Early stage water quality data collection and water quality image collection;

[0008] In the early stage, we prepared the water quality data and water quality images required for the establishment of the hyperspectral water quality inversion model. According to the needs, we selected multiple points at different locations and depths in the area to be collected, and used a hyperspectral camera to collect spectral data at different points. At the same time, we used sensors and chemical methods to obtain the true values ​​of the contents of seven substances, including total nitrogen, total phosphorus, ammonia nitrogen, dissolved oxygen, turbidity, suspended liquid, and COD, at multiple locations and depths in the collection area, to provide basic data for the subsequent establishment of the hyperspectral water quality inversion model.

[0009] Step 2: data preprocessing;

[0010] The water quality images and spectral data collected at different points are preprocessed, and the preprocessing includes reflectivity correction, noise reduction, and preprocessing operations. The details are as follows:

[0011] The reflectance correction is mainly used to analyze the water quality image collected in step 1, in order to convert the DN value (digital quantization value) into the reflectance value (reflectance). In the formula, R ref is the reflectance value of the black and white corrected water quality image, DN raw is the DN value of the original water quality image, DN white is the white frame data of the reference board, DN dark is the camera’s system error DN value, Rwhite is the reflectivity coefficient of the reference plate. After processing the spectral data at different points through reflectivity correction, accurate reflectivity data is obtained.

[0012] The noise reduction described: After the reflectance correction, accurate reflectance data is obtained, and these reflectance data will serve as the basis for subsequent processing. However, there may be other interference and noise factors in the water quality image in step one, which will affect the accurate extraction of the sample's chemical information, so the data needs to be further processed. Savitzky-Golay filtering is used for noise reduction. By performing a sliding window operation on the spectrum of the water quality image, polynomial fitting is used to smooth the signal. In the spectrum, the Savitzky-Golay filter is used to remove high-frequency noise, making the spectrum smoother and helping to extract the main features of the sample. Use SG filtering to denoise the reflectance data to make the data smoother and reduce fluctuations caused by noise. Specifically:

[0013] Savitzky-Golay: Use the following formula for smoothing filtering to improve the smoothness of the spectrum and reduce noise interference.

[0014]

[0015] In the formula, is the smoothed data set; is the collected data set; a ｉ , a0 is the smoothing coefficient, i.e. the original data of period i The weight in the smoothing window; n is the number of data in the sliding window; m is the window width, where n = 2m + 1.

[0016] The pre-processing operation described above: For the reflectance data after noise reduction, some of the reflectance data with obvious scattering differences are processed using multivariate scattering correction to enhance the correlation between the spectrum and the reflectance data. The processed data still has the problem of deviation in the overall spectral shape, and the standard normal transformation is selected to highlight the detail information. Specifically:

[0017] Multivariate scatter correction normalizes each spectrum and adjusts the spectral shape of each band to a standard shape to offset the deviation caused by scattering. It can effectively eliminate the spectral differences caused by different scattering levels, thereby enhancing the correlation between the spectrum and the data. It helps to more accurately extract the chemical information of water quality images.

[0018] First, the average value of all spectra obtained is taken as the "ideal spectrum" , where Data ij is a single sample spectrum vector, Σ represents the sum of all sample spectra, n is the number of calibration samples, is the average value of all spectra.

[0019] Then, the spectrum of each sample is subjected to a univariate linear regression with the average spectrum, and the least squares problem is solved to obtain the baseline shift and offset of each sample. , where Data i For each sample spectrum, k i is the baseline shift for each sample, b i The offset for each sample.

[0020] Continue to correct the spectrum of each sample: subtract the obtained baseline shift and divide it by the offset to obtain the corrected spectrum. , where k i is the baseline shift for each sample, b i The offset of each sample, Data i For each sample spectrum, Data i (MSC) is the spectrum after multivariate radiation correction.

[0021] The standard normal transformation centers and normalizes each spectrum so that its mean in each band is zero and its standard deviation is one. This helps eliminate differences in overall shape and highlights the details in the spectrum.

[0022] Subtract the mean of the elements in the column where each element is located from each element of the original data set and then divide it by the standard deviation of the elements in the column. Where, X ik represents the original spectral value of the kth variable of the ith sample, represents the mean of the kth variable on all samples, S k It represents the standard deviation of the k-th variable over all samples, that is, the standard deviation of all samples at the k-th wavelength position.

[0023] The denoised reflectance is processed through preprocessing operations to enhance the correlation between the spectrum and the data and remove the overall spectral shape caused by scattering and offset in the spectrum.

[0024] Step 3: Data dimensionality reduction method and fusion dimensionality reduction;

[0025] In hyperspectral water quality analysis, dimensionality reduction is a key step that aims to reduce the dimension of data while retaining the most important information. This paper adopts three advanced dimensionality reduction methods: principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE) and uniform manifold approximation and projection (UMAP), and performs fusion dimensionality reduction through parameter trade-off selection to achieve the best dimensionality reduction effect. The details are as follows:

[0026] The principal component analysis (PCA) method described above: PCA is a statistical method that transforms a set of possibly correlated variables into a set of linearly uncorrelated variables, called principal components, through orthogonal transformation. Its goal is to find the direction of maximum variance in the data, namely the principal component. The mathematical expression of PCA is: new =XP , where X new The new data matrix, XP is the original data matrix, and P is the matrix composed of the eigenvectors of the covariance matrix. PCA has higher accuracy for cases where there are linear relationships in the data and strong correlations between water quality parameters.

[0027] The t-distributed stochastic neighbor embedding (t-SNE) described: t-SNE is a dimensionality reduction method based on manifold learning, which reduces the dimensionality of data by calculating the similarity between data points in a high-dimensional space and then retaining this similarity in a low-dimensional space. t-SNE can well capture the local structure in the data and map it to a low-dimensional space. t-SNE is suitable for situations where there are complex nonlinear relationships between data points, and is suitable for situations where the relationship between water quality parameters is complex and the accuracy of the description using a linear hyperspectral water quality inversion model is low.

[0028] The Uniform Manifold Approximation and Projection (UMAP): UMAP is a new manifold learning algorithm that combines local and global structural information to reduce the dimensionality of data. UMAP achieves dimensionality reduction by constructing a topological representation of high-dimensional data points and retaining this topological structure in a low-dimensional space. UMAP is suitable for situations where there is a lot of noise and redundant information in water quality data, and can effectively extract useful information and reduce the dimensionality of the data.

[0029] Step 3.1: For the data preprocessed in step 2, use PCA, t-SNE and UMAP to perform independent dimensionality reduction on the data:

[0030] PCA dimensionality reduction: Calculate the eigenvalues ​​and eigenvectors of the covariance matrix, select the eigenvectors corresponding to the first k largest eigenvalues, and project the data into the subspace composed of these eigenvectors.

[0031] t-SNE Dimensionality Reduction: Use the t-SNE algorithm to map high-dimensional data into two-dimensional or three-dimensional space, maintaining the local structure between data points.

[0032] UMAP Dimensionality Reduction: Use the UMAP algorithm to map high-dimensional data to a low-dimensional space while preserving both global and local structures.

[0033] Step 3.2, after independent dimensionality reduction processing, in order to determine the weight of each method in the fusion, the cross-validation method is used. Divide the data set into several subsets, set aside one subset each time as the test set, and the rest as the training set. Repeat this process multiple times and calculate the performance indicators (accuracy, recall) of each dimensionality reduction method at different compromises. For each method, calculate the average value of its performance indicators in cross-validation. According to the average value of the performance indicators, assign weights to each method. The better the performance of the method, the higher the weight assigned.

[0034] Step 3.3, perform fusion dimensionality reduction:

[0035] The dimensionality reduction results of the three methods are weighted averaged according to the determined weights to obtain the final fusion dimensionality reduction result. The formula for weighted average is: fused =w pca X pca +w t-SNE X t-SNE +w UMAP X UMAP Among them, X fused is the dimension reduction result after fusion, X pca , X t-SNE and X UMAP They are the dimensionality reduction results of PCA, t-SNE and UMAP, w PCA 、w t-SNE and w UMAP is the corresponding weight.

[0036] In this step, the spectral data preprocessed in step 2 is subjected to dimensionality reduction processing through step 3, which effectively reduces the dimension of the data, removes noise and redundant information, makes the data structure more compact, and retains key water quality parameter information. At the same time, the optimized data helps to improve the performance of the subsequent machine learning hyperspectral water quality inversion model, making it more accurate and reliable in classifying or predicting water quality parameters.

[0037] Step 4: Hyperspectral water quality inversion model training;

[0038] After completing the dimensionality reduction process in step 3, the present invention uses three machine learning algorithms, namely support vector machine (SVM), random forest (RandomForest) and decision tree (Decision Tree), to train and test the hyperspectral water quality inversion model on the spectral data after dimensionality reduction. The following is a detailed description of the processing methods, training process and selection strategies of these three methods.

[0039] Support Vector Machine (SVM): Support vector machine is a binary classification algorithm based on the maximum margin principle, which can be extended to a multi-classification algorithm. It finds an optimal hyperplane so that the two types of samples are on both sides of the hyperplane and are farthest from the hyperplane. When processing spectral data after dimension reduction, SVM maps the data to a high-dimensional space through a kernel function to find the optimal classification surface or regression function. The SVM hyperspectral water quality inversion model is trained using a training data set, and the optimal hyperplane is determined by optimizing the objective function.

[0040] Random Forest: Random Forest is an ensemble learning method consisting of multiple decision trees. It improves the accuracy and robustness of the hyperspectral water quality inversion model by constructing multiple decision trees and averaging or voting their results. The spectral data after dimension reduction is used as input features and the water quality parameters are used as target variables. Multiple decision trees are constructed, and each decision tree randomly selects some features and samples for training during training to reduce the overfitting of the hyperspectral water quality inversion model. The random forest hyperspectral water quality inversion model is tested using the test set to evaluate the performance of the hyperspectral water quality inversion model. The final prediction result is obtained by combining the prediction results of multiple decision trees.

[0041] Decision Tree: Decision tree is an algorithm for classification and regression based on tree structure. It recursively selects the best features for partitioning to construct a tree structure for data classification or regression prediction. When processing spectral data after dimension reduction, the decision tree selects the best features for partitioning to obtain the final prediction result. Similar to SVM and random forest, the spectral data after dimension reduction is used as input features and the water quality parameters are used as target variables. The training set is trained using the decision tree algorithm to construct a decision tree hyperspectral water quality inversion model. The decision tree hyperspectral water quality inversion model is tested using the test set to evaluate the performance of the hyperspectral water quality inversion model. In order to avoid overfitting, the decision tree is pruned according to the results to remove unnecessary branches.

[0042] Step 5: Selection and optimization of hyperspectral water quality inversion model;

[0043] Select the coefficient of determination (R 2 ) and root mean square error (RMSE) are used as test indicators for the fitting accuracy of the hyperspectral water quality inversion model. Through the training of the hyperspectral water quality inversion model, the prediction results of the concentration of water quality parameters at different locations and depths in the acquisition area can be obtained, including the concentration values ​​of seven water quality parameters such as total nitrogen, total phosphorus, ammonia nitrogen, dissolved oxygen, turbidity, suspended solids, and COD at each point. Accurate and intuitive information on each water quality parameter can be obtained.

[0044] The coefficient of determination (R 2), also known as goodness of fit, is an important indicator of goodness of fit, and its change reflects the fitting effect of the hyperspectral water quality inversion model. The size of the determination coefficient determines the relationship between the actual value and the predicted value. The calculation and derivation process is as follows:

[0045]

[0046]

[0047] =

[0048] Where x i is the sample data, is the mean value of the sample data, y i To fit the data, SSreg represents the regression sum of squares, and SStot represents the total sum of squares of deviations. 2 The larger the value, the higher the degree of explanation of the independent variable on the dependent variable, and the greater the proportion of the change caused by the independent variable in the total change. 2 The closer it is to 1, the better the fitting degree of the hyperspectral water quality inversion model is and the higher the reference value is. On the contrary, if R 2 The closer it is to 0, the poorer the fitting degree of the hyperspectral water quality inversion model is, and the hyperspectral water quality inversion model does not have a good reference value.

[0049] The root mean square error (RMSE) is a performance indicator for evaluating the hyperspectral water quality inversion model, which can intuitively reflect the degree of deviation between the predicted value and the actual value of the hyperspectral water quality inversion model. The root mean square error is used to evaluate and test the hyperspectral water quality inversion model.

[0050]

[0051] Where Y i is the actual value, is the predicted value, and n is the total number of samples. By calculating the square root of the average of the squared differences between the predicted and observed values, RMSE can clearly show how the hyperspectral water quality inversion model performs in terms of prediction accuracy. RMSE is used to test the accuracy of the hyperspectral water quality inversion model. The smaller the value, the higher the accuracy of the hyperspectral water quality inversion model. Conversely, the larger the value, the lower the accuracy of the hyperspectral water quality inversion model. According to the RMSE and R 2 It is important to select the hyperspectral water quality inversion model for suitable waters for water quality monitoring.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The present invention adopts an innovative fusion strategy in data dimension reduction processing, and combines three methods, namely principal component analysis (PCA), t-distributed random neighbor embedding (t-SNE) and uniform manifold approximation and projection (UMAP), to weigh and select parameters, thereby achieving a better data dimension reduction effect. This fusion dimension reduction strategy not only reduces the dimension of the data, effectively removes noise and redundant information, but also better retains key water quality parameter information, providing a more accurate and reliable data basis for subsequent model training and prediction. The present invention also has high flexibility and accuracy in model training and testing. It can use a variety of machine learning algorithms such as support vector machine (SVM), random forest (Random Forest) and decision tree (Decision Tree) to train and test the spectral data after dimension reduction, and select the most suitable method for model training and prediction according to various indicators. This multi-algorithm training and testing method enables the present invention to make full use of the advantages of different algorithms and improve the accuracy and reliability of water quality analysis. In summary, the integrated hyperspectral water quality analysis method of the present invention is superior to the prior art in data dimension reduction processing, model training and testing, and has significant beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is the overall flow chart of the system. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention is clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Figure 1 shown.

[0056] The integrated hyperspectral water quality analysis method is implemented based on the integrated hyperspectral water quality analysis system. The spectral range of the hyperspectral water quality analysis system is 400nm~800nm. This method introduces an innovative fusion strategy in the field of data dimensionality reduction processing, combining three methods: principal component analysis (PCA), t-distributed random neighbor embedding (t-SNE) and uniform manifold approximation and projection (UMAP). By selecting the parameters of each method, better data dimensionality reduction results can be achieved. At the same time, this method also shows a high degree of flexibility and accuracy in the training and testing stage of the hyperspectral water quality inversion model. It can flexibly use a variety of machine learning algorithms such as support vector machine (SVM), random forest (Random Forest) and decision tree (Decision Tree) to conduct in-depth training and testing of spectral data after dimensionality reduction processing, and accurately select the most suitable hyperspectral water quality inversion model training and prediction methods based on various performance indicators. And analyze the content of seven substances in the water body: total phosphorus, total nitrogen, ammonia nitrogen, dissolved oxygen, turbidity, suspended solids, and COD (chemical oxygen demand).

[0057] The integrated hyperspectral water quality analysis system comprises the following steps:

[0058] Step 1. Water quality data collection and water quality image collection: Water quality images are collected through hyperspectral equipment with a spectral range of 400nm~1100nm. When collecting water quality data for Xishan Reservoir, a total of 100 sampling points were determined in the central area of ​​the reservoir and different locations around it. These sampling points cover different water depths, ranging from 0.5 meters to 5 meters below the water surface, to ensure that representative water quality data is collected. For each sampling point, a hyperspectral camera is used to collect spectral data to obtain spectral data of water quality at different depths. These spectral data will serve as the basis for subsequent analysis. Relevant physiological and chemical instruments are used to obtain the true value content of total phosphorus, total nitrogen, ammonia nitrogen turbidity, suspended solids, COD and dissolved oxygen for the collected samples.

[0059] Step 2: data preprocessing;

[0060] The water quality images and spectral data collected at different points in the Xishan Reservoir are preprocessed, mainly including reflectance correction, noise reduction, and preprocessing.

[0061] Reflectivity correction:

[0062] The main purpose is to analyze the water quality image of Xishan Reservoir collected in step 1, in order to convert the DN value (digital quantization value) into reflectance value (reflectance) After processing the spectral data at different points through reflectivity correction, accurate reflectivity data were obtained.

[0063] Noise Cancelling:

[0064] After the reflectivity correction, accurate reflectivity data was obtained. Since the water body near the shore of the Xishan Reservoir photographed contained more sediment and the water surface ripples in the middle were larger, there were other interference and noise factors in the water quality image, which would affect the accurate extraction of the sample chemical information. Therefore, the data needed to be further processed. The Savitzky-Golay filter was used for noise reduction. By performing a sliding window operation on the spectrum of the water quality image, a polynomial fit was used to smooth the signal. The SG filter was used to denoise the reflectivity data, making the data smoother and reducing the fluctuations caused by noise.

[0065] Preprocessing:

[0066] For the reflectance data after noise reduction, some of which have obvious scattering differences, multivariate scattering correction is used to enhance the correlation between the spectrum and the reflectance data. The processed data also has the problem of deviation in the overall spectral shape, and the standard normal transformation is selected to highlight the detail information.

[0067] First, the average value of all spectra obtained is taken as the "ideal spectrum" , and then perform a univariate linear regression on the spectrum of each sample and the average spectrum, and solve the least squares problem to obtain the baseline shift and offset of each sample. .

[0068] Then calibrate the spectrum of each sample, subtract the obtained baseline shift and divide it by the offset to obtain the corrected spectrum. Through the standard normal transformation, each spectrum is centered and standardized so that its mean in each band is zero and its standard deviation is one. The difference in the overall shape is eliminated and the detailed information in the spectrum is highlighted. Then, each element in the original data set is subtracted from the mean of the element in the column where the element is located and then divided by the standard deviation of the element in the column.

[0069] The denoised reflectance is processed by preprocessing to enhance the correlation between the spectrum and the data and remove the overall spectral shape caused by scattering and offset in the spectrum.

[0070] Step 3: Data dimensionality reduction method and fusion dimensionality reduction;

[0071] Obtain the data after preprocessing in step 2 and perform data dimensionality reduction on it. In order to obtain data with higher accuracy, select a data dimensionality reduction method and perform fusion dimensionality reduction.

[0072] PCA dimensionality reduction: First, PCA was used to reduce the data dimension. The covariance matrix was calculated by preprocessing the spectral data of 100 sampling points in Xishan Reservoir, and the eigenvectors corresponding to the first five largest eigenvalues ​​were selected for dimensionality reduction. These five principal components explained more than 95% of the total variance of the data, effectively retaining the key information of water quality parameters.

[0073] t-SNE dimensionality reduction: The next step is to use the t-SNE algorithm to map the high-dimensional data in the preprocessed spectral data into a two-dimensional space. In the Xishan Reservoir example, the perplexity parameter of the t-SNE algorithm is set to 25, the learning rate is 180, and the number of iterations is 700. The t-SNE algorithm can well capture the local structure in the water quality data and is very effective in analyzing the complex nonlinear relationship between water quality parameters.

[0074] UMAP dimensionality reduction: Finally, the UMAP algorithm is used to map the high-dimensional data in the preprocessed spectral data to a two-dimensional space. The n_neighbors parameter of the UMAP algorithm is set to 19 and the min_dist parameter is set to 0.2. While retaining the global structure of the data, it also effectively extracts useful information from the water quality data.

[0075] Fusion Dimensionality Reduction:

[0076] The performance of three dimensionality reduction methods, PCA, t-SNE and UMAP, was evaluated by cross-validation. In the case of Xishan Reservoir, t-SNE was found to perform best in capturing local structure, with higher accuracy and recall than PCA and UMAP. (PCA accuracy: 0.85, recall: 0.80; t-SNE accuracy: 0.91, recall: 0.89; UMAP accuracy: 0.87, recall: 0.82) Therefore, in the fusion dimensionality reduction, t-SNE is given a higher weight (,w PCA =0.2, w t-SNE =0.5, w UMAP =0.3). The final fusion dimensionality reduction result was obtained by weighted averaging the dimensionality reduction results of the three methods. The result not only reduces the dimension of the data, removes noise and redundant information, but also retains the key water quality parameter information, laying a solid foundation for the subsequent hyperspectral water quality inversion model training and prediction.

[0077] Step 4: Hyperspectral water quality inversion model training;

[0078] After completing the dimensionality reduction process in step three, three machine learning algorithms, namely support vector machine (SVM), random forest and decision tree, were used to train and test the hyperspectral water quality inversion model on the spectral data after dimensionality reduction and the water quality data collected in step one.

[0079] SVM hyperspectral water quality inversion model training: In the data processing of Xishan Reservoir, SVM was selected as the classifier. The SVM hyperspectral water quality inversion model uses the radial basis function (RBF) as the kernel function, the C parameter is set to 1.5, and the gamma parameter is set to 0.3. The SVM hyperspectral water quality inversion model was trained with the training set, and a hyperspectral water quality inversion model that can accurately classify water quality parameters was obtained.

[0080] Random forest hyperspectral water quality inversion model training: In the Xishan Reservoir example, 100 sets of data were used to establish a random forest hyperspectral water quality inversion model. The random forest hyperspectral water quality inversion model consists of 50 decision trees. Each tree is trained by extracting samples from 100 sets of data through a self-service sampling method to improve the accuracy and robustness of the hyperspectral water quality inversion model. During the training process of each tree, a part of the features is randomly selected from all the features to find the best split point. In this way, the random forest hyperspectral water quality inversion model is trained using the training set to obtain a stable hyperspectral water quality inversion model for water quality parameter prediction. The prediction results of each tree are aggregated to form the final hyperspectral water quality inversion model output.

[0081] Decision tree hyperspectral water quality inversion model training: The decision tree hyperspectral water quality inversion model is performed by recursively dividing the data set into smaller subsets, and each division decision is based on a feature and a threshold to create the nodes of the tree. In the example of Xishan Reservoir, the maximum depth of the tree is set to 5 to prevent the hyperspectral water quality inversion model from overfitting. The decision tree is grown by recursively selecting features and thresholds until the maximum depth or the minimum number of samples of the node is reached. To further avoid overfitting, pruning is performed after the tree is fully grown to remove branches that do not contribute much to the performance of the hyperspectral water quality inversion model. Finally, the trained decision tree hyperspectral water quality inversion model can clearly show the relationship between features and water quality parameters.

[0082] Step 5: Selection and optimization of hyperspectral water quality inversion model;

[0083] After the hyperspectral water quality inversion model training is completed, the coefficient of determination (R 2 ) and root mean square error (RMSE) are used as evaluation indicators for the performance of the hyperspectral water quality inversion model. In the case of Xishan Reservoir, we evaluated the hyperspectral water quality inversion model for each water quality parameter. The comprehensive R 2 The and RMSE values ​​are shown in Table 1. Finally, combined with the actual situation of Xishan Reservoir, the support vector machine (SVM) hyperspectral water quality inversion model was selected to analyze the water quality parameters.

[0084] Table 1

[0085]

[0086] The above-described embodiments merely express the implementation methods of the present invention, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. Integrated hyperspectral water quality analysis method, characterized in that: The integrated hyperspectral water quality analysis method comprises the following steps: Step 1: Early stage water quality data collection and water quality image collection; In the early stage, the hyperspectral water quality inversion model is prepared to establish the required water quality data and water quality images; according to the needs, multiple points at different locations and depths in the area to be collected are selected, and the spectral data of different points are collected using a hyperspectral camera; at the same time, sensors and chemical methods are used to obtain water quality parameters at multiple points at different locations and depths in the collection area; Step 2: data preprocessing; Preprocessing the collected water quality images and spectral data at different points, wherein the preprocessing includes reflectivity correction, noise reduction, and preprocessing operations; Step 3: Data dimensionality reduction method and fusion dimensionality reduction; Three dimensionality reduction methods are used: principal component analysis (PCA), t-distributed random neighbor embedding (t-SNE), uniform manifold approximation and projected UMAP, and the best dimensionality reduction effect is achieved by fusion dimensionality reduction through parameter trade-off selection; the details are as follows: Step 3.1: For the data preprocessed in step 2, use PCA, t-SNE and UMAP to perform independent dimensionality reduction on the data: PCA dimensionality reduction: Calculate the eigenvalues ​​and eigenvectors of the covariance matrix, select the eigenvectors corresponding to the first k largest eigenvalues, and project the data into the subspace composed of these eigenvectors; t-SNE dimensionality reduction: Use the t-SNE algorithm to map high-dimensional data into two-dimensional or three-dimensional space, maintaining the local structure between data points; UMAP Dimensionality Reduction: Use the UMAP algorithm to map high-dimensional data to a low-dimensional space while preserving global and local structures; Step 3.2, use the cross-validation method to determine the weight of each method in the fusion; divide the data set into several subsets, set aside one subset each time as the test set, and the rest as the training set, repeat multiple times, and calculate the performance index of each dimensionality reduction method in different compromises; for each method, calculate the average value of its performance index in cross-validation; assign weights to each method according to the average value of the performance index; Step 3.3, perform fusion dimensionality reduction: The dimensionality reduction results of the three methods are weighted averaged according to the determined weights to obtain the final fusion dimensionality reduction result; the weighted average formula is: fused =w pca X pca +w t-SNE X t-SNE +w UMAP X UMAP Among them, X fused is the dimension reduction result after fusion, X pca , X t-SNE and X UMAP They are the dimensionality reduction results of PCA, t-SNE and UMAP, w PCA 、w t-SNE and w UMAP is the corresponding weight; Step 4: Hyperspectral water quality inversion model training; Three machine learning algorithms, support vector machine (SVM), random forest (Random Forest) and decision tree (Decision Tree), were used to train and test the hyperspectral water quality inversion model on the spectral data after dimension reduction. Step 5: Selection and optimization of hyperspectral water quality inversion model; Select the coefficient of determination R 2 The root mean square error (RMSE) is used as the test index of the fitting accuracy of the hyperspectral water quality inversion model; through the training of the hyperspectral water quality inversion model, the prediction results of the water quality parameter concentrations at different locations and depths in the collection area can be obtained, including the concentration values ​​of seven water quality parameters at each point; accurate and intuitive information of each water quality parameter can be obtained.

2. The integrated hyperspectral water quality analysis method according to claim 1, characterized in that: The water quality parameters in step 1 include total nitrogen, total phosphorus, ammonia nitrogen, dissolved oxygen, turbidity, suspension, and chemical oxygen demand.

3. The integrated hyperspectral water quality analysis method according to claim 1, characterized in that: The step 2 is specifically as follows: The reflectivity correction described above: after the spectral data at different points are processed through the reflectivity correction, accurate reflectivity data is obtained; The noise reduction described: using Savitzky-Golay filtering to perform noise reduction processing to reduce fluctuations caused by noise; The preprocessing operation described herein: using multivariate scatter correction to process the noise-reduced data to enhance the correlation between the spectrum and reflectance data; The processed data has the problem of overall spectral shape deviation, and the standard normal transformation is selected to highlight the detailed information.

4. The integrated hyperspectral water quality analysis method according to claim 3 is characterized in that: During the pre-processing operation, specifically: The multivariate scattering correction process is as follows: first, the average value of all spectra obtained is taken as the "ideal spectrum"; second, the spectrum of each sample is subjected to a univariate linear regression with the average spectrum, and the least squares problem is solved to obtain the baseline shift and offset of each sample; finally, the spectrum of each sample is corrected: the obtained baseline shift is subtracted and divided by the offset to obtain the spectrum after multivariate scattering correction; The standard normal transformation is performed by centering and standardizing each spectrum so that its mean in each band is zero and its standard deviation is one.

5. The integrated hyperspectral water quality analysis method according to claim 1, characterized in that: The step five is specifically as follows: The coefficient of determination R 2 : R 2 The closer it is to 1, the better the fitting degree of the hyperspectral water quality inversion model is and the higher the reference value is. On the contrary, if R 2 The closer it is to 0, the poorer the fitting degree of the hyperspectral water quality inversion model is; The root mean square error RMSE: the smaller the RMSE value, the higher the accuracy of the hyperspectral water quality inversion model, and vice versa, the larger the RMSE value, the lower the accuracy of the hyperspectral water quality inversion model; RMSE and R of three hyperspectral water quality inversion models 2 It is important to select the hyperspectral water quality inversion model for suitable waters for water quality monitoring.

Images