Marine chlorophyll concentration inversion method based on machine learning

Abstract

The invention provides a machine learning-based marine chlorophyll concentration inversion method, which comprises the following steps of: clustering remote sensing reflectivity data to obtain clusters corresponding to different water body types, and applying training sets to various machine learning models for training according to the water body types to obtain a plurality of machine learning models; and performing performance verification on different optimization machine models under different water body types by using the verification set to obtain optimal machine learning models under different water body types, and finally forming a chlorophyll concentration dynamic inversion model. And carrying out chlorophyll concentration inversion by adopting the optimal machine learning model of the corresponding water body type, so that the model has a wide application scene. The chlorophyll concentration dynamic inversion model provided by the invention has very strong universality, and the global sea area water color information monitoring capability is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of water color remote sensing, and in particular to a method for inverting ocean chlorophyll concentration based on machine learning. Background Art

[0002] Marine chlorophyll concentration is an important parameter for studying marine primary productivity, carbon cycle and climate change. In existing technologies, ships and other observation platforms are usually used to collect chlorophyll concentration data. However, due to the difficulty in calibration between different sensors and observation methods, existing technologies are difficult to achieve large-scale and long-term monitoring. The development of satellite remote sensing technology has made it possible to monitor chlorophyll concentration on a global scale.

[0003] In the prior art, the commonly used marine chlorophyll inversion model is the OCx algorithm developed by NASA, which is used to estimate chlorophyll concentration from water color satellite data. The maximum band ratio is used to estimate chlorophyll concentration and is widely used in marine chlorophyll concentration monitoring. The OCx algorithm has been extended to mainstream water color sensors such as MODIS and COCTS through band interpolation. Although the OCx algorithm has been widely used, it has certain limitations in some specific areas or water bodies with complex optical characteristics. Since the OCx algorithm was originally designed for open waters, the changes in optical characteristics in clear waters are mainly dominated by phytoplankton and associated substances. Therefore, the algorithm performs well in estimating chlorophyll concentration in a type of water body. In coastal and estuarine waters, due to the presence of suspended particles, colored dissolved organic matter and other interfering substances, the optical complexity of the water body is greatly increased, which in turn affects the accuracy of the algorithm. In addition, high chlorophyll concentrations in eutrophic waters are also prone to overestimation by the OCx algorithm. In order to cope with the differences in inversion results under different optical conditions, the researchers proposed an inversion algorithm based on optical water type (OWT). By classifying different optical water types, the chlorophyll concentration was adjusted according to the specific optical properties of each type. The algorithm combined with OWT classification significantly improved the inversion accuracy of chlorophyll concentration, but it still lacks universality.

[0004] Machine learning and deep learning technologies have shown great potential in the field of chlorophyll concentration inversion. These algorithms can learn complex nonlinear relationships from a large amount of remote sensing and measured data, effectively handle the complexity of data, and significantly improve the accuracy and robustness of inversion. In the existing technology, algorithms such as support vector machine (SVM), random forest (RF), Gaussian process regression (GPR), artificial neural network (ANN) and XGBoost have been proven to be able to effectively estimate chlorophyll concentration in open waters and local waters. However, different machine learning models have different mathematical foundations and assumptions, which will affect their ability to process different types of data. The existing chlorophyll concentration inversion models perform differently when processing various types of data. Some models perform better in processing nonlinear relationships, while others are more suitable for processing high-noise data environments. Therefore, in the process of chlorophyll concentration inversion, the versatility and scalability of a single model are still insufficient. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a general, scalable, high-precision method for inverting marine chlorophyll concentration based on machine learning.

[0006] Technical solution: A method for inverting marine chlorophyll concentration based on machine learning, comprising the following steps: (1) Collect remote sensing reflectance data at several different wavelengths in the target area and the corresponding in-situ measured chlorophyll concentration data to form a bio-optical in-situ data set, and randomly divide the bio-optical in-situ data set into a training set and a validation set according to a set ratio; (2) Using remote sensing reflectance data as the clustering basis, clustering algorithms were used to cluster the bio-optical in situ data sets to obtain several clusters, and each cluster was marked as a different water body type; (3) Dividing the training set according to the water body type obtained in step (2), and training each training set subset obtained after the division on a number of different machine learning models, to obtain a number of optimized machine learning models corresponding to each training set subset; (4) Divide the validation set according to the water body type obtained in step (2), input the validation set into several optimized machine learning models corresponding to each training set subset in step (3), obtain chlorophyll concentration inversion data, compare it with the corresponding in-situ measured chlorophyll concentration data, obtain performance evaluation indicators, and select the optimized machine learning model with the best performance evaluation indicator for each water body type as the optimal machine learning model; (5) Establish a dynamic inversion model for chlorophyll concentration, input the remote sensing reflectance data to be tested, identify the water body type corresponding to the remote sensing reflectance data to be tested, input the optimal machine learning model corresponding to the water body type, and output the corresponding chlorophyll concentration inversion data.

[0007] Specifically, in step (2), the remote sensing reflectance data is first standardized, and then the optimal number of clusters is determined by the elbow rule. The optimal number of clusters is applied to the K-means clustering algorithm to obtain a number of corresponding clusters.

[0008] Preferably, step (3) further includes: using a 1D CNN-Transformer model to extract features from the training set, and then using the feature-extracted training set as input to the machine learning model.

[0009] Specifically, in step (3), a 1D CNN-Transformer model structure is selected for feature extraction according to the water body type. For water body types with stable spectral characteristics, a sequential feature extraction structure is adopted. First, local features are extracted through 1D CNN, and then global features are captured through Transformer. Finally, they are connected to the feature output layer and the final output layer through the Flatten layer. For water body types with complex spectral characteristics, a parallel feature extraction structure is adopted. After the features are expanded through the fully connected layer, they enter the CNN and Transformer branches to extract local and global features respectively, and adaptively fuse them through the weight learning mechanism. Finally, the output is adjusted through the feature output layer and the final output layer.

[0010] Preferably, in step (3), during the training process of extracting features from the training set using the 1D CNN-Transformer model, the feature output layer extracts high-level features from the input data and provides them to the final output layer for predicting chlorophyll concentration. The final output layer transmits loss information to the feature output layer through error feedback, thereby guiding the feature output layer to optimize the feature extraction process.

[0011] Specifically, in step (3), several different machine learning models include: RF, SVM, MLP, XGBoost and CatBoost.

[0012] Preferably, in step (3), grid search and random search are used to optimize hyperparameters for each machine learning model.

[0013] Specifically, in step (4), the performance evaluation index is a comprehensive evaluation index obtained by normalizing the mean absolute error and the root mean square error and performing weighted averaging with the coefficient of determination. The larger the comprehensive evaluation index, the better the performance of the corresponding machine learning model.

[0014] Specifically, in step (4), the weight coefficients of weighted average are set in equal proportion.

[0015] Specifically, the remote sensing reflectance data includes remote sensing reflectance data at wavelengths of 412 nm, 443 nm, 490 nm, 510 nm, 555 nm, and 670 nm.

[0016] Beneficial effect: Compared with the prior art, the significant effect of the present invention is that the present invention clusters the remote sensing reflectance data, and the obtained clusters correspond to different water body types, and then the training sets are applied to a variety of machine learning models for training according to the water body types, and the performance of different optimized machine models under different water body types is verified using the verification set to obtain the optimal machine learning model under different water body types, and finally a chlorophyll concentration dynamic inversion model is formed. The input remote sensing reflectance data to be measured is first divided into water body types, and then the chlorophyll concentration is inverted using the optimal machine learning model corresponding to the water body type, so that the model has a wide range of applicable scenarios. At the same time, the present invention also uses a 1D CNN-Transforme model to extract features from bio-optical in-situ data, which further enhances the credibility and prediction accuracy of the model. The chlorophyll concentration dynamic inversion model provided by the present invention has strong universality and effectively improves the global sea water color information monitoring capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the method of the present invention.

[0018] Figure 2 It is a result diagram of remote sensing reflectance spectra of four types of water bodies after cluster analysis of the present invention.

[0019] Figure 3 This is a comparison chart of the average and normalized remote sensing reflectance spectra of different water body types.

[0020] Figure 4 This is a statistical result diagram of the training set and validation set under the optimal machine learning model corresponding to the OWT1 water body of the present invention.

[0021] Figure 5 This is a statistical result diagram of the training set and validation set under the optimal machine learning model corresponding to the OWT2 water body of the present invention.

[0022] Figure 6 This is a statistical result diagram of the training set and validation set under the optimal machine learning model corresponding to the OWT3 water body of the present invention.

[0023] Figure 7 This is a statistical result diagram of the training set and validation set under the optimal machine learning model corresponding to the OWT4 water body of the present invention.

[0024] Figure 8 It is a comparison chart between the chlorophyll concentration inversion results of the present invention on the test set and the operational inversion results of NASA. DETAILED DESCRIPTION

[0025] The present invention is further described below in conjunction with the accompanying drawings.

[0026] See also Figure 1 As shown, the present invention provides a method for inverting marine chlorophyll concentration based on machine learning, comprising the following steps: (1) Collect remote sensing reflectance data at wavelengths of 412nm, 443nm, 490nm, 510nm, 555nm, and 670nm in the target area and the corresponding in-situ measured chlorophyll concentration data to form a bio-optical in-situ data set. Preprocess the bio-optical in-situ data set to remove outliers and missing values. Use the train_test_split function to randomly divide the bio-optical in-situ data set into a training set and a validation set according to a set ratio. To facilitate the evaluation of the final model, before the training set and validation set are divided, 20% of the data in the bio-optical in-situ data set is randomly divided as a test set, and the remaining 80% of the data is used as a model training set. 75% of the data in the model training set is divided into a training set, and 25% of the data is divided into a validation set.

[0027] (2) The preprocessed remote sensing reflectance data are used as the clustering basis. The remote sensing reflectance data are first standardized by Z score, and then the optimal number of clusters is determined by the elbow rule. The optimal number of clusters is applied to the K-means clustering algorithm to obtain several corresponding clusters, and each cluster is marked as a different water body type.

[0028] The elbow rule increases the number of clusters from 1 to obtain the corresponding sum of square errors (SSE) under different cluster numbers, and performs visualization to obtain a cluster number-sum of square error trend graph. By observing the changing trend of the sum of square errors, the cluster number at the image inflection point, that is, the "elbow" position, is selected as the optimal cluster number. In this embodiment, the optimal cluster number is 4, and a random seed of 42 is used to ensure the reproducibility of the results. Four different clusters are obtained through the K-means clustering algorithm, and the corresponding water body types are marked as OWT1, OWT2, OWT3, and OWT4. The remote sensing reflectance spectra of different water body types after clustering are as follows: Figure 2 As shown, OWT1, OWT2, OWT3, and OWT4 represent eutrophic water bodies, turbid water bodies, clear water bodies, and moderately turbid water bodies, respectively. Figure 3 Shown are the average and normalized remote sensing reflectance of different water body types. Figure 3 Sub-image a is the average remote sensing reflectance. Figure 3Sub-figure b is the normalized remote sensing reflectance. OWT1 has a low reflectance at all wavelengths, reaching the highest point at 550 nm and then decreasing. This characteristic indicates that this type of water body usually has a high chlorophyll concentration and is a eutrophic water body; OWT2 reaches a reflectance peak near 550 nm and decreases gently with wavelength, reflecting the dominant influence of suspended matter and dissolved colored matter, and is a turbid water body. The reflectance of OWT3 reaches a maximum at 400-450 nm and then decreases significantly, indicating that the water body has high transparency and low chlorophyll concentration, and is a clear water body; OWT4 has a high reflectance at 450-500 nm and then decreases, showing that its optical properties are between turbid and clear, and is a moderately turbid water body.

[0029] (3) Using the remote sensing reflectance data at the above wavelengths as the basic features, select a suitable 1DCNN-Transformer model structure according to the water body type to extract features from the training set, and then use the training set after feature extraction as the input of the machine learning model. The specific steps are as follows: OWT1 and OWT3 water bodies have relatively stable spectral characteristics, and the input sequential feature extraction structure. The structure first extracts local features through the convolution layer in the 1D CNN module. Then it is input into the Transformer module, and the multi-head attention mechanism is used to capture global features. Finally, the Flatten layer is used to expand the output into a one-dimensional disconnected feature output layer and a final output layer. The feature output layer and the final output layer are configured through two fully connected layers. The feature output layer sets the number of searchable units, and the activation function is ReLU, which is used to adjust the number of features and feature output. The number of units in the final output layer is 1, and the activation function is linear, which is used to predict the target variable.

[0030] Water types OWT2 and OWT4 have more complex spectral features and are input into a parallel feature extraction structure. First, the original features are expanded through a fully connected layer to enhance the model's perception of band features. Next, the data enters two parallel branches: the CNN branch extracts local features through two layers of convolution and flattens them through a Flatten layer; the Transformer branch normalizes the data, captures global features using a multi-head attention layer, and flattens them through a Flatten layer. Then the feature dimensions of the two branches are adjusted through a fully connected layer to ensure that the output dimensions are consistent. Finally, the outputs of the two branches are adaptively fused through a weight learning mechanism. The weight learning layer contains two fully connected layers, which learn weights for the CNN and Transformer branches respectively, and use a Sigmoid function to limit the weight value between 0 and 1, which is used to dynamically adjust the fusion ratio of local and global features. The fused features are further adjusted through the feature output layer and the final output layer.

[0031] During the training process, the quality of feature extraction and prediction is improved through the coordinated optimization of the feature output layer and the final output layer. Specifically, the feature output layer is responsible for extracting high-level features from the input data, which are provided to the final output layer for predicting chlorophyll concentration. The final output layer passes the loss information to the feature output layer through error feedback to guide it to optimize the feature extraction process. Finally, the entire model improves the quality of feature extraction through continuous weight adjustment.

[0032] The training set after feature extraction is divided according to the four water body types obtained in step (2), and the four training set subsets obtained after the division are trained on different machine learning models respectively to obtain several optimized machine learning models corresponding to each training set subset.

[0033] In this embodiment, the machine learning models used include: RF (random forest), SVM (support vector machine), MLP (multi-layer perceptron), XGBoost (extreme boosting tree) and CatBoost (ordered boosting of classification and regression trees). For each machine learning model, grid search and random search are used to optimize the hyperparameters.

[0034] (4) Divide the validation set according to the water body type obtained in step (2), input the validation set into the optimized machine learning model corresponding to the four training set subsets in step (3), obtain chlorophyll concentration inversion data, compare it with the corresponding in situ measured chlorophyll concentration data, obtain performance evaluation indicators, and select the optimized machine learning model with the best performance evaluation indicator as the optimal machine learning model for each water body type.

[0035] In a typical scenario, the performance evaluation indicators are the normalized mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R 2 ) and then perform weighted averaging together to obtain a comprehensive evaluation index. The larger the comprehensive evaluation index, the better the performance of the corresponding machine learning model. Specifically, RMSE and MAE are normalized to [0, 1] to obtain Standardized RMSE and Standardized MAE, and then reversely converted to obtain Inverted RMSE and Inverted MAE. The weight coefficient of the weighted average is set in equal proportion, and the calculation formula for the comprehensive evaluation index Score is:

[0036] Based on the comprehensive evaluation index results of the four water body types, the optimal machine learning model corresponding to each water body type is determined. Figures 4 to 7 As shown, the optimal machine learning model corresponding to OWT1 is XGBoost ( Figure 4); The optimal machine learning model corresponding to OWT2 is XGBoost ( Figure 5 ): The optimal machine learning model corresponding to OWT3 is CatBoost ( Figure 6 ); The optimal machine learning model corresponding to OWT4 is MLP ( Figure 7 ).

[0037] (5) Establish a dynamic inversion model for chlorophyll concentration, input the remote sensing reflectance data in the test set, identify the water body type corresponding to each set of remote sensing reflectance data, input the remote sensing reflectance data into the optimal machine learning model corresponding to the water body type, and output the corresponding chlorophyll concentration inversion data.

[0038] In order to prove the improvement effect of the chlorophyll concentration dynamic inversion model provided by the present invention, under the same test conditions, a hybrid algorithm combining NASA's business inversion algorithm OCx and CI algorithm was used as a control group to conduct chlorophyll concentration inversion tests. Figure 8 As shown in the figure, the red dotted line indicates the error range of 2 times the actual chlorophyll concentration. Figure 8 Sub-figure a is the chlorophyll concentration inversion result obtained by using the chlorophyll concentration dynamic inversion model provided by the present invention. Figure 8 The four shapes of the data points correspond to four types of water bodies. The diamond, triangle, circle and square represent OWT1, OWT2, OWT3 and OWT4 water bodies respectively. The points of different colors in the figure represent the logarithmic value of the distance from the coastline (km). The color gradually changes from yellow (farther distance) to green and purple (closer distance), showing the model's ability to predict water bodies at different coastal distances. Overall, the model performs stably in waters close to or far from the coastline, with only slightly higher errors in individual areas, such as nearshore waters. Figure 8 Sub-figure b is the chlorophyll concentration inversion result obtained by the hybrid algorithm combining OCx and CI. It can be clearly seen that the chlorophyll concentration inversion results are seriously deviated from the open sea to the nearshore waters. At the same time, the chlorophyll concentration dynamic inversion model provided by the present invention has good performance in terms of mean absolute error (MAE), root mean square error (RMSE) and determination coefficient (R 2 ) are significantly better than the hybrid algorithm combining OCx and CI in all three indicators.

Claims

1. A method for inverting marine chlorophyll concentration based on machine learning, It is characterized in that The following steps are involved: (1) Collect remote sensing reflectance data at several different wavelengths in the target area and the corresponding in-situ measured chlorophyll concentration data to form a bio-optical in-situ data set, and randomly divide the bio-optical in-situ data set into a training set and a validation set according to a set ratio; (2) Using remote sensing reflectance data as the clustering basis, clustering algorithms were used to cluster the bio-optical in situ data sets to obtain several clusters, and each cluster was marked as a different water body type; (3) Dividing the training set according to the water body type obtained in step (2), and training each training set subset obtained after the division on a number of different machine learning models, to obtain a number of optimized machine learning models corresponding to each training set subset; (4) Divide the validation set according to the water body type obtained in step (2), input the validation set into several optimized machine learning models corresponding to each training set subset in step (3), obtain chlorophyll concentration inversion data, compare it with the corresponding in-situ measured chlorophyll concentration data, obtain performance evaluation indicators, and select the optimized machine learning model with the best performance evaluation indicator for each water body type as the optimal machine learning model; (5) Establish a dynamic inversion model for chlorophyll concentration, input the remote sensing reflectance data to be tested, identify the water body type corresponding to the remote sensing reflectance data to be tested, input the optimal machine learning model corresponding to the water body type, and output the corresponding chlorophyll concentration inversion data.

2. The method for inverting marine chlorophyll concentration according to claim 1, Features: In the step (2), the remote sensing reflectance data is first standardized, and then the optimal cluster number is determined by the elbow rule, and the optimal cluster number is applied to the K-means clustering algorithm to obtain a corresponding number of clusters.

3. The method for inverting marine chlorophyll concentration according to claim 1, Features: The step (3) further includes: using the 1D CNN-Transformer model to extract features from the training set, and then using the training set after feature extraction as input to the machine learning model.

4. The method for inverting marine chlorophyll concentration according to claim 3, Features: In the step (3), the corresponding 1D CNN-Transformer model structure is selected according to the water body type for feature extraction. For water body types with stable spectral characteristics, a sequential feature extraction structure is adopted, firstly local features are extracted by 1D CNN, then global features are captured by Transformer, and finally connected to the feature output layer and the final output layer through the Flatten layer; for water body types with complex spectral characteristics, a parallel feature extraction structure is adopted, after the features are expanded by the fully connected layer, they enter the CNN and Transformer branches, extract local and global features respectively, and adaptively fuse them through the weight learning mechanism, and finally adjust the output through the feature output layer and the final output layer.

5. The method for inverting marine chlorophyll concentration according to claim 4, Features: In the step (3), during the training process of extracting features from the training set using the 1D CNN-Transformer model, the feature output layer extracts high-level features from the input data and provides them to the final output layer for predicting chlorophyll concentration. The final output layer transmits loss information to the feature output layer through error feedback, thereby guiding the feature output layer to optimize the feature extraction process.

6. The method for inverting marine chlorophyll concentration according to claim 1, Features: In step (3), the several different machine learning models include: RF, SVM, MLP, XGBoost and CatBoost models.

7. The method for inverting marine chlorophyll concentration according to claim 1, Features: In step (3), grid search and random search are used to optimize hyperparameters for each machine learning model.

8. The method for inverting marine chlorophyll concentration according to claim 1, Features: In step (4), the performance evaluation index is: a comprehensive evaluation index obtained by normalizing the mean absolute error and the root mean square error and performing weighted averaging with the coefficient of determination. The larger the comprehensive evaluation index, the better the performance of the corresponding machine learning model.

9. The method for inverting marine chlorophyll concentration according to claim 8, Features: In the step (4), the weight coefficients of the weighted average are set in equal proportion.

10. The method for inverting marine chlorophyll concentration according to claim 1, Features: The remote sensing reflectivity data includes remote sensing reflectivity data at wavelengths of 412 nm, 443 nm, 490 nm, 510 nm, 555 nm, and 670 nm.

Images