Shallow lake algal bloom and aquatic plant extraction method based on deep learning model

Abstract

The invention relates to a shallow lake algal bloom and aquatic plant extraction method based on a deep learning model. The method comprises the following steps: acquiring Landsat and Sentinel-2 images covering a shallow lake; making, training and testing water bloom and aquatic plant samples of the deep learning model by using the optical image; constructing spectral features, and selecting features of which the importance ranks the first five by using a random forest model to participate in the construction of a deep learning model; and constructing, training and testing a deep learning model, and classifying the algal blooms and the aquatic plants in the shallow lake by using the trained deep learning model, specifically, classifying the algal blooms and the aquatic plants into four categories, namely a water body, the algal blooms, submerged plants and floating-leaf emergent aquatic plants. The spatial distribution of the shallow lake algal blooms and the aquatic plants can be automatically and accurately obtained, and important technical support is provided for dynamic monitoring of the lake algal blooms and the aquatic plants.

Description

Technical Field

[0001] The invention relates to a method for extracting water blooms and aquatic plants in shallow lakes based on a deep learning model, and belongs to the technical field of remote sensing application. Background Art

[0002] Aquatic plants are plants that can grow in water, usually divided into floating-leaf emergent plants and submerged plants. They play a vital role in lake ecosystems, not only helping to improve the transparency of lakes, but also effectively inhibiting the excessive growth of harmful algae, thereby purifying water quality. In addition, aquatic plants provide food sources and habitats for aquatic animals, which is of great significance for maintaining lake biodiversity and ecological balance. However, due to the combined effects of climate change and human activities, the eutrophication problem of shallow lakes is becoming increasingly serious, which has led to an accelerated extinction of aquatic plants. Given the decisive influence of aquatic plants on lake ecosystems, accurately obtaining the temporal and spatial distribution of water blooms and aquatic plants plays a very important role in repairing lake ecosystems.

[0003] At present, the main monitoring methods for water blooms and aquatic plants are field surveys and remote sensing monitoring. Field surveys have high accuracy, but they are labor-intensive and time-sensitive. Remote sensing technology has the advantages of short revisit time and the ability to monitor spatial distribution, and has gradually become an effective means of monitoring water blooms and aquatic plants.

[0004] The main methods for remote sensing monitoring of water blooms and aquatic plants are the index method and the supervised classification method. Since the index method is simple in principle and easy to implement, it has been widely used in remote sensing monitoring of water blooms and aquatic plants. The supervised classification method has high accuracy and automation, but it has certain requirements on the data and quality of training samples. However, the existing methods cannot meet the accuracy requirements well in the process of monitoring water blooms and aquatic plants. On the one hand, since aquatic plants are affected by the lake water environment, the spectral characteristics of aquatic plants are small, which makes it difficult to distinguish different types of aquatic plants. On the other hand, the spectral characteristics of water blooms and aquatic plants are extremely similar, which causes great interference in distinguishing the two. In this case, the use of new intelligent algorithms for distinction has become a research hotspot.

[0005] In recent years, the emergence of deep learning methods has brought new ideas and methods for the extraction of water blooms and aquatic plants. However, when facing inland shallow lakes, the situation becomes more complicated. These lakes are often severely eutrophic. Not only do they grow a variety of aquatic plants, but they are often accompanied by the appearance of water blooms. This coexistence of water blooms and aquatic plants poses a higher challenge to the extraction accuracy of deep learning models. At present, the accuracy of most algorithms in extracting water blooms and aquatic plants is not ideal. Summary of the invention

[0006] The technical problem to be solved by the present invention is: in view of the fact that the accuracy of most current algorithms in extracting water blooms and aquatic plants is not ideal, a method for extracting water blooms and aquatic plants in shallow lakes based on a deep learning model is proposed. This method uses Landsat images as a data source, designs a deep learning model for extracting water blooms and aquatic plants, and applies this network to shallow lakes with severe eutrophication to achieve accurate classification of water blooms and aquatic plants.

[0007] In order to solve the above technical problems, the technical solution proposed in the present invention is: a method for extracting water blooms and aquatic plants in shallow lakes based on a deep learning model, comprising the following steps:

[0008] The first step is to obtain the Landsat reflectance image and Sentinel-2 image of the study area within the specified time range. The imaging time difference between the Landsat reflectance image and the corresponding Sentinel-2 image should not exceed one day.

[0009] The second step is to visually interpret and sample the Landsat reflectivity images with the help of Sentinel-2 high-resolution images to accurately produce training samples and test samples, which correspond to the categories in the visual interpretation results;

[0010] The third step is to construct characteristic indices for extracting water blooms and aquatic plants, and input the constructed characteristic indices and each band of the Landsat reflectance image into the random forest model, obtain the importance of the characteristic indices and rank the importance, select the top five characteristic indices, and construct the characteristic image of the Landsat reflectance image based on the five characteristic indices for the training samples; (the size is 5×256×256 (5 bands, the number of rows and columns is 256×256)).

[0011] Step 4: Build a deep learning model for water bloom and aquatic plant extraction, including encoder, pyramid pooling module ASPP and decoder;

[0012] Step 5: The feature images of the training samples and the corresponding visual interpretation results (labels) are divided into training sets and validation sets in proportion, and input into the constructed deep learning model for training. After that, multiple iterations are performed and the training model with the highest accuracy is saved to obtain appropriate training hyperparameters.

[0013] Step 6: Test and evaluate the trained deep learning model based on the test samples;

[0014] Step 7. Input the characteristic images of Landsat reflectivity images of the study area at different periods into the trained deep learning model for processing to obtain the spatial distribution maps of water blooms and aquatic plants at different periods.

[0015] Furthermore, in the third step, the top five characteristic indices are the normalized difference water index MNDWI, green light band G, shortwave infrared band SWIR, floating algae index FAI and enhanced vegetation index EVI, among which,

[0016]

[0017] In the formula, NIR is, R is the red light band, NIR is the near infrared band, B is the blue light band, λ NIR is the wavelength of the near-infrared band, λ R is the wavelength of the red light band.

[0018] Furthermore, in the deep learning model, the encoder includes a convolution layer and a four-layer ConvNeXt network. The feature image input to the encoder is convolved and normalized to obtain a feature map and sent to the ConvNeXt network. Each layer of the ConvNeXt network performs convolution and pooling operations on the feature map in turn, and inputs the feature image to the ASPP module for processing. The ASPP module uses a multi-scale strategy to perform hole convolutions with different expansion rates on the input feature map to obtain a multi-scale feature map, and then fuses the multi-scale feature map to obtain a fused feature map and sends it to the decoder; the Scse attention module is added to the decoder, and the decoder performs three iterative processes on the fused feature map, each iteration includes bilinear interpolation upsampling and fusion with the feature map of the corresponding layer in the encoder, and uses the Scse attention module for enhancement processing, and then performs convolution operation again. After the iteration is completed, bilinear interpolation upsampling is performed again to the size of the original input image, and convolution processing is performed to generate a single-band output image.

[0019] The present invention is mainly aimed at eutrophic shallow lakes, and can automatically and accurately obtain the temporal and spatial distribution of water blooms and aquatic plants, providing important technical support for dynamic monitoring of lake water blooms and aquatic plants, and further providing important reference for lake eutrophication control and water environment restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be further described below in conjunction with the accompanying drawings.

[0021] Figure 1 It is to study the geographical location of lakes and false color Landsat images.

[0022] Figure 2 It is a schematic diagram of the process of the present invention.

[0023] Figure 3 It is a schematic diagram of data augmentation.

[0024] Figure 4 This is the network structure of AAENet, a deep learning model for water bloom and aquatic plant extraction.

[0025] Figure 5 It is the network structure of the ASPP module.

[0026] Figure 6 The training loss and accuracy curves during model training. (a) Training loss. (b) Accuracy curve.

[0027] Figure 7 Landsat false color images of the three test lakes and the corresponding classification results. (a) Landsat false color image. (b) Classification results of AAENet, (c) UNet, (d) Deeplab v3, (e) PSPNet, and (f) VBI algorithms.

[0028] Figure 8 This is the result map of the five-feature image and model output of the Taihu Lake area on May 27, 2017. DETAILED DESCRIPTION

[0029] The present invention is described in detail below with reference to the accompanying drawings to make the steps and effects of the present invention clearer.

[0030] Figure 1 The locations and Landsat images of the three study lakes, including Chaohu Lake, Taihu Lake, and Changdang Lake, are serious eutrophication problems. Data source: Landsat reflectivity image data and Landsat images obtained through the Google Earth Engine cloud platform.

[0031] like Figure 2 As shown, it is a flow chart of a method for extracting water blooms and aquatic plants from shallow lakes based on a deep learning model according to an embodiment of the present invention. The method of this embodiment specifically includes the following steps:

[0032] The first step is image preparation: Landsat 8 / 9 reflectance image data from 2013 to 2023 were selected on the Google Earth Engine cloud platform. These images should not only cover the three research lakes, but also ensure that the cloud cover is less than 30% to ensure data quality. Subsequently, Sentinel-2 images with no cloud coverage, high imaging quality, and imaging time that is no more than one day different from the imaging time of the target Landsat image were selected on the Google Earth Engine cloud platform.

[0033] The second step is to create training and test samples: visually interpret the Landsat images with the assistance of Sentinel-2 high-resolution images to accurately create training samples and test samples, which correspond to the categories in the visual interpretation results. The visual interpretation results have four categories: water bodies, water blooms, submerged plants, and floating-leaf emergent plants, which are corresponded to 1, 2, 3, and 4 respectively and saved in the form of labels. The image size of the training samples is 256×256, and the test samples are shapefile point data. The training samples and test samples come from three eutrophic shallow lakes, multiple imaging periods, and multiple scenes. In order to obtain sufficient training samples, the produced training samples are subjected to data enhancement processing. Data enhancement includes horizontal flipping, vertical flipping, and diagonal mirroring operations on the image and the corresponding label image. Figure 3 This is the data enhancement operation.

[0034] The third step is feature construction and selection: a series of feature indices for extracting water blooms and aquatic plants are constructed, and the constructed feature indices and Landsat reflectivity bands are input into the random forest model to obtain the importance of the features. The five most important features are selected to train the deep learning model; the formulas and importance of a series of features input into the random forest model are shown in Table 1.

[0035] Table 1

[0036]

[0037] Based on the importance ranking in Table 1, the present invention selects five features, MNDWI, G, SWIR, FAI and EVI, to input into the deep learning model, where λ NIR is the wavelength of the near-infrared band, λ R is the wavelength of the red light band.

[0038] Step 4: Build a deep learning model (AAENet) for water bloom and aquatic plant extraction: Figure 4 The convolutional neural network structure diagram for extracting water blooms and aquatic plants includes an encoder, a pyramid pooling module ASPP, and a decoder. In this model, the encoder contains a convolutional layer and a four-layer ConvNeXt network. ConvNeXt can achieve the best balance between accuracy and computational complexity. The feature image input to the encoder is convolved and normalized to obtain a feature map and sent to the ConvNeXt network. Each layer of the ConvNeXt network performs convolution and pooling operations on the feature map in turn, and inputs the feature image to the ASPP module for processing. The ASPP module uses a multi-scale strategy to perform hole convolutions with different expansion rates on the input feature map to obtain a multi-scale feature map to extract deep semantic features. The multi-scale feature maps are then fused to obtain a fused feature map and sent to the decoder. Figure 5It is the network structure of the ASPP module. The decoding part restores the small-sized feature map to its original size, thereby obtaining the water bloom and aquatic plant segmentation map. In this embodiment, the Scse attention module is added to the decoder, and the decoder performs 3 iterations on the fused feature map. Each iteration includes bilinear interpolation upsampling and fusion with the feature map of the corresponding layer in the encoder, and the Scse attention module is used for enhancement processing, and then the convolution operation is performed again. After the iteration is completed, bilinear interpolation is performed again. Upsampling to the size of the original input image, and convolution processing is performed to generate a single-band output image. During the decoding process, the output feature map of the bilinear interpolation is input into the Scse attention module, thereby further improving the performance of the model.

[0039] The ConvNeXt network proposed in 2022 is an innovative deep learning model that combines the design elements of convolutional neural networks and draws on the advantages of the Transformer architecture to explore the performance limits of convolutional neural networks. When processing complex image tasks, ConvNext can better cope with transformations such as translation, rotation, and scaling in images, improving the robustness and generalization of the model. With its high efficiency and accuracy, ConvNext has demonstrated excellent performance in fields such as image classification and semantic segmentation. Therefore, this article uses the ConvNext network as a feature extraction network.

[0040] The ASPP module uses a multi-scale strategy to perform hole convolutions with different dilation rates on the input feature map, and then fuses these multi-scale feature maps. This design not only effectively restores the detailed information of the image, but also significantly improves the model's ability to capture deep semantic features, thereby significantly enhancing the accuracy and robustness of semantic segmentation while maintaining a high resolution. In view of the relatively small size of the feature map output by the ConvNext encoder, in order to more effectively extract deep semantic features, the present invention specifically adjusts the hole convolution dilation rate of the ASPP module to 1, 2, 3, and 6. This smaller dilation rate setting can still maintain the edge details of key information such as water blooms and aquatic plants while reducing the parameters of the deep learning model, thereby further optimizing the performance of the model.

[0041] The attention mechanism plays an important role in deep learning models. It enables the model to focus on and strengthen important features, thereby significantly improving the accuracy and operating efficiency of the model. This mechanism has been widely used in semantic segmentation tasks. The present invention introduces the Scse attention mechanism to further improve the accuracy of model prediction. The Scse module does not change the dimensionality of the feature map, but significantly improves the model's ability to finely correct the feature map, especially the edges of water blooms and aquatic plants.

[0042] A five-band image with an image size of 256×256 is input into the AAENet model. The image is first processed through four ConvNext blocks to obtain a feature map with an image size of 768×8×8. The obtained feature map with an image size of 768×8×8 is then input into the ASPP module, which effectively extracts deep semantic features through multi-scale operations without changing the dimension size of the feature map. Then, the model uses three iterative upsampling processes. Each iteration includes: (1) bilinear interpolation upsampling of the feature map; (2) connecting the upsampled feature map with the feature map output by the ConvNext block of the corresponding level to fuse features at different levels; (3) using the Scse module to enhance the upsampled features. (4) These features are further integrated through convolution operations; after these three iterations, the size of the feature map is restored to 96×64×64, which not only restores the spatial resolution but also significantly enhances the expressiveness of the features. Finally, in order to generate the final output of the model, the restored feature map is again bilinearly interpolated and upsampled to the size of the original input image, i.e., 96×256×256, and then processed through a convolutional layer to generate a single-band output image with an image size of 256×256.

[0043] The following is a detailed description of the steps combined with the change in image size. The model processing process is as follows:

[0044] 1. The input to the AAENet model is a 5×256×256 feature map (the feature map is 5 bands, and the rows and columns are 256×256 pixels). The 5 bands here are selected based on "feature selection".

[0045] 2. After “4×4 convolution and normalization”, the size of the feature map becomes 96×64×64.

[0046] 3. After being processed by the four ConvNext blocks, the feature map sizes become 96×64×64, 192×32×32, 384×16×16, and 768×8×8 respectively. The ConvNext block contains two Conv operations.

[0047] 4. Input the feature map of image size 768×8×8 into the ASPP module, and the output feature map size is still 768×8×8.

[0048] 5. The three-step upsampling process includes: bilinear interpolation, fusion with the feature map output by the corresponding ConvNext block, Scse module, and two convolutions.

[0049] The size of the input feature map of the first upsampling is 768×8×8; the size of the feature map after bilinear interpolation is 384×16×16; the size of the feature map after fusion with the feature map output by the corresponding ConvNext block is 768×16×16; the Scse module does not change the size of the feature map, and the feature map size remains 768×16×16; the size of the feature map after two convolutions is 384×16×16.

[0050] The feature map sizes in the second upsampling process change to 192×32×32, 384×32×32, 384×32×32, and 192×32×32.

[0051] The feature map size changes in the third upsampling process are 96×64×64, 192×64×64, 192×64×64, 96×64×64.

[0052] 6. Finally, the feature map with a size of 96×64×64 is upsampled by bilinear interpolation, and the feature map size changes to 96×256×256. It is then processed through a convolutional layer to generate a single-band output image with an image size of 256×256.

[0053] The fifth step is to train the deep learning model: the characteristic images of the training samples and the corresponding visual interpretation results are randomly divided into training sets and validation sets at a ratio of 80% and 20% (the training samples and the corresponding visual interpretation results produced in the second step can be divided first, and the model training is performed based on the characteristic images of the training samples). The training set and validation set are respectively input into the built deep learning model for training. After that, multiple iterative experiments are performed, and the training model with the highest accuracy is saved to obtain the appropriate training hyperparameters. That is, the number of iterations is set to 50, the batch size is set to 16, AdamW is selected as the optimizer, and the initial learning rate is set to 10 -4 , the cross entropy loss function and Dice Loss are combined for model training.

[0054]

[0055] Loss = 0.5 × Loss CE +0.5×Loss Dice

[0056] Among them, Loss CE 、Loss Dice and Loss represent the cross entropy loss function, Dice loss, and the loss function used by the model, respectively. i and p iRepresent the true value and the model prediction value respectively, and n represents the total number of samples. After multiple rounds of iterative training, the model finally converged, and the trained model showed excellent performance on the validation set. The model training curve is shown in Figure 6 shown.

[0057] Step 6, test the performance of the deep learning model: select three typical eutrophic shallow lakes as test lakes, and input the feature images of the selected three lakes into the deep learning model trained in step 5. And use three indicators, Accuracy, F1 score and Frequency Weighted Intersection over Union (FWIoU), to quantitatively evaluate the performance of the model. And compare the classification results of the trained deep learning model AAENet (the present invention) with the other four methods in these three lakes. The calculation formulas of the three indicators are:

[0058]

[0059] Among them, acc, F1 and FWIoU represent precision, F1 score and frequency weighted intersection over union respectively; i and recall i Respectively represent the precision and recall of category i; TP i Indicates the number of pixels predicted by the model as category i and the true category is also i; FP i Indicates the number of pixels predicted by the model as category i and whose true category is not i; FN i Indicates the number of pixels that the model predicts are not class i and the true class is i; TN i represents the number of pixels that the model predicts are not class i and the true class is not i; N represents the number of all classes. The other four methods include UNet, Deeplab v3, PSPNet deep learning method and VBI algorithm. The extraction accuracy comparison of these five methods is shown in Table 2 and Figure 7 shown.

[0060] Table 2

[0061]

[0062] The results show that compared with the other four methods, the AAENet deep learning model of the present invention has the best effect in the extraction of water blooms and aquatic plants, the highest accuracy, and complete target edge information. Not only the water blooms and aquatic plants in large lakes (Taihu Lake and Chaohu Lake) are accurately extracted, but even the small and medium-sized lakes (Changdang Lake) are completely extracted. The extraction accuracy of the three test lakes is above 86%. This shows that the AAENet model can not only accurately extract water blooms and aquatic plants in large lakes (Taihu Lake and Chaohu Lake), but also has a good performance for small and medium-sized lakes (Changdang Lake).

[0063] The seventh step is to obtain the spatiotemporal distribution of water blooms and aquatic plants in typical shallow lakes: input the long-term characteristic images of the selected typical eutrophic shallow lakes into the deep learning model trained in the fifth step to obtain the long-term spatial distribution map of water blooms and aquatic plants. Figure 8 The figure shows the five-feature image of the Taihu Lake area on May 27, 2017 and the result of the model output.

[0064] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope required by the present invention.

Claims

1. A method for extracting water blooms and aquatic plants in shallow lakes based on a deep learning model, comprising the following steps: The first step is to obtain the Landsat reflectance image and Sentinel-2 image of the study area within the specified time range. The imaging time difference between the Landsat reflectance image and the corresponding Sentinel-2 image should not exceed one day. The second step is to visually interpret and sample the Landsat reflectivity images with the help of Sentinel-2 high-resolution images to accurately produce training samples and test samples, which correspond to the categories in the visual interpretation results; The third step is to construct characteristic indexes for extracting water blooms and aquatic plants, and input the constructed characteristic indexes and each band of the Landsat reflectance image into the random forest model, obtain the importance of the characteristic indexes and sort the importance, select the top five characteristic indexes, and construct the characteristic image of the Landsat reflectance image based on the five characteristic indexes for the training samples; Step 4: Build a deep learning model for water bloom and aquatic plant extraction, including encoder, pyramid pooling module ASPP and decoder; Step 5: Divide the feature images of the training samples and the corresponding visual interpretation results into training sets and validation sets in proportion, input them into the constructed deep learning model for training, and then perform multiple iterations, and save the training model with the highest accuracy, so as to obtain appropriate training hyperparameters; Step 6: Test and evaluate the trained deep learning model based on the test samples; Step 7. Input the characteristic images of Landsat reflectivity images of the study area at different periods into the trained deep learning model for processing to obtain the spatial distribution maps of water blooms and aquatic plants at different periods.

2. The method for extracting water blooms and aquatic plants from shallow lakes based on a deep learning model according to claim 1, characterized in that: In the first step, the Landsat reflectivity image and the Sentinel-2 image are obtained from the Google EarthEngine cloud platform, the Landsat reflectivity image covers the study lake and has a cloud cover of less than 30%, and the Sentinel-2 image has no cloud coverage.

3. The method for extracting water blooms and aquatic plants from shallow lakes based on a deep learning model according to claim 1, characterized in that: In the second step, the training samples and test samples are derived from several shallow lakes with severe eutrophication.

4. The method for extracting water blooms and aquatic plants from shallow lakes based on a deep learning model according to claim 1, characterized in that: In the third step, the top five characteristic indices are the normalized difference water index MNDWI, green band G, shortwave infrared band SWIR, floating algae index FAI and enhanced vegetation index EVI, among which, In the formula, NIR is, R is the red light band, NIR is the near infrared band, B is the blue light band, λ NIR is the wavelength of the near-infrared band, λ R is the wavelength of the red light band.

5. The method for extracting water blooms and aquatic plants from shallow lakes based on a deep learning model according to claim 4, characterized in that: In the deep learning model, the encoder includes a convolution layer and a four-layer ConvNeXt network. The feature image input to the encoder is convolved and normalized to obtain a feature map and sent to the ConvNeXt network. Each layer of the ConvNeXt network performs convolution and pooling operations on the feature map in turn, and inputs the feature image to the ASPP module for processing. The ASPP module performs hole convolution with different expansion rates on the input feature map through a multi-scale strategy to obtain a multi-scale feature map, and then fuses the multi-scale feature map to obtain a fused feature map and sends it to the decoder; the Scse attention module is added to the decoder, and the decoder performs three iterations on the fused feature map, each iteration includes bilinear interpolation upsampling and fusion with the feature map of the corresponding layer in the encoder, and uses the Scse attention module for enhancement processing, and then performs convolution operation again. After the iteration is completed, bilinear interpolation upsampling is performed again to the size of the original input image, and convolution processing is performed to generate a single-band output image.

6. The method for extracting water blooms and aquatic plants from shallow lakes based on a deep learning model according to claim 5, characterized in that: The dilation rate of the atrous convolution of the ASPP module is adjusted to 1, 2, 3 and 6.

7. The method for extracting water blooms and aquatic plants from shallow lakes based on a deep learning model according to claim 1, characterized in that: In the fifth step, the number of iterations is 50, the batch size is 16, AdamW is used as the optimizer, and the initial learning rate is 10 -4 , the optimal hyperparameters combining the cross entropy loss function and Dice Loss.

8. The method for extracting water blooms and aquatic plants from shallow lakes based on a deep learning model according to claim 1, characterized in that: In the sixth step, several typical eutrophic shallow lakes were selected as tests, and the feature images of the selected lakes were input into the deep learning model trained in the fifth step. The performance of the model was quantitatively evaluated by using three indicators: Accuracy, F1 score, and frequency-weighted intersection over union (FWIoU), compared with the recognition results of existing water bloom and aquatic plant extraction methods.

9. The method for extracting water blooms and aquatic plants from shallow lakes based on a deep learning model according to claim 8, characterized in that: Existing water bloom and aquatic plant extraction methods include: UNet algorithm, Deeplab v3 algorithm, PSPNet algorithm, and VBI algorithm.

10. The method for extracting water blooms and aquatic plants from shallow lakes based on a deep learning model according to claim 1, characterized in that: The categories of visual interpretation results include: water bodies, water blooms, submerged plants, and floating-leaf emergent plants.

Images