A refined method for extracting surface water boundaries

Abstract

The present invention discloses a method for fine-grained extraction of surface water boundaries. The method for fine-grained extraction of surface water boundaries includes: constructing a surface water image data set; constructing a convolutional neural network, the convolutional neural network including a shallow feature extraction module, an SE attention module and a feature fusion module; the shallow feature extraction module is used to perform SA module processing on the features processed by the downsampled convolution block in the convolution block processing channel; the SE attention module is used to perform SE module processing on the input image of the convolution block processing channel; the feature fusion module is used to perform feature fusion; the convolutional neural network is trained using the surface water image data set to obtain a trained surface water boundary extraction model; the trained model is used to automatically extract the surface water boundary on the surface image to be processed. The method for fine-grained extraction of surface water boundaries of the present invention realizes the annotation of narrow rivers and small water bodies in the surface image, and realizes the fine extraction of surface water boundaries.

Description

Technical Field

[0001] The present invention relates to a surface image processing technology, and in particular to a surface water boundary refinement extraction method. Background Art

[0002] Surface water is one of the most important components of the ecosystem and is also very important for human production and life. Therefore, monitoring and understanding the situation of surface water is very important for effectively protecting the earth's ecological environment.

[0003] In recent years, with the development of deep learning technology, especially convolutional neural networks, it has been widely used in image segmentation, classification and target detection tasks. In 2017, Isikdogan first proposed a new CNN architecture called DeepWaterMap. This model uses a fully convolutional network structure to combine spectral, spatial, texture and shape features to distinguish between water bodies and disturbance features such as snow, ice and terrain shadows. However, when designing the convolutional neural network, this method aims to focus the network on the foreground (water body) and ultimately achieve an overall good effect. However, for narrower rivers and small water bodies, it is currently difficult to extract the water body type, which often results in missed detection or false detection. Summary of the Invention

[0004] The purpose of the present invention is to provide a surface water boundary refinement extraction method, which can mark relatively narrow rivers and small water bodies in surface images, thereby realizing the refinement extraction of surface water boundaries.

[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0006] A surface water boundary refinement extraction method, the surface water boundary refinement extraction method comprising:

[0007] Construct a surface water image dataset, which includes a training set and a test set;

[0008] Construct a convolutional neural network, which includes a convolutional block processing channel, a shallow feature extraction module, an SE attention module, and a feature fusion module;

[0009] The convolution block processing channel is used to achieve feature learning of the input image;

[0010] The shallow feature extraction module is used to perform SA module processing on the features processed by the previous downsampling convolution block in the convolution block processing channel, and then add the processed features to each upsampling convolution block in the convolution block processing channel;

[0011] The SE attention module is used to perform SE module processing on the input image of the convolution block processing channel and transmit the processed image to the output of the convolution block processing channel;

[0012] The feature fusion module is used to perform feature fusion on the features processed by the downsampling convolution block in the convolution block processing channel, and then add the fused features to each upsampling convolution block in the convolution block processing channel;

[0013] The convolutional neural network is trained using a surface water image dataset. The Sobel edge loss function is used to calculate the loss during training to obtain a trained surface water boundary extraction model.

[0014] The trained surface water boundary extraction model is used to automatically extract the surface water boundary on the surface image to be processed to obtain a surface water map.

[0015] Furthermore, seven convolution blocks are provided in the convolution block processing channel, of which four convolution blocks are downsampling convolution blocks and the other three convolution blocks are upsampling convolution blocks, and an ASPP processing module is provided between the four downsampling convolution blocks and the three upsampling convolution blocks.

[0016] Furthermore, the number of shallow feature extraction modules set in the convolutional neural network is 2, and the 2 shallow feature extraction modules are used to perform SA module processing on the features processed by the first and second downsampling convolution blocks in the convolution block processing channel, respectively.

[0017] Furthermore, the feature fusion module is used to perform feature fusion on the features processed by the first, second and fourth downsampling convolution blocks in the convolution block processing channel.

[0018] Furthermore, the surface water boundary refinement extraction method also includes: setting a sigmoid activation function for the output of the convolutional neural network, and the sigmoid activation function is used to perform sigmoid activation function processing on each pixel in the convolutional neural network output image to obtain a classification result for each pixel in the image.

[0019] Furthermore, when training the convolutional neural network, the convolutional neural network is iterated 200 times to complete the training.

[0020] Furthermore, an adaptive gradient optimizer and a momentum algorithm are used when training the convolutional neural network.

[0021] Furthermore, the image data in the surface water image dataset are selected from the public dataset ESWKB.

[0022] In the surface water boundary refinement extraction method of the present invention, a shallow feature extraction module, an SE attention module and a feature fusion module are set in the convolutional neural network, the shallow feature extraction module is used to perform SA module processing on the features processed by the down-sampled convolution block in the convolution block processing channel, and then add the processed features to the up-sampled convolution block in the convolution block processing channel, the SE attention module is used to perform SE module processing on the input image of the convolution block processing channel, and the feature fusion module is used to perform feature fusion on the features processed by the down-sampled convolution block in the convolution block processing channel, and then add the fused features to the up-sampled convolution block in the convolution block processing channel. In this way, the SE attention module and the feature fusion module can explore boundary semantic information, and the feature fusion module can combine features extracted from different depths, and use the Sobel edge loss function to enhance the network's attention to boundary information. The surface water boundary extraction model obtained after training can mark relatively narrow rivers and small water bodies in surface images, thereby realizing refined extraction of surface water boundaries.

[0023] Compared with the existing technology, the surface water boundary refinement extraction method of the present invention has the following beneficial effects: the surface water boundary refinement extraction method can mark relatively narrow rivers and small water bodies in the surface image, thereby realizing the refinement extraction of surface water boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of a convolutional neural network in a method for extracting refined surface water boundaries according to one embodiment of the present invention;

[0025] Figure 2 The following are partial magnified images of the braided river area and the densely distributed river area under different models;

[0026] Figure 3 The figure shows the comparison of the effects of six test images using different methods. DETAILED DESCRIPTION

[0027] The present invention will be further described below with specific embodiments:

[0028] This embodiment provides a method for extracting surface water boundaries in a refined manner, which can mark relatively narrow rivers and small water bodies in surface images.

[0029] The surface water boundary refinement extraction method of this embodiment includes the following steps S1 to S4.

[0030] Specifically,

[0031] S1, pre-construct a surface water image dataset, where the image data is selected from the public dataset ESWKB (Earth surface water knowledge base), which is built from atmospherically corrected Sentinel-2 multispectral instrument (MSI) images.

[0032] It should be noted that the ESWKB mentioned earlier is a public dataset that contains Sentinel-2 images from all over the world and throughout the year. The entire ESWKB dataset consists of 95 scenes captured from 31 images.

[0033] In this implementation, six scenes with different water body types are selected from ESWKB as the test set, and the remaining 89 scenes are used as the training set.

[0034] In other implementations, the number of selected scenes may be determined as needed.

[0035] S2, build a convolutional neural network.

[0036] See also Figure 1 ,The constructed convolutional neural network includes: a convolution block processing channel, an SE attention module (SE Attention), two shallow feature extraction modules (SA), and a feature fusion module (MFF).

[0037] The convolution block processing channel is used to realize the feature learning function of the input image.

[0038] Specifically,

[0039] The convolution block processing channel is provided with a plurality of convolution blocks (Conv Block) connected in series. In this embodiment, the number of convolution blocks provided in the convolution block processing channel is 7, wherein the 4 convolution blocks at the most upstream end of the convolution block processing channel are used to perform downsampling operations, and are referred to as downsampling convolution blocks, and the 3 convolution blocks at the most downstream end of the convolution block processing channel are used to perform upsampling operations, and are referred to as upsampling convolution blocks.

[0040] In addition, an ASPP processing module is also provided in the convolution block processing channel. The ASPP processing module is arranged between the 4th convolution block and the 5th convolution block, that is, between the downsampling convolution block and the upsampling convolution block. The ASPP processing module is a multi-scale feature extraction module in the prior art. Its role in the convolution block processing channel is to capture contextual information at different scales to improve the network's receptive field and semantic understanding ability.

[0041] The convolution block set in the convolution block processing channel has the following main operation steps: a convolution operation with a convolution kernel size of 3*3, a BN normalization operation, a Relu activation function operation and a jump connection operation.

[0042] It should be noted that, in this embodiment, the convolution block at the most upstream side in the convolution block processing channel (the convolution block at the input) is called the "first convolution block in the convolution block processing channel", and the subsequent convolution blocks are respectively called the second, third, fourth, ... ... convolution blocks, and the convolution block at the most downstream side in the convolution block processing channel (the convolution block at the output) is called the "last convolution block in the convolution block processing channel".

[0043] The SE attention module is used to perform SE module processing on the input image at the input of the convolution block processing channel, and then transmit the processed image to the output of the convolution block processing channel to assist in the generation of the final result image.

[0044] The two shallow feature extraction modules are respectively set at the first and second convolution blocks in the convolution block processing channel. The two shallow feature extraction modules are respectively used to perform SA module processing on the features processed by the first and second convolution blocks (downsampling convolution blocks) in the convolution block processing channel, and input the processed features into the ASPP processing module. Then, the ASPP processing module adds the features processed by the SA module to each upsampling convolution block to assist in image restoration.

[0045] The feature fusion module is used to perform feature fusion on the features processed by the first convolution block, the second convolution block and the fourth convolution block (3 downsampling convolution blocks) in the convolution block processing channel, and then input the fused features into the ASPP processing module. The ASPP processing module adds the features fused by the feature fusion module to each upsampling convolution block. In this way, the combination of the three scale features is conducive to helping the network learn better and assisting in better image restoration.

[0046] In addition, a sigmoid activation function is set for the output of the constructed convolutional neural network. The sigmoid activation function is used to perform sigmoid activation function processing on each pixel in the output image of the convolutional neural network, thereby obtaining the classification result of each pixel in the image.

[0047] The SE attention module mentioned earlier has the following basic structure: First, the convolutional layer: The input passes through a series of convolutional layers to extract features. Second, the squeeze operation compresses each channel of the input feature map into a scalar through global pooling, capturing the global statistical information of the feature map. This helps identify key regions in the input and preserve object boundary information. Next, the excitation operation takes the output of the squeeze operation and passes it through a fully connected network to produce a set of weight vectors. These vectors are used to weight the feature maps of each channel, thereby amplifying the response of key regions. This further strengthens the model's attention to object boundary information, making it easier to distinguish different objects. Finally, the rescaling layer: Finally, a value between 0 and 1 is multiplied back to the original feature map to weight its importance and enhance the influence of key information. After the SE attention module processes the input 6-channel image, it is introduced into the final upsampling layer of the network, helping the network to obtain more abstract information at the bottleneck layer. The SE attention module consists of four main steps: first, a feature map is generated from the given feature map, then a vector is obtained through global average pooling. At the same time, a global compressed feature quantity is also obtained for each feature channel. After that, it assigns a new weight to each feature channel through two fully connected layers. Finally, the new weight is multiplied with the initial feature map to obtain the desired feature map.

[0048] The feature fusion module mentioned earlier. This module improves the network's ability to model spatial and channel information by integrating spatial attention and channel attention. First, the input is processed by 3x3 convolution, and the feature maps are input into the spatial attention module and the channel attention module respectively. The channel attention module performs global maximum pooling and global average pooling on the input features to generate two feature maps, which are then processed by a multi-layer perceptron and added together. The resulting features are then passed through a sigmoid function to obtain a weight matrix, which is multiplied with the input feature map to enhance the most relevant channels. The spatial attention module performs global maximum pooling and global average pooling on the input features to generate a feature map, which is then processed through 7x7 convolution and s-type activation connection to obtain a weight matrix. The weight matrix is ​​then multiplied with the input feature map to enhance the most relevant spatial positions. Finally, the feature maps enhanced by channel attention and spatial attention are added together.

[0049] The feature fusion module mentioned earlier. This feature fusion module is designed to combine the advantages of shallow features and deep features. When the features are in the early layers of the encoder, they contain rich object boundary details, despite having only undergone a small amount of convolution. In contrast, the features of deeper encoder layers are more abstract at the semantic level and highlight the location of the target. By fusing shallow features and deep features, we hope to combine their advantages and obtain an initial saliency feature map with rough boundaries and precise locations. This map can then be introduced into each upsampling layer to help the network recover boundary details during decoding.

[0050] The Sobel edge loss function mentioned earlier uses the Sobel operator to calculate the horizontal and vertical image gradients. This function can effectively capture boundary information, especially for small water bodies, because water boundaries are usually continuous and their gradient directions should be consistent in space. Compared with the binary cross entropy loss, the Sobel edge loss function can reduce missegmentation by enforcing the consistency of gradient directions.

[0051] Subsequent experiments were conducted using 89 scenes as a training set, and the trained models were evaluated on six test sets. Various ablation experiments were conducted to analyze the contributions of the various components of EB-Net. Since the network backbone of this method is ResNet, three variants of the EB-Net architecture were compared with ResNet to verify the effectiveness of each module proposed in this model.

[0052] Table 1 and Figure 2 Indicates the evaluation metrics and prediction results obtained from the module configuration of each model, where Model No. 1: ResNet+BCE, Model No. 2: ResNet+SA+BCE, Model No. 3: ResNet+SA+MFF+SEA+BCE, Model No. 4: ResNet+SA+MFF+SEA+MSE, Model No. 5: ResNet+SA+MFF+SEA+Focal, Model No. 6: ResNet+SA+MFF+SE+Sobel (method of this embodiment). In Table 1, SA, MFF, and SEA represent the shallow attention module, multi-feature fusion module, and squeeze-excitation attention module, respectively. BCE represents the binary cross entropy loss function, MSE represents the mean square error loss function, focal represents the focal loss function, and Sobel represents the Sobel edge loss function. The evaluation value is the average result of 6 test images.

[0053] The feature fusion module aims to extract edge semantics from two shallow features. To test the effectiveness of SA, we added the SA module to the ResNet backbone and compared it with an encoder-decoder architecture with a ResNet backbone. As shown in Table 1, precision and recall improved slightly by 0.38% and 0.26%, respectively.

[0054] like Figure 2 As shown in the figure, for braided river areas, Model 1 and Model 2, it can be seen that after adding the SA module, the small branched water bodies previously omitted by ResNet have a certain shape, and the effect is significant. In densely distributed river areas, for Model 1 and Model 2, it can be seen that after adding the SA module, the previously disconnected branched water bodies are now clearly continuous, and the effect is significant.

[0055] Based on the backbone network, MFF is introduced to aggregate three shallow, medium and deep features to generate an initial target map, which is integrated into the decoder to gradually guide the final saliency prediction. Then, the SEA module is applied to the upsampled output of the last layer of the network to make the output channel more directional, which helps to incorporate the key direction into the final saliency prediction result. As shown in Table 1 (Model 2, 3), it can be observed that the recall rate increased by 1.72%, but the precision rate decreased slightly, and the F1 comprehensive index increased by 0.29%. Figure 2 In the figure, for braided rivers and densely distributed river areas, it can be seen that after adding the MFF and SEA models, some discontinuous branching water bodies have improved continuity.

[0056] The Sobel edge loss function uses two filters to continuously push the model, optimize the solution learning, and focus on the error of the segmentation boundary during the network optimization process. As shown in Table 1 (Models 3 and 6), it can be seen that the precision, recall and F1 comprehensive indicators have increased by 1.2%, 0.66% and 0.93% respectively, indicating that the model has been significantly improved. Considering that there are still some commonly used loss functions in the water segmentation task, the Sobel edge loss function is replaced by MSE and focal loss in the network for comparative experiments. Figure 2 As shown in (Model 4, 5), it can be clearly seen that when faced with small water information, MSE and focal loss cannot effectively help the network explore these small features. Therefore, an appropriate loss function can help each module give full play to its role and make target recognition more directional. Figure 2As shown in the various images in Figure 3, the proposed model (Model 6) significantly improves the extraction of small water bodies, further enhancing the relationships between small water bodies and improving previously disconnected tributaries. Furthermore, it can be observed that the model significantly improves the water features that were previously ignored after changing the loss function, indicating that the Sobel edge loss function continuously focuses the model on the target boundary. In summary, the Sobel edge loss function has superior performance in detecting slender water features.

[0057] Table 1. Ablation experiment of the model on the ESWKB dataset

[0058]

[0059] S3, inputting the surface water image data set constructed in step S1 into the convolutional neural network constructed in step S2, and training the convolutional neural network using the surface water image data set, and finally obtaining a trained model, which is called a surface water boundary extraction model.

[0060] It should be noted that when training convolutional neural networks, the Sobel edge loss function is used to calculate the loss.

[0061] It should be noted that when training convolutional neural networks, adaptive gradient optimizer (ADAM) and momentum algorithm can be used to optimize model parameters.

[0062] In addition, a training iteration number N can be predetermined. For the training of the convolutional neural network, the convolutional neural network can be iterated N times to complete the training, thereby obtaining a finally trained surface water boundary extraction model.

[0063] In this embodiment, the number of training iterations N is pre-set to 200, that is, after the convolutional neural network iterates 200 times, the final trained surface water boundary extraction model is obtained.

[0064] After training the surface water boundary extraction model, the surface water boundary extraction model is used to extract water bodies from test image data of size 512*512*6 to evaluate the effectiveness of the surface water boundary extraction model.

[0065] The evaluation method selected four commonly used semantic segmentation metrics: accuracy, precision, recall, and the F1 composite score. Accuracy represents the ratio of all correctly classified pixels to the total number of pixels. Precision represents the ratio of correctly classified water pixels to all pixels classified as water. Recall represents the ratio of detected water pixels to all true water pixels. The F1 composite metric is a comprehensive metric that combines precision and recall and is often used to measure the overall performance of a method. F1 composite metric values ​​close to 1 indicate better method performance, while values ​​close to 0 indicate worse performance.

[0066] The results of different methods are listed in Table 2 and Figure 3 middle. Figure 3 The types of the six images are: [A] lakes with irregular boundaries, [B] braided rivers, [C] linear rivers scattered among ponds, [D] densely distributed rivers, [E] small rivers, and [F] lakes with regular boundaries.

[0067] As can be seen from the tables and graphs, traditional water index-based methods, such as MNDWI, are noisy and easily interfered with by non-water features. The SVM classifier has the highest recall rate, but low precision and a high number of false detections. U-net is sensitive to small water bodies but lacks continuity, which can be seen in the intermittent identification of small tributaries in some areas. ResNet and DeepWaterMapV2 are affected by cloud cover, which increases false positives (non-water pixels) and are insensitive to small water bodies surrounded by land features. The EB-net proposed in this method combines the encoder structure with the deep convolution method and integrates key information of deep and shallow features in the encoding to assist in decoding. The receptive fields of different layers of the convolutional layer are transferred to the network to guide the network to learn the characteristics of small water bodies in a targeted manner, which can achieve the best water extraction effect compared with existing deep learning methods.

[0068] Table 2. Evaluation metrics of different methods. The listed values ​​are the average results of 6 test images.

[0069]

[0070] Table 3 compares the accuracy of different methods across six test images. As can be seen from the figure and table, the proposed EB-Net achieves the best accuracy across all test images, reaching 98.73% for lakes with irregular boundaries, 97.81% for braided rivers, 99.04% for linear rivers interspersed between ponds, 97.96% for densely populated rivers, 97.94% for small rivers, and 99.58% for lakes with regular boundaries. Overall, the accuracy improvement is particularly significant for complex water bodies such as lakes with irregular boundaries, braided rivers, linear rivers interspersed between ponds, densely populated rivers, and small rivers. Compared to the lowest accuracy, the accuracy in these scenarios increases by approximately 0.78%, 4.14%, 1.95%, 6.43%, and 8.10%, respectively. For lakes with regular boundaries, all methods, except the SVM method, achieve satisfactory results, with accuracy exceeding 99%.

[0071] Table 3. Accuracy, precision, recall, and F1 composite index of each test method on the water map.

[0072]

[0073]

[0074] S4. Finally, the trained surface water boundary extraction model is used to automatically extract the surface water boundary on the surface image to be processed, thereby obtaining a surface water map.

[0075] Those skilled in the art will understand that a surface water map is an image in which surface water boundaries are marked based on a surface image.

[0076] In the surface water boundary refinement extraction method of this embodiment, a shallow feature extraction module, an SE attention module, and a feature fusion module are set in the convolutional neural network. The shallow feature extraction module is used to perform SA module processing on the features processed by the downsampled convolution block in the convolution block processing channel, and then add the processed features to the upsampled convolution block in the convolution block processing channel. The SE attention module is used to perform SE module processing on the input image of the convolution block processing channel. The feature fusion module is used to perform feature fusion on the features processed by the downsampled convolution block in the convolution block processing channel, and then add the fused features to the upsampled convolution block in the convolution block processing channel. In this way, the SE attention module and the feature fusion module can explore boundary semantic information, and the feature fusion module can combine features extracted from different depths and use the Sobel edge loss function to enhance the network's attention to boundary information. The surface water boundary extraction model obtained after training can mark relatively narrow rivers and small water bodies in surface images, thereby realizing refined extraction of surface water boundaries.

[0077] The above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for extracting surface water boundaries in a refined manner, characterized by: The surface water boundary refinement extraction method comprises: Construct a surface water image dataset, which includes a training set and a test set; Construct a convolutional neural network, which includes a convolutional block processing channel, a shallow feature extraction module, an SE attention module, and a feature fusion module; The convolution block processing channel is used to achieve feature learning of the input image; The shallow feature extraction module is used to perform SA module processing on the features processed by the previous downsampling convolution block in the convolution block processing channel, and then add the processed features to each upsampling convolution block in the convolution block processing channel; The SE attention module is used to perform SE module processing on the input image of the convolution block processing channel and transmit the processed image to the output of the convolution block processing channel; The feature fusion module is used to perform feature fusion on the features processed by the downsampling convolution block in the convolution block processing channel, and then add the fused features to each upsampling convolution block in the convolution block processing channel; The convolutional neural network is trained using a surface water image dataset. The Sobel edge loss function is used to calculate the loss during training to obtain a trained surface water boundary extraction model. The surface water boundary extraction model that has been trained is used to automatically extract the surface water boundary on the surface image to be processed to obtain a surface water map; The convolution block processing channel is provided with 7 convolution blocks, of which 4 convolution blocks are downsampling convolution blocks and the other 3 convolution blocks are upsampling convolution blocks, and an ASPP processing module is provided between the 4 downsampling convolution blocks and the 3 upsampling convolution blocks; The number of shallow feature extraction modules set in the convolutional neural network is 2, and the two shallow feature extraction modules are used to perform SA module processing on the features processed by the first and second downsampling convolution blocks in the convolution block processing channel respectively; The feature fusion module is used to perform feature fusion on the features processed by the first, second and fourth down-sampling convolution blocks in the convolution block processing channel.

2. The method for extracting surface water boundaries in detail according to claim 1, characterized in that: The surface water boundary refinement extraction method further includes: setting a sigmoid activation function for the output of the convolutional neural network, wherein the sigmoid activation function is used to perform sigmoid activation function processing on each pixel in the output image of the convolutional neural network to obtain a classification result for each pixel in the image.

3. The method for extracting surface water boundaries in detail according to claim 1, wherein: When training the convolutional neural network, the convolutional neural network is iterated 200 times before completing the training.

4. The method for extracting surface water boundaries in detail according to claim 1, wherein: When training convolutional neural networks, an adaptive gradient optimizer and momentum algorithm are used.

5. The method for extracting surface water boundaries in detail according to claim 1, characterized in that: The image data in the surface water image dataset are selected from the public dataset ESWKB.

Images