Inland water body extraction method fusing hollow spatial pyramid pooling

Abstract

The application provides an inland water body extraction method fusing a cavity space pyramid pooling. The method comprises the following steps: constructing a water body extraction network; the water body extraction network adopts a SegFormer network as a framework, and a new decoder is obtained by improving a decoder in the SegFormer network; the new decoder comprises a first 1x1 convolution layer, an ASPP module, a first upsampling module and a semantic fusion module which are sequentially connected from top to bottom; after a remote sensing image is subjected to feature extraction by an encoder, the remote sensing image is subjected to processing by the new decoder; wherein the semantic fusion module takes the feature map extracted by the bottommost layer of the encoder and the feature map output by the first upsampling module as inputs, and takes the output after fusion as a water body extraction result; the water body extraction network is trained to obtain a water body extraction network model; and the trained water body extraction network model is used to extract water bodies from input remote sensing images. The method has good effects on water body edge integrity detection, reduction of small water bodies, branch leakage detection and false detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of water body extraction, and particularly relates to an inland water body extraction method fusing a hollow spatial pyramid pooling. BACKGROUND

[0002] Water body information extraction and distribution research is of great significance for water resource investigation, ecological protection and disaster monitoring. Traditional water body extraction methods are mainly based on the spectral characteristics of water bodies and use prior information to distinguish water bodies from non-water bodies. In a large number of complex background scenes, especially in images with dense distribution of vegetation and artificial buildings and with occlusion, the spatial context information is seriously lost in the traditional method, and the water body extraction result is relatively fragmented. Moreover, the traditional method is strongly dependent on the scene and has low automation, which is difficult to meet the requirements of real-time updating and large-scale scene application.

[0003] In recent years, classification methods based on convolutional neural networks have also been studied in water color remote sensing. Related research summarizes that deep learning methods use information fusion technology and various networks to construct water body extraction models. The FCN network proposes a feature extraction module to realize multi-scale segmentation of water body features through consecutive up-sampling and maximum pooling layers. The UNet network uses a skip connection to combine the high-level semantic information of the decoder and the detailed low-level features of the encoder. The Hrnet network effectively reduces the loss of feature information by cascading multi-scale resolution features. The PSPNet network couples different regional semantic features through a pyramid pooling module and a pyramid scene analysis module, and the water body edge segmentation accuracy is improved. The Deeplab series network adopts an asymmetric structure to aggregate shallow detail information and deep semantic information. The SegFormer network increases the receptive field by pruning position encoding and using an MLP (Multilayer Perceptron) to extract features, thereby realizing the expression of high-order water body semantics. Although the above water body extraction networks have high detection accuracy, the information loss problem has not been effectively improved, and in practical applications, the branch water body is seriously lost and the water body edge is relatively rough, and the water body extraction effect in complex environments needs to be further improved. SUMMARY

[0004] In view of the problem that the detection result output by the existing water body extraction network seriously lacks water body detail feature information, the present application provides an inland water body extraction method fusing a hollow spatial pyramid pooling.

[0005] The present application provides an inland water body extraction method fusing a hollow spatial pyramid pooling, comprising:

[0006] Step 1: constructing a water body extraction network; the water body extraction network adopts a SegFormer network as a framework, and a new decoder is obtained by improving the decoder in the SegFormer network; the new decoder comprises, from top to bottom, a first 1x1 convolutional layer, an ASPP module, a first upsampling module and a semantic fusion module connected in sequence; after the input remote sensing image is subjected to feature extraction by an encoder, the remote sensing image is processed in the new decoder; wherein the semantic fusion module outputs, as a water body extraction result, the feature map fused after the feature map extracted by the bottommost layer of the encoder and the feature map output by the first upsampling module are fused;

[0007] Step 2: training the water body extraction network to obtain a water body extraction network model;

[0008] Step 3: using the trained water body extraction network model to perform water body extraction on the input remote sensing image.

[0009] Further, the data processing process of the ASPP module is as follows:

[0010] The input feature F is augmented into i groups, denoted as ASPP i , i = 1, 2, 3, 4; the grouped features ASPP i are respectively processed by a dilated convolution layer with a dilated rate of 1, 3, 11 and 17 to obtain the processed grouped features ASPP i .

[0011] The input feature F is subjected to pooling and then parallel operation of upsampling and splicing fusion with all the grouped features ASPP i , and the feature after splicing fusion is the final output of the ASPP module.

[0012] Further, the semantic fusion module comprises a second 1x1 convolutional layer, a second upsampling module, a network concatenation layer and a third 1x1 convolutional layer.

[0013] The feature map extracted by the bottommost layer of the encoder is taken as the input of the second 1x1 convolutional layer.

[0014] The output of the second 1x1 convolutional layer and the output of the first upsampling module are processed by the network concatenation layer, and then are sequentially input to the second upsampling module and the third 1x1 convolutional layer, and the output of the third 1x1 convolutional layer is the detection result.

[0015] Further, in step 2, a transfer learning strategy is adopted, and the water body extraction network is trained using the pre-training weights of the ImageNet-1K dataset.

[0016] The present application has the following advantages:

[0017] In view of the problems of missing semantic and detail feature information, and missing or misextracting non-water body information in water body extraction of remote sensing images, the application improves the SegFormer detection network, adopts an ASPP module, applies multi-sampling rate atrous convolution and multi-receptive field convolution on the input feature map to explore multi-scale context information, captures rich context information through pooling operation at different resolutions, gradually recovers spatial information to capture clear target boundaries, overcomes the problems of missing water body feature information and lack of correlation of long-distance information, and realizes the response of semantic features while covering more geometric information.

[0018] Furthermore, a semantic fusion module is used in the output part to aggregate shallow rough information and deep fine information, realizes the interaction of feature information of different layers through Shuffle, effectively alleviates the missing of feature information caused by the resolution difference of images due to the redundancy of data sources, solves the problems of missing small water bodies, branch loss and inaccurate edges in water body extraction, and suppresses the influence of complex interference information on the extraction result. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A flowchart of the inland water body extraction method provided by the embodiment of the application is shown in the figure.

[0020] Figure 2 A structure diagram of the water body extraction network provided by the embodiment of the application is shown in the figure.

[0021] Figure 3 A reasonable dilatation rate atrous convolution diagram provided by the embodiment of the application is shown in the figure.

[0022] Figure 4 A diagram of the "grid effect" of the atrous convolution provided by the embodiment of the application is shown in the figure.

[0023] Figure 5 A data processing process diagram of the ASPP module provided by the embodiment of the application is shown in the figure.

[0024] Figure 6 A Shuffle mixing diagram provided by the embodiment of the application is shown in the figure.

[0025] Figure 7 A structure diagram of the semantic fusion module provided by the embodiment of the application is shown in the figure.

[0026] Figure 8 A data set example provided by the embodiment of the application is shown in the figure.

[0027] Figure 9The experimental prediction results of the existing network and the method of the present application for the embodiment of the present application are as follows: (a) is the original image, (h) is the data set label, (b)-(f) are the detection results of the existing network, which are Hrnet, PSPNet, UNet, Deeplabv3+, and original SegFormer respectively, and (g) is the detection result of the improved network of the present application. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0029] Embodiment 1

[0030] As shown in the figure, the inland water body extraction method fusing the hollow space pyramid pooling is provided, which comprises the following steps: Figure 1

[0031] S101: Constructing a water body extraction network;

[0032] Specifically, as shown in the figure, the water body extraction network adopts a SegFormer network as a framework, comprising an encoder and a decoder. The encoder comprises four feature extraction stages. In this embodiment, the decoder in the SegFormer network is improved to obtain a new decoder; the new decoder comprises a first 1x1 convolutional layer, an ASPP module, a first upsampling module and a semantic fusion module (SFM) connected in sequence from top to bottom; after the input remote sensing image is subjected to feature extraction by the encoder, it is processed by the new decoder; wherein the semantic fusion module fuses the feature map extracted by the bottommost layer of the encoder and the feature map output by the first upsampling module, and the output after fusion is taken as the water body extraction result; ASPP refers to hollow space pyramid pooling. Figure 2

[0033] S102: Training the water body extraction network to obtain a water body extraction network model;

[0034] S103: Using the trained water body extraction network model to extract water body from the input remote sensing image.

[0035] ​​In view of the problem that the existing network model has poor applicability in water body extraction in remote sensing images, the inland water body extraction method provided in the embodiment of the present application optimizes the SegFormer network structure, fuses the ASPP module, overcomes the problems of small water body information loss and long-distance information lack of correlation, obtains different scale feature information under the premise of not using a pooling layer, captures clear high-resolution water area edge information, and further solves the information loss problem caused by insufficient multi-scale feature acquisition.

[0036] Embodiment 2

[0037] On the basis of the above-mentioned embodiments, the ASPP module is further designed in the embodiment of the present application.

[0038] As Figure 5 shown, the data processing process of the ASPP module in the embodiment is as follows: the input feature F is augmented into i groups, denoted as ASPP i , i = 1, 2, 3, 4; the grouped features ASPP i are respectively processed by the dilated convolution layers with dilated rates of 1, 3, 11 and 17 to obtain the processed grouped features ASPP i '; the input feature F is pooled and then up-sampled and operated in parallel, and then spliced and fused with all the grouped features ASPP i '; the spliced and fused feature is the final output of the ASPP module.

[0039] Specifically, the deep network features are expanded according to different dilated rates, the ASPP module is used to capture multi-scale feature context information, and the global average pooling module is introduced to supplement and improve the context information.

[0040] Compared with the traditional convolution layer and the pooling layer, the advantages of the dilated convolution are as follows: ① after replacing the traditional convolution layer with the dilated convolution, the increase of the calculation parameters is avoided as much as possible, and the reasonably dilated new parameters can be trained, and the performance can be continuously optimized in the training process. ② the feature maps under different resolutions can be obtained by setting different dilated rates to change the size of the receptive field, and the loss of position information in the down-sampling process is reduced.

[0041] In the embodiment, the conventional convolution is replaced with the dilated convolution, and the receptive field in the pooling process is changed by changing the dilated rate of the dilated convolution. It is found in experiments that different dilated rates also have certain influence on the performance of the model. Through research, it is found that: ① using the dilated convolution with the same dilated rate will cause the convolution kernel to be discontinuous, resulting in a "grid effect", as shown in Figure 4 . ② the dilated rate combination cannot contain a common divisor greater than 1, and the value of the dilated rate R iFormula (1) must be satisfied as follows. ③ Assume that the dilation rates corresponding to N dilated convolutions with kernel size k×k are [r1,...,r...]. i ,...r n If the condition is met, then R² ≤ k, where r i R represents the dilation rate of the i-th dilated convolution. i Rn represents the maximum dilation rate of the i-th dilated convolution layer, with the default value being r. n Based on the above conclusions, this embodiment designs dilated convolution combinations with dilation rates of 1, 3, 11, and 17.

[0042] R i =max[R i+1 -2r i ,R i+1 -2(R i+1 -r i ),r i (1)

[0043] In this embodiment, the ASPP module consists of a series of dilated convolutions with different dilation rates and spatial pyramid pooling structures. It extracts multi-scale information from the image by paralleling multiple dilated convolutions with different dilation rates. The ASPP module overcomes the shortcomings of using a single dilated convolution, such as the loss of local information and the lack of correlation of long-distance information due to the grid effect. It can obtain feature information at different scales and obtain clear water body edge information without using pooling layers.

[0044] Example 3

[0045] Based on the above embodiments, in order to achieve effective fusion of shallow and deep features, this embodiment provides a structure for a semantic fusion module.

[0046] like Figure 7 As shown, the semantic fusion module includes a second 1×1 convolutional layer, a second upsampling module, a network cascade layer, and a third 1×1 convolutional layer; the feature map extracted from the lowest layer of the encoder is used as the input of the second 1×1 convolutional layer; the output of the second 1×1 convolutional layer and the output of the first upsampling module are processed by the network cascade layer and then input to the second upsampling module and the third 1×1 convolutional layer in sequence, and the output of the third 1×1 convolutional layer is the detection result.

[0047] In this embodiment, the semantic fusion module fuses the detailed information obtained from shallow, coarse feature extraction with the deep, refined semantic information obtained after ASPP multi-scale hollow pyramid pooling network computation, and performs channel shuffling on the fused information (e.g., Figure 6As shown in FIG. 1, the network autonomously learns the "semantic flow" information between different resolution features through the network, corrects the coarse features to fine features with higher resolution, and harmonizes the multi-scale information by mixing different channel information to strengthen the information exchange between different grouping feature channels, so as to effectively transmit the semantic information from the deep layer to the shallow layer, and realize effective fusion between different resolution features.

[0048] Embodiment 4

[0049] In order to verify the effectiveness and generalization performance of the method proposed in the present application, the following experimental data is also provided in this embodiment.

[0050] I. Experimental data set

[0051] In view of the problem that there is currently a lack of data sets for inland water body extraction research, in this study, the Dengzhou region of Henan Province with numerous lakes and rich water systems in the Danjiangkou reservoir area was selected as the study area, and unmanned aerial vehicle remote sensing images were used to construct the inland water body extraction data set. The unmanned aerial vehicle remote sensing images were obtained by UltraCamXp aerial camera, the image aerial photography time was May 2019, the imaging scale (96DPI) was 1:1000, the ground resolution was 0.1 meters, and 1109 images were included. After cutting the single sample image, the size was 200 to 5000 pixels.

[0052] II. Experimental environment and parameter setting

[0053] The experimental operating system was Windows 11, the CPU version was 12th Gen Intel(R) Core(TM) i9-12900H, the GPU was NVIDIA GeForce RTX 3080TI, and the deep learning framework was torch 1.11.0+cu115. The experiment used a transfer learning strategy, and the pre-trained weights on the ImageNet-1K data set were used for training. Through transfer learning, the network model relied on its unsaturated continuous learning ability to improve the accuracy of the detection results. The training configuration information is shown in Table 1:

[0054] Table 1: Data set training configuration

[0055] Class Dataset Input size 200-5000 Train size 512*512 Test size 512*512 Number of iterations 500 Number of single training 8 Optimizer Sgd Learning rate decay Cos Weight decay 0.0005 Learning rate 1.00E-05

[0056] III. Ablation experiment

[0057] (I) Different mechanism ablation experiment

[0058] To explore the contribution of the improved strategy made in this paper to the network model, an ablation experiment was conducted on the Henan Dengzhou inland water dataset using SegFormer as the baseline system, and the experimental parameters and experimental environment were kept consistent. The performance improvement of the model was evaluated by several evaluation indicators such as mIoU (mean Intersection over Union), F1 score, Precision, and Recall. The mIoU is the average percentage of the intersection and union of the predicted water pixel number and the true water pixel; the F1 score takes into account the influence of precision and recall; the precision is the percentage of pixels that the model correctly predicts as water; and the recall is the percentage of true water that is correctly predicted.

[0059] The comparison of evaluation indicators in different mechanism ablation experiments is shown in Table 2.

[0060] Table 2: Precision comparison of ablation experiment

[0061]

[0062]

[0063] From the experimental results in Table 2, when conducting experiments on the Henan Dengzhou inland water dataset, the improved network of the present application has the best improvement in precision indicators: mIoU is improved by 2.12%, F1 is improved by 1.29%, Precision is improved by 0.82%, and Recall is improved by 1.75%. After optimizing the network, the improved network of the present application has stronger feature extraction and feature fusion capabilities, can better recover and focus on different layer semantic information and feature information, and effectively improves the overall segmentation precision of the model.

[0064] (II) Ablation experiment of dilated rate of cavity spatial pyramid pooling decoding module

[0065] This experiment also verifies the influence of different dilated rate combinations on the performance of the ASPP module. From Table 3, we can know that: (1) Do not use the same consecutive convolution: when the combination is 1, 3, 6, 9, the precision is improved by 0.49% compared with the combination 1, 2, 2, 2. (2) Large dilation kernel has better effect: when using 1, 6, 12, 18 dilated rate, the precision is improved by 0.43% compared with using 1, 3, 6, 9 combination. (3) The combination of coprime numbers has better effect: when using 1, 5, 11, 17 dilated rate, the precision is improved by 0.62% compared with using 1, 6, 12, 18 combination. (4) The dilated rate of the second convolution kernel cannot be larger than the maximum size of the convolution kernel: when using 1, 3, 11, 17 dilated rate, the precision is improved by 0.92% compared with using 1, 5, 11, 17 combination.

[0066] Therefore, the combination of 1, 3, 11, and 17 hollow convolutions with discontinuity, a common divisor of 1, and a large expansion rate is selected. In the experiment, the accuracy of this combination is improved by 0.92%-3.74% compared with other combinations.

[0067] Table 3: Influence of different expansion rates on the performance of the ASPP module

[0068] Dilation rate combination mIoU 1，3，3，3 90.78 1，3，6，9 91.17 1，6，12，18 91.6 1，5，11，17 92.22 1，3，11，17 93.14

[0069] Four, comparative experiment

[0070] Based on the water data set of 0.1-meter high-resolution unmanned aerial vehicle remote sensing images, high-precision pixel-level label analysis can be realized, and the range, distribution, and geometric contour of the water body can be more accurately represented. In this experiment, the label information of the original data set is visualized on the image and compared with the experimental results. In order to verify the effectiveness of the improved scheme, several classical and mainstream deep learning semantic segmentation algorithms such as Hrnet, PSPNet, UNet, Deeplabv3+, and the original SegFormer network are selected for comparison. The comparison results are as follows:

[0071] Table 4: Accuracy comparison of different methods

[0072]

[0073] From Table 4, it can be seen that compared with the mainstream methods in recent years and the original SegFormer model, the improved method of the present application is superior to the comparison method in overall experimental accuracy. Among them, the mIoU reaches 93.14%, the F1 score is 96.83%, the Precision is 97.62%, and the recall rate is 96.06%. The mIoU value of the improved method of the present application is 7.81% higher than that of the comparison method. After network improvement, the application of multi-scene images is enhanced, and the performance of complex scenes and shadow shielding is more robust, verifying the effectiveness of the present application scheme.

[0074] Figure 8 Fig. 6 shows the experimental prediction results of Hrnet, PSPNet, UNet, Deeplabv3+, the original SegFormer algorithm, and the improved scheme of the present application. Based on the detection result visualization diagram, the performance of different algorithms is evaluated, wherein column (a) is the original image, column (h) is the data set label, columns (b)-(f) are the detection results of the comparison network, and column (g) is the detection result of the improved network of the present application. From Figure 8 As can be seen from Fig. 6, the method of the present application can more effectively capture the context information of different scales in the image, and the semantic fusion module can effectively fuse the context information and spatial detail information, verifying the high-efficiency semantic segmentation performance of the proposed method.

[0075] The experimental results show that the method can realize sufficient mining of water body element information under multi-element and multi-scene optical images, pixel-level deconstruction of water body information, effective integration of deep water body semantics and shallow surface feature shape, structure, texture, hue and other information, optimization and improvement of the detection network to establish a detection model with higher accuracy and deeper level than the original detection network, and high-precision extraction of inland water bodies. The method can be applied to inland water body remote sensing extraction and water resource distribution detection, and provides strong support for water body observation and water resource mapping production.

[0076] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for inland water body extraction fusing empty spatial pyramid pooling, characterized in that, The application relates to a water body extraction method, which comprises the following steps: Step 1: constructing a water body extraction network; the water body extraction network adopts a SegFormer network as a framework, and a new decoder is obtained by improving a decoder in the SegFormer network; the new decoder comprises a first 1x1 convolution layer, an ASPP module, a first upsampling module and a semantic fusion module which are sequentially connected from top to bottom; after a remote sensing image is subjected to feature extraction by an encoder, the remote sensing image is subjected to processing by the new decoder; wherein the semantic fusion module takes the feature map extracted by the bottommost layer of the encoder and the feature map output by the first upsampling module as inputs, and takes the output after fusion as a water body extraction result; The data processing process of the ASPP module is as follows: The input feature F is augmented into i groups, denoted as ASPP i , i = 1, 2, 3, 4; the grouped feature ASPP i is respectively processed by a dilated convolution layer with an expansion rate of 1, 3, 11, and 17 to obtain the processed grouped feature ASPP i '; The input feature F is pooled and resampled in parallel operation and combined with all the grouped features ASPP i The spliced and fused features are the final output of the ASPP module. The semantic fusion module comprises a second 1x1 convolution layer, a second upsampling module, a network concatenation layer and a third 1x1 convolution layer; The feature map extracted by the bottommost layer of the encoder is taken as the input of the second 1x1 convolution layer; The output of the second 1x1 convolution layer and the output of the first upsampling module are subjected to processing by the network concatenation layer, and then are subjected to channel shuffling, and then are sequentially input into the second upsampling module and the third 1x1 convolution layer; the output of the third 1x1 convolution layer is a detection result; Step 2: training the water body extraction network to obtain a water body extraction network model; Step 3: using the trained water body extraction network model to perform water body extraction on an input remote sensing image.

2. The method of claim 1, wherein the method further comprises: In step 2, a transfer learning strategy is adopted, and the water body extraction network is trained by using pre-training weights of an ImageNet-1K data set.

Images