SAR (Synthetic Aperture Radar) image water body segmentation system and method combined with optical characteristics

Abstract

The invention relates to the technical field of image processing, in particular to an SAR (Synthetic Aperture Radar) image water body segmentation system and method combined with optical features, and the method comprises the steps: respectively carrying out the feature extraction of an input optical water body image and an SAR water body image through an optical coding branch and an SAR coding branch; the bimodal fusion module is used for fusing the shallow-layer features and the deep-layer features of the two extracted features; the multi-scale fusion module performs multi-scale fusion on the fused shallow features and deep features; and the decoding branch is used for decoding the fusion features to obtain a water body segmentation map. According to the invention, by designing the shallow guidance module, the cross-modal attention module, the multi-scale context fusion module and the like, high-precision segmentation of the water body region in the remote sensing scene is realized, and the recognition precision and the boundary retention capability of the water body category are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a SAR image water body segmentation system and method combined with optical features. BACKGROUND

[0002] Remote sensing image semantic segmentation is a key technology in geographic information extraction, resource investigation and environmental monitoring. In recent years, with the enrichment of high-resolution optical and radar remote sensing data, segmentation methods based on deep learning have become a research hotspot. Among them, synthetic aperture radar (SAR) images have all-weather and all-day imaging capabilities, and are an important data source for water body identification; while optical images have higher expression in texture details and spectral differences. Optical-SAR joint semantic segmentation has significant value in practical applications due to its complementarity. Current mainstream remote sensing image segmentation methods such as DeepLabV3+, U-Net, etc. have achieved good performance on optical images, but when migrated to SAR images, the precision is significantly reduced due to the low contrast, speckle noise and other characteristics of SAR. To solve this problem, existing research attempts to use optical images as auxiliary modalities to guide feature extraction of SAR.

[0003] The existing SAR image segmentation network mainly has the following shortcomings: (1) Insufficient fusion between multi-modal images: early splicing or late result fusion fails to fully utilize the complementary guiding ability of optical images for SAR; (2) Poor boundary recognition ability: in complex water body contour or narrowband structure scenes, it is easy to appear broken and over-smoothed; (3) Poor large-scale water body feature extraction ability: existing networks are difficult to model large-scale lakes and narrow water systems at the same time; (4) Weak feature expression ability: using fixed backbone structure (such as ResNet), without designing shallow enhancement mechanism for water body texture features. SUMMARY

[0004] Therefore, the present application creates a SAR image water body segmentation system and method combined with optical features, which realizes high-precision segmentation of water body regions in remote sensing scenes by designing shallow guidance, dual-modal fusion module and multi-scale fusion module.

[0005] To achieve the above purpose, the technical solution of the present application is as follows: The application discloses a SAR image water body segmentation system combined with optical features, and belongs to the technical field of SAR image water body segmentation.

[0006] Further, in the optical encoding branch or the SAR encoding branch: the input water body image is subjected to continuous encoding operation for not less than 4 times, the shallow feature obtained through the second encoding operation is input into the shallow dual-modal fusion module, and the deep feature obtained through the last encoding operation is input into the deep dual-modal fusion module.

[0007] Further, in each encoding operation, the input water body image or feature is subjected to convolution operation, the feature obtained through the convolution is subjected to ReLU activation, and then is subjected to maximum pooling operation.

[0008] Further, in the shallow dual-modal fusion module or the deep dual-modal fusion module: the two input features are subjected to convolution operation respectively to obtain respective value features; the two input features are subjected to channel splicing, and then the spliced feature is subjected to convolution operation to realize feature compression; the compressed feature is subjected to global average pooling and global maximum pooling respectively, then the corresponding elements of the two pooled features are multiplied, the multiplied feature is subjected to softmax operation to obtain an attention score; the attention score and the two value features are subjected to Hadamard product operation respectively, then the two features obtained through the Hadamard product operation are subjected to Hadamard product operation to obtain corresponding fusion features.

[0009] Further, in the multi-scale fusion module: the deep fusion feature is subjected to hollow convolution operation with different expansion rates, and the deep fusion feature is subjected to global maximum pooling; the features obtained through the hollow convolution operation with different expansion rates are subjected to channel splicing and convolution operation, then the feature obtained through the channel splicing and convolution operation and the shallow fusion feature are subjected to channel splicing and convolution operation to obtain the fusion feature.

[0010] Further, after the deep fusion feature is subjected to the pooling of the hollow spatial convolution pyramid, the deep fusion feature enters the multi-scale fusion module.

[0011] Further, in the decoding branch: after the fused features are decoded for no less than 2 times successively, the obtained features are bilinearly interpolated and upsampled to obtain a water body segmentation map consistent with the resolution of the optical water body image.

[0012] Further, in each decoding operation, a ReLU activation operation is performed after the convolution operation on the features.

[0013] A SAR image water body segmentation method combining optical features, comprising: S1: obtaining an optical-SAR dataset and pre-processing the optical-SAR dataset to obtain a training set; S2: constructing a SAR image water body segmentation system combining optical features as provided by the present application, training the water body segmentation system using the training set obtained in step S1 to obtain a water body segmentation model; S3: adjusting the hyperparameters of the training, and repeating step S2 until an optimal water body segmentation model is obtained; S4: inputting an optical water body image to be segmented and its corresponding SAR water body image into the optimal water body segmentation model obtained in step S3 to obtain a predicted water body segmentation map.

[0014] Further, in step S2, a cross-entropy loss function is used for training.

[0015] Compared with the prior art, the present application can achieve the following beneficial effects: The SAR image water body segmentation system and method combining optical features provided by the present application combine the features of optical images and SAR images, the former provides texture details, and the latter has all-weather penetration capability, thereby improving the detection effect on water bodies in different scenes. In addition, the dual-modal fusion module adaptively fuses optical and SAR features after the shallow and deep stages of the encoding branch, solves the problem of misalignment of modal feature scales, and retains the advantages of different modalities. The multi-scale fusion module combines with the skip connection features, captures water body features of different scales through multi-scale convolution, and enhances the perception ability of water bodies. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and their Figure 1 The overall block diagram of the SAR image water body segmentation system combining optical features described in the embodiments of the present application; Figure 2 The schematic diagram of the SAR image water body segmentation system combining optical features described in the embodiments of the present application; Figure 3 a schematic diagram of a shallow dual-modal fusion module or a deep dual-modal fusion module according to an embodiment of the present invention; Figure 4 a schematic diagram of a multi-scale fusion module according to an embodiment of the present invention; Figure 5 a flowchart of a method for water body segmentation of a SAR image combined with optical features according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and do not constitute a limitation on the present invention.

[0018] It should be noted that the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0019] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present invention and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second" and the like are only used for description purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited by "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0020] In the description of the present invention, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present invention can be understood through specific circumstances.

[0021] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0022] As Figures 1 to 2As shown, the SAR image water body segmentation system combining optical features described in the embodiments of the present application includes an optical encoding branch, a SAR encoding branch and a decoding branch. The optical encoding branch extracts features from the input optical water body image to obtain shallow optical features and deep optical features; the SAR encoding branch extracts features from the input SAR water body image to obtain shallow SAR features and deep SAR features. A shallow dual-modal fusion module and a deep dual-modal fusion module are provided between the optical encoding branch and the SAR encoding branch, which respectively fuse the shallow features and the deep features of the two kinds of features. Specifically, the shallow dual-modal fusion module exchanges and fuses the shallow optical features and the shallow SAR features to obtain shallow fusion features; the deep dual-modal fusion module exchanges and fuses the deep optical features and the deep SAR features to obtain deep fusion features. The obtained shallow fusion features and deep fusion features are input into a multi-scale fusion module for feature fusion. Specifically, the multi-scale fusion module extracts multi-scale features from the deep fusion features, fuses the extracted multi-scale features, and obtains the fusion features by fusing the obtained features with the shallow fusion features. The decoding branch decodes the fusion features to obtain a water body segmentation map.

[0023] In the embodiments of the present application, the resolution of the optical water body image and the SAR water body image is 256x256, the channel number of the optical water body image is 3, and the channel number of the SAR water body image is 1.

[0024] In some embodiments, the optical encoding branch and the SAR encoding branch have consistent structures, and a pseudo-twin structure water body segmentation system is obtained. Specifically, in the optical encoding branch or the SAR encoding branch, the input water body image is subjected to continuous encoding operation not less than 4 times, the shallow features obtained by the second encoding operation are input into the shallow dual-modal fusion module, and the deep features obtained by the last encoding operation are input into the deep dual-modal fusion module. In each encoding operation, the input water body image or features are subjected to convolution operation, and the features obtained by convolution are subjected to ReLU activation and then maximum pooling operation.

[0025] In the optical coding branch or the SAR coding branch provided in the embodiments of the present application, the input optical water body image or SAR water body image is subjected to five continuous coding operations, the first two layers are low-level features, and the last three layers are high-level semantics, the shallow features obtained through the second coding operation are input into the shallow dual-modal fusion module, and the deep features obtained through the last coding operation are input into the deep dual-modal fusion module. In each coding operation, the input water body image or feature is subjected to convolution operation, and after the ReLU activation of the features obtained through the convolution, maximum pooling operation is performed. The first coding operation includes convolution operation, batch normalization operation, ReLU activation operation and maximum pooling operation performed in sequence, to obtain features with a size of 128x128 and a channel number of 64; the second coding operation includes convolution operation, batch normalization operation, ReLU activation operation and maximum pooling operation performed in sequence, to obtain shallow features with a size of 64x64 and a channel number of 128; the third coding operation includes convolution operation, batch normalization operation, ReLU activation operation and maximum pooling operation performed in sequence, to obtain features with a size of 32x32 and a channel number of 256; the fourth coding operation includes convolution operation, batch normalization operation and ReLU activation operation performed in sequence, to obtain features with a size of 32x32 and a channel number of 512; and the fifth coding operation includes convolution operation, batch normalization operation and ReLU activation operation performed in sequence, to obtain deep features with a size of 32x32 and a channel number of 1024.

[0026] In some embodiments, the structures of the shallow dual-modal fusion module and the deep dual-modal fusion module are consistent, as shown in FIG. 2. Figure 3 As shown in FIG. 2, in the shallow dual-modal fusion module or the deep dual-modal fusion module: the two kinds of input features are subjected to convolution operation respectively to obtain respective value features; the two kinds of input features are subjected to channel splicing, and the spliced features are subjected to convolution operation to realize feature compression; the compressed features are subjected to global average pooling and global maximum pooling respectively, then the corresponding elements of the two kinds of pooled features are multiplied, and after the softmax operation on the multiplied features, attention scores are obtained. The attention scores and the two kinds of value features are subjected to Hadamard product operation respectively, and the two features obtained through the Hadamard product operation are subjected to Hadamard product operation to obtain corresponding fusion features. The present application realizes the interaction of dual-modal information through cross-modal cooperation, and enhances the performance of subsequent tasks. The present application adopts Hadamard product mainly because it can efficiently and accurately weight the features point by point, and realize the selection of important information between different modalities. In the present application, the attention weight and the feature value are multiplied point by point, the feature value at the position with high attention weight is amplified, and the feature value at the position with low weight is suppressed. Unlike matrix multiplication, Hadamard product directly operates on the original feature space, and the locality and spatial relationship of the features are preserved.

[0027] In the embodiment of the present application, the shallow optical features and the shallow SAR features with the size of 64x64 and the channel number of 128 are input into the shallow dual-modal fusion module, the obtained shallow fusion features are subjected to convolution operation, and the shallow fusion features with the size of 64x64 and the channel number of 48 are obtained; the deep optical features and the deep SAR features with the size of 32x32 and the channel number of 1024 are input into the deep dual-modal fusion module, the obtained deep fusion features are subjected to convolution operation, and the deep fusion features with the size of 64x64 and the channel number of 256 are obtained.

[0028] In some embodiments, the multi-scale fusion module is as shown in FIG. 6, wherein the deep fusion features are subjected to the dilated convolution operation with different expansion rates, and the global maximum pooling operation is performed on the deep fusion features. Figure 4 The obtained multi-scale features are subjected to channel splicing and convolution operation, the obtained features are subjected to channel splicing and convolution operation with the shallow fusion features, and the enhanced fusion features of the low and high layers of the context are obtained.

[0029] In the multi-scale fusion module provided in the embodiment of the present application, the dilated convolution operation with the expansion rates of 1, 2, 4 and 6 is performed on the deep fusion features, and the global maximum pooling operation is performed on the deep fusion features, the obtained five features with different scales are subjected to channel splicing and convolution operation, and the features with the size of 64x64 and the channel number of 256 are obtained. The obtained features are subjected to channel splicing and convolution operation with the shallow fusion features, and the fusion features with the size of 64x64 and the channel number of 304 are obtained.

[0030] In some embodiments, after the deep fusion features are pooled by a Atrous Spatial Pyramid Pooling (ASPP), the pooled features enter a multi-scale fusion module. The ASPP is composed of a 1x1 convolution, a pooling pyramid, and an ASPP Pooling in parallel. The output features are obtained by channel splicing and convolution compression of the output features of the 1x1 convolution, the pooling pyramid, and the ASPP Pooling. The pooling pyramid includes multiple parallel convolutional layers with the same kernel size and different dilation factors. The dilation factors of the convolutional layers can be self-defined, so as to realize free multi-scale feature extraction. In the embodiments of the present application, the pooling pyramid includes three parallel convolutional layers, the kernel size of the three convolutional layers is 3x3, and the dilation factors are 1, 2, 4, and 6 respectively. The ASPP Pooling is a AdaptiveAvgPool2d layer, which is an adaptive average pooling. The adaptive average pooling does not need to specify the pooling size and step, but only needs to specify the final output size, which is 1x1 in the embodiments of the present application. By compressing the feature maps of each channel to 1x1, the features of each channel are extracted, and the global features are obtained. Then, a 1x1 convolution layer is used to further extract the features obtained in the previous step and reduce the dimension.

[0031] In some embodiments, the decoding branch performs continuous decoding operations on the fusion features for no less than 2 times, and then performs a bilinear interpolation upsampling operation on the obtained features to obtain a water body segmentation map consistent with the resolution of the optical water body image. In each decoding operation, a ReLU activation operation is performed after the convolution operation on the features.

[0032] In the embodiments of the present application, the decoding branch performs continuous decoding operations on the fusion features for 3 times, and then performs a bilinear interpolation upsampling operation on the obtained features to obtain a water body segmentation map with a resolution of 256x256. In each decoding operation, a ReLU activation operation is performed after the convolution operation, batch normalization, and ReLU activation operation on the features.

[0033] The present application also provides a SAR image water body segmentation method combined with optical features, as shown in Figure 5 , comprising: S1: obtaining an optical-SAR dataset and preprocessing the optical-SAR dataset to obtain a training set.

[0034] In the embodiment of the present application, the optical-SAR dataset in step S1 is from the public WHU-OPT-SAR dataset, which contains 100 optical images with a resolution of 5556x3704 and SAR images of the same area, covering an area of about 50000km 2 in Hubei Province (30°N-33°N, 108°E-117°E) in China. The preprocessing process in the embodiment of the present application includes random horizontal flipping, random vertical flipping, random 90-degree rotation and size adjustment of the data.

[0035] S2: Construct a SAR image water body segmentation system combined with optical features as provided in the present application, train the water body segmentation system using the training set obtained in step S1 to obtain a water body segmentation model. In step S2, the cross-entropy loss function is used for training.

[0036] S3: Adjust the hyperparameters of the training, and repeat step S2 until the optimal water body segmentation model is obtained.

[0037] In the embodiment of the present application, the training hyperparameters include: the training batch size train-batch-size is 8, the validation batch size train-batch-size is 8, the total training rounds total-epochs is 40, the optimizer used is Adam, the weight decay is 1e-4, the initial learning rate is 1e-3, the learning rate decay step is 20, and the learning rate decay factor is 0.1, i.e. the learning rate decays once every 20 times of training, and the learning rate decays by 0.1 each time.

[0038] S4: Input the optical water body image to be segmented and its corresponding SAR water body image into the optimal water body segmentation model obtained in step S3 to obtain a predicted water body segmentation map.

[0039] In the embodiment of the present application, the overall classification accuracy (OA), the average intersection over union (mIoU) and the kappa coefficient are used to evaluate the segmentation effect The OA index is used to measure the consistency of the segmentation result and the true label, and is the ratio of the number of correctly classified pixels (or samples) to the total number of pixels (or total samples). The average intersection over union is calculated for each class in the multi-class image segmentation task, and then the average of these IoUs is calculated. The mIoU takes into account the average performance of all class segmentation, and can more comprehensively evaluate the performance of the multi-class image segmentation model. The kappa coefficient takes a value between -1 and 1, 0.00 to 0.20 indicates very low consistency, and 1.00 indicates almost complete consistency.

[0040] The method provided by the present application can be accelerated by a GPU, supports batch optical and SAR image input, has good scalability, and is suitable for various complex remote sensing scenes such as urban water bodies, river channels, lakes and the like. To verify the superiority of the method provided by the present application, the present application further provides experimental results of Deeplabv3, U-Net and Segnet for water body segmentation, and compares the method provided by the present application, and the comparison results are shown in Table 1. Table 1: Comparison of results of different water body segmentation methods

[0041] As can be seen from the experimental results in Table 1, compared with several common neural networks, the accuracy and kappa coefficient of the method provided by the present application are also higher than those of other methods, which shows that the SAR water body segmentation of the method provided by the present application combined with optical images can obtain more meaningful water body information by designing different attention modules and fusion modules, and has significant advantages in actual application.

[0042] It should be understood that the various forms of the flow shown above can be used to reorder, add or delete steps. For example, each step described in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which is not limited herein.

[0043] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement and improvement within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A SAR image water segmentation system combining optical features, characterized by: include: The optical encoding branch extracts features from the input optical water image to obtain shallow optical features and deep optical features; SAR encoding branch, which extracts features from the input SAR water body image to obtain shallow SAR features and deep SAR features; A shallow dual-modal fusion module performs feature exchange and feature fusion on the shallow optical features and the shallow SAR features to obtain shallow fusion features; A deep dual-modal fusion module performs feature exchange and feature fusion on the deep optical features and the deep SAR features to obtain deep fusion features; A multi-scale fusion module extracts multi-scale features from the deep fusion features, fuses the extracted multi-scale features, and combines the obtained features with the shallow fusion features to obtain fusion features; The decoding branch performs a decoding operation on the fused features to obtain a water body segmentation map.

2. The SAR image water segmentation system combined with optical features according to claim 1, characterized in that: In the optical encoding branch or the SAR encoding branch: The input water body image is subjected to at least four consecutive encoding operations, the shallow features obtained by the second encoding operation are input into the shallow bimodal fusion module, and the deep features obtained by the last encoding operation are input into the deep bimodal fusion module.

3. The SAR image water segmentation system combined with optical features according to claim 2, characterized in that: In each encoding operation, a convolution operation is performed on the input water body image or feature, and the convolutional features are activated by ReLU, followed by a maximum pooling operation.

4. The SAR image water segmentation system combined with optical features according to claim 1, characterized in that: In the shallow bimodal fusion module or the deep bimodal fusion module: Perform convolution operations on the two input features to obtain their respective value features; The two input features are channel-spliced, and the spliced ​​features are then convolved to achieve feature compression; the compressed features are subjected to global average pooling and global maximum pooling respectively, and then the corresponding elements of the two pooled features are multiplied, and the multiplied features are subjected to softmax operation to obtain the attention score; the attention score is subjected to Hadamard product operation with the two value features respectively, and the two features are subjected to Hadamard product operation again to obtain the corresponding fusion feature.

5. The SAR image water segmentation system combined with optical features according to claim 1, characterized in that: In the multi-scale fusion module: The deep fusion features are subjected to multiple dilation rate dilation convolution operations, and global maximum pooling is performed on the deep fusion features. After channel splicing and convolution operations are performed on the obtained features of multiple scales, the obtained features are subjected to channel splicing and convolution operations with the shallow fusion features to obtain the fusion features.

6. The SAR image water segmentation system combined with optical features according to claim 1, characterized in that: The deep fusion features are pooled by the dilated spatial convolution pooling pyramid and then enter the multi-scale fusion module.

7. The SAR image water segmentation system combined with optical features according to claim 1, characterized in that: In the decoding branch: after performing a continuous decoding operation on the fusion feature for at least 2 times, a bilinear interpolation upsampling operation is performed on the obtained feature to obtain a water body segmentation map with a resolution consistent with the optical water body image.

8. The SAR image water segmentation system combined with optical features according to claim 7, characterized in that: In each decoding operation, a ReLU activation operation is performed after the convolution operation on the features.

9. A SAR image water segmentation method combining optical features, characterized in that: include: S1: Acquire an optical-SAR dataset and preprocess the optical-SAR dataset to obtain a training set; S2: Constructing the SAR image water segmentation system combined with optical features according to any one of claims 1 to 8, and training the water segmentation system using the training set obtained in step S1 to obtain a water segmentation model; S3: Adjust the training hyperparameters and repeat step S2 until the optimal water body segmentation model is obtained; S4: Input the optical water body image to be segmented and its corresponding SAR water body image into the optimal water body segmentation model obtained in step S3 to obtain a predicted water body segmentation map.

10. The SAR image water body segmentation method combined with optical features according to claim 9, characterized in that: In step S2, the cross entropy loss function is used for training.

Images