SAR image water body segmentation system and method incorporating optical features

Abstract

The present application relates to the technical field of image processing, and more particularly to a SAR image water body segmentation system and method combined with optical features, wherein an optical encoding branch and a SAR encoding branch respectively perform feature extraction on an input optical water body image and a SAR water body image; a dual-mode fusion module fuses the shallow features and deep features of the two extracted features; a multi-scale fusion module performs multi-scale fusion on the fused shallow features and deep features; a decoding branch performs decoding operation on the fused features to obtain a water body segmentation map. Through the design of shallow guiding, cross-modal attention and multi-scale context fusion modules, the present application realizes high-precision segmentation of water body regions in a remote sensing scene and improves the recognition accuracy and boundary preservation ability of water body categories.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a SAR image water body segmentation system and method combined with optical features. BACKGROUND

[0002] Remote sensing image semantic segmentation is a key technology in geographic information extraction, resource investigation and environmental monitoring. In recent years, with the enrichment of high-resolution optical and radar remote sensing data, segmentation methods based on deep learning have become a research hotspot. Among them, synthetic aperture radar (SAR) images have all-weather and all-day imaging capabilities, and are an important data source for water body identification; while optical images have higher expression in texture details and spectral differences. Optical-SAR joint semantic segmentation has significant value in practical applications due to its complementarity. The current mainstream remote sensing image segmentation methods such as DeepLabV3+, U-Net, etc. have achieved good performance on optical images, but when migrated to SAR images, the accuracy is significantly reduced due to the low contrast, speckle noise and other characteristics of SAR. To solve this problem, existing research attempts to use optical images as auxiliary modalities to guide feature extraction of SAR.

[0003] The existing SAR image segmentation network mainly has the following shortcomings:

[0004] (1) Insufficient fusion between multi-modal images: early splicing or late result fusion fails to fully utilize the complementary guiding ability of optical images for SAR;

[0005] (2) Poor boundary recognition ability: in complex water body contour or narrowband structure scenes, it is easy to appear problems such as cracking and excessive smoothing;

[0006] (3) Poor large-scale water body feature extraction ability: existing networks are difficult to model large-scale lakes and narrow water systems at the same time;

[0007] (4) Weak feature expression ability: using fixed backbone structure (such as ResNet), without designing shallow enhancement mechanism for water body texture features. SUMMARY

[0008] Therefore, the present application aims to provide a SAR image water body segmentation system and method combined with optical features, which realizes high-precision segmentation of water body regions in remote sensing scenes by designing shallow guiding, dual-modal fusion module and multi-scale fusion module.

[0009] To achieve the above purpose, the technical scheme of the present application is as follows:

[0010] The application discloses a SAR image water body segmentation system combined with optical features, and belongs to the technical field of SAR image water body segmentation.

[0011] Further, in the optical encoding branch or the SAR encoding branch: the input water body image is subjected to continuous encoding operation for not less than 4 times, the shallow feature obtained through the second encoding operation is input into the shallow dual-modal fusion module, and the deep feature obtained through the last encoding operation is input into the deep dual-modal fusion module.

[0012] Further, in each encoding operation, the input water body image or feature is subjected to convolution operation, the feature obtained through the convolution is subjected to ReLU activation, and then is subjected to maximum pooling operation.

[0013] Further, in the shallow dual-modal fusion module or the deep dual-modal fusion module: the two input features are subjected to convolution operation respectively to obtain respective value features; the two input features are subjected to channel splicing, and then the spliced feature is subjected to convolution operation to realize feature compression; the compressed feature is subjected to global average pooling and global maximum pooling respectively, then the corresponding elements of the two pooled features are multiplied, the multiplied feature is subjected to softmax operation to obtain an attention score; the attention score and the two value features are subjected to Hadamard product operation respectively, then the two features obtained through the Hadamard product operation are subjected to Hadamard product operation to obtain corresponding fusion features.

[0014] Further, in the multi-scale fusion module: the deep fusion feature is subjected to hollow convolution operation with different expansion rates, and the deep fusion feature is subjected to global maximum pooling; the obtained features with different scales are subjected to channel splicing and convolution operation, then the obtained feature is subjected to channel splicing and convolution operation with the shallow fusion feature to obtain the fusion feature.

[0015] Further, after the deep fusion feature is subjected to the pooling of the hollow spatial convolution pyramid, the deep fusion feature enters the multi-scale fusion module.

[0016] Further, in the decoding branch: after the fused features are decoded for not less than 2 times successively, the obtained features are bilinearly interpolated and upsampled to obtain a water body segmentation map consistent with the resolution of the optical water body image.

[0017] Further, in each decoding operation, a ReLU activation operation is performed after the convolution operation on the features.

[0018] A SAR image water body segmentation method combined with optical features comprises:

[0019] S1: An optical-SAR dataset is obtained, and the optical-SAR dataset is preprocessed to obtain a training set;

[0020] S2: A SAR image water body segmentation system combined with optical features is constructed, the training set obtained in step S1 is used to train the water body segmentation system, and a water body segmentation model is obtained;

[0021] S3: The hyperparameters of the training are adjusted, and step S2 is repeated until an optimal water body segmentation model is obtained;

[0022] S4: An optical water body image to be segmented and its corresponding SAR water body image are input into the optimal water body segmentation model obtained in step S3 to obtain a predicted water body segmentation map.

[0023] Further, in step S2, a cross-entropy loss function is used for training.

[0024] Compared with the prior art, the present application can achieve the following beneficial effects:

[0025] The SAR image water body segmentation system and method combined with optical features provided by the present application combine the features of optical images and SAR images, the former provides texture details, and the latter has all-weather penetration capability, thereby improving the detection effect on water bodies in different scenes. In addition, the dual-modal fusion module adaptively fuses optical and SAR features after the shallow and deep stages of the encoding branch, solves the problem of misalignment of modal feature scales, and retains the advantages of different modalities. The multi-scale fusion module combines with the skip connection features, captures water body features of different scales through multi-scale convolution, and enhances the perception ability of water bodies. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and their

[0027] Figure 1 The overall block diagram of the SAR image water body segmentation system combined with optical features described in the embodiments of the present application;

[0028] Figure 2 a schematic diagram of a SAR image water body segmentation system with optical features according to an embodiment of the present invention;

[0029] Figure 3 a schematic diagram of a shallow dual-modal fusion module or a deep dual-modal fusion module according to an embodiment of the present invention;

[0030] Figure 4 a schematic diagram of a multi-scale fusion module according to an embodiment of the present invention;

[0031] Figure 5 a flowchart of a SAR image water body segmentation method with optical features according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation on the present invention.

[0033] It should be noted that the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0034] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present invention and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second" and the like are only used for description purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited by "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0035] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0036] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0037] As Figures 1 to 2 shown, the SAR image water body segmentation system combining optical features according to the embodiments of the present application includes an optical encoding branch, a SAR encoding branch and a decoding branch. The optical encoding branch extracts features from the input optical water body image to obtain shallow optical features and deep optical features; the SAR encoding branch extracts features from the input SAR water body image to obtain shallow SAR features and deep SAR features. A shallow dual-modal fusion module and a deep dual-modal fusion module are provided between the optical encoding branch and the SAR encoding branch to fuse the shallow features and the deep features of the two kinds of features, specifically, the shallow dual-modal fusion module exchanges and fuses the shallow optical features and the shallow SAR features to obtain shallow fusion features; the deep dual-modal fusion module exchanges and fuses the deep optical features and the deep SAR features to obtain deep fusion features. The obtained shallow fusion features and deep fusion features are input into a multi-scale fusion module for feature fusion, specifically, the multi-scale fusion module extracts multi-scale features from the deep fusion features, fuses the extracted multi-scale features, and obtains the fusion features by combining the obtained features with the shallow fusion features. The decoding branch decodes the fusion features to obtain a water body segmentation map.

[0038] In the embodiments of the present application, the resolution of the optical water body image and the SAR water body image is 256x256, the channel number of the optical water body image is 3, and the channel number of the SAR water body image is 1.

[0039] In some embodiments, the structures of the optical encoding branch and the SAR encoding branch are consistent, obtaining a water body segmentation system with pseudo-twin structure. Specifically, in the optical encoding branch or the SAR encoding branch, the input water body image is subjected to continuous encoding operation not less than 4 times, the shallow features obtained by the second encoding operation are input into the shallow dual-modal fusion module, and the deep features obtained by the last encoding operation are input into the deep dual-modal fusion module. In each encoding operation, the input water body image or features are subjected to convolution operation, and after ReLU activation of the features obtained by convolution, maximum pooling operation is performed.

[0040] In the optical coding branch or the SAR coding branch provided by the embodiment of the application, the input optical water body image or SAR water body image is subjected to five continuous coding operations, the first two layers are low-level features, and the last three layers are high-level semantics, the shallow features obtained through the second coding operation are input into the shallow dual-modal fusion module, and the deep features obtained through the last coding operation are input into the deep dual-modal fusion module. In each coding operation, the input water body image or feature is subjected to a convolution operation, and after the ReLU activation of the feature obtained through the convolution, a maximum pooling operation is performed. The first coding operation includes convolution operation, batch normalization operation, ReLU activation operation and maximum pooling operation performed in sequence, and a feature with a size of 128x128 and a channel number of 64 is obtained; the second coding operation includes convolution operation, batch normalization operation, ReLU activation operation and maximum pooling operation performed in sequence, and a shallow feature with a size of 64x64 and a channel number of 128 is obtained; the third coding operation includes convolution operation, batch normalization operation, ReLU activation operation and maximum pooling operation performed in sequence, and a feature with a size of 32x32 and a channel number of 256 is obtained; the fourth coding operation includes convolution operation, batch normalization operation and ReLU activation operation performed in sequence, and a feature with a size of 32x32 and a channel number of 512 is obtained; and the fifth coding operation includes convolution operation, batch normalization operation and ReLU activation operation performed in sequence, and a deep feature with a size of 32x32 and a channel number of 1024 is obtained.

[0041] In some embodiments, the structures of the shallow dual-modal fusion module and the deep dual-modal fusion module are consistent, as shown in Figure 3 In the shallow dual-modal fusion module or the deep dual-modal fusion module, the two kinds of input features are respectively subjected to convolution operations to obtain respective value features; the two kinds of input features are subjected to channel splicing, and the spliced features are subjected to convolution operation to realize feature compression; the compressed features are respectively subjected to global average pooling and global maximum pooling, then the corresponding elements of the two kinds of pooled features are multiplied, and after the softmax operation on the multiplied features, attention scores are obtained. The attention scores and the two kinds of value features are respectively subjected to Hadamard product operation, and the two features obtained through the Hadamard product operation are subjected to Hadamard product operation to obtain corresponding fusion features. The application realizes the interaction of dual-modal information through cross-modal cooperation, and enhances the performance of subsequent tasks. The application adopts Hadamard product mainly because it can efficiently and accurately weight the features point by point, and realize the selection of important information between different modalities. In the application, the attention weight and the feature value are multiplied point by point, the feature value at the position with high attention weight is amplified, and the feature value at the position with low weight is suppressed. Unlike matrix multiplication, Hadamard product directly operates on the original feature space, and the locality and spatial relationship of the features are preserved.

[0042] In the embodiment of the present application, the shallow optical features and the shallow SAR features with the size of 64x64 and the channel number of 128 are input into the shallow dual-modal fusion module, the obtained shallow fusion features are subjected to convolution operation, and the shallow fusion features with the size of 64x64 and the channel number of 48 are obtained; the deep optical features and the deep SAR features with the size of 32x32 and the channel number of 1024 are input into the deep dual-modal fusion module, the obtained deep fusion features are subjected to convolution operation, and the deep fusion features with the size of 64x64 and the channel number of 256 are obtained.

[0043] In some embodiments, the multi-scale fusion module is as shown in FIG. 5. Figure 4 As shown in FIG. 5, the deep fusion features are subjected to the dilated convolution operation with multiple different expansion rates, and the global maximum pooling operation is performed on the deep fusion features, the obtained features with multiple scales are subjected to the channel splicing and the convolution operation, the obtained features are subjected to the channel splicing and the convolution operation with the shallow fusion features, and the enhanced fusion features of the context low-high layer are obtained.

[0044] In the multi-scale fusion module provided in the embodiment of the present application, the deep fusion features are subjected to the dilated convolution operation with the expansion rates of 1, 2, 4 and 6, and the global maximum pooling operation is performed on the deep fusion features, the obtained five features with different scales are subjected to the channel splicing and the convolution operation, and the features with the size of 64x64 and the channel number of 256 are obtained. The obtained features are subjected to the channel splicing and the convolution operation with the shallow fusion features, and the fusion features with the size of 64x64 and the channel number of 304 are obtained.

[0045] In some embodiments, after the deep fusion features are pooled by a Atrous Spatial Pyramid Pooling (ASPP), the pooled features enter a multi-scale fusion module. The ASPP is composed of a 1x1 convolution, a pooling pyramid, and an ASPP Pooling in parallel. The output features are obtained by channel splicing and convolution compression of the output features of the 1x1 convolution, the pooling pyramid, and the ASPP Pooling. The pooling pyramid includes multiple parallel convolutional layers with the same kernel size and different dilation factors. The dilation factors of the convolutional layers can be self-defined, so as to realize free multi-scale feature extraction. In the embodiments of the present application, the pooling pyramid includes three parallel convolutional layers, the kernel size of the three convolutional layers is 3x3, and the dilation factors are 1, 2, 4, and 6 respectively. The ASPP Pooling is a AdaptiveAvgPool2d layer, which is an adaptive average pooling. The adaptive average pooling does not need to specify the pooling size and step, but only needs to specify the final output size, which is 1x1 in the embodiments of the present application. By compressing the feature maps of each channel to 1x1, the features of each channel are extracted, and the global features are obtained. Then, a 1x1 convolution layer is used to further extract the features obtained in the previous step and reduce the dimension.

[0046] In some embodiments, the decoding branch performs continuous decoding operation on the fusion features for not less than 2 times, and then performs bilinear interpolation upsampling operation on the obtained features to obtain a water body segmentation map with the same resolution as the optical water body image. In each decoding operation, ReLU activation operation is performed after convolution operation on the features.

[0047] In the embodiments of the present application, the decoding branch performs continuous decoding operation on the fusion features for 3 times, and then performs bilinear interpolation upsampling operation on the obtained features to obtain a water body segmentation map with a resolution of 256x256. In each decoding operation, convolution operation, batch normalization, and ReLU activation operation are performed on the features in sequence.

[0048] The present application also provides a SAR image water body segmentation method combined with optical features, as shown in Figure 5 , comprising:

[0049] S1: Obtain an optical-SAR dataset, and pre-process the optical-SAR dataset to obtain a training set.

[0050] In the embodiment of the present application, the optical-SAR data set in step S1 is from a public WHU-OPT-SAR data set, which contains 100 optical images with a resolution of 5556x3704 and SAR images of the same area. The preprocessing process in the embodiment of the present application includes random horizontal flipping, random vertical flipping, random 90-degree rotation and size adjustment of the data.

[0051] S2: Construct a water body segmentation system combined with optical features as provided by the present application, train the water body segmentation system using the training set obtained in step S1 to obtain a water body segmentation model. In step S2, the cross-entropy loss function is used for training.

[0052] S3: Adjust the hyperparameters of the training, and repeat step S2 until the optimal water body segmentation model is obtained.

[0053] In the embodiment of the present application, the training hyperparameters include: the training batch size train-batch-size is 8, the verification batch size train-batch-size is 8, the total training round total-epochs is 40, the adopted optimizer is Adam, the weight decay is 1e-4, the initial learning rate is 1e-3, the learning rate decay step is 20, and the learning rate decay factor is 0.1, that is, the learning rate decays once every 20 times of training, and the learning rate decays by 0.1 each time.

[0054] S4: Input the optical water body image to be segmented and its corresponding SAR water body image into the optimal water body segmentation model obtained in step S3 to obtain a predicted water body segmentation map.

[0055] In the embodiment of the present application, the overall classification accuracy (OA), the average intersection over union (mIoU) and the kappa coefficient are used to evaluate the segmentation effect

[0056] The OA index is used to measure the consistency of the segmentation result and the true label, and is the ratio of the number of correctly classified pixels (or samples) to the total number of pixels (or total samples). The average intersection over union is calculated for each class in the multi-class image segmentation task, and then the average of these IoUs is obtained. The mIoU comprehensively considers the average performance of all class segmentation, and can more comprehensively evaluate the performance of the multi-class image segmentation model. The kappa coefficient takes a value between -1 and 1, 0.00 to 0.20 represents very low consistency, and 1.00 represents almost complete consistency.

[0057] The method provided by the present application can be accelerated by a GPU, supports batch optical and SAR image input, has good scalability, and is suitable for various complex remote sensing scenes such as urban water bodies, river channels, lakes and the like. To verify the superiority of the method provided by the present application, the embodiment of the present application further provides experimental results of Deeplabv3, U-Net and Segnet for water body segmentation, and compares the method provided by the present application, and the comparison results are shown in Table 1:

[0058] Table 1: Comparison of results of different water body segmentation methods

[0059]

[0060] From the experimental results in Table 1, it can be seen that compared with several common neural networks, the accuracy and kappa coefficient of the method provided by the present application are also higher than those of other methods, which shows that the SAR water body segmentation of the method provided by the present application combined with optical images can obtain more meaningful water body information by designing different attention modules and fusion modules, and has significant advantages in actual application.

[0061] It should be understood that the various forms of the flow shown above can be used to reorder, add or delete steps. For example, each step described in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which is not limited herein.

[0062] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A SAR image water segmentation system incorporating optical features, characterized in that, include: The optical coding branch extracts features from the input optical water image to obtain shallow optical features and deep optical features; The SAR coding branch extracts features from the input SAR water image to obtain shallow SAR features and deep SAR features; The shallow dual-modal fusion module performs feature exchange and feature fusion on the shallow optical features and the shallow SAR features to obtain shallow fused features. The deep dual-modal fusion module performs feature exchange and feature fusion on the deep optical features and the deep SAR features to obtain deep fused features; The multi-scale fusion module extracts multi-scale features from the deep fusion features, fuses the extracted multi-scale features, and then combines the resulting features with the shallow fusion features to obtain the fused features. The decoding branch performs a decoding operation on the fused features to obtain a water body segmentation map; In the optical coding branch or the SAR coding branch: the input water image is subjected to no less than 4 consecutive coding operations, the shallow features obtained from the second coding operation are input into the shallow dual-modal fusion module, and the deep features obtained from the last coding operation are input into the deep dual-modal fusion module. In the multi-scale fusion module: the deep fusion features are subjected to dilated convolution operations with different dilation rates, and global max pooling is performed on the deep fusion features. The features obtained at multiple scales are then subjected to channel concatenation and convolution operations. The resulting features are then subjected to channel concatenation and convolution operations with the shallow fusion features to obtain the fusion features. In the shallow bimodal fusion module or the deep bimodal fusion module: convolution operations are performed on the two input features respectively to obtain their respective value features; the two input features are concatenated by channels, and then convolution operations are performed on the concatenated features to achieve feature compression; the compressed features are subjected to global average pooling and global max pooling respectively, and then the corresponding elements of the two pooled features are multiplied together, and a softmax operation is performed on the multiplied features to obtain the attention score; the attention score is multiplied by the two value features respectively, and the two features are then multiplied by a Hadamard product to obtain the corresponding fused features.

2. The SAR image water segmentation system combining optical features according to claim 1, characterized in that, In each encoding operation, a convolution operation is performed on the input water body image or features. After ReLU activation, max pooling is performed on the convolutional features.

3. The SAR image water segmentation system combining optical features according to claim 1, characterized in that, The deep fusion features are pooled by the hollow spatial convolutional pooling pyramid and then enter the multi-scale fusion module.

4. The SAR image water segmentation system combining optical features according to claim 1, characterized in that, In the decoding branch: after performing at least two consecutive decoding operations on the fused features, the obtained features are then subjected to bilinear interpolation upsampling to obtain a water body segmentation map with the same resolution as the optical water body image.

5. The SAR image water segmentation system combining optical features according to claim 4, characterized in that, In each decoding operation, a ReLU activation operation is performed after the convolution operation on the features.

6. A method for water body segmentation in SAR images combining optical features, characterized in that, include: S1: Obtain the optical-SAR dataset and preprocess the optical-SAR dataset to obtain the training set; S2: Construct a SAR image water segmentation system combining optical features as described in any one of claims 1 to 5, and train the water segmentation system using the training set obtained in step S1 to obtain a water segmentation model; S3: Adjust the hyperparameters of the training and repeat step S2 until the optimal water segmentation model is obtained; S4: Input the optical water body image to be segmented and its corresponding SAR water body image into the optimal water body segmentation model obtained in step S3 to obtain the predicted water body segmentation map.

7. The SAR image water segmentation method combining optical features according to claim 6, characterized in that, In step S2, the cross-entropy loss function is used for training.

Images