Remote sensing image water segmentation method, system and equipment based on multi-path fusion of water index and polarization information

Abstract

A method, system, and device for water segmentation in remote sensing images using multi-path fusion of water index and polarization information utilizes two types of multi-source remote sensing data, multispectral images and SAR images, introduces a water index into the original UNet algorithm, designs a multi-path and multi-source data fusion encoder, and utilizes a Gated Channel Transform (GCT) attention mechanism to address the adverse effects of complex land environments in remote sensing images on water segmentation, thereby improving the accuracy of water segmentation with fewer computing resources. The system and device are used to implement water segmentation in remote sensing images using multi-path fusion of water index and polarization information. The present invention has the advantages of low computing cost, short detection time, and high segmentation accuracy, and can be iteratively updated at any time to improve model performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of water segmentation, and in particular relates to a remote sensing image water segmentation method, system and equipment for multi-path fusion of water body index and polarization information. Background Art

[0002] Detecting the extent of surface water is crucial for water resource management and flood monitoring. Remote sensing imagery captured by satellites is often used to analyze water extent. Water segmentation involves segmenting water areas pixel by pixel from remote sensing images to create a water extent map. Data for mapping water extents often comes from optical satellites and synthetic aperture radar (SAR) [Gka B, Svk A, Skaa C. Exploring Sentinel-1 and Sentinel-2 diversity for flood inundation mapping using deep learning [J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 180:163-173.]. Spectral data is the preferred data for mapping water areas in cloud-free conditions, providing multi-band, high-resolution, and low-noise remote sensing imagery. A limitation of spectral imagery is that spectral imagers are passive imaging instruments that require sunlight for light, thus failing to provide satellite imagery around the clock, and their light can be obscured by cloud cover. SAR, based on backscatter polarization imaging, can detect water without cloud interference and capture ground images around the clock. However, polarization information cannot effectively distinguish between water, water-like surfaces, and noise-like spots [Shen X, Wang D, Mao K, et al. Inundation extent mapping by synthetic aperture radar: A review [J]. Remote Sensing, 2019, 11(7): 879]. These shortcomings may limit the application of polarization information in water segmentation. Researchers have found that the combination of water index and polarization information performs better in flood inundation mapping. Therefore, combining water index and polarization information can help improve the effect of automatic water segmentation.

[0003] Manual water segmentation is time-consuming and labor-intensive, and relies heavily on the expertise of the annotators. Once trained, a neural network model eliminates the need for excessive human intervention, making it a viable approach for automated water segmentation in remote sensing images. This paper addresses the task of water segmentation in remote sensing images by studying and improving a U-shaped convolutional network and designing a network model capable of fusing two types of remote sensing data: multispectral imagery and SAR imagery, to enhance water segmentation performance.

[0004] Existing technical solutions include water area segmentation methods based on traditional threshold filtering and water area segmentation methods based on deep learning.

[0005] In early studies, water segmentation methods were limited by remote sensing technology and image quality [Su Longfei, Li Zhenxuan, Gao Fei, et al. A review of water extraction from remote sensing images [J]. Remote Sensing of Land and Resources, 2021, 33(1): 9-11]. In low-resolution remote sensing images, only the different radiation intensities of water bodies in different bands can be used for threshold filtering. However, direct threshold filtering on the original imaging data cannot produce an accurate segmentation map in most cases. For this reason, a water index method combining multi-band data has emerged. This method uses imaging data from multiple bands for combined calculation to achieve the effect of highlighting water bodies and suppressing other land features. After calculating the water index, the new data is filtered using the threshold method to extract and segment water areas. The disadvantage of the threshold method for water segmentation is that the threshold is difficult to determine and cannot adapt to diverse environments. During the threshold method operation, researchers need to set the threshold by themselves for segmentation. Under different scenes and different water indexes, the size of the threshold is not uniform. Remote sensing image water segmentation technology has undergone multiple generations of development. Fully automatic water segmentation based on deep learning technology has gradually replaced traditional manual extraction methods. In order to address the shortcomings of the threshold method, some researchers have begun to explore how to apply deep learning methods to the field of water segmentation. For example: Zhang Jinsong et al. [Zhang Jinsong, Xing Mengdao, Sun Guangcai. A polarization information water segmentation algorithm based on dense depth separation convolution [J]. Journal of Radar, 2019, 8(3):13] proposed a deep separation convolution segmentation network to solve the difficulties in using polarization information to segment water areas. Lv Wentao et al. invented a SAR image water segmentation method based on sinusoidal SPF distribution and level set model, which overcomes the inherent characteristics of SAR images such as multiplicative coherent speckle noise and heterogeneous changes in target intensity, and improves the segmentation performance of SAR images.

[0006] Currently, remote sensing images typically cover a wide range of ground, contain a large amount of ground object information, and have a large background proportion, while the water area is generally smaller. Compared with natural images, remote sensing images have more interference and noise, which affects the accuracy of water segmentation. In the case of high-altitude photography, due to cloud cover, relying solely on multispectral imagery cannot effectively segment the water area under the cloud shadow, and the help of synthetic aperture radar data is required. In addition, the available annotated data for water segmentation in multimodal remote sensing images is limited, the data source is single, and the complementary information of multi-source remote sensing data is not fully exploited. The model is too dependent on the quality of a single data source, and the segmentation effect is less than ideal. In addition, manual annotation is expensive and insufficient to support the training of large and complex networks.

[0007] Existing multimodal water segmentation methods generally use simple superposition of raw data or simple superposition of features to utilize multimodal data. This simple superposition method cannot effectively extract the complementary features between different modalities, but instead produces many redundant features, affecting the final segmentation results.

[0008] As a special feature selection method, existing methods rarely use deep networks to mine water body index information. In addition, the existing technical network structure design is too complex, with many modules and high requirements for hardware equipment, which is not conducive to deployment and real-time detection. Summary of the Invention

[0009] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to propose a remote sensing image water segmentation method, system and equipment that integrates multi-path fusion of water index and polarization information. By utilizing two types of multi-source remote sensing data, multispectral images and SAR images, the water index is introduced into the original UNet algorithm, a multi-path multi-source data fusion encoder is designed, and the GatedChannel Transform (GCT) attention mechanism is utilized to cope with the adverse effects of the complex ground environment in remote sensing images on water segmentation, thereby improving the accuracy of water segmentation with fewer computing resources.

[0010] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] A remote sensing image water segmentation method based on multi-path fusion of water index and polarization information includes the following steps:

[0012] S1. Collect images acquired by SAR remote sensing satellites and multispectral remote sensing satellites in the same area and similar time period from flood event datasets in different regions;

[0013] S2. Calculate water body index based on remote sensing images collected by multispectral remote sensing satellites;

[0014] S3. Construct a multi-source remote sensing dataset (WIPI) containing water index and polarization information;

[0015] S4. The WIPI dataset in step S3 is randomly divided into a training set, a validation set, and a test set based on regions and a custom ratio;

[0016] S5. Construct an input processing module for the water segmentation network consisting of invalid area shielding, data normalization, data standardization, data enhancement, and random sampling. Perform normalization and standardization preprocessing on the entire WIPI dataset, and perform data enhancement on the training set and validation set.

[0017] S6. Design a deep network model for water segmentation based on multi-path fusion of water index and polarization information;

[0018] S7, training the water segmentation model;

[0019] S8, using the test set in step S5 to test the optimal model trained in step S7, and using the F1 coefficient, the number of model parameters, and the amount of model calculation to evaluate the segmentation performance and complexity of the water segmentation model;

[0020] S9. Compare the segmentation effects of different fusion structures and give the best fusion solution.

[0021] The specific method of step S2 is:

[0022] S201. Band selection: Select a multispectral remote sensing satellite image that can provide blue band, green band, red band, near infrared band or short-wave infrared band;

[0023] S202, calculate the water index according to the multispectral image data set of the band selected in step S201, the water index includes the normalized water index (NDMI), the normalized vegetation index (NDVI), the normalized difference water index (NDWI), the improved normalized difference water index (MNDWI), the automatic water extraction index (AWEI), the normalized difference water index (NDWI), the normalized difference water index (MD ... nsh and AWEI sh ) and water body index of linear discriminant analysis (LDA);

[0024] S203 , saving the multiple water body indices calculated from each multispectral image as an image with the same number of channels as the water body indices, for subsequent water area segmentation.

[0025] The specific method of step S3 is:

[0026] Polarization information is obtained from the image collected by the SAR remote sensing satellite in step S1. The SAR image containing polarization information and the water index calculated in step S203 are combined into a data pair to construct a water index and polarization information dataset (WIPI) for subsequent training, verification, and testing of the segmentation model.

[0027] The specific method of step S4 is:

[0028] First, the WIPI datasets collected from different regions were grouped by region. Second, each group was randomly divided into training, validation, and test sets according to a custom ratio. Finally, the training, validation, and test sets were merged to obtain the final partitioning results.

[0029] The specific method of step S5 is: performing data preprocessing and data enhancement operations on the data image obtained in step S4 before inputting it into the water area segmentation model. The specific method is:

[0030] S501, generating an invalid area mask: according to step S4, the invalid value pixel position of each image is obtained, and an invalid area mask is obtained. The data is normalized and standardized according to the invalid area mask to shield the invalid value pixels and remove the influence of the invalid area;

[0031] S502, data normalization processing: according to the channel dimension direction, calculate the overall maximum value and overall minimum value of the same channel in all images after step S501, and use the two overall maximum value and overall minimum value indicators to normalize the data of the channel. The specific formula is as follows:

[0032]

[0033]

[0034] in, represents the j-th pixel value in the c-th channel of the i-th image, express Normalized pixel value, min c is the minimum pixel value of the c-th channel layer, max c is the maximum pixel value of the c-th channel layer;

[0035] S503. After normalization, all images with the same channel layer are grouped together, and the overall mean and overall variance are calculated within the group. The calculation formulas for the overall mean and overall variance are as follows:

[0036]

[0037]

[0038] Among them, Mean c Indicates the pixel mean of the c-th channel, Std c represents the pixel standard deviation of the c-th channel, represents the jth pixel value of the cth channel of the i-th image, N represents the number of images, and M represents the number of pixels in a single image;

[0039] S504, data standardization processing: the normalized data obtained in step S502 Perform channel dimension standardization. The standardization formula is as follows:

[0040]

[0041] in, represents the j-th pixel value in the c-th channel of the i-th image after normalization;

[0042] S505, data enhancement: After normalization and standardization, perform data enhancement including horizontal flipping, vertical flipping, random angle (45°-180°) counterclockwise rotation, adding random noise, and enhancing contrast to expand the training data and verification data;

[0043] S506, random sampling: Randomly sample image blocks with a width and height of w×h from the training set and validation set of the data augmented and expanded WIPI data. Traverse each image and extract [25, 50] image blocks from each image, where h is greater than or equal to 32 and less than the height of the image, and w is greater than 32 and less than the width of the image.

[0044] Generate two sets of data X and Y using a random number function. Each set includes [25, 50] numbers, which are combined into the coordinates of the upper left corner of the image block to be intercepted according to the subscript correspondence. After determining the coordinates of the upper left corner of the image block to be intercepted and the interception width and height, the image block can be generated. The specific implementation formula is as follows:

[0045]

[0046] Where N is the total number of images, H is the vertical axis coordinate sequence, W is the horizontal axis coordinate sequence, Range(·) is the sequence generation function, the formula can generate an arithmetic progression with a step size of [10, 30] within the specified range, RandomSample(·) is the random sampling function, and randomly extracts [25, 50] numbers from the sequence generated by Range(·), [H j ,W j ] represents the upper left corner coordinate of the jth image block, ImageBlock i,j represents the jth image block of the i-th image.

[0047] The specific method of step S6 is:

[0048] Improvements to the original UNet algorithm include introducing a multi-path multi-source data fusion encoder and using the Gated Channel Transform (GCT) attention mechanism to construct a multi-path fusion network. The specific steps are as follows:

[0049] S601. Design a deep network model for water segmentation in remote sensing images that integrates multi-path fusion of water index and polarization information. The water segmentation network uses a convolutional network structure, including: a multi-path contraction network on one side, which serves as an encoder for multimodal feature extraction; a middle skip connection combined with GCT, which performs channel weighting on multi-path features and inputs the weighted features into a decoder; and a decoder on the other side of the network, which serves as an expansion network for restoring image size and fusing deep and shallow features.

[0050] S602, constructing a CBR (Convolution-BatchNormalization-ReLU water segmentation network) module including a convolution layer, a batch normalization layer, and a ReLU nonlinear activation layer. The CBR module in the encoder keeps the feature resolution unchanged and increases the number of channels. The CBR module in the decoder keeps the feature resolution unchanged and reduces the number of channels.

[0051] S603, constructing a classifier module including a convolutional layer, a batch normalization layer, and a ReLU nonlinear activation layer, for reducing the number of channels of the feature map to the number of categories as the output result of the entire water segmentation deep network model;

[0052] S604: Constructing an encoder module for the water segmentation network based on the CBR module of step S602 and the maximum pooling layer in the water segmentation network. This module alternately uses the CBR module and the maximum pooling layer to form a multi-path and multi-scale contraction network structure, which performs multi-path layer-by-layer feature extraction on the input multimodal data. Each layer of the encoder module receives the feature input of the previous layer, and the generated feature output has two destinations: downward input to the maximum pooling layer and horizontal transmission to the GCT module.

[0053] S605. Construct a decoder module of the water segmentation network based on the CBR module and deconvolution layer of step S602. This module alternately uses the CBR module and the deconvolution layer to form an expanded network structure that fuses multi-scale deep and shallow features. This structure accepts multi-path features extracted from the encoder through skip connections, fuses the shallow features from the encoder with the deep features extracted from the previous layer, and expands the resolution of the feature map layer by layer.

[0054] S606. According to the polarization information of the SAR image and the number of water body indices, the number of input paths of the multi-path encoder is set on the WIPI dataset; the VV and VH polarization information of the SAR image are used as an input path, the NDWI and MNDWI of the water body indices are used as an input path, and the AWEI sh and LDA as an input path, NDVI, NDMI and AWEI nsh Input into the network through separate input paths respectively;

[0055] S607. The multi-path encoder receives input of different modal data and extracts features layer by layer to obtain intermediate features. The specific formula is as follows:

[0056]

[0057]

[0058] Where n represents the nth layer of the encoder, p represents the pth path of the encoder, represents the input of the nth layer in the pth path of the encoder, represents the output features of the nth layer in the pth path of the encoder, and MP() represents the 2×2 maximum pooling operation;

[0059] Then, the intermediate features of different paths but the same layer are spliced ​​in the channel dimension to form the output features of each layer of the encoder module in step S604, that is, the spliced ​​features F n , the specific formula is as follows:

[0060]

[0061] Among them, F n It is formed by concatenating the output features of all paths in the nth layer of the encoder;

[0062] S608, perform GCT channel attention calculation on each layer output feature of the multi-path encoder, GCT n There are three sets of learnable parameters: α, γ, β, which can be the n-th layer splicing feature F n Each channel generates a learnable parameter; the specific formula is as follows:

[0063] GCE=α×(L2Norm(F n ))

[0064]

[0065] CSF n =tanh(γ×(CN)+β)×F n +F n

[0066] Among them, GSF n is the channel selection feature of the nth layer, L2Norm() is the L2 normalization function, GCE is the global context embedding, and CN is the channel normalization.

[0067] S702: Use the training set and validation set processed in step S5 to train the model designed in step S6. During each training process, the model with the highest F1 coefficient on the validation set is saved as the optimal model.

[0068] The specific method of step S9 is:

[0069] Design three sets of experiments: "Multi-path vs. single-path," "Multi-modality vs. single-modality," and "Water index vs. multispectral raw bands." Use the dataset in step S5 and the method in step S7 to train models for each experiment. Finally, evaluate the model to obtain the optimal fusion solution. The specific steps are as follows:

[0070] S901. Design a "multi-path vs. single-path" comparison scheme. Separate the polarization information and water index into multiple paths and input them into the water segmentation model. In this case, the encoder is a "multi-path encoder." Concatenate the polarization information and water index along the channel dimension and input them into the water segmentation model via a single path. In this case, the encoder is a "single-path encoder." Train the two models separately and compare their prediction results on the test set.

[0071] S902. Design a "multimodal vs. single-modal" comparison scheme. Train models using only polarization information, only water index, and both types of data, and compare the prediction results on the test set.

[0072] S903. Design a comparison scheme for "water index and multispectral original bands". This scheme does not use polarization information and adopts a "single-path encoder". Train two models separately and compare the prediction results on the test set.

[0073] A segmentation system for implementing a remote sensing image water area segmentation method that implements multi-path fusion of water body index and polarization information in steps 1 to 9 includes:

[0074] A data set construction module is used to extract water body indices from collected remote sensing images and combine them with SAR images to construct a data set for completing the training, verification and testing tasks of the water segmentation model in the present invention;

[0075] Data input processing module, used to complete invalid area shielding, data normalization, data standardization, data enhancement and random sampling tasks;

[0076] Multi-path encoder module, used to perform multi-path layer-by-layer feature extraction on the input multimodal data;

[0077] The channel-weighted skip connection module is used to redistribute the weights between different channels, so that the model pays more attention to the channels with important data;

[0078] The decoder module is used to fuse the multi-path features extracted by the multi-path encoder with the deep features extracted by the previous layer, and to expand the resolution of the feature map layer by layer;

[0079] The classifier module is used to reduce the number of channels of the feature map to the number of categories as the output result of the entire water segmentation deep network model.

[0080] A segmentation device for implementing a remote sensing image water area segmentation method that implements multi-path fusion of water index and polarization information in steps 1 to 9 includes:

[0081] Image collector, used to collect images collected by SAR remote sensing satellites and multispectral remote sensing satellites in the same area and similar time period from flood event datasets in different regions;

[0082] A program processor is used to store a computer program and implement the remote sensing image water area segmentation method of multi-path fusion of water body index and polarization information described in steps 1 to 9 when executing the computer program.

[0083] Compared with the prior art, the present invention has the following beneficial effects:

[0084] 1. The features extracted from the water index and polarization information in this invention are more separable. In this invention, the water index and polarization information are integrated to extract feature information in a multi-path and multi-scale manner, which can help the model more easily segment the water area. In complex environments, if one modal data is incomplete, complementary information can be extracted from another modality, ensuring that the model segmentation accuracy is not significantly reduced.

[0085] 2. The feature fusion method of the present invention is more effective. In the present invention, a multi-path encoder network is first used to extract features of water index and polarization information. Then, the intermediate features of different modal data at the same level are preliminarily fused by splicing. Finally, weighted skip connections are used to input them into the decoder for deep and shallow feature fusion. This can extract features at different levels of multimodal data, which is better used for water segmentation tasks.

[0086] 3. This paper proposes a simple U-shaped convolutional network consisting of an encoder, skip connections, a decoder, and a classifier. The multi-path contraction network on the left is the encoder part for multimodal feature extraction. The skip connections in the middle, combined with GCT, are used to perform channel weighting on the multi-path features, which are then fed into the decoder. The decoder, located on the right side of the network, acts as an expansion network to restore image size and fuse deep and shallow features. The classifier, located in the last layer of the network, converts the number of channels in the feature map output by the previous layer into the number of categories. This network allows for rapid training and deployment, low computational cost, and short detection times. Compared to existing technologies, the network proposed in this paper requires less computation and can perform well even on devices with lower performance.

[0087] 4. The model in the present invention adopts a modular design concept. The basic network structures in the multi-path encoder, decoder and classifier can be replaced, and the weight calculation mechanism in the weighted jump connection can also be replaced. Modules can be added or modified according to the shortcomings of the basic network in the prediction task. With the development of emerging technologies and the proposal of better network modules, the present invention can be iteratively updated at any time to improve model performance.

[0088] 1) Iteration of basic feature extraction block

[0089] To achieve efficient and high-performance models, this paper selects CBR, the most suitable feature extraction building block for the task. Subsequently, ResNet-34 building blocks can be used in conjunction with depthwise separable convolutions to further improve the accuracy of water segmentation. As networks with improved performance and higher efficiency are proposed, the technology can be upgraded by using more optimized building blocks.

[0090] 2) Iteration of multimodal fusion methods

[0091] The encoder of the present invention uses a multi-path, multi-scale feature extraction structure to fuse multimodal information. This, to a certain extent, avoids interference between different modal data when extracting features, improving the performance of water segmentation. Therefore, the present invention can achieve technical iteration by updating the specific implementation methods in the multimodal fusion framework. For example, the present invention uses a multi-path, multi-scale fusion approach. When a better multimodal feature fusion method is proposed, it is replaced with a better method to further improve the performance of the model.

[0092] 3) Iteration of channel attention

[0093] When fusing multi-path features, the present invention uses the GCT module to perform attention weighting on different channels. Therefore, the present invention can achieve technical iteration by updating the specific implementation method in the channel attention module. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 Flowchart of the present invention.

[0095] Figure 2 This is the overall structure diagram of the water body index and polarization information fusion segmentation network of the present invention.

[0096] Figure 3 This is the structural diagram of the Gated Channel Transform (GCT) attention mechanism of the present invention.

[0097] Figure 4 It is a training flow chart of the present invention.

[0098] Figure 5 This is a test flow chart of the present invention.

[0099] Figure 6 This is the comparison model Model V1 of the present invention.

[0100] Figure 7 This is the comparison model Model V2 of the present invention. DETAILED DESCRIPTION

[0101] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0102] A remote sensing image water segmentation method based on multi-path fusion of water index and polarization information includes the following steps: (See Figure 1 、 Figure 2 )

[0103] S1. Images collected by the SAR remote sensing satellite Sentinel-1 and the multispectral remote sensing satellite Sentinel-2 were collected from 11 flood events in 11 regions over the same area and in a similar time period.

[0104] Because Sentinel-1 and Sentinel-2 imagery comes from two different satellites, if the time between data acquisition is too long, the features in the images will change, and the two data types will not provide complementary information, causing interference during the water segmentation process. Therefore, when using multimodal data, the images collected by Sentinel-1 and Sentinel-2 are first screened to ensure that the time interval between the two modal data acquisition times for the same area is within two days.

[0105] S2. Calculate the water index based on Sentinel-2 multispectral images.

[0106] S201. Band selection. The remote sensing images produced by the Sentinel-2 satellite have a total of 13 channels, namely B1, B2, B3, B4, B5, B6, B7, B8, B8A, B9, B10, B11, and B12, corresponding to the imaging information of the ground objects in 13 different bands. Since the water body index can highlight the water body area and help distinguish the water body from other ground objects by combining the information of different bands, this paper uses the normalized moisture index, normalized vegetation index, normalized difference water body index, improved normalized difference water body index, automatic water body extraction index, and linear discriminant analysis water body index. The bands required for the calculation of these indices are B2, B3, B4, B8, B8A, B11, and B12. The specific meaning of each band is as follows:

[0107] B2: 10-meter spatial resolution blue band.

[0108] B3: 10-meter spatial resolution green band.

[0109] B4: 10-meter spatial resolution red band.

[0110] B8: 10-meter spatial resolution near-infrared band (NIR).

[0111] B8A is: 20-meter spatial resolution near infrared band (NIR).

[0112] B11 and B12: 20-meter spatial resolution shortwave infrared (SWIR) band.

[0113] S202, water body index calculation: Read the multispectral image in step S1, and then combine different bands according to the calculation formula in Table 1 to obtain seven different water body indices.

[0114] Table 1 Water index used in the present invention

[0115]

[0116] S203. Seven different water body indices can be calculated from each multispectral image, and these seven water body indices are saved as a seven-channel image for subsequent water area segmentation.

[0117] S3. Construct a multi-source remote sensing dataset (WIPI) containing water index and polarization information;

[0118] The SAR image of the Sentinel-1 satellite contains two polarization modes, VV and VH. Therefore, two polarization information can be obtained. The SAR image containing polarization information in step S1 and the water index calculated in step S203 are combined into a data pair to construct the Water Index and Polarization Information Dataset (WIPI) for subsequent training, verification, and testing of the segmentation model.

[0119] S4. The WIPI dataset in step S3 is randomly divided into a training set, a validation set, and a test set in a ratio of 7:1:2 based on the region.

[0120] First, the WIPI dataset, collected from 11 different regions, was divided into 11 groups based on region. Next, each group was randomly split into training, validation, and test sets in a ratio of 7:1:2. Finally, all 11 training sets, 11 validation sets, and 11 test sets were combined to obtain the final partitioning results.

[0121] S5. Construct an input processing module for the water segmentation network, which consists of invalid area shielding, data normalization, data standardization, data enhancement, and random sampling. Perform normalization and standardization preprocessing on the WIPI full dataset, and perform data enhancement on the training set and validation set.

[0122] The data obtained in step S4 is preprocessed and enhanced before being input into the water segmentation model. The specific method is as follows:

[0123] S501: Generate an invalid area mask. According to the invalid value pixel positions of each image, an invalid area mask is obtained. During data normalization and standardization, invalid value pixels are shielded according to the invalid area mask to remove the influence of the invalid area.

[0124] S502, data normalization processing. According to the channel dimension direction, calculate the overall maximum and minimum values ​​of the same channel in all images, and use the two obtained indicators to normalize the data of the channel. The specific formula is as follows:

[0125]

[0126]

[0127] in: represents the j-th pixel value in the c-th channel of the i-th image, express Normalized pixel value, min c is the minimum pixel value of the c-th channel layer, max c is the maximum pixel value of the c-th channel layer;

[0128] S503. After normalization, all images with the same channel layer are grouped together, and the overall mean and overall variance are calculated within the group. The calculation formulas for the overall mean and overall variance are as follows:

[0129]

[0130]

[0131] Among them, Mean c Indicates the pixel mean of the c-th channel, Std c represents the pixel standard deviation of the c-th channel, represents the jth pixel value of the cth channel of the i-th image, N represents the number of images, and M represents the number of pixels in a single image;

[0132] S504, data standardization processing; the normalized data obtained in S502 Perform channel dimension standardization. The standardization formula is as follows:

[0133]

[0134] in, represents the j-th pixel value in the c-th channel of the i-th image after normalization;

[0135] S505: Data enhancement.

[0136] Currently, there is a limited amount of multimodal, accurately labeled data for water segmentation tasks. Using data augmentation to generate more training and validation data helps to make the augmented data as close to the true distribution as possible, thereby effectively improving the generalization ability of the model. This paper augments the normalized and standardized data with five methods: horizontal flipping, vertical flipping, random angle (45°-180°) counterclockwise rotation, random noise addition, and contrast enhancement. This method expands the training and validation data by six times.

[0137] S506: Random sampling. Randomly sample image blocks of 128×128 in width and height from the training and validation sets of the data augmented WIPI data. Traverse each image and extract 16 image blocks from each image.

[0138] A random number function is used to generate two sets of data, X and Y. Each set contains 16 numbers, which are combined into the coordinates of the upper left corner of the image block to be intercepted according to the corresponding subscripts. After determining the coordinates of the upper left corner of the image block to be intercepted and the interception width and height, the image block can be generated. The specific implementation formula is as follows:

[0139]

[0140] Where N is the total number of images, H is the vertical axis coordinate sequence, W is the horizontal axis coordinate sequence, Range(·) is the sequence generation function, and the formula can generate an arithmetic sequence with a step size of 10 in the range [0,384). RandomSample(·) is the random sampling function, which randomly extracts 32 numbers from the sequence generated by Range(·), [H j ,W j ] represents the upper left corner coordinate of the jth image block, ImageBlocki ,h represents the jth image block of the i-th image.

[0141] S6. Design a deep network model for water segmentation that integrates multi-path fusion of water body index and polarization information.

[0142] In order to design a lightweight network that can fuse water index and polarization information, this paper makes a series of improvements to the original UNet algorithm, including introducing a multi-path multi-source data fusion encoder and using the Gated Channel Transform (GCT) attention mechanism to construct a multi-path fusion network. Figure 3 The specific steps are as follows:

[0143] S601. Design a deep network model for water segmentation of remote sensing images based on multi-path fusion of water index and polarization information, such as Figure 2 The water segmentation network adopts a U-shaped convolutional network structure, which mainly consists of three parts. The multi-path contraction network on one side is the encoder, which is used for multimodal feature extraction; the jump connection in the middle is combined with GCT to perform channel weighting on the multi-path features and input the weighted features into the decoder; the decoder is on the other side of the network and acts as an expansion network to restore the image size and fuse the deep and shallow features.

[0144] S602. Construct a CBR (Convolution-BatchNormalization-ReLU water segmentation network) module consisting of 1 convolution layer, 1 batch normalization layer, and 1 ReLU nonlinear activation layer. The CBR module in the encoder can keep the feature resolution unchanged and double the number of channels. The CBR module in the decoder can keep the feature resolution unchanged and halve the number of channels.

[0145] S603. Construct a classifier module including one convolutional layer, one batch normalization layer, and one ReLU nonlinear activation layer to reduce the number of channels of the feature map to the number of categories as the output result of the entire water segmentation deep network model.

[0146] S604. An encoder module for the water segmentation network is constructed based on the CBR module and the maximum pooling layer of step S602. This module alternately uses the CBR module and the maximum pooling layer to form a multi-path and multi-scale contraction network structure, which performs multi-path layer-by-layer feature extraction on the input multimodal data. Each layer of the encoder module receives the feature input of the previous layer, and the generated feature output has two destinations: downward input to the maximum pooling layer and horizontal transmission to the GCT module.

[0147] S605. Construct a decoder module of the water segmentation network based on the CBR module and deconvolution layer of step S602. This module alternately uses the CBR module and the deconvolution layer to form an expanded network structure that fuses multi-scale deep and shallow features. This structure accepts multi-path features extracted from the encoder through jump connections, fuses the shallow features from the encoder with the deep features extracted from the previous layer, and expands the resolution of the feature map layer by layer.

[0148] S606. In the present invention, the number of input paths of the multi-path encoder is determined by the polarization information of the SAR image and the number of water body indices, which is set to 6 on the WIPI dataset; the VV and VH polarization information of the SAR image are used as an input path, the NDWI and MNDWI of the water body index are used as an input path, and the AWEI shand LDA as an input path, NDVI, NDMI and AWEI nsh They are input into the network via separate input paths.

[0149] S607: The multi-path encoder receives input of different modal data and extracts features layer by layer to obtain intermediate features. The specific formula is as follows:

[0150]

[0151]

[0152] Where n represents the nth layer of the encoder, p represents the pth path of the encoder, represents the input of the nth layer in the pth path of the encoder, represents the output features of the nth layer in the pth path of the encoder, and MP() represents the 2×2 maximum pooling operation;

[0153] Then, the intermediate features of different paths but the same layer are spliced ​​in the channel dimension to form the output features of each layer of the encoder module in step S604, that is, the spliced ​​features F n The specific formula is as follows:

[0154]

[0155] Among them, F n It is formed by concatenating the output features of all paths in the nth layer of the encoder.

[0156] S608, perform GCT channel attention calculation on each layer output feature of the multi-path encoder, GCT n There are three sets of learnable parameters: α, γ, β, which can be the n-th layer splicing feature F n Each channel generates a learnable parameter; the specific formula is as follows:

[0157] GCE=α×(L2Norm(F n ))

[0158]

[0159] CSF n =tanh(γ×(CN)+β)×F n +F n

[0160] Among them, CSF nis the channel selection feature of the nth layer, L2Norm() is the L2 normalization function, GCE is the global context embedding, and CN is the channel normalization.

[0161] S7. Train the water segmentation model (see Figure 4 )

[0162] S701, during the training of the segmentation model, the loss function uses the cross entropy loss function L log (y,p), defined as:

[0163] L log (y,p)=-(ylog(p)+(1-y)log(1-p))

[0164] Among them, y is the label and p is the predicted probability of the segmentation model.

[0165] S702, using the training set and validation set processed in S5 to train the model designed in S6, during each training process, the model with the highest F1 coefficient on the validation set is saved as the optimal model;

[0166] S8. Use the test set in S5 to test the optimal model trained in S7, and use the F1 coefficient, model parameters and model calculation amount to evaluate the segmentation performance and complexity of the water segmentation model; (See Figure 5 )

[0167] S9. Compare the segmentation effects of different fusion structures and give the best fusion solution;

[0168] By designing three sets of experiments, namely “multi-path vs. single-path”, “multi-modal vs. single-modal”, and “water index vs. multi-spectral original band”, and using the dataset in S5 and the method in S7 to train models for different experiments; finally, through model evaluation, the best fusion solution is obtained (see Figure 2 、 Figure 6 、 Figure 7 ); the specific method is as follows:

[0169] S901. Design a "multi-path versus single-path" comparison scheme. Divide the polarization information and water index into multiple paths and input them into the water segmentation model. In this case, the encoder is a "multi-path encoder." Connect the polarization information and water index according to the channel dimension and input them into the water segmentation model via one path. In this case, the encoder is a "single-path encoder." Train the two models separately and compare the prediction results on the test set.

[0170] S902. Design a "multimodal vs. unimodal" comparison scheme. This involves training models for "using only polarization information," "using only water index," and "using both types of data," and comparing the prediction results on the test set.

[0171] S903. Design a comparison scheme for "water index and multispectral original bands". This scheme does not use polarization information and adopts a "single-path encoder". Train two models separately and compare the prediction results on the test set.

[0172] A segmentation system for implementing a remote sensing image water area segmentation method that implements multi-path fusion of water body index and polarization information in steps 1 to 9 includes:

[0173] A data set construction module is used to extract water body indices from collected remote sensing images and combine them with SAR images to construct a data set for completing the training, verification and testing tasks of the water segmentation model in the present invention;

[0174] Data input processing module, used to complete invalid area shielding, data normalization, data standardization, data enhancement and random sampling tasks;

[0175] Multi-path encoder module, used to perform multi-path layer-by-layer feature extraction on the input multimodal data;

[0176] The channel-weighted skip connection module is used to redistribute the weights between different channels, so that the model pays more attention to the channels with important data;

[0177] The decoder module is used to fuse the multi-path features extracted by the multi-path encoder with the deep features extracted by the previous layer, and to expand the resolution of the feature map layer by layer;

[0178] The classifier module is used to reduce the number of channels of the feature map to the number of categories as the output result of the entire water segmentation deep network model.

[0179] A segmentation device for implementing a remote sensing image water area segmentation method that implements multi-path fusion of water index and polarization information in steps 1 to 9 includes:

[0180] Image collector, used to collect images collected by SAR remote sensing satellites and multispectral remote sensing satellites in the same area and similar time period from flood event datasets in different regions;

[0181] A program processor is used to store a computer program and implement the remote sensing image water area segmentation method of multi-path fusion of water body index and polarization information described in steps 1 to 9 when executing the computer program.

[0182] The present invention utilizes two types of multi-source remote sensing data, multispectral images and SAR images, and proposes a remote sensing image water segmentation method that integrates multi-path fusion of water index and polarization information. This method makes a series of improvements to the original UNet algorithm, including the introduction of water index, the design of a multi-path multi-source data fusion encoder, and the use of the Gated Channel Transform (GCT) attention mechanism, to cope with the adverse effects of complex ground objects in remote sensing images on water segmentation, thereby improving the accuracy of water segmentation with fewer computing resources.

Claims

1. A remote sensing image water segmentation method based on multi-path fusion of water index and polarization information, characterized in that: The specific steps include: S1. Collect images acquired by SAR remote sensing satellites and multispectral remote sensing satellites in the same area and similar time period from flood event datasets in different regions; S2. Calculate the water index based on the remote sensing images collected by multispectral remote sensing satellites: S3. Construct a multi-source remote sensing dataset (WIPI) containing water index and polarization information; S4. The WIPI dataset in step S3 is randomly divided into a training set, a validation set, and a test set based on regions and a custom ratio; S5. Construct an input processing module for the water segmentation network consisting of invalid area shielding, data normalization, data standardization, data enhancement, and random sampling. Perform normalization and standardization preprocessing on the entire WIPI dataset, and perform data enhancement on the training set and validation set. S6. Design a water segmentation deep network model that integrates water index and polarization information multi-path fusion; specifically: S601. Design a deep network model for water segmentation in remote sensing images that integrates multi-path fusion of water index and polarization information. This water segmentation network uses a convolutional network structure, including: a multi-path contraction network on one side, which serves as an encoder for multimodal feature extraction; a middle skip connection combined with GCT, which performs channel weighting on multi-path features and inputs the weighted features into a decoder; and a decoder on the other side of the network, which serves as an expansion network for restoring image size and fusing deep and shallow features. S602, constructing a CBR (Convolution-BatchNormalization-ReLU water segmentation network) module including a convolution layer, a batch normalization layer, and a ReLU nonlinear activation layer. The CBR module in the encoder keeps the feature resolution unchanged and increases the number of channels. The CBR module in the decoder keeps the feature resolution unchanged and reduces the number of channels. S603, constructing a classifier module including a convolutional layer, a batch normalization layer, and a ReLU nonlinear activation layer, for reducing the number of channels of the feature map to the number of categories as the output result of the entire water segmentation deep network model; S604: Constructing an encoder module for the water segmentation network based on the CBR module of step S602 and the maximum pooling layer in the water segmentation network. This module alternately uses the CBR module and the maximum pooling layer to form a multi-path and multi-scale contraction network structure, which performs multi-path layer-by-layer feature extraction on the input multimodal data. Each layer of the encoder module receives the feature input of the previous layer, and the generated feature output has two destinations: downward input to the maximum pooling layer and horizontal transmission to the GCT module. S605. Construct a decoder module of the water segmentation network based on the CBR module and deconvolution layer of step S602. This module alternately uses the CBR module and the deconvolution layer to form an expanded network structure that fuses multi-scale deep and shallow features. This structure accepts multi-path features extracted from the encoder through skip connections, fuses the shallow features from the encoder with the deep features extracted from the previous layer, and expands the resolution of the feature map layer by layer. S606. According to the polarization information of the SAR image and the number of water body indices, the number of input paths of the multi-path encoder is set on the WIPI dataset; the VV and VH polarization information of the SAR image are used as an input path, the NDWI and MNDWI of the water body indices are used as an input path, and the AWEIn sh and LDA as an input path, NDVI, NDMI and AWEI nsh Input into the network through separate input paths respectively; S607, the multi-path encoder receives input of different modal data and extracts features layer by layer to obtain intermediate features; then, the intermediate features of different paths but the same layer are spliced ​​in the channel dimension to form the output features of each layer of the encoder module in step S604, that is, the spliced ​​features F n ; S608, perform GCT channel attention calculation on each layer output feature of the multi-path encoder, GCT n There are three sets of learnable parameters: α, γ, β, which can be the n-th layer splicing feature F n Each channel of generates a learnable parameter; S7, training the water segmentation model; S8, using the test set in step S5 to test the optimal model trained in step S7, and using the F1 coefficient, the number of model parameters, and the amount of model calculation to evaluate the segmentation performance and complexity of the water segmentation model; S9. Compare the segmentation effects of different fusion structures and give the best fusion solution.

2. The method for water segmentation of remote sensing images by multi-path fusion of water index and polarization information according to claim 1 is characterized in that: The specific method of step S2 is: S201. Band selection: Select multispectral remote sensing satellite images that can provide blue band, green band, red band, near infrared band, and shortwave infrared band; S202, calculate the water index according to the multispectral image data set of the band selected in step S201, the water index includes the normalized water index (NDMI), the normalized vegetation index (NDVI), the normalized difference water index (NDWI), the improved normalized difference water index (MNDWI), the automatic water extraction index (AWEI), the normalized difference water index (NDWI), the normalized difference water index (MD ... nsh and AWEI sh ) and water body index of linear discriminant analysis (LDA); S203 , saving the multiple water body indices calculated from each multispectral image as an image with the same number of channels as the water body indices, for subsequent water area segmentation.

3. The method for water segmentation of remote sensing images by multi-path fusion of water index and polarization information according to claim 1, characterized in that: The specific method of step S3 is: Polarization information is obtained from the image collected by the SAR remote sensing satellite in step S1. The SAR image containing polarization information and the water index calculated in step S203 are combined into a data pair to construct a water index and polarization information dataset (WIPI) for subsequent training, verification, and testing of the segmentation model.

4. The method for water segmentation of remote sensing images by multi-path fusion of water index and polarization information according to claim 1, characterized in that: The specific method of step S4 is: First, the WIPI datasets collected from different regions were grouped by region. Second, each group was randomly divided into training, validation, and test sets according to a custom ratio. Finally, the training, validation, and test sets were merged to obtain the final partitioning results.

5. The method for water segmentation of remote sensing images by multi-path fusion of water index and polarization information according to claim 1, characterized in that: The specific method of step S5 is: performing data preprocessing and data enhancement operations on the data image obtained in step S4 before inputting it into the water area segmentation model. The specific method is: S501, generating an invalid area mask: according to step S4, the invalid value pixel position of each image is obtained, and an invalid area mask is obtained. The data is normalized and standardized according to the invalid area mask to shield the invalid value pixels and remove the influence of the invalid area; S502, data normalization processing: according to the channel dimension direction, calculate the overall maximum and overall minimum values ​​of the same channel in all images after step S501, and use the two obtained overall maximum and overall minimum indexes to normalize the data of the channel. The specific formula is as follows: in: represents the j-th pixel value in the c-th channel of the i-th image, express Normalized pixel value, min c is the minimum pixel value of the c-th channel layer, max c is the maximum pixel value of the c-th channel layer; S503. After normalization, all images with the same channel layer are grouped together, and the overall mean and overall variance are calculated within the group. The calculation formulas for the overall mean and overall variance are as follows: Among them, Mean c Indicates the pixel mean of the c-th channel, Std c represents the pixel standard deviation of the c-th channel, represents the jth pixel value of the cth channel of the i-th image, N represents the number of images, and M represents the number of pixels in a single image; S504, data normalization processing: the normalized data X obtained in step S502 is normalized in the channel dimension. The normalization formula is as follows: in, represents the j-th pixel value in the c-th channel of the i-th image after normalization; S505, data enhancement: After normalization and standardization, perform data enhancement including horizontal flipping, vertical flipping, random angle 45°-180° counterclockwise rotation, adding random noise, and enhancing contrast to expand the training data and verification data; S506, random sampling: Randomly sample image blocks of width w × height h from the training set and validation set of the data augmented and expanded WIPI data. Traverse each image and extract [25, 50] image blocks from each image, where h is greater than or equal to 32 and less than the height of the image, and w is greater than or equal to 32 and less than the width of the image. Generate two sets of data X and Y using a random number function. Each set includes [25, 50] numbers, which are combined into the coordinates of the upper left corner of the image block to be intercepted according to the subscript correspondence. After determining the coordinates of the upper left corner of the image block to be intercepted and the interception width and height, the image block can be generated. The specific implementation formula is as follows: Where N is the total number of images, H is the vertical axis coordinate sequence, W is the horizontal axis coordinate sequence, Range(·) is the sequence generation function, the formula can generate an arithmetic progression with a step size of [10, 30] within the specified range, RandomSample(·) is the random sampling function, and randomly extracts [25, 50] numbers from the sequence generated by Range(·), [H j , W j ] represents the upper left corner coordinate of the jth image block, ImageBlock i，j represents the jth image block of the i-th image.

6. The method for water segmentation of remote sensing images by multi-path fusion of water index and polarization information according to claim 1, characterized in that: The specific method of step S7 is: S701. During the training of the segmentation model, the loss function uses the cross entropy loss function. L log (y,p), defined as: L log (y,p)=-(ylog(p)+(1-y)log(1-p)) Where y is the label and p is the predicted probability of the segmentation model; S702: Use the training set and validation set processed in step S5 to train the model designed in step S6. During each training process, the model with the highest F1 coefficient on the validation set is saved as the optimal model.

7. The method for water segmentation of remote sensing images by multi-path fusion of water index and polarization information according to claim 1, characterized in that: The specific method of step S9 is as follows: design three sets of experiments: "multi-path vs. single-path", "multi-modal vs. single-modal", and "water index vs. multispectral original band", and use the data set in step S5 and the method in step S7 to train models for different experiments; finally, obtain the optimal fusion solution through model evaluation. The specific steps are as follows: S901. Design a "multi-path vs. single-path" comparison scheme. Separate the polarization information and water index into multiple paths and input them into the water segmentation model. In this case, the encoder is a "multi-path encoder." Concatenate the polarization information and water index along the channel dimension and input them into the water segmentation model via a single path. In this case, the encoder is a "single-path encoder." Train the two models separately and compare their prediction results on the test set. S902. Design a "multimodal vs. single-modal" comparison scheme. Train models using only polarization information, only water index, and both data types, and compare the prediction results on the test set. S903. Design a comparison scheme for "water index and multispectral raw bands." This scheme does not use polarization information and adopts a "single-path encoder." Train two models separately and compare the prediction results on the test set.

8. A segmentation system for implementing the remote sensing image water area segmentation method of multi-path fusion of water index and polarization information according to any one of claims 1 to 7, characterized in that: The segmentation system comprises: A data set construction module is used to extract water body indices from collected remote sensing images and combine them with SAR images to construct a data set for completing the training, verification and testing tasks of the water segmentation model in the present invention; Data input processing module, used to complete invalid area shielding, data normalization, data standardization, data enhancement and random sampling tasks; Multi-path encoder module, used to perform multi-path layer-by-layer feature extraction on the input multimodal data; The channel-weighted skip connection module is used to redistribute the weights between different channels, so that the model pays more attention to the channels with important data; The decoder module is used to fuse the multi-path features extracted by the multi-path encoder with the deep features extracted by the previous layer, and to expand the resolution of the feature map layer by layer; The classifier module is used to reduce the number of channels of the feature map to the number of categories as the output result of the entire water segmentation deep network model.

9. A segmentation device for implementing the remote sensing image water area segmentation method of multipath fusion of water index and polarization information according to any one of claims 1 to 7, characterized in that: The segmentation device comprises: Image collector, used to collect images collected by SAR remote sensing satellites and multispectral remote sensing satellites in the same area and similar time period from flood event datasets in different regions; A program processor is used to store a computer program and implement the remote sensing image water area segmentation method of multi-path fusion of water body index and polarization information described in steps 1 to 9 when executing the computer program.

Images