Deep learning river ice extraction method based on optical and SAR fusion image

Abstract

The invention discloses a deep learning river ice extraction method based on an optical and SAR fusion image, and belongs to the technical field of remote sensing geoscience application. The method comprises the following steps: acquiring an optical image and an SAR image which are in the same area and contain river ice distribution, preprocessing the optical image and the SAR image, marking river ice areas of the preprocessed optical image and the preprocessed SAR image, and constructing a training data set; a deep learning extraction model is constructed, and the training data set is used for training; and obtaining an optical image and an SAR image of a to-be-extracted river ice distribution region, obtaining a binary image by using the trained deep learning extraction model, converting the binary image into vector data, removing a misjudged river ice region, obtaining a river ice region in the region, and completing river ice extraction. According to the method, advantages of optics and SAR images are combined, an attention mechanism and a multi-level feature fusion strategy are constructed, and technical support is provided for refined river ice drawing in the cold and cold mountainous area.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and remote sensing geoscience application technology, and specifically relates to a deep learning method for extracting river ice based on optical and SAR fusion images. Background Art

[0002] River ice, the freezing of river water during cold seasons, is a crucial component of the cryosphere. Its formation and dissipation directly impact watershed water circulation and river channel stability. Global warming is significantly altering river ice phenology, with factors such as a shortened freezing period increasing the risk of river ice disasters. River ice extraction in high-altitude mountainous areas is susceptible to interference from clouds, snow, and topography, significantly complicating its extraction. High-precision extraction of regional river ice distribution information is a pressing challenge. With the advancement of remote sensing technology, Sentinel-1 and Sentinel-2 satellite data provide data support for comprehensive, rapid, and high-temporal and spatial resolution river ice information acquisition and detailed mapping.

[0003] While Sentinel-1 data offers all-weather observation capabilities and cloud penetration, geometric distortion and speckle noise caused by terrain can reduce river ice extraction accuracy. While Sentinel-2 data offers higher accuracy compared to Sentinel-1 data, it is significantly affected by cloud and snow spectral aliasing, resulting in reduced classification accuracy. Currently, there is a lack of methods to fuse Sentinel-1 and Sentinel-2 data to complement their respective data features and improve river ice extraction accuracy in alpine mountainous areas. Summary of the Invention

[0004] In order to solve the problems in the prior art, the present invention provides a river ice extraction method based on deep learning of optical and SAR fusion images.

[0005] The technical solutions of the present invention are as follows:

[0006] The present invention discloses a method for extracting river ice based on deep learning of optical and SAR fusion images, which includes the following steps:

[0007] Obtain optical and SAR images of the same area containing river ice distribution and preprocess them. Then, label the river ice areas in the preprocessed optical and SAR images to construct training datasets. Build a deep learning extraction model and train it using the training datasets.

[0008] Obtain optical and SAR images of the river ice distribution area to be extracted, use the trained deep learning extraction model to obtain binary images, convert the binary images into vector data, remove the misjudged river ice areas, obtain the river ice area in the area, and complete the river ice extraction;

[0009] Wherein, the deep learning extraction model includes:

[0010] Two improved U-Net models are used to extract feature information from annotated optical and SAR images, respectively, to obtain two upsampled feature maps. Each improved U-Net model includes a decoder and an encoder. The encoder consists of multiple convolutional blocks connected in sequence for downsampling. Each convolutional block consists of a maximum pooling layer, a dilated spatial pyramid pooling layer, a convolutional attention module, and a multi-layer perceptron. The encoder of one improved U-Net model outputs optical features, and the other outputs SAR features.

[0011] The fusion and sampling module is used to fuse the outputs of the two encoders to obtain fused features, and upsample the fused features to obtain a fused upsampled feature map;

[0012] Two decision-level fusion modules, one for fusing the two upsampled feature maps, and the other for fusing the fusion result of the two upsampled feature maps with the fused upsampled feature map to obtain a binary image.

[0013] Furthermore, removing the misjudged river ice area includes:

[0014] Obtain distribution vector data of rivers, lakes, and glaciers at the location of the optical image of the river ice distribution area to be extracted (which can also be a SAR image of the river ice distribution area to be extracted), and remove areas in the vector data that are mistakenly identified as river ice based on the obtained distribution vector data of rivers, lakes, and glaciers.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] The present invention can extract the features of optical, SAR and DEM data in a multi-scale and multi-channel manner based on the attention mechanism, and perform multi-level fusion of the features of optical, SAR and DEM data at the feature level and decision level, thereby achieving full extraction of feature information and complementary advantages between different features, reducing the interference of clouds, snow and terrain in the identification of river ice in high-altitude mountainous areas, and then extracting river ice information in high-altitude mountainous areas with high precision and good reliability, and realizing refined river ice mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the river ice extent extraction method based on optical and SAR image fusion provided by the present invention.

[0018] Figure 2 is a structural diagram of the deep learning extraction model of the present invention;

[0019] Figure 3 It is a structural diagram of the feature-level fusion module provided by the present invention;

[0020] Figure 4 It is a structural diagram of the decision-level fusion module provided by the present invention;

[0021] Figure 5 This is a folder structure diagram of the river ice training dataset and test dataset provided by the present invention;

[0022] Figure 6 This is a river ice extent prediction display diagram based on the deep learning extraction model provided by the present invention;

[0023] Figure 7 This is a data display diagram of river ice distribution range in a certain mountainous area provided by the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described and illustrated below in conjunction with specific embodiments. The embodiments are merely illustrative of the present disclosure and do not limit its scope. The technical features of the various embodiments of the present invention may be combined accordingly, provided that there is no conflict between them.

[0025] The present invention mainly provides a deep learning method for extracting river ice based on the fusion of optical and SAR (Synthetic Aperture Radar) images. It can fully extract the features of optical images and SAR images and perform multi-level fusion, realize the complementary advantages of different features, reduce the interference of clouds, snow and terrain when identifying river ice in high-altitude mountainous areas, and improve the extraction accuracy of river ice distribution range.

[0026] The technical solution adopted by the present invention is to select 30MB of optical satellite data (Sentinel-2 satellite), Sentinel-1 satellite data, and SRTM DEM (Shuttle Radar Topography Mission Digital Elevation Model) data on river ice distribution. First, the data is preprocessed to create river ice training and test datasets. Then, the parameters of the deep learning extraction model are set and the deep learning extraction model is trained. The test dataset is predicted based on the trained deep learning extraction model to obtain prediction results. Finally, the raster-to-vector tool in ArcGIS (Geographic Information System) Pro or QGIS (Quantum Geographic Information System) software is used to extract rasters with a median value of 1 from the predicted binary image into vector data. Finally, based on the distribution data of rivers, lakes, glaciers, etc., misjudged non-river ice information is removed to form complete river ice distribution range data for the target area.

[0027] like Figure 1 As shown, the technical solutions disclosed in the present invention are as follows:

[0028] Step 1: Data Preparation

[0029] Depending on factors such as river ice width and accuracy requirements, optical imagery (i.e., Sentinel-2 satellite imagery) is collected using the Sentinel-2 satellite. These optical imagery is typically high-spatial-resolution Sentinel-2 imagery. High spatial resolution refers to a resolution of 30 meters or less, and the optical imagery consists of green, red, near-infrared, and shortwave infrared spectral bands. Furthermore, SAR imagery (i.e., Sentinel-1 satellite imagery) is collected using the Sentinel-1 satellite. These SAR imagery is typically high-temporal-resolution Sentinel-1 imagery. High-spatial-resolution refers to a spatial resolution of 30 meters or less, and a temporal resolution greater than 10 days. Some high-altitude mountainous areas have large terrain fluctuations that can interfere with the extraction of ground river ice information. Therefore, high-resolution DEM data, such as the SRTM DEM30M, is obtained as auxiliary data for the same area as the optical imagery. Due to the temporal differences between different satellites, optical and SAR images should be acquired with similar transit times (imaging times), that is, the acquisition time interval between optical and SAR images should be less than 10 days to reduce the impact of asymmetric river ice information in different images caused by changes in river ice extent in different temporal phases.

[0030] Step 2: Data Preprocessing

[0031] The DEM data in the first step was visualized to obtain a DEM image, and then the VH polarization bands of the optical image, DEM image, and SAR image were resampled. The spatial resolution of the VH polarization bands of the resampled optical image, DEM image, and SAR image was 10 m.

[0032] Next, the resampled optical image is processed using the Normalized Difference Snow Index (NDSI) algorithm to generate the NDSI image. The resampled optical image, the DEM image, the VH polarimetric bands of the SAR image, and the NDSI image are then unified into the EPSG:4326 geographic coordinate system. The VH polarimetric bands of the unified geographic coordinate system SAR image are then topographically corrected using the unified geographic coordinate system DEM image to obtain the corrected SAR image. The unified geographic coordinate system NDSI image, DEM image, and corrected SAR image are then composited into a new image. The method for compositing the new image is essentially to merge different bands to create the desired multi-band image. Common methods include: 1) using the Band Composite function (.addBands) in GEE; 2) using the Band Composite tool (Layer Stacking) in ENVI; and 3) using the Band Composite tool in ArcGIS. A specific embodiment of the present invention is to complete the pre-processing of each band in GEE, and then directly apply the .addBands function in GEE to synthesize the new image.

[0033] Then, the near-infrared spectrum band, the red spectrum band and the green spectrum band in the optical image after the unified geographic coordinate system are combined into a new optical image, and then the new optical image is processed by multi-scene image synthesis to obtain a scene average image to obtain a processed image.

[0034] Finally, the new image and the processed image are resized to complete the preprocessing. The new image after resizing is the preprocessed SAR image, and the processed image after resizing is the preprocessed optical image. Data preprocessing can be performed using remote sensing software such as Envi, or using a more efficient remote sensing data processing cloud platform.

[0035] Step 3: Preparation of river ice training and test datasets

[0036] The river ice distribution range of the pre-processed optical images and SAR images were annotated using ArcGIS Pro to create a river ice training dataset. The annotated optical images and SAR images in the river ice training dataset are all 1024×1024 size TIFF files, such as Figure 5As shown, the annotated optical images, preprocessed SAR images, and labels are stored in three folders, each with a consistent image name and one-to-one correspondence. This constitutes the river ice training dataset. Optical and SAR images of the river ice distribution area to be extracted are obtained and preprocessed identically to create a test dataset. The test dataset contains two folders: the preprocessed optical images and the preprocessed SAR images. The image format, including band order and size, in the test dataset must be identical to that in the river ice training dataset. Data processing can be performed using ArcGIS Pro or by writing code in Python.

[0037] Step 4: Deep learning extraction model based on dual-branch feature fusion and attention mechanism

[0038] The deep learning extraction model consists of two improved U-Net models, a fusion and sampling module, and two decision-level fusion modules. The two improved U-Net models are used to extract feature information from the annotated optical image and SAR image, respectively, to obtain two upsampled feature maps. The improved U-Net model includes a decoder and an improved encoder. Each improved encoder consists of multiple sequentially connected convolutional blocks for downsampling. Each convolutional block consists of a sequentially connected max pooling layer, a dilated spatial pyramid pooling layer, a convolutional attention module, and a multi-layer perceptron. The encoder of one improved U-Net model outputs optical features, and the encoder of the other outputs SAR features. The fusion and sampling module fuses the outputs of the two improved encoders to obtain fused features, which are then upsampled to obtain a fused upsampled feature map. One decision-level fusion module fuses the two upsampled feature maps, and the other decision-level fusion module fuses the fused result of the two upsampled feature maps with the fused upsampled feature map to obtain a binary image.

[0039] like Figure 2 As shown in the figure, the improved U-Net model uses the U-Net model as the basic architecture. The encoder of the improved U-Net model performs four-layer downsampling on the input image (i.e., the annotated optical image and SAR image). That is, the encoder includes four sequentially connected convolution blocks for downsampling. Each convolution block consists of a sequentially connected maximum pooling layer, an atrous spatial pyramid pooling (ASPP), a convolutional block attention module (CBAM), and a multi-layer perceptron (MLP), which is used to extract multi-scale and multi-channel feature information and strengthen the association between local details and global features.

[0040] The fusion and sampling module includes a feature-level fusion module and an upsampling network, wherein the upsampling network has the same decoder structure as the improved U-Net model. Figure 3 As shown, the feature-level fusion module is used to fuse the outputs of the two improved encoders, and includes two multi-head attention modules and two multi-head cross-attention modules; the multi-head cross-attention module is a multi-head self-attention module based on the fusion key module. The construction method of the multi-head cross-attention module is as follows: the SAR features and optical features output by the two encoders are layer-normalized respectively, and then the layer-normalized SAR features and the layer-normalized optical features are added together. After the addition, point convolution, depth convolution, and point convolution are performed in sequence to obtain a fusion key. The fusion key is then used to replace the key in the multi-head self-attention module to obtain the multi-head cross-attention module. Among them, the layer network for point convolution, the layer network for depth convolution, and the layer network for point convolution connected in sequence constitute the fusion key module.

[0041] The feature-level fusion module fuses the optical and SAR features output by the two improved encoders and outputs a fused feature. This module uses a multi-head self-attention module and a multi-head cross-attention module to process the optical and SAR features, respectively. The multi-head cross-attention module is constructed by inputting the optical and SAR features (keys) into a joint space through a fusion key module. This fusion key dimension is reduced using point convolution, followed by a 3×3 depthwise convolution to aggregate the spatial features. Finally, a point-wise convolution is used to reconstruct the dimension to obtain the fusion key. This fusion key is then used to replace the key in the multi-head self-attention module, completing the construction of the multi-head cross-attention module. The reference for the feature-level fusion module is: LIU R, LING J, ZHANG H. SoftFormer: SAR-optical fusion transformer for urban land use and land cover classification[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2024, 218: 277-93.

[0042] The feature-level fusion module first receives the optical features and SAR features output by the two improved encoders, and then performs layer normalization on the optical features and SAR features respectively. The layer-normalized optical features are input into a multi-head self-attention module for feature transformation to obtain the first optical features, and the layer-normalized SAR features are input into another multi-head self-attention module for feature transformation to obtain the first SAR features. At the same time, the layer-normalized optical features are also input into a multi-head cross-attention module for feature transformation to obtain the second optical features, and the layer-normalized SAR features are also input into another multi-head cross-attention module for feature transformation to obtain the second SAR features. Next, the first optical features and the second optical features are fused to obtain the third optical feature, and the third optical feature is fused with the optical feature output by the improved encoder to obtain the fourth optical feature. At the same time, the first SAR feature and the second SAR feature are fused to obtain the third SAR feature, and the third SAR feature is fused with the SAR feature output by the improved encoder to obtain the fourth SAR feature. The fourth optical feature and the fourth SAR feature are then fused to obtain the fused feature, and the fused feature is then subjected to layer normalization, point convolution, depthwise convolution, and point convolution in sequence to obtain the fused feature. Finally, the fused features obtained by the feature-level fusion module are output to the upsampling network for upsampling to obtain a fused upsampling feature map.

[0043] Among them, the calculation formula of the multi-head attention module is:

[0044] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0045]

[0046] Where Q, K, and V are query vectors, key vectors, and value vectors respectively, and h is the number of heads. i represents the output of the i-th head, W O is the output transformation matrix, W i k , The query, key, and value transformation matrices of the i-th head are respectively, Attention is the attention function calculation, d k is the dimension of the key vector.

[0047] In the present invention, the calculation formula for obtaining the first SAR feature is:

[0048]

[0049] Y SAR1 =MHa self (QSAR 1 , K SAR 1 , V SAR 1 )

[0050] Among them, X SAR is the SAR feature after layer normalization, B is the batch size, L is the sequence length, d is the feature dimension, Q sAR 1 , K SAR 1 、V sAR 1 They are the first SAR query matrix, the first SAR key matrix, and the first SAR value matrix, respectively. They are the first SAR query projection weight matrix, the first SAR key projection weight matrix, the first SAR value projection weight matrix, MHA self (*) is multi-head self-attention, Y SAR1 It is the SAR feature after multi-head self-attention enhancement, that is, the first SAR feature.

[0051] The SAR features after layer normalization are also input into a multi-head cross attention module for feature transformation to obtain the calculation formula of the second SAR feature:

[0052]

[0053] K joint =Conv fusion (Concat(X 光 ,X sAR ))

[0054]

[0055] Y SAR2 =MHA cross (Q SAR 2 ,K joint ,V SAR 2 )

[0056] Among them, X 光 is the optical feature after layer normalization, Concat(*) means connection; Conv fusion (*) indicates the function of the fusion key module; K joint is a fusion bond; Q SAR 2 、V SAR 2 are the second SAR query matrix and the second SAR value matrix respectively, They are the second SAR query projection weight matrix and the second SAR value projection weight matrix, MHA cross (*) is multi-head cross attention, Y SAR2 It is the SAR feature after multi-head cross attention enhancement, that is, the second SAR feature.

[0057] In the present invention, the calculation formula for obtaining the first optical characteristic is:

[0058]

[0059] Y 光1 =MHA self (Q 光 1 , K 光 1 , V 光 1 )

[0060] Among them, X 光 is the optical feature sequence after layer normalization, B is the batch size, L is the sequence length, d is the feature dimension, Q 光 1 , K 光 1 、V 光 1 are respectively the first optical query matrix, the first optical key matrix, and the first optical value matrix, They are respectively the first optical query projection weight matrix, the first optical key projection weight matrix, the first optical value projection matrix, and MGA self (*) is multi-head self-attention, Y 光1 It is the optical feature after multi-head self-attention enhancement, that is, the first optical feature.

[0061] The optical features after layer normalization are also input into a multi-head cross attention module for feature transformation to obtain the calculation formula of the second optical feature:

[0062]

[0063] K joint =Conv fusion (Concat(X 光 ,X SAR ))

[0064]

[0065] Y 光2 =MHA cross (Q 光 ,K joint ,V 光 )

[0066] Among them, Q 光 2 、V 光 2 The second is the optical query matrix and the second optical value matrix, V 光 2 They are the second optical query projection weight matrix and the second optical value projection weight matrix, MHA cross (*) is multi-head cross attention, Y 光2 It is the optical feature enhanced by multi-head cross attention, that is, the second optical feature.

[0067] Then the two upsampled feature maps are input into a decision-level fusion module (Decision level Fusion), where the two upsampled feature maps are optical upsampled feature maps and SAR upsampled feature maps. The decision-level fusion module uses the input dimension as the group number for group convolution to calculate the weight parameter. After that, the optical upsampled feature map and the SAR upsampled feature map are respectively multiplied by the weight parameter to output a fusion result. The fusion result and the fused upsampled feature map are input into another decision-level fusion module. Similarly, the decision-level fusion module also uses the input dimension as the group number for group convolution to calculate the weight parameter. After that, the fusion result and the fused upsampled feature map are respectively multiplied by the weight parameter to output the final result, which is a binary image.

[0068] like Figure 4 As shown in Figure 1, it is a structural diagram of the decision-level fusion module. Regarding the decision-level fusion module fusing two upsampled feature maps, it includes the following steps: first, a decision-level fusion module receives two upsampled feature maps output by two improved U-Net models and splices them to obtain a spliced ​​feature map. Next, the spliced ​​feature map is grouped and convolved. The group convolution result is then mapped using the Sigmoid function to obtain the first attention weight of the group channel, that is, an attention weight corresponding to each of the two upsampled feature maps. The two upsampled feature maps are then multiplied by the corresponding attention weights, and finally the two multiplication results are added to obtain the fusion result of the two upsampled feature maps. Among them, the decision-level fusion module and the feature-level fusion module reference are the same.

[0069] Regarding the decision-level fusion module fusing the fusion result of the two up-sampled feature maps with the fused up-sampled feature map, it includes the following steps: first, the decision-level fusion module splices the fusion result of the two up-sampled feature maps with the fused up-sampled feature map to obtain a spliced ​​fused feature map, then performs group convolution on the spliced ​​fused feature map, and then uses the Sigmoid function to map the group convolution result to obtain the second attention weight of the group channel, that is, obtains an attention weight corresponding to each of the fusion result and the fused up-sampled feature map; then, the fusion result and the fused up-sampled feature map are multiplied by the corresponding attention weights respectively, and finally the two multiplication results are added to obtain a binary image.

[0070] In one embodiment of the present invention, the ASPP in the improved U-Net model adopts a layered strategy, with the dilated convolution parameters of the first layer set to 1, 3, and 6, the second layer to 6, 12, and 18, and the third layer to 12, 24, and 36. The channel attention parameter in the CBAM has a channel reduction ratio of eduction_ratio = 8, and the spatial attention convolution kernel size is 7. The depthwise separable convolution kernel size of the MLP is 3. In the feature-level fusion module, the embedding dimension embed_dim is 1024, the number of attention heads num_heads is 8, the random dropout rate of the attention weight matrix attn_drop is 0.1, the random dropout rate of the feedforward neurons is dropout 0.5, the kilogram network hidden layer dimension expansion multiple mlp_ratio is 4.0, and in the decision-level fusion module, the number of optical image channels opt_chans is 3, the number of SAR image channels sar_chans is 3, and the number of categories n_classes is 1. This decision-level fusion module is deployed in the Pytorch framework.

[0071] Step 5: Deep learning extraction model training and prediction;

[0072] First, set the parameters of the deep learning extraction model training code train.py. The opt_dir, sar_dir, and mask_dir parameters correspond to the paths of the annotated optical image, annotated SAR image, and label files in the river ice training dataset, respectively. Set the deep learning extraction model save path (dir_checkpoint) and the deep learning extraction model accuracy evaluation information save path (output_dir) in the code. Then set the multi-threading parameters (num_workers), number of iterations (epochs), batch size (batch-size), learning rate (learning-rate), and validation set ratio (validation). After all settings are completed, use the binary cross entropy loss (BCEWithLogitsLoss), Adam optimizer, and cosine annealing scheduler (CosineAnnealingWarmRestarts) to train the deep learning extraction model. Then set the parameters of the prediction code predict.py. The s1_folder, s2_folder, model, and output parameters correspond to the SAR image path, optical image path, deep learning extraction model weight path to be used, and output path of the prediction results in the test dataset, respectively. After setting, run the code to generate the prediction results of the test dataset.

[0073] Among them, the formula of the binary cross entropy loss function is:

[0074]

[0075] in, represents the binary cross entropy loss function; y is the actual label, is the probability predicted by the model.

[0076] Step 6: Organize the prediction results

[0077] Because the prediction results of the test dataset are all in 1024×1024 TIFF format, the 1024×1024 prediction results (binary images) were merged using the Merge Rasters tool in ArcGIS Pro or QGIS. The raster to vector tool was then used to extract the rasters with a median value of 1 from the binary images into vector data. Finally, based on the distribution vector data of rivers, lakes, glaciers, and other areas, misidentified non-river ice information was removed to form complete river ice distribution data for the target area. High-precision, high-temporal and high-resolution river ice distribution data can provide data support for research on river ice melt processes, river ice disaster prevention and control, and ice flood risk assessment.

[0078] In a specific embodiment of the present invention, the satellite data used in this example is Sentinel-1 and Sentinel-2 data from a mountainous area between January 1, 2024, and April 1, 2024. These data are provided free of charge by the United States Geological Survey (USGS) (website: http: / / glovis.usgs.gov / ; datasets can also be directly accessed on the remote sensing data processing cloud platform). The above-mentioned images can completely cover the mountainous area. Sentinel-1 data has a spatial resolution of 10 meters and a minimum revisit period of 6 days. Sentinel-2 data has spatial resolutions of 10 meters, 20 meters, and 60 meters in the visible, near-infrared, and short-wave infrared spectrum, respectively. Data from the simultaneous operation of two satellites can have a revisit period of up to 5 days. This can be used to extract the distribution range of river ice in the region, with both temporal and spatial resolutions being relatively suitable.

[0079] The specific implementation steps of the river ice extraction method based on deep learning of optical and SAR fusion images of the present invention include the following contents.

[0080] Step 1: On a remote sensing data processing cloud platform, we selected Sentinel-1 and Sentinel-2 satellite remote sensing images and SRTM DEM 30M data from a mountainous area between January 1, 2024, and April 1, 2024. The datasets provided by the cloud platform had undergone basic preprocessing and been converted into surface reflectance data. We selected the VH band of the Sentinel-1 satellite data and the B11, B8, B4, and B3 bands of the Sentinel-2 satellite data. We first resampled the B11, B8, B4, B3, DEM, and VH bands to 10 meters. We then calculated the corresponding NDSI band for the B3 and B11 bands using the NDSI formula. We then defined the B11, B8, B4, B3, DEM, VH, and NDSI bands in the EPSG:4326 geographic coordinate system. We then used this DEM data to perform terrain correction on the VH band of the Sentinel-1 satellite data. The B8, B4, and B3 bands are superimposed together and output as optical images, and the NDSI, DEM, and VH bands are superimposed together and output as SAR images. In order to reduce the impact of river ice information caused by the phase difference between the two satellites, the pixel mean synthesis image method is adopted. The mean of all images is calculated and synthesized, and then downloaded as TIFF format images to the local computer.

[0081] Step 2: Open the downloaded optical image and SAR image in ArcGIS Pro, draw the river ice distribution range vector on the image through visual interpretation or manual correction of the existing river ice dataset, complete the annotation of the river ice distribution range, and then use the tool "Export training dataset for deep learning", set the slice and block size to 1024, the stride to 768, the reference system to map space, and the metadata format to classified slices to produce optical and SAR image training datasets respectively. The generated files contain the image folder images and the label folder labels. Put the three folders of images, label in the optical image folder, and image in the SAR image folder in the same folder. Since the optical image and SAR image have been geo-referenced, only one copy of the label folder needs to be retained. The three folders are placed in the same folder to form a training dataset. The images in the three folders must be named the same and correspond one to one. This training set has a total of 1,000 groups of images.

[0082] Step 3: Select the region to be predicted and download the optical and SAR images to your local computer using the same preprocessing method as in Step 1. Use the image cropping code to crop each optical and SAR image to 1024×1024 size. Name each file the same and create a one-to-one correspondence between them. Place both folders in the same folder to create the test dataset. The training and test datasets must be in exactly the same format.

[0083] Step 4: Set the parameters of the model training code train.py. The opt_dir, sar_dir, and mask_dir parameters correspond to the paths of the optical image, SAR image, and label files in the training dataset, respectively. Set the model save path (dir_checkpoint) and the model accuracy evaluation information save path (output_dir) in the code. Then set the multi-threading parameter num_workers to 4, the number of epochs to 400, the batch size to 1, the learning rate to 0.0001, and the validation set ratio to 30.0. The num_workers and batch-size can be set according to the performance of your computer. After all settings are completed, train the model.

[0084] Step 5: Predict the test set (397 images) based on the trained best model (best_model.pth). Modify the path parameters in the prediction code predict.py, including the optical image to be predicted and its corresponding SAR image path, the model weight input path, and the prediction result output path. Then run predict.py to obtain the prediction results. Based on the prediction results and their corresponding river ice ground truth data (labels), the model prediction results were verified using six different accuracy evaluation metrics. The results showed that the model achieved an overall accuracy of 0.9975, precision of 0.8822, recall of 0.9023, F1 score of 0.8921, intersection over union (IOU) of 0.8053, and kappa of 0.8909 for river ice recognition in alpine mountainous areas.

[0085] Step 6: Open the river ice prediction results in QGIS software and merge all the prediction results into a raster image using the raster merge tool. Convert the portion of the merged raster with a median value of 1 into vector data (.shp) using the code raster_vectorize_py to obtain the final river ice distribution range data for the target area. Then, based on the Third Pole 1:1 million water system dataset, delete misjudged non-river ice information, such as lake ice and glaciers, to ensure the accuracy of the river ice distribution range data.

[0086] Figure 6 The result of the prediction of the test set is displayed. Based on the trained model, the image prediction of the test set is performed to obtain the river ice recognition result corresponding to the image of the test set, where the label is the true value of the river ice. Figure 6 The prediction results show that the proposed method can accurately identify river ice of different sizes in complex scenes in high-altitude mountainous areas. At the same time, the results also prove that the method is highly robust to factors such as terrain undulation, cloud cover and snow interference.

[0087] Figure 7 This is the result of extracting a large range of river ice in a certain mountain area and a local magnified image. This result is generated by outputting a binary image from the trained river ice recognition model and then post-processing (image stitching, raster-to-vector conversion). Figure 7 As shown in the results, this method shows high accuracy in large-scale river ice extraction tasks, can effectively maintain the detailed integrity and spatial continuity of river ice morphology, and can provide reliable technical support for refined river ice dynamic mapping.

[0088] The present invention discloses a deep learning method for extracting river ice based on optical and SAR fusion images, which can effectively overcome the interference of clouds, snow, and terrain, and achieve high-precision extraction of river ice range in high-altitude mountainous areas. The present invention selects optical images and SAR images with river ice distribution, then preprocesses the optical images and SAR images, and produces river ice training data sets and test data sets. Then, the parameters of the deep learning extraction model are set, and the training data set is input into the deep learning extraction model for training. After that, the images in the test data set are predicted based on the trained deep learning extraction model to obtain prediction results. Finally, the 1024×1024 prediction results are merged using the merge raster tool in ArcGIS Pro or QGIS software, and the raster to vector tool is used to extract the raster with a value of 1 as vector data. Finally, based on the distribution data of rivers, lakes, glaciers, etc., the misjudged non-river ice information is removed to form complete river ice distribution range data for the target area. The high-precision, high temporal and spatial resolution river ice distribution range data obtained by the method of the present invention can provide data support for river ice melting process research, river ice disaster prevention and control, and ice flood risk assessment.

[0089] This invention breaks through the limitations of a single data source and the data stacking paradigm, embeds multiple attention mechanisms to extract data features, and adopts a multi-level feature fusion strategy to integrate Sentinel-1, Sentinel-2, and DEM data features. This allows for the full extraction of feature information and the complementary advantages of different features. This allows for high-precision and reliable extraction of river ice distribution in alpine mountainous environments, enabling refined river ice mapping. The complex parameters of the deep learning extraction model of the present invention require relatively high hardware configuration (GPU memory of 12GB or more). For better extraction results, a river ice training dataset with a large number of training images, a full range of types, and high label accuracy is also required.

[0090] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they are not to be construed as limiting the scope of the present invention. Persons skilled in the art will readily appreciate that variations and modifications may be made without departing from the spirit of the present invention, and all such variations and modifications fall within the scope of protection of the present invention.

Claims

1. A deep learning method for extracting river ice based on optical and SAR fusion images, characterized by: The following steps are involved: Obtain optical and SAR images of the same area containing river ice distribution and preprocess them. Then, label the river ice areas in the preprocessed optical and SAR images to construct training datasets. Build a deep learning extraction model and train it using the training datasets. Obtain optical and SAR images of the river ice distribution area to be extracted, use the trained deep learning extraction model to obtain binary images, convert the binary images into vector data, remove the misjudged river ice areas, obtain the river ice area in the area, and complete the river ice extraction; Wherein, the deep learning extraction model includes: Two improved U-Net models are used to extract feature information from annotated optical and SAR images, respectively, to obtain two upsampled feature maps. Each improved U-Net model includes a decoder and an encoder. The encoder consists of multiple convolutional blocks connected in sequence for downsampling. Each convolutional block consists of a maximum pooling layer, a dilated spatial pyramid pooling layer, a convolutional attention module, and a multi-layer perceptron. The encoder of one improved U-Net model outputs optical features, and the other outputs SAR features. The fusion and sampling module is used to fuse the outputs of the two encoders to obtain fused features, and upsample the fused features to obtain a fused upsampled feature map; Two decision-level fusion modules, one for fusing the two upsampled feature maps, and the other for fusing the fusion result of the two upsampled feature maps with the fused upsampled feature map to obtain a binary image.

2. The method for extracting river ice based on deep learning of optical and SAR fusion images according to claim 1 is characterized in that: The spatial resolution of the optical image is less than or equal to 30 meters, and the optical image is collected by the Sentinel-2 satellite. The optical image consists of a green spectrum band, a red spectrum band, a near-infrared spectrum band, and a short-wave infrared spectrum band. The spatial resolution of the SAR image is less than or equal to 30 meters, the temporal resolution is greater than 10 days, and the SAR image is collected by the Sentinel-1 satellite. Moreover, the time interval between the collection of the optical image and the SAR image is less than 10 days.

3. The method for extracting river ice based on deep learning of optical and SAR fusion images according to claim 2 is characterized in that: Preprocessing the optical image and the SAR image includes: Obtain DEM data for the same area as the optical image and visualize the DEM data to obtain a DEM image. Resample the VH polarization bands of the optical image, DEM image, and SAR image. The spatial resolution of the VH polarization bands of the resampled optical image, DEM image, and SAR image is 10 meters. The resampled optical image is processed based on the normalized snow cover index algorithm to obtain the NDSI image; the geographic coordinate system of the resampled optical image, DEM image, VH polarization band of the SAR image, and the NDSI image is unified to EPSG:4326; the VH polarization band of the SAR image in the unified geographic coordinate system is then topographically corrected using the DEM image in the unified geographic coordinate system to obtain the corrected SAR image; and the NDSI image in the unified geographic coordinate system, the DEM image, and the corrected SAR image are then synthesized into a new image; Combining the near-infrared spectrum band, the red spectrum band, and the green spectrum band in the optical image after unifying the geographic coordinate system into a new optical image, and then performing multi-view image synthesis into a single-view average image processing on the new optical image to obtain a processed image; Finally, the new image and the processed image are subjected to image size consistency adjustment to complete the preprocessing, wherein the new image after image size consistency adjustment is the preprocessed SAR image, and the processed image after image size consistency adjustment is the preprocessed optical image.

4. The method for extracting river ice based on deep learning of optical and SAR fusion images according to claim 1 is characterized in that: The fusion and sampling module includes a feature-level fusion module and an upsampling network, and the upsampling network has the same decoder structure as the improved U-Net model; The feature-level fusion module is used to fuse the outputs of the two encoders and includes two multi-head self-attention modules and two multi-head cross-attention modules; The construction method of the multi-head cross-attention module is as follows: the SAR features and optical features output by the two encoders are layer-normalized respectively, and then the layer-normalized SAR features and the layer-normalized optical features are added together. After the addition, point convolution, depth convolution and point convolution are performed in sequence to obtain a fusion key, and then the fusion key is used to replace the Key in the multi-head self-attention module to obtain the multi-head cross-attention module.

5. The method for extracting river ice based on deep learning of optical and SAR fusion images according to claim 4 is characterized in that: The feature-level fusion module first receives the optical features and SAR features output by the two encoders, and then performs layer normalization on the optical features and SAR features respectively. The layer-normalized optical features are input into a multi-head self-attention module for feature transformation to obtain the first optical features, and the layer-normalized SAR features are input into another multi-head self-attention module for feature transformation to obtain the first SAR features. At the same time, the layer-normalized optical features are also input into a multi-head cross-attention module for feature transformation to obtain the second optical features, and the layer-normalized SAR features are also input into another multi-head cross-attention module for feature transformation to obtain the second SAR features. Next, the first optical features and the second optical features are fused to obtain the third optical feature, and the third optical feature is fused with the optical feature output by the encoder to obtain the fourth optical feature. At the same time, the first SAR feature and the second SAR feature are fused to obtain the third SAR feature, and the third SAR feature is fused with the SAR feature output by the encoder to obtain the fourth SAR feature. The fourth optical feature and the fourth SAR feature are then fused to obtain the fused feature, and the fused feature is then subjected to layer normalization, point convolution, depth convolution, and point convolution in sequence to obtain the fused feature.

6. The method for extracting river ice based on deep learning of optical and SAR fusion images according to claim 5 is characterized in that: The decision-level fusion module fuses the two upsampled feature maps, including: The decision-level fusion module receives two upsampled feature maps and concatenates them to obtain a concatenated feature map. Next, the concatenated feature map is grouped and convolved. The sigmoid function is used to map the group convolution result to obtain the first attention weight of the group channel, that is, an attention weight corresponding to each of the two upsampled feature maps. The two upsampled feature maps are then multiplied by the corresponding attention weights, and the two multiplication results are added to obtain the fusion result of the two upsampled feature maps. The decision-level fusion module fuses the fusion results of the two upsampled feature maps with the fused upsampled feature map, including: The decision-level fusion module splices the fusion result of the two upsampled feature maps with the fused upsampled feature map to obtain a spliced ​​fused feature map. Next, the spliced ​​fused feature map is group convolved, and then the group convolution result is mapped using the Sigmoid function to obtain the second attention weight of the group channel, that is, an attention weight corresponding to the fusion result and the fused upsampled feature map respectively is obtained; then the fusion result and the fused upsampled feature map are multiplied by the corresponding attention weights respectively, and finally the two multiplication results are added to obtain a binary image.

7. The method for extracting river ice based on deep learning of optical and SAR fusion images according to claim 1, characterized in that: The deep learning extraction model is trained using the binary cross entropy loss function as the loss function; and during the training process of the deep learning extraction model, the Adam optimizer and the cosine annealing learning rate scheduler are also used.

8. The method for extracting river ice based on deep learning of optical and SAR fusion images according to claim 1, characterized in that: The removal of misjudged river ice areas includes: Obtain distribution vector data of rivers, lakes, and glaciers at the location of the image of the river ice distribution area to be extracted, and remove areas in the vector data that are mistakenly identified as river ice based on the obtained distribution vector data of rivers, lakes, and glaciers.

Images