Vision transformer-based high-resolution remote sensing image water body extraction method and device

Abstract

The application discloses a high-resolution remote sensing image water body extraction method and equipment based on a Vision Transformer, first acquires multispectral remote sensing images and carries out pretreatment; visual water body interpretation is carried out on the pretreated images, water body label true values corresponding to the images are made, and a data set for deep learning is made; then, a spatial detail information reservation module, a multi-scale linear multi-head self-attention mechanism and a global multi-layer perceptron are designed, a Vision Transformer model is combined, and an MSMViT water body extraction method is constructed; the MSMViT model is trained and adjusted through a water body sample data set, the effect of the water body extraction method is checked on a test data set, and water body extraction of multispectral remote sensing images is completed. Global features and detail features of high-resolution remote sensing images can be considered, and better water body extraction effect is achieved.

Description

TECHNICAL FIELD

[0001] The application relates to a high-resolution remote sensing image water body extraction method, and aims at the deficiency of a Vision Transformer (ViT) network in water body extraction. BACKGROUND

[0002] Water is an important resource and plays a key role in human production and life. Therefore, understanding the distribution of water resources has very important guiding significance for the rational use and protection of water resources. Remote sensing technology has macroscopic, timely and real nature, and can obtain large-scale spectral image information of the earth's surface. Especially the data obtained by various types of remote sensing means represented by satellite remote sensing have greatly improved spatial resolution and spectral resolution. Using satellite remote sensing data to extract surface water body information has become one of the important contents of remote sensing application.

[0003] High-resolution remote sensing image water body extraction mainly includes single-band threshold method, multi-band spectral relationship method and water body index method. With the development of deep learning methods, great progress has been made in remote sensing image water body information extraction. VGG, ResNet and other classification networks and FCN and other segmentation networks have achieved good results in water body information extraction. Vision Transformer can segment remote sensing images into multiple patches, model the relationship between patches through a Transformer encoder, and ViT can capture global context information in remote sensing images, so it can improve the recognition ability of large-area water bodies. However, ViT needs a large amount of data for pre-training, and the computational complexity is high, which leads to the demand for a large amount of memory of ViT, which seriously affects its application potential. The improved Swin Transformer adopts a hierarchical structure and designs a multi-head self-attention mechanism to improve efficiency, but its complexity will increase quadratically with the increase of the window. ViT mainly focuses on capturing global context information, but ignores spatial detail features, so how to maintain the global information extraction ability of ViT and improve its spatial detail feature extraction ability without increasing the computational load of the model is very important for high-resolution remote sensing image water body extraction. SUMMARY

[0004] Based on the deficiency of the prior art, the application designs a high-resolution remote sensing image water body extraction method based on Vision Transformer, which can balance the global features and detail features of high-resolution remote sensing images and achieve better water body extraction effect.

[0005] The application designs a high-resolution remote sensing image water body extraction method based on a Vision Transformer, which comprises the following steps:

[0006] S1, obtaining high-resolution remote sensing images and performing preprocessing;

[0007] S2, visually interpreting the water body of the preprocessed high-resolution remote sensing image, and drawing the image water body boundary for making a data set for training a deep learning model;

[0008] S3, constructing an MSMViT network model, which comprises an encoder and a decoder; wherein the encoder comprises a detail information reservation module and a water body feature extraction module; the multi-scale water body feature extraction module has multiple parts, and the jump connection part contains a dynamic information fusion module;

[0009] The detail information reservation module first acquires an image feature map, and then processes the image feature map to obtain spatial detail information;

[0010] The multi-scale water body feature extraction module comprises a mixW-Block module, which is composed of an LMSMHSA, a convolution-based global multi-layer perceptron CGMLP, a normalization layer and a residual connection, wherein the LMSMHSA is improved from the multi-head self-attention mechanism MHSA of an ordinary ViT, and the improvement includes replacing the full connection layer in the ordinary ViT with a convolution, and replacing the Softmax function in the ordinary ViT with a Taylor formula;

[0011] S4, training the constructed MSMViT network model using the data set in step S2, and using the trained MSMViT network model for water body extraction.

[0012] Preferably, the preprocessing in S1 comprises radiation correction, geometric correction, fusion, inlaying and cropping of the high-resolution remote sensing image.

[0013] Preferably, the water body on the multi-spectral image is visually interpreted in S2, and the image water body boundary is drawn using Labelme software and converted into water body true value label data.

[0014] Preferably, in S2, after reading the high-resolution remote sensing image data and the corresponding water body label data, the sliding window method is used to cut the sample data into the same size, and the entire data set is divided into a training set, a validation set and a test set according to the proportion.

[0015] Preferably, the spatial detail information reservation module comprises two parts A and B, the A part provides a feature map for the main structure, which mainly comprises two down-sampling modules and a fusion module, and the B part is a spatial detail information extraction module.

[0016] Further, the spatial detail information acquisition module is implemented by multiple layers of modules, and each layer of module processing includes convolution, batch normalization and activation function, and each layer obtains detailed spatial information through stepwise down-sampling.

[0017] Preferably, the encoder further comprises a patch merging module for adjusting the size of the feature map, and the specific process comprises: respectively using convolution and pooling operation to down-sample the feature map to half of the original feature map, and expanding the dimension of the feature map to twice the original, and then splicing the feature maps obtained by the two operations.

[0018] Preferably, the multi-scale water feature extraction module has 4.

[0019] Preferably, the decoder uses the basic self-attention mechanism to recover the resolution of the feature map by layer-by-layer up-sampling.

[0020] Based on the same inventive concept, the present application also designs an electronic device comprising:

[0021] one or more processors;

[0022] a storage device for storing one or more programs;

[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the high-resolution remote sensing image water extraction method based on the Vision Transformer.

[0024] Based on the same inventive concept, the present application also designs a computer readable medium having a computer program stored thereon, and the program is executed by a processor to implement the high-resolution remote sensing image water extraction method based on the Vision Transformer.

[0025] The special features of the MSMViT network model proposed by the present application include the following points:

[0026] 1) In the MSMViT network model, a spatial detail information reservation module is designed, which not only provides feature maps for the main structure, but also provides more detailed spatial detail information for the network model, and strengthens the ability of the model to extract detailed features.

[0027] 2) In the MSMViT network model, a multi-scale module mixW-Block is designed to extract water feature information in the image. The improvement point of the module is the linear multi-scale multi-head self-attention module LMSMHSA and the convolution-based global multi-layer perceptron CGMLP. LMSMHSA can obtain more rich spatial information, avoid the lack of feature diversity caused by the single feature head of the self-attention mechanism, and optimize the calculation method of the multi-head attention mechanism, effectively reducing the calculation amount of the self-attention, which is more suitable for large-scale high-resolution remote sensing image processing; CGMLP strengthens the cross-window interaction, further enhances the ability of the model to capture global context information.

[0028] 3) The MSMViT network model can retain spatial detail information while considering the spatial position relationship of water in the remote sensing image, dynamically balances the local spatial detail information and global context information in the segmentation process, and improves the accuracy of water body segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The figure is the overall structure diagram of the MSMViT network in the application.

[0030] Figure 2 The figure is the structure diagram of the spatial detail information retention module in the application.

[0031] Figure 3 The figure is the structure diagram of the mixW-Block module in the application.

[0032] Figure 4 The figure is the structure diagram of the global multi-layer perceptron in the application.

[0033] Figure 5 The figure is an example of image data, corresponding label and water extraction result of each method. DETAILED DESCRIPTION

[0034] In order to make the technical means, creative features, work flow, use method, purpose and effect of the application easy to understand, the application will be further described below with reference to the drawings.

[0035] Embodiment one

[0036] A high-resolution remote sensing image water extraction method based on Vision Transformer, comprising the following steps:

[0037] S1, obtaining high-resolution remote sensing images of the research area, performing radiation correction, geometric correction, image fusion, inlaying, cutting and other pretreatments on the high-resolution remote sensing images;

[0038] S2, visual interpretation of the pretreated high-resolution remote sensing image, using Labelme software to outline the water mask, and converting the mask data into water true value label data, registering the image data and the corresponding water label data, using the sliding window method to cut the image and label data into 512*512 size, obtaining many pairs of image and label data blocks, and dividing the whole data set into training set, validation set and test set according to the proportion of 7:2:1, wherein the training set is used to train the model, that is, the model learns the characteristics and rules in the data through the data of the training set; the validation set is used to adjust the model hyperparameters and evaluate the model performance to avoid model overfitting; the test set is used to evaluate the performance of the model finally, simulating the performance of the model in the real scene.

[0039] S3, the MSMViT network model is constructed, the overall structure is as shown in Figure 1 , the main part still follows the network structure of encoder and decoder. The encoder part is composed of 1 spatial detail information reservation module mix-stem and 4 multi-scale water feature extraction modules w-mix stage 1、2、3、4 ; the middle jump connection part contains three dynamic information fusion modules WFstage 1、2、3 ; the decoder part is composed of 4 water feature recovery modules w-mix stage 5、6、7、8 and a fusion module merge; finally, the water extraction result is output through the head part Head;

[0040] The role of the encoder is to extract image features and capture context information. In order to not lose spatial detail information in the network downsampling process, the application designs a spatial detail information reservation module mix-stem at the front end of the encoder, and the specific structure is as shown in Figure 2 , the spatial detail information reservation module is composed of A and B two parts, A mainly provides feature maps for the main structure through downsampling, which is composed of two downsampling modules and a fusion module, which is composed of convolution kernel 4*4 step 4, convolution kernel 1*1 step 1 and an activation layer in series; B mainly provides more detailed spatial detail information for the network model, which is composed of 6 parts, each part is composed of convolution, batch normalization and activation function, and detailed spatial information is obtained by stepwise downsampling. After the spatial detail information reservation module, there are 4 multi-scale water feature extraction modules w-mix stage. The encoder specifically includes:

[0041] 1) patch merging, which mainly adjusts the size of the feature map in the downsampling process, completes the task of downsampling, and includes 3 patch merging layers in the whole encoder, in w-mix stage 2、3、4To complete the three downsampling tasks. First, respectively use convolution and pooling operation to downsample the feature map to half of the original feature map, and expand the dimension of the feature map to 2 times of the original, then splice the feature maps obtained by the two operations, which is beneficial to enhance the foreground information, and finally use the standard deep convolution and residual connection operation to strengthen the adjusted information after fusion.

[0042] 2) mixW-Block module, the core component of extracting features in the encoder, obtaining high-level semantic information at each level, containing mixW-Block module in w-mix stage 1、2、3、4 The mixW-Block contains LMSMHSA, convolution-based global multilayer perceptron CGMLP, normalization layer and residual connection setting, as shown in Figure 3 After the spatial detail information is input into the mixW-Block, the LMSMHSA first calculates the weight information of the feature information to improve the network's information perception ability of global water features, and outputs detailed water feature maps. Compared with the ordinary ViT's multi-head self-attention mechanism MHSA, LMMHSA has certain advantages in calculation method and amount. The ordinary ViT directly maps the input feature map into Q (Query), K (Key), and V (Value), calculates the attention score through QKV, and inputs after weighted summation. LMMHSA uses convolution instead of the full connection layer in ordinary ViT, which is conducive to the preservation of the spatial relationship between features in the image. At the same time, different sizes of convolution layers are constructed to obtain different sizes of feature information, avoiding the information loss problem caused by single scale and enhancing the diversity of information, which is more suitable for extracting cross-scale water features. In addition, the Taylor formula is used to replace the Softmax function in ordinary ViT, and the calculation complexity of LMSMHSA is reduced from N 2 to N. It can be seen that improving the QKV generation module can improve the efficiency and effect of feature extraction. CGMLP uses multi-scale convolution operation to enhance local information fusion, as shown in Figure 4 First, the 1*1 convolution combination module and the 3*3 convolution combination module are connected in parallel and respectively increase the dimension of the feature map by 2 times, and after superposition, the dimension of the feature map becomes 4 times the dimension of the input feature map. Then a 3*3 convolution combination module is performed, and finally the dimension of the feature map is restored to the input dimension through a 1*1 convolution combination module, and a drop layer is used to alleviate the overfitting problem. Therefore, the CGMLP module directly outputs the feature map after strengthening the information interaction of the water feature map and the spatial detail information. The role of the normalization layer is to stabilize the training process and accelerate the convergence of the model; the main role of the residual connection is to relieve gradient disappearance and reduce feature information loss.

[0043] The role of the decoder part is to restore the resolution and fuse the features to generate accurate output. The application uses the basic self-attention mechanism to restore the resolution of the feature map layer by layer, and the whole decoder uses three basic self-attention mechanisms, which fully utilizes the global features and spatial detail features in the decoding process. The skip connection between the encoder and the decoder uses different connection methods according to the characteristics of the feature map. When connecting the global information features and the up-sampled features, the addition connection method is used. When connecting the spatial detail features and the up-sampled features, the application designs a special feature fusion method, which controls the addition of the two feature maps according to a dynamic training weight, so that the fusion of different types of feature maps is more scientific. Finally, the feature map is restored to the original size through the segmentation head to complete the segmentation task.

[0044] S4, training the constructed MSMViT network model using the data set in step S2, and using the trained MSMViT network model for water body extraction.

[0045] Experimental setup

[0046] In order to ensure the repeatability of the experiment and the fairness of the comparative experiment, all the comparative experiments are carried out in the same implementation environment, in which the hardware configuration is: the display uses NVIDIA GeForce RTX 3090 GPU, and the memory size is 64GB; the software configuration is: the network model is built based on pytorch1.10 framework under python3.9 version, and the gradient descent optimization method used is AdamW optimizer; the loss function uses binary cross entropy loss function; the enhancement methods for the training set data in the training process include flipping, random cropping, random cropping combination and mirroring.

[0047] Case display

[0048] The data used in the experiment is a self-made data set and an open source GID data set. The self-made water body data set selects the test area in Chongqing and Yueyang cities in China, including urban and rural areas, and uses high-resolution Gaofen 2 satellite image data with a resolution of 1m. The water body vector data is obtained by visual interpretation and converted into a binary label. A total of 4910 data sets with a size of 512*512 are obtained by image cropping, and the data is divided into training set, validation set and test set according to the ratio of 7:2:1. The GID data set is a high-resolution remote sensing image provided by Gaofen 2, and the image resolution reaches 1m after image fusion. The image includes a total of 150 remote sensing images, and the original data set is a 5-class ground object classification data. In this experiment, only the water body information is retained and set as the classification information, and the others are set as background information. At the same time, the missing water bodies in the label are supplemented accordingly.

[0049] The five evaluation indexes, i.e., overall accuracy (OA), precision (P), recall (R), F1-score (F1) and intersection over union (IoU), can effectively evaluate the comprehensive extraction capability of the semantic segmentation network. The overall accuracy (OA) represents the ratio of the correct water body pixel value extracted by the network model to the total pixel value; the precision (P) represents the ratio of the correct water body pixel value extracted by the network model to all water body extraction pixel values; the recall (R) represents the ratio of the water body pixel value extracted by the network model to the water body pixel value in the real label; the F1-score (F1) represents the harmonic mean of the precision and the recall; and the intersection over union (IoU) represents the ratio of the intersection of the real label water body pixel and the predicted water body pixel to the union. In order to verify the effectiveness and superiority of the proposed water body extraction method, the present application selects several commonly used and better water body extraction methods and deep learning methods for comparative experiments, and the selected comparison methods include Unet, PSPNet, SegNet, MWEN and ViT. Table 1 is a progress comparison table of the six water body extraction methods, Figure 5 Image data, corresponding labels and water body extraction result samples of each method are shown.

[0050] Table 1. Precision comparison of MSMViT and other methods

[0051]

[0052] Example Two

[0053] Based on the same inventive concept, the present application also provides an electronic device comprising one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in Example One.

[0054] Since the device introduced in Example Two of the present application is an electronic device used to implement the high-resolution remote sensing image water body extraction method based on VisionTransformer of Example One of the present application, the specific structure and modifications of the electronic device can be understood by those skilled in the art based on the method introduced in Example One of the present application, and therefore will not be described here. Any electronic device used in the method of Example One of the present application belongs to the scope of protection of the present application.

[0055] Example Three

[0056] Based on the same inventive concept, the present application also provides a computer readable medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method described in Example One.

[0057] Since the equipment introduced in the embodiment three of the present application is the computer readable medium for implementing the high-resolution remote sensing image water body extraction method based on the VisionTransformer of the embodiment one of the present application, therefore, based on the method introduced in the embodiment one of the present application, the person skilled in the art can understand the specific structure and deformation of the electronic equipment, therefore, it will not be repeated here. Any electronic equipment used in the method of the embodiment one of the present application belongs to the scope of protection of the present application.

[0058] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for water body extraction from high-resolution remote sensing images based on Vision Transformer, characterized in that, Includes the following steps: S1, acquire high-resolution remote sensing images and perform preprocessing; S2 performs visual interpretation of water bodies in the preprocessed high-resolution remote sensing images, delineates the water body boundaries in the images, and is used to create a dataset for training deep learning models. S3, construct the MSMViT network model, which includes an encoder and a decoder; the encoder includes a detail information preservation module and a water feature extraction module; there are 4 multi-scale water feature extraction modules, and their skip connection parts include a dynamic information fusion module; The detail information preservation module first obtains the image feature map, and then processes the image feature map to obtain spatial detail information; The detailed information retention module consists of two parts, A and B. Part A provides feature maps for the main structure and mainly consists of two downsampling modules and one fusion module. Part B is the spatial detail information extraction module, which is implemented by a multi-layer module. The processing of each layer module includes convolution, batch normalization and activation function. Each layer obtains detailed spatial information through stepwise downsampling. The multi-scale water feature extraction module includes a mixW-Block module, which consists of LMSMHSA, a convolution-based global multilayer perceptron, a normalization layer, and residual connections. The LMSMHSA is an improvement on the multi-head self-attention mechanism MHSA of ordinary ViT. The specific improvements include replacing the fully connected layers in ordinary ViT with convolution and replacing the Softmax function in ordinary ViT with Taylor formula. S4. The constructed MSMViT network model is trained using the dataset from step S2, and the trained MSMViT network model is used for water body extraction.

2. The method for water body extraction from high-resolution remote sensing images based on Vision Transformer according to claim 1, characterized in that: The preprocessing in S1 includes radiometric correction, geometric correction, image fusion, mosaicking, and cropping of high-resolution remote sensing images.

3. The method for water body extraction from high-resolution remote sensing images based on Vision Transformer according to claim 1, characterized in that: In step S2, the water bodies in the high-resolution image are visually interpreted, the boundaries of the water bodies in the image are delineated using Labelme software, and then converted into ground truth label data for the water bodies.

4. The method for extracting water bodies from high-resolution remote sensing images based on Vision Transformer according to claim 1, characterized in that: In step S2, after reading the high-resolution remote sensing image data and the corresponding water body label data, the data is cut into samples of the same size using the sliding window method, and the entire dataset is divided into training set, validation set and test set according to the proportion.

5. The method for extracting water bodies from high-resolution remote sensing images based on Vision Transformer according to claim 1, characterized in that: The encoder also includes a patch merging module, which is used by the water feature extraction module to adjust the size of the feature map. The specific process includes: using convolution and pooling operations to downsample the feature map to half of the original feature map, and expanding the dimension of the feature map to twice the original, and then concatenating the feature maps obtained from the two operations.

6. The method for extracting water bodies from high-resolution remote sensing images based on Vision Transformer according to claim 1, characterized in that: The decoder utilizes a basic self-attention mechanism to recover the resolution of the feature map through layer-by-layer upsampling.

7. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the high-resolution remote sensing image water body extraction method as described in any one of claims 1-6.

8. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the high-resolution remote sensing image water body extraction method as described in any one of claims 1-6.

Images