A satellite image super-resolution method based on deep learning

Abstract

The invention discloses a satellite image super-resolution method based on deep learning, which belongs to the field of image processing. The present invention builds an extensible deep super-resolution network, performs 4 times super-resolution on remote sensing satellite images, and can well reconstruct the characteristics of transmission line tower insulators, with the characteristics of high fidelity and fast processing speed; in order to improve the ability of the super-resolution network to learn the characteristics of remote sensing satellite images, further, a data augmentation method is adopted, and the weight normalization layer replaces the batch normalization layer to more accurately learn the characteristics of remote sensing satellite images; further, a jump-connected pixel reorganization and amplification module is introduced, which speeds up the calculation speed without a series of additional convolutions, and is well applied to power grid remote sensing image processing. It can be used for subsequent transmission line inspection, transmission line tower identification, insulator detection and other purposes.

Description

Technical Field

[0001] The invention belongs to the field of satellite image processing, in particular to a super-resolution method for satellite images. Background Art

[0002] In order to identify and monitor the faults of transmission line insulators more accurately and extensively, Yunnan Power Grid has carried out a number of key technologies for the "sky-ground coordination" inspection of power grids based on satellite technology. With the development of key technologies for power grid inspections based on satellite technology, the power grid department has accumulated a large amount of remote sensing satellite image data, drone inspection data and transmission line insulator ledger data. However, in the current transmission line insulator fault monitoring process, only drone inspection data and part of the transmission line insulator ledger data are used, and high-resolution remote sensing satellite image data is not used. In the identification and monitoring of transmission line insulators, the traditional method is to extract insulator features from drone inspection data, generally only considering features such as shape and color. There are also methods based on deep learning, which use target detection networks to automatically learn insulator features. However, in remote sensing satellite images, these methods are not applicable because the resolution of remote sensing satellite images is limited and cannot accurately reflect the characteristics of transmission line insulators. Summary of the invention

[0003] The purpose of the present invention is to provide a satellite image super-resolution method based on deep learning, which is used to solve the problem of fine feature extraction of transmission line insulator identification based on remote sensing satellite images. By increasing the resolution by 4 times, the transmission line insulator features can be reconstructed more clearly. The present invention solves the problem of insufficient resolution of transmission line insulator identification based on remote sensing satellite images, and solves the problem of difficulty in extracting transmission line insulator features based on remote sensing satellite images.

[0004] To achieve the above object, the technical solution of the present invention is: a satellite image super-resolution method based on deep learning, the method comprising:

[0005] Step 1: Obtain satellite images;

[0006] Step 2: Orthorectify the satellite image using 90m×90m elevation data;

[0007] Step 3: Perform multispectral and panchromatic band fusion on the orthorectified remote sensing satellite image. The fused image is a four-channel image of red band, green band, blue band and near infrared band;

[0008] Step 4: Use the bicubic interpolation downsampling algorithm to downsample the orthorectified remote sensing satellite image by 4 times, and then perform data augmentation by horizontal flipping and vertical flipping. The original multispectral image is used as the true value of the label to obtain the training image pair;

[0009] Step 5: Build a super-resolution network;

[0010] The super-resolution network has 4 channels, corresponding to the red band, green band, blue band and near-infrared band. Each channel is divided into two branches after input, wherein the main branch passes through the first convolution layer, ten residual network modules, the fourth convolution layer, and the pixel reorganization upsampling 4 times layer I in sequence; the branch first passes through the fifth convolution layer, and then is divided into two branches, wherein one branch passes through the sixth convolution layer and the pixel reorganization upsampling 4 times layer II in sequence, and the other branch passes through the seventh convolution layer, the eighth convolution layer and the pixel reorganization upsampling 4 times layer III in sequence; the output of the pixel reorganization upsampling 4 times layer I, the pixel reorganization upsampling 4 times layer II, and the pixel reorganization upsampling 4 times layer III are added together as the output of the entire super-resolution network; the residual network module includes: the first convolution layer, the weight normalization layer, the linear rectification layer, the second convolution layer and the weight normalization layer;

[0011] Step 6: Use the training image pairs obtained in step 4 as the input of the super-resolution network in step 5 to train a super-resolution network;

[0012] Step 7: When super-resolving the super-resolved image, firstly, the image is segmented and cropped, then each cropped regional image is super-resolved, and finally the super-resolved regional images are spliced;

[0013] Step 8: Perform three-channel false color and 1% linear stretching on the red band, near-infrared band and blue band of the super-resolution image, and output it.

[0014] Furthermore, in the residual network module of step 5, the convolution kernel of the first convolution layer has a side length of 3, the number of output channels is 192, and the convolution step length is 1; the convolution kernel of the second convolution layer has a side length of 3, the number of output channels is 192, and the convolution step length is 1; the convolution kernel of the third convolution layer of the main branch has a side length of 3, the output channels is 32, the convolution step length is 1, and the convolution kernel of the fourth convolution layer has a side length of 3, the output channels is 12, and the convolution step length is 1. The convolution kernel of the fifth convolution layer of the first side branch has a side length of 3, the number of output channels is 96, the convolution step length is 1, the convolution kernel of the sixth convolution layer has a side length of 5, the number of output channels is 12, and the convolution step length is 1. The convolution kernel of the seventh convolution layer of the second side branch has a side length of 3, the number of output channels is 32, the convolution step length is 1, and the convolution kernel of the eighth convolution layer has a side length of 5, the number of output channels is 12, and the convolution step length is 1;

[0015] Furthermore, in step 6, the number of training rounds is set to 100, the initial learning rate is set to 0.0002, and the learning rate is decayed by 0.5 at the 25000th, 50000th, and 75000th single cycles, respectively, and the batch size is 16.

[0016] The present invention adopts a multi-source remote sensing satellite image super-resolution method based on deep learning, builds an extensible deep super-resolution network, and performs 4-fold super-resolution on SuperView-1 and WorldView-3 remote sensing satellite images, which can well reconstruct the insulator features of transmission line towers, and has the advantages of high fidelity and fast processing speed; the present invention also performs data augmentation on limited remote sensing satellite images, so that the remote sensing satellite image features are more and wider, and the insulator feature learning is better. In order to avoid the effect of grayscale stretching after image super-resolution, a weight normalization layer is used after the convolution layer, which plays the role of accelerating convergence, preventing gradient disappearance and explosion, and preventing overfitting; further, a jump-connected pixel reorganization upsampling 4-fold layer is introduced, without a series of additional convolutions, thereby reducing the amount of calculation and speeding up the calculation speed. The present invention proposes a multi-source remote sensing satellite image super-resolution method based on deep learning. Starting from the perspective of deep learning, the method designs a lightweight super-resolution network, and according to the characteristics of two types of remote sensing satellite image data, a set of automated super-resolution processes are studied and formulated, and a high-fidelity remote sensing satellite image super-resolution model is constructed. It has been applied to remote sensing satellite image processing of Yunnan power grid and can be used for subsequent transmission line inspection, transmission line tower identification and insulator detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the remote sensing satellite image insulator super-resolution method based on deep learning of the present invention;

[0018] Figure 2 is a super-resolution network structure diagram of the present invention;

[0019] Figure 3 It is a super-resolution result image of the tower part in the SuperView-1 image of the present invention;

[0020] Figure 4 It is a super-resolution result image of the tower part in the WordView-3 image of the present invention;

[0021] Figure 5 This is a large-scale super-resolution result diagram of the SuperView-1 image of the present invention. DETAILED DESCRIPTION

[0022] The present invention is described in detail below in conjunction with specific implementation modes.

[0023] The technical solution of the present invention is as follows Figure 1 , taking a transmission line in Kunming, Yunnan Province as an example, the following steps are included:

[0024] Step 1: Data Processing

[0025] First, the collected SuperView-1 and Worldview-3 multispectral images and panchromatic band images of the designated area were opened in Envi software, and the default elevation data (DEM) was selected for orthorectification. Then, the Gram-Schmidt remote sensing satellite image fusion algorithm was used to obtain four-channel (RGBN) remote sensing satellite images, of which the resolution of SuperView-1 fusion images was 0.5m and that of WorldView-3 fusion images was 0.3m. The image data was then compressed to a range of 0-255.

[0026] Step 2: Dataset creation

[0027] The ortho-reflected SuperView-1 and WorldView-3 images were cropped with a step size of 512 pixels to obtain a high-resolution (HR) training data set as the true value. The original images were downsampled 4 times according to the bicubic interpolation downsampling algorithm, and the 4-fold downsampled data were cropped with a step size of 128 pixels to obtain a low-resolution (LR) training data set. The two constituted a training data pair.

[0028] Step 3: Super-resolution network construction

[0029] How to build a super-resolution network Figure 2 , mainly using ten improved residual network modules that replace batch normalization layers with weight normalization layers to build the network. The residual network module is mainly composed of the first convolution layer, weight normalization layer, linear rectification layer, second convolution layer and weight normalization layer. Among them, the convolution kernel of the first convolution layer has a side length of 3, the number of output channels is 192, and the convolution step is 1. The convolution kernel of the second convolution layer has a side length of 3, the number of output channels is 32, and the convolution step is 1. The input of the residual network module is added to the output of the last weight normalization layer by jumping to form the final output of the residual network module. At the same time, three pixel reorganization amplification modules are built, one is mainly composed of the fourth convolution layer and the pixel reorganization upsampling 4 times layer I, the second and third are branches, and the second branch is mainly composed of the fifth convolution layer, the sixth convolution layer and the pixel reorganization upsampling 4 times layer II. Among them, the convolution kernel of the fifth convolution layer has a side length of 3, the output channel is 96, and the convolution step is 1. The convolution kernel of the sixth convolution layer has a side length of 5, an output channel of 12, and a convolution step of 1. The third branch is mainly composed of the seventh convolution layer, the eighth convolution layer, and the pixel reorganization upsampling 4 times layer III. The convolution kernel of the seventh convolution layer has a side length of 3, a channel of 32, and a convolution step of 1. The convolution kernel of the eighth convolution layer has a side length of 5, an output channel of 12, and a convolution step of 1. A pixel reorganization amplification module is connected in series after the residual network module, and the other two are jumped at the network input end, and finally added to obtain the final output.

[0030] Step 4: Super-resolution network training

[0031] The training data pairs obtained in step 2 are put into the super-resolution network built in step 3, and the final training parameters are set as follows: the initial learning rate is 0.0002, and the learning rate is decayed by 0.5 at the 25000th, 50000th, and 75000th single cycles respectively, the batch size is Batch size = 16, the optimization algorithm uses the Adam algorithm, the total number of training cycles is 100, and the final super-resolution model is obtained.

[0032] According to the obtained super-resolution training image pairs, the original images were cropped to 512×512 pixel size, and the downsampled images were cropped to 128×128 pixel size; then, data augmentation was performed by horizontal flipping and vertical flipping, and a total of 9900 pairs of training images were generated.

[0033] Step 5: Model Testing

[0034] The SuperView-1 and WorldView-3 images (different from the training set images) fused by Gram-Schmidt in step 1 are first cropped to 512×512 pixels and input into the trained super-resolution model in batches to obtain the super-resolution output. The SuperView-1 resolution of the output image is 0.125m and the WorldView-3 resolution is 0.75m. At the same time, the coordinate information is retained with the help of the Gdal library, and finally the cropped super-resolution output images are merged into the original images.

[0035] Step 6: Analytical Evaluation and Indicator Assessment

[0036] The three evaluation indicators of Universal Image Quality Index (UQI), Mean Similarity Index (MSSIM) and Weighted Peak Signal-to-Noise Ratio (WPSNR) are used to evaluate the similarity between the 2m resolution image after 4 times downsampling and the original 0.5m high-resolution image after 4 times super-resolution of each model (since there is no true value after super-resolution of the image after Gram-Schmidt fusion, the evaluation indicators are calculated here using the super-resolution of the downsampled image and the original high-resolution image). The results are shown in Table 1. It can be seen that the trained super-resolution model can achieve very high evaluation indicators on SuperView-1 and WorldView-3 satellite images, indicating that the super-resolution images can obtain higher fidelity. At the same time, check the super-resolution result diagram of the transmission line tower. Figure 3 to Figure 4It can be seen that the transmission line tower has been more obviously enhanced in detail, the tower structure is more obvious, and the insulator features are also more obvious, which is beneficial to the subsequent application of remote sensing image transmission line target detection.

[0037] Table 1 Super-resolution evaluation index table

[0038]

[0039] The beneficial effect of the present invention is that it can quickly super-resolution and amplify the remote sensing satellite image of the power transmission line by 4 times to obtain a clearer remote sensing satellite image of the power transmission line. The super-resolution model trained by the present invention has been well verified for the super-resolution visualization results of two types of remote sensing satellite images. Evaluation indicators such as UQI are all higher than 0.99, MSSIM are all higher than 0.91, and WPSNR are all higher than 32, indicating that the results have high credibility and have been applied to the remote sensing satellite processing of the Yunnan power grid, which can provide clearer images for transmission line inspection and insulator target detection.

Claims

1. A satellite image super-resolution method based on deep learning, the method comprising: Step 1: Obtain satellite images; Step 2: Orthorectify the satellite image using 90m×90m elevation data; Step 3: Perform panchromatic band fusion on the orthorectified remote sensing satellite image. The fused image is a four-channel image of red band, green band, blue band and near infrared band; Step 4: Use the bicubic interpolation downsampling algorithm to downsample the orthorectified remote sensing satellite image by 4 times, and then perform data augmentation by horizontal flipping and vertical flipping. The original multispectral image is used as the true value of the label to obtain the training image pair; Step 5: Build a super-resolution network; The super-resolution network has 4 channels, corresponding to the red band, green band, blue band and near-infrared band. Each channel is divided into two branches after input, wherein the main branch passes through the first convolution layer, ten residual network modules, the fourth convolution layer, and the pixel reorganization upsampling 4 times layer I in sequence; the branch first passes through the fifth convolution layer, and then is divided into two branches, one of which passes through the sixth convolution layer and the pixel reorganization upsampling 4 times layer II in sequence, and the other passes through the seventh convolution layer, the eighth convolution layer and the pixel reorganization upsampling 4 times layer III in sequence; the result of adding the outputs of the pixel reorganization upsampling 4 times layer I, the pixel reorganization upsampling 4 times layer II, and the pixel reorganization upsampling 4 times layer III is used as the output of the entire super-resolution network; The residual network module includes, in sequence: a second convolutional layer, a first weight normalization layer, a linear rectification layer, a third convolutional layer and a second weight normalization layer; The convolution kernel side length of the first convolution layer is 3, the number of output channels is 192, and the convolution step length is 1. The convolution kernel side length of the second convolution layer is 3, the number of output channels is 192, and the convolution step length is 1; the convolution kernel side length of the third convolution layer is 3, the output channel is 32, the convolution step length is 1, the convolution kernel side length of the fourth convolution layer is 3, the output channel is 12, and the convolution step length is 1; the convolution kernel side length of the fifth convolution layer of the first side branch is 3, the number of output channels is 96, the convolution step length is 1, the convolution kernel side length of the sixth convolution layer is 5, the number of output channels is 12, and the convolution step length is 1; the convolution kernel side length of the seventh convolution layer of the second side branch is 3, the number of output channels is 32, the convolution step length is 1, and the convolution kernel side length of the eighth convolution layer is 5, the number of output channels is 12, and the convolution step length is 1; Step 6: Use the training image pairs obtained in step 4 as the input of the super-resolution network in step 5 to train a super-resolution network; Step 7: When super-resolving the super-resolved image, firstly, the image is segmented and cropped, then each cropped regional image is super-resolved, and finally the super-resolved regional images are spliced; Step 8: Perform three-channel false color and 1% linear stretching on the red band, near-infrared band and blue band of the super-resolution image, and output it.

2. The satellite image super-resolution method based on deep learning according to claim 1, characterized in that: In step 6, the number of training rounds is set to 100, the initial learning rate is set to 0.0002, and the learning rate is decayed by 0.5 at the 25000th, 50000th, and 75000th single cycles, respectively, and the batch size is 16.

Images