A remote sensing image texture enhancement method, device and storage medium

Abstract

The application relates to the field of image processing and discloses a remote sensing image texture enhancement method, equipment and a storage medium, a generation network and an adversarial network are constructed, and a loss function is improved to optimize the training effect. The generation network improves the feature extraction capability through methods such as multistage dense connection, Shuffle attention and residual connection. Meanwhile, a discriminant network composed of a convolution layer and an activation layer is designed, full connection layers in the discriminant network are removed, and the discriminant network can help the network to generate more abundant texture details through local discrimination. In the training process, a texture residual loss function is used for local statistics to distinguish false images and real texture details, and the network is helped to reduce the influence of false images. The application has the beneficial effects that super-resolution remote sensing images with more real texture details can be generated, the generation of false images is reduced, and better visual effects are achieved.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a method, apparatus, and storage medium for enhancing the texture of remote sensing images. Background Technology

[0002] The increasing maturity of remote sensing technology has enabled researchers to obtain abundant surface observation images, serving various sectors of society. However, due to limitations in remote sensing image imaging hardware, technology, and observation environment, obtaining high-resolution remote sensing images requires a significant amount of work. The quality of low-resolution remote sensing images cannot meet the needs of map updating, semantic segmentation, and target detection. Therefore, super-resolution reconstruction technology for remote sensing images has become an important processing technique for improving the clarity and reliability of remote sensing images at low cost. Through super-resolution reconstruction, observation results in remote sensing images can be made more apparent, providing more reliable data for subsequent remote sensing image processing.

[0003] Image super-resolution reconstruction is a technique that reconstructs high-resolution images with richer information from low-resolution images. In recent years, deep learning-based image super-resolution reconstruction methods, especially those based on generative adversarial networks (GANs), have gradually become the mainstream research approach. GAN-based methods offer superior reconstruction results for texture details, and many researchers have proposed various improvement schemes based on this idea. However, it should be noted that adversarial training still suffers from instability in this field, which may produce unpleasant artifacts during reconstruction. Generating complex textures and effectively removing artifacts remain among the challenges of remote sensing image super-resolution reconstruction technology. Summary of the Invention

[0004] The purpose of this invention is to propose a method, device, and storage medium for enhancing the texture of remote sensing images, thereby solving the artifact problem generated in the super-resolution generation process of existing technologies and generating high-resolution remote sensing images with more realistic texture details.

[0005] A method for enhancing texture in remote sensing images includes the following steps:

[0006] S1. Construct a dense generative adversarial network (GAN) for enhanced texture details; the GAN includes a generator network and a discriminator network; the generator network includes a feature extraction module and a reconstruction module;

[0007] S2. Train the generator network and the discriminator network, and optimize them using a loss function based on texture residuals to achieve a dynamic balance between the generator network and the discriminator network, thereby obtaining a trained generative adversarial network.

[0008] S3. Use the trained generative adversarial network to enhance the texture details of the remote sensing image, and obtain the remote sensing image with enhanced texture details.

[0009] A storage medium storing instructions and data for implementing a remote sensing image texture enhancement method.

[0010] A remote sensing image texture enhancement device includes: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a remote sensing image texture enhancement method.

[0011] The beneficial effects of this invention are: it enables the reconstruction of high-resolution remote sensing images with richer and more realistic texture details. This invention improves the feature extraction capability of the generative network by enhancing its capabilities, while the discriminative network performs local discrimination, making the network more focused on local structure and helping the model generate richer texture details. However, reducing the impact of artifacts is a challenge in the reconstruction process of generative adversarial networks. Therefore, this invention introduces a texture residual loss function and uses EMA technology for local statistics to reduce the impact of artifacts and generate images with higher visual quality. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0013] Figure 2 This is a schematic diagram of the structure of the texture detail enhancement dense generative adversarial network of the present invention;

[0014] Figure 3 This is a schematic diagram of the texture enhancement result of the present invention;

[0015] Figure 4 This is a schematic diagram of the hardware device of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0017] Please refer to Figures 1-2 , Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the structure of the dense generative adversarial network for texture detail enhancement according to the present invention. The present invention provides a method for enhancing the texture of remote sensing images, comprising the following steps:

[0018] S1. Construct a dense generative adversarial network (GAN) for enhanced texture details; the GAN includes a generator network and a discriminator network; the generator network includes a feature extraction module and a reconstruction module;

[0019] It should be noted that the present invention utilizes multi-level dense connections, Shuffle attention, residual connections, and local discrimination to construct its generator network and discriminator network.

[0020] Specifically, the generative network consists of a feature extraction module and a reconstruction module. We designed a dense residual block with Shuffle Attention (SADRDB) and used 23 densely connected SADRDBs as the main part of the feature extraction module. The reconstruction module consists of two sets of upsampling layers to perform image upscaling.

[0021] As one embodiment, the specific structure of the generator network is as follows:

[0022] First layer: A convolutional layer with a 3x3 kernel and a stride of 1;

[0023] As one example, the function call for the first layer is as follows:

[0024] torch.nn.Conv2d(in_channels=num_in_ch, out_channels=num_feat, kernel_size=3, stride=1, padding=1, bias=True);

[0025] Where in_channels represents the number of channels in the input fused image; out_channels represents the number of channels in the output image, which is also the number of convolution kernels; kernel_size represents the size of the convolution kernel; and stride represents the stride of the convolution kernel.

[0026] The second layer: The feature image is input into a densely connected SADRDB with 23 layers to further extract features;

[0027] As one example, the first layer is implemented by building 23 SADRDB modules using the make_layer method of models.modules.module_util.

[0028] The third layer: The feature image passes through a convolutional layer with a 3x3 kernel and a stride of 1;

[0029] The third layer: The feature image passes through a convolutional layer with a 3x3 kernel and a stride of 1;

[0030] As one example, the function call for the third layer is as follows:

[0031] torch.nn.Conv2d(in_channels=num_in_ch, out_channels=num_feat, kernel_size=3, stride=1, padding=1, bias=True).

[0032] Fourth layer: Further feature extraction is performed using the Shuffle attention layer, and residual connections are added;

[0033] It is particularly important to note that the core implementation code of the Shuffle attention layer is as follows:

[0034] b,c,h,w = x.shape;

[0035] x=x.reshape(b,groups,-1,h,w);

[0036] x = x.permute(0,2,1,3,4);

[0037] x = x.reshape(b, -1, h, w);

[0038] Where x is the input feature image; b is the batch size; c is the number of channels; h is the height of the image; w is the width of the image; the shape function obtains the shape of the image tensor; the permute function performs dimensional permutation; and the reshape function reshapes the image tensor.

[0039] Fifth layer: Upsampling layer, 3x3 convolutional kernel, convolutional layer with stride of 1 and leaky ReLU activation layer;

[0040] As one example, the function call for the fifth layer is as follows:

[0041] F.interpolate(fea,scale_factor=2,mode='nearest');

[0042] torch.nn.LeakyReLU(negative_slope=0.2, inplace=True);

[0043] Where fea represents the input feature image; scale_factor is the magnification factor; mode is the scaling algorithm; negative_slope is the negative slope; and inplace is a boolean parameter.

[0044] Sixth layer: Upsampling layer, 3x3 convolutional kernel, convolutional layer with stride of 1 and leaky ReLU activation layer;

[0045] It should be noted that the function called here is the same as that in the fifth layer.

[0046] The seventh layer consists of a 3x3 convolutional kernel, a convolutional layer with a stride of 1, and a leaky ReLU activation layer.

[0047] It should be noted that this layer still uses torch.nn.Conv2d and torch.nn.LeakyReLU.

[0048] Eighth layer: A convolutional layer with a 3x3 kernel and a stride of 1.

[0049] It should be noted that this layer still uses torch.nn.Conv2d.

[0050] It should be noted that the specific structure of SADRDB used in the generative network is as follows:

[0051] First layer: 3 densely connected residual dense blocks (RDBs);

[0052] The second layer: a convolutional layer with a 3x3 kernel and a stride of 1;

[0053] The third layer is the Shuffle attention layer, with residual connections added.

[0054] It should be noted that the specific structure of the RDB used above is as follows:

[0055] The first layer consists of four densely connected CL modules (composed of 3x3 convolutional kernels, convolutional layers with a stride of 1, and leaky ReLU activation layers), with residual connections added after the second and fourth CL layers, respectively.

[0056] The second layer: a convolutional layer with a 3x3 kernel and a stride of 1.

[0057] The processing steps of the input image in the generative network are as follows:

[0058] Step 1: Use a convolutional layer with a 3x3 kernel and a stride of 1 to extract shallow features from the image;

[0059] Step 2: Input the feature image into a 23-densely-connected SADRDB to further extract features;

[0060] Step 3: The feature image is further processed by a convolutional layer with a 3x3 kernel, a stride of 1, and a Shuffle attention layer to extract features, and residual connections are added.

[0061] Step 4: The image is magnified four times by using two upsampling layers, a 3x3 convolutional kernel, a convolutional layer with a stride of 1, and a leaky ReLU activation layer;

[0062] Step 5: A convolutional layer with a 3x3 kernel and a stride of 1, a leaky ReLU activation layer, a convolutional layer with a 3x3 kernel and a stride of 1, and finally output the feature image.

[0063] The discriminative network consists of multiple convolutional layers, batch normalization layers, and leaky ReLU activation layers. It has a certain receptive field and uses local discrimination to determine the similarity between the network-generated image and the high-resolution image, thus helping the network generate higher-quality high-resolution images.

[0064] It should be noted that the specific structure of the discriminant network is as follows:

[0065] First layer: A convolutional layer with 64 output channels, 4x4 kernels, and a stride of 2;

[0066] Preferably, the function call for the first layer is as follows:

[0067] torch.nn.Sequential(torch.nn.Conv2d(in_channels=3, out_channels=64, kernel_size=4, stride=2, padding=1), torch.nn.LeakyReLU());

[0068] The second layer consists of a convolutional layer with 128 output channels, a 4x4 kernel, a stride of 2, a batch normalization layer, and a leaky ReLU activation layer.

[0069] Preferably, the function call for the second layer is as follows:

[0070] torch.nn.Sequential(torch.nn.Conv2d(in_channels=64, out_channels=128, kernel_size=4, stride=2, padding=1), torch.nn.LeakyReLU());

[0071] The third time: a convolutional layer with 256 output channels, a 4x4 kernel, and a stride of 2, a batch normalization layer, and a leaky ReLU activation layer;

[0072] Preferably, the function call for the third layer is as follows:

[0073] torch.nn.Sequential(torch.nn.Conv2d(in_channels=128, out_channels=256, kernel_size=4, stride=2, padding=1), torch.nn.LeakyReLU());

[0074] The fourth layer consists of a convolutional layer with 512 output channels, a 4x4 kernel, a stride of 2, a batch normalization layer, and a leaky ReLU activation layer.

[0075] Preferably, the function call for the fourth layer is as follows:

[0076] torch.nn.Sequential(torch.nn.Conv2d(in_channels=256, out_channels=512, kernel_size=4, stride=2, padding=1), torch.nn.LeakyReLU());

[0077] Fifth layer: Convolutional layer with 512 output channels, 4x4 kernel, stride of 1, batch normalization layer, and leaky ReLU activation layer;

[0078] Preferably, the function call for the fifth layer is as follows:

[0079] torch.nn.Sequential(torch.nn.Conv2d(in_channels=512, out_channels=512, kernel_size=4, stride=1, padding=1), torch.nn.LeakyReLU());

[0080] The sixth layer consists of a convolutional layer with 512 output channels, a 4x4 kernel, a stride of 1, a batch normalization layer, and a leaky ReLU activation layer.

[0081] Preferably, the function call for the sixth layer is as follows:

[0082] torch.nn.Sequential(torch.nn.Conv2d(in_channels=512, out_channels=512, kernel_size=4, stride=1, padding=1), torch.nn.LeakyReLU());

[0083] Layer 7: A convolutional layer with 1 output channel, 4x4 kernel, and stride of 1.

[0084] Preferably, the function call for the sixth layer is as follows:

[0085] torch.nn.Conv2d(in_channels=512, out_channels=1, kernel_size=4, stride=1, padding=1);

[0086] S2. Train the generator network and the discriminator network, and optimize them using a loss function based on texture residuals to achieve a dynamic balance between the generator network and the discriminator network, thereby obtaining a trained generative adversarial network.

[0087] It should be noted that the loss function used in training is as follows:

[0088] L total =0.01*L pixel +L percep +0.005*L GA +L artif #(1)

[0089] In equation (1), L total It is the total loss function; L pixel It is a pixel loss function that calculates the difference between pixels in two images; L percep It is the perceptual loss function, which calculates the perceptual loss using the high-frequency feature maps before activation of the perceptual network; L GA It is an adversarial loss function that, through the supervision of the discriminative network, encourages the generation of more realistic images; L artif It is a texture residual loss function that reduces the impact of artifacts through local statistics.

[0090] The specific calculation process of the texture residual loss function is as follows:

[0091] Step 1: Calculate the residual image R between the real high-resolution feature image and the generated feature image;

[0092] Step 2: Perform local statistics to calculate the pixel difference M(i,j) within a certain area, as shown in the following formula:

[0093]

[0094] In equation (2), n is the size of the statistical region, and var(·) is the operation for calculating the variance of pixels;

[0095] Step 3: Use a global patch N to optimize local statistics, as shown in the following formula:

[0096]

[0097] In equation (3), α is the global patch parameter;

[0098] Step 4: Optimize using the Exponential Moving Average (EMA) technique, as shown in the formula below:

[0099]

[0100] In equation (4), R1 is the residual image between the generated feature image and the real high-definition image, R2 is the residual image between the generated feature image and the real high-definition image after applying an exponential moving average, and M... r These are the texture residual loss weights;

[0101] Step 5: Calculate the texture residual loss, using the following formula:

[0102]

[0103] As one example, the code for the local statistics implementation of the texture residual loss function is as follows:

[0104] pad = (ksize - 1) / / 2

[0105] residual_pad=torch.nn.functional.pad(residual,pad=[pad,pad,pad,pad],mode='reflect')

[0106] unfolded_residual=residual_pad.unfold(2,ksize,1).unfold(3,ksize,1)

[0107] pixel_level_weight=torch.var(unfolded_residual,dim=(-1,-2),unbiased=True,keepdim=True).squeeze(-1).squeeze(-1)

[0108] Where, ksize is the statistical window size; pad is the padding pixel size; residual is the calculated residual image; residual_pad is the padded residual image; torch.nn.functional.pad is the padding operation; unfolded_residual is the residual image after padded along dimensions 2 and 3; the torch.var function performs residual statistics according to a window of size ksize; and pixel_level_weight is the loss weight.

[0109] S3. Use the trained generative adversarial network to enhance the texture details of the remote sensing image, and obtain the remote sensing image with enhanced texture details.

[0110] To verify the method of this invention, please refer to... Figure 3 , Figure 3 This is a schematic diagram of the texture enhancement result of the present invention.

[0111] Figure 3 In the image, a is a high-resolution image, b is a bicubic interpolated image, and c is the super-resolution result image of this invention. It is clear that after texture detail enhancement, the result of this invention possesses richer texture details compared to the original image.

[0112] Please see Figure 4 , Figure 4 This is a schematic diagram of the hardware device in operation according to an embodiment of the present invention. The hardware device specifically includes: a remote sensing image texture enhancement device 401, a processor 402, and a storage medium 403.

[0113] A remote sensing image texture enhancement device 401: The remote sensing image texture enhancement device 401 implements the remote sensing image texture enhancement method.

[0114] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the remote sensing image texture enhancement method.

[0115] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the remote sensing image texture enhancement method.

[0116] The beneficial effects of this invention are: it can reconstruct high-resolution remote sensing images with richer and more realistic texture details. This invention improves the feature extraction capability of the generative network by enhancing its capabilities, while the discriminative network performs local discrimination, making the network more focused on local structure and helping the model generate richer texture details. However, reducing the impact of artifacts is a challenge in the reconstruction process of generative adversarial networks. Therefore, this invention introduces a texture residual loss function and uses EMA technology for local statistics to reduce the impact of artifacts and generate images with higher visual quality.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for enhancing texture in remotely sensed images, characterized in that: The method includes the following steps: S1. Construct a dense generative adversarial network to enhance texture details; The generative adversarial network includes a generator network and a discriminator network; The generative network includes: a feature extraction module and a reconstruction module; S2. Train the generator network and the discriminator network, and optimize them using a loss function based on texture residuals to achieve a dynamic balance between the generator network and the discriminator network, thus obtaining a trained generative adversarial network. S3. Use the trained generative adversarial network to enhance the texture details of the remote sensing image, and obtain the remote sensing image with enhanced texture details; The loss functions based on texture residuals mentioned in step S2 include: pixel loss function, perceptual loss function, adversarial loss function, and texture residual loss function; The specific calculation process of the texture residual loss function is as follows: Step 1: Calculate the residual image R between the real high-resolution feature image and the generated feature image; Step 2: Perform local statistics to calculate pixel differences within a certain area. The formula is as follows: In equation (2), n To calculate the size of the statistical area, var (·) is the operation for calculating the variance of pixels; Step 3: Use global patch N To optimize local statistics, the formula is as follows: In equation (3), α These are global patch parameters; Step 4: Optimize using the Exponential Moving Average (EMA) technique, as shown in the formula below: In equation (4), It is the residual image between the generated feature image and the real high-resolution image. It is the residual image between the generated feature image after applying an exponential moving average and the real high-resolution image. These are the texture residual loss weights; Step 5: Calculate the texture residual loss, using the following formula: 。 2. The remote sensing image texture enhancement method as described in claim 1, characterized in that: The feature extraction module has residual dense blocks with shuffle attention; the structure of the residual dense blocks with shuffle attention is as follows: First layer: 3 densely connected residual dense blocks (RDBs); The second layer: a convolutional layer with a 3x3 kernel and a stride of 1; The third layer: Shuffle attention layer, with added residual connections; The reconstruction module consists of multiple upsampling layers.

3. The remote sensing image texture enhancement method as described in claim 2, characterized in that: The structure of the residual dense block RDB is as follows: First layer: 4 densely connected CL modules, with residual connections added after the second and fourth CL layers respectively; The second layer: a convolutional layer with a 3x3 kernel and a stride of 1.

4. The remote sensing image texture enhancement method as described in claim 1, characterized in that: The discriminant network consists of multiple convolutional layers, batch normalization layers, and leaky ReLU activation layers.

5. A method for enhancing texture in remote sensing images as described in claim 2 or 3, characterized in that: The structure of the generative network is as follows: First layer: A convolutional layer with a 3x3 kernel and a stride of 1; Second layer: Multiple densely connected residual dense blocks with shuffle attention; The third layer: a convolutional layer with a 3x3 kernel and a stride of 1; Fourth layer: Shuffle attention layer, with residual connections added; Fifth layer: Upsampling layer, 3x3 convolutional kernel, convolutional layer with stride of 1 and leaky ReLU activation layer; Sixth layer: Upsampling layer, 3x3 convolutional kernel, convolutional layer with stride of 1 and leaky ReLU activation layer; The seventh layer consists of a 3x3 convolutional kernel, a convolutional layer with a stride of 1, and a leaky ReLU activation layer. Eighth layer: A convolutional layer with a 3x3 kernel and a stride of 1.

6. The remote sensing image texture enhancement method as described in claim 4, characterized in that: The specific structure of the discriminant network is as follows: First layer: A convolutional layer with 64 output channels, 4x4 kernels, and a stride of 2; The second layer consists of a convolutional layer with 128 output channels, a 4x4 kernel, a stride of 2, a batch normalization layer, and a leakyReLU activation layer. The third layer consists of a convolutional layer with 256 output channels, a 4x4 kernel, a stride of 2, a batch normalization layer, and a leakyReLU activation layer. The fourth layer consists of a convolutional layer with 512 output channels, a 4x4 kernel, a stride of 2, a batch normalization layer, and a leaky ReLU activation layer. Fifth layer: Convolutional layer with 512 output channels, 4x4 kernel, stride of 1, batch normalization layer, and leakyReLU activation layer; The sixth layer consists of a convolutional layer with 512 output channels, a 4x4 kernel, a stride of 1, a batch normalization layer, and a leaky ReLU activation layer. Layer 7: A convolutional layer with 1 output channel, 4x4 kernel, and stride of 1.

7. A storage medium, characterized in that: The storage medium stores instructions and data to implement the remote sensing image texture enhancement method according to any one of claims 1 to 6.

8. A remote sensing image texture enhancement device, characterized in that: include: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the remote sensing image texture enhancement method according to any one of claims 1 to 6.

Images