A model self-tuning remote sensing image super-resolution reconstruction method, device and medium

Abstract

The present application relates to the field of remote sensing image reconstruction, and discloses a model self-tuning remote sensing image super-resolution reconstruction method, equipment and medium, comprising: constructing a remote sensing image super-resolution reconstruction network; inputting a low-resolution remote sensing image into the remote sensing image super-resolution reconstruction network for first-stage training to obtain a first-stage super-resolution reconstruction result; importing the first-stage remote sensing image super-resolution reconstruction network training parameters into the second stage for second-stage remote sensing image super-resolution reconstruction network training, fine-tuning the remote sensing image super-resolution reconstruction network parameters, and obtaining a second-stage super-resolution reconstruction result. The method can obtain a more realistic remote sensing image super-resolution reconstruction result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of remote sensing image reconstruction, in particular to a model self-tuning remote sensing image super-resolution reconstruction method, device and medium. BACKGROUND

[0002] Remote sensing image is a detection system installed on a satellite, an airplane or other flight platform to obtain ground object information on the earth's surface. After interpretation through a series of technical means, it can effectively provide effective image support for applications that need related ground object information.

[0003] Remote sensing image is usually the surface information photographed by a satellite in outer space. The resolution of the image is usually limited by the long detection distance. The detailed ground object information is usually small and difficult to distinguish in detail, which brings great obstacles to the application in related fields.

[0004] In order to solve the above problems, the current mainstream method is to use deep learning method to super-resolution enlarge the remote sensing image, and to improve the resolution of the remote sensing image to improve the resolution of the related ground object information, and to provide effective technical support for the application work in related fields. Generally, the existing deep learning method adopts a one-stage training method to train the model. After reaching the fitting state, the network model with the highest PSNR and SSIM indicators is selected as the trained model. This method has obtained good super-resolution enlargement results in actual application. However, this training method is affected by the difference between the true value and the input. If the difference between the true value and the input is too large, the one-stage training method is difficult to learn the high-frequency detail information in the real high-resolution image, which will lead to the phenomenon of fuzzy details in the super-resolution reconstruction result. SUMMARY

[0005] The purpose of the present application is to provide a model self-tuning remote sensing image super-resolution reconstruction method, device and medium, which solves the problem that the existing remote sensing image super-resolution reconstruction network is difficult to learn the high-frequency detail information in the real high-resolution image due to the too large difference between the input and the true value, resulting in fuzzy details in the super-resolution reconstruction result.

[0006] Specifically, the present application provides a model self-tuning remote sensing image super-resolution reconstruction method, device and medium, which comprises the following steps:

[0007] S1, constructing a remote sensing image super-resolution reconstruction network;

[0008] S2, inputting a low-resolution remote sensing image into the remote sensing image super-resolution reconstruction network for first-stage training to obtain a first-stage super-resolution reconstruction result;

[0009] S3, the first stage remote sensing image super-resolution reconstruction network training parameters are imported into the second stage, and the second stage remote sensing image super-resolution reconstruction network is trained, the remote sensing image super-resolution reconstruction network parameters are fine-tuned, and the second stage super-resolution reconstruction result is obtained.

[0010] A storage medium stores instructions and data for implementing a model self-fine-tuning remote sensing image super-resolution reconstruction method.

[0011] A model self-fine-tuning remote sensing image super-resolution reconstruction device comprises a processor and the storage medium; the processor loads and executes the instructions and data in the storage medium to implement a model self-fine-tuning remote sensing image super-resolution reconstruction method.

[0012] The present application provides beneficial effects: the present application provides a two-stage self-fine-tuning model remote sensing image super-resolution reconstruction method, in the training process of the first stage, learn as much as possible high resolution remote sensing image restoration information, then use the model of the first stage as the feature loss function in the second stage and joint visual loss and adversarial loss as the overall loss function to fine-tune the super-resolution network trained in the first stage, further to obtain the high-frequency feature information that the first stage cannot learn, so as to obtain more real high-resolution remote sensing image. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a simple process flow diagram of the method of the present application;

[0014] Figure 2 is a super-resolution network structure diagram;

[0015] Figure 3 is a deep feature extraction module diagram;

[0016] Figure 4 is a deep feature extraction function diagram;

[0017] Figure 5 is a super-resolution reconstruction result diagram;

[0018] Figure 6 is a hardware device working diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described below with reference to the drawings.

[0020] Before formally describing the present application, first of all, the scheme of the present application is described generally, which is convenient for understanding.

[0021] Please refer to Figure 1The application provides a model self-tuning remote sensing image super-resolution reconstruction method, comprising:

[0022] S1, constructing a remote sensing image super-resolution reconstruction network;

[0023] It should be noted that the remote sensing image super-resolution reconstruction network in the application is based on the SRNet architecture, and the specific structure is shown in Figure 2 , comprising a shallow feature extraction module, a plurality of sequentially connected deep feature extraction modules and an up-sampling module.

[0024] S2, inputting a low-resolution remote sensing image into the remote sensing image super-resolution reconstruction network for first-stage training to obtain a first-stage super-resolution reconstruction result;

[0025] It should be noted that the specific process of obtaining the first-stage super-resolution reconstruction result in step S2 is as follows:

[0026] S21, inputting a low-resolution remote sensing image into the shallow feature extraction module to obtain a high-dimensional shallow feature vector;

[0027] S22, inputting the high-dimensional shallow feature vector into the plurality of sequentially connected deep feature extraction modules to obtain a sufficient high-dimensional feature vector required for super-resolution reconstruction;

[0028] S23, amplifying the sufficient high-dimensional feature vector required for super-resolution reconstruction through the up-sampling module to obtain high-resolution residual information;

[0029] S24, after the low-resolution remote sensing image is amplified by bilinear interpolation, the high-resolution residual information is fused to obtain the first-stage super-resolution reconstruction result.

[0030] As a specific embodiment, the training process of the first stage only uses the L1 loss function to perform gradient backpropagation to achieve network fitting, so that the super-resolution network learns as much super-resolution restoration information as possible, and the overall method process is shown in Figure 1 . Here, the low-resolution remote sensing image is input into the super-resolution network to obtain the first super-resolution reconstruction result of the first stage, as shown below:

[0031] (1)

[0032] wherein, , are a low-resolution remote sensing image and a first super-resolution reconstruction result of the first stage, is a super-resolution network adopted, and the specific structure is shown in Figure 2 .

[0033] Next, the super-resolution network Detailed description is unfolded. First, the low-resolution remote sensing image is input to the shallow feature extraction module, which is converted into a high-dimensional shallow feature vector, as shown below:

[0034] (2)

[0035] wherein, is the extracted high-dimensional shallow feature vector, is a 3*3 convolution operation (shallow feature extraction module), the input channel is 3, and the output channel is 64.

[0036] Next, the extracted shallow feature vector is input into N deep feature extraction modules for deep feature vector extraction. In the embodiment of the present application, N is set to N=32. Correspondingly, a schematic diagram of the deep feature extraction module is shown in Figure 3 .

[0037] First, the shallow feature vector is input into a 1*1 convolution layer, as shown below:

[0038] (3)

[0039] wherein, is the extracted high-dimensional deep feature vector, is a 1*1 convolution operation.

[0040] Next, the high-dimensional deep feature vector is input into a 3*3 convolution layer and an activation function, specifically:

[0041] (4)

[0042] wherein, is the further extracted high-dimensional deep feature vector, is a 1*1 convolution operation, is a torch.nn.ReLU function in pytorch. Then, the high-dimensional deep feature vector is input into a 1*1 convolution layer and added to the shallow feature vector input into this module, to obtain a fully fused deep feature vector, as shown below:

[0043] (5)

[0044] wherein, is the fully fused deep feature vector, is a 1*1 convolution operation. Then, Again, through the convolution operation, activation function and residual connection, further to obtain the required deep feature vector. Specifically, first input to 1*1 convolution layer and activation function, as follows:

[0045] (6)

[0046] wherein, is the extracted high-dimensional deep feature vector, is a 1*1 convolution operation, is the torch.nn.ReLU function in pytorch. Then, input to the 1*1 convolution layer plus , get the fully fused deep feature vector, as follows:

[0047] (7)

[0048] wherein, is the deep feature vector output by the deep feature extraction module, is a 1*1 convolution operation.

[0049] Next, the deep feature vector extracted by the deep feature extraction module is input again to the next deep feature extraction module, and the operation of formula (3)-(7) is repeated N=32 times, so that the deep features are fully fused and contain the image information required for super-resolution reconstruction.

[0050] Through a series of deep feature extraction modules, the sufficient high-dimensional feature vector required for super-resolution reconstruction is obtained. Next, it is obtained through the up-sampling module, and the high-resolution residual information is obtained, which is added to the interpolation enlarged result of the original low-resolution remote sensing image, to obtain the final super-resolution reconstruction result, as follows:

[0051] (8)

[0052] wherein, is a 3*3 convolution operation, is the torch.nn.PixelShuffle function in pytorch, bic is a bilinear interpolation operation, is the first super-resolution reconstruction result of the first stage.

[0053] The above describes the super-resolution network in detail, and through the super-resolution network , the low-resolution remote sensing image can be converted into a high-resolution remote sensing image. Here, the training process of the first-stage super-resolution network is described in detail. In the training process of the first stage, the super-resolution network The restoration information of as many high-resolution remote sensing images as possible is learned to achieve the maximum learning ability of the network itself. Therefore, only the pixel loss of L1 is used to perform gradient back propagation on the super-resolution network in the first stage, and the model fitting outputs corresponding network parameters. The specific L1 loss is as follows:

[0054] (9)

[0055] wherein, , are the low-resolution remote sensing images and high-resolution remote sensing images used in training, is the L1 norm, is the loss function used in the one-stage training process.

[0056] The formula (9) is used as the loss function in network training. When the number of iterations is considered to reach the fitting state of the model, the current network parameters are output as the second-stage fine-tuning.

[0057] S3, the first-stage remote sensing image super-resolution reconstruction network training parameters are imported into the second stage, and the second-stage remote sensing image super-resolution reconstruction network is trained, the remote sensing image super-resolution reconstruction network parameters are fine-tuned, and the second-stage super-resolution reconstruction result is obtained.

[0058] It should be noted that in the second-stage training process of step S3, the up-sampling module and the residual connection in the remote sensing image super-resolution reconstruction network trained in the first stage are removed, and the remaining part is used as a deep feature extraction function for the feature loss function in the fine-tuning stage. The corresponding parameters of the deep feature extraction function are inherited from the network parameters trained in the first stage, and the following loss function is used for gradient update:

[0059]

[0060] wherein, , are the low-resolution remote sensing images and high-resolution remote sensing images used in training, is the part of the network trained in the second stage after removing the sampling module and the residual connection; is the classical VGG pre-trained visual loss, is the classical UNetDiscriminatorSN adversarial loss, are the weight parameters of the corresponding loss function part, respectively; is the L1 norm.

[0061] As an embodiment, through a one-stage training process, one-stage trained network parameters are obtained , and a two-stage fine-tuning process is unfolded using the training parameters to obtain high-frequency feature information that the first stage learning cannot learn, so as to obtain a more real high-resolution remote sensing image.

[0062] The specific process is to remove the sampling module and the residual connection in the one-stage trained super-resolution network , and use the remaining part as a deep feature extraction function as a feature loss function in the model fine-tuning stage, and the corresponding structure is shown in Figure 4 . The corresponding parameters of the deep feature extraction function are inherited from the one-stage trained network parameters , and are not changed by gradient update in the two-stage fine-tuning process. In the second stage training process, the one-stage trained super-resolution network is further trained using the following loss function for gradient update only, as follows:

[0063] (10)

[0064] wherein, is a low-resolution remote sensing image used in training, is the remaining part of the two-stage trained network after removing the sampling module and the residual connection (the detailed structure is shown in Figure 4 ); is a classic VGG pre-trained visual loss, is a classic UNetDiscriminatorSN adversarial loss, are weight parameters of the corresponding loss function parts, which are set to here.

[0065] Taking equation (10) as the loss function in the two-stage fine-tuning process, when the iteration number is considered to reach the fitting state of the model, the current network parameters are output as the final fine-tuning result.

[0066] Finally, using the fine-tuned , the low-resolution remote sensing image can be converted into a clear high-resolution remote sensing image, as shown below:

[0067] (11)

[0068] wherein, , respectively are low-resolution remote sensing images, super-resolution reconstruction results of the two-stage network after fine-tuning, is a two-stage super-resolution network after fine-tuning.

[0069] Please refer to Figure 5 , Figure 5 is a schematic diagram of the result of an embodiment of the present application. Figure 5 In (a) of FIG. 1, (a) represents a low-resolution remote sensing image; (b) represents a first-stage super-resolution reconstruction result; and (c) represents a second-stage super-resolution reconstruction result. Figure 5 It can be seen from the result that in the training process of the first stage, the present application learns as much as possible the restoration information of high-resolution remote sensing images, and then in the second stage, the model of the first stage is used as a feature loss function and the visual loss and the adversarial loss are used as the overall loss function to fine-tune the super-resolution network trained in the first stage, further to obtain the high-frequency feature information that cannot be learned in the first stage, so that a more real high-resolution remote sensing image can be obtained.

[0070] Please refer to Figure 6 , Figure 6 is a hardware device working schematic diagram of an embodiment of the present application, and the hardware device specifically comprises: a model self-fine-tuning remote sensing image super-resolution reconstruction device 401, a processor 402 and a storage medium 403.

[0071] The model self-fine-tuning remote sensing image super-resolution reconstruction device 401: the model self-fine-tuning remote sensing image super-resolution reconstruction device 401 realizes the model self-fine-tuning remote sensing image super-resolution reconstruction method.

[0072] The processor 402: the processor 402 loads and executes the instructions and data in the storage medium 403 to realize the model self-fine-tuning remote sensing image super-resolution reconstruction method.

[0073] The storage medium 403: the storage medium 403 stores instructions and data; the storage medium 403 is used to realize the model self-fine-tuning remote sensing image super-resolution reconstruction method.

[0074] In general, the beneficial effects of the present application are: the present application proposes a two-stage self-fine-tuning model remote sensing image super-resolution reconstruction method, in the training process of the first stage, as much as possible the restoration information of high-resolution remote sensing images is learned, and then in the second stage, the model of the first stage is used as a feature loss function and the visual loss and the adversarial loss are used as the overall loss function to fine-tune the super-resolution network trained in the first stage, further to obtain the high-frequency feature information that cannot be learned in the first stage, so that a more real high-resolution remote sensing image can be obtained.

[0075] The above merely describes preferred embodiments of the present application and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A model self-fine-tuning remote sensing image super-resolution reconstruction method, characterized in that: The method comprises the following steps: S1, constructing a remote sensing image super-resolution reconstruction network; S2, inputting a low-resolution remote sensing image into the remote sensing image super-resolution reconstruction network for first-stage training to obtain a first-stage super-resolution reconstruction result; S3, importing first-stage remote sensing image super-resolution reconstruction network training parameters into a second stage for training of a second-stage remote sensing image super-resolution reconstruction network, fine-tuning remote sensing image super-resolution reconstruction network parameters, and obtaining a second-stage super-resolution reconstruction result; In the second stage of the step S3, the up-sampling module and the residual connection in the remote sensing image super-resolution reconstruction network trained in the first stage are removed, and the remaining part is used as a deep feature extraction function , which is used as the feature loss function in the fine-tuning stage. The corresponding parameters of the deep feature extraction function are inherited from the network parameters trained in the first stage, and only gradient update is performed using the following loss function: wherein, , are low-resolution remote sensing images and high-resolution remote sensing images used in training, is a network trained in two stages , the remaining part of the removal sampling module and the residual connection in; is a visual loss using a classic VGG pre-training, is a classic UNetDiscriminatorSN adversarial loss, are the weight parameters of the corresponding loss function part, respectively; is the L1 norm.

2. The model self-tuning remote sensing image super-resolution reconstruction method of claim 1, wherein: The remote sensing image super-resolution reconstruction network in step S1 is based on an SRNet architecture and comprises a shallow feature extraction module, a plurality of sequentially connected deep feature extraction modules, and an upsampling module.

3. The model self-tuning remote sensing image super-resolution reconstruction method of claim 2, wherein: The specific process of obtaining the first-stage super-resolution reconstruction result in step S2 is as follows: S21, inputting a low-resolution remote sensing image into the shallow feature extraction module to obtain a high-dimensional shallow feature vector; S22, inputting the high-dimensional shallow feature vector into the plurality of sequentially connected deep feature extraction modules to obtain a high-dimensional feature vector sufficient for super-resolution reconstruction; S23, magnifying the high-dimensional feature vector sufficient for super-resolution reconstruction through the upsampling module to obtain high-resolution residual information; S24, magnifying the low-resolution remote sensing image through bilinear interpolation, and then fusing the high-resolution residual information to obtain the first-stage super-resolution reconstruction result.

4. The model self-tuning remote sensing image super-resolution reconstruction method of claim 1, wherein: The first-stage training in step S2 only uses an L1 loss function to train the remote sensing image super-resolution reconstruction network.

5. The model self-tuning remote sensing image super-resolution reconstruction method of claim 1, wherein: Each deep feature extraction module comprises, in sequence, a first 1*1 convolutional layer, a 3*3 convolutional layer, an activation function, and a second 1*1 convolutional layer.

6. A storage medium characterized by: The storage medium stores instructions and data for implementing the model self-fine-tuning remote sensing image super-resolution reconstruction method of any one of claims 1-5.

7. A model self-tuning remote sensing image super-resolution device, characterized in that: It comprises: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the model self-fine-tuning remote sensing image super-resolution reconstruction method of any one of claims 1-5.

Images