A lightweight remote sensing image super-resolution reconstruction method, device and equipment

Abstract

The present invention relates to the technical field of image processing, and discloses a lightweight remote sensing image super-resolution reconstruction method, device and equipment. The present invention performs random haze processing and interpolation processing on each image in the constructed high-definition image set to obtain low-resolution images; constructs a lightweight image super-resolution reconstruction network, which includes a low-frequency reconstruction module and a high-frequency reconstruction module; the low-frequency reconstruction module is composed of bicubic interpolation method; the high-frequency reconstruction module is a neural network structure, including a shallow feature extraction part, a deep residual feature extraction part and a feature reconstruction part; the feature reconstruction part adopts a lightweight reconstruction structure, mainly composed of nearest neighbor interpolation method, convolutional layer and non-linear activation layer; finally, the low-resolution images are used to train the reconstruction network to obtain a trained reconstruction model. The network model structure of the present invention has a low complexity, and can achieve a good reconstruction effect in the super-resolution reconstruction task of remote sensing images in special scenarios such as haze.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a lightweight remote sensing image super-resolution reconstruction method, device, and equipment. Background Art

[0002] Super-resolution reconstruction is an image processing technology that reconstructs a low-resolution image into a high-resolution image with rich texture details. Applying the super-resolution reconstruction technology in the field of remote sensing can improve the visual effect of remote sensing images.

[0003] Existing remote sensing image super-resolution reconstruction methods can be mainly divided into two categories. One category mainly performs super-resolution reconstruction based on a deep residual network; and the other category mainly performs super-resolution reconstruction based on a generative adversarial network. In order to pursue better reconstruction effects, related technologies usually adopt the method of deepening the network structure or adding an attention mechanism. For example, Patent CN202110730777.3 provides a remote sensing image super-resolution reconstruction method based on a lightweight generation model. Based on the idea of a generative adversarial network, it proposes a new generative adversarial network model based on remote sensing images. This network model integrates an attention saliency mechanism and a depthwise separable convolutional network in the generator part. Patent CN 202011563791.0 provides a remote sensing image super-resolution reconstruction method based on self-attention fusion. It adds an attention module for attention fusion before the feature extraction module and the re-module respectively, so that the features passing through the attention module are weighted, establishing a long-distance constraint relationship between features, thereby solving the limitation of the receptive field of the convolutional operation, strengthening the feature map useful for super-resolution reconstruction, and weakening the feature map useless for the super-resolution reconstruction task. Patent CN202110546005.4 provides a remote sensing image super-resolution reconstruction method based on a deep convolutional neural network. This invention example adopts an upsampling branch network and a main network. The upsampling branch learns global residual information, and the main network learns local detail information, and introduces an attention mechanism in the residual attention block. This mechanism fuses channel attention and spatial attention, enabling the network to more effectively focus on important feature information.

[0004] However, the above-mentioned remote sensing image super-resolution reconstruction methods do not consider the cost problem of the model, resulting in a very high complexity of the model and gradually stringent requirements for the hardware platform, further delaying the speed of reconstructing images. In addition, these existing super-resolution reconstruction models only adopt a single interpolation degradation model to obtain corresponding high-definition and low-definition image training pairs. This simple degradation model cannot effectively fit the degradation process of real-world remote sensing images. The imaging of remote sensing images is usually affected by haze or thin clouds, resulting in these methods not being able to achieve good reconstruction effects when dealing with the super-resolution reconstruction task of remote sensing images in special scenarios such as haze. Summary of the Invention

[0005] The present invention provides a lightweight remote sensing image super-resolution reconstruction method, device, and equipment, which solve the technical problems that the network model structure of the existing remote sensing image super-resolution reconstruction method is relatively complex and the reconstruction effect achieved in the super-resolution reconstruction task of remote sensing images in special scenarios such as haze is relatively poor.

[0006] In the first aspect of the present invention, a lightweight remote sensing image super-resolution reconstruction method is provided, including:

[0007] Obtain high-resolution remote sensing image data, and construct a high-definition image set according to the high-resolution remote sensing image data;

[0008] Perform random haze processing on each high-resolution remote sensing image in the high-definition image set, and then perform interpolation processing to obtain corresponding low-resolution images;

[0009] Construct a lightweight image super-resolution reconstruction network; the lightweight image super-resolution reconstruction network includes a low-frequency reconstruction module and a high-frequency reconstruction module; the low-frequency reconstruction module is used for reconstructing image low-frequency data, which is composed of bicubic interpolation; the high-frequency reconstruction module is used for reconstructing image high-frequency data, which is a neural network structure, including a shallow feature extraction part, a deep residual feature extraction part, and a feature reconstruction part; the feature reconstruction part adopts a lightweight reconstruction structure, mainly composed of nearest neighbor interpolation, a convolutional layer, and a non-linear activation layer;

[0010] Use the low-resolution images to train the lightweight image super-resolution reconstruction network to obtain a trained lightweight remote sensing image super-resolution reconstruction model.

[0011] According to an implementable manner of the first aspect of the present invention, the shallow feature extraction part is mainly composed of a convolutional layer; the deep residual feature extraction part is mainly composed of RCAB modules, the overall structure of the RCAB module adopts a residual form, and the RCAB module includes a first convolutional layer, a LeakyReLu non-linear activation layer, a second convolutional layer, and an HCA high-frequency perception channel attention module connected in sequence, and the HCA high-frequency perception channel attention module is mainly composed of global average pooling, the solution of global variance, a first linear layer, a LeakyReLu non-linear activation layer, a second linear layer, and a Sigmoid non-linear activation layer.

[0012] According to an implementable manner of the first aspect of the present invention, the convolutional layer in the shallow feature extraction part has a convolutional kernel of 3×3, an input channel of 3, and an output channel of 64; both convolutional layers of the RCAB module are convolutional layers with a convolutional kernel of 3×3 and both input and output channels of 64.

[0013] In an implementable manner according to the first aspect of the present invention, the feature reconstruction part includes a 2-fold adjacent interpolation method connected in sequence, a convolutional layer with a 3×3 convolutional kernel and input and output channels of 64 and 24 respectively, a convolutional layer with a 3×3 convolutional kernel and input and output channels of 24 and 24 respectively, and a LeakyReLu non-linear activation layer.

[0014] In an implementable manner according to the first aspect of the present invention, the construction of the high-definition image set based on the high-resolution remote sensing image data includes:

[0015] Preprocessing the high-resolution remote sensing image data; the preprocessing includes horizontal flipping, vertical flipping, and / or normalization processing.

[0016] In an implementable manner according to the first aspect of the present invention, the random haze processing of each high-resolution remote sensing image in the high-definition image set includes:

[0017] Performing random haze processing on each high-resolution remote sensing image in the high-definition image set according to the following standard optical model:

[0018] I(x) = J(x)t(x) + L[1 - t(x)]

[0019] Wherein, I(x) is the synthesized hazy image, x is the coordinate value of the image pixel, J(x) is the original image to be fogged, L is the global atmospheric light component, and t(x) is the transmittance.

[0020] In an implementable manner according to the first aspect of the present invention, the training of the lightweight image super-resolution reconstruction network using the low-resolution image includes:

[0021] Inputting the low-resolution image into the lightweight image super-resolution reconstruction network to obtain a reconstructed image;

[0022] Calculating the reconstructed image and the corresponding high-resolution image in the high-definition image set through the L1 loss function, calculating the backward gradient of the network according to the obtained loss function value, and updating the network model parameters according to the obtained gradient information. Through continuous iteration, the network model is optimized.

[0023] The second aspect of the present invention provides a lightweight remote sensing image super-resolution reconstruction device, including:

[0024] A high-definition image set construction module for acquiring high-resolution remote sensing image data and constructing a high-definition image set based on the high-resolution remote sensing image data;

[0025] An image processing module, configured to perform random haze processing on each high-resolution remote sensing image in the high-definition image set, and then perform interpolation processing to obtain corresponding low-resolution images;

[0026] A reconstruction network construction module, configured to construct a lightweight image super-resolution reconstruction network; the lightweight image super-resolution reconstruction network includes a low-frequency reconstruction module and a high-frequency reconstruction module; the low-frequency reconstruction module is used for reconstructing image low-frequency data, which is composed of bicubic interpolation method; the high-frequency reconstruction module is used for reconstructing image high-frequency data, which is a neural network structure, including a shallow feature extraction part, a deep residual feature extraction part, and a feature reconstruction part; the feature reconstruction part adopts a lightweight reconstruction structure, mainly composed of nearest neighbor interpolation method, convolutional layer, and non-linear activation layer;

[0027] A reconstruction network training module, configured to train the lightweight image super-resolution reconstruction network by using the low-resolution images to obtain a trained lightweight remote sensing image super-resolution reconstruction model.

[0028] According to an implementable manner of the second aspect of the present invention, the shallow feature extraction part is mainly composed of convolutional layers; the deep residual feature extraction part is mainly composed of RCAB modules, the overall structure of the RCAB module adopts a residual form, and the RCAB module includes a first convolutional layer, a LeakyReLu non-linear activation layer, a second convolutional layer, and an HCA high-frequency perception channel attention module connected in sequence, and the HCA high-frequency perception channel attention module is mainly composed of global average pooling, solving global variance, a first linear layer, a LeakyReLu non-linear activation layer, a second linear layer, and a Sigmoid non-linear activation layer.

[0029] According to an implementable manner of the second aspect of the present invention, the convolutional layer in the shallow feature extraction part has a convolutional kernel of 3×3, an input channel of 3, and an output channel of 64; both convolutional layers of the RCAB module are convolutional layers with a convolutional kernel of 3×3 and both input and output channels of 64.

[0030] According to an implementable manner of the second aspect of the present invention, the feature reconstruction part includes 2 times of nearest neighbor interpolation method, a convolutional layer with a convolutional kernel of 3×3 and input and output channels of 64 and 24 respectively, a convolutional layer with a convolutional kernel of 3×3 and input and output channels of 24 and 24 respectively, and a LeakyReLu non-linear activation layer connected in sequence.

[0031] According to an implementable manner of the second aspect of the present invention, the high-definition image set construction module includes:

[0032] An image preprocessing unit for preprocessing the high-resolution remote sensing image data; the preprocessing includes horizontal flipping, vertical flipping, and / or normalization processing.

[0033] According to an implementable manner of the second aspect of the present invention, the image processing module includes:

[0034] A haze processing unit for randomly performing haze processing on each high-resolution remote sensing image in the high-definition image set according to the following standard optical model:

[0035] I(x) = J(x)t(x) + L[1 - t(x)]

[0036] In the formula, I(x) is the synthesized hazy image, x is the coordinate value of the image pixel, J(x) is the original image to be fogged, L is the global atmospheric light component, and t(x) is the transmittance.

[0037] According to an implementable manner of the second aspect of the present invention, the reconstruction network training module includes:

[0038] An image reconstruction unit for inputting the low-resolution image into the lightweight image super-resolution reconstruction network to obtain a reconstructed image;

[0039] A training unit for calculating the reconstructed image and the corresponding high-resolution image in the high-definition image set through the L1 loss function, calculating the backward gradient of the network according to the obtained loss function value, and updating the network model parameters according to the obtained gradient information. By continuous iteration, the network model is optimized.

[0040] The third aspect of the present invention provides a lightweight remote sensing image super-resolution reconstruction device, including:

[0041] A memory for storing instructions; wherein, the instructions are used to implement the lightweight remote sensing image super-resolution reconstruction method described in any of the above implementable manners;

[0042] A processor for executing the instructions in the memory.

[0043] The fourth aspect of the present invention is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the lightweight remote sensing image super-resolution reconstruction method described in any of the above implementable manners.

[0044] From the above technical solutions, it can be seen that the present invention has the following advantages:

[0045] The present invention performs random haze processing on each high-resolution remote sensing image in the constructed high-definition image set, and then performs interpolation processing to obtain corresponding low-resolution images; constructs a lightweight image super-resolution reconstruction network, which includes a low-frequency reconstruction module and a high-frequency reconstruction module; the low-frequency reconstruction module is composed of bicubic interpolation method; the high-frequency reconstruction module is a neural network structure, including a shallow feature extraction part, a deep residual feature extraction part and a feature reconstruction part; the feature reconstruction part adopts a lightweight reconstruction structure, mainly composed of nearest neighbor interpolation method, convolutional layer and non-linear activation layer; finally, the lightweight image super-resolution reconstruction network is trained using the low-resolution images to obtain a trained lightweight remote sensing image super-resolution reconstruction model; the present invention optimizes the reconstruction network structure, so that the reconstruction effect can still be close to or even exceed the existing large-scale super-resolution networks while keeping the number of parameters of the overall model very low, and is optimized for the characteristics that the imaging quality of remote sensing images is easily affected by the atmosphere such as haze, fog, thin clouds, etc., so that the network can still have good imaging effect and quality in special scenarios such as haze weather, and solves the technical problems of the high complexity of the network model structure of the existing remote sensing image super-resolution reconstruction method and the poor reconstruction effect achieved in the remote sensing image super-resolution reconstruction task for special scenarios such as haze. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0047] Figure 1 It is a flowchart of a lightweight remote sensing image super-resolution reconstruction method provided by an optional embodiment of the present invention;

[0048] Figure 2 It is a schematic structural diagram of a lightweight image super-resolution reconstruction network provided by an optional embodiment of the present invention;

[0049] Figure 3 It is a schematic structural diagram of an RCAB module provided by an optional embodiment of the present invention;

[0050] Figure 4 It is a schematic structural diagram of an HCA high-frequency perception channel attention module provided by an optional embodiment of the present invention;

[0051] Figure 5 It is a schematic structural diagram of a feature reconstruction part provided by an optional embodiment of the present invention;

[0052] Figure 6 The structural connection block diagram of a lightweight remote sensing image super-resolution reconstruction device provided by an alternative embodiment of the present invention.

[0053] Reference numerals:

[0054] 1 - High-definition image set construction module; 2 - Image processing module; 3 - Reconstruction network construction module; 4 - Reconstruction network training module. Detailed implementation manners

[0055] The embodiments of the present invention provide a lightweight remote sensing image super-resolution reconstruction method, device and equipment, which are used to solve the technical problems that the network model structure of the existing remote sensing image super-resolution reconstruction method is relatively complex and the reconstruction effect is poor when performing the super-resolution reconstruction task of remote sensing images in special scenarios such as haze.

[0056] In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described below are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0057] The present invention provides a lightweight remote sensing image super-resolution reconstruction method.

[0058] Please refer to Figure 1 , Figure 1 , which shows the flowchart of a lightweight remote sensing image super-resolution reconstruction method provided by an embodiment of the present invention.

[0059] A lightweight remote sensing image super-resolution reconstruction method provided by an embodiment of the present invention includes steps S1 - S4.

[0060] Step S1, obtain high-resolution remote sensing image data, and construct a high-definition image set according to the high-resolution remote sensing image data.

[0061] In an implementable manner, the constructing a high-definition image set according to the high-resolution remote sensing image data includes:

[0062] Perform preprocessing on the high-resolution remote sensing image data; the preprocessing includes horizontal flipping, vertical flipping and / or normalization processing.

[0063] Through the data preprocessing operation, the enhancement of image data can be realized.

[0064] Step S2: Randomly haze the high-resolution remote sensing images in the high-definition image set, and then perform interpolation processing to obtain corresponding low-resolution images.

[0065] In an implementable manner, the randomly hazing the high-resolution remote sensing images in the high-definition image set includes:

[0066] Randomly haze the high-resolution remote sensing images in the high-definition image set according to the following standard optical model:

[0067] I(x) = J(x)t(x) + L[1 - t(x)]

[0068] In the formula, I(x) is the synthesized hazy image, x is the coordinate value of the image pixel, J(x) is the original image to be hazed, L is the global atmospheric light component, and t(x) is the transmittance.

[0069] In the embodiment of the present invention, a remote sensing image degradation model for haze weather is used, that is, a standard optical model is introduced to synthesize low-definition and hazed image training pairs. Based on the standard optical model, haze weather or foggy weather is synthesized, so that the training set has haze characteristics. Through the training of the subsequent model with this training set, the trained network model can be applicable to the remote sensing image super-resolution reconstruction task for processing scenes such as haze.

[0070] Step S3: Construct a lightweight image super-resolution reconstruction network; the lightweight image super-resolution reconstruction network includes a low-frequency reconstruction module and a high-frequency reconstruction module; the low-frequency reconstruction module is used for reconstructing the low-frequency data of the image, which is composed of bicubic interpolation; the high-frequency reconstruction module is used for reconstructing the high-frequency data of the image, which is a neural network structure, including a shallow feature extraction part, a deep residual feature extraction part, and a feature reconstruction part; the feature reconstruction part adopts a lightweight reconstruction structure, mainly composed of nearest neighbor interpolation, a convolutional layer, and a non-linear activation layer.

[0071] In the embodiment of the present invention, bicubic interpolation is used to complete the reconstruction of the low-frequency image, and a deep neural network is used to complete the reconstruction of the high-frequency image. Under the condition of a certain number of parameters, better reconstruction quality can be obtained.

[0072] Figure 2 Fig. shows the structural schematic diagram of the lightweight image super-resolution reconstruction network provided by an optional embodiment of the present invention. Among them, "Conv3×3" represents a convolutional kernel of 3×3, "Upsampler" represents the feature reconstruction part, and "Bicubic" represents bicubic interpolation.

[0073] Specifically, as Figure 2As shown, the convolution kernel of the convolution layer in the shallow feature extraction part is 3×3, the input channels are 3, and the output channels are 64.

[0074] In an implementable manner, the shallow feature extraction part is mainly composed of convolution layers; the deep residual feature extraction part is mainly composed of RCAB modules.

[0075] Figure 3 The structural schematic diagram of the RCAB module provided by an optional embodiment of the present invention is shown. Among them, "Conv3×3" represents a convolution kernel of 3×3, "LeakyReLu" represents a LeakyReLu non-linear activation layer, and "HCA" represents an HCA high-frequency perception channel attention module.

[0076] As Figure 3 shown, the overall structure of the RCAB module adopts a residual form. The RCAB module includes a first convolution layer, a LeakyReLu non-linear activation layer, a second convolution layer, and an HCA high-frequency perception channel attention module connected in sequence. Among them, both convolution layers of the RCAB module are convolution layers with a convolution kernel of 3×3 and both input and output channels being 64.

[0077] Figure 4 The structural schematic diagram of the HCA high-frequency perception channel attention module provided by an optional embodiment of the present invention is shown. Among them, "Avgpool" represents global average pooling, "Variance" represents the solution of global variance, "Tied FC" represents a linear layer, and "Sigmoid" represents a Sigmoid non-linear activation layer.

[0078] As Figure 4 shown, the HCA high-frequency perception channel attention module is mainly composed of global average pooling, the solution of global variance, a first linear layer, a LeakyReLu non-linear activation layer, a second linear layer, and a Sigmoid non-linear activation layer.

[0079] In the embodiment of the present invention, the HCA high-frequency perception channel attention module can complete the calculation of the mean and variance of each channel, and then reallocate the weights, making the network pay more attention to the high-frequency channels, which can effectively improve the reconstruction details of the network and is beneficial to improving the reconstruction effect.

[0080] Figure 5 The structural schematic diagram of the feature reconstruction part provided by an optional embodiment of the present invention is shown. Among them, "Nearest×2" represents the nearest neighbor interpolation method by 2 times, "Conv3×3" represents a convolution kernel of 3×3, and "LeakyReLu" represents a LeakyReLu non-linear activation layer.

[0081] In an implementable manner, as Figure 5 shown, the feature reconstruction part includes a 2-fold nearest neighbor interpolation method connected in sequence, a convolutional layer with a 3×3 convolutional kernel and input and output channels of 64 and 24 respectively, a convolutional layer with a 3×3 convolutional kernel and input and output channels of 24 and 24 respectively, and a LeakyReLu non-linear activation layer.

[0082] In the embodiment of the present invention, the 2-fold magnification of the feature is completed by the 2-fold nearest neighbor interpolation method, and then the feature fusion is completed by the convolutional layer, and then the relatively good values of the feature are retained through the LeakyReLu non-linear activation layer. The structure of this feature reconstruction part is simple and can effectively realize the reconstruction of the feature.

[0083] It should be noted that the parameters of the above network structure, such as the convolutional kernel parameters, the number of input and output channels, etc., can be adjusted according to actual needs.

[0084] Step S4, use the low-resolution image to train the lightweight image super-resolution reconstruction network to obtain a trained lightweight remote sensing image super-resolution reconstruction model.

[0085] In an implementable manner, the using the low-resolution image to train the lightweight image super-resolution reconstruction network includes:

[0086] Input the low-resolution image into the lightweight image super-resolution reconstruction network to obtain a reconstructed image;

[0087] Calculate the reconstructed image and the corresponding high-resolution image in the high-definition image set through the L1 loss function, calculate the backward gradient of the network according to the obtained loss function value, and update the network model parameters according to the obtained gradient information. Through continuous iteration, the network model is optimized.

[0088] Among them, the specific calculation formula of the L1 loss function is as follows, which represents the solution of the L1 norm of the reconstructed image and the corresponding high-resolution image in the high-definition image set:

[0089] L = ||I SR - I HR ||1

[0090] In the formula, I SR represents the reconstructed image obtained after being processed by the lightweight image super-resolution reconstruction network, and I HR is the high-resolution image corresponding to I SR .

[0091] The present invention also provides a lightweight remote sensing image super-resolution reconstruction device, which can be used to implement the lightweight remote sensing image super-resolution reconstruction method described in any one of the above embodiments of the present invention.

[0092] Please refer to Figure 6 , Figure 6 which shows the structural connection block diagram of a lightweight remote sensing image super-resolution reconstruction device provided by an embodiment of the present invention.

[0093] A lightweight remote sensing image super-resolution reconstruction device provided by an embodiment of the present invention includes:

[0094] A high-definition image set construction module 1, configured to obtain high-resolution remote sensing image data and construct a high-definition image set according to the high-resolution remote sensing image data;

[0095] An image processing module 2, configured to perform random haze processing on each high-resolution remote sensing image in the high-definition image set, and then perform interpolation processing to obtain corresponding low-resolution images;

[0096] A reconstruction network construction module 3, configured to construct a lightweight image super-resolution reconstruction network; the lightweight image super-resolution reconstruction network includes a low-frequency reconstruction module and a high-frequency reconstruction module; the low-frequency reconstruction module is used for reconstructing image low-frequency data, which is composed of bicubic interpolation method; the high-frequency reconstruction module is used for reconstructing image high-frequency data, which is a neural network structure, including a shallow feature extraction part, a deep residual feature extraction part, and a feature reconstruction part; the feature reconstruction part adopts a lightweight reconstruction structure, mainly composed of nearest neighbor interpolation method, convolutional layer, and non-linear activation layer;

[0097] A reconstruction network training module 4, configured to train the lightweight image super-resolution reconstruction network by using the low-resolution images to obtain a trained lightweight remote sensing image super-resolution reconstruction model.

[0098] In an implementable manner, the shallow feature extraction part is mainly composed of convolutional layers; the deep residual feature extraction part is mainly composed of RCAB modules, the overall structure of the RCAB module adopts a residual form, the RCAB module includes a first convolutional layer, a LeakyReLu non-linear activation layer, a second convolutional layer, and an HCA high-frequency perception channel attention module connected in sequence, and the HCA high-frequency perception channel attention module is mainly composed of global average pooling, solving global variance, a first linear layer, a LeakyReLu non-linear activation layer, a second linear layer, and a Sigmoid non-linear activation layer.

[0099] In an implementable manner, the convolutional layer in the shallow feature extraction part has a convolutional kernel of 3×3, an input channel of 3, and an output channel of 64; both convolutional layers of the RCAB module are convolutional layers with a convolutional kernel of 3×3 and both input and output channels of 64.

[0100] In one possible implementation, the feature reconstruction part includes a 2-fold nearest neighbor interpolation method connected in sequence, a convolutional layer with a 3×3 convolutional kernel and input and output channels of 64 and 24 respectively, a convolutional layer with a 3×3 convolutional kernel and input and output channels of 24 and 24 respectively, and a LeakyReLu non-linear activation layer.

[0101] In one possible implementation, the high-definition image set construction module 1 includes:

[0102] An image preprocessing unit for preprocessing the high-resolution remote sensing image data; the preprocessing includes horizontal flipping, vertical flipping, and / or normalization processing.

[0103] In one possible implementation, the image processing module 2 includes:

[0104] A haze processing unit for randomly haze-processing each high-resolution remote sensing image in the high-definition image set according to the following standard optical model:

[0105] I(x) = J(x)t(x) + L[1 - t(x)]

[0106] Where, I(x) is the synthesized hazy image, x is the coordinate value of the image pixel, J(x) is the original image to be fogged, L is the global atmospheric light component, and t(x) is the transmittance.

[0107] In one possible implementation, the reconstruction network training module 4 includes:

[0108] An image reconstruction unit for inputting the low-resolution image into the lightweight image super-resolution reconstruction network to obtain a reconstructed image;

[0109] A training unit for calculating the reconstructed image and the corresponding high-resolution image in the high-definition image set through the L1 loss function, calculating the backward gradient of the network according to the obtained loss function value, and updating the network model parameters according to the obtained gradient information. By continuous iteration, the network model is optimized.

[0110] The present invention also provides a lightweight remote sensing image super-resolution reconstruction device, including:

[0111] A memory for storing instructions; wherein, the instructions are used to implement the lightweight remote sensing image super-resolution reconstruction method described in any one of the above embodiments;

[0112] A processor for executing the instructions in the memory.

[0113] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the lightweight remote sensing image super-resolution reconstruction method described in any one of the above embodiments.

[0114] In the above embodiments of the present invention, the reconstruction network structure is optimized, so that when the number of parameters of the overall model remains very low, the reconstructed effect can still be close to or even exceed the existing large-scale super-resolution networks. Moreover, aiming at the characteristics that the imaging quality of remote sensing images is easily affected by the atmosphere, such as haze, fog, thin clouds, etc., the network is optimized so that it can still have good imaging effects and quality in special scenarios such as haze weather, solving the technical problems that the network model structure of the existing remote sensing image super-resolution reconstruction method is relatively complex and the reconstruction effect is poor when performing the super-resolution reconstruction task of remote sensing images in special scenarios such as haze.

[0115] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices, equipment, and modules can refer to the corresponding processes in the foregoing method embodiments, and the specific beneficial effects of the above-described devices, equipment, and modules can refer to the corresponding beneficial effects in the foregoing method embodiments, which will not be elaborated herein.

[0116] In several embodiments provided by the present application, it should be understood that the disclosed devices, equipment, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0117] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they can be located in one place, or they can be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0118] In addition, in each embodiment of the present invention, the functional modules can be integrated in one processing module, or each module can exist physically alone, or two or more modules can be integrated in one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0119] When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0120] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of various embodiments of the present invention.

Claims

1. A lightweight remote sensing image super-resolution reconstruction method, characterized in that, Including: Obtain high-resolution remote sensing image data, and construct a high-definition image set according to the high-resolution remote sensing image data; Perform random haze processing on each high-resolution remote sensing image in the high-definition image set, and then perform interpolation processing to obtain corresponding low-resolution images; Construct a lightweight image super-resolution reconstruction network; the lightweight image super-resolution reconstruction network includes a low-frequency reconstruction module and a high-frequency reconstruction module; the low-frequency reconstruction module is used for reconstructing image low-frequency data, which is composed of bicubic interpolation; the high-frequency reconstruction module is used for reconstructing image high-frequency data, which is a neural network structure, including a shallow feature extraction part, a deep residual feature extraction part, and a feature reconstruction part; The feature reconstruction part adopts a lightweight reconstruction structure, mainly composed of nearest neighbor interpolation, a convolutional layer, and a non-linear activation layer; Use the low-resolution images to train the lightweight image super-resolution reconstruction network to obtain a trained lightweight remote sensing image super-resolution reconstruction model; The shallow feature extraction part is mainly composed of convolutional layers; the deep residual feature extraction part is mainly composed of RCAB modules. The overall structure of the RCAB module adopts a residual form. The RCAB module includes a first convolutional layer, a LeakyReLu non-linear activation layer, a second convolutional layer, and an HCA high-frequency perception channel attention module connected in sequence. The HCA high-frequency perception channel attention module is mainly composed of global average pooling, the solution of global variance, a first linear layer, a LeakyReLu non-linear activation layer, a second linear layer, and a Sigmoid non-linear activation layer.

2. The lightweight remote sensing image super-resolution reconstruction method according to claim 1, characterized in that, The convolutional layer in the shallow feature extraction part has a convolutional kernel of 3×3, an input channel of 3, and an output channel of 64; both convolutional layers of the RCAB module are convolutional layers with a convolutional kernel of 3×3 and both input and output channels of 64.

3. The lightweight remote sensing image super-resolution reconstruction method according to claim 1, characterized in that, The feature reconstruction part includes nearest neighbor interpolation multiplied by 2, a convolutional layer with a convolutional kernel of 3×3 and input and output channels of 64 and 24 respectively, a convolutional layer with a convolutional kernel of 3×3 and input and output channels of 24 and 24 respectively, and a LeakyReLu non-linear activation layer connected in sequence.

4. The lightweight remote sensing image super-resolution reconstruction method according to claim 1, characterized in that, The constructing the high-definition image set according to the high-resolution remote sensing image data includes: Perform preprocessing on the high-resolution remote sensing image data; the preprocessing includes horizontal flipping, vertical flipping, and / or normalization processing.

5. The lightweight remote sensing image super-resolution reconstruction method according to claim 1, characterized in that, The performing random haze processing on each high-resolution remote sensing image in the high-definition image set includes: Perform random haze processing on each high-resolution remote sensing image in the high-definition image set according to the following standard optical model: ； In the formula, is the synthesized foggy image, is the coordinate value of the image pixel, is the original image to be fogged, is the global atmospheric light component, is the transmittance.

6. The lightweight remote sensing image super-resolution reconstruction method according to claim 1, characterized in that, The using the low-resolution images to train the lightweight image super-resolution reconstruction network includes: Input the low-resolution images into the lightweight image super-resolution reconstruction network to obtain reconstructed images; Calculate the reconstructed images and the corresponding high-resolution images in the high-definition image set through the L1 loss function, calculate the backward gradient of the network according to the obtained loss function value, and update the network model parameters according to the obtained gradient information. Through continuous iteration, optimize the network model.

7. A lightweight remote sensing image super-resolution reconstruction system, characterized in that, It includes: A high-definition image set construction module, which is used to obtain high-resolution remote sensing image data and construct a high-definition image set according to the high-resolution remote sensing image data; An image processing module, which is used to perform random haze processing on each high-resolution remote sensing image in the high-definition image set, and then perform interpolation processing to obtain corresponding low-resolution images; A reconstruction network construction module, which is used to construct a lightweight image super-resolution reconstruction network; the lightweight image super-resolution reconstruction network includes a low-frequency reconstruction module and a high-frequency reconstruction module; the low-frequency reconstruction module is used for reconstructing image low-frequency data, which is composed of bicubic interpolation; the high-frequency reconstruction module is used for reconstructing image high-frequency data, which is a neural network structure, including a shallow feature extraction part, a deep residual feature extraction part, and a feature reconstruction part; The feature reconstruction part adopts a lightweight reconstruction structure, which is mainly composed of nearest neighbor interpolation, a convolutional layer, and a non-linear activation layer; A reconstruction network training module, which is used to train the lightweight image super-resolution reconstruction network with the low-resolution images to obtain a trained lightweight remote sensing image super-resolution reconstruction model; The shallow feature extraction part is mainly composed of a convolutional layer; the deep residual feature extraction part is mainly composed of RCAB modules. The overall structure of the RCAB module adopts a residual form. The RCAB module includes a first convolutional layer, a LeakyReLu non-linear activation layer, a second convolutional layer, and an HCA high-frequency perception channel attention module connected in sequence. The HCA high-frequency perception channel attention module is mainly composed of global average pooling, solving global variance, a first linear layer, a LeakyReLu non-linear activation layer, a second linear layer, and a Sigmoid non-linear activation layer.

8. A lightweight remote sensing image super-resolution reconstruction device, characterized in that, It includes: A memory, which is used to store instructions; wherein, the instructions are used to implement the lightweight remote sensing image super-resolution reconstruction method according to any one of claims 1-6; A processor, which is used to execute the instructions in the memory.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, it implements the lightweight remote sensing image super-resolution reconstruction method according to any one of claims 1-6.

Images