High-fidelity remote sensing image super-resolution reconstruction method, equipment and medium

Abstract

The invention relates to the field of remote sensing image reconstruction, and discloses a high-fidelity remote sensing image super-resolution reconstruction method and device and a medium, and the method comprises the steps: constructing a coding information network and a super-resolution reconstruction network; inputting the low-resolution remote sensing image into a super-resolution reconstruction network to obtain a super-resolution reconstructed remote sensing image; respectively inputting the high-resolution remote sensing image and the super-resolution reconstructed remote sensing image into a coding information network, and carrying out information coding and information cleaning to obtain a high-resolution remote sensing image after information cleaning and a super-resolution reconstructed high-resolution remote sensing image after information cleaning; establishing a loss function for training the super-resolution reconstruction network to train the super-resolution reconstruction network to obtain a trained super-resolution reconstruction network; obtaining a reconstruction result by using the trained network; according to the invention, the high-resolution remote sensing image can be recovered with high fidelity.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image reconstruction, and in particular to a high-fidelity remote sensing image super-resolution reconstruction method, equipment and medium. Background Art

[0002] Remote sensing is a common Earth observation technology, with optical remote sensing images being the mainstream imaging result. They are widely used in various industries, including agriculture and geological disaster monitoring. Super-resolution reconstruction of optical remote sensing images is a fundamental research direction in this field, serving as the cornerstone for other related research, such as object detection and instance segmentation.

[0003] Currently, research on super-resolution reconstruction primarily employs deep learning techniques, demonstrating promising results. However, much of this research focuses on pixel mapping between ground truth and low-resolution images in the constructed datasets, often using the L1-norm distance with pixel loss. Consequently, some high-frequency information in super-reconstructed remote sensing images cannot be effectively restored using pixel loss, resulting in blurry, unrealistic, and missing high-frequency information in the reconstruction results. Summary of the Invention

[0004] The purpose of the present invention is to propose a high-fidelity remote sensing image super-resolution reconstruction method, equipment and medium to solve the technical problem that some high-frequency information in the existing remote sensing images after super-resolution reconstruction cannot be effectively restored through pixel loss, resulting in blurred, unrealistic and missing high-frequency information in the reconstruction results.

[0005] Specifically, the present invention provides a high-fidelity remote sensing image super-resolution reconstruction method, device and medium, the method comprising the following steps: S1. Construct the coding information network CodeNet and the super-resolution reconstruction network UNetSR; S2, input the low-resolution remote sensing image LR into the super-resolution reconstruction network UNetSR to obtain the super-resolution reconstructed remote sensing image out; S3, input the high-resolution remote sensing image HR and the super-resolution reconstructed remote sensing image out into the coding information network CodeNet respectively, perform information encoding and information cleaning, and obtain the high-resolution remote sensing image after information cleaning HR c Super-resolution reconstruction of high-resolution remote sensing images after information cleaning out c ; S4. High-resolution remote sensing images after information cleaning HR c Super-resolution reconstruction of high-resolution remote sensing images after information cleaning out cEstablishing a loss function for super-resolution reconstruction network training to train the super-resolution reconstruction network and obtain a trained super-resolution reconstruction network; S5. Input the low-resolution remote sensing image to be processed into the trained super-resolution reconstruction network to obtain a final reconstructed high-fidelity super-resolution reconstruction result. A storage medium stores instructions and data for implementing a high-fidelity remote sensing image super-resolution reconstruction method.

[0006] A high-fidelity remote sensing image super-resolution reconstruction device comprises: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a high-fidelity remote sensing image super-resolution reconstruction method.

[0007] The beneficial effects provided by the present invention are: proposing a coding information network to encode and clean information of the input high-resolution remote sensing image, removing relevant information in the high-resolution remote sensing image that cannot be learned through pixel loss, and retaining only relevant information that can be learned through pixel loss, thereby solving the ambiguity problem in the reconstruction result caused by the use of pixel loss in the super-resolution reconstruction network, and being able to restore the high-resolution remote sensing image with high fidelity. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a simple flow chart of the method of the present invention; Figure 2 This is a schematic diagram of the CodeNet structure; Figure 3 It is a schematic diagram of the information coding unit structure; Figure 4 It is a structural diagram of the information cleaning unit; Figure 5 This is the corresponding structural diagram of NAFblock; Figure 6 This is a schematic diagram of the UNetSR structure; Figure 7 It is a schematic diagram of the results of cleaning the coded information network; Figure 8 This is a schematic diagram of the 4x super-resolution reconstruction results of the high-fidelity remote sensing image super-resolution reconstruction method based on the coded information network; Figure 9 It is a schematic diagram of the working of the hardware device of an embodiment of the present invention. DETAILED DESCRIPTION

[0009] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0010] Before formally explaining the present invention, the scheme of the present invention is first generally explained for easy understanding.

[0011] Please refer to Figure 1 The present invention provides a high-fidelity remote sensing image super-resolution reconstruction method, comprising: S1. Construct the coding information network CodeNet and the super-resolution reconstruction network UNetSR; Please refer to Figure 2 , Figure 2 It is a schematic diagram of the structure of the coded information network CodeNet; It should be noted that the coded information network CodeNet in step S1 includes: an information coding unit and an information cleaning unit.

[0012] In the present invention, a high-resolution remote sensing image is input into a coding information network, and information coding and information cleaning processes are performed to obtain an image after information cleaning. The specific operations are as follows: (1) in, 、 They are respectively the input high-resolution remote sensing image and the output remote sensing image after information cleaning.

[0013] For details, please refer to Figure 3-Figure 4 , Figure 3 It is a schematic diagram of the information coding unit structure; Figure 4 It is a structural diagram of the information cleaning unit; The information encoding unit includes: a shallow feature extraction module, a downsampling module and an information encoding module connected in sequence; the information encoding module is composed of three consecutive 128-channel NAFblock structures.

[0014] The information cleaning unit includes: a shallow feature extraction module, an information cleaning module and an upsampling module connected in sequence; the information cleaning module is composed of three consecutive 64-channel NAFblock structures.

[0015] The processing process of the coding information network CodeNet for any input image is as follows: The input image is input to the shallow feature extraction module of the information encoding unit for feature extraction, and is converted into a first high-dimensional feature vector; The first high-dimensional feature vector is processed by the downsampling module to obtain a downsampled high-dimensional feature vector; The downsampled high-dimensional feature vector is processed by the information encoding module to obtain a low-resolution remote sensing image after information encoding; The low-resolution remote sensing image after information encoding is input into the shallow feature extraction module of the information cleaning unit for feature extraction, and is converted into a second high-dimensional feature vector; The second high-dimensional feature vector is processed by the information cleaning module to obtain a high-dimensional feature vector after information cleaning; The high-dimensional feature vector after information cleaning is input into the upsampling module to obtain a high-resolution output remote sensing image after information cleaning.

[0016] As an embodiment, in the information encoding unit, the input high-resolution remote sensing image is firstly encoded, and the distribution information contained in the high-resolution image is encoded to obtain a low-resolution remote sensing image after information encoding. The structure of the corresponding information encoding module is as follows: Figure 2 shown.

[0017] Specifically, we first perform shallow feature extraction on the input high-resolution remote sensing image and convert it into a high-dimensional feature vector, as shown below: (2) in, 、 They are the input high-resolution remote sensing image and the extracted shallow high-dimensional feature vector respectively. Conv1 is a 3*3 convolution operation with 3 input channels and 64 output channels.

[0018] Then, The input is sent to the downsampling module, which reduces its spatial resolution so that the spatially relevant information is encoded into the channel. The specific operations are as follows: (3) in, is a high-dimensional feature vector with 128 channels after downsampling, PixelUnshuffle is the torch.nn.PixelUnshuffle function in pytorch, and scale is the downsampling scale, which is 4 here.

[0019] Then, the deep feature vector Input it into the information encoding module to perform the information encoding process, encode the distribution information contained in the high-resolution image, and obtain the low-resolution remote sensing image after information encoding, as shown below: (4) Among them, DeepCode is a deep feature coding module composed of three consecutive 128-channel NAFblocks. The NAFblock structure used is as follows: Figure 4 As shown; Conv2 is a 3*3 convolution operation, with 128 input channels and 3 output channels; The low-resolution remote sensing image after information encoding is output by the information encoding module, and its size is (H / scale)*(W / scale)*3.

[0020] The above describes in detail the specific processing process of the information encoding unit. The following describes in detail the detailed processing process of the information cleaning unit.

[0021] Specifically, the low-resolution remote sensing image after the information encoding module outputs the information encoding Input it into the shallow feature extraction module and convert it into a high-dimensional feature vector as shown below: (5) in, 、 They are respectively the low-resolution remote sensing image after information encoding and the extracted shallow high-dimensional feature vector. Conv3 is a 3*3 convolution operation with 3 input channels and 64 output channels.

[0022] Then, the shallow high-dimensional feature vector The data is input into the information cleaning module for information cleaning. The relevant information that cannot be learned through pixel loss is removed from the low-resolution remote sensing image after information encoding. The high-dimensional feature vector after information cleaning is obtained as shown below: (6) InfoClean is an information cleaning module composed of three consecutive 64-channel NAFblocks. Figure 4 As shown; It is the high-dimensional feature vector after information cleaning, and its size is (H / scale)*(W / scale)*64.

[0023] Next, the cleaned high-dimensional feature vector is input into the upsampling module, and the cleaned high-dimensional feature vector is restored to the spatial scale of the original high-resolution remote sensing image to obtain the cleaned high-resolution remote sensing image, as shown below: (7) in, The image is a high-resolution remote sensing image after information cleaning, with a size of H*W*3. Pixelshuffle is the torch.nn.Pixelshuffle function in pytorch. conv4 is a 3*3 convolution operation with 64 input channels and 3*scale*scale output channels. Here, scale=4 is used.

[0024] The above process explains the process of information encoding and information cleaning of high-resolution remote sensing images by the coded information network, removing the relevant information in the high-resolution remote sensing images that cannot be learned through pixel loss, and only retaining the relevant information that can be learned through pixel loss.

[0025] S2, input the low-resolution remote sensing image LR into the super-resolution reconstruction network UNetSR to obtain the super-resolution reconstructed remote sensing image out; S3, input the high-resolution remote sensing image HR and the super-resolution reconstructed remote sensing image out into the coding information network CodeNet respectively, perform information encoding and information cleaning, and obtain the high-resolution remote sensing image after information cleaning HR c Super-resolution reconstruction of high-resolution remote sensing images after information cleaning out c ; S4. High-resolution remote sensing images after information cleaning HR c Super-resolution reconstruction of high-resolution remote sensing images after information cleaning out c Establishing a loss function for super-resolution reconstruction network training to train the super-resolution reconstruction network and obtain a trained super-resolution reconstruction network; S5. Input the low-resolution remote sensing image to be processed into the trained super-resolution reconstruction network to obtain the final high-fidelity super-resolution reconstruction result.

[0026] It should be noted that the process of performing high-fidelity super-resolution reconstruction using the above-mentioned coding information network is described in detail below.

[0027] Assume there is a training data pair (HR, LR), where HR is a high-resolution remote sensing image and LR is a low-resolution remote sensing image. The low-resolution remote sensing image is input into the super-resolution reconstruction network to obtain a high-resolution remote sensing image reconstructed by super-resolution, as shown below: (8) Among them, out is the high-resolution remote sensing image reconstructed by super-resolution, and UNetSR is the super-resolution reconstruction network used. It is a super-resolution reconstruction network with a 4-layer UNet structure composed of NAFblock as the basic module. The corresponding structure of NAFblock is as follows Figure 5 As shown. The structure of UNetSR is as follows Figure 6 shown.

[0028] Next, the high-resolution remote sensing image HR in the training data pair and the super-resolution reconstructed high-resolution remote sensing image out are respectively input into the coding information network CodeNet, and information encoding and information cleaning are performed on HR and out respectively, and the relevant information that cannot be learned through pixel loss is removed. The high-resolution remote sensing image after information cleaning and the high-resolution remote sensing image reconstructed by super-resolution after information cleaning are obtained, as shown below: (9) (10) in, 、 They are respectively the high-resolution remote sensing image after information cleaning and the high-resolution remote sensing image reconstructed by super-resolution after information cleaning. CodeNet is the coding information network described in formula (1).

[0029] Then, the image pair after information cleaning is used ( 、 ) to establish the loss function and mix the visual sense loss at the same time, so that the super-reconstructed high-resolution remote sensing image is more consistent with the distribution information of the real high-resolution remote sensing image in the training data, and a high-fidelity super-resolved image result is obtained, as shown below: (11) Here, is the L1 norm, VGG is the classic VGG19 feature extractor, >0, >0 is the weight coefficient, which is taken as =1, =1.

[0030] Then, the loss function of the above formula is used to perform reverse gradient propagation on the super-resolution reconstruction network UNetSR. When the number of iterations is greater than 10^{6}, the corresponding trained model is output. Finally, using the trained model Inference process, low-resolution remote sensing images are input into The high-resolution remote sensing image is obtained as follows: (12) Among them, input is the input low-resolution remote sensing image, The final reconstructed high-fidelity super-resolution result.

[0031] Please refer to Figure 7-Figure 8 , Figure 7 It is a schematic diagram of the results of cleaning the coded information network; Figure 8 This is a schematic diagram of the 4x super-resolution reconstruction results of the high-fidelity remote sensing image super-resolution reconstruction method based on the coded information network.

[0032] Figure 7 (a) in the figure represents a low-resolution remote sensing image; Figure 7 (b) in the figure represents a real high-resolution remote sensing image; Figure 7 (c) in the figure shows the high-resolution remote sensing image reconstructed by super-resolution; Figure 7 (d) in the figure represents the real high-resolution remote sensing image after cleaning the coded information network information; Figure 7(e) in the figure shows the high-resolution remote sensing image reconstructed by super-resolution after cleaning of the encoded information network.

[0033] Figure 8 (a) is the input low-resolution remote sensing image; Figure 8 (b) in the figure is a remote sensing image reconstructed by 4 times super resolution; Figure 8 (c) in the figure is a real high-resolution remote sensing image; from Figure 7-Figure 8 It can be seen from the results in that after the present invention uses the coded information network for super-resolution reconstruction, its reconstruction effect is close to the real high-resolution remote sensing image.

[0034] See Figure 9 , Figure 9 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically comprises: a high-fidelity remote sensing image super-resolution reconstruction device 401, a processor 402 and a storage medium 403.

[0035] A high-fidelity remote sensing image super-resolution reconstruction device 401: The high-fidelity remote sensing image super-resolution reconstruction device 401 implements the high-fidelity remote sensing image super-resolution reconstruction method.

[0036] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the high-fidelity remote sensing image super-resolution reconstruction method.

[0037] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the high-fidelity remote sensing image super-resolution reconstruction method.

[0038] In general, the beneficial effects of the present invention are: proposing an information coding network to perform information encoding and information cleaning on the input high-resolution remote sensing image, removing relevant information in the high-resolution remote sensing image that cannot be learned through pixel loss, and retaining only relevant information that can be learned through pixel loss, thereby solving the ambiguity problem in the reconstruction results caused by the use of pixel loss in the super-resolution reconstruction network, and can restore the high-resolution remote sensing image with high fidelity.

[0039] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A high-fidelity remote sensing image super-resolution reconstruction method, characterized by: The following steps are involved: S1. Construct the coding information network CodeNet and the super-resolution reconstruction network UNetSR; S2, input the low-resolution remote sensing image LR into the super-resolution reconstruction network UNetSR to obtain the super-resolution reconstructed remote sensing image out; S3, input the high-resolution remote sensing image HR and the super-resolution reconstructed remote sensing image out into the coding information network CodeNet respectively, perform information encoding and information cleaning, and obtain the high-resolution remote sensing image after information cleaning HR c Super-resolution reconstruction of high-resolution remote sensing images after information cleaning out c ; S4. High-resolution remote sensing images after information cleaning HR c Super-resolution reconstruction of high-resolution remote sensing images after information cleaning out c Establishing a loss function for super-resolution reconstruction network training to train the super-resolution reconstruction network and obtain a trained super-resolution reconstruction network; S5. Input the low-resolution remote sensing image to be processed into the trained super-resolution reconstruction network to obtain the final high-fidelity super-resolution reconstruction result.

2. The high-fidelity remote sensing image super-resolution reconstruction method according to claim 1, characterized in that: The coded information network CodeNet in step S1 includes: an information encoding unit and an information cleaning unit.

3. The high-fidelity remote sensing image super-resolution reconstruction method according to claim 2, characterized in that: The information encoding unit includes: a shallow feature extraction module, a downsampling module and an information encoding module connected in sequence; the information encoding module is composed of three consecutive 128-channel NAFblock structures.

4. The high-fidelity remote sensing image super-resolution reconstruction method according to claim 2, wherein: The information cleaning unit includes: a shallow feature extraction module, an information cleaning module and an upsampling module connected in sequence; the information cleaning module is composed of three consecutive 64-channel NAFblock structures.

5. A high-fidelity remote sensing image super-resolution reconstruction method according to claim 3 or 4, characterized in that: The processing process of the coding information network CodeNet for any input image is as follows: The input image is input to the shallow feature extraction module of the information encoding unit for feature extraction, and is converted into a first high-dimensional feature vector; The first high-dimensional feature vector is processed by the downsampling module to obtain a downsampled high-dimensional feature vector; The downsampled high-dimensional feature vector is processed by the information encoding module to obtain a low-resolution remote sensing image after information encoding; The low-resolution remote sensing image after information encoding is input into the shallow feature extraction module of the information cleaning unit for feature extraction, and is converted into a second high-dimensional feature vector; The second high-dimensional feature vector is processed by the information cleaning module to obtain a high-dimensional feature vector after information cleaning; The high-dimensional feature vector after information cleaning is input into the upsampling module to obtain a high-resolution output remote sensing image after information cleaning.

6. The high-fidelity remote sensing image super-resolution reconstruction method according to claim 1, wherein: The loss function in step S4 is as follows: is the L1 norm, VGG is the classic VGG19 feature extractor, >0, >0 is the weight coefficient.

7. A storage medium, characterized in that: The storage medium stores instructions and data for implementing a high-fidelity remote sensing image super-resolution reconstruction method as described in any one of claims 1 to 6.

8. A high-fidelity remote sensing image super-resolution reconstruction device, characterized by: include: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a high-fidelity remote sensing image super-resolution reconstruction method as described in any one of claims 1 to 6.

Images