A method, equipment and medium for super-resolution reconstruction of high-fidelity remote sensing images

Abstract

This invention relates to the field of remote sensing image reconstruction, and discloses a high-fidelity super-resolution reconstruction method, device, and medium for remote sensing images. The method includes: constructing an coded information network and a super-resolution reconstruction network; inputting a low-resolution remote sensing image into the super-resolution reconstruction network to obtain a super-resolution reconstructed remote sensing image; inputting a high-resolution remote sensing image and the super-resolution reconstructed remote sensing image into the coded information network respectively, performing information encoding and information cleaning to obtain an information-cleaned high-resolution remote sensing image and an information-cleaned super-resolution reconstructed high-resolution remote sensing image; establishing a loss function for training the super-resolution reconstruction network to train the network, obtaining a trained super-resolution reconstruction network; and using the trained network to obtain the reconstruction result. This invention can restore high-resolution remote sensing images with high fidelity.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image reconstruction, and in particular to a method, device and medium for super-resolution reconstruction of high-fidelity remote sensing images. Background Technology

[0002] Remote sensing, as a commonly used Earth observation technique, relies heavily on optical remote sensing images, which are the mainstream imaging results and are widely used in various industries such as military, agriculture, and geological disaster monitoring. The super-resolution reconstruction of optical remote sensing images is a fundamental research direction in this field, serving as the cornerstone of other related research, such as target detection and instance segmentation.

[0003] Currently, most research on super-resolution reconstruction employs deep learning techniques, demonstrating promising results. However, this research largely focuses on pixel mapping between the ground truth and low-resolution images in the constructed dataset, often using L1 norm distance based on pixel loss. Consequently, some high-frequency information in the reconstructed remote sensing image cannot be effectively restored through pixel loss, resulting in blurry, inaccurate, and missing high-frequency information in the reconstruction results. Summary of the Invention

[0004] The purpose of this invention is to propose a high-fidelity remote sensing image super-resolution reconstruction method, device, and medium to solve the technical problem that some high-frequency information in the existing super-resolution reconstructed remote sensing images cannot be effectively restored through pixel loss, resulting in blurry, unrealistic, and missing high-frequency information in the reconstruction results.

[0005] Specifically, the present invention provides a high-fidelity remote sensing image super-resolution reconstruction method, device, and medium, the method comprising the following steps:

[0006] S1. Construct the coding information network CodeNet and the super-resolution reconstruction network UNetSR;

[0007] S2. Input the low-resolution remote sensing image LR into the super-resolution reconstruction network UNetSR to obtain the super-resolution reconstructed remote sensing image out.

[0008] S3. Input the high-resolution remote sensing image HR and the super-reconstructed remote sensing image OUT into the CodeNet coding information network for information encoding and cleaning to obtain the cleaned high-resolution remote sensing image. HR c High-resolution remote sensing images after information cleaning and super-resolution reconstruction out c ;

[0009] S4. Based on the high-resolution remote sensing images after information cleaning HRc High-resolution remote sensing images after information cleaning and super-resolution reconstruction out c Establish a loss function for training the super-resolution reconstruction network to train the super-resolution reconstruction network and obtain the trained super-resolution reconstruction network.

[0010] S5. Input the low-resolution remote sensing image to be processed into the trained super-resolution reconstruction network to obtain the final high-fidelity super-resolution reconstruction result. A storage medium is used to store instructions and data, which are loaded and executed by the processor to implement a high-fidelity remote sensing image super-resolution reconstruction method.

[0011] A high-fidelity remote sensing image super-resolution reconstruction device includes: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a high-fidelity remote sensing image super-resolution reconstruction method.

[0012] The beneficial effects provided by this invention are: it proposes an information encoding network to encode and clean the input high-resolution remote sensing image, removes relevant information that cannot be learned through pixel loss from the high-resolution remote sensing image, and retains only the relevant information that can be learned through pixel loss, thereby solving the blurring problem in the reconstruction result caused by the use of pixel loss in super-resolution reconstruction network, and can restore the high-resolution remote sensing image with high fidelity. Attached Figure Description

[0013] Figure 1 This is a simplified flowchart of the method of the present invention;

[0014] Figure 2 This is a schematic diagram of the CodeNet architecture;

[0015] Figure 3 This is a schematic diagram of the information encoding unit structure;

[0016] Figure 4 This is a schematic diagram of the information cleaning unit structure;

[0017] Figure 5 This is a schematic diagram of the corresponding NAFblock structure;

[0018] Figure 6 This is a schematic diagram of the UNetSR structure;

[0019] Figure 7 This is a schematic diagram of the results of cleaning coded information network information;

[0020] Figure 8 This is a schematic diagram of the 4x super-resolution reconstruction result of a high-fidelity remote sensing image super-resolution reconstruction method based on coded information networks;

[0021] Figure 9This is a schematic diagram of the hardware device operation according to an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0023] Before formally describing the present invention, a general description of the solution of the present invention will be given first to facilitate understanding.

[0024] Please refer to Figure 1 The present invention provides a high-fidelity remote sensing image super-resolution reconstruction method, comprising:

[0025] S1. Construct the coding information network CodeNet and the super-resolution reconstruction network UNetSR;

[0026] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the CodeNet structure;

[0027] It should be noted that the CodeNet information network mentioned in step S1 includes: an information encoding unit and an information cleaning unit.

[0028] In this invention, a high-resolution remote sensing image is input into a coding information network for information encoding and cleaning to obtain a cleaned image. The specific operations are as follows:

[0029] (1)

[0030] in, , These are the input high-resolution remote sensing image and the output cleaned remote sensing image, respectively.

[0031] Specifically, please refer to Figures 3-4 , Figure 3 This is a schematic diagram of the information encoding unit structure; Figure 4 This is a schematic diagram of the information cleaning unit structure;

[0032] The information encoding unit comprises, in sequence, a shallow feature extraction module, a downsampling module, and an information encoding module; the information encoding module consists of three consecutive 128-channel NAFblock structures.

[0033] The information cleaning unit includes, in sequence, a shallow feature extraction module, an information cleaning module, and an upsampling module; the information cleaning module consists of three consecutive 64-channel NAFblock structures.

[0034] The CodeNet encoding information network processes any input image as follows:

[0035] The input image is fed into the shallow feature extraction module of the information encoding unit for feature extraction, and is transformed into a first high-dimensional feature vector;

[0036] The first high-dimensional feature vector is processed by the downsampling module to obtain the downsampled high-dimensional feature vector;

[0037] The downsampled high-dimensional feature vectors are processed by the information encoding module to obtain the information-encoded low-resolution remote sensing image;

[0038] The low-resolution remote sensing image after information encoding is input into the shallow feature extraction module of the information cleaning unit for feature extraction, and is transformed into a second high-dimensional feature vector;

[0039] The second high-dimensional feature vector is processed by the information cleaning module to obtain the information-cleaned high-dimensional feature vector.

[0040] The vector of high-dimensional features after information cleaning is input into the upsampling module to obtain a high-resolution output remote sensing image after information cleaning.

[0041] As one embodiment, in the information encoding unit, the input high-resolution remote sensing image is first encoded to encode the distribution information contained in the high-resolution image, resulting in an information-encoded low-resolution remote sensing image. The structure of the corresponding information encoding module is as follows: Figure 2 As shown.

[0042] Specifically, the input high-resolution remote sensing image is first subjected to shallow feature extraction, which transforms it into a high-dimensional feature vector, as shown below:

[0043] (2)

[0044] in, , The input high-resolution remote sensing image and the extracted shallow high-dimensional feature vector are respectively. Conv1 is a 3*3 convolution operation with 3 input channels and 64 output channels.

[0045] Next, regarding The input is fed into the downsampling module, which reduces its spatial resolution, thus encoding spatially relevant information onto the channel. The specific operation is as follows:

[0046] (3)

[0047] in, This is a high-dimensional feature vector with 128 channels after downsampling. PixelUnshuffle is the torch.nn.PixelUnshuffle function in PyTorch, and scale is the downsampling scale, which is set to scale=4 here.

[0048] Then, the deep feature vector The information is input into the information encoding module for encoding. This process encodes the distribution information contained in the high-resolution image, resulting in an encoded low-resolution remote sensing image, as shown below:

[0049] (4)

[0050] DeepCode is a deep feature encoding module composed of three consecutive 128-channel NAFblocks. The NAFblock structure used is as follows: Figure 4 As shown; Conv2 is a 3*3 convolution operation with 128 input channels and 3 output channels; The low-resolution remote sensing image output by the information encoding module has a size of (H / scale)*(W / scale)*3.

[0051] The above details the specific processing steps of the information encoding unit. The following section details the processing steps of the information cleaning unit.

[0052] Specifically, the low-resolution remote sensing image output by the information encoding module is encoded. The input is fed into the shallow feature extraction module, which transforms it into a high-dimensional feature vector, as shown below:

[0053] (5)

[0054] in, , These are the low-resolution remote sensing image after information encoding and the extracted shallow high-dimensional feature vector, respectively. Conv3 is a 3*3 convolution operation with 3 input channels and 64 output channels.

[0055] Then, the shallow high-dimensional feature vectors The data is input into the information cleaning module, where it undergoes a cleaning process. This process removes relevant information from the low-resolution remote sensing image after information encoding that cannot be learned through pixel loss, resulting in a cleaned high-dimensional feature vector, as shown below:

[0056] (6)

[0057] InfoClean is an information cleaning module consisting of three consecutive 64-channel NAFblocks. The NAFblocks used are as follows: Figure 4 As shown; The high-dimensional feature vector after information cleaning has a size of (H / scale)*(W / scale)*64.

[0058] Next, the cleaned high-dimensional feature vector is input into the upsampling module to restore the cleaned high-dimensional feature vector to the spatial scale of the original high-resolution remote sensing image, resulting in the cleaned high-resolution remote sensing image, as shown below:

[0059] (7)

[0060] in, This is a high-resolution remote sensing image after information cleaning, with a size of H*W*3. Pixelshuffle is the torch.nn.Pixelshuffle function in PyTorch; conv4 is a 3*3 convolution operation with 64 input channels and 3*scale*scale output channels. Here, scale=4 is chosen.

[0061] The above process describes the information encoding and cleaning process of high-resolution remote sensing images by the coding information network. It removes relevant information in high-resolution remote sensing images that cannot be learned through pixel loss, and retains only the relevant information that can be learned through pixel loss.

[0062] S2. Input the low-resolution remote sensing image LR into the super-resolution reconstruction network UNetSR to obtain the super-resolution reconstructed remote sensing image out.

[0063] S3. Input the high-resolution remote sensing image HR and the super-reconstructed remote sensing image OUT into the CodeNet coding information network for information encoding and cleaning to obtain the cleaned high-resolution remote sensing image. HR c High-resolution remote sensing images after information cleaning and super-resolution reconstruction out c ;

[0064] S4. Based on the high-resolution remote sensing images after information cleaning HR c High-resolution remote sensing images after information cleaning and super-resolution reconstruction out c Establish a loss function for training the super-resolution reconstruction network to train the super-resolution reconstruction network and obtain the trained super-resolution reconstruction network.

[0065] S5. Input the low-resolution remote sensing image to be processed into the trained super-resolution reconstruction network to obtain the final high-fidelity super-resolution reconstruction result.

[0066] It should be noted that the process of high-fidelity super-resolution reconstruction using the above-mentioned coded information network will be described in detail below.

[0067] Suppose we have training data pairs (HR, LR), where HR is a high-resolution remote sensing image and LR is a low-resolution remote sensing image. The low-resolution remote sensing image is input into the super-resolution reconstruction network to obtain the super-reconstructed high-resolution remote sensing image, as shown below:

[0068] (8)

[0069] Where, out represents the high-resolution remote sensing image reconstructed by super-resolution, and UNetSR is the super-resolution reconstruction network used. It is a 4-layer UNet super-resolution reconstruction network composed of NAFblock as the basic module. The corresponding structure of NAFblock is as follows: Figure 5 As shown. The structure of UNetSR is as follows. Figure 6 As shown.

[0070] Next, the high-resolution remote sensing image HR and the super-reconstructed high-resolution remote sensing image out from the training data pair are respectively input into the CodeNet coding information network. Information encoding and information cleaning are performed on HR and out respectively, removing relevant information that cannot be learned through pixel loss, resulting in the information-cleaned high-resolution remote sensing image and the information-cleaned super-reconstructed high-resolution remote sensing image, as shown below:

[0071] (9)

[0072] (10)

[0073] in, , These are the high-resolution remote sensing image after information cleaning and the high-resolution remote sensing image after super-resolution reconstruction after information cleaning, respectively. CodeNet is the coded information network described in formula (1).

[0074] Next, the cleaned image pairs are used ( , To establish a loss function, a visual loss is mixed in to make the super-reconstructed high-resolution remote sensing image more consistent with the distribution information of the real high-resolution remote sensing images in the training data, resulting in a high-fidelity super-reconstruction image, as shown below:

[0075] (11)

[0076] Here, Using the L1 norm, VGG is the classic VGG19 feature extractor. >0、 >0 represents the weighting coefficient, which takes the value in this paper. =1、 =1.

[0077] Then, the loss function in the above formula is used to perform backpropagation of the super-resolution reconstruction network UNetSR. When the number of iterations is greater than 10^{6}, the corresponding trained model is output. Finally, the trained model is used. The inference process involves inputting low-resolution remote sensing images into... The resulting high-resolution remote sensing image is shown below:

[0078] (12)

[0079] Wherein, input is the low-resolution remote sensing image being input. This is the final high-fidelity super-resolution reconstruction result.

[0080] Please refer to Figures 7-8 , Figure 7 This is a schematic diagram of the results of cleaning coded information network information; Figure 8 This is a schematic diagram of the 4x super-resolution reconstruction result of a high-fidelity remote sensing image super-resolution reconstruction method based on coded information networks.

[0081] Figure 7 (a) in the image represents a low-resolution remote sensing image; Figure 7 (b) in the image represents a true high-resolution remote sensing image; Figure 7 (c) in the image represents a high-resolution remote sensing image reconstructed by super-resolution; Figure 7 In the image, (d) represents the actual high-resolution remote sensing image after the information network information has been cleaned. Figure 7 In the image, (e) represents the high-resolution remote sensing image reconstructed by super-resolution after cleaning with the coded information network.

[0082] Figure 8 (a) in the image is the input low-resolution remote sensing image; Figure 8 (b) in the image is a remote sensing image reconstructed by 4x super-resolution; Figure 8 (c) in the image is a real high-resolution remote sensing image;

[0083] from Figures 7-8 The results show that after super-resolution reconstruction using the coded information network, the reconstruction effect of this invention is close to that of a real high-resolution remote sensing image.

[0084] Please see Figure 9 , Figure 9 This is a schematic diagram of the hardware device in operation according to an embodiment of the present invention. The hardware device specifically includes: a high-fidelity remote sensing image super-resolution reconstruction device 401, a processor 402, and a storage medium 403.

[0085] A high-fidelity remote sensing image super-resolution reconstruction device 401: The high-fidelity remote sensing image super-resolution reconstruction device 401 implements the high-fidelity remote sensing image super-resolution reconstruction method.

[0086] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the high-fidelity remote sensing image super-resolution reconstruction method.

[0087] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the high-fidelity remote sensing image super-resolution reconstruction method.

[0088] In summary, the beneficial effects of this invention are: it proposes an information encoding network to encode and clean the input high-resolution remote sensing image, removes relevant information that cannot be learned through pixel loss from the high-resolution remote sensing image, and retains only the relevant information that can be learned through pixel loss, thereby solving the blurring problem in the reconstruction result caused by using pixel loss in super-resolution reconstruction networks, and can restore high-resolution remote sensing images with high fidelity.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A high-fidelity remote sensing image super-resolution reconstruction method, characterized in that: Includes the following steps: S1. Construct the coding information network CodeNet and the super-resolution reconstruction network UNetSR; S2. Input the low-resolution remote sensing image LR into the super-resolution reconstruction network UNetSR to obtain the super-resolution reconstructed remote sensing image out. S3. Input the high-resolution remote sensing image HR and the super-reconstructed remote sensing image OUT into the CodeNet coding information network for information encoding and cleaning to obtain the cleaned high-resolution remote sensing image. HR c High-resolution remote sensing images after information cleaning and super-resolution reconstruction out c ; S4. Based on the high-resolution remote sensing images after information cleaning HR c High-resolution remote sensing images after information cleaning and super-resolution reconstruction out c Establish a loss function for training the super-resolution reconstruction network to train the super-resolution reconstruction network and obtain the trained super-resolution reconstruction network. S5. Input the low-resolution remote sensing image to be processed into the trained super-resolution reconstruction network to obtain the final high-fidelity super-resolution reconstruction result. The CodeNet information network mentioned in step S1 includes: an information encoding unit and an information cleaning unit; The information encoding unit comprises, in sequence, a shallow feature extraction module, a downsampling module, and an information encoding module; The information encoding module consists of three consecutive 128-channel NAFblock structures; The information cleaning unit includes, in sequence, a shallow feature extraction module, an information cleaning module, and an upsampling module; the information cleaning module consists of three consecutive 64-channel NAFblock structures. The CodeNet encoding information network processes any input image as follows: The input image is fed into the shallow feature extraction module of the information encoding unit for feature extraction, and is transformed into a first high-dimensional feature vector; The first high-dimensional feature vector is processed by the downsampling module to obtain the downsampled high-dimensional feature vector; The downsampled high-dimensional feature vectors are processed by the information encoding module to obtain the information-encoded low-resolution remote sensing image; The low-resolution remote sensing image after information encoding is input into the shallow feature extraction module of the information cleaning unit for feature extraction, and is transformed into a second high-dimensional feature vector; The second high-dimensional feature vector is processed by the information cleaning module to obtain the information-cleaned high-dimensional feature vector. The vector of high-dimensional features after information cleaning is input into the upsampling module to obtain a high-resolution output remote sensing image after information cleaning.

2. The high-fidelity remote sensing image super-resolution reconstruction method as described in claim 1, characterized in that: The loss function described in step S4 is as follows: Using the L1 norm, VGG is the classic VGG19 feature extractor. >0、 >0 represents the weighting coefficient.

3. A storage medium, characterized in that: The storage medium stores instructions and data, which are loaded and executed by the processor to implement the high-fidelity remote sensing image super-resolution reconstruction method according to any one of claims 1 to 2.

4. A high-fidelity remote sensing image super-resolution reconstruction device, characterized in that: include: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the high-fidelity remote sensing image super-resolution reconstruction method according to any one of claims 1 to 2.

Images