Cross-platform remote sensing image super-resolution reconstruction method and system
By constructing a cross-platform GAN network and using UAV and satellite imagery data for pre-training and fine-tuning, the problems of large-scale observation by UAV remote sensing and fine-tuning of field scale by satellite remote sensing were solved, realizing the reconstruction of high-resolution images and support for large-scale remote sensing data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENYANG AGRI UNIV
- Filing Date
- 2026-02-04
- Publication Date
- 2026-05-15
AI Technical Summary
Unmanned aerial vehicle (UAV) remote sensing is difficult to conduct large-scale observations, satellite remote sensing cannot perform fine monitoring at the field scale, and existing technologies cannot achieve high-resolution image reconstruction across platforms.
By constructing a cross-platform GAN network, using UAV and satellite imagery data for pre-training and refined training, adding a residual module with PN layers, and performing layer-by-layer feature extraction and decoding in the generator and discriminator, the mapping pattern learning from low resolution to high resolution is realized, generating a 16x super-resolution reconstruction model.
It enables the reconstruction of large-scale, high-resolution imagery, solves the problem of missing images from UAV image stitching, reduces costs, and provides high-precision, large-area remote sensing data support, serving fields such as agricultural remote sensing, surveying and mapping, and emergency management.
Smart Images
Figure CN122048668A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of remote sensing monitoring technology, specifically relating to a cross-platform method and system for super-resolution reconstruction of remote sensing images. Background Technology
[0002] The precise monitoring of crop growth through remote sensing has a significant impact on global food security and economic development. With the development of remote sensing technology and the improvement of hardware, monitoring crops using various sensing devices mounted on drones and satellite platforms has become the main method.
[0003] Existing satellite remote sensing offers the advantage of large-scale imaging and is widely used in crop retrieval, monitoring, and yield estimation. However, multispectral satellites, widely used for agricultural monitoring, often have low resolution, and acquiring high-resolution imagery is costly, hindering precise field observation. With the rapid development of drone technology, it has become an important tool in smart agriculture. Drone remote sensing technology, equipped with multiple sensors such as visible light, near-infrared, thermal infrared, and lidar, is a crucial component of agricultural remote sensing, enabling the acquisition of massive amounts of near-ground farmland remote sensing information. Compared to satellite remote sensing, which can achieve detailed monitoring at the field level, its detection range is typically limited by the aircraft's battery power and flight altitude, preventing drones from acquiring large-area crop growth data at a low cost. Given the limitations of single-mode monitoring, it is necessary to leverage the unique advantages of both drone imagery and traditional satellite remote sensing imagery to address these shortcomings and achieve refined and large-area cross-platform super-resolution image reconstruction. Summary of the Invention
[0004] To address the challenges of large-scale observations using drones and the inability of satellite remote sensing to perform detailed field-scale monitoring in large-scale remote sensing applications, this invention provides a cross-platform method and system for super-resolution reconstruction of remote sensing images.
[0005] To achieve the above objectives, the present invention provides the following technical solution: A cross-platform method for super-resolution reconstruction of remote sensing images includes the following steps: UAV images and raw remote sensing images of farmland at the same time and location are acquired; the raw remote sensing images are downsampled to obtain a first remote sensing image; the raw remote sensing images, the first remote sensing image and the UAV images are cropped respectively, and the cropped images are geo-registered based on the image geographic coordinates. The cropped raw remote sensing images and the first remote sensing images constitute a pre-training dataset, and the cropped raw remote sensing images and the UAV images constitute a real dataset. Sixteen residual modules with PN layers are added after the convolutional layers of the original convolutional neural network to serve as the generator of the GAN network. The GAN network is initialized and trained using the pre-trained dataset to learn the mapping rules from low resolution to high resolution, resulting in a pre-trained model for 16x super-resolution reconstruction. The pre-trained model for 16x super-resolution reconstruction is then trained using the real dataset to obtain a cross-platform super-resolution reconstruction network. The remote sensing image to be processed is input into the cross-platform super-resolution reconstruction network to obtain a super-resolution reconstructed image.
[0006] Preferably, when training the pre-trained model for the 16x super-resolution reconstruction using the real dataset, the original remote sensing image is input into the pre-trained model. A residual module with PN layers extracts deep features from the original remote sensing image layer by layer, and the feature vector at each pixel location is normalized to obtain a 16x ultra-high resolution image. The 16x ultra-high resolution image and the UAV image are input into a discriminator to decode each resolution layer step by step. Each resolution layer is connected through downsampling, and then convolution and loss calculations are performed to evaluate the similarity between the 16x ultra-high resolution image and the UAV image. The parameters of the pre-trained model are optimized to obtain a cross-platform super-resolution reconstruction network.
[0007] Preferably, the residual module with PN layer includes a convolutional layer, a pixel normalization PN layer, and a ReLU layer.
[0008] Preferably, the original remote sensing image is downsampled using a bicubic downsampling method to obtain a first remote sensing image.
[0009] Preferably, the calculation method for the PN layer is as follows: ; Where x represents the image tensor, and ε takes the value of 1E. -8 , For the number of pixels, The tensor values of the original image. This represents the normalized tensor value.
[0010] This invention also provides a cross-platform remote sensing image super-resolution reconstruction system, specifically comprising: The data module is used to acquire UAV images and raw remote sensing images of farmland at the same time and location; downsample the raw remote sensing images to obtain a first remote sensing image; crop the raw remote sensing images, the first remote sensing image, and the UAV images respectively, and perform georegistration on the cropped images based on the image geographic coordinates. The cropped raw remote sensing images and the first remote sensing images constitute a pre-training dataset, and the cropped raw remote sensing images and UAV images constitute a real dataset.
[0011] The model training module is used to add 16 residual modules with PN layers after the convolutional layers of the original convolutional neural network as the generator of the GAN network; the GAN network is initialized and trained using the pre-training dataset to learn the mapping rules from low resolution to high resolution, and a pre-trained model of 16x super-resolution reconstruction is obtained; the pre-trained model of 16x super-resolution reconstruction is then trained using the real dataset to obtain a cross-platform super-resolution reconstruction network.
[0012] The reconstruction module is used to input the remote sensing image to be processed into the cross-platform super-resolution reconstruction network to obtain a super-resolution reconstructed image.
[0013] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps described in the cross-platform remote sensing image super-resolution reconstruction method.
[0014] The present invention also provides a computer-readable storage medium storing a computer program that, when loaded by a processor, can execute the steps described in the cross-platform remote sensing image super-resolution reconstruction method.
[0015] The cross-platform remote sensing image super-resolution reconstruction method provided by this invention has the following beneficial effects: This invention acquires UAV and remote sensing imagery; processes the UAV and remote sensing imagery to obtain pre-training and real datasets, generating a standardized dataset suitable for model training and reducing data costs. Sixteen residual modules with PN layers are added after the convolutional layers of the original convolutional neural network as the generator of the GAN network, enhancing the extraction and representation capabilities of deep features. The GAN network is pre-trained and refined using the pre-training and real datasets respectively, resulting in a cross-platform super-resolution reconstruction network. This yields large-area study area images with high spatiotemporal resolution, achieving cross-platform remote sensing image super-resolution reconstruction. It also addresses the problem of missing points in image stitching when acquiring large-scale UAV images, which may be due to the inability to find similar points, by using satellite data reconstruction to fill in the gaps. This reduces the time, manpower, and economic costs of continuously acquiring large-area low-altitude UAV images to a certain extent, and can further provide data support for acquiring high-precision, large-area remote sensing data in other fields. It better serves multiple fields such as agricultural remote sensing, surveying and mapping, and emergency management. Attached Figure Description
[0016] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is a flowchart illustrating a cross-platform remote sensing image super-resolution reconstruction method according to an embodiment of the present invention.
[0018] Figure 2 This is a schematic diagram of the Res-PGGAN network in an embodiment of the present invention.
[0019] Figure 3 This is a schematic diagram of the overall study area in an embodiment of the present invention.
[0020] Figure 4 This is a comparison image showing the overall reconstruction in an embodiment of the present invention; wherein, Figure 4 (a) is the original Sentinel-2 image. Figure 4 (b) is the reconstructed image. Figure 4 (c) is the original image from the drone.
[0021] Figure 5 This is an example of a portion of the real dataset images in an embodiment of the present invention; wherein, Figure 5 (a) is a remote sensing image. Figure 5 (b) is an image from a drone. Detailed Implementation
[0022] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.
[0023] Example This invention provides a cross-platform method for super-resolution reconstruction of remote sensing images, such as... Figure 1 As shown, the specific steps include: Step 1: Satellite and UAV simultaneously acquire images of the same date and location. Using experimental station A as the super-resolution reconstruction test area, the overall study area is as follows: Figure 3 As shown, the images were obtained by DJI M300RTK drones during the rice growing season, and simultaneously by Sentinel-2 satellite remote sensing images at a resolution of 10m, i.e., satellite images.
[0024] Step 2: Preprocess the UAV and remote sensing images from Step 1 to create a 16x pre-training dataset and a real dataset of satellite imagery and UAV imagery.
[0025] First, the 10-meter resolution remote sensing imagery was bicubic downsampled to 160m resolution, resulting in over 39,000 sample pairs covering various terrain features. The low-resolution imagery for each sample pair was cropped to 8 pixels × 8 pixels, and the corresponding high-resolution sample was cropped to 128 pixels × 128 pixels, used as a pre-training dataset. Georegistration was performed on the UAV imagery and the cropped 10-meter resolution remote sensing imagery using their geographic coordinates in the ENVI image processing software's image registration module. The UAV imagery was processed using the same cropping method. The cropped UAV imagery and the cropped 10-meter resolution remote sensing imagery constituted the real dataset. An example image is shown below. Figure 5 As shown.
[0026] Step 3: Construct a cross-platform super-resolution reconstruction network (i.e., Res-PGGAN network) based on GAN networks, such as... Figure 2 As shown, the constructed deep learning network was pre-trained and refined using simulated and real datasets.
[0027] The network consists of two parts: a generator and a discriminator. The inputs to the generator and discriminator are a cropped low-resolution image and a high-resolution image, respectively. Sixteen residual modules with PN layers are added to the input of the generator to extract deep features from 8×8 image patches. Each residual block consists of a convolutional layer, a pixel normalization (PN) layer, and a ReLU layer. This enhances the stability and convergence speed during high-magnification training while strengthening the extraction and representation capabilities of deep features. The specific calculation method for the PN layers is as follows:
[0028] ; Where x represents a tensor, and ε takes the value of 1E. -8 , For the number of pixels, These are the original tensor values. This represents the normalized tensor value.
[0029] A progressive training approach was used to generate 128×128 images, representing a 16x super-high resolution reconstruction. For each progressive resolution level, a three-unit structure similar to residual blocks was used, with each block consisting of a convolutional layer, an LRelu layer, and a PN layer to facilitate the deep propagation of network parameters.
[0030] The high-resolution imagery and UAV imagery output from the generator are input into the discriminator. Each resolution layer of the discriminator is decoded sequentially, and each resolution layer consists of three units, each composed of a convolutional layer and an LReLU layer. Each resolution layer is connected via downsampling. After the final decoding stage, a loss is calculated using 1×1 convolutions and Dence generation maps.
[0031] When performing high-magnification super-resolution reconstruction, the significant difference in swath width between UAV imagery and satellite imagery results in a very limited number of effective training samples. Reconstructing images based on such a small sample size easily leads to overfitting in GAN networks. To address this issue, a transfer learning approach is employed. The GAN model is pre-trained on a large scale using pre-trained data samples obtained through bicubic downsampling, resulting in a pre-trained model suitable for 16x super-resolution reconstruction. This allows the subsequent generative network to learn from more data and distributions, thus mitigating the problem caused by the small sample size.
[0032] Subsequently, the pre-trained model for 16x super-resolution reconstruction was refined using the registered real dataset to obtain a cross-platform super-resolution reconstruction network.
[0033] Step 4: Use a cross-platform super-resolution reconstruction network to reconstruct the resolution of satellite images and obtain remote sensing observation data at the decimeter or centimeter level over a wide area.
[0034] like Figure 4 As shown in the table, the proposed method achieves resolution upgrade, detail reconstruction, and style transfer. Table 1 shows the quantitative evaluation results of the comparison algorithm and the proposed method. The Res-PGGAN network achieves the best results in all three evaluation metrics: SSIM, PSNR, and UIQ. Compared to ESRGAN and PGGAN networks, the Res-PGGAN network improves SSIM accuracy by 0.0062 and 0.0039 respectively, PSNR accuracy by 0.9266 and 0.4742 respectively, and UIQ accuracy by 0.0151 and 0.0005 respectively. Quantitative comparison demonstrates that the expected results can be achieved, resulting in a 16x reconstruction of cross-platform remote sensing imagery.
[0035] Table 1. Calculation results of reconstruction indexes for the four methods This invention also provides a cross-platform remote sensing image super-resolution reconstruction system, comprising: The data module is used to acquire UAV images and raw remote sensing images of farmland at the same time and location; downsample the raw remote sensing images to obtain the first remote sensing image; crop the raw remote sensing image, the first remote sensing image and the UAV image respectively, and georegulate the cropped images based on the image geographic coordinates. The cropped raw remote sensing image and the first remote sensing image constitute a pre-training dataset, and the cropped raw remote sensing image and the UAV image constitute a real dataset.
[0036] The model training module is used to add 16 residual modules with PN layers after the convolutional layers of the original convolutional neural network as the generator of the GAN network. The GAN network is initialized and trained using a pre-training dataset to learn the mapping rules from low resolution to high resolution, resulting in a pre-trained model for 16x super-resolution reconstruction. The pre-trained model for 16x super-resolution reconstruction is then trained using a real dataset to obtain a cross-platform super-resolution reconstruction network.
[0037] The reconstruction module is used to input the remote sensing images to be processed into a cross-platform super-resolution reconstruction network to obtain super-resolution reconstructed images.
[0038] The modules in the aforementioned cross-platform remote sensing image super-resolution reconstruction system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.
[0039] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps in an embodiment of a cross-platform remote sensing image super-resolution reconstruction method. Specific implementation methods can be found in the method embodiments, and will not be repeated here.
[0040] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions on which a computer program is stored. For example, a memory containing instructions that can be executed by a processor of a computer device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the computer program is executed by the processor, it can implement the steps in an embodiment of a cross-platform remote sensing image super-resolution reconstruction method. Specific implementation methods can be found in the method embodiments, which will not be repeated here.
[0041] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0042] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0043] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0044] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0045] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims. Any simple variations or equivalent substitutions of technical solutions that can be readily obtained by those skilled in the art within the scope of the technology disclosed in the present invention are within the protection scope of the present invention.
Claims
1. A cross-platform method for super-resolution reconstruction of remote sensing images, characterized in that, Includes the following steps: UAV images and raw remote sensing images of farmland at the same time and location are acquired; the raw remote sensing images are downsampled to obtain a first remote sensing image; the raw remote sensing images, the first remote sensing image and the UAV images are cropped respectively, and the cropped images are geo-registered based on the image geographic coordinates. The cropped raw remote sensing images and the first remote sensing images constitute a pre-training dataset, and the cropped raw remote sensing images and the UAV images constitute a real dataset. Sixteen residual modules with PN layers are added after the convolutional layers of the original convolutional neural network to serve as the generator of the GAN network. The GAN network is initialized and trained using the pre-trained dataset to learn the mapping rules from low resolution to high resolution, resulting in a pre-trained model for 16x super-resolution reconstruction. The pre-trained model for 16x super-resolution reconstruction is then trained using the real dataset to obtain a cross-platform super-resolution reconstruction network. The remote sensing image to be processed is input into the cross-platform super-resolution reconstruction network to obtain a super-resolution reconstructed image.
2. The cross-platform remote sensing image super-resolution reconstruction method according to claim 1, characterized in that, When training the pre-trained model for the 16x super-resolution reconstruction using the real dataset, the original remote sensing image is input into the pre-trained model. The deep features of the original remote sensing image are extracted layer by layer through a residual module with PN layers, and the feature vector of each pixel position is normalized to obtain the 16x super-high resolution image. The 16x super-high resolution image and the UAV image are input into the discriminator to decode each resolution layer step by step. Each resolution layer is connected by downsampling, and then convolution and loss calculation are performed to evaluate the similarity between the 16x super-high resolution image and the UAV image. The parameters of the pre-trained model are optimized to obtain a cross-platform super-resolution reconstruction network.
3. The cross-platform remote sensing image super-resolution reconstruction method according to claim 1, characterized in that, The residual module with PN layers includes a convolutional layer, a pixel normalization PN layer, and a ReLU layer.
4. The cross-platform remote sensing image super-resolution reconstruction method according to claim 1, characterized in that, The original remote sensing image was downsampled using a bicubic downsampling method to obtain the first remote sensing image.
5. The cross-platform remote sensing image super-resolution reconstruction method according to claim 3, characterized in that, The specific method for calculating the PN layer is as follows: ； Where x represents the image tensor, and ε takes the value of 1E. -8 , For the number of pixels, The tensor values of the original image. This represents the normalized tensor value.
6. A cross-platform remote sensing image super-resolution reconstruction system, characterized in that, include: The data module is used to acquire UAV images and raw remote sensing images of farmland at the same time and location; downsample the raw remote sensing images to obtain a first remote sensing image; crop the raw remote sensing images, the first remote sensing images, and the UAV images respectively, and perform georegistration on the cropped images based on the image geographic coordinates. The cropped raw remote sensing images and the first remote sensing images constitute a pre-training dataset, and the cropped raw remote sensing images and the UAV images constitute a real dataset. The model training module is used to add 16 residual modules with PN layers after the convolutional layers of the original convolutional neural network as the generator of the GAN network; the GAN network is initialized and trained using the pre-training dataset to learn the mapping rules from low resolution to high resolution, and a pre-trained model of 16x super-resolution reconstruction is obtained; the pre-trained model of 16x super-resolution reconstruction is then trained using the real dataset to obtain a cross-platform super-resolution reconstruction network. The reconstruction module is used to input the remote sensing image to be processed into the cross-platform super-resolution reconstruction network to obtain a super-resolution reconstructed image.
7. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by the processor, it is able to perform the steps of the method according to any one of claims 1 to 5.