Remote sensing image super-resolution deep learning method and device

Abstract

The application relates to a remote sensing image super-resolution deep learning method and device. First, Sentinel-2 images and Planet satellite images of the same time and place are acquired, and the Sentinel-2 images are up-sampled. Then, the up-sampled Sentinel-2 images and the Planet satellite images are concatenated to form labels as inputs of a deep learning network model, and the deep learning network model is trained. Finally, spectral loss SAM is calculated by using the up-sampled Sentinel-2 images and the output of the deep learning network model, spatial loss UIQI is calculated by using the Planet satellite images and the output of the deep learning network model, and then the network is returned through back propagation. After the network model converges, the output of the network model is a super-resolved image. By concatenating the original high-spatial-resolution image and the up-sampled image to be super-resolved as labels and by controlling the spatial loss and the spectral loss of the network output, the network output can improve the spatial resolution while keeping the spectral distortion small.

Description

Technical Field

[0001] This invention relates to the field of remote sensing mapping technology, and specifically to a method and apparatus for super-resolution deep learning of remote sensing images using spatial loss function and spectral loss function. Background Technology

[0002] Remote sensing imagery is increasingly used in industrial and agricultural production, military reconnaissance and strike operations, urban planning, and resource exploration. Simultaneously, with the continuous development of information and communication technologies, more and more satellites are being launched, resulting in a massive increase in the amount of remote sensing imagery available. However, due to limitations in sensors, acquiring remote sensing images with high spatial resolution and hyperspectral density is extremely difficult. Currently, commonly used satellites carry different sensors to collect multimodal data, which is then fused to obtain the required high spatial resolution and hyperspectral imagery.

[0003] Traditional fusion methods are generally divided into component substitution and multi-resolution analysis. Component substitution has the advantage of rich spatial details in the fused image, but it is prone to spectral distortion. Multi-resolution analysis can maintain good spectral information, but sometimes loses some spatial details in the fused image. Furthermore, traditional methods require both high spatial resolution and hyperspectral images simultaneously, but acquiring both types of images simultaneously is not easy in practical production applications.

[0004] Image super-resolution reconstruction (SR), or super-resolution for short, refers to the reconstruction of a corresponding high-resolution image from an observed low-resolution image. It is applied in fields such as satellite remote sensing, where high spatial resolution hyperspectral images can be obtained by creating simulated training sets without simultaneously requiring both high spatial resolution and hyperspectral images. Therefore, the super-resolution concept has significant practical application value.

[0005] Traditional super-resolution techniques have been maturely applied in operational use. However, in the field of remote sensing, these techniques fail to meet operational needs in many aspects. First, compared to RGB images, remote sensing images have more bands and richer spectral information, which is crucial for some operational needs. Traditional super-resolution methods are prone to spectral distortion, negatively impacting operational applications. Second, traditional methods struggle to address issues such as missing prior information and over-reliance on degradation estimation models, leading to noise and loss of high-frequency information in reconstructed images. Deep learning-based methods can effectively learn the correlations between bands and achieve good reconstruction results; however, they lack real high-spatial-resolution images as references. To learn the mapping from LR to HR, simulated datasets need to be created according to the Wald protocol. This method does not involve learning from real images and ignores the differences in spectral and spatial details between simulated data and real LR images.

[0006] Furthermore, when using deep learning algorithms for super-resolution of remote sensing data, the loss function often uses mean squared error (MSE). MSE is simple to calculate and fits the data well, but it penalizes large errors strongly but weakly for small errors, sometimes ignoring the influence of the image content itself. Additionally, its partial derivatives are very small when the output probability value is close to 0 or close to 1, which may cause the partial derivative values ​​to disappear at the beginning of model training. To prevent overfitting, sparsity terms and regularization terms are sometimes added to the loss function, such as the l0-norm penalty function, l1-norm penalty function (parameter sparsity penalty function), and l2-norm penalty function (weight decay penalty function).

[0007] The loss function described above measures the difference between the target value and the true value and attempts to minimize this difference. However, in reality, ground truth images are often unavailable as reference images. A common approach is to create a simulated training set. This method is feasible, but it learns a mapping between simulated and real images, which differs somewhat from the true LR-HR mapping. Summary of the Invention

[0008] This invention addresses the shortcomings of most existing deep learning algorithms, which use simple loss functions and fail to consider the spatial and spectral characteristics of remote sensing images. By incorporating remote sensing image quality evaluation functions, a loss function is designed that can simultaneously control the spectral and spatial losses of the network output. Through comparative experiments, a simpler network model and suitable sample size are selected, and a deep learning method for remote sensing image super-resolution using both spatial and spectral loss functions is designed. This provides technical support for rapidly acquiring high spatial resolution remote sensing image data with rich spatial details and low spectral distortion.

[0009] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0010] In a first aspect, the present invention provides a remote sensing image super-resolution deep learning method, comprising:

[0011] S1, acquire Sentinel-2 imagery and Planet satellite imagery at the same time and location, and upsample the Sentinel-2 imagery to make it have the same spatial resolution as the Planet satellite imagery;

[0012] S2, the upsampled Sentinel-2 image and Planet satellite image are concatenated to form labels, which are then used as input to the deep learning network model for training.

[0013] S3 calculates the spectral loss SAM using the upsampled Sentinel-2 imagery and the output of the deep learning network model, and calculates the spatial loss UIQI using Planet satellite imagery and the output of the deep learning network model. Then, the results are returned to the network via backpropagation. Once the network model converges, its output is the super-resolution imagery.

[0014] Furthermore, step S3 also includes weighted summation of UIQI and SAM to obtain the final loss function.

[0015] Furthermore, the spatial loss UIQI is calculated using the following formula:

[0016]

[0017] In the formula, x and y are the output of the deep learning network model and the Planet satellite image, respectively, and σ xy Let σ be the covariance of x and y. x 2 and σ y 2 Let x and y be the variances, respectively. and Let x and y be the mean values, respectively. The spatial loss UIQI ranges from [-1, 1], with an optimal value of 1.

[0018] The spectral loss SAM is calculated using the following formula:

[0019]

[0020] In the formula, x and z are the output of the deep learning network model and the upsampled Sentinel-2 image, respectively, with an optimal value of 0.

[0021] Furthermore, the deep learning network model adopts a three-layer convolutional neural network similar to SRCNN, wherein the number of filters and the filter kernel size of the feature extraction layer, nonlinear mapping layer, and reconstruction layer are 64, 32, 4 and 9, 5, 5, respectively.

[0022] Furthermore, step S2 includes:

[0023] S201, which concatenates upsampled Sentinel-2 images and Planet satellite images to form labels;

[0024] S202, the label is segmented to obtain image blocks of a preset size, and the image blocks are used as input to a deep learning network model for training.

[0025] Furthermore, when segmenting the label, the overlap rate of adjacent image blocks is 50%, and the image block size is 64×64.

[0026] Furthermore, after acquiring Sentinel-2 and Planet satellite images at the same time and location, preprocessing operations are performed on the Sentinel-2 and Planet satellite images, including radiometric calibration, atmospheric correction, registration, and data augmentation.

[0027] In a second aspect, the present invention provides a remote sensing image super-resolution deep learning device, comprising:

[0028] The data acquisition and sampling module acquires Sentinel-2 images and Planet satellite images at the same time and location, and upsamples the Sentinel-2 images to make them have the same spatial resolution as the Planet satellite images.

[0029] The label creation and model training module uses the concatenation of upsampled Sentinel-2 images and Planet satellite images to form labels, which are then used as input to the deep learning network model for training.

[0030] The loss calculation module calculates the spectral loss SAM using the upsampled Sentinel-2 imagery and the output of the deep learning network model, and calculates the spatial loss UIQI using Planet satellite imagery and the output of the deep learning network model. Then, the results are returned to the network through backpropagation. After the network model converges, its output is the super-resolution imagery.

[0031] Thirdly, the present invention provides an electronic device, comprising:

[0032] Memory, used to store computer software programs;

[0033] A processor is used to read and execute the computer software program, thereby implementing the remote sensing image super-resolution deep learning method described in the first aspect of the present invention.

[0034] Fourthly, the present invention provides a non-transitory computer-readable storage medium, characterized in that the storage medium stores a computer software program for implementing a remote sensing image super-resolution deep learning method as described in the first aspect of the present invention.

[0035] The beneficial effects of this invention are:

[0036] 1. This invention provides technical support for rapidly acquiring high spatial resolution remote sensing images with low spectral distortion.

[0037] 2. The newly designed label creation method allows the label to include both the original high spatial resolution image and the upsampled image to be super-resolved. Previous super-resolution schemes simply used the original image to be super-resolved as the label. This technology concatenates the original high spatial resolution image and the upsampled image to be super-resolved as the label, enabling the label to simultaneously possess rich spatial detail and spectral information.

[0038] 3. The training set of this method is directly made from the original image to be super-resolved. In the past, the training set of super-resolved schemes required filtering and sampling the original image to be super-resolved according to the Wald protocol. This method simplifies the training set production process and saves manpower to a certain extent.

[0039] 4. The loss function design is more reasonable. Compared with the mean square error commonly used in previous schemes, this method takes into account the spatial and spectral characteristics of remote sensing images. By writing spatial and spectral loss functions, it simultaneously controls the spatial and spectral losses of the network output. This allows the network output to improve spatial resolution while minimizing spectral distortion. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of a remote sensing image super-resolution deep learning method provided in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of label manufacturing provided in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the loss function calculation process provided in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the structure of a remote sensing image super-resolution deep learning device provided in an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of an embodiment of the electronic device provided in this invention;

[0045] Figure 6 This is a schematic diagram of an embodiment of a computer-readable storage medium provided in this invention. Detailed Implementation

[0046] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0047] Figure 1 This is a schematic diagram illustrating a remote sensing image super-resolution deep learning method according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0048] S1. Acquire Sentinel-2 imagery and Planet satellite imagery at the same time and location, and upsample the Sentinel-2 imagery to make it have the same spatial resolution as the Planet satellite imagery.

[0049] This step affects the acquisition of accurate data. Sentinel-2 and Planet images with a spatial resolution of 10m were acquired from the U.S. Geological Survey (USGS) website and the Planet website, respectively. Preprocessing was performed on the images, including radiometric calibration, atmospheric correction, and registration. Then, the 10m Sentinel-2 images were upsampled to 3m.

[0050] To facilitate storage and improve efficiency in subsequent image processing, the preprocessed large-scale image is divided into medium-sized image blocks, making it easier for manual inspection of the image for outliers and null values. Image pairs containing outliers are then removed.

[0051] After examining the preprocessed images, select images with appropriate features to create a training set. The training set should include as many features as possible, such as urban, farmland, mountain, ocean, lake, and forest features. Furthermore, select images of the same feature from different time periods to create the training set to improve the model's generalization ability. After selecting appropriate features, perform data augmentation operations such as flipping and rotating them to expand the number of samples in the training set.

[0052] S2 involves concatenating upsampled Sentinel-2 and Planet satellite images to create labels, which are then used as input to train the deep learning network model. The label creation process is as follows: Figure 2 As shown.

[0053] Specifically, after concatenating upsampled Sentinel-2 images and Planet satellite images to form labels, these labels are segmented to obtain image patches of a preset size. These image patches are then used as input to a deep learning network model for training. Comparative verification shows that when segmenting the labels, an overlap rate of 50% between adjacent image patches and an image patch size of 64×64 pixels yields relatively good model training results.

[0054] Due to the complexity of calculating the loss function, the network structure used in this invention cannot be too deep to avoid NAN values ​​during training. Therefore, the network model of this invention adopts a three-layer convolutional neural network similar to SRCNN, wherein the number of filters and the filter kernel size of the feature extraction layer, nonlinear mapping layer, and reconstruction layer are 64, 32, and 4, and 9, 5, and 5, respectively.

[0055] S3 calculates the spectral loss SAM using the upsampled Sentinel-2 imagery and the output of the deep learning network model, and calculates the spatial loss UIQI using Planet satellite imagery and the output of the deep learning network model. Then, the results are returned to the network via backpropagation. Once the network model converges, its output is the super-resolution imagery.

[0056] The spatial loss UIQI is calculated using the following formula:

[0057]

[0058] In the formula, x and y are the output of the deep learning network model and the Planet satellite image, respectively, and σ xy Let σ be the covariance of x and y. x 2 and σ y 2 Let x and y be the variances, respectively. and Let x and y be the mean values, respectively. The spatial loss UIQI ranges from -1 to 1, with an optimal value of 1. The closer the UIQI is to 1, the less spatial structure distortion there is between the network output and the Planet image.

[0059] The spectral loss SAM is calculated using the following formula:

[0060]

[0061] In the formula, x and z are the output of the deep learning network model and the upsampled Sentinel-2 image, respectively; <·,·> denote the inner product; and ||·||2 denotes the vector L2 norm. The optimal value of SAM is 0. SAM measures the similarity of spectral information between two images; the smaller the SAM, the less spectral distortion in the network output.

[0062] To ensure that the spatial loss and spectral loss values ​​follow the same trend when calculating the loss, UIQI and SAM are weighted and summed to obtain the final loss function. Generally, the weights differ for different regions, land cover types, and training stages. For example, the weight coefficient of SAM can be configured to 0.7, and the weight coefficient of UIQI can be configured to -0.3. In this embodiment, the weight coefficients of UIQI and SAM are -1 and 1, respectively. The loss function calculation process is as follows... Figure 3 As shown.

[0063] Figure 4 This is a schematic diagram of a remote sensing image super-resolution deep learning device provided in an embodiment of the present invention. Figure 4 As shown, the device includes:

[0064] The data acquisition and sampling module acquires Sentinel-2 images and Planet satellite images at the same time and location, and upsamples the Sentinel-2 images to make them have the same spatial resolution as the Planet satellite images.

[0065] The label creation and model training module uses the concatenation of upsampled Sentinel-2 images and Planet satellite images to form labels, which are then used as input to the deep learning network model for training.

[0066] The loss calculation module calculates the spectral loss SAM using the upsampled Sentinel-2 imagery and the output of the deep learning network model, and calculates the spatial loss UIQI using Planet satellite imagery and the output of the deep learning network model. Then, the results are returned to the network through backpropagation. After the network model converges, its output is the super-resolution imagery.

[0067] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 5 As shown, this embodiment of the invention provides an electronic device 500, including a memory 510, a processor 520, and a computer program 511 stored in the memory 520 and executable on the processor 520. When the processor 520 executes the computer program 511, it performs the following steps:

[0068] S1, acquire Sentinel-2 imagery and Planet satellite imagery at the same time and location, and upsample the Sentinel-2 imagery to make it have the same spatial resolution as the Planet satellite imagery;

[0069] S2, the upsampled Sentinel-2 image and Planet satellite image are concatenated to form labels, which are then used as input to the deep learning network model for training.

[0070] S3 calculates the spectral loss SAM using the upsampled Sentinel-2 imagery and the output of the deep learning network model, and calculates the spatial loss UIQI using Planet satellite imagery and the output of the deep learning network model. Then, the results are returned to the network via backpropagation. Once the network model converges, its output is the super-resolution imagery.

[0071] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by an embodiment of the present invention. For example... Figure 6As shown, this embodiment provides a computer-readable storage medium 600, on which a computer program 611 is stored. When the computer program 611 is executed by a processor, it performs the following steps:

[0072] S1, acquire Sentinel-2 imagery and Planet satellite imagery at the same time and location, and upsample the Sentinel-2 imagery to make it have the same spatial resolution as the Planet satellite imagery;

[0073] S2, the upsampled Sentinel-2 image and Planet satellite image are concatenated to form labels, which are then used as input to the deep learning network model for training.

[0074] S3 calculates the spectral loss SAM using the upsampled Sentinel-2 imagery and the output of the deep learning network model, and calculates the spatial loss UIQI using Planet satellite imagery and the output of the deep learning network model. Then, the results are returned to the network via backpropagation. Once the network model converges, its output is the super-resolution imagery.

[0075] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0076] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0081] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A remote sensing image super-resolution deep learning method, characterized in that, include: S1. Acquire Sentinel-2 imagery and Planet satellite imagery at the same time and location. Perform preprocessing operations on the Sentinel-2 imagery and Planet satellite imagery. The preprocessing operations include radiometric calibration, atmospheric correction, registration, and data augmentation. Upsample the Sentinel-2 imagery to make it have the same spatial resolution as the Planet satellite imagery. S2, the upsampled Sentinel-2 image and Planet satellite image are concatenated to form labels, which are then used as input to the deep learning network model for training. S3 calculates the spectral loss SAM using the upsampled Sentinel-2 imagery and the output of the deep learning network model, and calculates the spatial loss UIQI using Planet satellite imagery and the output of the deep learning network model. The weights of SAM and UIQI are configured according to different regions and land cover types. The UIQI is weighted and summed with SAM to obtain the final loss function, which is then returned to the network through backpropagation. Once the network model converges, its output is the super-resolution image. Step S2 includes: S201, which concatenates upsampled Sentinel-2 images and Planet satellite images to form labels; S202, the label is segmented to obtain image blocks of a preset size, with an overlap rate of 50% between adjacent image blocks, and the image blocks are used as input to a deep learning network model for training.

2. The method of claim 1, wherein, The spatial loss UIQI is calculated using the following formula: In the formula, and These are the output of a deep learning network model and Planet satellite imagery, respectively. for and covariance, and They are respectively and variance and They are respectively and The mean value of the spatial loss UIQI is [-1, 1], and its optimal value is 1. The spectral loss SAM is calculated using the following formula: In the formula, and These are the output of the deep learning network model and the upsampled Sentinel-2 image, respectively, with an optimal value of 0.

3. The method according to claim 1, characterized in that, The deep learning network model adopts a three-layer convolutional neural network similar to SRCNN, in which the number of filters and the filter kernel size of the feature extraction layer, nonlinear mapping layer and reconstruction layer are 64, 32, 4 and 9, 5, 5 respectively.

4. The method according to claim 1, characterized in that, When segmenting the label, the image block size is 64×64.

5. A remote sensing image super-resolution deep learning device, characterized in that, include: The data acquisition and sampling module acquires Sentinel-2 images and Planet satellite images at the same time and location, performs preprocessing operations on the Sentinel-2 images and Planet satellite images, including radiometric calibration, atmospheric correction, registration, and data augmentation, and upsamples the Sentinel-2 images to make them have the same spatial resolution as the Planet satellite images. The label creation and model training module uses the concatenation of upsampled Sentinel-2 images and Planet satellite images to form labels, which are then used as input to the deep learning network model for training. The loss calculation module calculates the spectral loss (SAM) using upsampled Sentinel-2 imagery and the output of the deep learning network model, and the spatial loss (UIQI) using Planet satellite imagery and the output of the deep learning network model. Weights for SAM and UIQI are configured according to different regions and land cover types. The UIQI is then weighted and summed with SAM to obtain the final loss function, which is then returned to the network via backpropagation. Once the network model converges, its output is the super-resolution image. The tag creation and model training module is specifically used for: Upsampled Sentinel-2 images and Planet satellite images are concatenated to form labels; The label is segmented to obtain image blocks of a preset size, with an overlap rate of 50% between adjacent image blocks. These image blocks are then used as input to a deep learning network model for training.

6. An electronic device, characterized in that, include: Memory, used to store computer software programs; A processor is configured to read and execute the computer software program, thereby implementing the remote sensing image super-resolution deep learning method according to any one of claims 1-4.

7. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores a computer software program for implementing the remote sensing image super-resolution deep learning method according to any one of claims 1-4.

Images