Remote sensing image super-resolution reconstruction method and device, storage medium and equipment

Abstract

The application discloses a kind of remote sensing image super-resolution reconstruction method, device, storage medium and equipment, it is related to computer vision technical field, can solve the technical problem of poor remote sensing image super-resolution reconstruction effect.Method includes: obtaining sample remote sensing image and multimodal auxiliary data, extract the spectral feature, texture feature and geometric feature of preprocessed multimodal auxiliary data, generate conditional feature tensor by fusion;The forward diffusion is executed to sample remote sensing image, and generates noisy image sequence;In the process of training UNet diffusion network using noisy image sequence, conditional feature tensor is injected into the preset level of the encoder and decoder of UNet diffusion network, network parameter is optimized by multi-objective loss, and the UNet diffusion network trained is obtained;Obtain the noise map to be reconstructed, based on the UNet diffusion network trained, the noise of noise map is predicted, and the target noise is obtained, the reverse sampling is executed to noise map, and the target reconstruction image is generated by iterative denoising.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, storage medium, and device for super-resolution reconstruction of remote sensing images. Background Technology

[0002] Remote sensing images are images of the Earth's surface acquired through remote sensing technology. They contain a wealth of important geographical, environmental, and resource information and are widely used in fields such as geological exploration, agricultural management, urban planning, environmental monitoring, hydrology, and forestry management.

[0003] Currently, deep learning-based methods are used for super-resolution reconstruction, such as CNN (Convolutional Neural Network). However, these methods are insufficient in terms of structural and texture reconstruction quality, resulting in poor super-resolution reconstruction results for remote sensing images. Another method is GAN (Generative Adversarial Network), which has a high rate of artifacts, and these artifacts are concentrated at the edges and cannot be eliminated through post-processing, leading to poor super-resolution reconstruction results for remote sensing images. Summary of the Invention

[0004] In view of this, the present invention provides a method, apparatus, storage medium and device for super-resolution reconstruction of remote sensing images, which can solve the technical problem of poor super-resolution reconstruction effect of remote sensing images.

[0005] According to one aspect of the present invention, a method for super-resolution reconstruction of remote sensing images is provided, the method comprising:

[0006] Acquire sample remote sensing images and multimodal auxiliary data, and preprocess the multimodal auxiliary data, which includes multispectral images, panchromatic images, and synthetic aperture radar images;

[0007] Spectral features, texture features, and geometric features are extracted from the preprocessed multimodal auxiliary data. The spectral features, texture features, and geometric features are then fused using an image pyramid to generate a conditional feature tensor.

[0008] The sample remote sensing images are forward diffused based on a linear noise scheduling strategy, and Gaussian noise is gradually added to generate a noisy image sequence.

[0009] During the training of the UNet diffusion network using the noisy image sequence, the conditional feature tensor is injected into the preset layers of the encoder and decoder of the UNet diffusion network through an adaptive instance normalization mechanism. The network parameters are optimized through multi-objective loss to obtain the trained UNet diffusion network.

[0010] A noise map to be reconstructed is obtained. The noise is predicted based on the trained UNet diffusion network to obtain the target noise. The noise map is backsampled based on the denoising diffusion implicit model algorithm and the target noise. The target reconstructed image is generated by iterative denoising.

[0011] Preferably, the preprocessed multimodal auxiliary data includes preprocessed multispectral images, preprocessed panchromatic images, and preprocessed synthetic aperture radar images. The extraction of spectral features, texture features, and geometric features from the preprocessed multimodal auxiliary data includes:

[0012] Extract spectral features from the preprocessed multispectral image based on the band histogram;

[0013] Texture features of the preprocessed panchromatic image are extracted based on the gray-level co-occurrence matrix;

[0014] Geometric features of the preprocessed synthetic aperture radar image are extracted using the Sobel operator.

[0015] Preferably, the step of performing forward diffusion on the sample remote sensing images based on a linear noise scheduling strategy to gradually add Gaussian noise and generate a noisy image sequence includes:

[0016] The sample remote sensing image is forward diffused by linearly increasing the noise coefficient from the initial value to the target value, and Gaussian noise is gradually added to generate a noisy image sequence.

[0017] Preferably, the UNet diffusion network includes: an encoder, a bottleneck layer, a decoder, and a convolutional layer connected in sequence, wherein the decoder has the same number of layers as the encoder, the encoder layer above the bottleneck layer is skipped to the decoder layer below it, the encoder layers above the bottleneck layer are skipped to the decoder layers below it, and so on, until the last encoder layer above the bottleneck layer is skipped to the last decoder layer below it. During the training of the UNet diffusion network using the noisy image sequence, the conditional feature tensor is injected into the preset layers of the encoder and decoder of the UNet diffusion network through an adaptive instance normalization mechanism, and the network parameters are optimized through multi-objective loss to obtain the trained UNet diffusion network, including:

[0018] Based on the preset level of the encoder in the UNet diffusion network into which the conditional feature tensor is to be injected, the encoder to be injected is determined. Based on the noisy image sequence, the previous output feature of the encoder to be injected is taken as the first feature. The conditional feature tensor is injected into the first feature based on adaptive instance normalization to obtain the first injected feature. The first injected feature is input into the encoder to be injected, and downsampling is continued until the second feature output by the last encoder is input into the bottleneck layer.

[0019] Obtain the third feature extracted after the bottleneck layer, and determine the decoder to be injected according to the preset level of the decoder to be injected into the conditional feature tensor in the UNet diffusion network;

[0020] Based on the third feature, the previous output feature of the decoder to be injected is used as the fourth feature. The conditional feature tensor is injected into the fourth feature based on adaptive instance normalization to obtain the second injected feature. The second injected feature is input into the decoder to be injected and upsampling is continued until the fifth feature output by the last decoder is input into the convolutional layer to obtain the predicted image.

[0021] The pixel values ​​of the predicted image are determined as the prediction noise. Based on the noisy image sequence, the sample remote sensing image, and the prediction noise, a multi-target loss is calculated. The network parameters are optimized through the multi-target loss to obtain the trained UNet diffusion network.

[0022] Preferably, the step of calculating multi-target loss based on the noisy image sequence, the sample remote sensing image, and the predicted noise, and optimizing network parameters through the multi-target loss to obtain the trained UNet diffusion network includes:

[0023] Based on the noisy image sequence, the sample remote sensing image and the predicted noise, calculate the spatial structure loss, spectral angle mapping loss, normalized vegetation index bias loss, edge preservation loss and multimodal fusion guidance loss.

[0024] The multi-objective loss is obtained by calculating the weighted sum of the spatial structure loss, the spectral angle mapping loss, the normalized vegetation index bias loss, the edge preservation loss, and the multimodal fusion guidance loss.

[0025] Determine whether the multi-target loss is less than a first preset value. If not, adjust the network parameters until the multi-target loss is less than the first preset value to obtain the trained UNet diffusion network.

[0026] Preferably, the step of injecting the conditional feature tensor into the first feature based on adaptive instance normalization to obtain the first injected feature includes:

[0027] Calculate the first mean and first standard deviation of the first feature in each channel dimension;

[0028] Calculate the first scale factor and the first offset factor corresponding to the conditional feature tensor;

[0029] For each channel dimension, the first feature is normalized based on the first mean and the first standard deviation to obtain the first normalized feature.

[0030] Calculate the first product of the first normalized feature and the first scale factor, and calculate the sum of the first product and the first offset factor to obtain the first injected feature;

[0031] The step of injecting the conditional feature tensor into the fourth feature based on adaptive instance normalization to obtain the second injected feature includes:

[0032] Calculate the second mean and second standard deviation of the fourth feature in each channel dimension;

[0033] Calculate the second scale factor and the second offset factor corresponding to the conditional feature tensor;

[0034] For each channel dimension, the fourth feature is normalized based on the second mean and the second standard deviation to obtain the second normalized feature;

[0035] Calculate the second product of the second normalized feature and the second scale factor, and calculate the sum of the second product and the second offset factor to obtain the second injected feature.

[0036] Preferably, the step involves predicting noise in the noise map based on the trained UNet diffusion network to obtain target noise, performing backsampling on the noise map based on the denoising diffusion implicit model algorithm and the target noise, and generating a target reconstructed image through iterative denoising, including:

[0037] The noise map is input into the trained UNet diffusion network to predict the target noise at step t. Based on the denoising diffusion implicit model algorithm, the image corresponding to step t-1 is calculated according to the target noise at step t and the noise map.

[0038] The image corresponding to step t-1 is input into the trained UNet diffusion network to predict the target noise corresponding to step t-1. Based on the denoising diffusion implicit model algorithm, the image corresponding to step t-2 is calculated according to the target noise at step t-1 and the image corresponding to step t-1. The prediction and calculation are iterated until the target reconstruction image corresponding to step 0 is obtained.

[0039] According to another aspect of the present invention, a remote sensing image super-resolution reconstruction apparatus is provided, the apparatus comprising:

[0040] The preprocessing module is used to acquire sample remote sensing images and multimodal auxiliary data, and to preprocess the multimodal auxiliary data, which includes multispectral images, panchromatic images, and synthetic aperture radar images.

[0041] The fusion module is used to extract the spectral features, texture features, and geometric features of the preprocessed multimodal auxiliary data, and fuse the spectral features, texture features, and geometric features through an image pyramid to generate a conditional feature tensor.

[0042] The diffusion module is used to perform forward diffusion on the sample remote sensing images based on a linear noise scheduling strategy, gradually adding Gaussian noise to generate a noisy image sequence;

[0043] The training module is used to inject the conditional feature tensor into the preset layers of the encoder and decoder of the UNet diffusion network through an adaptive instance normalization mechanism during the training of the UNet diffusion network using the noisy image sequence, and optimize the network parameters through multi-objective loss to obtain the trained UNet diffusion network.

[0044] The generation module is used to acquire the noise map to be reconstructed, predict the noise of the noise map based on the trained UNet diffusion network to obtain the target noise, perform backsampling on the noise map based on the denoising diffusion implicit model algorithm and the target noise, and generate the target reconstructed image through iterative denoising.

[0045] According to another aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described remote sensing image super-resolution reconstruction method.

[0046] According to another aspect of the present invention, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described remote sensing image super-resolution reconstruction method.

[0047] By employing the above technical solutions, the present invention provides a method, apparatus, storage medium, and device for super-resolution reconstruction of remote sensing images. Through the technical solutions of the present invention, the conditional feature tensor corresponding to multimodal auxiliary data, which includes detailed features of spectral, textural, and geometric information, can be input into different levels of the UNet diffusion network to achieve precise guidance of structural information at different scales, thereby accurately realizing super-resolution reconstruction of remote sensing images. Through multi-target loss, the consistency of geometric and textural structures can be maintained, the physical consistency of reflectivity can be guaranteed, the spectral consistency between multiple bands can be maintained, the spatial fidelity of geometric structures can be maintained, and spectral distortion or texture misalignment can be prevented.

[0048] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of this application. In the drawings:

[0050] Figure 1 A flowchart illustrating a remote sensing image super-resolution reconstruction method provided by an embodiment of the present invention is shown.

[0051] Figure 2 A flowchart illustrating another remote sensing image super-resolution reconstruction method provided by an embodiment of the present invention is shown.

[0052] Figure 3 This diagram illustrates the structure of a remote sensing image super-resolution reconstruction device according to an embodiment of the present invention.

[0053] Figure 4 A schematic diagram of another remote sensing image super-resolution reconstruction device provided in an embodiment of the present invention is shown. Detailed Implementation

[0054] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0055] This embodiment provides a method for super-resolution reconstruction of remote sensing images, such as... Figure 1 As shown, the method includes:

[0056] 101. Acquire sample remote sensing images and multimodal auxiliary data, and preprocess the multimodal auxiliary data, which includes multispectral images, panchromatic images, and synthetic aperture radar images.

[0057] In this embodiment, as one implementation method, the sample remote sensing image can be acquired by a remote sensing sensor.

[0058] The multimodal auxiliary data includes panchromatic images, multispectral images, and synthetic aperture radar (SAR) images. Panchromatic images offer high spatial resolution, providing more detailed surface information and clearly displaying the outlines, shapes, and details of features, such as clearly distinguishing the boundaries of buildings and roads in a city. Multispectral images offer high spectral resolution, composed of multiple spectral bands, each capturing radiation within a specific wavelength range, covering multiple bands such as visible and infrared light. This provides detailed information about surface features, and by combining information from different bands, it can more comprehensively reflect surface characteristics, helping to identify different types of vegetation, soil, and water bodies. Synthetic aperture radar (SAR) images offer high image data resolution, achieving high geometric resolution in both azimuth and range, breaking through the resolution limits of classical radar and providing relatively detailed surface information.

[0059] Preprocessing of multimodal auxiliary data includes preprocessing of multispectral images, panchromatic images, and synthetic aperture radar images respectively.

[0060] The panchromatic images, multispectral images, and synthetic aperture radar images exhibit scale differences (different image resolutions), positional deviations (at the same scale, the same feature, such as a building, may have different pixel positions in two images), and noise interference (the images are affected by various noises, making them unclear). For subsequent feature fusion, preprocessing is required, including scale adaptation (addressing scale differences), geometric registration (addressing positional deviations), and noise suppression (addressing noise interference). Then, the image formats are unified, resulting in preprocessed panchromatic images, preprocessed multispectral images, and preprocessed synthetic aperture radar images.

[0061] Specifically, for scale differences, a four-layer image pyramid is used, with each layer scaling by 1, 2, 4, and 8 times the original size, covering a scale range of 4-8 times to ensure that images of different sizes can be adapted. A standard scale is determined, and panchromatic images, multispectral images, and synthetic aperture radar images are all scaled to the standard scale. For example, if the resolution of the multispectral image and the panchromatic image is both 3.2 meters, and the resolution of the synthetic aperture radar image is 12.8 meters (each pixel corresponds to 12.8 meters × 12.8 meters on the ground), the synthetic aperture radar image is magnified by 4 times (from 12.8 meters to 3.2 meters) through a bilinear interpolation algorithm, thus achieving scale consistency among the three.

[0062] Specifically, for positional deviations, the Scale-Invariant Feature Transform (SIFT) algorithm extracts invariant feature points from panchromatic images, multispectral images, and synthetic aperture radar images. These feature points remain stable under changes in scale, rotation, and illumination, thereby achieving accurate matching of panchromatic images, multispectral images, and synthetic aperture radar images under different conditions.

[0063] Specifically, regarding noise interference, synthetic aperture radar images contain speckle noise. This is addressed using a 3×3 pixel Gamma filter window. Gamma filter windowing is a technique in image processing that combines Gamma transformation with local window operations. Its core is applying Gamma correction within a local window (neighborhood) of the image to achieve adaptive image enhancement or noise suppression. Panchromatic and multispectral images are affected by haze, aerosols, and the angle of the sun, resulting in darker or blurrier images. These interferences are removed using an atmospheric correction algorithm (Fast Line-of-sight Atmospheric Analysis of Spectral Hypercubes, FLAASH).

[0064] Specifically, after scale adaptation, geometric registration and noise suppression, the intermediate panchromatic image, intermediate multispectral image and intermediate synthetic aperture radar image are obtained. The unified image format includes: (1) cropping: the intermediate panchromatic image, intermediate multispectral image and intermediate synthetic aperture radar image are all cut into small blocks of 512×512 pixels (to avoid the influence of edge pixel error), to avoid the image being too large and causing a waste of computing resources. The blocks overlap by 64 pixels so that there will be no gaps when stitching. (2) Normalization: The pixel value ranges of different types of images vary greatly (e.g., the pixel value of the intermediate synthetic aperture radar image is 0-2000, the intermediate panchromatic image and the intermediate multispectral image are 0-65535, the pixel value ranges of the intermediate synthetic aperture radar image and the intermediate panchromatic image are very different, and the pixel value ranges of the intermediate synthetic aperture radar image and the intermediate multispectral image are very different). Directly inputting them into the UNet diffusion network will cause the feature weights to become unbalanced. They need to be unified to the [0,1] interval. By normalization, the difference in the numerical range is eliminated, and the preprocessed panchromatic image, the preprocessed multispectral image and the preprocessed synthetic aperture radar image are obtained, so that the pixel values ​​of the three are on the same order of magnitude, which is convenient for the UNet diffusion network to learn features.

[0065] 102. Extract the spectral features, texture features, and geometric features of the preprocessed multimodal auxiliary data, and fuse the spectral features, texture features, and geometric features through an image pyramid to generate a conditional feature tensor.

[0066] To extract texture features from the preprocessed panchromatic image, a gray-level co-occurrence matrix is ​​calculated. Four types of texture features are extracted in four directions: 0°, 45°, 90°, and 135°: energy (reflecting texture uniformity, with higher values ​​indicating more regular texture); entropy (reflecting texture complexity, with higher values ​​indicating richer details); contrast (reflecting local gray-level differences, with higher values ​​indicating clearer edges); and correlation (reflecting texture direction consistency, with higher values ​​indicating more continuous ridge / road direction). These features are concatenated into a 4×H×W tensor and weighted using an attention mechanism network structure such as SENet. Preferably, a suppression weight of 0.3±0.05 can be assigned to noisy regions to enhance the effective texture signal.

[0067] To extract spectral features from the preprocessed multispectral images, a 128-bin spectral histogram is calculated to statistically analyze the pixel value distribution of the preprocessed multispectral images under each band. The 128-bin spectral histogram is then processed through a 3-layer 3×3 convolutional network to extract band correlations. Layer 1: 64 convolutional kernels, outputting 64×H×W features (extracting local spectral patterns in a single band); Layer 2: 128 convolutional kernels, outputting 128×H×W features (capturing low-order correlations between bands); Layer 3: 256 convolutional kernels, outputting 256×H×W features (modeling global spectral dependencies, such as the ratio of vegetation reflectance between the red and near-infrared bands).

[0068] To extract the geometric features of the preprocessed synthetic aperture radar image, the horizontal and vertical gradients are calculated based on edge detection operators such as the Sobel Operator (Sobel). The edge intensity map is then extracted based on the horizontal and vertical gradients. Thresholding is performed on the edge intensity map to preserve strong edge features such as building outlines and road boundaries, and a 1×H×W geometric feature is output.

[0069] The preprocessed texture features, preprocessed spectral features, and preprocessed geometric features are fused to obtain a conditional feature tensor. This includes downsampling high-resolution features from the preprocessed texture features, preprocessed spectral features, and preprocessed geometric features to low-resolution features, or upsampling low-resolution features to high-resolution features, through an image pyramid to achieve scale alignment. The aligned texture features, aligned spectral features, and aligned geometric features are then fused into a 256×H×W conditional feature tensor by channel splicing, where H and W are consistent with the size of the sample remote sensing image (e.g., 512×512).

[0070] The problems of scale difference, positional deviation and noise interference were solved, so that the feature alignment error of the conditional feature tensor is less than or equal to 1 pixel, which improves the accuracy of subsequent injection into the UNet diffusion network for UNet diffusion network training, so as to accurately perform resolution enhancement.

[0071] 103. Based on a linear noise scheduling strategy, perform forward diffusion on the sample remote sensing images and gradually add Gaussian noise to generate a noisy image sequence.

[0072] 104. In the process of training the UNet diffusion network using the noisy image sequence, the conditional feature tensor is injected into the preset layers of the encoder and decoder of the UNet diffusion network through an adaptive instance normalization mechanism, and the network parameters are optimized through multi-target loss to obtain the trained UNet diffusion network.

[0073] In this embodiment, the conditional feature tensor is injected into the preset level of the encoder and the preset level of the decoder of the UNet diffusion network through an adaptive instance normalization mechanism. The preset level of the encoder is used as the encoder to be injected, and the preset level of the decoder is used as the decoder to be injected. For example, the encoder to be injected is the second layer encoder and the fourth layer encoder, and the decoder to be injected is the first layer decoder and the third layer decoder.

[0074] The output of the UNet diffusion network is the predicted noise. The noisy image sequence = sample remote sensing image + real noise. The real noise is the cumulative value of Gaussian noise added step by step. As one implementation method, the difference between the real noise and the predicted noise is calculated to calculate the L1 norm. It is determined whether the result is less than a second preset value. If not, the network parameters are adjusted until the loss is less than the second preset value, and the trained UNet diffusion network is obtained.

[0075] 105. Obtain the noise map to be reconstructed, predict the noise based on the trained UNet diffusion network to obtain the target noise, perform backsampling on the noise map based on the denoising diffusion implicit model algorithm and the target noise, and generate the target reconstructed image through iterative denoising.

[0076] In this embodiment, after the trained UNet diffusion network is obtained, it can be applied. The noise map to be reconstructed is a low-resolution remote sensing image, denoted by LR. The trained UNet diffusion network can predict the target noise to be subtracted from the noise map at each step until the target reconstructed image is obtained, thus completing the super-resolution reconstruction of the remote sensing image.

[0077] This invention provides a method, apparatus, storage medium, and device for super-resolution reconstruction of remote sensing images. Through the technical solution of this invention, the conditional feature tensor corresponding to multimodal auxiliary data, which includes detailed features of spectral, textural, and geometric information, can be input into different layers of a UNet diffusion network. This allows for precise guidance of structural information at different scales, thereby accurately achieving super-resolution reconstruction of remote sensing images. By using multi-target loss, the consistency of geometric and textural structures can be maintained, ensuring physical consistency of reflectivity, maintaining spectral consistency across multiple bands, preserving the spatial fidelity of geometric structures, and preventing spectral distortion or texture misalignment.

[0078] Furthermore, as a refinement and extension of the specific implementation methods described above, and to fully illustrate the specific implementation process in this embodiment, another method for super-resolution reconstruction of remote sensing images is provided, such as... Figure 2 As shown, the method includes:

[0079] 201. Acquire sample remote sensing images and multimodal auxiliary data, and preprocess the multimodal auxiliary data, which includes multispectral images, panchromatic images, and synthetic aperture radar images.

[0080] The specific implementation steps in this embodiment are the same as those in step 101 of the embodiment, and will not be repeated here.

[0081] 202. Extract the spectral features, texture features, and geometric features of the preprocessed multimodal auxiliary data, and fuse the spectral features, texture features, and geometric features through an image pyramid to generate a conditional feature tensor.

[0082] The preprocessed multimodal auxiliary data includes preprocessed multispectral images, preprocessed panchromatic images, and preprocessed synthetic aperture radar images. The extraction of spectral, texture, and geometric features from the preprocessed multimodal auxiliary data includes: extracting spectral features from the preprocessed multispectral images based on band histograms; extracting texture features from the preprocessed panchromatic images based on the gray-level co-occurrence matrix; and extracting geometric features from the preprocessed synthetic aperture radar images based on the Sobel operator.

[0083] The specific implementation steps in this embodiment are the same as those in step 102 of the embodiment, and will not be repeated here.

[0084] 203. Based on a linear noise scheduling strategy, perform forward diffusion on the sample remote sensing images and gradually add Gaussian noise to generate a noisy image sequence.

[0085] The step of performing forward diffusion on the sample remote sensing image based on a linear noise scheduling strategy and gradually adding Gaussian noise to generate a noisy image sequence includes: performing forward diffusion on the sample remote sensing image with the noise coefficient linearly increasing from an initial value to a target value, and gradually adding Gaussian noise to generate a noisy image sequence.

[0086] For example, the initial value is 1e-4, which increases linearly to the target value 2e-2.

[0087] Specifically, Gaussian noise is added stepwise based on a Markov chain to simulate the degradation process of remote sensing images. After t-step diffusion, the sample remote sensing images yield a noisy image sequence. Wherein, the noisy image at step t. From the noisy image at step t-1 Gaussian noise is added to generate the noise; the formula is:

[0088]

[0089] in, For the noisy image at step t, For the noisy image at step t-1, The retention coefficient at step t is a preset value that controls the proportion of original data information retained. The closer the retention coefficient is to 1, the more original information is retained. It is the preset noise figure. The Gaussian noise added at step t-1 follows a normal distribution with a mean of 0 and a covariance matrix equal to the identity matrix. A linear strategy can be used, for example, at t=1000. The speckle noise of the synthetic aperture radar image in step 201 of the typical embodiment increases linearly from 1e-4 at t=1 to 2e-2 at t=1000. Starting from 1e-4 corresponding to t=1, the speckle noise of the synthetic aperture radar image can be covered, and the noise is rapidly enhanced in the later stage to simulate a severely degraded scene.

[0090] 204. In the process of training the UNet diffusion network using the noisy image sequence, the conditional feature tensor is injected into the preset layers of the encoder and decoder of the UNet diffusion network through an adaptive instance normalization mechanism, and the network parameters are optimized through multi-target loss to obtain the trained UNet diffusion network.

[0091] The UNet diffusion network includes an encoder, a bottleneck layer, a decoder, and a convolutional layer connected in sequence. The number of layers of the decoder is the same as that of the encoder. The encoder layer above the bottleneck layer is skipped to the decoder layer below it. The encoder layers above the bottleneck layer are skipped to the decoder layers below it, and so on, until the encoder layer above the bottleneck layer is skipped to the decoder layer below it.

[0092] In the process of training the UNet diffusion network using the noisy image sequence, the conditional feature tensor is injected into the preset layers of the encoder and decoder of the UNet diffusion network through an adaptive instance normalization mechanism. The network parameters are optimized through multi-objective loss to obtain the trained UNet diffusion network. This includes: determining the encoder to be injected based on the preset layers of the encoders in the UNet diffusion network into which the conditional feature tensor is to be injected; using the previous output feature of the encoder to be injected as a first feature based on the noisy image sequence; injecting the conditional feature tensor into the first feature based on adaptive instance normalization to obtain a first injected feature; inputting the first injected feature into the encoder to be injected; and continuing downsampling until the second feature output by the last encoder is input into the bottleneck layer. The process involves: obtaining the third feature extracted by the bottleneck layer; determining the decoder to be injected based on the preset level of the decoder to be injected with the conditional feature tensor in the UNet diffusion network; using the third feature as the previous output feature of the decoder to be injected, injecting the conditional feature tensor into the fourth feature based on adaptive instance normalization to obtain the second injected feature; inputting the second injected feature into the decoder to be injected and continuing upsampling until the fifth feature output by the last decoder is input into the convolutional layer to obtain the predicted image; determining the pixel values ​​of the predicted image as prediction noise; calculating the multi-target loss based on the noisy image sequence, the sample remote sensing image, and the prediction noise; optimizing the network parameters through the multi-target loss to obtain the trained UNet diffusion network.

[0093] Taking a 4-layer encoder and decoder as an example, the injection locations of the conditional feature tensor (i.e., the preset layers of the encoder and decoder) are as follows: Second layer encoder (1 / 4 resolution, 128×128 pixels): guides medium-scale structures such as road networks and farmland plots; Fourth layer encoder (1 / 16 resolution, 32×32 pixels): controls large-scale structures such as city outlines and mountain ranges; First layer decoder (1 / 2 resolution, 256×256 pixels): fine-grained optimization of fine-scale details such as building edges and roof textures; Third layer decoder (1 / 8 resolution, 64×64 pixels): assists in the recovery of medium-scale features such as vegetation edges and river boundaries.

[0094] Accordingly, the UNet diffusion network includes: a first layer encoder, a second layer encoder, a third layer encoder, a fourth layer encoder, a bottleneck layer, a first layer decoder, a second layer decoder, a third layer decoder, a fourth layer decoder, and a convolutional layer connected in sequence. The first layer decoder is also skip-connected to the fourth layer encoder, the second layer decoder is also skip-connected to the third layer encoder, the third layer decoder is also skip-connected to the second layer encoder, and the fourth layer decoder is also skip-connected to the first layer encoder. The step of training the UNet diffusion network based on the noisy image sequence, the real noise, and the conditional feature tensor to obtain the trained UNet diffusion network includes: (1) determining that the encoder to be injected is the second layer encoder and the fourth layer encoder, and using the first layer encoder to downsample the noisy image sequence to obtain a first downsampled image. The sampling features are obtained by injecting the conditional feature tensor into the first downsampled feature based on adaptive instance normalization to obtain the first downsampled injected feature. The first downsampled injected feature is then downsampled using the second layer encoder to obtain the second downsampled feature. The second downsampled feature is then downsampled using the third layer encoder to obtain the third downsampled feature. The conditional feature tensor is then injected into the third downsampled feature based on adaptive instance normalization to obtain the second downsampled injected feature. The second downsampled injected feature is then downsampled using the fourth layer encoder to obtain the fourth downsampled feature. (Correspondingly, when the encoder to be injected is the second layer encoder and the fourth layer encoder, the first feature includes the first downsampled feature and the third downsampled feature, the first injected feature includes the first downsampled injected feature and the second downsampled injected feature, and the second feature is the fourth downsampled feature.)

[0095] (2) Use the bottleneck layer to extract features from the fourth downsampling feature to obtain the bottleneck layer extracted features (correspondingly, the third feature is the bottleneck layer extracted features).

[0096] (3) Determine that the decoder to be injected is the first layer decoder and the third layer decoder. Based on adaptive instance normalization, inject the conditional feature tensor into the bottleneck layer extracted features to obtain the first upsampled injected features. Use the first layer decoder to upsample the first upsampled injected features. Concatenate the obtained first upsampled features with the fourth downsampled features to obtain the first concatenated features. Use the second layer decoder to upsample the first concatenated features. Concatenate the obtained second upsampled features with the third downsampled features to obtain the second concatenated features. Inject the conditional feature tensor into the second concatenated features based on adaptive instance normalization. The system obtains a second upsampled injected feature, then upsamples the second upsampled injected feature using the third layer decoder, concatenates the obtained third upsampled feature with the second downsampled feature to obtain a third concatenated feature, then upsamples the third concatenated feature using the fourth layer decoder, and concatenates the obtained fourth upsampled feature with the first downsampled feature to obtain a fourth concatenated feature, and inputs the fourth concatenated feature into the convolutional layer to obtain the predicted image (correspondingly, the fourth feature is the feature extracted from the bottleneck layer, the second injected feature includes the first upsampled injected feature and the second upsampled injected feature, and the fifth feature is the fourth concatenated feature).

[0097] Preferably, in order to reduce memory usage and computing power consumption, a preset proportion (e.g., 30%) of channel pruning is performed on the unconditionally injected convolutional layers in the UNet diffusion network, while retaining the channels of the second-layer encoder, the fourth-layer encoder, the first-layer decoder, and the third-layer decoder. After pruning, the number of parameters is reduced and the peak memory usage for inference is reduced.

[0098] The step of calculating multi-target loss based on the noisy image sequence, the sample remote sensing image, and the predicted noise, and optimizing network parameters through multi-target loss to obtain a trained UNet diffusion network includes: calculating spatial structure loss, spectral angle mapping loss, normalized vegetation index bias loss, edge preservation loss, and multimodal fusion guidance loss based on the noisy image sequence, the sample remote sensing image, and the predicted noise; calculating a weighted sum of the spatial structure loss, the spectral angle mapping loss, the normalized vegetation index bias loss, the edge preservation loss, and the multimodal fusion guidance loss to obtain the multi-target loss; determining whether the multi-target loss is less than a first preset value; if not, adjusting the network parameters until the multi-target loss is less than the first preset value to obtain the trained UNet diffusion network.

[0099] Wherein, the noisy image sequence = sample remote sensing image HR + real noise (real noise is the cumulative value of added Gaussian noise, for example, If Gaussian noise is added in step t-1, then the actual noise = + +...+ + ), Denoising image =Noisy image sequence - Predicted noise, spatial structure loss is used to constrain the HR of the noisy image sequence and the denoised image. Consistency in geometric and textural structure, including L1 loss and perceptual loss, specifically:

[0100] L1= This indicates a concern for HR and To calculate the L1 norm from the difference, it should be noted that... -HR = Noisy image sequence - Predicted noise + Real noise - Noisy image sequence = Real noise - Predicted noise, used to ensure overall structural consistency.

[0101] Perceived loss:

[0102] , This represents the features extracted using the m-th layer of the VGG19 network. Indicates to and The difference is used to calculate the square of the L2 norm, which is used to improve the fidelity of high-level semantic structures (such as road connectivity and building textures).

[0103] Spectral angle mapping loss , , This represents the inner product of vectors, used to control the angle error between multi-band vectors and ensure physical consistency of reflectivity.

[0104] Normalized Difference Vegetation Index (NDV) Deviation Loss , , , For the pixel values ​​in the near-infrared and red bands of the denoised image, , These are the near-infrared and red band pixel values ​​of the sample remote sensing image, used to maintain spectral consistency among multiple bands.

[0105] Edge preservation loss , = , The image gradient (edge ​​information) calculated for the edge detection operator is used to enhance the edge sharpness of targets such as buildings and roads, while maintaining the spatial fidelity of the geometric structure.

[0106] Multimodal fusion guided loss , = , Denoising images Obtained through cross-modal conversion, Denoising images Obtained through cross-modal conversion, ( ) represents the 512-dimensional feature vector extracted by ResNet-18, used to constrain the panchromatic image. With multispectral images Consistency of the fusion results in the feature space prevents spectral distortion or texture misalignment.

[0107] L= L1+ +

[0108] L represents the multi-objective loss. By using the multi-objective joint loss, the optimization and reconstruction quality of the UNet diffusion network can be comprehensively guided. It can maintain the consistency of geometric and texture structures, ensure the physical consistency of reflectivity, maintain the spectral consistency between multiple bands, maintain the spatial fidelity of the geometric structure, and prevent spectral distortion or texture misalignment.

[0109] The step of injecting the conditional feature tensor into the first feature based on adaptive instance normalization to obtain the first injected feature includes: calculating the first mean and first standard deviation of the first feature in each channel dimension; calculating the first scale factor and first offset factor corresponding to the conditional feature tensor; performing instance normalization on the first feature according to the first mean and first standard deviation for each channel dimension to obtain the first normalized feature; calculating the first product of the first normalized feature and the first scale factor; and calculating the sum of the first product and the first offset factor to obtain the first injected feature.

[0110] For each channel of the first feature, calculate the first mean and the first standard deviation of all elements in that channel. The shape of the first feature F is (C, H, W), where C is the number of channels, H is the feature map height corresponding to the first feature, and W is the feature map width corresponding to the first feature.

[0111] First mean =

[0112] First standard deviation =

[0113] Using 1 for generating scale factor The first scale factor corresponding to the conditional feature tensor of the convolutional layer is calculated using a 1-dimensional convolutional layer. Using the 1 of the generated offset factor The first offset factor corresponding to the conditional feature tensor of the convolutional layer is calculated. Among them, the two 1s The convolutional layers in step 1 are independent of each other, with different weight parameters.

[0114]

[0115]

[0116] Characteristics after the first injection:

[0117]

[0118] First normalized feature:

[0119]

[0120] The step of injecting the conditional feature tensor into the fourth feature based on adaptive instance normalization to obtain the second injected feature includes: calculating the second mean and second standard deviation of the fourth feature in each channel dimension; calculating the second scale factor and second offset factor corresponding to the conditional feature tensor; performing instance normalization on the fourth feature according to the second mean and second standard deviation for each channel dimension to obtain the second normalized feature; calculating the second product of the second normalized feature and the second scale factor; and calculating the sum of the second product and the second offset factor to obtain the second injected feature.

[0121] 205. Obtain the noise map to be reconstructed, input the noise map into the trained UNet diffusion network, predict the target noise at step t, and calculate the image corresponding to step t-1 based on the denoising diffusion implicit model algorithm and the target noise at step t and the noise map.

[0122] In this embodiment, the step of calculating the image corresponding to step t-1 based on the target noise at step t and the target low-resolution image includes: determining a first coefficient based on a preset preservation coefficient at step t; calculating a third product of the first coefficient and the target noise at step t; calculating the difference between the target low-resolution image and the third product; determining a second coefficient based on a preset preservation coefficient at step t; calculating a fourth product of the second coefficient and the difference; calculating a fifth product of a preset noise coefficient and random noise; and calculating the sum of the fourth product and the fifth product to obtain the image corresponding to step t-1.

[0123]

[0124] In the formula, To reconstruct the noise map (The noise map to be reconstructed can also be represented by LR) The image corresponding to step t-1 after stripping the target noise corresponding to step t-1 at step t-1, As the first coefficient, As the second coefficient, These are noise control parameters. The square of the pre-set hold-over coefficient at step t. The preset retention coefficient for step t. The variance of the preset noise figure. The preset noise figure, and It can be set to be the same as the forward noise addition process (adding real noise to the sample remote sensing image), for example, when t=1, When t=1000, , The cumulative noise control parameter represents the overall noise level up to step t. The target noise at step t corresponds to step t. This represents random noise, which is a standard Gaussian distributed noise and follows a standard normal distribution.

[0125] 206. Continue to input the image corresponding to step t-1 into the trained UNet diffusion network to predict the target noise corresponding to step t-1. Based on the denoising diffusion implicit model algorithm, calculate the image corresponding to step t-2 according to the target noise at step t-1 and the image corresponding to step t-1. Iterate the prediction and calculation until the target reconstruction image corresponding to step 0 is obtained.

[0126] The image corresponding to step t-2 is The process of calculating the image corresponding to step t-2 is the same as the process of calculating the image corresponding to step t-1 in step 204 of the embodiment, and will not be repeated here. After calculating the image corresponding to step t-2, the image corresponding to step t-2 is input into the trained UNet diffusion network to predict the target noise corresponding to step t-3. The third image corresponding to step t-3 is calculated based on the target noise and the image corresponding to step t-2, until the target reconstruction image is obtained. .

[0127] As another implementation, after obtaining the image corresponding to step t-1, the image corresponding to step t-1 is input into the trained UNet diffusion network to predict the target noise corresponding to step ts. Based on the target noise at step ts and the image corresponding to step t-1, the image corresponding to step t-2s is calculated. This prediction and calculation is iterated until the target reconstruction image corresponding to step 0 is obtained. Here, s is a preset number of steps, which can be dynamically configured. For example, s can be preset to 25 steps, which is suitable for real-time disaster monitoring scenarios; it can be preset to 50 steps, which is suitable for conventional remote sensing interpretation scenarios; or it can be preset to 100 steps, which is suitable for mapping and map updating scenarios.

[0128] This invention provides a method, apparatus, storage medium, and device for super-resolution reconstruction of remote sensing images. Through the technical solution of this invention, the conditional feature tensor corresponding to multimodal auxiliary data, which includes detailed features of spectral, textural, and geometric information, can be input into different layers of a UNet diffusion network. This allows for precise guidance of structural information at different scales, thereby accurately achieving super-resolution reconstruction of remote sensing images. By using multi-target loss, the consistency of geometric and textural structures can be maintained, ensuring physical consistency of reflectivity, maintaining spectral consistency across multiple bands, preserving the spatial fidelity of geometric structures, and preventing spectral distortion or texture misalignment.

[0129] Furthermore, as Figure 1 and Figure 2 The specific implementation of the method shown in this invention provides a remote sensing image super-resolution reconstruction device, such as... Figure 3 As shown, the device includes: a preprocessing module 31, a fusion module 32, a diffusion module 33, a training module 34, and a generation module 35;

[0130] Preprocessing module 31 is used to acquire sample remote sensing images and multimodal auxiliary data, and to preprocess the multimodal auxiliary data, which includes multispectral images, panchromatic images, and synthetic aperture radar images.

[0131] The fusion module 32 is used to extract the spectral features, texture features and geometric features of the preprocessed multimodal auxiliary data, and fuse the spectral features, texture features and geometric features through an image pyramid to generate a conditional feature tensor;

[0132] The diffusion module 33 is used to perform forward diffusion on the sample remote sensing image based on a linear noise scheduling strategy, and gradually add Gaussian noise to generate a noisy image sequence;

[0133] Training module 34 is used to inject the conditional feature tensor into the preset layers of the encoder and decoder of the UNet diffusion network through an adaptive instance normalization mechanism during the training of the UNet diffusion network using the noisy image sequence, and optimize the network parameters through multi-objective loss to obtain the trained UNet diffusion network.

[0134] The generation module 35 is used to acquire the noise map to be reconstructed, predict the noise of the noise map based on the trained UNet diffusion network to obtain the target noise, perform backsampling on the noise map based on the denoising diffusion implicit model algorithm and the target noise, and generate the target reconstructed image through iterative denoising.

[0135] Accordingly, the preprocessed multimodal auxiliary data includes preprocessed multispectral images, preprocessed panchromatic images, and preprocessed synthetic aperture radar images. In order to extract the spectral features, texture features, and geometric features of the preprocessed multimodal auxiliary data, the fusion module 32 is specifically used to extract the spectral features of the preprocessed multispectral images based on the band histogram; extract the texture features of the preprocessed panchromatic images based on the gray-level co-occurrence matrix; and extract the geometric features of the preprocessed synthetic aperture radar images based on the Sobel operator.

[0136] Accordingly, in order to perform forward diffusion on the sample remote sensing image based on the linear noise scheduling strategy and gradually add Gaussian noise to generate a noisy image sequence, the diffusion module 33 is specifically used to perform forward diffusion on the sample remote sensing image with the noise coefficient linearly increasing from the initial value to the target value and gradually add Gaussian noise to generate a noisy image sequence.

[0137] Accordingly, the UNet diffusion network includes: an encoder, a bottleneck layer, a decoder, and a convolutional layer connected in sequence. The decoder has the same number of layers as the encoder. The encoder layer above the bottleneck layer is skipped to the decoder layer below it. The two encoder layers above the bottleneck layer are skipped to the two decoder layers below it, until the last encoder layer above the bottleneck layer is skipped to the last decoder layer below it. To inject the conditional feature tensor into the preset layers of the encoder and decoder of the UNet diffusion network using an adaptive instance normalization mechanism during the training of the UNet diffusion network with the noisy image sequence, and to optimize the network parameters through multi-objective loss to obtain the trained UNet diffusion network, the training module 34 is specifically used to determine the encoder to be injected based on the preset layers of the encoder in the UNet diffusion network into which the conditional feature tensor is to be injected. Based on the noisy image sequence, the previous output feature of the encoder to be injected is used as the first feature, and the conditional feature tensor is normalized based on the adaptive instance normalization mechanism. A conditional feature tensor is injected into the first feature to obtain a first injected feature. This first injected feature is then input into the encoder to be injected, and downsampling continues until the second feature output by the last encoder is input into the bottleneck layer. A third feature extracted by the bottleneck layer is obtained. Based on the preset level of the decoder in the UNet diffusion network into which the conditional feature tensor is to be injected, the decoder to be injected is determined. Based on the third feature, the previous output feature of the decoder to be injected is used as the fourth feature. The conditional feature tensor is injected into the fourth feature based on adaptive instance normalization to obtain a second injected feature. This second injected feature is then input into the decoder to be injected, and upsampling continues until the fifth feature output by the last decoder is input into the convolutional layer to obtain a predicted image. The pixel values ​​of the predicted image are determined as prediction noise. Based on the noisy image sequence, the sample remote sensing image, and the prediction noise, a multi-target loss is calculated. The network parameters are optimized using the multi-target loss to obtain the trained UNet diffusion network.

[0138] Accordingly, in order to calculate the multi-target loss based on the noisy image sequence, the sample remote sensing image, and the predicted noise, and to optimize the network parameters through the multi-target loss to obtain the trained UNet diffusion network, the training module 34 is specifically used to calculate the spatial structure loss, spectral angle mapping loss, normalized vegetation index bias loss, edge preservation loss, and multimodal fusion guidance loss based on the noisy image sequence, the sample remote sensing image, and the predicted noise; calculate the weighted sum of the spatial structure loss, the spectral angle mapping loss, the normalized vegetation index bias loss, the edge preservation loss, and the multimodal fusion guidance loss to obtain the multi-target loss; determine whether the multi-target loss is less than a first preset value; if not, adjust the network parameters until the multi-target loss is less than the first preset value to obtain the trained UNet diffusion network.

[0139] To inject the conditional feature tensor into the first feature based on adaptive instance normalization to obtain the first injected feature, the training module 34 is specifically used to calculate the first mean and first standard deviation of the first feature in each channel dimension; calculate the first scale factor and first offset factor corresponding to the conditional feature tensor; perform instance normalization operation on the first feature according to the first mean and first standard deviation for each channel dimension to obtain the first normalized feature; calculate the first product of the first normalized feature and the first scale factor; and calculate the sum of the first product and the first offset factor to obtain the first injected feature. To inject the conditional feature tensor into the fourth feature based on adaptive instance normalization to obtain the second injected feature, the training module 34 is specifically used to calculate the second mean and second standard deviation of the fourth feature in each channel dimension; calculate the second scale factor and second offset factor corresponding to the conditional feature tensor; perform instance normalization operation on the fourth feature according to the second mean and second standard deviation for each channel dimension to obtain the second normalized feature; calculate the second product of the second normalized feature and the second scale factor; and calculate the sum of the second product and the second offset factor to obtain the second injected feature.

[0140] In order to predict noise based on the trained UNet diffusion network and obtain the target noise, the noise map is backsampled based on the denoising diffusion implicit model algorithm and the target noise, and the target reconstructed image is generated by iterative denoising. The generation module 35 includes: a first generation unit 351 and a second generation unit 352.

[0141] The first generation unit 351 is specifically used to input the noise map into the trained UNet diffusion network, predict the target noise at step t, and calculate the image at step t-1 based on the denoising diffusion implicit model algorithm and the target noise at step t and the noise map.

[0142] The second generation unit 352 is specifically used to continue inputting the image corresponding to the (t-1)th step into the trained UNet diffusion network, predict the target noise corresponding to the (t-1)th step, calculate the image corresponding to the (t-2)th step based on the denoising diffusion implicit model algorithm, and iterate the prediction and calculation until the target reconstruction image corresponding to the 0th step is obtained.

[0143] It should be noted that other corresponding descriptions of the functional units involved in the remote sensing image super-resolution reconstruction device provided in this embodiment can be found in [reference]. Figures 1 to 2 The corresponding description will not be repeated here.

[0144] Based on the above, Figures 1 to 2 Accordingly, this embodiment also provides a storage medium, which may be volatile or non-volatile, storing a computer program that, when executed by a processor, implements the above-described method. Figures 1 to 2 The method for super-resolution reconstruction of remote sensing images is shown.

[0145] Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present invention.

[0146] Based on the above, Figures 1 to 2 The method shown and Figure 3 , Figure 4 To achieve the above objectives, the present application also provides a computer device, specifically a personal computer, server, network device, etc., as shown in the virtual device embodiment. This computer device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to achieve the above-described objectives. Figure 1 and Figure 2 The method for super-resolution reconstruction of remote sensing images is shown.

[0147] Optionally, the computer device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0148] Those skilled in the art will understand that the computer device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0149] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned computer device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the non-volatile storage medium, as well as communication with other hardware and software in the information processing entity device.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platform, or it can be implemented by hardware.

[0151] This invention provides a method, apparatus, storage medium, and device for super-resolution reconstruction of remote sensing images. Through the technical solution of this invention, the conditional feature tensor corresponding to multimodal auxiliary data, which includes detailed features of spectral, textural, and geometric information, can be input into different layers of a UNet diffusion network. This allows for precise guidance of structural information at different scales, thereby accurately achieving super-resolution reconstruction of remote sensing images. By using multi-target loss, the consistency of geometric and textural structures can be maintained, ensuring physical consistency of reflectivity, maintaining spectral consistency across multiple bands, preserving the spatial fidelity of geometric structures, and preventing spectral distortion or texture misalignment.

[0152] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or they can be located in one or more apparatuses different from this embodiment, with corresponding changes. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0153] The serial numbers used above are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenarios. The above disclosures are merely a few specific implementation scenarios of the present invention; however, the present invention is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for super-resolution reconstruction of remote sensing images, characterized in that, The method includes: Acquire sample remote sensing images and multimodal auxiliary data, and preprocess the multimodal auxiliary data, which includes multispectral images, panchromatic images, and synthetic aperture radar images; Spectral features, texture features, and geometric features are extracted from the preprocessed multimodal auxiliary data. The spectral features, texture features, and geometric features are then fused using an image pyramid to generate a conditional feature tensor. The sample remote sensing images are forward diffused based on a linear noise scheduling strategy, and Gaussian noise is gradually added to generate a noisy image sequence. During the training of the UNet diffusion network using the noisy image sequence, the conditional feature tensor is injected into the preset layers of the encoder and decoder of the UNet diffusion network through an adaptive instance normalization mechanism. The network parameters are optimized through multi-objective loss to obtain the trained UNet diffusion network. A noise map to be reconstructed is obtained, and noise is predicted based on the trained UNet diffusion network to obtain the target noise. The noise map is then backsampled based on the denoising diffusion implicit model algorithm and the target noise, and the target reconstructed image is generated through iterative denoising. The UNet diffusion network comprises: an encoder, a bottleneck layer, a decoder, and a convolutional layer connected sequentially. The decoder has the same number of layers as the encoder. The encoder layer above the bottleneck layer is skipped to the decoder layer below it. The two encoder layers above the bottleneck layer are skipped to the two decoder layers below it, until the last encoder layer above the bottleneck layer is skipped to the last decoder layer below it. During training the UNet diffusion network using the noisy image sequence, the conditional feature tensor is injected into the preset layers of the encoder and decoder of the UNet diffusion network through an adaptive instance normalization mechanism. The network parameters are optimized using multi-objective loss to obtain the trained UNet diffusion network. Based on the preset level of the encoder in the UNet diffusion network into which the conditional feature tensor is to be injected, the encoder to be injected is determined. Based on the noisy image sequence, the previous output feature of the encoder to be injected is taken as the first feature. The conditional feature tensor is injected into the first feature based on adaptive instance normalization to obtain the first injected feature. The first injected feature is input into the encoder to be injected, and downsampling is continued until the second feature output by the last encoder is input into the bottleneck layer. Obtain the third feature extracted after the bottleneck layer, and determine the decoder to be injected according to the preset level of the decoder to be injected into the conditional feature tensor in the UNet diffusion network; Based on the third feature, the previous output feature of the decoder to be injected is used as the fourth feature. The conditional feature tensor is injected into the fourth feature based on adaptive instance normalization to obtain the second injected feature. The second injected feature is input into the decoder to be injected and upsampling is continued until the fifth feature output by the last decoder is input into the convolutional layer to obtain the predicted image. The pixel values ​​of the predicted image are determined as the prediction noise. Based on the noisy image sequence, the sample remote sensing image, and the prediction noise, a multi-target loss is calculated. The network parameters are optimized through the multi-target loss to obtain the trained UNet diffusion network.

2. The method according to claim 1, characterized in that, The preprocessed multimodal auxiliary data includes preprocessed multispectral images, preprocessed panchromatic images, and preprocessed synthetic aperture radar images. The extraction of spectral features, texture features, and geometric features from the preprocessed multimodal auxiliary data includes: Extract spectral features from the preprocessed multispectral image based on the band histogram; Texture features of the preprocessed panchromatic image are extracted based on the gray-level co-occurrence matrix; Geometric features of the preprocessed synthetic aperture radar image are extracted using the Sobel operator.

3. The method according to claim 1, characterized in that, The step of performing forward diffusion on the sample remote sensing images based on a linear noise scheduling strategy, gradually adding Gaussian noise to generate a noisy image sequence, includes: The sample remote sensing image is forward diffused by linearly increasing the noise coefficient from the initial value to the target value, and Gaussian noise is gradually added to generate a noisy image sequence.

4. The method according to claim 1, characterized in that, The step of calculating multi-target loss based on the noisy image sequence, the sample remote sensing image, and the predicted noise, and optimizing network parameters through multi-target loss to obtain the trained UNet diffusion network includes: Based on the noisy image sequence, the sample remote sensing image and the predicted noise, calculate the spatial structure loss, spectral angle mapping loss, normalized vegetation index bias loss, edge preservation loss and multimodal fusion guidance loss. The multi-objective loss is obtained by calculating the weighted sum of the spatial structure loss, the spectral angle mapping loss, the normalized vegetation index bias loss, the edge preservation loss, and the multimodal fusion guidance loss. Determine whether the multi-target loss is less than a first preset value. If not, adjust the network parameters until the multi-target loss is less than the first preset value to obtain the trained UNet diffusion network.

5. The method according to claim 1, characterized in that, The step of injecting the conditional feature tensor into the first feature based on adaptive instance normalization to obtain the first injected feature includes: Calculate the first mean and first standard deviation of the first feature in each channel dimension; Calculate the first scale factor and the first offset factor corresponding to the conditional feature tensor; For each channel dimension, the first feature is normalized based on the first mean and the first standard deviation to obtain the first normalized feature. Calculate the first product of the first normalized feature and the first scale factor, and calculate the sum of the first product and the first offset factor to obtain the first injected feature; The step of injecting the conditional feature tensor into the fourth feature based on adaptive instance normalization to obtain the second injected feature includes: Calculate the second mean and second standard deviation of the fourth feature in each channel dimension; Calculate the second scale factor and the second offset factor corresponding to the conditional feature tensor; For each channel dimension, the fourth feature is normalized based on the second mean and the second standard deviation to obtain the second normalized feature; Calculate the second product of the second normalized feature and the second scale factor, and calculate the sum of the second product and the second offset factor to obtain the second injected feature.

6. The method according to claim 1, characterized in that, The trained UNet diffusion network predicts noise in the noise map to obtain target noise. Based on the denoising diffusion implicit model algorithm and the target noise, backsampling is performed on the noise map to generate a target reconstructed image through iterative denoising. This includes: The noise map is input into the trained UNet diffusion network to predict the target noise at step t. Based on the denoising diffusion implicit model algorithm, the image corresponding to step t-1 is calculated according to the target noise at step t and the noise map. The image corresponding to step t-1 is input into the trained UNet diffusion network to predict the target noise corresponding to step t-1. Based on the denoising diffusion implicit model algorithm, the image corresponding to step t-2 is calculated according to the target noise at step t-1 and the image corresponding to step t-1. The prediction and calculation are iterated until the target reconstruction image corresponding to step 0 is obtained.

7. A remote sensing image super-resolution reconstruction device, characterized in that, The device includes: The preprocessing module is used to acquire sample remote sensing images and multimodal auxiliary data, and to preprocess the multimodal auxiliary data, which includes multispectral images, panchromatic images, and synthetic aperture radar images. The fusion module is used to extract the spectral features, texture features, and geometric features of the preprocessed multimodal auxiliary data, and fuse the spectral features, texture features, and geometric features through an image pyramid to generate a conditional feature tensor. The diffusion module is used to perform forward diffusion on the sample remote sensing images based on a linear noise scheduling strategy, gradually adding Gaussian noise to generate a noisy image sequence; The training module is used to inject the conditional feature tensor into the preset layers of the encoder and decoder of the UNet diffusion network through an adaptive instance normalization mechanism during the training of the UNet diffusion network using the noisy image sequence, and optimize the network parameters through multi-objective loss to obtain the trained UNet diffusion network. The generation module is used to acquire the noise map to be reconstructed, predict the noise of the noise map based on the trained UNet diffusion network to obtain the target noise, and perform backsampling on the noise map based on the denoising diffusion implicit model algorithm and the target noise to generate the target reconstructed image through iterative denoising. The UNet diffusion network comprises an encoder, a bottleneck layer, a decoder, and a convolutional layer connected in sequence. The decoder has the same number of layers as the encoder. The encoder layer above the bottleneck layer is skipped to the decoder layer below it. The two encoder layers above the bottleneck layer are skipped to the two decoder layers below it, until the last encoder layer above the bottleneck layer is skipped to the last decoder layer below it. The training module determines the encoder to be injected based on a preset layer of the encoder in the UNet diffusion network into which the conditional feature tensor is to be injected. Based on the noisy image sequence, the previous output feature of the encoder to be injected is used as the first feature. The conditional feature tensor is injected into the first feature using adaptive instance normalization to obtain the first injected feature. The first injected feature is then input into the encoder to be injected, and downsampling continues until… The second feature output by the last encoder is input into the bottleneck layer; the third feature extracted by the bottleneck layer is obtained, and the decoder to be injected is determined according to the preset level of the decoder to be injected with the conditional feature tensor in the UNet diffusion network; according to the third feature, the previous output feature of the decoder to be injected is used as the fourth feature, and the conditional feature tensor is injected into the fourth feature based on adaptive instance normalization to obtain the second injected feature, and the second injected feature is input into the decoder to be injected, and upsampling is continued until the fifth feature output by the last decoder is input into the convolutional layer to obtain the predicted image; the pixel values ​​of the predicted image are determined as prediction noise, and multi-target loss is calculated according to the noisy image sequence, the sample remote sensing image and the prediction noise, and the network parameters are optimized by multi-target loss to obtain the trained UNet diffusion network.

8. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the remote sensing image super-resolution reconstruction method according to any one of claims 1 to 6.

9. A computer device comprising a memory, a processor, and a computer program stored on a storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the remote sensing image super-resolution reconstruction method according to any one of claims 1 to 6.

Images