A multi-space knowledge distillation remote sensing image super-resolution method

Abstract

The application discloses a multi-space knowledge distillation remote sensing image super-resolution method, and belongs to the field of deep learning image processing. The method comprises the following steps: acquiring a high-resolution training image and generating a low-resolution image according to a continuous scale; constructing a lightweight student model comprising a prior encoder, a generator and a continuous scale up sampler, loading and freezing a pre-trained latent diffusion continuous scale teacher model, and setting a training auxiliary discriminator; using a teacher mean and a single sampling result to jointly distill hidden variables, noise prediction and low-noise clean hidden variable reconstruction; training in combination with pixel, perception, texture, adversarial and distillation loss, and supervising clean hidden variables according to time steps; and during deployment, only loading the student model and an exponential moving average weight, and generating a target scale image through a single forward pass. The application can reduce inference delay and deployment overhead, and is suitable for edge devices. The application also discloses an electronic device and a storage medium for implementing the above method.

Description

Technical Field

[0001] This invention relates to a method for super-resolution of remote sensing images, specifically a method for super-resolution of remote sensing images based on multi-spatial knowledge distillation, belonging to the field of deep learning image processing technology. Background Technology

[0002] Remote sensing images have significant applications in military reconnaissance, urban planning, environmental monitoring, and disaster assessment. However, due to limitations in imaging distance and sensor hardware, the spatial resolution of acquired remote sensing images is often insufficient. Super-resolution (SR) technology aims to reconstruct high-resolution (HR) images from low-resolution (LR) images. In practical applications, different scenarios have varying magnification requirements; continuous-scale super-resolution can cover multiple needs with a single model, avoiding the need to train a separate model for each fixed magnification.

[0003] Furthermore, remote sensing SR is often deployed on resource-constrained edge devices, such as drones and on-orbit satellites, which have strict requirements on model size and inference speed. Edge computing platforms are typically constrained by power consumption, memory, and real-time performance, making it difficult for large-scale diffusion models to run efficiently on such devices for extended periods.

[0004] In scenarios where a pre-trained latent diffusion continuous-scale super-resolution teacher model is compressed into a lightweight student model constrained by the training objective of a Generative Adversarial Network (GAN) for use in continuous-scale super-resolution of remote sensing images, existing distillation methods typically only perform teacher-student alignment in the pixel space or a single feature space. This results in the teacher model's supervision signals in the Variational Autoencoder (VAE) latent variable space, the noise prediction space of the Conditional Denoising U-shaped Network (U-Net), and the clean latent variable reconstruction space of the Denoising Diffusion Probabilistic Model (DDPM) not being fully utilized. The distilled lightweight model is prone to problems such as decreased structural fidelity, smoothed texture, and insufficient generalization at high magnification scales. Furthermore, if the clean latent variables obtained by inversely calculating DDPM in one step are directly used as the distillation target for all time steps, the noise prediction error in high-noise time steps will be amplified when divided by the cumulative retention coefficient, thus contaminating the student model's training gradient. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-spatial knowledge distillation method for remote sensing image super-resolution, addressing the technical problems in existing continuous-scale remote sensing image super-resolution: while diffusion models offer high perception quality, they suffer from long inference times, high memory and power consumption; lightweight models have high inference efficiency but struggle to maintain the texture generation capabilities of diffusion models; single-spatial knowledge distillation cannot fully transfer multi-layer representation knowledge from the teacher model; and instability exists in the clean latent variable inverse supervision during high-noise time steps. This invention obtains supervision signals from multiple representation spaces of the pre-trained diffusion teacher model during the student model training phase, including the latent variable space, the noisy prediction space, and the clean latent variable reconstruction space. It updates the parameters of the lightweight student model by jointly using reconstruction loss and adversarial loss, and introduces a selective gating mechanism in the clean latent variable reconstruction space to avoid inaccurate reconstruction of contaminated gradients at high-noise time steps, thereby reducing distillation degradation and improving perception quality and inference efficiency.

[0006] To achieve the above objectives, this invention provides a remote sensing image super-resolution method based on multi-spatial knowledge distillation, comprising the following steps:

[0007] Step 1: Obtain high-resolution remote sensing image training samples, perform cropping, enhancement and normalization processing on the high-resolution remote sensing image training samples, and generate corresponding low-resolution remote sensing image training samples according to the training scale.

[0008] Step 2: Construct a lightweight student model, a training-stage auxiliary discriminator, and a pre-trained diffusion teacher model;

[0009] Step 3: Input the remote sensing image training samples into the lightweight student model and the pre-trained diffusion teacher model, and jointly train the lightweight student model through the latent variable space, the noisy prediction space and the clean latent variable reconstruction space to obtain the trained student model.

[0010] Step 4: Input the low-resolution remote sensing image to be processed into the trained student model, and give the target scale and target output size to obtain the high-resolution remote sensing image.

[0011] Further, in step 2, the lightweight student model is a generative super-resolution network constrained by the training objective of a Generative Adversarial Network (GAN), comprising a prior encoder, a generator, and a Continuous Scale Upsampling Module (CSUM). The discriminator is an auxiliary network during the training phase, used only to provide adversarial supervision and is not part of the student model in the inference deployment phase. The teacher model adopts a latent diffusion continuous scale super-resolution model, comprising a first-stage image encoder and image decoder of a Variational Autoencoder (VAE), a Conditional Denoising U-shaped Network (U-Net), a Conditional Construction Module, a Continuous Scale Upsampling Module, and a Differential Prior Encoder and Differential Prior Decoder.

[0012] Further, step 3 includes:

[0013] The remote sensing image training data is input into the student model and the teacher model for processing to obtain the student over-score result and the teacher over-score result.

[0014] The student super-resolution results and the teacher super-resolution results are respectively input into the first stage image encoder of the variational autoencoder in the teacher model for processing to obtain the student latent variables and the teacher latent variables, and the first distillation loss is calculated.

[0015] The latent variables of students and teachers after applying noise are processed by the conditional denoising U-shaped network in the teacher model to obtain the student prediction noise and the teacher prediction noise, and the second distillation loss is calculated.

[0016] Based on the predicted noise, the latent variables after noise is applied, and the cumulative retention coefficient of the diffusion process, the clean latent variables of students and teachers are recovered respectively through the one-step inverse formula of the denoised diffusion probability model, and the third distillation loss is calculated by combining the selective gating mechanism.

[0017] Based on the total loss at the generator end, the student model is trained and its parameters are updated to obtain the trained student model. The total loss at the generator end includes reconstruction loss, generator adversarial loss, and various distillation losses. The discriminator loss is only used to update the auxiliary discriminator during the training phase.

[0018] Furthermore, the reconstruction loss is expressed as:

[0019] ,

[0020] ,

[0021] ,

[0022] ,

[0023] Where sr represents the super-resolution image output by the student model, and hr represents the high-resolution ground truth image. This represents the total amount of reconstruction losses. Indicates pixel reconstruction loss. This represents the perceived reconstruction loss. This represents the texture statistics loss. Charbonnier represents the Charbonnier robust pixel loss, with its internal smoothing constant set to... ; , , represents the features of the pre-trained visual geometry network at layers c34, c44, and c54, respectively; Gram represents the Gram matrix composed of the inner products of the feature channels. This represents the L1 norm. In the formula, 2.0, 0.1, and 20.0 are the weights for pixel, perceptual, and texture statistics, respectively. The 0.5 and 1.0 in parentheses are the intra-layer weights of the feature difference terms in the c34th and c54th layers of VGG, respectively, used to adjust the relative contributions of the features in the two layers.

[0024] Furthermore, the distillation loss includes a first distillation loss, a second distillation loss, and a third distillation loss;

[0025] First distillation loss Represented as: ,

[0026] in, This represents the pixel-wise mean of multiple sampling results from the teacher diffusion model; This represents the frozen variational autoencoder in the teacher model, specifically the first-stage image encoder; `sg` indicates stopping gradient operations, allowing the teacher's latent variables to serve only as supervision targets and not participate in gradient backpropagation; `SmoothL1` represents smoothing the L1 loss; the coefficient 1.0 is the loss weight for this distillation term. The gradient of this loss is passed back to the student's super-resolution results via the frozen encoder, but the teacher encoder parameters are not updated.

[0027] The second distillation loss is expressed as: ,

[0028] in, This represents the second distillation loss, i.e., the noise prediction spatial distillation loss; This represents the loss weight corresponding to time step t; This represents the noise residual of the teacher-conditional denoising U-shaped network's prediction of noisy latent variables in students. This represents the noise residual of the teacher branch after the gradient stops; the coefficient 0.4 is the loss weight of the noise prediction distillation term.

[0029] The third distillation loss is expressed as follows: ,

[0030] in, This represents the third distillation loss, i.e., the distillation loss in the clean hidden variable reconstruction space; This represents the estimate of the clean latent variables of the student, obtained by inversely deducing the student's noisy latent variables and student prediction noise through a denoising diffusion probability model in one step. This represents the estimate of the clean latent variables of teachers obtained from the noisy latent variables of teachers and the teacher prediction noise recovery after stopping the gradient; This represents the time-step gating threshold for the clean latent variable reconstruction space. When t ≤ Calculate the smoothed L1 loss at 0.2 times the normal value when t> This term is set to 0 at this time. This gating is used to prevent the one-step backpropagation error of a high-noise time step from contaminating the student model gradient.

[0031] Furthermore, the implementation process of the selective gating mechanism includes:

[0032] First, time step t is sampled from the time step set of the teacher diffusion model, and the cumulative signal retention coefficient is calculated based on noise scheduling. Cumulative noise variance ratio and signal-to-noise ratio .in, The noise variance coefficient at time step t is determined by noise scheduling. middle, Indicates the single-step signal retention coefficient; , This represents the cumulative signal retention coefficient up to time step t. Indicates the time step index in a multiplication operation; This indicates the proportion of cumulative noise variance.

[0033] Subsequently, the student latent variables and teacher latent variables are added to the same noise at the same time step, and the student prediction noise and teacher prediction noise are obtained through the frozen teacher conditional denoising U-shaped network. Then, it is determined whether the time step is less than or equal to the preset low noise time step threshold, or whether the signal-to-noise ratio is higher than the preset signal-to-noise ratio threshold. If the judgment result is yes, the student clean latent variables and teacher clean latent variables are recovered respectively according to the one-step back-calculation formula of the denoising diffusion probability model, and the third distillation loss is calculated. If the judgment result is no, the calculation of the third distillation loss is skipped, the loss is set to zero value which does not produce gradients, and the subsequent training steps are continued.

[0034] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0035] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] 1. This invention transfers structural priors, denoising dynamics, clean latent variable reconstruction capabilities, and perceptual texture information from the diffusion teacher model to a lightweight student model through joint distillation of the latent variable space, noisy prediction space, and clean latent variable reconstruction space. This technique avoids the problem of single-pixel space distillation only learning the averaging result, enabling the student model to achieve near-diffusion model-like structural recovery and texture generation capabilities while maintaining a lightweight inference path. Its theoretical basis lies in the fact that different representation spaces respectively constrain the image's compressed semantics, diffusion trajectory, low-noise reconstruction, and visual perception statistics, with complementary supervisory signals.

[0038] 2. This invention introduces a selective gating mechanism into the clean latent variable reconstruction space, using the clean latent variables obtained through one-step back-calculation only as the distillation target under low-noise time steps or high signal-to-noise ratio conditions. This technique can avoid the amplification of noise prediction errors in high-noise time steps by the back-calculation formula, thereby reducing the impact of erroneous gradients on the student model. The theoretical basis is that the signal-to-noise ratio of the diffusion process changes with time steps, and the estimation of clean samples in high-noise time steps is unstable and not suitable as a strong supervision target.

[0039] 3. This invention restricts the use of the diffusion teacher model and discriminator to the training phase, while retaining only the student model consisting of the prior encoder, generator, and continuous-scale upsampling unit during the inference and deployment phase. This technique eliminates the variational autoencoder encoding and decoding, conditional denoising U-shaped network multi-step sampling, and discriminator computation during the inference phase, enabling a single lightweight model to cover multiple continuous scaling scales. This reduces model size, memory usage, and response latency, making it suitable for resource-constrained scenarios such as UAVs, mobile mapping platforms, and edge computing nodes. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0041] Figure 2 This is a schematic diagram of the three-space distillation data stream of the present invention;

[0042] Figure 3 This is a schematic diagram of the time-step gating decision-making process of the present invention;

[0043] Figure 4 This is a schematic diagram of the student model inference deployment process of the present invention. Detailed Implementation

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0045] Example 1:

[0046] This embodiment provides a multi-spatial knowledge distillation method for super-resolution of remote sensing images. The method flowchart is as follows: Figure 1 The technical solution is as follows:

[0047] Step 1: Prepare training data.

[0048] The training set of the Aerial Image Dataset (AID) was used for partitioning. This dataset was publicly constructed by Xia et al. in "AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification". The original images are from Google Earth and contain images of various remote sensing scenes.

[0049] This embodiment reads high-resolution images from the training set and randomly crops 256×256 pixel image blocks as high-resolution ground truth images. For each high-resolution image block, a low-resolution image is generated according to the training scale factor s. When s=2, the low-resolution size is 128×128; when s=4, the low-resolution size is 64×64; and when s=8, the low-resolution size is 32×32. During the training phase, the scale factor s is uniformly and randomly sampled within the continuous interval [1.0, 8.0], and resampled at the beginning of each training round. The same scale factor is shared within the same global training batch to ensure consistent output size across devices during distributed training. Data augmentation includes random horizontal flipping, random vertical flipping, and random right-angle rotation. Before inputting into the network, the image is converted from the 0 to 255 pixel values ​​of the Red, Green, and Blue (RGB) channels into tensors and normalized to the [0,1] value range by dividing by 255.

[0050] Furthermore, bicubic interpolation (Bicubic) downsampling is implemented using the INTER_CUBIC interpolation method in the OpenCV image processing library, which involves performing cubic convolutional interpolation resampling on the high-resolution image based on the target low-resolution size.

[0051] Step 2: Load the pre-trained diffusion teacher model.

[0052] Load the pre-trained Latent Diffusion for Continuous-Scale Super-Resolution (LDCSR) teacher model. This teacher model can be obtained by reproducing the model and its open-source license published by Wu, Mo, Sun, and Ma in "Latent Diffusion, Implicit Amplification: Efficient Continuous-Scale Super-Resolution for Remote Sensing Images" published in IEEE Transactions on Geoscience and Remote Sensing 2025, and includes the following components:

[0053] (a) The first stage variational autoencoder image encoder and image decoder, wherein the image encoder maps the high-resolution or super-resolution RGB image together with the corresponding low-resolution conditions to low spatial resolution latent variables, and the image decoder reconstructs the latent variables obtained by diffusion sampling into an RGB super-resolution image under low-resolution conditions and target size constraints.

[0054] (b) A conditional denoising U-shaped network is used to perform denoising diffusion probability model sampling in the latent space. Noise scheduling is determined by the initial noise intensity. and termination noise intensity Sure; This represents the noise variance coefficient corresponding to the starting point of the diffusion time step sequence. This represents the noise variance coefficient corresponding to the end point of the diffusion time step sequence, for each time step. It is generated linearly between the two and further used for calculation. and cumulative retention factor ;

[0055] (c) Continuous Scale Upsampling Module (CSUM) is used to fuse multi-level features based on the target scale and output size during the teacher decoding stage, amplify the latent space difference prior to the target high-resolution space, and generate continuous scale super-resolution results together with the structural information of the low-resolution conditional image.

[0056] (d) Differential prior encoder and differential prior decoder are used to learn the high-frequency differential information missing in high-resolution images relative to low-resolution inputs; wherein the differential prior encoder extracts the true differential prior when training the teacher model, and the differential prior decoder completes image reconstruction in conjunction with the continuous scale upsampling module after obtaining the differential latent variables through diffusion sampling.

[0057] The teacher model keeps its parameters frozen during the student training phase. Specifically, after initializing the trainer and loading the teacher weights, all trainable parameters of the teacher model are set to not receive gradients, and the teacher model is switched to evaluation mode. This setting prevents the teacher model from updating weights during backpropagation, while fixing training-state behaviors such as normalized layers and randomly deactivated layers, ensuring the stability of the teacher supervision signal under the same input conditions, and only allowing gradients to be passed back to the student super-resolution results via the frozen encoder.

[0058] Step 3: Build a lightweight generative student model.

[0059] The student model adopts a three-stage structure, including a prior encoder, a generator, and a continuous-scale upsampler; the discriminator is an auxiliary module during the training phase and does not enter the three-stage inference chain.

[0060] The prior encoder employs a lightweight multi-level convolutional pyramid structure to extract multi-scale prior features from low-resolution remote sensing images. It includes a Galerkin attention module to enhance global context modeling capabilities and outputs multi-scale prior feature maps, global prior vectors, and spatial prior maps. The parameter size of this module is significantly smaller than that of the representation path corresponding to the teacher model.

[0061] Furthermore, the spatial attention module can be implemented using Galerkin attention, self-attention, or channel attention. Preferably, an attention form that can model long-range spatial dependencies with a low number of parameters is adopted to better support the latent space supervision of the teacher model.

[0062] The generator performs feature extraction and residual generation in the target resolution space. The input is a low-resolution image upsampled to the target size and the result of concatenating the spatial prior image along the channel dimension. Local texture is extracted through shallow convolution and several lightweight feature blocks. Then, the multi-scale prior features and global prior vectors are injected into the generator through the prior fusion module. Finally, the residual head generates a residual feature map for use by the continuous-scale upsampler.

[0063] Furthermore, lightweight feature blocks can employ depthwise separable convolutions, channel-reduced convolutions, residual connections, or attention-enhanced convolutions to maintain the ability to model local details when the number of parameters is limited.

[0064] The continuous-scale upsampler generates multi-level feature pyramids from the residual feature map step by step. The scale-aware multilayer perceptron generates fusion weights for each level of features based on the scale factor. After each level of features is adjusted to the target output size, they are weighted and fused. Finally, after convolution mapping, they are added pixel by pixel to the upsampled low-resolution image to obtain the super-resolution output.

[0065] In this embodiment, the number of basic feature channels of the student model, the number of pyramid levels of the prior encoder, the number of lightweight feature blocks of the generator, the number of continuous scale upsampling levels, the hidden dimension of the scale-aware multilayer perceptron, the number of prior map channels, and the low-resolution conditional modulation method are all determined based on the deployed computing power and the target output size.

[0066] The preferred implementation employs a three-layer prior pyramid, four generator lightweight feature blocks, four-level continuous scale upsampling, scale-aware weight prediction, multi-level prior map fusion, and low-resolution conditional feature modulation. These parameters pertain to model capacity and engineering deployment configuration and do not alter the multi-space distillation training method of this invention; in this preferred implementation, the number of student model parameters is approximately 2.49 million.

[0067] Step 4: Set up the training environment and hyperparameters.

[0068] Training was implemented using the PyTorch deep learning framework. To reduce memory usage and improve training throughput, regular forward and backward computations of the student model and discriminator were performed using half-precision floating-point (FP16) with automatic mixed precision. For forward computations of the Galerkin attention module and teacher diffusion model, which are sensitive to numerical stability, single-precision floating-point (FP32) was used to avoid underflow and non-numerical results caused by half-precision computations. Multi-GPU training employed a distributed data parallel (DDP) strategy, allowing different GPUs to process different training batches and synchronize gradients after backpropagation. The optimizer preferred was the Adaptive Moment Estimation (Adam) optimizer, and the learning rate was gradually decayed from the initial value using a cosine annealing strategy. Training could continue initialization from the converged student model weights, but historical optimizer states were not loaded to avoid momentum from old training phases affecting the current distillation objective.

[0069] Step 5: Execute the training loop.

[0070] Each training batch contains low-resolution remote sensing images, high-resolution ground truth images, and scale factors s. The training loop is executed in the order of teacher-supervised generation, discriminator update, student model update, and exponential moving average weight update. The data relationships between these steps are as follows.

[0071] (a) In the student forward stage, the low-resolution remote sensing image is first upsampled by interpolation according to the target output size as the base image for residual reconstruction. Then, the prior encoder extracts multi-scale prior features and spatial prior map from the low-resolution image. The generator fuses the low-resolution base image, spatial prior map, scale embedding and multi-scale prior features in the target resolution space to obtain residual features. Finally, the continuous scale upsampler fuses multi-level features according to the scale factor and the target output size and adds them to the upsampled base image to obtain the student super-resolution result.

[0072] (b) In the teacher feedforward phase, without updating the teacher parameters, the same low-resolution image, scale factor, and target output size are input into the pre-trained diffusion teacher model. The teacher model performs multiple diffusion samplings to obtain a set of teacher super-resolution samples; the mean value of the teacher super-resolution is obtained pixel by pixel from this set of samples, which is used for distillation of the pixel-stable latent variable space, the noise prediction space, and the clean latent variable reconstruction space; single sampling results are extracted from this set of samples as teacher super-resolution samples for high-frequency texture supervision; the standard deviation of the teacher is calculated pixel by pixel from this set of samples to obtain the teacher uncertainty result, which is used to characterize the teacher's confidence in generating different regions.

[0073] (c) The reconstruction loss is used to constrain the pixel accuracy, perceptual semantics, and texture statistics of the student's super-resolution results relative to the high-resolution ground truth image. In the formula, the Visual Geometry Group 19-layer Network (VGG19) can use the ImageNet pre-trained weights from "Very Deep Convolutional Networks for Large-Scale Image Recognition" proposed by Simonyan and Zisserman; the Gram matrix represents the inner product statistics between feature channels, used to characterize the texture distribution.

[0074] ,

[0075] in, ,

[0076] ,

[0077] ,

[0078] ,

[0079] Where sr represents the super-resolution image output by the student model, and hr represents the high-resolution ground truth image. This represents the total amount of reconstruction losses. Indicates pixel reconstruction loss. This represents the perceived reconstruction loss. This represents the texture statistics loss. Charbonnier represents the Charbonnier robust pixel loss, with its internal smoothing constant set to... ; , , represents the features of layers c34, c44, and c54 of the pre-trained visual geometry network, respectively; Gram represents the Gram matrix composed of the inner products of the feature channels; Laplacian represents the Laplacian high-frequency operator. This represents the L1 norm. In the formula, 2.0, 0.1, 20.0, and 0.05 are the weights for pixel, perceptual, and texture statistics, respectively. The 0.5 and 1.0 within parentheses are the intra-layer weights of the feature difference terms in VGG layers c34 and c54, respectively, used to adjust the relative contributions of the features in the two layers.

[0080] (d) Three-space distillation loss is used to transfer knowledge from the diffuse teacher model at different representation levels to the lightweight student model. Specifically, the student super-resolution results and the teacher super-resolution mean are first input into the frozen first-stage teacher encoder to obtain teacher and student latent variables; then, the same Gaussian noise is added to the teacher and student latent variables at the same time step, and the noise residuals are predicted using the frozen teacher conditional denoising U-shaped network; finally, under low-noise conditions, a one-step backpropagation is performed based on the noise residuals to obtain clean latent variable reconstruction supervision. In the formula, stopping gradient indicates that the teacher-side target does not backpropagate gradients. The cumulative retention coefficient up to time step t represents the diffusion process and is pre-calculated by the noise scheduling. For the overall mechanism of three-space distillation loss, see [link to relevant documentation]. Figure 2 .

[0081] The first distillation space is the latent variable space. The student super-resolution results and the teacher super-resolution mean are mapped to student latent variables and teacher latent variables respectively by the first-stage image encoder of the variational autoencoder frozen in the teacher model, and aligned using smooth L1 loss, so that the student results maintain the same structural prior in the teacher compressed representation.

[0082] ,

[0083] ,

[0084] ,

[0085] ,

[0086] in, represents the first distillation loss, i.e., the latent variable space distillation loss; sr represents the student super-resolution result; This represents the pixel-wise mean of multiple sampling results from the teacher diffusion model; both are... Figure 2 The corresponding Chinese character is and ; This represents the latent variable obtained by encoding the student's super-resolution results using the teacher's variational autoencoder. The latent variable representing the teacher mean image after the gradient has stopped; This represents the frozen variational autoencoder in the teacher model, specifically the first-stage image encoder; `sg` indicates stopping gradient operations, allowing the teacher's latent variables to serve only as supervision targets and not participate in gradient backpropagation; `SmoothL1` represents smoothing the L1 loss; the coefficient 1.0 is the loss weight for this distillation term. The gradient of this loss is passed back to the student's super-resolution results via the frozen encoder, but the teacher encoder parameters are not updated.

[0087] The second distillation space is the noise prediction space. The same Gaussian noise is added to the student and teacher latent variables at the same time step t, resulting in noisy student and teacher latent variables. These noisy variables are then input into a frozen teacher-conditional denoising U-shaped network along with the same low-resolution, scale, and target size conditions to obtain the student and teacher predicted noise, respectively, and are constrained to be consistent. The second and third distillation spaces share the same forward computation of the teacher denoising network to reduce training overhead.

[0088] ,

[0089] , ,

[0090] , ,

[0091] ,

[0092] in, This represents the second distillation loss, i.e., the noise-predicted spatial distillation loss; t represents the diffusion time step, distributed according to the classification in the formula. Sample from {0,1,2,3}; Represents the loss weight corresponding to time step t; η represents Gaussian noise that follows a standard normal distribution; , These represent the noisy latent variables obtained by the student branch and the teacher branch at the same time step and under the same noise, respectively. This represents the cumulative retention coefficient up to time step t during the diffusion process; This represents the noise residual of the teacher-conditional denoising U-shaped network's prediction of noisy latent variables in students. This represents the noise residual of the teacher branch after the gradient stops; the coefficient 0.4 is the loss weight of the noise prediction distillation term.

[0093] The third distillation space is a clean hidden variable reconstruction space; the mechanism of this space can be found in [reference needed]. Figure 3 According to the one-step inverse formula of the denoised diffusion probability model, the clean latent variables are recovered from the noisy latent variables, the predicted noise, and the cumulative retained coefficients; this space is only enabled when the selective gating is determined to be a low-noise time step, so as to avoid the inverse calculation error of the high-noise time step being amplified and contaminating the student model gradient.

[0094] , ,

[0095] ,

[0096] in, This represents the third distillation loss, i.e., the distillation loss in the clean hidden variable reconstruction space; This represents the estimate of the clean latent variables of the student, obtained by inversely deducing the student's noisy latent variables and student prediction noise through a denoising diffusion probability model in one step. This represents the estimate of the clean latent variables of teachers obtained from the noisy latent variables of teachers and the teacher prediction noise recovery after stopping the gradient; This represents the time-step gating threshold for the clean latent variable reconstruction space. When t ≤ Calculate the smoothed L1 loss at 0.2 times the normal value when t> This term is set to 0 at this time. This gating is used to prevent the one-step backpropagation error of a high-noise time step from contaminating the student model gradient.

[0097] Furthermore, the second and third distillation spaces share the same teacher-conditional denoised U-shaped network forward computation and use the same diffusion time step and the same Gaussian noise; therefore, during training, only one additional noise prediction for the student branch and one additional noise prediction for the teacher branch are required to simultaneously obtain noise prediction distillation supervision and clean latent variable reconstruction distillation supervision.

[0098] Furthermore, the specific process of selective gating is as follows: First, time step t is sampled from the time step set of the teacher diffusion model, and the cumulative signal retention coefficient is calculated based on the noise scheduling. Cumulative noise variance ratio and signal-to-noise ratio .in, The noise variance coefficient at time step t is determined by noise scheduling. middle, Indicates the single-step signal retention coefficient; , This represents the cumulative signal retention coefficient up to time step t. Indicates the time step index in a multiplication operation; This indicates the proportion of cumulative noise variance.

[0099] Subsequently, the student latent variables and teacher latent variables are added to the same noise at the same time step, and the student prediction noise and teacher prediction noise are obtained through the frozen teacher conditional denoising U-shaped network. Then, it is determined whether the time step is less than or equal to the preset low noise time step threshold, or whether the signal-to-noise ratio is higher than the preset signal-to-noise ratio threshold. If the judgment result is yes, the student clean latent variables and teacher clean latent variables are recovered respectively according to the one-step back-calculation formula of the denoising diffusion probability model, and the third distillation loss is calculated. If the judgment result is no, the calculation of the third distillation loss is skipped, the loss is set to zero value which does not produce gradients, and the subsequent training steps are continued.

[0100] (e) The discriminator update phase is performed before each student model update. The discriminator is part of the auxiliary network during the training phase and is not part of the inference component of the lightweight student model; its role is to distinguish between real high-resolution images or teacher-sampled images and student super-resolution results, thereby providing adversarial supervision at the texture distribution level for the student model.

[0101] During discriminator updates, real samples are input to obtain real discrimination scores, while student super-resolution results, after gradient truncation, are used as pseudo-samples input to obtain pseudo-sample discrimination scores. Subsequently, the discriminator is updated based on the relative average hinge adversarial loss to ensure that real sample scores are higher than pseudo-sample scores. To limit excessive gradients of the discriminator to the input image, a first-order gradient regularization term can be applied to the real samples at preset intervals. This stage only updates the discriminator parameters, not the student model parameters.

[0102] ,

[0103] ,

[0104] ,

[0105] Where D represents the auxiliary discriminator during the training phase, The discrimination score represents the true sample. The discrimination score represents the score of the spurious sample. This represents a high-resolution ground truth image or a single-sampled image of the teacher. This represents the student super-resolution results after gradient truncation. This indicates that the gradient is calculated over the real samples. represents the gradient regularization weight, and K represents the training interval that triggers gradient regularization.

[0106] (f) The generator update phase is used to update all trainable parameters of the lightweight student model. In this phase, the discriminator parameters are fixed, the student super-resolution results are recalculated, and the reconstruction loss, multispatial distillation loss, and generator adversarial loss are weighted and summed to obtain the generator total loss used only for updating the student model.

[0107] ,

[0108] in, This represents the total loss at the generator end and is only used to update the lightweight student model. This indicates that the generation end is subject to adversarial loss. This represents the adversarial loss weights; the discriminator loss is not included in the total loss at the generator end, and the discriminator parameters are updated in a separate discriminator update phase. The adversarial loss weights can be dynamically adjusted based on the difference in discrimination scores between real and pseudo-samples and the proportion of adversarial loss contribution to reduce the risk of training imbalance.

[0109] (g) The Exponential Moving Average (EMA) update phase is executed after each update of the student model parameters. This phase maintains a smoothed copy of the student model parameters, incorporating the current student parameters into the historical smoothed parameters at a small ratio for validation and inference deployment, thereby reducing weight jitter caused by a single gradient update.

[0110] ,

[0111] ,

[0112] in, Indicates the current student model parameters. This represents the parameters of the student model after the exponential moving average. This represents the exponential moving average decay coefficient. After each parameter update of the student model, a large proportion of the historical smoothed parameters are retained first, and then a small proportion of the current parameters are incorporated; after training, the parameters are exported first. The corresponding student model weights are used for validation and deployment, and teacher diffusion sampling, discriminator discrimination, or distillation loss calculation are no longer performed during the inference phase.

[0113] Step 6: Deployment by reasoning.

[0114] After training, only export the weight files of the student model. (Refer to...) Figure 4It can be saved as a PyTorch weight file or an Open Neural Network Exchange (ONNX) file, including a prior encoder, generator, and scale upsampling unit, and deployed to desktop workstations, UAV-based edge computing platforms, or other target inference environments. During the inference stage, there is no need to load the teacher model diffusion link or the student model discriminator; for the input low-resolution remote sensing image, only the target scale and target output size need to be given, and the student model can obtain the corresponding scale's super-resolution result through a single forward computation.

[0115] Step 7: Experimental verification.

[0116] To verify the effectiveness of this invention, quality experiments, ablation experiments, and deployment efficiency experiments were conducted. The quality experiments used one thousand images from the AID test set and were extended to test subsets of the Dataset for Object Detection in Aerial Images (DOTA) and the Dataset for Object Detection in Optical Remote Sensing Images (DIOR). All methods used bicubic interpolation to generate low-resolution inputs, and peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learned perceptual image patch similarity (LPIPS), and Frechet inception distance (FID) were evaluated at magnification scales of 2x, 4x, and 8x. PSNR and SSIM were used to measure pixel and structural fidelity, while LPIPS and FID were used to measure perceptual quality and distribution consistency.

[0117] In the quality experiment, the student model of this invention was compared with the frozen LDCSR teacher model on the same scale. As shown in Table 1, the experimental results show that the images generated by the student model, with a significant reduction in inference paths, can approach the PSNR and perceptual metrics of the images generated by the teacher model; this indicates that multi-spatial distillation can transfer the global structural prior of the teacher model while preserving its texture generation capabilities. Arrows (↓ or ↑) are used to indicate whether the metrics tend to take smaller or larger values. The bolded values ​​in Table 1 represent that the student model outperforms the teacher model at that value, and the bolded values ​​in Table 3 represent that the model of this invention performs best among open-source models under the specified metrics.

[0118] Table 1. Cross-dataset evaluation of teacher and student models based on the reproduced LDCSR model.

[0119]

[0120] The ablation experiments sequentially added reconstruction loss, latent variable spatial distillation, noisy prediction spatial distillation, and clean latent variable reconstruction spatial distillation, while maintaining consistency in the student model structure, training data, and optimization settings. The comparison results in Table 2 show that the student model, due to its lightweight design, exhibits poor learning ability across all scales. Adding reconstruction loss to this model subsequently improved its overall capabilities, but due to the reduction in LR image priors at high magnifications, the model still struggles to establish a connection between LR and HR. With the gradual addition of the three-space distillation loss, although the model degrades somewhat at low magnifications, the significant improvement compared to its performance at medium to high magnifications is still acceptable.

[0121] Table 2. FID ablation experiments of the student model on the AID dataset.

[0122]

[0123] The deployment efficiency experiment used a uniform scale of 4x magnification, from 64×64 low-resolution input to 256×256 high-resolution output. Average generation speed, cold start first image latency, energy consumption per image, and peak memory usage were tested on desktop graphics processors (such as the RTX 4090D) and edge devices (such as the Jetson Orin NX) with a batch size of 1. Experimental results show that this invention retains only the student model during the inference stage, thus exhibiting lower latency, fewer parameters, and lower deployment complexity compared to the teacher model, which requires variational autoencoders, conditional construction, diffusion sampling, and image decoding. Performance tests on the RTX 4090D and Jetson Orin NX show that the student model's inference speed is 17 times and 4 times faster than the teacher model, respectively, leading inference performance among existing open-source reproducible models. Specific data are shown in Table 3. This test uniformly used a 4x magnification, from 64×64 to 256×256, with a batch size of 1. On the RTX 4090D, student latency is 9.8ms and teacher latency is 166.1ms, a 16.95x improvement; on the Jetson Orin NX, student latency is 169.42ms and teacher latency is 632.58ms, a 3.73x improvement. On the Jetson, student / teacher CUDA memory allocation is 356.0 / 579.4 MiB, with power consumption per image of 3.549 / 10.827J, a reduction of approximately 67.2%.

[0124] Table 3 Jetson Orin NX Edge Device Benchmarks

[0125]

[0126] The sources of the existing open-source reproducible models are as follows:

[0127] The EDiffSR model is derived from Xiao Y, Yuan QQ, Jiang K, et al. EDiffSR: Anefficient diffusion probabilistic model for remote sensing image super-resolution[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 62:1-14;

[0128] The IDM-40 model originates from Liu XH, Gao SC, Zeng BH, Zhang LP, Wang T, Liu JZ, Zhang B C. Implicit Diffusion Models for Continuous Super-Resolution[J]. International Journal of Computer Vision, 2025, 133(9):6535-6557;

[0129] The SR3-40 model originates from Saharia C, Ho J, Chan W, Salimans T, Fleet DJ, Norouzi M. Image super-resolution via iterative refinement[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(4):4713-4726;

[0130] The HAT-L model is derived from Chen XY, Wang XT, Zhou JT, Qiao Y, Dong C. Activating more pixels in image super-resolution transformer[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023:22367-22377;

[0131] The LIIF model originates from Chen YB, Liu SF, Wang X L. Learning continuous image representation with local implicit image function[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021:8628-8638;

[0132] The SPSR model is derived from Ma C, Rao YM, Cheng Y, Chen C, Lu JW, Zhou J.Structure-preserving super resolution with gradient guidance[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020:7769-7778;

[0133] The SADN model is derived from Wu HL, Ni N, Zhang L B. Learning dynamic scaleawareness and global implicit functions for continuous-scale super-resolution of remote sensing images[J]. IEEE Transactions on Geoscience and RemoteSensing, 2023, 61:1-15.

[0134] In summary, the quality experiments verified that multi-space knowledge distillation can transfer the structure, denoising dynamics, and perceptual texture knowledge of the diffusion teacher model to the lightweight student model; the ablation experiments verified that each distillation space and the selective gating mechanism contribute to the final result; and the deployment experiments verified that the lightweight student model can solve the constraints of real-time response, power consumption, and memory on desktop and edge devices for remote sensing super-resolution.

[0135] Example 2:

[0136] This invention provides an electronic device including a processor and a memory. The memory stores program instructions for implementing the remote sensing image super-resolution method of multi-spatial knowledge distillation as described in Embodiment 1. The processor executes the program instructions stored in the memory to perform remote sensing image super-resolution.

[0137] The processor can also be referred to as a central processing unit (CPU). The processor can be a central processing unit, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a general-purpose microprocessor or other conventional processor capable of executing the steps of the method of this invention.

[0138] The methods described in Embodiment 1 of this invention are applicable to this electronic device embodiment. The specific functions implemented by this electronic device embodiment are the same as those in the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0139] Example 3:

[0140] This embodiment provides a storage medium storing a computer program or instructions capable of implementing the above-described multi-spatial knowledge distillation method for super-resolution of remote sensing images. When the computer program or instructions are run on a computer, the steps of the above-described multi-spatial knowledge distillation method for super-resolution of remote sensing images are executed.

[0141] The computer program or instructions may be stored in the storage medium in the form of a software product, including several instructions for causing a computer device or processor to execute all or part of the steps of the methods of various embodiments of the present invention. The computer device may be a personal computer, a server, or a network device. The storage medium includes media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, and may also be storage components in computers, servers, mobile phones, or tablet terminals.

[0142] The methods described in Embodiment 1 of this invention are all applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0143] This invention provides a remote sensing image super-resolution method based on multi-spatial knowledge distillation. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A multi-spatial knowledge distillation method for super-resolution of remote sensing images, characterized in that, Includes the following steps: Step 1: Obtain training samples of remote sensing images; Step 2: Build a lightweight student model and load the pre-trained teacher model; Step 3: Input the remote sensing image training samples into the lightweight student model and the pre-trained teacher model, and jointly train the lightweight student model through the latent variable space, the noisy prediction space and the clean latent variable reconstruction space to obtain the trained student model. Step 4: Input the low-resolution remote sensing image to be processed into the trained student model, and give the target scale and target output size to obtain the high-resolution remote sensing image.

2. The remote sensing image super-resolution method based on multi-spatial knowledge distillation according to claim 1, characterized in that, In step 2, the lightweight student model is a generative super-resolution network constrained by the training objective of the generative adversarial network, including: a prior encoder, a generator, and a continuous scale upsampling module; the discriminator network of the generative adversarial network is set independently of the lightweight student model, and only receives images and provides adversarial loss during the training phase. The teacher model employs a latent diffusion continuous-scale super-resolution model, comprising: a first-stage image encoder and image decoder of variational autoencoder, a conditional denoising U-shaped network, a conditional construction module, a continuous-scale upsampling module, a differential prior encoder, and a differential prior decoder.

3. The remote sensing image super-resolution method based on multi-spatial knowledge distillation according to claim 1, characterized in that, Step 3 includes: The remote sensing image training data is input into the student model and the teacher model for processing to obtain the student over-score result and the teacher over-score result. The student super-resolution results and the teacher super-resolution results are respectively input into the first stage image encoder of the variational autoencoder in the teacher model for processing to obtain the student latent variables and the teacher latent variables, and the first distillation loss is calculated. The latent variables of students and teachers after applying noise are processed by the conditional denoising U-shaped network in the teacher model to obtain the student prediction noise and the teacher prediction noise, and the second distillation loss is calculated. Based on the predicted noise, the latent variables after noise is applied, and the cumulative retention coefficient of the diffusion process, the clean latent variables of students and teachers are recovered respectively through the one-step inverse formula of the denoised diffusion probability model, and the third distillation loss is calculated by combining the selective gating mechanism. The student model is trained and its parameters are updated based on the total loss at the generator end to obtain the trained student model; the total loss at the generator end includes reconstruction loss, adversarial loss and various distillation losses.

4. The remote sensing image super-resolution method based on multi-spatial knowledge distillation according to claim 3, characterized in that, The reconstruction loss is expressed as: ， ， ， ， Where sr represents the super-resolution image output by the lightweight student model, and hr represents the high-resolution ground truth image. This represents the total amount of reconstruction losses. Indicates pixel reconstruction loss. This represents the perceived reconstruction loss. Represents texture statistics loss; Charbonnier represents Charbonnier robust pixel loss; , , ... This represents the L1 norm.

5. The remote sensing image super-resolution method based on multi-spatial knowledge distillation according to claim 3, characterized in that, The distillation losses include the first distillation loss, the second distillation loss, and the third distillation loss; First distillation loss Represented as: , in, This represents the pixel-wise mean of multiple sampling results from the teacher model; denoted as the first-stage image encoder in the variational autoencoder of the teacher model; sg represents the stopping gradient operation; SmoothL1 represents the smoothing L1 loss. Second distillation loss Represented as: , Where t represents the diffusion time step; This represents the loss weight corresponding to time step t; This represents the noise residual of the teacher-conditional denoising U-shaped network's prediction of noisy latent variables in students. This represents the teacher branch noise residual after the gradient stops; The third distillation loss Represented as: , in, This represents the estimate of the clean latent variables of the student, obtained by inversely deducing the student's noisy latent variables and student prediction noise through a denoising diffusion probability model in one step. This represents the estimate of the clean latent variables of teachers obtained from the noisy latent variables of teachers and the teacher prediction noise recovery after stopping the gradient; The time-step gating threshold represents the space for reconstructing clean latent variables.

6. The remote sensing image super-resolution method based on multi-spatial knowledge distillation according to claim 3, characterized in that, The implementation process of the selective gating mechanism includes: For each training sample, time steps are sampled from the time step set of the teacher diffusion model, and the cumulative retention coefficient, noise figure, and signal-to-noise ratio are calculated based on the noise scheduling. The student latent variable and the teacher latent variable are added to the same noise at the same time step to obtain the noisy student latent variable and the noisy teacher latent variable. The student prediction noise and the teacher prediction noise are obtained respectively by the frozen teacher conditional denoising U-shaped network. Determine whether the time step is less than or equal to the preset low-noise time step threshold, or determine whether the signal-to-noise ratio is higher than the preset signal-to-noise ratio threshold; If the judgment condition is met, the student clean latent variables and teacher clean latent variables are recovered respectively according to the one-step back-reasoning formula of the denoising diffusion probability model, and the third distillation loss is calculated; if the judgment condition is not met, the calculation of the third distillation loss is skipped, the loss is set to zero value which does not produce gradients, and the subsequent training steps are continued.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the remote sensing image super-resolution method of multi-spatial knowledge distillation as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, The system stores a computer program or instructions that, when executed on a computer, perform the steps of the remote sensing image super-resolution method of multi-spatial knowledge distillation as described in any one of claims 1 to 6.

Images