A method for repairing missing bands in Landsat 7 multispectral images
By constructing a multispectral conditional diffusion model and utilizing band-wise shared feature encoding and cross-band feature interaction, the problems of external reference dependence and spectral consistency in the restoration of missing strips in SLC-off images were solved, achieving high-quality multispectral image restoration.
Patent Information
- Application Number
- CN202611376000.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-07
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-09-07
AI Technical Summary
Existing SLC-off stripe loss restoration methods have limited applicability when relying on external reference images, and conventional image diffusion networks fail to effectively and explicitly model the inter-band relationships of multispectral images, resulting in poor spectral consistency of the restoration results.
A multispectral conditional diffusion model is constructed, including a noise prediction backbone network and a multispectral conditional coding branch. Through band-by-band shared feature coding, cross-band feature interaction, and spectral statistical conditional coding, the multi-scale conditional features and spectral statistical features of the image to be restored are used for reverse denoising, thereby achieving joint restoration of multiple reflection bands.
Without the need for external reference images, it improves the multispectral consistency of restoration results and the reconstruction quality of spatial structure, making it suitable for scenarios with rapid surface changes or insufficient reference data, and reducing the deviation of spectral relationships between bands.
Smart Images

Figure CN122865936B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of remote sensing image processing technology, specifically a method for repairing missing stripes in Landsat 7 multispectral images. Background Technology
[0002] Landsat 7's Enhanced Thematic Mapper Plus (ETM+) employs six reflection bands with a spatial resolution of 30 meters: blue band B1, green band B2, red band B3, near-infrared band B4, shortwave infrared 1 band B5, and shortwave infrared 2 band B7, covering the visible, near-infrared, and shortwave infrared spectral ranges. Imagery acquired by Landsat 7 after a permanent failure of its scan-line corrector (SLC) is called SLC-off imagery. SLC failure results in spatially corresponding striped data gaps in the aforementioned reflection bands, limiting the imagery's applications in land cover identification, spectral index calculation, and quantitative inversion.
[0003] Existing SLC-off stripe restoration methods mainly include non-reference methods that rely solely on the image itself and reference methods that utilize imagery from other temporal phases or other sensors. Traditional non-reference methods, such as inverse distance weighted interpolation and Kriging interpolation, are typically based on spatial interpolation for reconstruction, making it difficult to effectively restore the spatial structure and texture of missing areas. Reference methods, such as local histogram matching based on multi-temporal Landsat imagery and spatial-spectral radial basis function interpolation based on Sentinel-2 multispectral imagery, can utilize auxiliary imagery to provide missing information, but they require high temporal, spatial, and radiometric consistency between the reference image and the image to be restored, limiting their applicability in rapidly changing areas or scenarios where reliable reference data is insufficient.
[0004] Diffusion models, capable of learning image data distribution through progressive denoising, have been used for inpainting natural and remote sensing images. However, when conventional image diffusion networks are directly applied to multispectral remote sensing images, it is difficult to explicitly constrain the inter-band spectral relationships of the current image to be inpainted if only multiple spectral bands are used as common channels for joint input. Landsat 7's reflectance bands correspond to fixed visible, near-infrared, and shortwave infrared wavelength ranges. The reflectance relationship of the same pixel in different bands is determined by the spectral characteristics of the ground cover and is used for spectral index calculation and quantitative analysis. Therefore, even if the individual bands after inpainting have a continuous spatial structure, the resulting multi-band spectral vectors may still exhibit inter-band relationship deviations, affecting the multispectral consistency of the inpainted image. Furthermore, different images exhibit varying multispectral response characteristics due to factors such as ground cover type, imaging season, and atmospheric conditions. Relying solely on general features learned from training data to reconstruct missing areas is insufficient to adequately adapt to the differences in spectral distribution among different images to be inpainted, easily leading to inconsistencies between the inpainted results and the original image's multispectral features.
[0005] Therefore, without relying on external reference images, making full use of the correlation between different reflection bands of the Landsat 7 multispectral image to be restored, jointly reconstructing the strip missing areas of multiple reflection bands, and reducing the inter-band spectral deviation in the restoration results is of great significance for improving the restoration quality of SLC-off multispectral images. Summary of the Invention
[0006] The purpose of this invention is to provide a method for repairing missing bands in Landsat 7 multispectral images. This method aims to address the limitations of existing reference-based repair methods, which rely on external reference images and have limited applicability in scenarios with rapidly changing terrain or a lack of reliable reference data. Furthermore, when conventional image diffusion networks are directly used for multispectral image repair, they only treat each spectral band as a common channel input and lack an explicit modeling mechanism for the relationships between bands, which can easily lead to deviations in the spectral relationships between bands in the repair results.
[0007] The technical solution provided by this invention is: A method for restoring missing bands in Landsat 7 multispectral images includes the following steps: Step S1: Construct the training dataset; Obtain the stripe missing mask of the complete Landsat 7 six-band multispectral image and the real SLC-off image. Apply the stripe missing mask to the complete multispectral image to generate a simulated SLC-off multispectral image. Combine the complete multispectral image, the simulated SLC-off multispectral image and the corresponding stripe missing mask to form training samples.
[0008] Step S2: Construct a multispectral conditional diffusion model; The multispectral conditional diffusion model includes a forward diffusion process, a multispectral conditional noise prediction network, and a reverse denoising process; the multispectral conditional noise prediction network includes a noise prediction backbone network and a multispectral conditional coding branch; the multispectral conditional coding branch includes a band-by-band shared feature coding module, a cross-band feature interaction module, and a spectral statistical conditional coding module.
[0009] Step S3: Train the multispectral conditional noise prediction network; A six-channel noisy image is obtained by forward denoising the complete multispectral image. Multiscale conditional features and spectral statistical conditional vectors are obtained based on the simulated SLC-off multispectral image and the strip missing mask. The multispectral conditional noise prediction network is trained using the noise prediction mean square error loss, the missing region pixel reconstruction loss and the missing region spectral direction consistency loss.
[0010] Step S4: Perform reverse denoising on the SLC-off image to be repaired; Based on the SLC-off multispectral image to be repaired and its missing strip mask, the trained multispectral conditional noise prediction network is used to predict noise step by step and perform inverse denoising to obtain a complete multispectral predicted image.
[0011] Step S5: Generate the final restored image; The original effective pixels in the SLC-off multispectral image to be repaired are retained, and the missing strip regions are filled only using the complete multispectral prediction image to obtain the final repaired image.
[0012] Through the above scheme, the present invention introduces multi-scale conditional features extracted from missing multispectral images and their strip missing masks and obtained through cross-band feature interaction, as well as spectral statistical conditional vectors of known effective areas, in the reverse denoising process of the multispectral conditional diffusion model. This makes the denoising process subject to multi-dimensional conditional constraints, thereby achieving joint restoration of multiple reflection bands without the need for external reference images.
[0013] Further, in step S1, the complete Landsat 7 six-band multispectral image includes blue band B1, green band B2, red band B3, near-infrared band B4, shortwave infrared band 1 B5, and shortwave infrared band 2 B7; the band missing mask is a real band missing mask extracted from the Landsat 7 SLC-off image; the band missing mask is a binary mask, and the missing band position and the known valid position are identified by different values. By using the real SLC-off mask to construct training samples, the model can fully learn the real band missing morphology during the training phase, which is beneficial to improving the repair accuracy during the inference phase.
[0014] Further, in step S2, the noise prediction backbone network adopts a residual U-Net structure, including an encoding part, a decoding part, and skip connections. The encoding part extracts multi-scale features through step-by-step downsampling, the decoding part recovers spatial scale through step-by-step upsampling, and the skip connections fuse features corresponding to the encoding scale. The noise prediction backbone network takes the multi-channel noisy image at the current time step as input and outputs predicted noise with the same number of input channels. The residual U-Net structure preserves the detailed features of the encoding stage through skip connections, which is beneficial for recovering spatial details in the denoising output.
[0015] Specifically, in step S2, the multispectral conditional coding branch extracts multi-scale neighborhood features for each band as follows: For each band, the corresponding single-band image and the stripe missing mask are combined into a multi-channel input. Neighborhood features at different scales are extracted using a parameter-shared convolutional encoder, and corresponding learnable band codes are added to each scale to obtain band-by-band multi-scale features that retain the band axis. Using a parameter-shared convolutional encoder reduces the number of model parameters, while learnable band codes ensure that each band retains an independent feature representation, avoiding feature confusion between different bands.
[0016] Further, in step S2, the cross-band feature interaction module obtains the query matrix, key matrix, and value matrix through linear projection, calculates the correlation weights between different bands using scaled dot product attention, and adds the cross-band interaction results to the original band features using a residual method. The multi-band features after cross-band interaction at each scale are combined along the channel direction and mapped to multi-scale conditional features of the corresponding encoding scale through pointwise convolution. The scaled dot product attention mechanism can adaptively learn the correlation weights between different bands, while the residual connection method preserves the original band features, preventing information loss during the cross-band interaction process.
[0017] Further, step S3 includes: S3.1 Prepare training samples and add noise forward; Select complete multispectral image samples from the training dataset. Corresponding simulated SLC-off multispectral image samples And a single-channel stripe missing mask m; copy m along the band dimension to obtain a six-channel mask. ; Based on simulated SLC-off multispectral image samples Using the stripe missing mask m, we can calculate the multi-scale conditional features and the spectral statistical conditional vector. In each training iteration, a time step t is uniformly and randomly selected from the time step set {1,2,…T} to generate a time step that is consistent with the training time step set. The six-channel noisy image at time step t is calculated using six-channel standard Gaussian noise ε of the same size. ; S3.2 Predict noise and estimate complete multispectral imagery; Six-channel noisy images By inputting a multispectral conditional noise prediction network and introducing multi-scale conditional features and a joint conditional vector formed by time-step embedding and spectral statistical conditional vectors, a six-channel predicted noise is obtained. Calculate the prediction of the complete multispectral image corresponding to the current time step. .
[0018] S3.3 Calculate the loss function; The training loss function of the multispectral conditional noise prediction network comprehensively considers the mean square error loss of noise prediction. Loss of pixel reconstruction in missing areas loss of spectral orientation consistency with missing regions ; The total loss function is calculated as follows: in, and These are the weights for the pixel reconstruction loss and the spectral orientation consistency loss in the missing region, respectively. The mean square error loss of noise prediction ensures the basic denoising capability of the diffusion model, the pixel reconstruction loss in the missing region directly constrains the pixel-level reconstruction accuracy of the missing region, and the spectral orientation consistency loss constrains the multi-band joint prediction results from the spectral vector direction dimension. The synergistic optimization of the three losses is beneficial to simultaneously improve the spatial reconstruction quality and multispectral consistency. S3.4 Optimize network parameters Based on the total loss function End-to-end joint optimization of learnable parameters in a multispectral conditional noise prediction network is performed.
[0019] Furthermore, in step S4, the reverse denoising starts with multi-channel standard Gaussian noise, selects multiple non-zero time steps at equal intervals from a preset diffusion time step, and performs reverse sampling in descending order of time steps; each reverse sampling step does not introduce additional random noise; the multi-scale conditional features and the spectral statistical conditional vector remain unchanged during the reverse sampling process, and the time step embedding is generated based on the current time step and updated with the current time step. This deterministic sampling strategy, which does not introduce additional random noise, makes the reverse denoising process deterministic, which helps ensure the stability of the repair results.
[0020] Further, in step S5, the strip missing mask is copied along the band dimension to obtain a six-channel mask. Outside the strip missing area, the original effective value in the SLC-off multispectral image to be repaired is used, and inside the strip missing area, the predicted value of the complete multispectral prediction image obtained in step S4 is used to obtain the final repaired image.
[0021] Beneficial effects Compared with the prior art, the present invention has the following advantages: (1) In the model reasoning stage, this invention only requires inputting the single-scene SLC-off multispectral image to be repaired and its strip missing mask, without needing to obtain other temporal or other sensor-assisted images of the same area. This fundamentally eliminates the dependence of the reference-type repair method on the temporal, spatial and radiometric consistency of external reference images, and has a wider range of applicability in areas with rapid changes in the ground surface or in scenarios where reliable reference data is difficult to obtain.
[0022] (2) The present invention uses a multi-channel multispectral image composed of multiple reflection bands as a diffusion variable, and performs forward noise addition and reverse noise removal in the multi-band joint image space, so that each reflection band can be reconstructed in the same reverse noise removal process. Compared with the method of repairing band by band independently, it is beneficial to maintain the spatial correspondence between different bands in the repair result.
[0023] (3) The present invention sets up a multispectral conditional coding branch, extracts multi-scale neighborhood features from missing multispectral images and strip missing masks through a band-by-band shared feature coding module, and uses the spatial context information near the strip missing area to provide multi-scale conditional constraints for the repair of missing areas.
[0024] (4) In the conditional coding process, the present invention retains the independent feature representation of each band and explicitly models the correlation between the neighborhood features of each band through the cross-band feature interaction module, thereby enhancing the utilization of relevant information between different bands and helping to reduce the spectral relationship deviation between bands in the repair results.
[0025] (5) The present invention uses the spectral statistical characteristics of the known effective area of the image to be repaired as a condition to characterize the multispectral distribution characteristics of the current input image, and adjusts the denoising process accordingly. This is beneficial to enhance the model's ability to adapt to the differences in spectral distribution of different images, and to reduce the deviation between the repair results of missing areas and the spectral characteristics of the current input image. Attached Figure Description
[0026] Figure 1 This is a flowchart illustrating the method of the present invention.
[0027] Figure 2 This is a schematic diagram of the overall structure of the multispectral conditional diffusion model of the present invention. Detailed Implementation
[0028] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The described embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.
[0029] like Figure 1This invention discloses a method for repairing missing bands in Landsat 7 multispectral images, comprising the following steps: Step S1: Construct the training dataset This embodiment selects representative Landsat 7 images from different geographical regions worldwide. The selection of regions takes into account different climatic conditions and land cover types, including urban areas, farmland, vegetation, water bodies, bare land, and arid zones, to cover different spatial textures and spectral response conditions. The data used to construct the complete multispectral image set is selected from complete images without missing bands acquired before the scan line corrector malfunctioned. The images include blue band B1, green band B2, red band B3, near-infrared band B4, shortwave infrared 1 band B5, and shortwave infrared 2 band B7, with a spatial resolution of 30 meters for each of the six bands. The stored pixel values of the above six reflection bands are converted to floating-point surface reflectance according to the scaling factor and offset specified in the product; in this embodiment, the scaling factor is 0.0000275, and the offset is -0.2. Regions in the selected images that are cloudless, cloud-free, radiation-saturated, and have valid data in all six bands are selected to form a complete multispectral image set.
[0030] Landsat 7 ETM+ Collection 2 Level-2 SLC-off images acquired after a scan line corrector malfunction were used. Within the effective imaging range of the image, SLC-off stripe missing regions were extracted based on the missing data locations of the six reflection bands, forming a set of true stripe missing mask codes. The stripe missing mask is a single-channel binary mask M, where the stripe missing locations are set to 1, and known locations are set to 0.
[0031] Complete multispectral images and stripe missing masks are selected from the complete multispectral image set and the stripe missing mask set, respectively. The stripe missing mask is applied synchronously to the six reflection bands of the complete multispectral image to construct a simulated missing region in the complete multispectral image that has the same stripe missing shape as the real SLC-off image, thereby generating a simulated SLC-off multispectral image.
[0032] Let the complete multispectral image be denoted as X, and the mask for missing single-channel stripes be denoted as M. When performing element-wise operations with the six-band multispectral image, M is copied along the band dimension to obtain the six-channel mask. Let Y be the simulated SLC-off image, then: Equation (1) in, This indicates element-wise multiplication. According to equation (1), the six band pixel values corresponding to the strip missing regions in the simulated SLC-off multispectral image Y are all set to 0 as input placeholder values, while the reflectance values of the remaining regions remain unchanged as they are in the complete multispectral image X.
[0033] The complete multispectral image X, the simulated SLC-off multispectral image Y, and the corresponding stripe-deficient mask M are simultaneously cropped at the same spatial location to obtain a 256×256 pixel sub-image that simultaneously contains both known valid regions and stripe-deficient regions. The cropped complete multispectral sub-image is denoted as... The corresponding simulated SLC-off multispectral sub-image is denoted as The corresponding stripe loss mask is denoted as m. From , The training samples are composed of m and m, and the resulting training samples form the training dataset. (Note: In this paper, sub-images are represented by lowercase letters, such as m.) , and m; use uppercase letters for full images, such as X, and M) Step S2: Construct a multispectral conditional diffusion model A multispectral conditional diffusion model for stripe loss repair in Landsat 7 SLC-off multispectral images was constructed, and its overall structure is as follows: Figure 2 As shown. The multispectral conditional diffusion model includes a forward diffusion process, a multispectral conditional noise prediction network, and a reverse denoising process. The multispectral conditional noise prediction network includes a noise prediction backbone network and a multispectral conditional coding branch. The multispectral conditional coding branch includes a band-by-band shared feature coding module, a cross-band feature interaction module, and a spectral statistical conditional coding module. Specifically, it includes the following steps.
[0034] S2.1 Setting up the forward diffusion process Forward noise addition is performed directly within the complete six-band multispectral image space. The total diffusion time steps are set to T, and the single-step noise variance corresponding to each time step is determined according to a preset noise scheduling. In this embodiment, the total diffusion time steps T are set to 1000, and linear noise scheduling is used, where the initial noise variance... The variance of the termination noise is 1×10⁻⁴. The variance of the single-step noise at time step t is 2 × 10⁻². Determined according to the following formula: Equation (2) Determined based on the single-step noise variance: Equation (3) in, The single-step signal retention coefficients at time step t are... The cumulative signal retention coefficients from the 1st time step to the tth time step.
[0035] For complete multispectral images At time step t, add six channels of Gaussian noise of the same size. The corresponding six-channel noisy image was obtained. : Equation (4) This yields six-channel noisy images of the complete multispectral image at different diffusion time steps.
[0036] S2.2 Constructing a noise prediction backbone network The noise prediction backbone network employs a residual U-Net structure with three feature scales, taking the six-channel noisy image at the current time step as input. The input uses 3×3 convolutions to map the six-channel image into 64-channel features. The encoding part sets three feature channel scales: 64, 128, and 256. Each scale includes two residual convolutional blocks, and adjacent scales are downsampled using 3×3 convolutions. In this embodiment, the stride of the ordinary 3×3 convolution is set to 1, and the padding is set to 1; the stride of the downsampled convolution is set to 2, and the padding is set to 1. For a 256×256 pixel input image, represented as "number of channels × height × width," the feature sizes of the three encoding scales are 64×256×256, 128×128×128, and 256×64×64, respectively.
[0037] Each residual convolutional block includes two sets of group normalization layers, SiLU activation functions, and 3×3 convolutional layers arranged sequentially, with residual connections. Input features first pass through the first set of normalization layers, the first SiLU activation function, and the first 3×3 convolutional layer to obtain intermediate features. The conditional vector used to adjust the residual convolutional block is added to these intermediate features, then passes through the second set of normalization layers, the second SiLU activation function, and the second 3×3 convolutional layer, and is added to the residual branch to obtain the output of the residual convolutional block. In this embodiment, the number of groups in the group normalization layer is set to 8, and the numerical stability parameter is set to 1×10⁻⁵; the stride and padding of each 3×3 convolutional layer are set to 1. Within the same scale, the number of input channels and output channels of the residual convolutional block are the same, and the residual branches use identity connections. The formation of the conditional vector and its function in the residual convolutional block are explained in S2.6.
[0038] The decoding part employs a two-stage upsampling structure corresponding to the encoding part. The outputs of the second residual convolutional blocks in the first and second encoding scales serve as the skip connection features for the corresponding decoding scales, respectively; the output of the second residual convolutional block in the third encoding scale serves as the input to the decoding part.
[0039] The first stage upsamples the 256-channel features by bilinear interpolation by a factor of two, and then maps them to 128 channels through a 3×3 convolution. This 128-channel feature is then concatenated with the 128-channel skip connection feature corresponding to the second encoding scale along the channel direction to form a 256-channel feature, which is then mapped to 128 channels through a 3×3 convolution. The second stage upsamples the 128-channel feature by bilinear interpolation by a factor of two, and then maps it to 64 channels through a 3×3 convolution. This 64-channel feature is then concatenated with the 64-channel skip connection feature corresponding to the first encoding scale along the channel direction to form a 128-channel feature, which is then mapped to 64 channels through a 3×3 convolution.
[0040] After the two 3×3 convolutional layers used for feature fusion, a group normalization layer and a SiLU activation function are sequentially set, with the number of groups in the group normalization layer set to 8. In the decoding part, the stride of the above 3×3 convolutional layers is set to 1 and the padding is set to 1. No additional residual convolutional blocks are set in the decoding part.
[0041] At the end of the network, a group normalization layer, a SiLU activation function, and a 3×3 convolutional layer are set in sequence. The number of groups in the group normalization layer is set to 8, and the stride and padding of the 3×3 convolutional layer at the end are set to 1. The 64-channel features are mapped to 6 channels, and the output is a six-channel prediction noise with the same spatial size as the input image.
[0042] S2.3 Constructing a band-by-band shared feature coding module The band-by-band shared feature encoding module takes missing multispectral imagery and single-channel stripe missing mask as input. The missing multispectral imagery is 6×256×256 in size, and the stripe missing mask is 256×256 in size. For each band, the corresponding single-band imagery and stripe missing mask are combined along the channel direction to form a two-channel input, and neighborhood features are extracted by a parameter-shared convolutional encoder. The same convolution parameters are used for all six bands.
[0043] In this embodiment, the band-wise shared feature encoding module employs three scales. The first scale uses a 3×3 convolution with 2 input channels and 16 output channels, with a stride of 1 and padding of 1. The second scale uses a 3×3 convolution with 16 input channels and 32 output channels for downsampling, with a stride of 2 and padding of 1. The third scale uses a 3×3 convolution with 32 input channels and 64 output channels for downsampling, with a stride of 2 and padding of 1. Each scale's convolutional layer is followed by a group normalization layer and a SiLU activation function, with the group normalization layer having 8 groups and a numerical stability parameter of 1×10⁻⁵.
[0044] Learnable band codes are set at each scale. The size of the band coding parameter matrix for the three scales is 6×16, 6×32, and 6×64, respectively. Within the same scale, each of the six bands corresponds to an independent band coding vector, and band coding parameters are not shared between different scales.
[0045] At each scale, the corresponding band encoding vector is expanded along the spatial dimension and then added to the band's features after convolution, group normalization, and SiLU activation processing to obtain the band-wise neighborhood features for that scale. The band-wise neighborhood features from the first two scales are used as input to the convolutional encoding at the next scale. A feature tensor preserving the band axis is formed at the l-th scale. Its dimensions are 6× × × ,in, Let l be the feature dimension of the l-th scale. and These represent the height and width of the feature map at that scale, respectively. In this embodiment, the input image size is 256×256 pixels, and the feature tensor sizes for the three scales are 6×16×256×256, 6×32×128×128, and 6×64×64×64, respectively.
[0046] S2.4 Constructing a cross-band feature interaction module The three-scale band-wise neighborhood features obtained by the band-wise shared feature encoding module described in S2.3 are subjected to cross-band feature interaction. For any scale and spatial location p, the neighborhood features of the six bands at that location are arranged into a band feature matrix Z. For the l-th scale, the size of Z is 6× Linear projections are performed on the band feature matrix Z to obtain the query matrix Q, key matrix K, and value matrix V: , , Equation (5) in, , and For learnable projection matrices, all are of size 1. × The dimensions of Q, K, and V are all 6× .
[0047] In this embodiment, a single-head scaling dot product attention method is used for cross-band feature interaction: Equation (6) For each band, its correlation value with the six corresponding bands is Softmax normalized so that the sum of the six weights is 1; the weighted cross-band interaction result is added to the original band feature matrix Z using a residual method.
[0048] For each scale, the six band features after cross-band interaction are combined along the feature channel direction to obtain the combined features for that scale, and then channel mapping is performed through 1×1 convolution. The combined features for the three scales are mapped to 64, 128, and 256 channels through 1×1 convolution, respectively, to obtain the multi-scale conditional features for the three coding scales of the corresponding noise prediction backbone network. The stride of each 1×1 convolutional layer is set to 1, and the padding is set to 0.
[0049] S2.5 Constructing the Spectral Statistical Conditional Coding Module The spectral statistical condition encoding module generates a spectral statistical condition vector based on the spectral statistical information of the known effective region of the current input multispectral sample. Let the set of known effective pixels of the current input multispectral sample be... The number of valid pixels is known to be Calculate the mean and standard deviation of each of the six bands within the known effective area, and calculate the correlation coefficient between different bands. The mean is the arithmetic mean of all effective pixels in the known effective area, and the standard deviation is the population standard deviation of all effective pixels in the known effective area.
[0050] The correlation coefficient between the i-th and j-th bands Calculate according to the following formula: Equation (7) in, and Let be the mean and standard deviation of the i-th band within the known effective area, respectively. and Let be the mean and standard deviation of the j-th band within the known effective area, respectively. and Let be the reflectance of the i-th and j-th bands at position p, respectively. To prevent the constant stabilization term with a denominator of zero, in this embodiment, Set to 1×10⁻⁶.
[0051] For the six bands, the six means, six standard deviations, and 15 independent correlation coefficients (excluding the diagonal lines) of the upper triangular region of the correlation coefficient matrix are combined to form a 27-dimensional spectral statistical vector. The means and standard deviations of each band are arranged in the band order of B1, B2, B3, B4, B5, and B7, and the 15 correlation coefficients are arranged row-by-row according to the upper triangular region of the correlation coefficient matrix.
[0052] The 27-dimensional spectral statistical vector is input into a multilayer perceptron. In this embodiment, the multilayer perceptron includes two fully connected layers. The input and output dimensions of the first fully connected layer are 27 and 128, respectively. After passing through the SiLU activation function, the vector is input into the second fully connected layer. The input and output dimensions of the second fully connected layer are 128 and 256, respectively, resulting in a 256-dimensional spectral statistical conditional vector.
[0053] S2.6 Constructing a multispectral conditional noise prediction network The noise prediction backbone network described in S2.2 is integrated with the multispectral conditional coding branches described in S2.3 to S2.5. The three-scale band-wise neighborhood features extracted by the band-wise shared feature coding module described in S2.3 are input into the cross-band feature interaction module described in S2.4. Cross-band interaction is performed on the six band features at the same spatial location at each scale. The six interacted band features are then combined along the channel direction and passed through 1×1 convolutions to obtain multi-scale conditional features with 64, 128, and 256 channels respectively. These multi-scale conditional features are then input into the corresponding coding scales of the noise prediction backbone network, with the number of channels matching the number of channels in the three coding scales of the backbone network. The multi-scale conditional features are fused once before entering the first residual convolution block at each coding scale. At the three coding scales, the backbone features and the corresponding multi-scale conditional features are concatenated along the channel direction and channel compression is performed through 1×1 convolutions. The number of feature channels after concatenation at the three scales are 128, 256, and 512 respectively, which are mapped to 64, 128, and 256 channels after 1×1 convolution. The stride of each of the three 1×1 convolutional layers is set to 1, and the padding is set to 0. After conditional feature fusion, the resulting features are sequentially passed through two residual convolutional blocks at the corresponding scales.
[0054] The time step t is converted into a 64-dimensional time feature vector using a sine-cosine time step encoding method. In this embodiment, the frequency base of the sine-cosine time step encoding is set to 10000, where the sine and cosine components are each 32-dimensional. The resulting 64-dimensional time feature vector is input into two fully connected layers. The first fully connected layer maps it to 256-dimensional features, which are then activated by the SiLU activation function and input into the second fully connected layer. The second fully connected layer has both input and output dimensions of 256, resulting in a 256-dimensional time step embedding. The 256-dimensional time step embedding is concatenated along the feature dimensions with the 256-dimensional spectral statistical condition vector obtained by the spectral statistical condition encoding module described in S2.5, resulting in a 512-dimensional joint condition vector. Each of the three encoding scales includes two residual convolutional blocks, with a total of six independent linear mapping layers. The two linear mapping layers in the first encoding scale map the 512-dimensional joint condition vector to 64 dimensions; the two linear mapping layers in the second encoding scale map it to 128 dimensions; and the two linear mapping layers in the third encoding scale map it to 256 dimensions. For each residual convolutional block, the joint conditional vector is extended along the spatial dimension by the conditional vector obtained through the corresponding linear mapping layer, so that its spatial size is consistent with the intermediate feature output by the first 3×3 convolutional layer of that residual convolutional block, and then added element-wise to the intermediate feature. The added feature is then passed sequentially through the second set of normalization layers, the SiLU activation function, and the second 3×3 convolutional layer, and added to the residual branch. The joint conditional vector only applies to the six residual convolutional blocks in the encoding part.
[0055] The noise prediction backbone network and the multispectral conditional coding branch together constitute the multispectral conditional noise prediction network.
[0056] S2.7 Setting the reverse noise reduction process like Figure 2 As shown, the reverse denoising process is set up, starting with six-channel standard Gaussian noise with the same spatial size as the current model input image. K non-zero time steps are selected at equal intervals from the full diffusion time step defined in S2.1 to perform reverse sampling, and the reverse sampling sequence is formed in descending order of time steps.
[0057] Let the selected time step be , … and order , and Then the backsampled sequence satisfies: In this embodiment, the total diffusion time step T = 1000, the number of backsampling steps K = 100, and the interval between adjacent selected time steps is set to 10.
[0058] In each backsampling step, based on the current six-channel noisy image and corresponding time steps By combining multi-scale conditional features and spectral statistical conditional vector constraints, a multispectral conditional noise prediction network is used to obtain the six-channel prediction noise for the current time step. The predicted noise is further used to calculate the full multispectral image prediction corresponding to the current time step and update it to the next backsampling time step. For the same input image, the multi-scale conditional features and spectral statistical conditional vectors are calculated once before backsampling begins and remain unchanged throughout all backsampling steps; the time step embedding is generated based on the current time step and updated with the current time step.
[0059] For any time step t, calculate the full multispectral image prediction corresponding to the current time step based on the prediction noise. : Equation (8) At the current backsampling time step According to equation (8), we get And update the current image status to the next reverse sampling time step. : Equation (9) Each backsampling step does not introduce additional random noise. The above steps are repeated in descending order of time steps until the desired result is reached. To obtain complete multispectral predicted images.
[0060] The forward diffusion process described in S2.1, the multispectral conditional noise prediction network described in S2.2 to S2.6, and the reverse denoising process described in S2.7 together constitute... Figure 2 The multispectral conditional diffusion model is shown.
[0061] Step S3: Train the multispectral conditional noise prediction network The training dataset constructed in step S1 is used to train the multispectral conditional noise prediction network in the multispectral conditional diffusion model described in step S2, specifically including the following steps.
[0062] S3.1 Prepare training samples and add forward noise. Select complete multispectral image samples from the training dataset. Corresponding simulated SLC-off multispectral image samples And the single-channel stripe missing mask m. and The dimensions of each element are 6×256×256, and the dimension of m is 256×256. A six-channel mask is obtained by copying m along the band dimension. .
[0063] Following the methods described in S2.3 to S2.5, based on the simulated SLC-off multispectral image samples... By combining the stripe missing mask m, we obtain multi-scale conditional features and spectral statistical conditional vectors.
[0064] In each training iteration, a time step t is uniformly and randomly selected from the time step set {1,2,…T} to generate a time step that is consistent with the training time step set. Six-channel standard Gaussian noise of the same size is used, and the six-channel noisy image corresponding to time step t is obtained according to equation (4). .
[0065] S3.2 Predict noise and estimate complete multispectral image Six-channel noisy images Input the multispectral conditional noise prediction network, and introduce multi-scale conditional features and a joint conditional vector formed by time-step embedding and spectral statistical conditional vectors as described in S2.6 to obtain the six-channel predicted noise. Based on the six-channel prediction noise, the predicted complete multispectral image for the current time step is calculated according to equation (8). .
[0066] S3.3 Calculate the loss function The training of the multispectral conditional noise prediction network uses the noise prediction mean square error loss. Loss of pixel reconstruction in missing areas loss of spectral orientation consistency with missing regions .
[0067] The noise prediction mean square error loss is calculated using the following formula: Equation (10) Where H and W are the height and width of the training sub-image, respectively.
[0068] Let the set of missing pixels in the strip be... Its number of pixels is The pixel reconstruction loss in the missing area is calculated using the following formula: Equation (11) in, This represents element-wise multiplication. To prevent the constant stabilization term with a denominator of zero, in this embodiment... Set to 1×10⁻⁶.
[0069] The loss of spectral orientation consistency in the missing region is calculated using the following formula: Equation (12) in, and Let p be the true spectral vector and the predicted spectral vector, respectively. To prevent the constant stabilization term with a denominator of zero, in this embodiment... Set to 1×10⁻⁶.
[0070] The total loss function is calculated as follows: Equation (13) in, and The weights for the pixel reconstruction loss and the spectral orientation consistency loss of the missing region are respectively set to 0.1 in this embodiment.
[0071] S3.4 Optimize network parameters Based on the total loss function End-to-end joint optimization of learnable parameters in a multispectral conditional noise prediction network is performed.
[0072] In this embodiment, the AdamW optimizer is used, with an initial learning rate of 1×10⁻⁴, a batch size of 2, and 500,000 training iterations. The first-order moment estimation coefficient of the AdamW optimizer is set to 0.9, the second-order moment estimation coefficient is set to 0.999, the numerical stability parameter is set to 1×10⁻⁸, the weight decay coefficient is set to 1×10⁻⁴, and the learning rate remains constant during training.
[0073] After reaching the preset number of training iterations, the trained multispectral conditional noise prediction network is obtained.
[0074] Step S4: Perform reverse denoising on the SLC-off image to be repaired. Obtain the Landsat 7 SLC-off multispectral image to be repaired and its band missing mask. The image to be repaired includes six reflection bands: B1, B2, B3, B4, B5, and B7. The six-band floating-point reflectance data are obtained using the same reflectance conversion method as in step S1. Set the input value for the missing band location to 0, and set the missing location in the band missing mask to 1 and the known location to 0. Divide the multispectral image to be repaired and its band missing mask into blocks according to the same spatial location. Starting from the upper left corner of the image, divide it into sub-images of size 256×256 pixels along the row and column directions with a step size of 256 pixels.
[0075] If the number of rows or columns in the remaining area of the image edge is less than 256 pixels, the corresponding edge block window is adjusted to the inside of the image so that the size of the resulting edge sub-image is maintained at 256×256 pixels, and its outer boundary coincides with the corresponding boundary of the original image to be repaired.
[0076] For each sub-image to be repaired, a spectral statistical condition vector is obtained based on the spectral statistical information of the known effective region, according to the method described in S2.5; multi-scale conditional features are obtained based on the sub-image to be repaired and its missing strip mask, according to the methods described in S2.3 and S2.4. Starting with six-channel standard Gaussian noise of the same size as the sub-image to be repaired, backsampling is performed according to the conditional fusion method described in S2.6 and the back denoising process described in S2.7. In each backsampling step, the six-channel predicted noise of the current time step is obtained using a multispectral conditional noise prediction network, and the corresponding complete multispectral image prediction is calculated according to Equation (8), and updated to the next backsampling time step according to Equation (9). The above process is repeated until the backsampling is completed, and the complete six-band multispectral predicted image corresponding to the current sub-image to be repaired is obtained.
[0077] After performing the above reverse denoising on all the segmented sub-images in sequence, proceed to step S5 for stitching.
[0078] Step S5: Generate the final restored image The complete six-band multispectral prediction sub-images obtained in step S4 are stitched together according to their spatial positions in the original image to be repaired. For overlapping areas formed due to edge block window adjustments, the prediction results of the edge multispectral prediction sub-images within these overlapping areas are not included in the final stitching. Their prediction results are only used for edge areas not covered when dividing the blocks according to a 256-pixel step size, thereby obtaining a complete multispectral prediction image with a spatial range consistent with the original image to be repaired. The SLC-off multispectral image to be repaired, where the missing band positions have been set to 0, is denoted as... The missing stripe mask is denoted as M, and a six-channel mask is obtained by copying it along the band dimension. The original valid pixels in the image to be restored are preserved, and the missing stripe areas are filled only using the complete multispectral prediction image to obtain the final restored image. : Equation (14) in, This indicates element-wise multiplication. Outside of areas with missing stripes, the output uses the original valid values from the image to be restored; within areas with missing stripes, the output uses the predicted values from the multispectral conditional diffusion model. The final output image maintains the original image to be restored, retaining its 30-meter spatial resolution, geographic transformation parameters, and projection information.
[0079] The restoration effect of the method of the present invention is evaluated by simulating SLC-off images with known true values. The restoration results are compared with the corresponding true values in the strip missing areas, and the restoration quality is evaluated by indicators such as peak signal-to-noise ratio, structural similarity and spectral angle.
[0080] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. After reading this specification, those skilled in the art can make various modifications or variations to the technical solutions without departing from the concept of the present invention, and all such modifications or variations fall within the protection scope of the present invention.
Claims
1. A method for repairing missing bands in Landsat 7 multispectral images, characterized in that, Includes the following steps: Step S1: Construct the training dataset; Obtain the stripe missing mask of the complete Landsat 7 six-band multispectral image and the real SLC-off image, apply the stripe missing mask to the complete multispectral image to generate the simulated SLC-off multispectral image, and use the complete multispectral image, the simulated SLC-off multispectral image and the corresponding stripe missing mask to form training samples. Step S2: Construct a multispectral conditional diffusion model; The multispectral conditional diffusion model includes a forward diffusion process, a multispectral conditional noise prediction network, and a reverse denoising process; the multispectral conditional noise prediction network includes a noise prediction backbone network and a multispectral conditional coding branch; the multispectral conditional coding branch includes a band-by-band shared feature coding module, a cross-band feature interaction module, and a spectral statistical conditional coding module. Step S3: Train the multispectral conditional noise prediction network; A six-channel noisy image is obtained by forward denoising the complete multispectral image. Multi-scale conditional features and spectral statistical conditional vectors are obtained based on the simulated SLC-off multispectral image and the strip missing mask. The multispectral conditional noise prediction network is trained using noise prediction mean square error loss, missing region pixel reconstruction loss and missing region spectral orientation consistency loss. Step S4: Perform reverse denoising on the SLC-off image to be repaired; Based on the SLC-off multispectral image to be repaired and its missing strip mask, the trained multispectral conditional noise prediction network is used to predict noise step by step and perform inverse denoising to obtain a complete multispectral prediction image. Step S5: Generate the final restored image; The original effective pixels in the SLC-off multispectral image to be repaired are retained, and the missing strip regions are filled only using the complete multispectral prediction image to obtain the final repaired image.
2. The method according to claim 1, characterized in that, In step S1, the complete Landsat 7 six-band multispectral image includes blue band B1, green band B2, red band B3, near-infrared band B4, shortwave infrared band 1 B5, and shortwave infrared band 2 B7; the band missing mask is the real band missing mask extracted from the Landsat 7 SLC-off image; the band missing mask is a binary mask, and the missing band position and the known valid position are identified by different values.
3. The method according to claim 1, characterized in that, In step S2, the noise prediction backbone network adopts a residual U-Net structure, including an encoding part, a decoding part, and skip connections. The encoding part extracts multi-scale features through stepwise downsampling, the decoding part recovers the spatial scale through stepwise upsampling, and fuses features corresponding to the encoding scale through the skip connections. The noise prediction backbone network takes the multi-channel noisy image at the current time step as input and outputs the same number of predicted noise as the number of input channels.
4. The method according to claim 1, characterized in that, In step S2, the method for extracting multi-scale neighborhood features of each band by the multispectral conditional coding branch is as follows: for each band, the corresponding single-band image and the strip missing mask are combined into a multi-channel input, and neighborhood features at different scales are extracted by a parameter-shared convolutional encoder. The corresponding learnable band code is added to each scale to obtain band-by-band multi-scale features that retain the band axis.
5. The method according to claim 1, characterized in that, In step S2, the cross-band feature interaction module obtains the query matrix, key matrix and value matrix through linear projection, calculates the association weights between different bands using scaling dot product attention, and adds the cross-band interaction results to the original band features in a residual manner; the multi-band features after cross-band interaction at each scale are combined along the channel direction and mapped to multi-scale conditional features of the corresponding coding scale through pointwise convolution.
6. The method according to claim 1, characterized in that, In step S2, the spectral statistical conditional encoding module combines the mean, standard deviation, and inter-band correlation coefficient of each band within the known effective region into a spectral statistical vector, and maps it into a spectral statistical conditional vector through a multilayer perceptron. The time step embedding is concatenated with the spectral statistical conditional vector along the feature dimension to form a joint conditional vector. After being mapped by the independent linear mapping layer corresponding to each residual convolutional block, the joint conditional vector is expanded along the spatial dimension and added element by element to the intermediate features of the residual convolutional block.
7. The method according to claim 1, characterized in that, Step S3 is as follows: S3.1 Prepare training samples and add noise forward; Select complete multispectral image samples from the training dataset. Corresponding simulated SLC-off multispectral image samples and single-channel stripe missing mask m; A six-channel mask is obtained by copying m along the band dimension. ; Based on simulated SLC-off multispectral image samples Using the stripe missing mask m, we can calculate the multi-scale conditional features and the spectral statistical conditional vector. In each training iteration, a time step t is uniformly and randomly selected from the time step set {1,2,…T} to generate a time step that is consistent with the training time step set. The six-channel noisy image at time step t is calculated using six-channel standard Gaussian noise ε of the same size. ; S3.2 Predict noise and estimate complete multispectral imagery; Six-channel noisy images By inputting a multispectral conditional noise prediction network and introducing multi-scale conditional features and a joint conditional vector formed by time-step embedding and spectral statistical conditional vectors, a six-channel predicted noise is obtained. Calculate the prediction of the complete multispectral image corresponding to the current time step. ; S3.3 Calculate the loss function; The training loss function of the multispectral conditional noise prediction network comprehensively considers the mean square error loss of noise prediction. Loss of pixel reconstruction in missing areas loss of spectral orientation consistency with missing regions ; The total loss function is calculated as follows: in, and These are the weights for the pixel reconstruction loss and the spectral orientation consistency loss in the missing region, respectively. The mean square error loss of noise prediction ensures the basic denoising capability of the diffusion model, the pixel reconstruction loss in the missing region directly constrains the pixel-level reconstruction accuracy of the missing region, and the spectral orientation consistency loss constrains the multi-band joint prediction results from the spectral vector direction dimension. The synergistic optimization of the three losses is beneficial to simultaneously improve the spatial reconstruction quality and multispectral consistency. S3.4 Optimize network parameters Based on the total loss function End-to-end joint optimization of learnable parameters in a multispectral conditional noise prediction network is performed.
8. The method according to claim 7, characterized in that, The noise prediction mean square error loss is calculated using the following formula: Where H and W are the height and width of the training sub-image, respectively; Let the set of missing pixels in the strip be... Its number of pixels is The pixel reconstruction loss in the missing area is calculated using the following formula: in, This represents element-wise multiplication. To prevent constant stabilizing terms with a denominator of zero; The loss of spectral orientation consistency in the missing region is calculated using the following formula: in, and Let p be the true spectral vector and the predicted spectral vector, respectively. To prevent the constant stable term with a denominator of zero.
9. The method according to claim 1, characterized in that, In step S4, the reverse denoising starts with multi-channel standard Gaussian noise, selects multiple non-zero time steps at equal intervals from the preset diffusion time step, and performs reverse sampling in descending order of time steps; Each backsampling step does not introduce additional random noise; the multi-scale conditional features and the spectral statistical conditional vector remain unchanged during the backsampling process, and the time step embedding is generated based on the current time step and updated with the current time step.
10. The method according to claim 1, characterized in that, In step S5, the strip missing mask is copied along the band dimension to obtain a six-channel mask. Outside the strip missing area, the original effective value in the SLC-off multispectral image to be repaired is used, and inside the strip missing area, the predicted value of the complete multispectral prediction image obtained in step S4 is used to obtain the final repaired image.
Citation Information
Patent Citations
Remote sensing image lost pixel recovery method for failure of Landsat-7 scanning line corrector
CN111476723A
Remote sensing image super-resolution reconstruction method and device, storage medium and equipment
CN121032800A
