Satellite image super-resolution reconstruction method and device

Abstract

This application proposes a method and apparatus for super-resolution reconstruction of satellite images, relating to the field of satellite image reconstruction technology. The method includes: performing parallel high-resolution reconstruction of original low-resolution satellite images using a deep generative super-resolution network to obtain a set of high-resolution candidate images; constructing pairs of high-resolution candidate images, calculating the normalized mutual information value of each high-resolution candidate image pair within a local window centered on each pixel, and generating a set of mutual information heatmaps based on the normalized mutual information values; calculating the pixel confidence score of each high-resolution candidate image based on the set of mutual information heatmaps using a majority voting mechanism to obtain a set of pixel-level confidence maps; performing weighted fusion of all high-resolution candidate images based on the set of pixel-level confidence maps to obtain an initial fused image; globally optimizing the initial fused image using an energy function to obtain a final super-resolution reconstructed image; and performing Earth observation based on the final super-resolution reconstructed image.

Description

Technical Field

[0001] This application relates to the field of satellite image reconstruction technology, and in particular to a method and apparatus for super-resolution reconstruction of satellite images. Background Technology

[0002] Satellite imagery super-resolution reconstruction technology aims to recover high-resolution images from low-resolution images and is an important means of improving the spatial resolution of satellite images.

[0003] Existing super-resolution reconstruction methods mostly employ a single reconstruction algorithm, making it difficult to maintain optimal performance across all land cover types simultaneously. Global uniform processing leads to quality degradation in some areas. There is a lack of effective quantitative monitoring and suppression mechanisms for artifacts, which can easily mislead subsequent land cover interpretation. Multi-image fusion lacks an adaptive weighting mechanism based on pixel-level reliability, making it impossible to achieve adaptive optimal fusion. The lack of global optimization for fused images easily leads to spectral distortion, resulting in low resolution of satellite image reconstruction and affecting the accuracy of Earth observation. Summary of the Invention

[0004] To address the aforementioned problems, in a first aspect, the present invention provides a satellite image super-resolution reconstruction method, comprising: The original low-resolution satellite images are acquired, and parallel high-resolution reconstruction is performed on the original low-resolution satellite images through a deep generative super-resolution network to obtain a set of high-resolution candidate images. Traverse each high-resolution candidate image in the high-resolution candidate image set, construct pairs of high-resolution candidate image pairs, calculate the normalized mutual information value of the high-resolution candidate image pairs within a local window centered on each pixel, and generate a set of mutual information heatmaps based on the normalized mutual information values. Based on the mutual information heatmap set, pixel confidence is calculated for each high-resolution candidate image through a majority voting mechanism to obtain a pixel-level confidence map set; We perform weighted fusion of all high-resolution candidate images based on the pixel-level confidence map set to obtain an initial fused image; construct an energy function and perform global optimization of the initial fused image using the energy function to obtain the final super-resolution reconstructed image; and conduct Earth observation based on the final super-resolution reconstructed image.

[0005] In one embodiment, a set of high-resolution candidate images is obtained by performing parallel high-resolution reconstruction on the original low-resolution satellite imagery using a deep generative super-resolution network, including: High-resolution reconstruction of original low-resolution satellite images is performed by embedding a variety of heterogeneous super-resolution reconstruction algorithms into a deep generative super-resolution network. High-resolution candidate images corresponding to each heterogeneous super-resolution reconstruction algorithm are obtained. The heterogeneous super-resolution reconstruction algorithms include diffusion super-resolution reconstruction algorithm, streaming super-resolution reconstruction algorithm and generative adversarial super-resolution reconstruction algorithm. A high-resolution candidate image set is constructed by using all high-resolution candidate images.

[0006] In one embodiment, the normalized mutual information value of high-resolution candidate image pairs is calculated within a local window centered on each pixel, and a set of mutual information heatmaps is generated based on the normalized mutual information values, including: For high-resolution candidate image pairs, a local window of a preset size is extracted, centered on each pixel in the high-resolution candidate image pair. Within a local window, extract the grayscale histogram of each high-resolution candidate image pair, calculate the normalized mutual information value based on the grayscale histogram, and assign the normalized mutual information value to the pixel. The mutual information heatmap of high-resolution candidate image pairs is constructed by using the normalized mutual information values ​​of all pixels, and the set of mutual information heatmaps is constructed by using all mutual information heatmaps.

[0007] In one embodiment, based on a set of mutual information heatmaps, a pixel-level confidence map set is obtained by calculating the confidence score of each high-resolution candidate image through a majority voting mechanism, including: For the i-th high-resolution candidate image Pixels High-resolution candidate images are obtained from a set of mutual information heatmaps. The normalized mutual information value between the image and the remaining N-1 high-resolution candidate images, where N is the total number of high-resolution candidate images; The remaining high-resolution candidate images with normalized mutual information values ​​greater than or equal to a preset mutual information threshold are assigned to the consensus set, while the remaining high-resolution candidate images with normalized mutual information values ​​less than the preset mutual information threshold are assigned to the divergence set. Pixels are then calculated based on the consensus set and the divergence set. Confidence level; Calculate high-resolution candidate images The confidence scores of all pixels are used to construct a high-resolution candidate image. The pixel-level confidence map is constructed by combining the pixel-level confidence maps of all high-resolution candidate images.

[0008] In one embodiment, pixels are calculated based on the consensus set and the divergence set. The confidence level includes: When the number of images contained in the divergence set is greater than When; will pixel The mean of the normalized mutual information values ​​of all high-resolution candidate images in the consensus set is used as the consensus strength; the pixel... The mean of the normalized mutual information values ​​of all high-resolution candidate images in the divergence set is used as the divergence strength; the product of the consensus strength and the divergence strength is used as the pixel value. Confidence level; When the number of images contained in the divergence set is less than or equal to When; will pixel The mean of the normalized mutual information values ​​with all other high-resolution candidate images is used as the average consensus score. ; to pixels The standard deviation of the normalized mutual information values ​​with all other high-resolution candidate images is used as the consensus volatility. ;Will As pixels The confidence level.

[0009] In one embodiment, a weighted fusion of all high-resolution candidate images is performed based on a set of pixel-level confidence maps to obtain an initial fused image, including: Perform Laplacian pyramid decomposition on each high-resolution candidate image in the high-resolution candidate image set to obtain the Laplacian pyramid coefficients of the high-resolution candidate image at each level. Gaussian pyramid decomposition is performed on each pixel-level confidence map in the set of pixel-level confidence maps to obtain the Gaussian pyramid coefficients of the pixel-level confidence map at each level; wherein, the number of levels of the Gaussian pyramid is the same as the number of levels of the Laplace pyramid. At each level, the weighted average of the Laplace pyramid coefficients is calculated based on the Gaussian pyramid coefficients to obtain the fused Laplace pyramid coefficient for each level. The initial fused image is obtained by reconstructing the Laplacian pyramid using the fused Laplacian pyramid coefficients of all layers.

[0010] In one embodiment, an energy function is constructed, and the initial fused image is globally optimized using the energy function to obtain the final super-resolution reconstructed image, including: An energy function is constructed, which includes a data fidelity term, a prior constraint term, and an edge preservation regularization term. The data fidelity term is used to constrain the current optimization result to remain consistent with the original low-resolution satellite image after downsampling. The prior constraint term is used to constrain the deviation of the current optimization result from the initial fused image. The edge preservation regularization term is used to keep the image edges sharp in the current optimization result. The initial fused image is iteratively optimized with the goal of minimizing the energy function. When the value of the energy function converges to a preset threshold, the current optimization result is used as the final super-resolution reconstructed image.

[0011] Secondly, the present invention provides a satellite image super-resolution reconstruction apparatus for implementing the aforementioned satellite image super-resolution reconstruction method, the apparatus comprising: The high-resolution candidate image set acquisition module is used to acquire the original low-resolution satellite imagery and perform parallel high-resolution reconstruction on the original low-resolution satellite imagery through a deep generative super-resolution network to obtain a high-resolution candidate image set. The mutual information heatmap set acquisition module is used to traverse each high-resolution candidate image in the high-resolution candidate image set, construct pairs of high-resolution candidate image pairs, calculate the normalized mutual information value of the high-resolution candidate image pairs within a local window centered on each pixel, and generate a mutual information heatmap set based on the normalized mutual information value. The pixel-level confidence map set acquisition module is used to calculate the pixel confidence of each high-resolution candidate image based on the mutual information heatmap set through a majority voting mechanism, and obtain the pixel-level confidence map set. The Earth observation module is used to perform weighted fusion of all high-resolution candidate images based on a set of pixel-level confidence maps to obtain an initial fused image; construct an energy function and use the energy function to globally optimize the initial fused image to obtain the final super-resolution reconstructed image; and perform Earth observation based on the final super-resolution reconstructed image.

[0012] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the satellite image super-resolution reconstruction method.

[0013] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the satellite image super-resolution reconstruction method.

[0014] The present invention has the following beneficial effects: 1. Parallel high-resolution reconstruction of original low-resolution satellite imagery is performed using a deep generative super-resolution network. This leverages the complementarity of heterogeneous super-resolution reconstruction algorithms based on various generative principles in texture restoration, spectral fidelity, and structural reconstruction, avoiding the reliability issues of single algorithms in complex terrain areas. By calculating normalized mutual information values ​​between high-resolution candidate image pairs and generating a set of mutual information heatmaps, the abstract differences between algorithms are transformed into a visualized spatial distribution, enabling unsupervised mutual verification between algorithms without relying on pre-trained artifact detection network models. A majority voting mechanism is used to calculate the confidence score of each high-resolution candidate image pixel-by-pixel, automatically identifying and suppressing artifacts generated by a few unreliable algorithms based on the consensus of the majority of algorithms, effectively solving the problem of difficulty in quantifying and directionally suppressing artifacts in deep learning methods. Weighted fusion of high-resolution candidate images is performed using a set of pixel-level confidence maps as weights, allowing different terrain areas to adaptively select the optimal algorithm features, achieving pixel-level adaptive optimal fusion. An energy function is constructed to globally optimize the initial fused image, ensuring that the optimization result remains consistent with the original low-resolution satellite imagery after downsampling under data fidelity constraints, solving the spectral distortion problem and improving the resolution of satellite image reconstruction.

[0015] 2. By extracting local windows pixel by pixel and calculating normalized mutual information values ​​based on grayscale histograms, the similarity between high-resolution candidate images is quantified into a normalized index, enabling precise expression of spatial consistency differences between the outputs of different algorithms. After normalization, the mutual information value is unaffected by changes in the grayscale range and size of the image within the local window. The mutual information values ​​between different candidate image pairs can be compared and thresholded on the same scale, providing a unified and comparable benchmark for subsequent majority voting mechanisms. By traversing all pixel positions to generate a mutual information heatmap, the spatial distribution of consistency between algorithms is presented intuitively in the form of a two-dimensional image. High mutual information regions correspond to the real texture of algorithm consensus, while low mutual information regions correspond to the potential artifact locations of algorithm divergences, providing a spatially discriminative basis for the accurate localization of artifact pixels and the calculation of confidence.

[0016] 3. By employing a majority voting mechanism and dual-path confidence calculation, a fine-grained quantification of pixel-level reliability for each high-resolution candidate image is achieved. When a majority of algorithms reach consensus at a certain pixel, the product of the average consensus score and the consensus stability factor is used as the confidence score. This reflects the overall level of consistency among algorithms while also penalizing unstable consensus situations where some algorithms have high support while others have low support, avoiding overestimation of confidence scores through simple arithmetic averaging. When an algorithm disagrees with the majority of algorithms, the product of the consensus strength and the disagreement strength is used as the confidence score. This not only utilizes the consensus information of the majority of algorithms to provide a reference benchmark for that pixel but also quantifies the degree of unreliability through the disagreement strength, assigning different levels of low confidence scores to extreme artifacts and critical artifacts, rather than simply assigning them to zero or using binary judgment. In subsequent weighted fusion, the confidence score can be used as a continuous weight to perform fine-grained weighting of candidate images, ensuring that artifact regions are smoothly suppressed rather than abruptly removed, effectively avoiding splicing marks or unnatural transitions in the fusion results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a structural diagram of the device according to an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0020] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0021] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0022] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0023] Reference Figure 1 The present invention provides a method for super-resolution reconstruction of satellite images, including steps S100-S400.

[0024] The S100 acquires raw low-resolution satellite imagery and performs parallel high-resolution reconstruction on the raw low-resolution satellite imagery using a deep generative super-resolution network to obtain a set of high-resolution candidate images.

[0025] In some embodiments, a high-resolution candidate image set is obtained by performing parallel high-resolution reconstruction on the original low-resolution satellite imagery using a deep generative super-resolution network, including: High-resolution reconstruction of original low-resolution satellite images is performed by embedding a variety of heterogeneous super-resolution reconstruction algorithms into a deep generative super-resolution network. High-resolution candidate images corresponding to each heterogeneous super-resolution reconstruction algorithm are obtained. The heterogeneous super-resolution reconstruction algorithms include diffusion super-resolution reconstruction algorithm, streaming super-resolution reconstruction algorithm and generative adversarial super-resolution reconstruction algorithm. A high-resolution candidate image set is constructed by using all high-resolution candidate images.

[0026] In some embodiments, the deep generative super-resolution network embeds three heterogeneous super-resolution reconstruction algorithms: diffusion super-resolution reconstruction, streaming super-resolution reconstruction, and generative adversarial super-resolution reconstruction. These three heterogeneous super-resolution reconstruction algorithms differ fundamentally in their generation principles: diffusion super-resolution reconstruction recovers high-resolution images from random noise through a stepwise denoising process; streaming super-resolution reconstruction establishes a bijective mapping between the latent space and the image space through an invertible transformation; and generative adversarial super-resolution reconstruction recovers high-frequency details through adversarial training between the generator and the discriminator.

[0027] The original low-resolution satellite imagery is denoted as ,in The elevation of the original low-resolution satellite image. The width of the original low-resolution satellite image. This represents the number of channels in the original low-resolution satellite image.

[0028] Original low-resolution satellite imagery Simultaneously, three heterogeneous super-resolution reconstruction algorithms—diffusion-based super-resolution reconstruction, streaming-based super-resolution reconstruction, and generative adversarial super-resolution reconstruction—are input. These three algorithms independently and in parallel perform high-resolution reconstruction and output corresponding high-resolution candidate images.

[0029] The high-resolution reconstruction process of the diffusion-based super-resolution reconstruction algorithm is as follows: First, the original low-resolution satellite imagery is... An initial high-resolution image is obtained by upsampling to the target resolution using bicubic interpolation. Then, with As a condition, a pre-trained stable diffusion model is used as a generation prior, and the structure and texture details of the remote sensing image are recovered through a multi-level prior guidance mechanism. After step-by-step iterative denoising (in this embodiment) This process is repeated multiple times to obtain a set of high-resolution candidate images corresponding to the diffusion-based super-resolution reconstruction algorithm. .

[0030] The high-resolution reconstruction process of the streaming super-resolution reconstruction algorithm is as follows: The streaming super-resolution reconstruction algorithm is a generative super-resolution network based on a flow model, which establishes a bijective mapping between the latent space and the image space through invertible transformation. First, the original low-resolution satellite image is... An initial high-resolution image is obtained by upsampling to the target resolution using bicubic interpolation. Then, through the encoder Will Mapping to the latent space yields the latent representation. ,in These are the parameters of the encoder. They represent the latent representation in the latent space. To improve the resolution, a high-resolution latent representation is obtained. Then through the decoder Will By inversely mapping back to image space, we obtain the high-resolution candidate image corresponding to the streaming super-resolution reconstruction algorithm. ,in These are the parameters of the decoder, and Repeat the above steps multiple times to obtain a set of high-resolution candidate images corresponding to the streaming super-resolution reconstruction algorithm. .

[0031] The high-resolution reconstruction process of the generative adversarial super-resolution reconstruction algorithm is as follows: The generative adversarial super-resolution reconstruction algorithm includes a generator. and discriminator Generator Receive raw low-resolution satellite imagery As input, the output is a high-resolution candidate image generated by the adversarial super-resolution reconstruction algorithm. Generator A residual dense network structure is employed to extract deep features and reconstruct high-resolution details through multiple layers of residual dense blocks. Discriminator This is used to distinguish the high-resolution candidate images output by the generator from real high-resolution satellite imagery, driving the generator to recover high-frequency textures that conform to the real data distribution. Repeating the above steps multiple times yields a set of high-resolution candidate images corresponding to the generative adversarial super-resolution reconstruction algorithm. .

[0032] The set of high-resolution candidate images corresponding to the diffusion super-resolution reconstruction algorithm High-resolution candidate image set corresponding to streaming super-resolution reconstruction algorithm High-resolution candidate image set corresponding to generative adversarial super-resolution reconstruction algorithm Together they constitute a high-resolution candidate image set. ; Let i be the i-th high-resolution candidate image, and the total number of high-resolution candidate images is . The height of all high-resolution candidate images is [missing information]. The width is The number of channels is .

[0033] S200 iterates through each high-resolution candidate image in the set of high-resolution candidate images, constructs pairs of high-resolution candidate images, calculates the normalized mutual information value of the high-resolution candidate image pairs within a local window centered on each pixel, and generates a set of mutual information heatmaps based on the normalized mutual information values.

[0034] In some embodiments, normalized mutual information values ​​for high-resolution candidate image pairs are calculated within a local window centered on each pixel, and a set of mutual information heatmaps is generated based on the normalized mutual information values, including: For high-resolution candidate image pairs, a local window of a preset size is extracted, centered on each pixel in the high-resolution candidate image pair. Within a local window, extract the grayscale histogram of each high-resolution candidate image pair, calculate the normalized mutual information value based on the grayscale histogram, and assign the normalized mutual information value to the pixel. The mutual information heatmap of high-resolution candidate image pairs is constructed by using the normalized mutual information values ​​of all pixels, and the set of mutual information heatmaps is constructed by using all mutual information heatmaps.

[0035] In some embodiments, a high-resolution candidate image set The number of all pairwise high-resolution candidate image pairs is For any two distinct high-resolution candidate images, let the high-resolution candidate image pair be denoted as . ,in and .

[0036] For high-resolution candidate image pairs Normalized mutual information values ​​are calculated pixel-by-pixel. The spatial location of the candidate image pair is used as the reference. pixels Extract a local window of a preset size centered on the target area. ,in , The height of the local window is... Width is , and All are odd numbers; in this embodiment, we take... Pixel.

[0037] In local window Within, extract high-resolution candidate images to medium-high resolution candidate images. The grayscale histogram is used to calculate the probability of each gray level appearing within a local window. Information entropy within a local window Similarly, extract high-resolution candidate images. The grayscale histogram is used to calculate... Information entropy within a local window Furthermore, the joint probability of grayscale value pairs appearing within a local window is calculated. and Joint information entropy within a local window .

[0038] According to information entropy , and joint information entropy Calculate high-resolution candidate image pairs In pixels The mutual information value at the location, the mutual information value is and Sum minus The mutual information values ​​are normalized to obtain normalized mutual information values. The normalization calculation formula is:

[0039] Normalized mutual information value The range of values ​​is .when When, it represents a high-resolution candidate image. and In pixels Highly consistent; when When, it indicates that the two are in pixels The differences are almost completely inconsistent.

[0040] The calculated normalized mutual information value Assign pixels Traverse high-resolution candidate images The mutual information heatmap of the high-resolution candidate image pair is composed of the normalized mutual information values ​​of all pixels at all spatial locations. .

[0041] Iterate through all pairs of high-resolution candidate images in the high-resolution candidate image set, performing the above process for each pair, generating a total of A set of mutual information heatmaps is composed of all mutual information heatmaps. .

[0042] The S300 uses a set of mutual information heatmaps and a majority voting mechanism to calculate the pixel confidence score of each high-resolution candidate image to obtain a set of pixel-level confidence maps.

[0043] In some embodiments, based on a set of mutual information heatmaps, pixel confidence scores are calculated for each high-resolution candidate image using a majority voting mechanism to obtain a set of pixel-level confidence maps, including: For the i-th high-resolution candidate image Pixels High-resolution candidate images are obtained from a set of mutual information heatmaps. The normalized mutual information value between the image and the remaining N-1 high-resolution candidate images, where N is the total number of high-resolution candidate images; The remaining high-resolution candidate images with normalized mutual information values ​​greater than or equal to a preset mutual information threshold are assigned to the consensus set, while the remaining high-resolution candidate images with normalized mutual information values ​​less than the preset mutual information threshold are assigned to the divergence set. Pixels are then calculated based on the consensus set and the divergence set. Confidence level; Calculate high-resolution candidate images The confidence scores of all pixels are used to construct a high-resolution candidate image. The pixel-level confidence map is constructed by combining the pixel-level confidence maps of all high-resolution candidate images.

[0044] In some embodiments, for the first High-resolution candidate images Pixels Obtain from the mutual information heatmap set With the rest The normalized mutual information values ​​between high-resolution candidate images will be excluded The set of indices of the remaining high-resolution candidate images is denoted as . .

[0045] Set a preset mutual information threshold In this embodiment, take The normalized mutual information value is greater than or equal to The remaining high-resolution candidate images were incorporated into the consensus set. The normalized mutual information value is less than The remaining high-resolution candidate images are assigned to the divergence set. ,Right now:

[0046] Let the number of images contained in the consensus set be... The number of images contained in the divergence set is Both satisfy .

[0047] In some embodiments, pixels are calculated based on a consensus set and a divergence set. The confidence level includes: When the number of images contained in the divergence set is greater than When; will pixel The mean of the normalized mutual information values ​​of all high-resolution candidate images in the consensus set is used as the consensus strength; the pixel... The mean of the normalized mutual information values ​​of all high-resolution candidate images in the divergence set is used as the divergence strength; the product of the consensus strength and the divergence strength is used as the pixel value. Confidence level; When the number of images contained in the divergence set is less than or equal to When; will pixel The mean of the normalized mutual information values ​​with all other high-resolution candidate images is used as the average consensus score. ; to pixels The standard deviation of the normalized mutual information values ​​with all other high-resolution candidate images is used as the consensus volatility. ;Will As pixels The confidence level.

[0048] In some embodiments, pixels are calculated in two cases based on the number of images contained in the consensus set and the divergence set. confidence level .

[0049] Scenario 1: When At that time, calculate pixels With consensus set The mean of the normalized mutual information values ​​of all high-resolution candidate images is used as the consensus strength. Calculate pixels With the set of disagreements The mean of the normalized mutual information values ​​of all high-resolution candidate images is used as the divergence strength. The product of consensus strength and divergence strength is used as the pixel value. The confidence level, i.e.:

[0050] Scenario 2: When At that time, calculate pixels The mean of the normalized mutual information values ​​with all other high-resolution candidate images is used as the average consensus score. Calculate pixels The standard deviation of the normalized mutual information values ​​with all other high-resolution candidate images is used as the consensus volatility. .Will As pixels The confidence level, i.e.:

[0051] in, This serves as a consensus stability factor. Confidence level. The range of values ​​is A larger value indicates that the reconstruction result at that pixel is more reliable.

[0052] Traversing high-resolution candidate images For each pixel at any spatial location, the confidence score is calculated pixel by pixel, and the confidence score is composed of the confidence scores of all pixels. Pixel-level confidence map The process iterates through all high-resolution candidate images in the high-resolution candidate image set, generating a corresponding pixel-level confidence map for each image, resulting in a total of [number missing]. A set of pixel-level confidence maps is formed. .

[0053] The S400 performs weighted fusion of all high-resolution candidate images based on a set of pixel-level confidence maps to obtain an initial fused image; it constructs an energy function and uses the energy function to globally optimize the initial fused image to obtain the final super-resolution reconstructed image; and it performs Earth observation based on the final super-resolution reconstructed image.

[0054] In some embodiments, weighted fusion of all high-resolution candidate images is performed based on a set of pixel-level confidence maps to obtain an initial fused image, including: Perform Laplacian pyramid decomposition on each high-resolution candidate image in the high-resolution candidate image set to obtain the Laplacian pyramid coefficients of the high-resolution candidate image at each level. Gaussian pyramid decomposition is performed on each pixel-level confidence map in the set of pixel-level confidence maps to obtain the Gaussian pyramid coefficients of the pixel-level confidence map at each level; wherein, the number of levels of the Gaussian pyramid is the same as the number of levels of the Laplace pyramid. At each level, the weighted average of the Laplace pyramid coefficients is calculated based on the Gaussian pyramid coefficients to obtain the fused Laplace pyramid coefficient for each level. The initial fused image is obtained by reconstructing the Laplacian pyramid using the fused Laplacian pyramid coefficients of all layers.

[0055] In some embodiments, Laplacian pyramid decomposition is performed on each high-resolution candidate image in the high-resolution candidate image set. For high-resolution candidate images , will the Layer initialization It itself. Regarding the first The layer is Gaussian smoothed and interlaced downsampling is performed to obtain the first layer. Image of the Gaussian pyramid; then the first The Gaussian pyramid image is upsampled to the level of the first Gaussian pyramid image through interpolation. The same size for the layers, will the first The difference between the layer Gaussian pyramid image and the upsampled image is used as the first... Layer-by-layer Laplace pyramid coefficients. Decomposed layer by layer to obtain... In each level of the Laplace pyramid coefficient, where The total number of layers in the pyramid is taken as in this embodiment. The highest layer is the top-level Gaussian pyramid image as the residual. All high-resolution candidate images are traversed to obtain the multi-layer Laplacian pyramid coefficients for each high-resolution candidate image.

[0056] Perform Gaussian pyramid decomposition on each pixel-level confidence map in the pixel-level confidence map set. For each pixel-level confidence map... , will the Layer initialization It itself. Regarding the first The layer is Gaussian smoothed and interlaced downsampling is performed to obtain the first layer. Layered Gaussian pyramid coefficients. Decomposed layer by layer to obtain... In the Gaussian pyramid coefficients of each level, the number of levels in the Gaussian pyramid is the same as the number of levels in the Laplace pyramid. By traversing all pixel-level confidence maps, the multi-level Gaussian pyramid coefficients for each pixel-level confidence map are obtained.

[0057] At each level of the pyramid, the Laplacian pyramid coefficients of the high-resolution candidate images are used as the fusion coefficients, and the Gaussian pyramid coefficients of the corresponding pixel-level confidence maps are used as weights. A weighted average is then applied to all the fusion coefficients of the high-resolution candidate images to obtain the fused Laplacian pyramid coefficients for that level. The formula for calculating the weighted average is as follows:

[0058] in, For the first The fusion Laplace pyramid coefficient of the layers, For the first The pixel-level confidence map in the first Gaussian pyramid coefficients of the layers, For the first High-resolution candidate images in the first The Laplace pyramid coefficient of the layer, This represents element-wise multiplication. To prevent small constants with denominators of zero, take At the highest level of the pyramid, the top-level Gaussian pyramid image is weighted using the corresponding Gaussian pyramid coefficients to obtain the top-level fused image.

[0059] Top-level fused image As the starting image for reconstruction, let For the first layer( ),Will Upsampled by interpolation to the same level as the first... Layer fusion Laplace pyramid coefficient Same size, and then with Adding them together yields the reconstructed image of the current layer, i.e.:

[0060] go through The recursive reconstruction process yields the initial fused image. ,Right now: in, This represents the reconstructed image at the k-th layer.

[0061] In some embodiments, an energy function is constructed, and the initial fused image is globally optimized using the energy function to obtain the final super-resolution reconstructed image, including: An energy function is constructed, which includes a data fidelity term, a prior constraint term, and an edge preservation regularization term. The data fidelity term is used to constrain the current optimization result to remain consistent with the original low-resolution satellite image after downsampling. The prior constraint term is used to constrain the deviation of the current optimization result from the initial fused image. The edge preservation regularization term is used to keep the image edges sharp in the current optimization result. The initial fused image is iteratively optimized with the goal of minimizing the energy function. When the value of the energy function converges to a preset threshold, the current optimization result is used as the final super-resolution reconstructed image.

[0062] In some embodiments, an energy function is constructed. The energy function is composed of data fidelity terms. Prior constraints Maintain regularization terms at the edges Three components, namely in, The optimization variable represents the high-resolution image to be solved; Let the equilibrium parameters of the prior constraint terms be taken as follows: ; To maintain the balance of the regularization term at the edges, take... .

[0063] The data fidelity term is used to constrain the current optimization result to remain consistent with the original low-resolution satellite image after downsampling. To simulate sensor degradation, two operations are included: blurring and downsampling. In this embodiment, the sensor degradation simulation model is implemented using a cascaded operation of Gaussian blurring and bicubic downsampling, with the Gaussian blur kernel size being [size missing]. The standard deviation is The data fidelity term is defined as the ratio of the optimized result, after degradation model processing, to the original low-resolution satellite imagery. The sum of squares of the differences between each pixel.

[0064] Prior constraints are used to constrain the current optimization result from the initial fused image. The degree of deviation is defined as the difference between the optimization result and... The sum of squares of the differences between each pixel.

[0065] An edge-preserving regularization term is used to maintain sharp image edges in the current optimization result, employing the total variation norm. The total variation norm is defined as the ratio of the image spatial gradient... The norm is calculated by taking the square root of the sum of the squares of the horizontal and vertical gradient components at each spatial location, and summing the gradient magnitudes over all spatial locations.

[0066] The initial fused image is iteratively optimized with the goal of minimizing the energy function. As the initial value for optimization, let In the first In the next iteration ( ), calculate the gradient of the energy function with respect to the current optimization result, update the current optimization result along the gradient descent direction, and take the iteration step size as . The gradient of the energy function is the sum of the gradient of the data fidelity term, the gradient of the prior constraint term, and the gradient of the edge-preserving regularization term.

[0067] Iteration stops when the energy function converges to a preset threshold. The preset convergence threshold is set to... The convergence condition is that the relative change in the energy function value between two consecutive iterations is less than 1. When the convergence condition is met, the current optimization result is used as the final super-resolution reconstructed image. Output. Under the data fidelity constraint of the original low-resolution satellite image, the final super-resolution reconstructed image retains the texture details restored by various heterogeneous super-resolution reconstruction algorithms, maintains the sharpness of image edges, and effectively suppresses artifacts.

[0068] Reference Figure 2 This invention provides a satellite image super-resolution reconstruction device 20 for implementing a satellite image super-resolution reconstruction method. The device includes: The high-resolution candidate image set acquisition module 21 is used to acquire the original low-resolution satellite image and perform parallel high-resolution reconstruction on the original low-resolution satellite image through a deep generative super-resolution network to obtain a high-resolution candidate image set. The mutual information heatmap set acquisition module 22 is used to traverse each high-resolution candidate image in the high-resolution candidate image set, construct pairs of high-resolution candidate image pairs, calculate the normalized mutual information value of the high-resolution candidate image pairs in a local window centered on each pixel, and generate a mutual information heatmap set based on the normalized mutual information value. The pixel-level confidence map set acquisition module 23 is used to calculate the pixel confidence of each high-resolution candidate image based on the mutual information heatmap set through a majority voting mechanism to obtain the pixel-level confidence map set. The Earth observation module 24 is used to perform weighted fusion of all high-resolution candidate images based on the pixel-level confidence map set to obtain an initial fused image; construct an energy function, and perform global optimization of the initial fused image through the energy function to obtain the final super-resolution reconstructed image; and perform Earth observation based on the final super-resolution reconstructed image.

[0069] This application provides an electronic device, including a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the satellite image super-resolution reconstruction method of any of the above schemes.

[0070] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0071] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0072] This application also provides a computer-readable medium storing a computer program thereon, which, when executed by a processor, implements the satellite image super-resolution reconstruction method of any of the above-described schemes. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method as described in the embodiments of this application.

[0073] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0074] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A satellite image super-resolution reconstruction method, characterized in that, include: The original low-resolution satellite images are acquired, and parallel high-resolution reconstruction is performed on the original low-resolution satellite images through a deep generative super-resolution network to obtain a set of high-resolution candidate images. Traverse each high-resolution candidate image in the high-resolution candidate image set, construct pairs of high-resolution candidate image pairs, calculate the normalized mutual information value of the high-resolution candidate image pairs within a local window centered on each pixel, and generate a set of mutual information heatmaps based on the normalized mutual information values. Based on the mutual information heatmap set, pixel confidence is calculated for each high-resolution candidate image through a majority voting mechanism to obtain a pixel-level confidence map set; The initial fused image is obtained by weighting and fusing all high-resolution candidate images based on the set of pixel-level confidence maps. An energy function is constructed, and the initial fused image is globally optimized using the energy function to obtain the final super-resolution reconstructed image; Earth observation is then performed based on the final super-resolution reconstructed image.

2. The method of claim 1, wherein, Parallel high-resolution reconstruction of the original low-resolution satellite imagery is performed using a deep generative super-resolution network to obtain a set of high-resolution candidate images, including: High-resolution reconstruction of original low-resolution satellite images is performed by embedding a variety of heterogeneous super-resolution reconstruction algorithms into a deep generative super-resolution network. High-resolution candidate images corresponding to each heterogeneous super-resolution reconstruction algorithm are obtained. The heterogeneous super-resolution reconstruction algorithms include diffusion super-resolution reconstruction algorithm, streaming super-resolution reconstruction algorithm and generative adversarial super-resolution reconstruction algorithm. A high-resolution candidate image set is constructed by using all high-resolution candidate images. 3.The method of claim 1, wherein, Normalized mutual information values ​​for high-resolution candidate image pairs are calculated within a local window centered on each pixel. A set of mutual information heatmaps is generated based on these normalized mutual information values, including: For high-resolution candidate image pairs, a local window of a preset size is extracted, centered on each pixel in the high-resolution candidate image pair. Within a local window, extract the grayscale histogram of each high-resolution candidate image pair, calculate the normalized mutual information value based on the grayscale histogram, and assign the normalized mutual information value to the pixel. The mutual information heatmap of high-resolution candidate image pairs is constructed by using the normalized mutual information values ​​of all pixels, and the set of mutual information heatmaps is constructed by using all mutual information heatmaps. 4.The method of Claim 1, wherein, Based on a set of mutual information heatmaps, pixel confidence scores are calculated for each high-resolution candidate image using a majority voting mechanism, resulting in a set of pixel-level confidence maps, including: For the i-th high-resolution candidate image Pixels High-resolution candidate images are obtained from a set of mutual information heatmaps. The normalized mutual information value between the image and the remaining N-1 high-resolution candidate images, where N is the total number of high-resolution candidate images; The remaining high-resolution candidate images with normalized mutual information values ​​greater than or equal to a preset mutual information threshold are assigned to the consensus set, while the remaining high-resolution candidate images with normalized mutual information values ​​less than the preset mutual information threshold are assigned to the divergence set. Pixels are then calculated based on the consensus set and the divergence set. Confidence level; Calculate high-resolution candidate images The confidence scores of all pixels are used to construct a high-resolution candidate image. The pixel-level confidence map is constructed by combining the pixel-level confidence maps of all high-resolution candidate images.

5. The satellite image super-resolution reconstruction method according to claim 4, characterized in that, Pixels are calculated based on the consensus set and the divergence set. The confidence level includes: When the number of images contained in the divergence set is greater than When; will pixel The mean of the normalized mutual information values ​​of all high-resolution candidate images in the consensus set is used as the consensus strength; the pixel... The mean of the normalized mutual information values ​​of all high-resolution candidate images in the divergence set is used as the divergence strength; the product of the consensus strength and the divergence strength is used as the pixel value. Confidence level; When the number of images contained in the divergence set is less than or equal to When; will pixel The mean of the normalized mutual information values ​​with all other high-resolution candidate images is used as the average consensus score. ; to pixels The standard deviation of the normalized mutual information values ​​with all other high-resolution candidate images is used as the consensus volatility. ;Will As pixels The confidence level.

6. The satellite image super-resolution reconstruction method according to claim 1, characterized in that, The initial fused image is obtained by weighting and fusing all high-resolution candidate images based on the pixel-level confidence map set, including: Perform Laplacian pyramid decomposition on each high-resolution candidate image in the high-resolution candidate image set to obtain the Laplacian pyramid coefficients of the high-resolution candidate image at each level. Gaussian pyramid decomposition is performed on each pixel-level confidence map in the set of pixel-level confidence maps to obtain the Gaussian pyramid coefficients of the pixel-level confidence map at each level; wherein, the number of levels of the Gaussian pyramid is the same as the number of levels of the Laplace pyramid. At each level, the weighted average of the Laplace pyramid coefficients is calculated based on the Gaussian pyramid coefficients to obtain the fused Laplace pyramid coefficient for each level. The initial fused image is obtained by reconstructing the Laplacian pyramid using the fused Laplacian pyramid coefficients of all layers.

7. The satellite image super-resolution reconstruction method according to claim 1, characterized in that, An energy function is constructed, and the initial fused image is globally optimized using the energy function to obtain the final super-resolution reconstructed image, including: An energy function is constructed, which includes a data fidelity term, a prior constraint term, and an edge preservation regularization term. The data fidelity term is used to constrain the current optimization result to remain consistent with the original low-resolution satellite image after downsampling. The prior constraint term is used to constrain the deviation of the current optimization result from the initial fused image. The edge preservation regularization term is used to keep the image edges sharp in the current optimization result. The initial fused image is iteratively optimized with the goal of minimizing the energy function. When the value of the energy function converges to a preset threshold, the current optimization result is used as the final super-resolution reconstructed image.

8. A satellite image super-resolution reconstruction apparatus, used to implement the satellite image super-resolution reconstruction method as described in any one of claims 1 to 7, characterized in that, The device includes: The high-resolution candidate image set acquisition module is used to acquire the original low-resolution satellite imagery and perform parallel high-resolution reconstruction on the original low-resolution satellite imagery through a deep generative super-resolution network to obtain a high-resolution candidate image set. The mutual information heatmap set acquisition module is used to traverse each high-resolution candidate image in the high-resolution candidate image set, construct pairs of high-resolution candidate image pairs, calculate the normalized mutual information value of the high-resolution candidate image pairs within a local window centered on each pixel, and generate a mutual information heatmap set based on the normalized mutual information value. The pixel-level confidence map set acquisition module is used to calculate the pixel confidence of each high-resolution candidate image based on the mutual information heatmap set through a majority voting mechanism, and obtain the pixel-level confidence map set. The Earth observation module is used to perform weighted fusion of all high-resolution candidate images based on a set of pixel-level confidence maps to obtain an initial fused image; construct an energy function and use the energy function to globally optimize the initial fused image to obtain the final super-resolution reconstructed image; and perform Earth observation based on the final super-resolution reconstructed image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the satellite image super-resolution reconstruction method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the satellite image super-resolution reconstruction method as described in any one of claims 1 to 7.

Images