A remote sensing image pan-sharpening method based on a flow-based deep network

Abstract

A flow-based method for super-resolution, which learns the conditional distribution of high-resolution multispectral images under the condition of given low-resolution multispectral images and panchromatic images, and alleviates the ill-posed problem. It includes: first, pre-processing the data set used, including image cropping, image enlargement, normalization, data augmentation, and constructing data pairs; second, constructing a flow-based deep network for super-resolution, which is composed of several flow blocks, and a flow block is composed of a reversible conditional affine coupling block CACB; third, inputting the data pairs into the network for training, learning the conditional distribution of high-resolution multispectral images, and saving the trained network model; then, the saved network model is used for inference, a series of high-resolution multispectral images are generated under the condition of random sampling of Gaussian noise and given low-resolution multispectral images and panchromatic images; finally, the model effect is evaluated by comparing with the real high-resolution multispectral images.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and relates to generalized sharpening of remote sensing images, and particularly to a method for generalized sharpening of remote sensing images based on flow-based deep networks. Background Technology

[0002] The satellite simultaneously captures multispectral (MS) and panchromatic (PAN) images, providing supplementary information for each modality. PAN images have high resolution, while MS images contain rich spectral information. The MS sensor reduces spatial resolution while maintaining the spectral richness of the MS image. To obtain MS images with both high spectral and high spatial resolution, generalized sharpening techniques aimed at fusing MS and PAN images have received widespread attention.

[0003] Regarding the quality of the generated fusion results, current research focuses primarily on model-based and deep neural network-based methods. Model-based methods typically optimize mathematical models that provide pre-existing spectral and spatial information, and most methods follow the assumption that a PAN image (or its gradients) can be modeled as a linear combination of all bands (or their gradients) of a high-resolution multispectral (HRMS) image. However, they heavily rely on assumptions about the relationship between HRMS and PAN images. Unfortunately, previous work has not accurately established this relationship, limiting further improvements in generalized sharpening. Furthermore, model-based methods are challenging to optimize, limiting their practical applications.

[0004] In the era of deep learning, convolutional neural networks (CNNs) have become an important tool for general sharpening. CNN-based methods train the network by minimizing the distance between the fused result and the HRMS reference image. Due to the strong nonlinear fitting ability of neural networks, this method often exhibits excellent performance. However, general sharpening is inherently an ill-posed problem because a given low-resolution multispectral (LRMS) image can be degraded from an infinite number of compatible HRMS images. This presents a significant challenge to designing deep learning-based general sharpening methods. While existing CNN-based methods, taking an LRMS image L and a PAN image P as input and outputting an average predicted HRMS image H from all reasonable output images, can achieve good results, they only learn a deterministic mapping from LRMS and PAN images to the HRMS image. The problem of instability has not been adequately addressed.

[0005] The following existing techniques learn the conditional distribution of high-resolution multispectral images and alleviate the ill-posed problem in the generalization sharpening process:

[0006] Existing technology 1: SRFlow: Learning the Super-Resolution Space with NormalizingFlow.

[0007] Existing technology 2: GUIDED IMAGE GENERATION WITH CONDITIONAL INVERTIBLE NEURALNETWORKS.

[0008] In prior art 1, a super-resolution method based on normalized flow, SRFlow, was proposed. The core idea of ​​this method is to map a low-resolution image to a high-resolution space and then perform super-resolution reconstruction in this space. Its advantages include the ability to generate high-quality super-resolution images and its interpretability and controllability.

[0009] In prior art 2, a guided image generation method based on conditional invertible neural networks was proposed. The core idea of ​​this method is to generate high-quality images by controlling the generation process in a conditional space. Its advantages lie in the high controllability and realism of the generated images, and its ability to guide image generation in various application scenarios.

[0010] Existing technique 1 can learn the conditional distribution of the output under low-resolution input, solving the ill-posed problem in super-resolution tasks, and has learned to predict various realistic high-resolution images. Existing technique 2 proposes a conditional invertible neural network, solving the task of natural image generation guided by conditional input. It combines a pure generative invertible neural network model with an unconstrained feedforward network, effectively preprocessing the conditional input into useful features. As can be seen from the literature, flow-based generative models demonstrate excellent ability to explicitly learn the probability density function of data.

[0011] However, the two existing technologies mentioned above also have typical problems in general sharpening tasks: a given LRMS image can be degraded from an infinite number of compatible HRMS images, that is, each given LRMS image may correspond to multiple reasonable HRMS images, which is an ill-posed problem; secondly, how to scientifically fuse MS and PAN images, and how to inject the detailed texture information of PAN images into the network, are also the focus of research on general sharpening tasks. Summary of the Invention

[0012] To overcome the shortcomings of the prior art, the present invention aims to provide a flow-based deep network-based remote sensing image generalization sharpening method. Given a low-resolution multispectral image and a high-resolution panchromatic image, the method learns the conditional distribution of the high-resolution multispectral image to alleviate the ill-posed problem in the generalization sharpening process.

[0013] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0014] A stream-based deep network-based method for generalized sharpening of remote sensing images includes the following steps:

[0015] Step 1: Construct a training dataset. The data in the training dataset consists of paired LRMS images, PAN images, and HRMS images, i.e., LRMS-PAN-HRMS image pairs. Each image is preprocessed and has the same size.

[0016] Step 2: Construct a flow-based deep network, which is a reversible neural network PanFlowNet that follows a probabilistic flow model. It consists of several stacked flow blocks, each containing a reversible conditional affine coupling block (CACB). Each CACB comprises two complementary affine coupling layers, corresponding to the transformation process f in the probabilistic flow model. θ One step of n, where θ is the network parameter of PanFlowNet;

[0017] PanFlowNet is trained using the training dataset; wherein, the LRMS-PAN-HRMS image pairs are used as input to PanFlowNet, and PanFlowNet parameterizes the HRMS conditional distribution by mapping the LRMS-PAN-HRMS image pairs to the latent variable space, thereby using an invertible function; and learns the parameters of the invertible function by training the negative log-likelihood of the LRMS-PAN-HRMS image pairs.

[0018] Step 3: Randomly sample Gaussian noise from the Gaussian distribution and use the trained deep network to generate the corresponding HRMS image.

[0019] In one embodiment, step 1 involves constructing a paired training dataset using the Wald protocol and preprocessing the image pairs. The preprocessing includes four steps: image cropping, image upscaling, normalization, and data augmentation.

[0020] The image cropping refers to cropping the MS image and its corresponding PAN image to the required size;

[0021] The image magnification refers to restoring the cropped image to the same size as the full-color image;

[0022] The normalization process involves normalizing each MS image block and PAN image block after the image is enlarged, so that the pixel range is from 0 to 1.

[0023] The data augmentation involves randomly performing horizontal flipping, vertical flipping, and rotation operations on the normalized image.

[0024] In one embodiment, in step 2 of the PanFlowNet, the transformation process for each flow block is as follows:

[0025] Let h n For a series of reversible transformations, the latent characteristic variables are the transformation process. The goal is to generate h based on LRMS and PAN images. n+1 As shown in the following formula:

[0026]

[0027] Reverse process of flow block With h n+1 As input, generate h using LRMS images and PAN images as conditions. n As shown in the following formula:

[0028]

[0029] Based on the above formula, a conditional affine coupling module CACB is given as... Its reverse process is Where L represents the LRMS image and P represents the PAN image.

[0030] In one embodiment, the conditional affine coupling block (CACB) is constructed through the following process:

[0031] Step 1: Affine Transformation Process

[0032] The definition of affine transformation is expressed by the following equation:

[0033]

[0034]

[0035] Where ⊙ represents element-wise multiplication, h n and h n+1 Let be two adjacent latent feature variables in an invertible transformation, where That is, each flow block consists of two complementary affine coupling layers. and These are two complementary radial coupling layers in the flow block.

[0036] s1(·), s2(·), t1(·), and t2(·) are the coefficients of the affine transformation determined by the HIN module, which is implemented by the HIN module; output It is reconnected and then passed to the next CACB. The internal function is represented by a neural network and is evaluated only in the positive direction.

[0037] Step 2: Affine Inverse Transformation Process

[0038] The definition of inverse affine transformation is expressed by the following equation:

[0039]

[0040]

[0041] in It is element-based division.

[0042] In one embodiment, step 2, which parameterizes the HRMS conditional distribution using an invertible function, is as follows:

[0043] Step 1: Distribution of Learning Conditions

[0044] The LRMS-PAN-HRMS training set is provided:

[0045]

[0046] Among them, L j P represents the j-th LRMS image. j H represents the j-th PAN image. j Let represent the j-th HRMS image, and m represent the total number of samples in the training set;

[0047] Use the invertible function f θ (·) Parameterized conditional distribution, f θ (·) Map the LRMS-PAN-HRMS image pair to a latent variable z, i.e.:

[0048] z = f θ (H;L,P)

[0049] f θ (H; L, P) is invertible with respect to the first parameter H. Given an LRMS image L and a PAN image P, the HRMS image H is obtained from the latent variable z:

[0050]

[0051] Where the latent variable z is the latent variable distribution P z (·) samples;

[0052] Step 2: Define the probability density function

[0053] The latent variable z is assumed to be Gaussian distributed. Where I s Let z be the identity matrix, s be the dimension of z, and s equal to H×W×B, where H×W×B represents the dimension of the HRMS image to ensure f θ Invertible, probability density function P H|L,P(H|L,P;θ) is defined using the variable substitution formula, i.e.:

[0054]

[0055] in, It is an invertible function f θ (·) is the Jacobian matrix at H, and det(·) is the determinant function.

[0056] In one embodiment, step 2 involves learning f by minimizing the negative log-likelihood of the training pair (L,P,θ). θ The parameters of (·) are as follows:

[0057]

[0058] f θ (·) can be decomposed into N reversible layer sequences, i.e. in It is the Nth reversible layer, which receives the features h from the previous layer. n As input, h is generated. n+1 As output, i.e. h 1 =H,h N+1 =z, and use the information of LRMS image L and PAN image P as conditional inputs to encode into each reversible layer to compensate for the details and structural information of each layer;

[0059] Using the chain rule and the multiplication property of determinants, the negative log-likelihood objective is calculated as follows:

[0060]

[0061] The final objective function on the training set is:

[0062]

[0063] The optimal parameter θ is obtained by minimizing the negative log-likelihood of the training pairs. * According to P H|L,P (H|L,P;θ * Apply an inverse network to sample z. Right now

[0064] In one embodiment, in step 2, during the training process, L1 loss is first used for pre-training, and then L1 loss is used only to train the entire network.

[0065] Compared with the prior art, the beneficial effects of the present invention are:

[0066] (1) This invention accurately learns the conditional distribution of HRMS images by using the given LRMS and PAN images, and the generated HRMS images are superior to other state-of-the-art generalized sharpening methods in both visual and quantitative aspects.

[0067] (2) This invention recognizes the diversity of HRMS images. The generated HRMS images are diverse. Each randomly sampled HRMS image focuses on different parts of the data. By learning the conditional distribution of HRMS images, the ill-posed problem in the generalized sharpening process is solved. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating the overall framework of the present invention.

[0069] Figure 2 This is a flowchart of the data preprocessing process of the present invention.

[0070] Figure 3 This is a diagram of the overall network structure of the present invention.

[0071] Figure 4 This is a detailed structural diagram of CACB.

[0072] Figure 5 This is a detailed structural diagram of the HIN Block.

[0073] Figure 6 This is a network flowchart.

[0074] Figure 7 Generate flowcharts for HRMS.

[0075] Figure 8 The image visual results are from the WorldView II dataset. Detailed Implementation

[0076] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0077] This invention presents a generalized sharpening method for remote sensing images based on flow-based deep networks. The goal of this generalized sharpening method is to sharpen images from a given low-resolution multispectral (LRMS) image, guided by a high-resolution panchromatic (PAN) image P. Medium recovery of high resolution multispectral (HRMS) images Where H×W×B represents the dimension of the HRMS image, h=H / s, w=W / s, and s is the resolution factor.

[0078] For this generalized sharpening task, a flow-based deep network, PanFlowNet, is built based on a probabilistic flow model. PanFlowNet utilizes an invertible network to learn the conditional distribution of HRMS images, transforming HRMS images with unknown conditional distributions into a given Gaussian distribution. This allows for the generation of different HRMS images, including LRMS images, PAN images, and Gaussian noise samples. Furthermore, the generated HRMS images are diverse because each HRMS image focuses on different parts of the data. Ultimately, this approach alleviates the ill-posed problem to some extent, achieving good results both visually and quantitatively.

[0079] Specifically, refer to Figure 1 The main steps of this invention are as follows:

[0080] Step 1: Construct the training dataset.

[0081] In this invention, the training dataset consists of paired LRMS images, PAN images, and HRMS images, i.e., LRMS-PAN-HRMS image pairs, where each image is preprocessed to be the same size.

[0082] Step 2: Construct a flow-based deep network, PanFlowNet, a reversible neural network that follows a probabilistic flow model. It consists of several stacked flow blocks, each containing a reversible conditional affine coupling block (CACB). The CACB comprises two complementary affine coupling layers, corresponding to the transformation process f in the probabilistic flow model. θ n One step is that the network structure of the reversible layer needs to be carefully designed to ensure good conditions for reversibility and an easily tractable Jacobian matrix determinant.

[0083] PanFlowNet is trained using the training dataset. The LRMS-PAN-HRMS image pairs are used as input to PanFlowNet. PanFlowNet parameterizes the HRMS conditional distribution using an invertible function by mapping the LRMS-PAN-HRMS image pairs to the latent variable space. The parameters of the invertible function are learned by training the negative log-likelihood of the LRMS-PAN-HRMS image pairs.

[0084] Through the above steps, it is possible to learn the conditional distribution of a high-resolution multispectral image given a low-resolution multispectral image and a high-resolution panchromatic image. Furthermore, by learning the conditional distribution of HRMS images, various different and reasonable HRMS images can be generated, including LRMS and PAN images as well as Gaussian noise samples.

[0085] Step 3: Randomly sample Gaussian noise from the Gaussian distribution and use the trained deep network to generate the corresponding HRMS image.

[0086] Furthermore, in step 1 of the present invention, the training dataset is used to construct pairings using the Wald protocol, and the image pairs are preprocessed. The preprocessing mainly includes four steps: image cropping, image upscaling, normalization, and data augmentation.

[0087] like Figure 2 As shown, specifically, firstly, the MS image and its corresponding PAN image are cropped to the required size, i.e., image cropping. Secondly, to facilitate fusion, the cropped image is restored to the same size as the panchromatic image, i.e., image magnification. Then, each MS image block and PAN image block after magnification is normalized so that the pixel range is from 0 to 1, i.e., normalization. Finally, the normalized image is randomly flipped horizontally, vertically, and rotated to complete data augmentation.

[0088] In this embodiment of the invention, normalization involves min-max standardization of the data, mapping all values ​​to the range [0.0, 1.0]. The conversion formula is as follows:

[0089]

[0090] Where x and x * These represent the values ​​of the data before and after min-max normalization, respectively, with max and min representing the maximum and minimum values ​​for all pixels.

[0091] In this embodiment of the invention, a random selection strategy is used for data augmentation. By introducing a certain degree of randomness and uncertainty, the robustness and generalization ability of the algorithm can be improved, thereby avoiding problems such as overfitting to a certain extent.

[0092] refer to Figure 3 In step 2 of this invention, PanFlowNet follows the design and implementation of a probabilistic flow model. Its overall construction process generates a reversible neural network (INN) with conditional affine coupling blocks. The architecture of the reversible neural network is as follows: Figure 5 As shown, the transformation process for each flow block of PanFlowNet is described below:

[0093] Let h n For a series of reversible transformations, the latent characteristic variables are the transformation process. The goal is to generate h based on LRMS and PAN images. n+1 As shown in the following formula:

[0094]

[0095] Reverse process of flow block With h n+1As input, generate h using LRMS images and PAN images as conditions. n As shown in the following formula:

[0096]

[0097] Based on the above formula, a conditional affine coupling module CACB is given as... Its reverse process is Where L represents the LRMS image, P represents the PAN image, and θ is the network parameter of PanFlowNet.

[0098] Further, refer to Figure 4 The conditional affine coupling block CACB of the present invention is constructed through the following process:

[0099] Step 1: Affine Transformation Process

[0100] The definition of affine transformation is expressed by the following equation:

[0101]

[0102]

[0103] Where ⊙ represents element-wise multiplication, h n and h n+1 Let be two adjacent latent feature variables in an invertible transformation, where That is, each flow block consists of two complementary affine coupling layers. and These are two complementary radial coupling layers in the flow block.

[0104] s1(·), s2(·), t1(·), and t2(·) are the coefficients of the affine transformation determined by the HIN module, which is implemented by the HIN module. This affine transformation has a triangular Jacobian matrix, so its determinant is easily calculated. Furthermore, the output... It is reconnected and then passed to the next CACB. The inner function can be represented by any neural network and is evaluated only in the positive direction.

[0105] Step 2: Affine Inverse Transformation Process

[0106] The definition of inverse affine transformation is expressed by the following equation:

[0107]

[0108]

[0109] in It's based on element-wise division; the logarithm of the determinant of the Jacobian matrix of this coupled block is s. i (·) and ti (·) The sum of the image dimensions. Considering L and P as s i (·) and t i Under the conditional input settings of (·), i = 1, 2, due to the invertibility of the affine transformation and the subnetwork s i (·) and t i Since (·) is irrelevant, the affine transformation remains invertible. Therefore, by concatenating the conditional data L and P with the latent feature h, s is generated. i (·) and t i The input of (·) does not affect the invertibility of the affine transformation.

[0110] As mentioned earlier, generalized sharpening is inherently an ill-posed problem because LRMS images can be degraded by an infinite number of HRMS images. To mitigate this ill-posed problem, this invention aims to learn the conditional distribution of HRMS image H, i.e., P. H|L,P (H|L,P;θ), given an LRMS image L and a PAN image P, this is a more difficult task because the conditional distribution model can generate an infinite number of possible HRMS images, rather than just predicting a single HRMS image output through deterministic mapping.

[0111] Therefore, the process of parameterizing the HRMS conditional distribution using an invertible function in this invention can be described as follows:

[0112] Step 1: Distribution of Learning Conditions

[0113] This invention proposes a probability flow method to learn the conditional distribution P. H|L,P (H|L,P;θ), given the LRMS-PAN-HRMS training set:

[0114]

[0115] Among them, L j P represents the j-th LRMS image. j H represents the j-th PAN image. j Let represent the j-th HRMS image, and m represent the total number of samples in the training set;

[0116] Due to the conditional distribution P H|L,P (H|L,P;θ) is unknown, therefore this invention employs a probabilistic flow model, which uses an invertible function f. θ (·) is used to parameterize the conditional distribution. Under this condition, f θ (·) can map LRMS-PAN-HRMS image pairs to a latent variable z, i.e.:

[0117] z = f θ (H;L,P)

[0118] f θ (H; L, P) is invertible with respect to the first parameter H. Given the LRMS image L and the PAN image P, the HRMS image H can be obtained exactly from the latent variable z:

[0119]

[0120] Where the latent variable z is the latent variable distribution P z (·) samples;

[0121] Step 2: Define the probability density function

[0122] For simplicity, we assume that the latent variable z follows a simple Gaussian distribution. Where I s Let z be the identity matrix, s be the dimension of z, and s equal to H×W×B to ensure f θ Invertible; under this setting, the probability density function P H|L,P (H|L,P;θ) can be precisely defined using the variable substitution formula, that is:

[0123]

[0124] in, It is an invertible function f θ (·) is the Jacobian matrix at H, and det(·) is the determinant function.

[0125] Step 3: Minimize the negative log-likelihood

[0126] This invention selects f θ (·), making the determinant of its Jacobian matrix easy to compute. Specifically, f is implemented using an inverse neural network (INN). θ (·). Therefore, f can be learned by minimizing the negative log-likelihood (NLL) of the training pair (L,P,θ). θ The parameters of (·) are as follows:

[0127]

[0128] Step 4: Jacobian determinant can be processed

[0129] Furthermore, to ensure It can be processed, f θ (·) can be decomposed into N reversible layer sequences, i.e. in It is the Nth reversible layer, which receives the features h from the previous layer. n As input, h is generated. n+1 As output, i.e. From the above formula, we can see that h 1 =H,hN+1 =z, and use the information of LRMS image L and PAN image P as conditional inputs to encode into each reversible layer, which can compensate for the details and structural information of each layer;

[0130] Using the chain rule and the multiplication property of determinants, the negative log-likelihood objective is calculated as follows:

[0131]

[0132] The final objective function on the training set is:

[0133]

[0134] To ensure that in Jacobian The invertibility and fast computation of the logarithmic determinant of each layer require careful design of the network architecture for each layer. The optimal parameters θ are obtained by minimizing the negative log-likelihood of the training pairs. * Therefore, we can directly use P H|L,P (H|L,P;θ * Apply an inverse network to sample z. Right now

[0135]

[0136] During the training process of this invention, the PanFlowNet network uses the log-likelihood of the conditional distribution as the loss function, i.e., as mentioned above. To improve training efficiency, we first pre-train the network using L1 loss, and then train the entire network using only L1 loss.

[0137] The complete process of this invention can be described as follows: Figure 6 As shown, the final HRMS is generated as follows: Figure 7 As shown.

[0138] In step 3, to evaluate the model's performance, the method of this invention is compared with various generalized sharpening methods. In this invention, once PanFlowNet is trained, different HRMS images can be sampled from the distribution of HRMS. Specifically, HRMS are generated using different noise samples. Four commonly used image quality metrics—Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Relative Global Error (ERGAS), and Spectral Angle Mapper (SAM)—are used to evaluate its performance. The first three metrics measure spatial distortion, and the fourth measures spectral distortion. The specific formulas are as follows:

[0139] Peak Signal-to-Noise Ratio (PSNR) describes and compares the difference between the original image and the compressed or processed image, measuring image quality. The specific calculation formula is as follows:

[0140]

[0141] Among them, MAX I This represents the maximum possible value of a pixel, and MSE represents the mean square error, calculated using the following formula:

[0142]

[0143] Where M and N represent the number of rows and columns of the image, respectively, I(i,j) represents the pixel value in the original image, and K(i,j) represents the pixel value in the processed image.

[0144] Structural Similarity (SSIM) is a metric used to evaluate the similarity between two images. It considers not only differences in pixel values ​​but also the similarity of image structure and texture. Its calculation formula is as follows:

[0145]

[0146] Where, μ x and μ y Let σ represent the pixel mean of the two images respectively. x and σ y Let σ represent the pixel standard deviation of the two images respectively. xy This represents the covariance of corresponding pixels in two images. C1 and C2 are constants used to avoid the case where the denominator is zero.

[0147] Relative Global Dimensionless Error (ERGAS) is an indicator used to evaluate the accuracy of remote sensing image classification. It takes into account the effects of spatial resolution and spectral resolution. The specific calculation formula is as follows:

[0148]

[0149] Where n is a pixel, x i and y i These represent the pixel values ​​of the original image and the reconstructed image, respectively. This represents the average value of the original image.

[0150] The Spectral Angle Mapping (SAM) method calculates the similarity between two arrays. The result can be viewed as the cosine angle between the two arrays. The specific calculation formula is as follows:

[0151]

[0152] Where, x i and y i These represent the pixel values ​​of the original image and the reconstructed image, respectively.

[0153] In the following embodiments of the present invention, the method of the present invention is described and the effects of the present invention are verified using image datasets WorldView II, WorldView III, and GaoFen2 from satellites as examples.

[0154] In this embodiment, the following is executed: Figure 2 During the preprocessing step, the MS image is cropped to 32×32 pixels, and the corresponding PAN image is cropped to 128×128 pixels.

[0155] This embodiment assigns values ​​to some hyperparameters in PanFlowNet. PanFlowNet uses a stack of 16 CACB modules, with each flow block consisting of a reversible block composed of two complementary CACBs. Since paired training samples are unavailable, a paired training dataset is constructed using the Wald protocol. To improve training efficiency, 1000 epochs of pre-training are performed using L1 loss, followed by 100 epochs of training on the entire network using only L1 loss. The optimization algorithm used during network training is crucial for improving the training speed and convergence of deep learning networks; therefore, this invention uses the Adam optimizer (β1 = 0.9, β2 = 0.999) to update network parameters for 1000 epochs during training, with an initial learning rate set to 5e-5, decaying by 0.5 every 200 epochs. During the inference phase, Gaussian noise is randomly selected for inference. All experiments were performed on an NVIDIA GeForce GTX3080Ti GPU.

[0156] In this embodiment, randomly sampled Gaussian noise and PAN-MS images are input to the model to generate a series of HRMS images. Evaluation metrics for the quality of the generated HRMS images are calculated and compared with other existing methods; the results are shown in Table 1. Visual results from the WorldView II dataset are also presented. Figure 8 .from Figure 8 As can be seen from the first two lines, the visual performance of PanFlowNet in this invention is significantly superior to other competing methods. To make the visual advantage even clearer, the last line shows the error mapping and average error between the output generalized sharpening result and the ground truth. Compared with other competing methods, PanFlowNet in this invention exhibits minimal spatial and spectral distortion.

[0157] Table 1

[0158]

[0159] In summary, unlike traditional flow models for image inverse problems (such as SRFlow), this invention embeds guided PAN image information. This invention uses a generative flow model to learn the explicit distribution of the output image for the generalized sharpening task, and outperforms existing state-of-the-art methods both visually and quantitatively.

[0160] Those skilled in the art will readily understand that the above description is merely an embodiment of the method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for remote sensing image pan-sharpening based on a flow-based deep network, characterized in that, The method comprises the following steps: Step 1: constructing a training data set, the data in the training data set being paired LRMS images, PAN images and HRMS images, i.e. LRMS-PAN-HRMS image pairs, wherein each image is pre-processed to have the same size; Step 2, constructing a flow-based deep network, which is a reversible neural network PanFlowNet following a probabilistic flow model, composed of several flow blocks stacked together, each flow block containing a reversible conditional affine coupling block CACB, which is composed of two complementary affine coupling layers corresponding to a step of the transformation process in the probabilistic flow model θ being the network parameters of the PanFlowNet; training PanFlowNet using the training data set; wherein the LRMS-PAN-HRMS image pairs are used as the input of PanFlowNet, PanFlowNet maps the LRMS-PAN-HRMS image pairs to a latent variable space, thereby parameterizing the HRMS conditional distribution using an invertible function; and the parameters of the invertible function are learned by training the negative log-likelihood of the LRMS-PAN-HRMS image pairs; Step 3: randomly sampling Gaussian noise from a Gaussian distribution, and generating a corresponding HRMS image using the trained deep network; wherein the conditional affine coupling block CACB is constructed by the following process: Step 1: affine transformation process The definition of affine transformation is represented by the following formula: where is the element-wise multiplication, h n and h n+1 are two adjacent latent feature variables in the invertible transform, where i.e. each flow block consists of two complementary affine coupling layers, and are the two complementary radiative coupling layers in the flow block, respectively. s1(·), s2(·), t1(·), and t2(·) are coefficients of the affine transformation determined by the HIN module, implemented by the HIN module; output are reconnected and then passed to the next CACB, the internal functions are represented with neural networks and are evaluated only in the forward direction; Step 2: affine inverse transformation process The definition of affine inverse transformation is represented by the following formula: wherein is an element-based division.

2. The method of claim 1, wherein the flow-based deep network is a U-Net. 2 In the step 1, a paired training data set is constructed using the Wald protocol, and the image pairs are pre-processed, the pre-processing comprising four steps of image cropping, image enlargement, normalization and data augmentation; the image cropping is to crop the MS images and their corresponding PAN images to the required size; the image enlargement is to restore the cropped images to the same size as the panchromatic image; the normalization is to normalize each MS image block and PAN image block after image enlargement, so that the pixel range is from 0 to 1; the data augmentation is to randomly perform horizontal flipping, vertical flipping and rotation operations on the normalized images.

3. The method of claim 1, wherein the flow-based deep network is a U-Net. 3 In the step 2, in the PanFlowNet, for each flow block, the conversion process is as follows: Let h n be a latent feature variable in a series of invertible transformations, the transformation process aims to generate h n+1 as follows: Inverse process of stream block h n+1 h n As shown in the following formula: According to the above formula, a conditional affine coupling block CACB is given as The inverse process is where L denotes the LRMS image and P denotes the PAN image.

4. The method of claim 1, wherein the flow-based deep network is a U-Net. 3 In the step 2, the HRMS conditional distribution is parameterized using an invertible function, and the process is as follows: Step 1: learning the conditional distribution give the LRMS-PAN-HRMS training set: where L j represents the jth LRMS image, P j represents the jth PAN image, H j represents the jth HRMS image, and m represents the total number of samples in the training set; Using a reversible function f θ (·) parameterize the conditional distribution, f θ (·) map the LRMS-PAN-HRMS image pairs to a latent variable z, i.e.: z = f θ (H; L, P)f θ (H; L, P) is invertible for the first parameter H, given LRMS image L and PAN image P, to get HRMS image H from latent variable z: where the latent variable z is a latent variable distribution P z a sample of (·) Step 2: define the probability density function The latent variable z is assumed to be Gaussian distributed where I s is the identity matrix and s is the dimension of z, s equals HxWxB, HxWxB represents the dimension of the HRMS image, to ensure that f θ is invertible, the probability density function P H|L,P (H|L, P; θ) is defined by the substitution formula, i.e.: wherein is a reversible function f θ (·) the Jacobian matrix at H, det(·) is the determinant function.

5. The method for generalized sharpening of remote sensing images based on flow-driven deep networks according to claim 4, characterized in that, The step 2, f is learned by minimizing the negative log-likelihood of the training pairs (L, P, θ) θ parameters of f as follows f θ (·) decomposed into N reversible layer sequences, i.e. where is the Nth reversible layer, which receives the features h n as input, generates h n+1 as output, i.e. h 1 = H, h N+1 = z, and encodes the information of the LRMS image L and the PAN image P as conditional input into each reversible layer, compensating for the details and structural information of each layer; using the chain rule and the multiplicative property of determinants, the negative log-likelihood target is calculated as follows: the final objective function on the training set is: The optimal parameters θ are obtained by minimizing the negative log-likelihood of the training pairs * , according to P H|L,P (H|L, P; θ * ), apply the inverse network to the sample z That is 6. The method of claim 5, wherein the flow-based deep network is a U-Net. in the training process of the step 2, first, pre-training is performed using the L1 loss, and then the entire network is trained only using the L1 loss.

Images