Panchromatic sharpening method based on multi-modal texture correction and adaptive edge detail fusion

Abstract

The invention discloses a panchromatic sharpening method based on multi-modal texture correction and adaptive edge detail fusion, which comprises the following steps of: fusing an up-sampled low-resolution multispectral image and a panchromatic image to obtain a fused image, respectively extracting intensity components of the low-resolution multispectral image and the fused image, and respectively extracting intensity components of the low-resolution multispectral image and the fused image; inputting the low-resolution multispectral image, the intensity component of the fused image and the panchromatic image into a multi-modal texture correction model, and performing optimization solution on the multi-modal texture correction model through an optimization method to obtain a texture correction image; performing detail extraction and edge protection on the texture correction image to obtain first image details; performing detail extraction and edge protection on the up-sampled low-resolution multispectral image to obtain second image details; and performing adaptive fusion on the first image details and the second image details to obtain detail information, and adding the detail information to the up-sampled low-resolution multispectral image to obtain a final high-resolution multispectral image.

Description

Technical Field

[0001] The invention belongs to the technical field of image fusion, and in particular relates to a full color sharpening method based on multi-modal texture correction and adaptive edge detail fusion. Background Art

[0002] Due to the limitation of hardware conditions of satellite imaging sensors, it is impossible to obtain multispectral images with high spatial resolution and high spectral resolution at the same time. However, spectral sensors can be used to obtain multispectral (MS) images with rich spectral information but low spatial resolution, and spatial sensors can be used to obtain panchromatic (PAN) images with high spatial resolution but poor spectral information. Therefore, panchromatic sharpening technology is used to improve the spatial resolution of low-resolution multispectral (LRMS) images. By fusing LRMS and PAN images and taking advantage of their respective advantages, high-resolution multispectral (HRMS) images are finally obtained.

[0003] Pan-sharpening is the process of fusing multispectral (MS) images and panchromatic (PAN) images to obtain high-resolution multispectral (HRMS) images. However, due to the low correlation and similarity between MS and PAN images, as well as inaccurate spatial information injection, HRMS images have severe spectral and spatial distortions.

[0004] With the rapid development of panchromatic sharpening technology, it can be divided into four categories: methods based on component substitution (CS), methods based on multiresolution analysis (MRA), methods based on variational optimization (VO), and methods based on deep learning (DL). Methods based on CS can usually better preserve spatial details, obtain higher spatial quality, are easy to implement, and have high computational efficiency, but are prone to serious spectral distortion. Methods based on MRA can better preserve spectral information, but the decomposition of spatial structure can easily cause spatial distortion. Methods based on VO can consider the problem of spectral and spatial distortion in images, impose spectral prior constraints and spatial prior constraints between MS, PAN and ideal HRMS images, and correct regularized prior constraints, build a reasonable degradation model, and solve the model through optimization algorithms. Methods based on VO usually better preserve spatial and spectral information than methods based on CS and MRA, and obtain better fusion results. However, once unreasonable model assumptions are made, unpredictable deviations usually occur. Therefore, this type of method needs to establish a more accurate mathematical model, and the efficiency needs to be further improved. DL-based methods can generally achieve good fusion effects, but they require a large number of images to train the network, which also consumes a lot of computing resources. In addition, the test images are highly correlated with the training data, and the parameters of the network after training are also fixed, which usually cannot adapt to other new data sets of different sensors. The accuracy of DL-based methods cannot be further improved.

[0005] At present, the above-mentioned panchromatic sharpening methods all have the problem of low correlation and similarity between MS and PAN images, resulting in inaccurate information such as extracted spatial details, or even only extracting spatial details from PAN images. It is difficult to balance spectral and spatial information during the fusion process, resulting in spatial and spectral distortion of the fused image, resulting in insufficient fusion effect of the final high-resolution multispectral image. Even if deep learning-based methods can be used to balance spectral and spatial information, for example, supervised training networks will only be applicable to this data set during testing, and frequent training on different data sets will lead to a sharp increase in training time and other costs. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a full color sharpening method based on multimodal texture correction and adaptive edge detail fusion to solve the problems existing in the above prior art.

[0007] To achieve the above object, the present invention provides a panchromatic sharpening method based on multimodal texture correction and adaptive edge detail fusion, comprising:

[0008] Acquire a low-resolution multispectral image and a panchromatic image, fuse the upsampled low-resolution multispectral image with the panchromatic image to obtain a fused image, extract intensity components of the low-resolution multispectral image and the fused image, respectively, input the intensity components of the low-resolution multispectral image and the fused image and the panchromatic image into a multimodal texture correction model, and optimize and solve the multimodal texture correction model by an optimization method to obtain a texture-corrected image, wherein the multimodal texture correction model is constructed based on a variational optimization model;

[0009] Detail extraction and edge protection are performed on the texture-corrected image to obtain first image details; detail extraction and edge protection are performed on the upsampled low-resolution multispectral image to obtain second image details; the first image details and the second image details are adaptively fused to obtain detail information, and the detail information is added to the upsampled low-resolution multispectral image to obtain a final high-resolution multispectral image.

[0010] Optionally, a linear weighted sum is performed on each band image of the low-resolution multispectral image and a linear weighted sum is performed on each band image of the fused image to extract intensity components of the low-resolution multispectral image and the fused image.

[0011] Optionally, the upsampled low-resolution multispectral image is fused with the panchromatic image through a target adaptive convolutional neural network based panchromatic sharpening (A-PNN) model to obtain a fused image.

[0012] Optionally, the multimodal texture correction model is:

[0013]

[0014] Among them, T C is the texture correction image, D represents the downsampling matrix, H represents the degradation filter, I0 represents the intensity component of the low-resolution multispectral image, α, β, γ, δ, θ represent the penalty parameters corresponding to different items, is the Laplace operator, P represents the full-color image, I net represents the intensity component of the fused image, ||·|| F represents the Frobenius norm, and ||·||1 represents the 1-norm.

[0015] Optionally, the degradation filter H is obtained by an adaptive degradation filter algorithm, wherein the degradation filter H adopts a Gaussian filter H A , the adaptive degradation filter algorithm is:

[0016]

[0017] Among them, DH A T C =DF -1 (H A (u,v)F(T C ));

[0018] Gaussian filter H A The frequency domain expression H A (u,v) is:

[0019]

[0020] Among them, D C (u, v) represents the distance from point (u, v) to the center of the frequency domain, σ represents the standard deviation, and σ is optimally obtained based on the correlation and similarity indicators. The optimal value of σ is σ best :

[0021]

[0022] Among them, ρ(DH A T C ,I0) is DH A T C CC index between I0 and S(DH A T C ,I0) is DH A T C The SSIM indicator between I0 and I1.

[0023] Optionally, the multimodal texture correction model is optimized and solved by an ADMM model.

[0024] Optionally, the process of extracting details from the texture-corrected image includes:

[0025]

[0026] Among them, D TC Image details for texture-corrected images, T CL A low-resolution version of the texture-corrected image,

[0027]

[0028] Among them, χ1 represents the weight coefficient, I UP represents the intensity component of the upsampled low-resolution multispectral image, T CD Represents the image after the texture correction image is processed by the Gaussian filter;

[0029]

[0030] Among them, x3 represents the normalized weight, x1 represents I UP The influence coefficient of x2 represents T CD The influence coefficient of x1 is T C and I UP The mean of the correlation and similarity, x2 value is T C and T CD The mean of the correlation and similarity.

[0031] Optionally, the process of adaptively fusing the first image details and the second image details includes:

[0032] The second image details are enhanced to the same level as the first image details according to the scaling factor ξ:

[0033]

[0034] Wherein, F2 represents the second image detail, F3 represents the enhanced second image detail, and the superscript or subscript i represents the band number corresponding to the image;

[0035] The enhanced second image details are fused with the first image details to obtain detail information F:

[0036]

[0037] Among them, χ2 is the weight coefficient, Among them, x1 represents I UP The influence coefficient of x1 is T C and IUP The mean of the correlation and similarity.

[0038] Optionally, the process of adding the detail information to the upsampled low-resolution multispectral image includes:

[0039]

[0040] Where g represents the scaling factor of the injected details, M UP is the upsampled low-resolution multispectral image, B represents the total number of bands, i represents the band number, the superscript or subscript i represents the band number corresponding to the image, F represents the detail information, and M HR High-resolution multispectral images.

[0041] Optionally, the scaling factor g for injected detail is:

[0042]

[0043] Among them, cov(.) is the covariance function, σ 2 is the variance function, T C represents the texture correction image, M UP is the upsampled low-resolution multispectral image, and the superscript or subscript i represents the band number corresponding to the image.

[0044] Compared with the prior art, the present invention has the following advantages and technical effects:

[0045] (1) In order to enhance the correlation and similarity between source images, a multimodal texture correction model is proposed. The intensity component of the LRMS image, the PAN image and the intensity component of the image after A-PNN fusion are taken as the input of the model, and the output is the texture corrected image. The model imposes intensity correction constraints between the texture corrected image, the intensity component of the LRMS image and the PAN image, and the A-PNN-based deep plug-and-play correction prior between the intensity component of the texture corrected image and the image after A-PNN fusion.

[0046] (2) Since the degradation filter is difficult to determine in the intensity correction constraint, in order to ensure the accuracy of the prior establishment of each constraint, an adaptive degradation filter algorithm is proposed. This algorithm can adaptively determine the degradation filter in the model, thereby enhancing the correlation and similarity between the texture corrected image and the source image in the multimodal texture correction model.

[0047] (3) In order to achieve the accuracy of spatial information injection, an adaptive edge detail fusion model is proposed to adaptively extract the detail information of the texture corrected image and apply edge protection. Similarly, the detail information of the upsampled multispectral image is extracted and edge protection is applied. The spatial information of the upsampled multispectral image is improved to the same level as the texture corrected image. Finally, the spatial information of the texture corrected image and the upsampled multispectral image are adaptively fused to obtain more accurate spatial information. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0049] Figure 1 is a flowchart of a method according to an embodiment of the present invention;

[0050] Figure 2 It is a schematic diagram of iterative convergence results of the WorldView-3 dataset according to an embodiment of the present invention;

[0051] Figure 3 This is a subjective evaluation fusion result diagram of downscaled images in the QuickBird dataset according to an embodiment of the present invention;

[0052] Figure 4 This is a subjective evaluation fusion result diagram of downscaled images in the WorldView-2 dataset of an embodiment of the present invention;

[0053] Figure 5 This is a diagram of the subjective evaluation fusion results of the downscaled images in the WorldView-3 dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0055] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] In order to solve the problems pointed out in the above technical background, the present invention proposes a panchromatic sharpening method based on multimodal texture correction and adaptive edge detail fusion. In order to obtain a texture-corrected image T that is highly correlated and similar to a multispectral (MS) image C, a panchromatic sharpening (A-PNN) fusion method based on target adaptive convolutional neural network is introduced. By constructing a multimodal texture correction model, intensity, gradient and A-PNN-based depth plug-and-play correction constraints are established between the texture correction image and the source image, and an adaptive degradation filter algorithm is proposed to ensure the accuracy of these constraints. Since the obtained texture correction image can replace the panchromatic (PAN) image, and the MS image also contains some spatial information, an adaptive edge detail fusion algorithm is proposed to adaptively extract the detail information of the texture correction image and the MS image respectively, and apply edge protection. Since the spatial information of the MS image is less, its spatial information is enhanced proportionally and then adaptively fused. The fused spatial information is injected into the upsampled multispectral (UPMS) image to obtain the final HRMS image. A large number of experimental results show that compared with other methods, the algorithm proposed in the present invention has achieved better results in both subjective visual effects and objective evaluation indicators, and maintains a high operating efficiency.

[0057] Related work and related technical basis involved in this invention:

[0058] Injection model:

[0059] The injection model is commonly used in the panchromatic sharpening method. It generates HRMS images by injecting the spatial detail information with high spatial resolution in the panchromatic (PAN) image into the original UPMS image with high spectral resolution, so as to solve the problem of missing a large amount of spatial information in the LRMS image. It is assumed that the size of the LRMS image is L×W×B (i.e., length×width×number of bands), and the size of the PAN image is L′×W′, where L′=L / r, W′=W / r, and r represents the compression ratio. Then the size of the upsampled multispectral (UPMS) and high-resolution multispectral (HRMS) images is L′×W′×B. The specific formula of the injection model can be uniformly expressed as follows.

[0060] M HR =M UP +GS D (1)

[0061] Among them, M HR is the HRMS image, M UP is the UPMS image, G is the injection gain, S D is the injected spatial detail information. Extract S D The methods can be uniformly divided into CS-based methods and MRA-based methods. For CS-based methods, S D It can be extracted using the following formula.

[0062] S D =P I-I UP (2)

[0063] Among them, P I Represents the intensity component of PAN and UPMS images (I UP ) images for histogram matching. Histogram matching ensures that the intensity and contrast of PAN and LRMS images are within the same grayscale value range, ensuring the accuracy of spatial information extraction. I The formula is as follows.

[0064]

[0065] Where P represents the original PAN image, μ P and μ I Represents P and I respectively UP The mean value of the image, σ P and σ I Represents P and I respectively UP Variance of the image. UP It is M UP It is obtained by linear weighting of each band, and the formula is as follows.

[0066]

[0067] Where ω represents the linear weighting coefficient, the superscript or subscript i represents the i-th band of the image, and B represents the total number of bands. For MRA-based methods, S D It can be extracted using the following formula.

[0068]

[0069] Among them, P D represents the degraded PAN image, which can be processed by applying a low-pass filter H to the PAN image P LP Get, H LP Has a blur effect on P.

[0070] However, problems such as inaccurate injected spatial detail information still exist. Since the missing spatial detail information in the LRMS image is generally inferred from the PAN image, the inaccurate inference and the inaccurate correspondence of spectral information that may occur during the fusion process make it impossible to maintain relatively accurate spectral fidelity and spatial fidelity at the same time, which in turn leads to spectral and spatial distortion in the fused image.

[0071] Variational Optimization Model:

[0072] Variational optimization methods have become increasingly popular in recent years. They can ensure that the spectral and spatial information of images is as accurate as possible by establishing a mathematical model. The established mathematical model can be regarded as a degradation model. The ideal HRMS image after fusion of the model is restored by the LRMS and PAN images, that is, the ideal HRMS image is degraded into an inverse process of the source image. Therefore, the variational optimization method can retain the spatial and spectral information of the LRMS and PAN images through various optimization algorithms, and finally restore the desired ideal HRMS image. In summary, the variational optimization method generally establishes an energy function E(M) between LRMS, PAN and the ideal HRMS image. HR ), = This energy function can be divided into three terms, the first term is the spectral fidelity term f spectral (M0,M HR ), + the second term is the spatial fidelity term +f spatial (P,M HR ), + the third term is the regular prior term +f prior (M HR ). The specific formula is as follows.

[0073] E(M HR ) = f spectral (M0,M HR )+f spatial (P,M HR )+f prior (M HR ) (6)

[0074] Where M0 is the LRMS image. MR After blurring and downsampling, we can get M0. HR The PAN image P can also be obtained by linear weighted combination. Therefore, the energy function in equation (6) can be simplified to the following commonly used form.

[0075] E(M HR )=λ1||(DH LP M HR -M0)||+||P-CM HR ||+λ2f prior (M HR ) (7)

[0076] Among them, λ1, λ2 are penalty parameters, D represents the downsampling matrix, and C represents the linear weighted combination matrix. By optimizing and solving the above formula, we can finally get M HR .

[0077] Although the variational optimization method can retain relatively accurate spectral and spatial information at the same time, it depends on the accuracy of the mathematical model. An unreasonable variational optimization model will ignore the correlation and similarity between MS and PAN images, and the spectral and spatial information obtained by the two may not match, which will lead to spectral and spatial distortion in the final HRMS image. In addition, the efficiency of most variational optimization models is also relatively low.

[0078] The specific related method processes involved in the present invention are described as follows:

[0079] In order to solve the problems of poor correlation and similarity between LRMS, PAN and HRMS images, and inaccurate spatial information of injected UPMS images, a panchromatic sharpening method with multimodal texture correction and adaptive edge detail fusion is proposed. This method can improve the spectral and spatial distortion of HRMS images.

[0080] The input of the multimodal texture correction model is the intensity component I0 of the LRMS image, the intensity component of the PAN image and the image after A-PNN fusion, and the output is the texture corrected image T C By establishing an intensity correction prior, I0 and T are corrected C Intensity constraints between images. By establishing a gradient correction prior, I0, PAN and T are corrected C Gradient constraints between images. By establishing a deep plug-and-play correction prior based on A-PNN, the correction I net With T C The intensity gradient constraint between images. These three correction priors form the basis of the multimodal texture correction model. In addition, an adaptive degradation filter algorithm is proposed, which can be used to obtain an accurate adaptive degradation filter H in the intensity correction prior. A To degenerate T C , making the degenerate T C The correlation and similarity between the image and I0 are the highest. Finally, the multimodal texture correction model is optimized using ADMM to obtain the texture correction image T C Since the texture correction image T C The high correlation and similarity with the source image, while keeping the spectral information of the LRMS image unchanged, inherits the gradient information from the PAN image, and the intensity component I of the image after A-PNN fusion net It has more image features and can further maintain the stability of texture information. Therefore, the texture corrected image T C It can be used to replace the PAN image to facilitate subsequent fusion operations.

[0081] After obtaining the texture correction image T CAfterwards, the texture correction image T is fused by the adaptive edge detail fusion model. C and the multispectral MS image is correlated and fused to generate the final HRMS image;

[0082] In the adaptive edge detail fusion model, since the spatial detail information is not only present in the texture correction image T that replaces the PAN image, C There is also some spatial detail information in the multispectral MS image. Therefore, the adaptive texture correction image T C The detail information of the upsampled multispectral (UPMS) image is extracted and edge protection is applied using a Gaussian filter matched with the modulation transfer function (MTF). The detail information of the upsampled multispectral (UPMS) image with edge protection is enhanced to the same level as the texture corrected image T C At the same level, and with the texture correction image T with edge protection applied C The detail information is adaptively fused to obtain spatial information with high correlation and similarity to the source image, and the spatial information is injected into the UPMS image in an appropriate proportion to obtain the final HRMS image. Figure 1 The specific process is shown in the subsequent content.

[0083] Specifically, the multimodal texture correction model mainly includes an intensity correction prior, a gradient correction prior and a deep plug-and-play correction prior based on A-PNN; the relevant filters in the intensity correction prior and the gradient correction prior are determined by an adaptive degradation filter algorithm, and the multimodal texture correction model is optimized and solved by an optimization model algorithm to obtain the final texture corrected image. The specific contents are shown below.

[0084] Intensity correction prior:

[0085] Based on the spectral fidelity term of the variational optimization model in the above technical foundation, the LRMS image can be obtained by blurring and downsampling the HRMS image. The specific formula is as follows:

[0086]

[0087] Where H is generally a Gaussian smoothing filter, ‖·‖ F In order to keep the intrinsic correlation and similarity between bands unchanged, the LRMS and ideal HRMS images of each band are linearly weighted summed using formula (4) to obtain the intensity component I0 and the ideal HRMS image HR , the specific formula is as follows.

[0088]

[0089] Because I HR is unknown, so assume that T C Close to I HR , and are highly correlated. Therefore, the intensity correction prior E intensity As shown below.

[0090]

[0091] Gradient correction prior:

[0092] In the intensity-corrected prior model, T C The invariance of spectral information is maintained, but spatial information should also be retained. Based on the spatial fidelity term of the variational optimization model in the above technical foundation, the gradient information of the PAN image is retained by establishing the spatial fidelity term. The specific formula is as follows.

[0093]

[0094] Among them, α is the penalty parameter. is the Laplace operator. Since T C The process of correcting the gradient information of the PAN image may cause T C In order to avoid deviations in the intensity correction of the I0 image, a spatial fidelity term needs to be established to ensure that the intensity correction does not deviate during the gradient correction process, further enhancing the T C The correlation and similarity with the I0 image, the specific formula is as follows.

[0095]

[0096] Among them, β is the penalty parameter. In summary, the gradient correction prior can be expressed as follows.

[0097]

[0098] A-PNN-based deep plug-and-play correction prior:

[0099] In order to generate more texture features, further improve T C The correlation and similarity between I0 and PAN images retain more spectral and spatial information. After fusing PAN and UPMS images through A-PNN, the HRMS image is obtained, which is denoted as MS net , through formula (4) to MS net After linear weighting, we get MS net The intensity component I netA-PNN is a well-known technology for technicians in the field. It is a full-color sharpening method based on a target adaptive convolutional neural network. Based on the spectral fidelity term of the variational optimization model in the above technical foundation, by C with I net Establish the spectral fidelity term and correct T C The specific formula is as follows.

[0100]

[0101] Among them, γ is the penalty parameter. C with I net Establish the spatial fidelity term and correct T C The gradient information of is as follows.

[0102]

[0103] Among them, δ is the penalty parameter. In summary, the deep plug-and-play correction prior based on A-PNN can be expressed as follows.

[0104]

[0105] Multimodal Texture Correction Model:

[0106] In order to ensure the sparsity of the output texture map and reduce artifacts, in addition to combining the above-mentioned intensity correction prior, gradient correction prior and A-PNN-based deep plug-and-play correction prior, the total variation regularization term (TV) is also used. To this end, the present invention proposes a multimodal texture correction model, and the specific formula is as follows.

[0107]

[0108] Among them, θ is the penalty parameter.

[0109] Adaptive degradation filter algorithm:

[0110] In the model shown in equation (17), in addition to the Gaussian filter H and the texture correction image T C All other factors have been determined, and the texture correction image T C Algorithm 2 can be used to determine H, but H is difficult to determine. Therefore, the present invention proposes an adaptive degradation filter algorithm, using a Gaussian filter as the degradation filter, defined as H A , can be determined by the following formula.

[0111]

[0112] From the above formula, we can see that when the texture correction image DH is processed by downsampling and degradation filter A TC When the difference between the intensity component I0 of the LRMS image is the smallest, that is, when the correlation and similarity between the two are the highest, the H A is the best degradation filter. Therefore, the adaptive degradation filter algorithm comprehensively considers the correlation and similarity between the two, and uses the correlation coefficient (CC) and structural similarity (SSIM) indicators to measure them respectively, and finally adaptively determines the best degradation filter. When the filter is processed in the spatial domain of the image, the convolution operation will greatly increase the computational complexity. When processing in the image frequency domain, the convolution operation is converted into the inner product operation, which will greatly reduce the computational complexity. Therefore, H A Choose to operate in the frequency domain, H A The frequency domain expression of is:

[0113]

[0114] Among them, D C (u,v) represents the distance from point (u,v) to the center of the frequency domain, and σ represents the standard deviation. A After conversion to the frequency domain, T C It is also necessary to calculate in the frequency domain, so the Fast Fourier Transform (FFT) is used to convert T C Converted into frequency domain, the inverse fast Fourier transform (IFFT) converts H A T C Then convert it back to the spatial domain to facilitate subsequent DH A T C The correlation and similarity calculation between I0 and I1 is as follows.

[0115] DH A T C =DF -1 (H A (u,v)F(T C )) (20)

[0116] Where F(.) represents the FFT operation, F -1 (.) represents the IFFT operation. In summary, determine H A The key is to determine the unknown parameter σ. Therefore, by correcting DH A T C The correlation and similarity between DH and I0 are used to find the best σ. The correlation is measured by the CC index, denoted as ρ(DH A T C ,I0). The similarity is measured by the SSIM index, denoted as S(DH A T C,I0). The two indicators are comprehensively considered using the average value rule, and different σ values ​​are used for iterative processing. The final result is the maximum value, and the σ at this time is the optimal value, recorded as σ best .

[0117] The specific formula is as follows.

[0118]

[0119] In summary, the overall process of the adaptive degradation filter algorithm is shown in Algorithm 1.

[0120] Algorithm 1:

[0121] Algorithm 1: Adaptive degradation filter algorithm

[0122] Input: Texture correction image T C , the intensity component I0 of the LRMS image

[0123] Initialization: Set σ (0) =1, step size s = 0.5, iteration step size k = 0,

[0124] By using formula (19), H A Converted to frequency domain,

[0125] Calculate (DH) by formula (20) A T C ) (0) ,

[0126] Calculate ρ (0) (DH A T C ,I0) and S (0) (DH A T C ,I0).

[0127] Cycle: σ (k+1) =σ (k) +s,k=k+1,

[0128] Optimize (DH) by formula (20) A T C ) (k+1) ,

[0129] Optimizing ρ (k+1) (DH A T C ,I0) and S (k+1) (DH A T C ,I0),

[0130] Calculated by formula (21)

[0131] Until σ is satisfied (k+1) <σ (k) , the iteration ends and the loop is exited

[0132] Output: σ best =σ (k) , adaptive degradation filter H A

[0133] Optimization model algorithm:

[0134] The optimization model algorithm uses ADMM to optimize, which is an optimization method that decomposes the original problem into multiple easy-to-handle sub-problems. In order to make the optimization process easier, different auxiliary variables are introduced That is, B=DA, then the model of formula (17) can be expressed as.

[0135]

[0136] The augmented Lagrangian function of the above formula can be expressed as.

[0137]

[0138] Among them, Λ1, Λ2, Λ3 are different Lagrange multipliers, μ1, μ2, μ3 are different penalty parameters. In order to minimize the energy function of the above formula, A is optimized by iterative optimization. (k+1) , B (k+1) , T C (k+1) , H A (k+1) , C (k+1) , Λ1 (k+1) , Λ2 (k+1) and Λ3 (k+1) , and finally get T C , k is the number of iterations, and the different parameters with superscript k+1 represent the corresponding parameters in the k+1th iteration process. The specific optimization process is as follows.

[0139] (1) Optimize A (k+1)

[0140] Fixing other variables, A (k+1) The sub-problems are as follows.

[0141]

[0142] A (k+1) The deviation is set to 0, that is Then A (k+1) Obtained by the following formula.

[0143]

[0144] Among them, U represents the identity matrix and the superscript T represents the transpose operator.

[0145] (2) Optimize B (k+1)

[0146] Fix other variables, B (k+1) The sub-problems are as follows.

[0147]

[0148] B (k+1) The deviation is set to 0, that is However, due to the existence of the Laplace operator, the computational complexity of the solution process increases. In order to improve the computational efficiency, FFT and IFFT are used to perform fast calculations in the frequency domain and then convert them back to the spatial domain. (k+1) After optimization, B (k+1) It can be obtained by the following formula.

[0149]

[0150] (3) Optimize T C (k+1)

[0151] Fix other variables, T C (k+1) The sub-problems are as follows.

[0152]

[0153] T C (k+1) The deviation is set to 0, that is Due to the existence of the Laplace operator, FFT and IFFT are also used for solving. Therefore, when A (k+1) After optimization, T C (k+1) It can be obtained by the following formula.

[0154]

[0155] (4) Optimize C (k+1)

[0156] Fix other variables, C (k+1) The sub-problems are as follows.

[0157]

[0158] The soft thresholding formula is further simplified to obtain the following formula.

[0159]

[0160] Among them, sgn(.) is the signal function and max(.) is the maximum value function.

[0161] (5) Optimizing the Lagrange multiplier Λ1 (k+1) , Λ2 (k+1) and Λ3 (k+1)

[0162] Fix other variables and obtain Λ1 by gradient ascent method (k+1) , Λ2 (k+1) and Λ3 (k+1) sub-problem of .

[0163]

[0164] in, is the step size required for gradient ascent, and the formula is as follows.

[0165]

[0166] Among them, τ is the penalty parameter. Generally, τ>1 can accelerate the convergence speed. In summary, the overall process of multimodal texture correction model optimization is shown in Algorithm 2. Among them, in the iterative process, H A (t+1) Optimized by Algorithm 1, when T C When the relative change (RelCha) of two consecutive iterations is less than the tolerance deviation ε, the iteration process is exited and T is finally obtained. C Image. The relative change judgment formula is as follows.

[0167]

[0168] As the iteration proceeds, the relative change value RelCha gradually decreases, so it is necessary to determine the parameter ε to make it slightly larger than RelCha in order to balance the efficiency and accuracy of the model. Figure 2 The iterative convergence results of the test images in the WorldView-3 dataset are shown. When the number of iterations reaches about 15, RelCha tends to converge and is close to 1×10 -4 , that is, ε can be assigned a value of 1×10 -4 .

[0169] Algorithm 2:

[0170] Algorithm 2: Optimization algorithm for multimodal texture correction model

[0171] Input: full-color image P, intensity component I0 of LRMS image

[0172] Initialization: k = 0, Through the initialization of Algorithm 1, we get τ=1.01

[0173] While RelCha>εdo

[0174] cycle:

[0175] Optimize A by formula (25) (k+1) ,

[0176] Optimize B by formula (27) (k+1) ,

[0177] Optimize by formula (29)

[0178] Optimize by Algorithm 1

[0179] Optimize C by formula (31) (k+1) ,

[0180] Optimize by formula (32)

[0181] k=k+1.

[0182] Until RelCha<=ε is satisfied, the iteration ends and the loop is exited

[0183] Output: Texture corrected image T C .

[0184] Adaptive edge detail fusion model:

[0185] Adaptive Extraction T C Image details and apply edge protection:

[0186] By using Algorithm 2, we can get T C Finally, the following formula is used to extract image details D TC .

[0187]

[0188] Among them, T CL T C A low-resolution version of the image. For more accurate C From equations (2) and (5), we can see that T CL It can be obtained by two methods. The first method is T C By applying Algorithm 1, we get the degraded image T CD , similar to the MRA-based method to extract details and better preserve spectral information. The second method is to use formula (4) to obtain I UP, similar to the CS-based method to extract details and better preserve spatial information. Therefore, considering the advantages of these two methods, adaptive extraction of D TC , and its algorithm is designed as follows.

[0189] T CL =χ1I UP +(1-χ1)T CD st0<χ1<1 (36)

[0190] Among them, χ1 represents the weight coefficient to be determined. Since the accuracy of detail extraction by the two methods is affected by the correlation and similarity between the source images, I UP The influence coefficient can be set as x1, T CD The influence coefficient can be set to x2, and its formula is as follows.

[0191]

[0192] Since x1 and x2 do not satisfy the normalization constraint of χ1, χ1 should be positively correlated with x1 and x2 and have a reasonable range, so χ1 can be obtained using the following formula.

[0193]

[0194] Substituting χ1 from the above equation into equation (36), we get T CL Then substitute into formula (35), and finally get D TC , completing the adaptive extraction of T C Image detail operations. In order to retain edge information while extracting details, the following edge detection matrix formula E is used TC Extract edges.

[0195]

[0196] in, is the gradient operator, η and is the modulation coefficient. Generally, η=1×10 -9 , Therefore, the edge protection T C The image detail information, namely the first image detail F1, is as follows.

[0197]

[0198] Extract UPMS image details and apply edge protection:

[0199] The following formula is used to extract the detail information D of the UPMS image. M .

[0200]

[0201] Among them, M UPL represents a low-resolution version of the UPMS image. UPL The MTF obtained from the MS sensor is introduced as an important indicator for extracting UPMS image details. Therefore, the MTF-matched Gaussian filter H is used. MG Degrade the UPMS image to obtain a low-resolution version of the UPMS image. The specific process is shown in the following formula.

[0202]

[0203] Substituting the above formula into formula (41), we can get the detail information of the UPMS image. At this time, we need to use the edge detection matrix formula E M Right D M Edge protection.

[0204]

[0205] Therefore, the detail information of the UPMS image with edge protection applied, that is, the second image detail F2 is as follows.

[0206]

[0207] Adaptive edge detail fusion process:

[0208] T C After the detail information of the UPMS image with edge protection is extracted, F1 and F2 can be fused. However, since the spatial resolution of the UPMS image is lower than that of the T C Therefore, F2 contains less detailed information than F1, and directly fusing them may lose detailed information. To avoid this, the information of F2 is enhanced to the same level as F1 before fusion. The specific formula is as follows.

[0209]

[0210] Where ξ is the scaling factor. ξ is solved by the linear regression model. Therefore, the spatial information F3 enhanced by ξ is expressed as:

[0211]

[0212] At this time, F1 and F3 can be adaptively fused to obtain detail information F. The specific algorithm is as follows.

[0213]

[0214] Among them, χ2 is the weight coefficient. The weight distribution of detail information is affected by the source image, i.e., T CThe influence of correlation and similarity with UPMS. Therefore, T can be set using formula (37) C with I UP The relationship x1 is ensured, while x2 is within a reasonable range, and x1 and x2 are positively correlated. The specific formula is shown below.

[0215]

[0216] Substitute χ2 from the above formula into formula (47) to obtain the final detail information F.

[0217] Final injection of spatial edge detail information:

[0218] Substitute the detail information F of formula (47) into the following injection model to obtain the final HRMS image.

[0219]

[0220] Where g represents the scaling factor of the injected details, which can be adaptively determined by the following formula.

[0221]

[0222] Among them, cov(.) is the covariance function, σ 2 is the variance function.

[0223] Experimental design and conclusions for the above technical solutions:

[0224] In order to illustrate the performance advantages and effectiveness of the proposed method, the proposed method is compared with eight methods, including GSA, NIHS, BDSD-PC, SFIM, ATWT-W3, DMPIF, CDIF and A-PNN, and a large number of experiments are carried out using three datasets, including QuickBird, WorldView-2 and WorldView-3. Each pair of images in each dataset contains one MS image and one PAN image. In the QuickBird dataset, the number of bands of the MS image is 4; while in the WorldView-2 and WorldView-3 datasets, the number of bands of the MS image is 8. PAN images in all datasets contain only one band.

[0225] According to the Wald protocol, this experiment uses the original MS image as the reference image, that is, the real ground object (GT) image. At this time, it is necessary to perform 4-fold downsampling degradation processing on the original MS and PAN images respectively. The degraded images can be used as downscaled source images. The algorithm proposed in the present invention is used to fuse the source images, and the obtained fused images are compared with the GT images. The smaller the difference, the better the effect. Therefore, this experiment crops the size of each band of the GT image to 256*256, the size of each band of the MS image to 64*64, and the size of the PAN image to 256*256.

[0226] This experiment summarizes the specific information of these three data sets, as shown in Table 1. Table 1 is the detailed information of the data sets used in this experiment.

[0227] Table 1

[0228]

[0229]

[0230] In order to evaluate and compare the image quality of different methods, a combination of subjective and objective evaluation criteria was adopted. Six commonly used objective evaluation indicators were used for objective evaluation. Among them, Q2n index (Q4 represents a 4-band data set, Q8 represents an 8-band data set) was selected to evaluate the spatial and spectral quality of the image, peak signal-to-noise ratio (PSNR) was used to measure the error degree between the image reconstruction and the reference image, universal image quality index (UIQI) was used to more comprehensively evaluate the quality difference and similarity between the image fusion and the reference image, relative average spectral error (RASE) was used to evaluate the average spectral difference before and after image fusion, comprehensive dimensionless relative global error (ERGAS) was used to indicate the degree of distortion of the image spatial and spectral information, and spectral correlation coefficient (SCC) was used to measure the ability to retain the image spectral information. Subjective evaluation is to visualize the fused MS image, extract the three bands of red (R), green (G), and blue (B) to display the true color fusion image, which can more intuitively reflect the quality difference of the image. Among the above evaluation indicators, the ideal values ​​of Q2n, UIQI and SCC are 1, RASE, ERGAS, and the ideal value of PSNR is +∞. All experiments in this section are run on a PC with an Inter Core i7-12700 CPU, a base speed of 2.10 GHz, and 32 GB of memory, and the experimental platform is MATLAB R2021b.

[0231] Comparative experiment:

[0232] QuickBird Dataset:

[0233] In the subjective evaluation of the QuickBird dataset, Figure 3 As shown in the figure, the subjective evaluation fusion results of the method proposed in the present invention and the comparative methods are shown, and the GT image is used as the reference image. In order to more clearly display the spatial and spectral information of the image, the local fusion results are enlarged. It can be seen from the local enlargement that the GSA method shows too much detail information on the top of the house. The images of the BDSD-PC and ATWT-M3 methods are relatively blurred and the brightness is relatively dark. Although the SFIM method retains the edge information in some areas relatively well, there is a problem of dark brightness. The edge information of the CDIF method is well preserved, but artifacts are generated. Although the DMPIF and A-PNN methods retain the spatial information relatively well, their edges generate redundant color information and the spectral distortion is relatively serious. The results of the method proposed in the present invention are closest to the GT image, and the spatial and spectral information are better preserved. The objective evaluation fusion results are shown in Table 2. Table 2 is the objective evaluation fusion results of the downscaled images in the QuickBird dataset. The ideal values ​​are marked in brackets, and the black bold indicates the optimal results. It can be seen that compared with the other 8 methods, the method proposed in the present invention has achieved the best results in all evaluation indicators, and the method takes a shorter time.

[0234] Table 2

[0235]

[0236] WorldView-2 Dataset:

[0237] like Figure 4 As shown in the figure, in the subjective evaluation of the fusion results of each comparison method in the WorldView-2 dataset. It can be seen from the enlarged local area that the image clarity in the GSA, BDSD-PC, and ATWT-M3 methods is poor, resulting in more serious spatial distortion and darker colors. In the NIHS method, there is a problem of too much spatial information injection in some areas. Compared with the GT image, the spatial information of the SFIM method still has a certain gap. The image spatial distortion and spectral distortion problems in the CDIF method are more serious. The image clarity of the DMPIF method is different from that of the GT image, and the image has a more serious artifact problem. The spectrum of some areas in the A-PNN method is distorted, and the spatial information is poorly retained. The method proposed in the present invention is closest to the GT image, and its visual effect is better than other comparison methods. The objective evaluation fusion results are shown in Table 3, which is the objective evaluation fusion results of the downscaled image in the WorldView-2 dataset. Obviously, compared with the other 8 methods, the method proposed in the present invention has achieved the best results in all evaluation indicators, and the time required for operation is also shorter.

[0238] Table 3

[0239]

[0240] WorldView-3 dataset

[0241] like Figure 5 As shown in the figure, in the subjective evaluation of the fusion results of each method in the WorldView-3 dataset. It can be seen from the enlarged local area that the top of the house of the GSA method is darker in color than the GT image. The image of the NIHS method produces certain artifacts, which affects the spatial information quality of the image. The images of the BDSD-PC, SFIM and ATWT-M3 methods are relatively blurred. Although the CDIF method retains spectral information well, its detail information is poorly retained, resulting in more serious spatial distortion. The color changes of the DMPIF and A-PNN methods are large, resulting in more serious spectral distortion. The method proposed in the present invention is closest to the GT image, and the subjective visual effect has achieved the best effect. The objective evaluation fusion results are shown in Table 4. Table 4 is the objective evaluation fusion results of the downscaled images in the WorldView-3 dataset. It can be seen that the method of the present invention has the best results in all evaluation indicators and the running time is also short.

[0242] Table 4

[0243]

[0244] Related conclusions:

[0245] Since MS and PAN images are obtained by different sensors, this pair of source images usually have low correlation and similarity, and direct fusion may cause serious spectral distortion and spatial distortion. Secondly, to obtain an ideal HRMS image, it is necessary to inject the spatial information of the PAN image into the UPMS image. However, inaccurate injected spatial information will lead to low spatial resolution of the HRMS image. In view of these main problems, the present invention proposes a fusion model based on multimodal texture correction and adaptive edge detail. In order to obtain a T image that is highly correlated and similar to MS and accurately inherits the spatial information of PAN, the HRMS image is highly correlated and similar to MS. C , by establishing T C The strength constraint between I0 and T C Gradient constraints between PAN, I0, and T C with I net A deep plug-and-play constraint based on A-PNN is proposed, and an adaptive degradation filter algorithm is proposed to accurately maintain the constraints of the model. Finally, a multimodal texture correction model is constructed, which uses the ADMM algorithm to solve T C , which is used to replace the function of PAN image. Since spatial detail information is not only present in T CThere are also some spatial detail information in the MS image, so an adaptive edge detail fusion model is proposed. The model extracts T C The details of the UPMS image are extracted and edge protection is applied. In order to extract the details more accurately, an adaptive extraction T C The algorithm is used to extract details, and the MTF matched Gaussian filter is used to extract the UPMS image. C The edge-protected detail information is adaptively fused with the enhanced detail information of the UPMS image. Finally, the fused spatial information is injected into the UPMS image using the injection model to obtain the final HRMS image. In the comparative experiment, the performance advantage of the algorithm of the present invention is illustrated, and the parameter analysis and ablation study prove the effectiveness of the algorithm of the present invention. The final results show that the algorithm proposed in the present invention can obtain better fusion results.

[0246] In the multimodal texture correction model, the solution efficiency is greatly improved because it is iteratively optimized in a two-dimensional image. The three correction priors set can better retain spatial and spectral information. However, this model still has shortcomings. There are unknown parameters in the correction priors that need to be determined through experiments, which may consume a lot of computing resources and time. In the adaptive edge detail fusion model, in order to obtain accurate spatial information, T is comprehensively considered. C The edge detail information of UPMS. However, the problem of the amount of injected spatial information and the ratio of spectral information of UPMS to injected spatial information still exists. Therefore, our future work will focus on adaptively determining other unknown parameters in the pan-sharpening model and exploring more appropriate injection model methods to improve the overall performance and efficiency.

[0247] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A full color sharpening method based on multimodal texture correction and adaptive edge detail fusion, characterized in that: include: Acquire a low-resolution multispectral image and a panchromatic image, fuse the upsampled low-resolution multispectral image with the panchromatic image to obtain a fused image, extract intensity components of the low-resolution multispectral image and the fused image, respectively, input the intensity components of the low-resolution multispectral image and the fused image and the panchromatic image into a multimodal texture correction model, and optimize and solve the multimodal texture correction model by an optimization method to obtain a texture-corrected image, wherein the multimodal texture correction model is constructed based on a variational optimization model; Detail extraction and edge protection are performed on the texture-corrected image to obtain first image details; detail extraction and edge protection are performed on the upsampled low-resolution multispectral image to obtain second image details; the first image details and the second image details are adaptively fused to obtain detail information, and the detail information is added to the upsampled low-resolution multispectral image to obtain a final high-resolution multispectral image.

2. The method according to claim 1, characterized in that Performing linear weighted summation on each band image of the low-resolution multispectral image and performing linear weighted summation on each band image of the fused image, and extracting intensity components of the low-resolution multispectral image and the fused image.

3. The method according to claim 1, characterized in that: The upsampled low-resolution multispectral image is fused with the panchromatic image through a panchromatic sharpening model based on a target adaptive convolutional neural network to obtain a fused image.

4. The method according to claim 1, characterized in that: The multimodal texture correction model is: Among them, T C is the texture correction image, D represents the downsampling matrix, H represents the degradation filter, I0 represents the intensity component of the low-resolution multispectral image, α, β, γ, δ, θ represent the penalty parameters corresponding to different items, is the Laplace operator, P represents the full-color image, I net represents the intensity component of the fused image, ||·|| F represents the Frobenius norm, and ||·||1 represents the 1-norm.

5. The method according to claim 4, characterized in that The degradation filter H is obtained by an adaptive degradation filter algorithm, wherein the degradation filter H adopts a Gaussian filter H A , the adaptive degradation filter algorithm is: Among them, DH A T C =DF -1 (H A (u,v)F(T C )); Gaussian filter H A The frequency domain expression H A (u,v) is: Among them, D C (u, v) represents the distance from point (u, v) to the center of the frequency domain, σ represents the standard deviation, and σ is optimally obtained based on the correlation and similarity indicators. The optimal value of σ is σ best : Among them, ρ(DH A T C ,I0) is DH A T C CC index between I0 and S(DH A T C ,I0) is DH A T C The SSIM indicator between I0 and I1.

6. The method according to claim 1, characterized in that The multimodal texture correction model is optimized and solved by the ADMM model.

7. The method according to claim 1, characterized in that The process of extracting details from the texture-corrected image includes: Among them, D TC Image details for texture-corrected images, T CL A low-resolution version of the texture-corrected image, T CL =χ1I UP +(1-χ1)T CD s.t.0<χ1<1; Among them, χ1 represents the weight coefficient, I UP represents the intensity component of the upsampled low-resolution multispectral image, T CD Represents the image after the texture correction image is processed by the Gaussian filter; Among them, x3 represents the normalized weight, x1 represents I UP The influence coefficient of x2 represents T CD The influence coefficient of x1 is T C and I UP The mean of the correlation and similarity, x2 value is T C and T CD The mean of the correlation and similarity.

8. The method according to claim 1, characterized in that The process of adaptively fusing the first image details and the second image details includes: The second image details are enhanced to the same level as the first image details according to the scaling factor ξ: Wherein, F2 represents the second image detail, F3 represents the enhanced second image detail, and the superscript or subscript i represents the band number corresponding to the image; The enhanced second image details are fused with the first image details to obtain detail information F: Among them, χ2 is the weight coefficient, Among them, x1 represents I UP The influence coefficient of x1 is T C and I UP The mean of the correlation and similarity.

9. The method according to claim 1, characterized in that: The process of adding the detail information to the upsampled low-resolution multispectral image includes: Where g represents the scaling factor of the injected details, M UP is the upsampled low-resolution multispectral image, B represents the total number of bands, i represents the band number, the superscript or subscript i represents the band number corresponding to the image, F represents the detail information, and M HR High-resolution multispectral images.

10. The method according to claim 1, characterized in that The scaling factor g for injected detail is: Among them, cov(.) is the covariance function, σ 2 is the variance function, T C represents the texture correction image, M UP is the upsampled low-resolution multispectral image, and the superscript or subscript i represents the band number corresponding to the image.

Images