A multispectral image panchromatic sharpening method and system based on variational theory

Abstract

The application belongs to the technical field of remote sensing image fusion, and discloses a multispectral image panchromatic sharpening method and system based on a variational theory; wherein the multispectral image panchromatic sharpening method comprises the following steps: obtaining a multispectral image and a panchromatic image to be fused; based on the obtained multispectral image and panchromatic image, an optimization function is solved iteratively by using an ADMM algorithm to obtain a high-resolution multispectral image after panchromatic sharpening. The technical scheme provided by the application is based on the solving of a newly proposed optimization function, and finally obtains a high-resolution multispectral image after panchromatic sharpening, which can solve the technical defect of insufficient utilization of spectral data in the prior art; in the optimization function proposed by the application, a detail fidelity term is designed in combination with a non-subsampled shearlet transform, which can make up for the defect of insufficient extraction of spatial information by a gradient operation.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image fusion technology, and specifically relates to a multispectral image panchromatic sharpening method and system based on variational theory. Background Technology

[0002] While multispectral images have lower spatial resolution, their unique advantage lies in their rich spectral information, making them highly effective in analyzing the spectral characteristics of different ground features. In contrast, panchromatic images are acquired using broadband remote sensing techniques. This method is characterized by its high spatial resolution and excellent ability to distinguish detailed features such as the shape and texture of ground objects; however, panchromatic images have relatively limited bands and lack the rich spectral information of multispectral images.

[0003] To overcome the limitations of the two types of images and make full use of their advantages, researchers adopted a pixel-level fusion technique for multispectral and panchromatic images. This technique can accurately fuse the information of the two images at the pixel level, thereby generating a multispectral image that has both rich spectral information and high spatial resolution. The fused image not only retains the spectral characteristics of the original multispectral image, but also improves its spatial resolution, making it more accurate and comprehensive in representing ground feature information.

[0004] In existing technologies, variational theory-based methods have demonstrated unique advantages in image fusion, attracting widespread attention from scholars. The core idea of ​​these methods is to construct a variational model incorporating both spatial and spectral information to balance their weights during the fusion process, thereby achieving optimal information fusion. Furthermore, by adding prior terms, this method can further enhance the fusion effect and improve the quality of the fused image. Existing variational theory-based methods typically achieve good results in processing edge information because the variational model can fully consider the local structure and edge features in the image, thus maintaining the integrity of this important information during the fusion process. However, despite their excellent edge processing performance, existing methods suffer from insufficient utilization of spectral data during spectral data clustering. Interpretatively, spectral data contains rich ground feature information, the potential of which has not been fully explored and utilized in existing methods, which to some extent limits the quality and accuracy of the fused image. In addition, existing methods also have certain limitations in extracting spatial information. Interpretationally, although some spatial information can be extracted through gradient operations, this information is often insufficient and cannot fully reflect the spatial structure and texture features in the original image. During the fusion process, this information may be lost or distorted, thus affecting the quality and visual effect of the fused image. Summary of the Invention

[0005] The purpose of this invention is to provide a multispectral image panchromatic sharpening method and system based on variational theory to solve one or more of the aforementioned technical problems. The technical solution provided by this invention, based on a newly proposed optimization function, ultimately obtains a high-resolution multispectral image after panchromatic sharpening, which can solve the technical deficiency of insufficient utilization of spectral data in existing technologies. Explained further, the optimization function proposed in this invention incorporates a detail-preserving term designed in conjunction with non-subsampled shear wave transform, which can compensate for the deficiency of insufficient spatial information extraction by gradient operations.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention provides a multispectral image panchromatic sharpening method based on variational theory, comprising the following steps:

[0008] Acquire the multispectral and panchromatic images to be fused;

[0009] Based on the acquired multispectral and panchromatic images, the ADMM algorithm is used to iteratively solve the optimization function to obtain a high-resolution multispectral image after panchromatic sharpening.

[0010] The expression for the optimization function is:

[0011]

[0012] In the formula, For spectral fidelity, For gradient fidelity, α1 and α2 represent the gradient fidelity term coefficient constant and the NSST term coefficient constant, respectively; X is the high-resolution multispectral image after panchromatic sharpening, Y is the multispectral image to be fused, and P is the panchromatic image to be fused; B represents the blurring operation, and D represents the downsampling operation. For gradient taking, ⊙ represents the Hadamard product operation; G is the gradient coefficient matrix. This represents the NSST transform, where K represents the NSST band coefficient; This represents the Frobenius norm.

[0013] A further improvement to the method of the present invention is that, in the optimization function, the calculation expression for the gradient coefficient matrix G is:

[0014]

[0015]

[0016] In the formula, M is the total number of pixel categories; p z G represents the probability that a pixel belongs to class z; z Y is the gradient coefficient matrix for pixel class z;z A multispectral image for pixel category z; D T This is an upsampling operation; This is a least squares regression.

[0017] A further improvement of the method of the present invention is that, in the optimization function, when calculating the gradient coefficient matrix G, a Gaussian mixture model clustering method is used to cluster the pixels in the image.

[0018] A further improvement of the method of the present invention is that the step of clustering pixels in the image using the Gaussian mixture model clustering method includes:

[0019] Upsample the multispectral image to the same size as the panchromatic image; connect the upsampled multispectral image and the panchromatic image along the band direction; cluster all pixels into multiple categories in multidimensional space using a Gaussian mixture model.

[0020] A further improvement to the method of this invention is that, in the optimization function, the calculation expression for the NSST band coefficient K is:

[0021]

[0022] In the formula, DCT(·) represents the discrete cosine transform; This is a least squares regression.

[0023] A further improvement of the method of the present invention is that the step of iteratively solving the optimization function using the ADMM algorithm includes:

[0024] By introducing an auxiliary variable U = XB and using the separation of the inverse transform operation within the FISTA framework, the optimized function is rewritten as follows:

[0025]

[0026]

[0027] In the formula, A is an auxiliary variable introduced to accelerate iteration under the FSITA framework; Li is... The Lipschitz constant;

[0028] Will Rewritten as Then, write the augmented Lagrange equation.

[0029]

[0030] In the formula, Λ is the Lagrange multiplier;

[0031] Within the ADMM algorithm framework, multiple subproblems are solved iteratively and alternately, with the variables in the iterative solution being S, X, U, A, and Λ; where,

[0032] The S subproblem is

[0033] The subproblem X is,

[0034]

[0035] In the formula, k represents the number of iterations;

[0036] The subproblem X can be rewritten using matrix differentiation as follows:

[0037]

[0038] The solution is obtained by converting the frequency domain to Fourier transform.

[0039] In the formula, F -1 (·) represents the inverse Fourier transform; F(·) represents the Fourier transform;

[0040]

[0041]

[0042] The U-subproblem is,

[0043]

[0044] The solution is obtained using matrix differentiation.

[0045] Update of dynamic step size t under the FISTA framework

[0046] Update of auxiliary variable A,

[0047] Update of the Lagrange multiplier Λ, Λ k+1 =X k+1 BU k+1 +Λ k ;

[0048] The fused image is obtained after the iteration is completed.

[0049] This invention provides a multispectral image panchromatic sharpening system based on variational theory, comprising:

[0050] The image acquisition module is used to acquire the multispectral and panchromatic images to be fused.

[0051] The fusion computing module is used to iteratively solve the optimization function based on the acquired multispectral and panchromatic images using the ADMM algorithm to obtain a high-resolution multispectral image after panchromatic sharpening.

[0052] The expression for the optimization function is:

[0053]

[0054] In the formula, For spectral fidelity, For gradient fidelity, α1 and α2 represent the gradient fidelity term coefficient constant and the NSST term coefficient constant, respectively; X is the high-resolution multispectral image after panchromatic sharpening, Y is the multispectral image to be fused, and P is the panchromatic image to be fused; B represents the blurring operation, and D represents the downsampling operation. For gradient taking, ⊙ represents the Hadamard product operation; G is the gradient coefficient matrix. This represents the NSST transform, where K represents the NSST band coefficient; This represents the Frobenius norm.

[0055] A further improvement to the system of the present invention is that, in the optimization function, the calculation expression for the gradient coefficient matrix G is:

[0056]

[0057]

[0058] In the formula, M is the total number of pixel categories; p z G represents the probability that a pixel belongs to class z; z Y is the gradient coefficient matrix for pixel class z; z A multispectral image for pixel category z; D T This is an upsampling operation; This is a least squares regression.

[0059] A further improvement of the system of the present invention is that, in the optimization function, when calculating the gradient coefficient matrix G, a Gaussian mixture model clustering method is used to cluster the pixels in the image.

[0060] A further improvement of the system of the present invention is that, in the optimization function, the calculation expression for the NSST band coefficient K is:

[0061]

[0062] In the formula, DCT(·) represents the discrete cosine transform; This is a least squares regression.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] The multispectral image panchromatic sharpening scheme provided by this invention uses a newly proposed optimization function to obtain a high-resolution multispectral image after panchromatic sharpening, which solves the problem of insufficient utilization of spectral data in existing methods. Specifically, the optimization function incorporates a detail-preserving term designed with non-subsampled shear wave transform, enabling full extraction of spatial information from the panchromatic image. Solving multiple optimization terms within a variational optimization framework yields a high-resolution fused image. In summary, the multispectral image panchromatic sharpening method based on variational theory disclosed in this invention, employing a newly designed variational optimization function and solving within the ADMM framework to obtain a fused image, fully utilizes the spectral and spatial information of the source image, resulting in a fused image of superior overall quality.

[0065] In a further preferred embodiment of the present invention, Gaussian mixture model clustering is introduced, and the fusion coefficient is recalculated using a probabilistic weighting method, which can overcome the problem of insufficient utilization of spectral information in existing methods. Further explanation: To address the problem of insufficient utilization of spectral data in the spectral data clustering process, the present invention uses Gaussian mixture model clustering to classify spectral data. The classification results obtained by this method better meet the need for separate calculation of gradient scale coefficients and better fit the distribution of spectral data.

[0066] In a further preferred embodiment of the present invention, a detail-preserving term based on NSST is designed, and a method for calculating K is provided, which can overcome the problem of insufficient spatial detail preservation in existing methods. Further explanation: to address the issue of insufficient spatial information extracted by gradient operations, the present invention uses an NSST term to replace the gradient term to constrain the spatial information of the fused image, thus allowing the fused image to inherit more spatial information from the panchromatic image. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0068] Figure 1 This is a flowchart illustrating a multispectral image panchromatic sharpening method based on variational theory, as described in an embodiment of the present invention.

[0069] Figure 2 This is a schematic diagram of the overall framework for image fusion in an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of the gradient term coefficient calculation process based on probability reweighting in an embodiment of the present invention;

[0071] Figure 4 This is a schematic diagram of the visualization results on the Pleiades dataset in an embodiment of the present invention; wherein, Figure 4 (a) is a multispectral image. Figure 4 Image (b) is a panchromatic image. Figure 4 Image (c) is a reference image. Figure 4 (d) is a schematic diagram of the fusion results of the IHS transform method. Figure 4 Image (e) is a schematic diagram of the sharpening result using the PCA transform method. Figure 4 Image (f) is a schematic diagram of the sharpening result using the multi-aperture wavelet transform method. Figure 4 Image (g) is a schematic diagram of the sharpening result using the non-subsampled shear wave transform method. Figure 4 The diagram in h shows the sharpening result using the total variation method. Figure 4 Image (i) is a schematic diagram of the sharpening result using the CDIF method. Figure 4 (j) is a schematic diagram of the sharpening result of the method provided by the present invention;

[0072] Figure 5 This is a schematic diagram of the visualization results on the GF-2 dataset in an embodiment of the present invention; wherein, Figure 5 (a) is a multispectral image. Figure 5 Image (b) is a panchromatic image. Figure 5 Image (c) is a reference image. Figure 5 (d) is a schematic diagram of the fusion results of the IHS transform method. Figure 5 Image (e) is a schematic diagram of the sharpening result using the PCA transform method. Figure 5 Image (f) is a schematic diagram of the sharpening result using the multi-aperture wavelet transform method. Figure 5 Image (g) is a schematic diagram of the sharpening result using the non-subsampled shear wave transform method. Figure 5 The diagram in middle (h) shows the sharpening result using the total variation method. Figure 5 Image (i) is a schematic diagram of the sharpening result using the CDIF method. Figure 5 (j) is a schematic diagram of the sharpening result of the method provided by the present invention;

[0073] Figure 6 This is a schematic diagram of the visualization results on QuickBird dataset 1 in an embodiment of the present invention; wherein, Figure 6 (a) is a multispectral image. Figure 6 Image (b) is a panchromatic image. Figure 6 (c) is a schematic diagram of the fusion results of the IHS transform method. Figure 6 Image (d) shows the sharpening result obtained using the PCA transform method. Figure 6 Image (e) is a schematic diagram of the sharpening result using the multi-aperture wavelet transform method. Figure 6 Image (f) is a schematic diagram of the sharpening result using the non-subsampled shear wave transform method. Figure 6(g) is a schematic diagram of the sharpening result using the total variation method. Figure 6 The image in middle (h) is a schematic diagram of the sharpening result using the CDIF method. Figure 6 Image (i) is a schematic diagram of the sharpening result provided by the method of the present invention;

[0074] Figure 7 This is a schematic diagram of the visualization results on the QuickBird dataset 2 in an embodiment of the present invention; wherein, Figure 7 (a) is a multispectral image. Figure 7 Image (b) is a panchromatic image. Figure 7 (c) is a schematic diagram of the fusion results of the IHS transform method. Figure 7 Image (d) shows the sharpening result obtained using the PCA transform method. Figure 7 Image (e) is a schematic diagram of the sharpening result using the multi-aperture wavelet transform method. Figure 7 Image (f) is a schematic diagram of the sharpening result using the non-subsampled shear wave transform method. Figure 7 (g) is a schematic diagram of the sharpening result using the total variation method. Figure 7 The image in middle (h) is a schematic diagram of the sharpening result using the CDIF method. Figure 7 Image (i) is a schematic diagram of the sharpening result provided by the method of the present invention;

[0075] Figure 8 This is a schematic diagram of a multispectral image panchromatic sharpening system based on variational theory in an embodiment of the present invention. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0077] Please see Figure 1 The present invention provides a multispectral image panchromatic sharpening method based on variational theory, comprising the following steps:

[0078] Step 1: Obtain the multispectral image and panchromatic image to be fused;

[0079] Step 2: Based on the multispectral image and panchromatic image to be fused obtained in Step 1, the ADMM algorithm is used to iteratively solve the optimization function to obtain the high-resolution multispectral image after panchromatic sharpening.

[0080] The expression for the optimization function is:

[0081]

[0082] In the formula, For spectral fidelity, For gradient fidelity, α1 represents the NSST term (non-downsampled shear wave transform term); α2 and α1 represent the constants of the gradient fidelity term and the NSST term, respectively; X is the high-resolution multispectral image after panchromatic sharpening, Y is the multispectral image to be fused, and P is the panchromatic image to be fused; B represents the blurring operation, and D represents the downsampling operation. For gradient taking, ⊙ represents the Hadamard product operation; G is the gradient coefficient matrix. This represents the NSST transform, where K represents the NSST band coefficient; This represents the Frobenius norm.

[0083] The technical solution provided in this invention uses a newly proposed optimization function to obtain a high-resolution multispectral image after panchromatic sharpening, which solves the problem of insufficient utilization of spectral data in existing methods. Specifically, a detail-preserving term is designed in conjunction with non-subsampled shear wave transform to fully extract spatial information from the panchromatic image. Multiple optimization terms are then combined and solved within a variational optimization framework to obtain the fused image. In summary, the variational theory-based multispectral image panchromatic sharpening method disclosed in this invention, which first designs a variational optimization function and then solves it within the ADMM framework to obtain the fused image, fully utilizes the spectral and spatial information of the source image, resulting in a fused image of superior overall quality.

[0084] In a preferred embodiment, the gradient fidelity term In the above, the steps to solve for the gradient coefficient matrix G are as follows:

[0085]

[0086] In the formula, M represents the total number of pixel categories; p z G represents the probability that a pixel belongs to class z; z Y is the gradient coefficient matrix for pixel class z; z A multispectral image for pixel category z; D T This is an upsampling operation; This is a least squares regression.

[0087] In a preferred embodiment of the present invention, after the total number of pixel categories is determined, the pixels in the image are clustered using a Gaussian mixture model clustering method based on the multispectral image and panchromatic image to be fused. Interpretationally, the Gaussian mixture model can better simulate the distribution of data. By clustering the pixels in the image, pixels with similar characteristics can be effectively grouped together, which facilitates subsequent processing steps.

[0088] In a preferred embodiment, the NSST item In the solution, the steps for calculating the NSST band coefficient K are as follows:

[0089]

[0090] In the formula, DCT(·) represents the discrete cosine transform; This is a least squares regression.

[0091] In a specific exemplary embodiment of the present invention, the calculation expression for the NSST band coefficient K(b,d) in the d-th direction of the b-th band is as follows:

[0092]

[0093] In the formula, Y represents the NSST transform in the d-th direction; b This is the multispectral image of the b-th band.

[0094] Specifically, according to a specific embodiment of the present invention, the steps of clustering pixels in an image using a Gaussian mixture model clustering method based on a multispectral image and a panchromatic image to be fused specifically include:

[0095] Step (1) Upsample the multispectral image to the same size as the panchromatic image;

[0096] Step (2): After step (1), the upsampled multispectral image and panchromatic image are connected in the band direction, so that each pixel is a multidimensional vector;

[0097] Step (3) After step (2), all pixels are clustered into multiple categories in multidimensional space using a Gaussian mixture model.

[0098] Please see Figure 2 The specific exemplary embodiment of the present invention, the multispectral image panchromatic sharpening method based on variational theory, specifically includes the following steps:

[0099] Step S1: First, the Gaussian mixture model clustering method is used to cluster the pixels in the image into multiple categories according to their characteristics;

[0100] Step S2: After completing the pixel clustering in Step S1, the gradient scaling factor for each pixel class is estimated using the least squares method. The purpose of this step is to extract the spatial structural characteristics of each pixel class. Then, a weighted calculation is performed using the probability of a pixel belonging to each class. This ensures that the information of each type of pixel can be fully utilized and balanced in the subsequent fusion process.

[0101] Step S3: To improve the quality of the fused image, a fidelity term based on the non-subsampled Shearlet transform (NSST) is added to the variational optimization function. Explain, NSST is a multi-scale, multi-directional transform method that can effectively extract high-frequency information and texture features from the image. By introducing it into the variational optimization function, the detailed information in the original image can be better preserved, and the loss and distortion of information can be reduced during the fusion process.

[0102] In the technical solution provided by the embodiments of the present invention, the image fusion method based on variational theory can be further improved through the above three steps. First, by clustering with Gaussian mixture models, the information in spectral data can be better utilized. Second, by estimating the gradient scaling coefficients and performing weighted calculations using the least squares method, spatial information can be extracted more accurately. Finally, by adding a fidelity term based on NSST, the quality and visual effect of the fused image can be further improved.

[0103] Please see Figure 3 To further explain, step S2 of the above embodiments of the present invention specifically includes:

[0104] S201: For the corresponding pixels of the gradient images of multispectral images and panchromatic images of the same type, the relationship between the low-pass filtered panchromatic image pixels and the upsampled multispectral image pixels is calculated using least squares regression to obtain the coefficient G corresponding to each pixel;

[0105] S202: For each pixel, its final coefficient G is the probability of it belonging to each class as the weight, multiplied by the coefficient of the class as a whole, and the coefficient obtained in the first calculation is replaced by the weighted coefficient.

[0106] S203: Multiply the obtained coefficient matrix by the gradient of the panchromatic image, subtract it from the gradient of the fused image, constrain it with the F norm, and set it as one of the terms under the variational optimization function.

[0107] To further explain, step S3 of the above embodiments of the present invention specifically includes:

[0108] S301: Perform NSST transformation on the upsampled multispectral and panchromatic images, perform discrete cosine transformation on the low-frequency coefficients obtained separately to the frequency domain, and estimate an overall coefficient K using the least squares method. That is, each band of the multispectral image shares a coefficient K.

[0109] S302: Multiply the NSST coefficients of each scale of the panchromatic image by the K coefficients, then perform the inverse NSST transform. Subtract the fused image from the image after the inverse transform, and set it as a term in the variational optimization function using the F norm constraint.

[0110] Specifically, the multispectral image panchromatic sharpening method based on variational theory, as exemplified in this embodiment of the invention, includes the following steps:

[0111] Step 1: Upsample the multispectral image to the same size as the panchromatic image, and then connect the upsampled multispectral image and the panchromatic image along the band direction;

[0112] Step 2: After step 1, cluster all pixels into 10 classes in multidimensional space using a Gaussian mixture model;

[0113] Step 3: After step 2, for corresponding pixels in the same type of multispectral and panchromatic images, use least squares regression to fit the relationship between the gradient pixels of the panchromatic image after low-pass filtering and the gradient pixels of the upsampled multispectral image. Obtain the coefficients corresponding to each pixel;

[0114] Step 4: For each pixel, its final coefficient G is the probability of it belonging to each class multiplied by the weight of the class as a whole, and the weighted coefficient is calculated using the weighted average. Replace the coefficients obtained in the first calculation;

[0115] Step 5: Multiply the obtained coefficient matrix by the gradient of the panchromatic image, subtract it from the gradient of the fused image, constrain it using the F-norm, and set it as one of the terms under the variational optimization function.

[0116] Step 6: Perform NSST transformation on the upsampled multispectral image and the low-pass filtered panchromatic image. Perform discrete cosine transformation on the high-frequency coefficients obtained at the coarsest scale to the frequency domain. Estimate an overall coefficient K using the least squares method. That is, each band of the multispectral image shares a coefficient K in each NSST direction.

[0117] Step 7: Multiply the NSST coefficients of the panchromatic image at each scale and in each direction by the K coefficients, then perform the inverse NSST transform. Subtract the fused image from the image after the inverse transform, and constrain it with the F norm as a term in the variational optimization function.

[0118] Step 8: Subtract the low-pass filtered fused image from the upsampled multispectral image, and constrain it using the F-norm, setting it as a term in the variational optimization function.

[0119] Based on the above steps, after obtaining the overall optimization function, iteratively solve it within the ADMM algorithm framework to obtain a high-resolution multispectral image after panchromatic sharpening.

[0120] In a specific and exemplary embodiment of the present invention, based on the obtained optimization function, the process of iteratively solving within the ADMM algorithm framework to obtain a high-resolution multispectral image after panchromatic sharpening includes the following specific solution steps:

[0121] Step 1: Introduce the auxiliary variable U = XB, and rewrite the optimization function in the following form:

[0122]

[0123] stU = XB;

[0124] Step 2: Use the inverse transform operation separated under the FISTA framework to avoid the transpose of NSST. Therefore, the rewritten optimization function is:

[0125]

[0126]

[0127] In the formula: A is an auxiliary variable introduced to accelerate iteration under the FSITA framework; Li is... The Lipschitz constant;

[0128] Step 3: Rewrite S as follows:

[0129] Step 4: Write the augmented Lagrange equation.

[0130]

[0131] In the formula: the hyperparameters are determined through multiple adjustments, with α1 being 0.00004; α2 being 0.000004; α3 being 0.00008; and Λ being the Lagrange multiplier.

[0132] Step 5: Solve multiple subproblems iteratively within the ADMM algorithm framework, involving five variables: S, X, U, A, and Λ.

[0133] S-subproblem:

[0134] Subproblem X: Subproblem X is expressed as the following optimization problem,

[0135]

[0136] In the formula, k represents the number of iterations;

[0137] Since the X subproblem only contains the Frobenius norm, it can be rewritten using matrix differentiation as follows:

[0138]

[0139] Solving this problem by performing a Fourier transform in the frequency domain avoids the difficulty of expressing gradient and fuzzy operations in matrix form in the spatial domain.

[0140] In the formula, F -1 (·) represents the inverse Fourier transform; F(·) represents the Fourier transform;

[0141]

[0142]

[0143] U-subproblem: The U-subproblem is expressed as the following optimization problem,

[0144]

[0145] The solution is also obtained using matrix differentiation.

[0146] Update of dynamic step size t under the FISTA framework

[0147] Update of auxiliary variable A,

[0148] Update of the Lagrange multiplier Λ, Λ k+1 =X k+1 BU k+1 +Λ k ;

[0149] After iterating 200 times, the above steps are completed to obtain the fused image X.

[0150] In summary, this invention specifically discloses a multispectral image panchromatic sharpening method based on variational theory. Building upon the classic multispectral image panchromatic sharpening method, it introduces Gaussian mixture model clustering and recalculates the fusion coefficients using a probabilistic weighting approach to overcome the problem of insufficient utilization of spectral information in the classic method. Furthermore, it designs a detail fidelity term based on NSST to overcome the problem of insufficient spatial detail preservation in the classic method.

[0151] Please see Figures 4 to 7In a specific embodiment of the present invention, a dataset with reference influence is selected from the Pleiades satellite and a dataset from the GF-2 satellite for a comparative experiment, and two datasets without reference influence are selected from the QuickBird satellite for a comparative experiment; wherein,

[0152] The Pleiades satellite has a reference image size of 256×256×4, a multispectral image size of 64×64×4, a panchromatic image size of 256×256, and a spatial resolution of 0.5m for the panchromatic image.

[0153] The reference image of the GF-2 satellite has a size of 256×256×4, the multispectral image has a size of 64×64×4, the panchromatic image has a size of 256×256, and the spatial resolution of the panchromatic image is 1m.

[0154] The QuickBird satellite has no reference imagery. Its multispectral image size is 64×64×4, and its panchromatic image size is 256×256. The spatial resolution of the panchromatic image is 0.61m.

[0155] In this embodiment of the invention, mainstream methods are selected from three types of image fusion methods (component replacement method, multi-resolution analysis method, and variational method) for comparison to illustrate the effectiveness of the method proposed in this embodiment of the invention; wherein,

[0156] Methods based on component substitution: IHS transformation method, principal component analysis method;

[0157] Methods based on multi-resolution analysis: multi-aperture wavelet transform method, non-subsampling shear wavelet transform method;

[0158] Variational methods: TVP method, CDIF method.

[0159] In this embodiment of the invention, the comparative experimental results are shown in Tables 1 to 4.

[0160] Table 1. Objective Evaluation Indicators of Experimental Results from the Pleiades Satellite Dataset

[0161]

[0162] Table 2. Objective Evaluation Indicators of Experimental Results for GF2 Satellite Dataset

[0163]

[0164] Table 3. Objective Evaluation Indicators of Experimental Results for QuickBird Satellite Dataset 1

[0165]

[0166] Table 4. Objective Evaluation Indicators of QuickBird Satellite Dataset 2 Experimental Results

[0167]

[0168] Based on the experimental results, it can be concluded that the method of the present invention performs well on all evaluation metrics with and without reference images, demonstrating its effectiveness and superiority in panchromatic sharpening of multispectral images.

[0169] In summary, the technical solution provided by the embodiments of the present invention firstly uses Gaussian mixture model clustering to cluster spectral data and constructs a gradient fidelity term accordingly; secondly, it combines non-subsampled shear wave transform to design a detail fidelity term to fully extract the spatial information of the panchromatic image, and then solves the fused image under a variational optimization framework by combining multiple optimization terms. Experimental results show that the method of the present invention can fully utilize the spectral and spatial information of the source image to obtain a fused image with better overall quality.

[0170] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0171] Please see Figure 8 In this embodiment of the invention, a multispectral image panchromatic sharpening system based on variational theory is provided, comprising:

[0172] The image acquisition module is used to acquire the multispectral and panchromatic images to be fused.

[0173] The fusion computing module is used to iteratively solve the optimization function based on the acquired multispectral and panchromatic images using the ADMM algorithm to obtain a high-resolution multispectral image after panchromatic sharpening.

[0174] The expression for the optimization function is:

[0175]

[0176] In the formula, For spectral fidelity, For gradient fidelity, α1 and α2 represent the gradient fidelity term coefficient constant and the NSST term coefficient constant, respectively; X is the high-resolution multispectral image after panchromatic sharpening, Y is the multispectral image to be fused, and P is the panchromatic image to be fused; B represents the blurring operation, and D represents the downsampling operation. For gradient taking, ⊙ represents the Hadamard product operation; G is the gradient coefficient matrix. This represents the NSST transform, where K represents the NSST band coefficient; This represents the Frobenius norm.

[0177] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or function. The processor described in this embodiment of the present invention can be used to execute the operation of a multispectral image panchromatic sharpening method based on variational theory.

[0178] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the multispectral image panchromatic sharpening method based on variational theory in the above embodiments.

[0179] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0180] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0181] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0182] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multispectral image panchromatic sharpening method based on variational theory, characterized in that, Includes the following steps: Acquire the multispectral and panchromatic images to be fused; Based on the acquired multispectral and panchromatic images, the ADMM algorithm is used to iteratively solve the optimization function to obtain a high-resolution multispectral image after panchromatic sharpening. The expression for the optimization function is: ； In the formula, For spectral fidelity, For gradient fidelity, For NSST items; , These are the constants of the gradient fidelity term coefficient and the constants of the NSST term coefficient, respectively. This is a high-resolution multispectral image after pancolor sharpening. For the multispectral images to be fused, The image to be merged is a panchromatic image. For fuzzy operations, This indicates a downsampling operation. For gradient taking operation, This is the Hadamard product operation; G is the gradient coefficient matrix. This represents the NSST transform, where K represents the NSST band coefficient; Represents the Frobenius norm; In the optimization function, the expression for calculating the gradient coefficient matrix G is: ； ； In the formula, M is the total number of pixel categories; is the probability that a pixel belongs to class z; Let z be the gradient coefficient matrix of pixel class z; A multispectral image for pixel category z; This is an upsampling operation; This is a least squares regression; In the optimization function, the expression for calculating the NSST band coefficient K is as follows: ； In the formula, DCT(·) represents the discrete cosine transform; This is a least squares regression.

2. The multispectral image panchromatic sharpening method based on variational theory according to claim 1, characterized in that, In the optimization function, when calculating the gradient coefficient matrix G, the Gaussian mixture model clustering method is used to cluster the pixels in the image.

3. The multispectral image panchromatic sharpening method based on variational theory according to claim 2, characterized in that, The steps of clustering pixels in an image using the Gaussian mixture model clustering method include: Upsample the multispectral image to the same size as the panchromatic image; connect the upsampled multispectral image and the panchromatic image along the band direction; cluster all pixels into multiple categories in multidimensional space using a Gaussian mixture model.

4. A multispectral image panchromatic sharpening system based on variational theory, characterized in that, include: The image acquisition module is used to acquire the multispectral and panchromatic images to be fused. The fusion computing module is used to iteratively solve the optimization function based on the acquired multispectral and panchromatic images using the ADMM algorithm to obtain a high-resolution multispectral image after panchromatic sharpening. The expression for the optimization function is: ； In the formula, For spectral fidelity, For gradient fidelity, For NSST items; , These are the constants of the gradient fidelity term coefficient and the constants of the NSST term coefficient, respectively. This is a high-resolution multispectral image after pancolor sharpening. For the multispectral images to be fused, The image to be merged is a panchromatic image. For fuzzy operations, This indicates a downsampling operation. For gradient taking operation, This is the Hadamard product operation; G is the gradient coefficient matrix. This represents the NSST transform, where K represents the NSST band coefficient; Represents the Frobenius norm; In the optimization function, the expression for calculating the gradient coefficient matrix G is: ； ； In the formula, M is the total number of pixel categories; is the probability that a pixel belongs to class z; Let z be the gradient coefficient matrix of pixel class z; A multispectral image for pixel category z; This is an upsampling operation; This is a least squares regression; In the optimization function, the expression for calculating the NSST band coefficient K is as follows: ； In the formula, DCT(·) represents the discrete cosine transform; This is a least squares regression.

5. The multispectral image panchromatic sharpening system based on variational theory according to claim 4, characterized in that, In the optimization function, when calculating the gradient coefficient matrix G, the Gaussian mixture model clustering method is used to cluster the pixels in the image.

Images