A hyperspectral fusion method based on dynamic gradient group sparsity and low-rank regularization

Abstract

The present invention proposes a hyperspectral fusion method based on dynamic gradient group sparsity and low-rank regularization. A low-resolution hyperspectral image is obtained by a hyperspectral sensor, and a low-resolution multispectral image and a high-resolution panchromatic image of the same picture are collected by a multispectral sensor. A fusion model among the low-resolution hyperspectral image, the low-resolution multispectral image, and the high-resolution panchromatic image is constructed by using dynamic gradient group sparsity regularization and low-rank regularization. The fusion model is solved by the alternating direction multiplier method to obtain a coefficient matrix, and then a high-resolution hyperspectral image is obtained by multiplying the coefficient matrix with the subspace matrix. The fusion method of the present invention combines the subspace regularization of the image with the image fusion, thereby converting the solution of the fusion target into the solution of the low-dimensional coefficient matrix, improving the computational efficiency, and obtaining a high-resolution hyperspectral image that is superior to the comparative method in both qualitative and quantitative aspects.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing image fusion, and in particular relates to a hyperspectral fusion method based on dynamic gradient group sparsity and low-rank regularization. Background Art

[0002] In the field of remote sensing, panchromatic (PAN), multispectral (MS) and hyperspectral (HS) data have a wide range of applications, including land classification, change detection, etc. Due to the cost limitations of optical sensors and the limitations of data storage and transmission bandwidth, the design of optical sensors requires a trade-off between spatial resolution and spectral resolution. For example, hyperspectral data with hundreds of bands provides high spectral resolution, but its spatial image is generally blurry. The spectral information of a panchromatic image with a single band is relatively simple, but its spatial information is richer. Due to the application demand for remote sensing data with both high spatial resolution and spectral resolution, a variety of image fusion technologies have been studied.

[0003] Multispectral fusion technology fuses low-resolution multispectral images (LR-MSI) with high-resolution panchromatic images (HR-PAN) to improve the spatial resolution of multispectral images. This fusion technology is currently relatively mature and is mainly divided into component replacement methods, multiresolution analysis methods, and methods based on variational methods and machine learning. Hyperspectral fusion technology usually fuses low-resolution hyperspectral images (LR-HSI) with high-resolution multispectral images (HR-MSI) to improve the spatial resolution of hyperspectral images. Compared with multispectral image fusion technology, the data dimension used in the hyperspectral image fusion process is higher and more variables need to be estimated. Hyperspectral image fusion technology mainly includes the extension of multispectral image fusion methods, Bayesian-based methods, and linear spectral decomposition-based methods.

[0004] Considering that the information contained in hyperspectral, multispectral and panchromatic images is complementary, fusing the three types of data taken from the same picture can better mine the spatial and spectral information contained therein. There are few studies on this type of data fusion method. In order to better utilize the total information of hyperspectral images, multispectral images and panchromatic images, this paper proposes a new model-based method that combines dynamic gradient group sparsification and low-rank regularization with hyperspectral, multispectral and panchromatic data fusion. Summary of the invention

[0005] Aiming at the shortcomings of existing remote sensing image fusion methods, the present invention proposes a hyperspectral fusion method based on dynamic gradient group sparsity and low-rank regularization.

[0006] Step 1: Obtain a low-resolution hyperspectral image through a hyperspectral sensor, and collect a low-resolution multispectral image and a high-resolution panchromatic image of the same picture through a multispectral sensor;

[0007] Step 2: construct a fusion model among the low-resolution hyperspectral image, the low-resolution multispectral image and the high-resolution panchromatic image through the low-resolution hyperspectral image, the low-resolution multispectral image and the high-resolution panchromatic image described in step 1;

[0008] Step 3: Based on the fusion model described in step 2, solve the coefficient matrix X by alternating direction multiplication method (3) , and then obtain the high-resolution hyperspectral image F by multiplying the coefficient matrix with the subspace matrix (3) ;

[0009] Preferably, the low-resolution hyperspectral image in step 1 is recorded as a tensor Its 3-mode expansion matrix is Indicates that the image has L h Bands, W h ×H h pixels;

[0010] The low-resolution multispectral image described in step 1 is recorded as a tensor Its 3-mode expansion matrix is Indicates that the image has L m Bands, W m ×H m pixels;

[0011] The high-resolution full-color image described in step 1 is recorded as a tensor Its 3-mode expansion matrix is Indicates that the image has 1 band, W p ×H p pixels;

[0012] The resolution ratio between the high-resolution panchromatic image and the low-resolution hyperspectral image is s h , the resolution ratio between the high-resolution panchromatic image and the low-resolution multispectral image is s m ,Right now

[0013] Preferably, the fusion model between the low-resolution hyperspectral image, the low-resolution multispectral image, and the high-resolution panchromatic image in step 2 is composed of a hyperspectral image fitting term, a multispectral image fitting term, a dynamic gradient group sparse regularization term, and a low-rank regularization term;

[0014] The hyperspectral image fitting term and multispectral image fitting term are obtained by fitting the low-resolution hyperspectral image H (3) , low-resolution multispectral image M (3) The spatial degradation relationship and spectral degradation relationship between the low-resolution hyperspectral image H and the high-resolution hyperspectral image H are constructed. (3) It is equivalent to a spatially downsampled version of a high-resolution hyperspectral image, a low-resolution multispectral image M (3) It is equivalent to a high-resolution hyperspectral image after spatial downsampling and spectral downsampling;

[0015] The hyperspectral image fitting term, multispectral image fitting term, dynamic gradient group sparse regularization term, and low-rank regularization term constitute a target energy function about a coefficient matrix;

[0016] Preferably, the specific implementation of constructing the fusion model among the hyperspectral image, the multispectral image, and the panchromatic image in step 2 includes the following sub-steps:

[0017] Step 2.1: The low-resolution hyperspectral image H(3) can be viewed as a spatially downsampled version of the high-resolution hyperspectral image F(3):

[0018] H (3) =F (3) B h S h

[0019] Among them, the matrix is the spatial fuzzy matrix. It is a spatial downsampling operation used to reduce the spatial resolution of hyperspectral images.

[0020] Step 2.2: Low-resolution multispectral image M (3) It can be regarded as a high-resolution hyperspectral image F (3) The spatially and spectrally downsampled version:

[0021] M (3) =Rm F (3) B m S m

[0022] in, is the spectral response matrix of the multi-spectral instrument. is the spatial fuzzy matrix. It is a spatial downsampling operation used to reduce the spatial resolution of hyperspectral images.

[0023] Step 2.3: The high-resolution panchromatic image P and the high-resolution hyperspectral image F have the same spatial edge information. Their gradient difference should satisfy the group sparsity feature. The regularization term of dynamic gradient group sparsity can be established:

[0024] The high-resolution hyperspectral image is recorded as a tensor Its 3-mode expansion matrix is Indicates that the image has L h Bands, W p ×H p pixels;

[0025]

[0026] in, Indicates copying a high-resolution full-color image to L m bands.

[0027] B in step 2.1 h Matrix, R in step 2.2 m , B m The matrix can be composed of a low-resolution hyperspectral image H (3) , low-resolution multispectral image M (3) , high-resolution full-color image P (3) It is estimated that the present invention uses an estimation method proposed in a hyperspectral image fusion method named HySure.

[0028] Step 2.4: Use low-rank regularization to build a 3D tensor The regularization term of :

[0029] ||X|| *

[0030] Step 2.5: Combining the product model of the subspace matrix and the coefficient matrix to transform the high-resolution hyperspectral image into a coefficient matrix model of the high-resolution hyperspectral image in the low-dimensional subspace;

[0031] The product model of the subspace matrix and the coefficient matrix in step 2.5 is:

[0032]

[0033] Among them, × 3 represents the third mode product, is a subspace matrix, wherein the subspace matrix represents a vector of D pure spectral features, and the subspace matrix is ​​directly obtained by decomposing a low-resolution hyperspectral image, and the image decomposition method selected by the present invention is vertex component analysis;

[0034] Tensor is the coefficient matrix, representing Each pixel is represented by the coefficients of a linear combination of the vector members in the subspace matrix, Represents pixels The corresponding coefficients are: i is the spatial horizontal coordinate of the pixel point, and j is the spatial vertical coordinate of the pixel point.

[0035] The coefficient matrix model of converting the high-resolution hyperspectral image into the high-resolution hyperspectral image in the low-dimensional subspace described in step 2.5 is:

[0036] F (3) =EX (3)

[0037] in, For tensors 3-mode expansion;

[0038] Step 2.6: Based on steps 2.1 to 2.5, establish a three-way fusion model:

[0039]

[0040] in represents the Frobenius norm, ||·|| 2，1 represents the L2,1 norm, ||·|| * represents the nuclear norm, λ m ,λ φ and λ l are the parameters of each balance. (3) is the two-dimensional matrix representation of the coefficients.

[0041] Preferably, the specific implementation of step 3 includes the following sub-steps:

[0042] Step 3.1: Introduce auxiliary variable O, satisfying O = X (3) B h ; Introduce auxiliary variable U, satisfying U=X (3) B m ; Introduce auxiliary variable V, satisfying V=X (3) ; Introduce auxiliary variable W, satisfying W=R mEV; introduce auxiliary variable Q, satisfying Q=X (3) The three-way fusion model is expressed as:

[0043]

[0044] stO=X (3) B h

[0045] U=X (3) B m

[0046] V=X (3)

[0047] W=R m EV

[0048] Q=X (3)

[0049] The augmented Lagrangian function of the three-way fusion model is expressed as:

[0050]

[0051] Among them, Y 1 , Y 2 , Y 3 , Y 4 and Y 5 is the scaled dual variable, represents the Frobenius norm, ||·|| 2，1 represents the L2,1 norm, ||·|| * represents the nuclear norm, λ m ,λ φ and λ l are the parameters of each balance. μ is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients. (3) It is a two-dimensional matrix representation of a low-resolution hyperspectral image. (3) It is a two-dimensional matrix representation of a low-resolution multispectral image. Indicates copying a high-resolution full-color image to L m bands. Represents the gradient symbol. is the spatial fuzzy matrix, is the spatial fuzzy matrix, is the spatial downsampling operation, is the spatial downsampling operation, is the spectral response matrix of the multi-spectral instrument, is the subspace matrix. The optimization problem of the augmented Lagrangian function of the three fusion models can be decomposed into X subproblem, O subproblem, U subproblem, V subproblem, W subproblem, Q subproblem and Y subproblem.

[0052] Step 3.2: Solve the X subproblem:

[0053]

[0054] Among them, t in the upper right corner represents the number of iterations, Y 1 , Y 2 , Y 3 and Y 5 is the scale dual variable, represents the Frobenius norm, μ is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients. is the spatial fuzzy matrix, is the spatial fuzzy matrix;

[0055] The solution to the above problem is:

[0056]

[0057] Step 3.3: Solve the O subproblem:

[0058]

[0059] Among them, Y 1 is the scale dual variable, represents the Frobenius norm, μ is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients. (3) It is a two-dimensional matrix representation of a low-resolution hyperspectral image. is the spatial fuzzy matrix, is the spatial downsampling operation, is the subspace matrix.

[0060] Divide O into OS h and in Indicates that the matrix S h The selected pixel point, then the solution of subproblem O is:

[0061]

[0062]

[0063] Step 3.4: Solve the U-problem:

[0064]

[0065] Among them, Y 2 is the scale dual variable, represents the Frobenius norm, λ m is the parameter for balancing each item. μ is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients. (3) It is a two-dimensional matrix representation of a low-resolution multispectral image. is the spatial fuzzy matrix, is the spatial downsampling operation, is the spectral response matrix of the multi-spectral instrument, is the subspace matrix.

[0066] Divide U into US m and in Indicates that the matrix S m For the selected pixel, the solution to the U subproblem is:

[0067]

[0068]

[0069] Step 3.5: Solve the V subproblem:

[0070]

[0071] Among them, Y 3 , Y 4 is the scale dual variable, represents the Frobenius norm, μ is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients. is the spectral response matrix of the multi-spectral instrument, is the subspace matrix.

[0072] The solution to the above problem is:

[0073]

[0074] Step 3.6: Solve the W subproblem:

[0075]

[0076] Among them, Y 4 is the scale dual variable, represents the Frobenius norm, ||·|| 2，1 represents the L2,1 norm,,λ φ is the parameter for balancing each item. μ is the penalty parameter. Indicates copying a high-resolution full-color image to Lm bands. Represents the gradient symbol. is the spectral response matrix of the multi-spectral instrument, is the subspace matrix.

[0077] Let G = WP, then the W subproblem becomes

[0078]

[0079] The W subproblem can be solved directly by the Vectorial Total Variation algorithm.

[0080] Step 3.7: Solve the Q subproblem:

[0081]

[0082] Among them, Y 5 is the scale dual variable, represents the Frobenius norm, ||·|| * represents the nuclear norm, λ l is the parameter for balancing each item. μ is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients.

[0083] The Q subproblem can be solved directly by the singular value contraction algorithm.

[0084] Step 3.8: Solve the Y subproblem. This subproblem can be directly updated by the following formula:

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] Among them, Y 1 , Y 2 , Y 3 , Y 4 and Y 5 is the scale dual variable, μ is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients. is the spatial fuzzy matrix, is the spatial fuzzy matrix, is the spectral response matrix of the multi-spectral instrument, is the subspace matrix. O, U, V, W and Q are auxiliary variables.

[0091] Step 3.9: After iteratively solving the above sub-problems, we get the coefficient tensor According to step 2 E and then obtain high-resolution hyperspectral images

[0092] The present invention is mainly aimed at the application demand of obtaining a high-resolution hyperspectral image by fusing a low-resolution hyperspectral image with a low-resolution multispectral and high-resolution panchromatic image. The fusion method of the present invention combines the subspace regularization of the image with the image fusion, thereby converting the solution of the fusion target into the solution of the low-dimensional coefficient matrix. The model proposed by the present invention includes two quadratic data fitting terms, a dynamic gradient group sparse regularization term and a low-rank regularization term, and is solved by the ADMM algorithm, thereby obtaining a hyperspectral fusion image that is superior to the comparative method in both qualitative and quantitative aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 : It is an experimental flow chart of the present invention.

[0094] Figure 2 : This is the fusion result diagram of the comparison method CNMF of the present invention.

[0095] Figure 3 : This is the fusion result diagram of the comparison method GSA of the present invention.

[0096] Figure 4 : is the fusion result diagram of the method proposed in the present invention.

[0097] Figure 5 : is a reference image of the present invention. DETAILED DESCRIPTION

[0098] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0099] The data used in the present invention comes from the urban area of ​​Pavia University photographed by the ROSIS aerial imaging spectrometer in Germany. The present invention selects 93 bands from the 115 bands of the data to obtain HR-HSI (320×320×93), which is used as the reference image Ref for quality evaluation after fusion. In order to generate LR-HSI (20×20×93), the present invention first uses a 7×7 Gaussian blur (mean zero, standard deviation 2) to blur the HR-HSI, and then downsamples every sixteen pixels in the width and height mode of the HR-HSI. In order to simulate the LR-MSI (80×80×4) of the same scene, a reflectance spectral response filter similar to IKONOS is used to sample the HR-HSI along the spectral mode, and then a 7×7 Gaussian blur (mean zero, standard deviation 2) is used to blur the HR-HSI, and finally downsamples every four pixels in the width and height mode of the HR-HSI. The method to generate HR-PAN (320×320) of the same scene is to first use a reflectance spectral response filter similar to IKONOS to sample HR-HSI into four bands along the spectral mode, and then linearly combine the first three bands according to the weights [0.114, 0.587, 0.299]. The flowchart of the fusion experiment is as follows Figure 1 shown.

[0100] This embodiment provides a hyperspectral fusion method based on dynamic gradient group sparsification and low-rank regularization, comprising the following steps:

[0101] Step 1: Obtain a low-resolution hyperspectral image through a hyperspectral sensor, and collect a low-resolution multispectral image and a high-resolution panchromatic image of the same picture through a multispectral sensor;

[0102] The low-resolution hyperspectral image in step 1 is recorded as a tensor Its 3-mode expansion matrix is Indicates that the image has L h =93 bands, W h ×H h =20×20=400 pixels;

[0103] The low-resolution multispectral image described in step 1 is recorded as a tensor Its 3-mode expansion matrix is Indicates that the image has L m = 4 bands, W m ×H m =80×80=6400 pixels;

[0104] The high-resolution full-color image described in step 1 is recorded as a tensor Its 3-mode expansion matrix is Indicates that the image has 1 band, W p ×H p =320×320=102400 pixels;

[0105] The resolution ratio between the high-resolution panchromatic image and the low-resolution hyperspectral image is s h , the resolution ratio between the high-resolution panchromatic image and the low-resolution multispectral image is s m ,Right now

[0106] Step 2: construct a fusion model among the low-resolution hyperspectral image, the low-resolution multispectral image and the high-resolution panchromatic image through the low-resolution hyperspectral image, the low-resolution multispectral image and the high-resolution panchromatic image described in step 1;

[0107] Preferably, the fusion model between the low-resolution hyperspectral image, the low-resolution multispectral image, and the high-resolution panchromatic image in step 2 is composed of a hyperspectral image fitting term, a multispectral image fitting term, a dynamic gradient group sparse regularization term, and a low-rank regularization term;

[0108] The hyperspectral image fitting term and multispectral image fitting term are obtained by fitting the low-resolution hyperspectral image H (3) , low-resolution multispectral image M (3) The spatial degradation relationship and spectral degradation relationship between the low-resolution hyperspectral image H and the high-resolution hyperspectral image H are constructed. (3) It is equivalent to a spatially downsampled version of a high-resolution hyperspectral image, a low-resolution multispectral image M (3) It is equivalent to a high-resolution hyperspectral image after spatial downsampling and spectral downsampling;

[0109] The hyperspectral image fitting term, multispectral image fitting term, dynamic gradient group sparse regularization term, and low-rank regularization term constitute a target energy function about a coefficient matrix.

[0110] The specific implementation of constructing the fusion model between the hyperspectral image, the multispectral image, and the panchromatic image in step 2 includes the following sub-steps:

[0111] Step 2.1: Low-resolution hyperspectral image H (3) It can be regarded as a high-resolution hyperspectral image F (3) A spatially downsampled version of:

[0112] H (3) =F (3) B h S h

[0113] Among them, the matrix is the spatial fuzzy matrix. It is a spatial downsampling operation used to reduce the spatial resolution of hyperspectral images.

[0114] Step 2.2: Low-resolution multispectral image M (3) It can be regarded as a high-resolution hyperspectral image F (3) The spatially and spectrally downsampled version:

[0115] M (3) =R m F (3) B m S m

[0116] in is the spectral response matrix of the multi-spectral instrument. is the spatial fuzzy matrix. It is a spatial downsampling operation used to reduce the spatial resolution of hyperspectral images.

[0117] Step 2.3: The high-resolution panchromatic image P and the high-resolution hyperspectral image F have the same spatial edge information. Their gradient difference should satisfy the group sparsity feature. The regularization term of dynamic gradient group sparsity can be established:

[0118] The high-resolution hyperspectral image is recorded as a tensor Its 3-mode expansion matrix is Indicates that the image has L h =93 bands, W p ×H p =320×320=102400 pixels;

[0119]

[0120] in, Indicates copying a high-resolution full-color image to L m =4 bands.

[0121] B in step 2.1 h Matrix, R in step 2.2 m , B m The matrix can be composed of a low-resolution hyperspectral image H (3) , low-resolution multispectral image M (3) , high-resolution full-color image P (3) To estimate, we use the estimation method proposed in the hyperspectral image fusion method called HySure.

[0122] Step 2.4: Use low-rank regularization to build a 3D tensor The regularization term of :

[0123] ||X|| *

[0124] Step 2.5: Combining the product model of the subspace matrix and the coefficient matrix to transform the high-resolution hyperspectral image into a coefficient matrix model of the high-resolution hyperspectral image in the low-dimensional subspace;

[0125] The product model of the subspace matrix and the coefficient matrix in step 2.5 is:

[0126]

[0127] Among them, × 3 represents the third mode product, is a subspace matrix, which is composed of vectors of D=10 pure spectral features. The subspace matrix is ​​directly obtained by image decomposition of low-resolution hyperspectral images. The image decomposition method we choose is vertex component analysis;

[0128] Tensor is the coefficient matrix, representing Each pixel is represented by the coefficients of a linear combination of the vector members in the subspace matrix, Represents pixels The corresponding coefficients are: i is the spatial horizontal coordinate of the pixel point, and j is the spatial vertical coordinate of the pixel point.

[0129] The coefficient matrix model of converting the high-resolution hyperspectral image into the high-resolution hyperspectral image in the low-dimensional subspace described in step 2.5 is:

[0130] F (3) =EX (3)

[0131] in, For tensors 3-mode expansion;

[0132] Step 2.6: Based on steps 2.1 to 2.5, establish a three-way fusion model:

[0133]

[0134] in represents the Frobenius norm, ||·|| 2，1 represents the L2,1 norm, ||·|| * represents the nuclear norm, λ m =0.01,λ φ = 0.01 and λ l =0.001 are the parameters for balancing each item.(3) is the two-dimensional matrix representation of the coefficients.

[0135] Step 3: Based on the fusion model described in step 2, solve the coefficient matrix X by alternating direction multiplication method (3) , and then obtain the high-resolution hyperspectral image F by multiplying the coefficient matrix with the subspace matrix (3) ;

[0136] The specific implementation of step 3 includes the following sub-steps:

[0137] Step 3.1: Introduce auxiliary variable O, satisfying O = X (3) B h ; Introduce auxiliary variable U, satisfying U=X (3) B m ; Introduce auxiliary variable V, satisfying V=X (3) ; Introduce auxiliary variable W, satisfying W=R m EV; introduce auxiliary variable Q, satisfying Q=X (3) The three-way fusion model is expressed as:

[0138]

[0139] stO=X (3) B h

[0140] U=X (3) B m

[0141] V=X (3)

[0142] W=R m EV

[0143] Q=X (3)

[0144] The augmented Lagrangian function of the three-way fusion model is expressed as:

[0145]

[0146] Among them, Y 1 , Y 2 , Y 3 , Y 4 and Y 5 is the scaled dual variable, represents the Frobenius norm, ||·|| 2，1 represents the L2,1 norm, ||·|| * represents the nuclear norm, λ m =0.01,λ φ= 0.01 and λ l =0.001 are the parameters for balancing each item. μ = 0.001 is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients. (3) It is a two-dimensional matrix representation of a low-resolution hyperspectral image. (3) It is a two-dimensional matrix representation of a low-resolution multispectral image. Indicates copying a high-resolution full-color image to L m =4 bands. Represents the gradient symbol. is the spatial fuzzy matrix, is the spatial fuzzy matrix, is the spatial downsampling operation, is the spatial downsampling operation, is the spectral response matrix of the multi-spectral instrument, is the subspace matrix. The optimization problem of the augmented Lagrangian function of the three fusion models can be decomposed into X subproblem, O subproblem, U subproblem, V subproblem, W subproblem, Q subproblem and Y subproblem.

[0147] Step 3.2: Solve the X subproblem:

[0148]

[0149] Among them, t in the upper right corner represents the number of iterations, Y 1 , Y 2 , Y 3 and Y 5 is the scale dual variable, represents the Frobenius norm, μ=0.001 is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients. is the spatial fuzzy matrix, is the spatial fuzzy matrix;

[0150] The solution to the above problem is:

[0151]

[0152] Step 3.3: Solve the O subproblem:

[0153]

[0154] Among them, Y 1 is the scale dual variable, represents the Frobenius norm, μ=0.001 is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients. (3)It is a two-dimensional matrix representation of a low-resolution hyperspectral image. is the spatial fuzzy matrix, is the spatial downsampling operation, is the subspace matrix.

[0155] Divide O into OS h and in Indicates that the matrix S h The selected pixel point, then the solution of subproblem O is:

[0156]

[0157]

[0158] Step 3.4: Solve the U-problem:

[0159]

[0160] Among them, Y 2 is the scale dual variable, represents the Frobenius norm, λ m =0.01 is the parameter for balancing each item. μ = 0.001 is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients. (3) It is a two-dimensional matrix representation of a low-resolution multispectral image. is the spatial fuzzy matrix, is the spatial downsampling operation, is the spectral response matrix of the multi-spectral instrument, is the subspace matrix.

[0161] Divide U into US m and in Indicates that the matrix S m For the selected pixel, the solution to the U subproblem is:

[0162]

[0163]

[0164] Step 3.5: Solve the V subproblem:

[0165]

[0166] Among them, Y 3 , Y 4 is the scale dual variable, represents the Frobenius norm, μ=0.001 is the penalty parameter.(3) is the two-dimensional matrix representation of the coefficients. is the spectral response matrix of the multi-spectral instrument, is the subspace matrix.

[0167] The solution to the above problem is:

[0168]

[0169] Step 3.6: Solve the W subproblem:

[0170]

[0171] Among them, Y 4 is the scale dual variable, represents the Frobenius norm, ||·|| 2，1 represents the L2,1 norm,,λ φ =0.01 is the parameter for balancing each item. μ = 0.001 is the penalty parameter. Indicates copying a high-resolution full-color image to L m =4 bands. Represents the gradient symbol. is the spectral response matrix of the multi-spectral instrument, is the subspace matrix.

[0172] Let G = WP, then the W subproblem becomes

[0173]

[0174] The W subproblem can be solved directly by the Vectorial Total Variation algorithm.

[0175] Step 3.7: Solve the Q subproblem:

[0176]

[0177] Among them, Y 5 is the scale dual variable, represents the Frobenius norm, ||·|| * represents the nuclear norm, λ l =0.001 is the parameter for balancing each item. μ = 0.001 is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients.

[0178] The Q subproblem can be solved directly by the singular value contraction algorithm.

[0179] Step 3.8: Solve the Y subproblem. This subproblem can be directly updated by the following formula:

[0180]

[0181]

[0182]

[0183]

[0184]

[0185] Among them, Y 1 , Y 2 , Y 3 , Y 4 and Y 5 is the scale dual variable, μ=0.001 is the penalty parameter. (3) is the two-dimensional matrix representation of the coefficients. is the spatial fuzzy matrix, is the spatial fuzzy matrix, is the spectral response matrix of the multi-spectral instrument, is the subspace matrix. O, U, V, W and Q are auxiliary variables.

[0186] Step 3.9: After iteratively solving the above sub-problems, we get the coefficient tensor According to step 2 Then we can get high-resolution hyperspectral images

[0187] The optimization solution process is shown in Table 1.

[0188] Table 1

[0189]

[0190] Step 4: Based on the above steps, the present invention uses the hyperspectral image Multispectral imagery Full color image The three data are fused to obtain a high-resolution hyperspectral image. In order to quantitatively and qualitatively evaluate the fused image, the present invention selects two hyperspectral fusion methods, CNMF and GSA, as comparison methods. The present invention uses these comparison methods to fuse the hyperspectral image and the multispectral image and compares the fusion results of the present invention. The present invention uses four quality evaluation indicators, ERGAS, SAM, RMSE, and PSNR. The quantitative analysis results are shown in Table 2. Bold indicates the best result. The visual comparison results are shown in Table 2. Figure 2-Figure 5 shown. Figure 2 This is the fusion result diagram of the comparison method CNMF. Figure 3 This is the fusion result diagram of the comparison method GSA. Figure 4 This is the fusion result diagram of the present invention, Figure 5 is the reference image. As you can see, Figure 2 and Figure 3 There is obvious blurring, and many details are blurred. Figure 4 Compare Figure 2 and Figure 3 It is much clearer. Take the small house in the upper right corner as an example. Figure 4 You can see the division of the grid, and Figure 2 and Figure 3 You can't see it. Figure 2 and Figure 3 , Figure 4 Closer to the reference image Figure 5 .

[0191] Table 2

[0192]

[0193] It can be seen that since the fusion method of the present invention utilizes the information in the full-color image, compared with the HSI / MSI fusion algorithm and the HSI / PAN fusion algorithm, the fusion result of the present invention has higher clarity, and each index is closest to the ideal value, which is superior to the comparative method in both qualitative and quantitative aspects.

[0194] It should be understood that parts not elaborated in detail in this description belong to the prior art.

[0195] It should be understood that the above description of the embodiments is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A hyperspectral fusion method based on dynamic gradient group sparsity and low-rank regularization, characterized in that: Step 1: Obtain a low-resolution hyperspectral image through a hyperspectral sensor, and collect a low-resolution multispectral image and a high-resolution panchromatic image of the same picture through a multispectral sensor; Step 2: construct a fusion model among the low-resolution hyperspectral image, the low-resolution multispectral image and the high-resolution panchromatic image through the low-resolution hyperspectral image, the low-resolution multispectral image and the high-resolution panchromatic image described in step 1; Step 3: Based on the fusion model described in step 2, the coefficient matrix X(3) is solved by the alternating direction multiplication method, and then the high-resolution hyperspectral image F is obtained by multiplying the coefficient matrix with the subspace matrix. (3) ; The low-resolution hyperspectral image in step 1 is recorded as a tensor Its 3-mode expansion matrix is Indicates that the image has L h Bands, W h ×H h pixels; The low-resolution multispectral image described in step 1 is recorded as a tensor Its 3-mode expansion matrix is Indicates that the image has L m Bands, W m ×H m pixels; The high-resolution full-color image described in step 1 is recorded as a tensor Its 3-mode expansion matrix is Indicates that the image has 1 band, W p ×H p pixels; The resolution ratio between the high-resolution panchromatic image and the low-resolution hyperspectral image is s h , the resolution ratio between the high-resolution panchromatic image and the low-resolution multispectral image is s m ,Right now The fusion model between the low-resolution hyperspectral image, the low-resolution multispectral image, and the high-resolution panchromatic image in step 2 is composed of a hyperspectral image fitting term, a multispectral image fitting term, a dynamic gradient group sparse regularization term, and a low-rank regularization term; The hyperspectral image fitting term and multispectral image fitting term are obtained by fitting the low-resolution hyperspectral image H (3) , low-resolution multispectral image M (3) The spatial degradation relationship and spectral degradation relationship between the low-resolution hyperspectral image H and the high-resolution hyperspectral image H are constructed. (3) It is equivalent to a spatially downsampled version of a high-resolution hyperspectral image, a low-resolution multispectral image M (3) It is equivalent to a high-resolution hyperspectral image after spatial downsampling and spectral downsampling; The hyperspectral image fitting term, multispectral image fitting term, dynamic gradient group sparse regularization term, and low-rank regularization term constitute a target energy function about a coefficient matrix; The specific implementation of constructing the fusion model between the hyperspectral image, the multispectral image, and the panchromatic image in step 2 includes the following sub-steps: Step 2.1: Low-resolution hyperspectral image H (3) It can be regarded as a high-resolution hyperspectral image F (3) A spatially downsampled version of: H (3) =F (3) B h S h Among them, the matrix is the spatial fuzzy matrix; It is a spatial downsampling operation used to reduce the spatial resolution of the hyperspectral image; Step 2.2: Low-resolution multispectral image M (3) It can be regarded as a high-resolution hyperspectral image F (3) The spatially and spectrally downsampled version: M (3) =R m F (3) B m S m in, is the spectral response matrix of the multi-spectral instrument; the matrix is the spatial fuzzy matrix; It is a spatial downsampling operation used to reduce the spatial resolution of the hyperspectral image; Step 2.3: The high-resolution panchromatic image P and the high-resolution hyperspectral image F have the same spatial edge information. Their gradient difference should satisfy the group sparse feature. The regularization term of dynamic gradient group sparse can be established: The high-resolution hyperspectral image is recorded as a tensor Its 3-mode expansion matrix is Indicates that the image has L h Bands, W p ×H p pixels; in, Indicates copying a high-resolution full-color image to L m Bands; B in step 2.1 h Matrix, R in step 2.2 m , B m The matrix can be composed of a low-resolution hyperspectral image H (3) , low-resolution multispectral image M (3) , high-resolution full-color image P (3) It was estimated using the estimation method proposed in the hyperspectral image fusion method named HySure; Step 2.4: Use low-rank regularization to build a 3D tensor The regularization term of : ||X|| * Step 2.5: Combining the product model of the subspace matrix and the coefficient matrix to transform the high-resolution hyperspectral image into a coefficient matrix model of the high-resolution hyperspectral image in the low-dimensional subspace; The product model of the subspace matrix and the coefficient matrix in step 2.5 is: Among them, ×3 represents the third mode product, is a subspace matrix, wherein the subspace matrix represents a vector of D pure spectral features, and the subspace matrix is ​​directly obtained by image decomposition of a low-resolution hyperspectral image, and the selected image decomposition method is vertex component analysis; Tensor is the coefficient matrix, representing Each pixel is represented by the coefficients of a linear combination of the vector members in the subspace matrix, Represents pixels The corresponding coefficients, i is the spatial horizontal coordinate of the pixel point, and j is the spatial vertical coordinate of the pixel point; The coefficient matrix model of converting the high-resolution hyperspectral image into the high-resolution hyperspectral image in the low-dimensional subspace described in step 2.5 is: F (3) =EX (3) in, For tensors 3-mode expansion; Step 2.6: Based on steps 2.1 to 2.5, establish a three-way fusion model: in represents the Frobenius norm, ||·|| 2,1 represents the L2,1 norm, ||·|| * represents the nuclear norm, λ m ,λ φ and λ l are the parameters of each balance; X (3) is the two-dimensional matrix representation of the coefficients; The specific implementation of step 3 includes the following sub-steps: Step 3.1: Introduce auxiliary variable O, satisfying O = X (3) B h ; Introduce auxiliary variable U, satisfying U=X (3) B m ; Introduce auxiliary variable V, satisfying V=X (3) ; Introduce auxiliary variable W, satisfying W=R m EV; introduce auxiliary variable Q, satisfying Q=X (3) ; The three-in-one fusion model is expressed as: s.t.O=X (3) B h U=X )3( B m V=X (3) W=R m EV Q=X (3) The augmented Lagrangian function of the three-way fusion model is expressed as: Among them, Y1, Y2, Y3, Y4 and Y5 are scaled dual variables. represents the Frobenius norm, ||·|| 2,1 represents the L2,1 norm, ||·|| * represents the nuclear norm, λ m ,λ φ and λ l are the parameters of each balance; μ is the penalty parameter; X (3) is the two-dimensional matrix representation of the coefficients; H (3) is a two-dimensional matrix representation of a low-resolution hyperspectral image; M (3) It is a two-dimensional matrix representation of a low-resolution multispectral image; Indicates copying a high-resolution full-color image to L m Bands; Represents the gradient symbol; is the spatial fuzzy matrix, is the spatial fuzzy matrix, is the spatial downsampling operation, is the spatial downsampling operation, is the spectral response matrix of the multi-spectral instrument, is the subspace matrix; the optimization problem of the augmented Lagrangian function of the three-in-one fusion model can be decomposed into X subproblem, O subproblem, U subproblem, V subproblem, W subproblem, Q subproblem, and Y subproblem; Step 3.2: Solve the X subproblem: Among them, t in the upper right corner represents the number of iterations, Y1, Y2, Y3 and Y5 are scale dual variables, represents the Frobenius norm, μ is the penalty parameter; X (3) is the two-dimensional matrix representation of the coefficients; is the spatial fuzzy matrix, is the spatial fuzzy matrix; The solution to the above problem is: Step 3.3: Solve the O subproblem: Among them, Y1 is the scale dual variable, represents the Frobenius norm, μ is the penalty parameter; X (3 is the two-dimensional matrix representation of the coefficients; H (3) It is a two-dimensional matrix representation of a low-resolution hyperspectral image; is the spatial fuzzy matrix, is the spatial downsampling operation, is the subspace matrix; Divide O into OS h and in Indicates that the matrix S h The solution to the O subproblem is: Step 3.4: Solve the U-problem: Among them, Y2 is the scale dual variable, represents the Frobenius norm, λ m is the parameter for balancing each item; μ is the penalty parameter; X (3) is the two-dimensional matrix representation of the coefficients; M (3) It is a two-dimensional matrix representation of a low-resolution multispectral image; is the spatial fuzzy matrix, is the spatial downsampling operation, is the spectral response matrix of the multi-spectral instrument, is the subspace matrix; Divide U into US m and in Indicates that the matrix S m For the selected pixel, the solution to the U subproblem is: Step 3.5: Solve the V subproblem: Among them, Y3 and Y4 are scale dual variables, represents the Frobenius norm, μ is the penalty parameter; X (3 is the two-dimensional matrix representation of the coefficients; is the spectral response matrix of the multi-spectral instrument, is the subspace matrix; The solution to the above problem is: Step 3.6: Solve the W subproblem: Among them, Y4 is the scale dual variable, represents the Frobenius norm, ||·|| 2,1 represents the L2,1 norm, λ φ is the parameter for balancing each item; μ is the penalty parameter; Indicates copying a high-resolution full-color image to L m Bands; Represents the gradient symbol; is the spectral response matrix of the multi-spectral instrument, is the subspace matrix; Let G = WP, then the W subproblem becomes The W subproblem can be solved directly using the Vectorial Total Variation algorithm; Step 3.7: Solve the Q subproblem: Among them, Y5 is the scale dual variable, represents the Frobenius norm, ||·|| * represents the nuclear norm, λ l is the parameter for balancing each item; μ is the penalty parameter; X (3) is the two-dimensional matrix representation of the coefficients; The Q subproblem can be solved directly by the singular value contraction algorithm; Step 3.8: Solve the Y subproblem; this subproblem can be directly updated by the following formula: Among them, Y1, Y2, Y3, Y4 and Y5 are scale dual variables, μ is the penalty parameter; X (3) is the two-dimensional matrix representation of the coefficients; is the spatial fuzzy matrix, is the spatial fuzzy matrix, is the spectral response matrix of the multi-spectral instrument, is the subspace matrix; O, U, V, W and Q are auxiliary variables; Step 3.9: After iteratively solving the above sub-problems, we get the coefficient tensor According to step 2 Then we can get high-resolution hyperspectral images

Images