High-order tensor spectral image super-resolution reconstruction method based on spectral information fusion

Abstract

The present invention relates to a high-order tensor spectral image super-resolution reconstruction method based on spectral information fusion, which is a method for realizing super-resolution reconstruction of hyperspectral images by weighted low Tensor-Train (TT) rank tensor analysis using weighted group sparsity and spectral unmixing. The present invention utilizes the high spatial-spectral correlation and non-local self-similarity of high-multispectral images, utilizes low TT rank tensor decomposition to approximate the recovery of block tensor data, and mines the non-local self-similarity contained in the block tensor data; weighted group sparsity is used as a regularization term to describe the spatial-spectral continuity of hyperspectral images; linear spectral decomposition based on weighted non-convex regularization is used as an effective spectral regularization to reduce spectral distortion. This method solves the problems of super-resolution reconstruction, high-multispectral fusion, noise removal, and image detail restoration, and its effectiveness and advancement are experimentally verified.

Description

Technical Field

[0001] The present invention relates to a method for super-resolution reconstruction of hyperspectral images, in particular to a method for super-resolution reconstruction of hyperspectral images by weighted low TT rank tensor analysis based on weighted group sparsity and spectral unmixing. Background Art

[0002] Hyperspectral (HS) images consist of images of a scene captured in dozens to hundreds of discrete spectral bands at specific frequencies. HS images possess extremely high spectral resolution and coverage, enabling precise identification of materials present in the scene. This facilitates characterization of the imaged scene and significantly improves performance in many applications, including remote sensing, surveillance, target detection, military applications, and tracking. However, despite their high spectral resolution, HS images often suffer from severe limitations in spatial resolution. HS imaging systems require a large number of exposures to simultaneously acquire multiple bands within a narrow spectral window. To ensure a sufficient signal-to-noise ratio (SNR), long exposure times are often required, often at the expense of spatial resolution. Consequently, the spatial resolution of HS images is typically low due to the large number of mixed pixels present in low-resolution HS images, which, to a certain extent, limits their widespread application. Improving the spatial resolution of HS images has therefore become an important topic and has received increasing attention in recent years.

[0003] Simply increasing the spatial resolution of the image sensor is not effective for HS imaging because the average number of photons reaching the sensor will further decrease, resulting in a lower signal-to-noise ratio. Therefore, improving the resolution through post-processing is a better option. Furthermore, multispectral (MS) imaging sensors are capable of capturing images with increasingly higher spatial resolutions. Therefore, a low-resolution HS image can be fused with a high-resolution (HR) MS image captured of the same scene to reconstruct a low-resolution HS image. This process, known as HS and MS image fusion, has attracted widespread attention. In fact, new HS imagers are also planned to include MS image sensors. This means that high-spectral-resolution HS images and high-spatial-resolution MS images can be captured simultaneously, and it is expected that more data sources for HS and MS image fusion will emerge.

[0004] The fusion of HS and MS images has seen a lot of fruitful research in recent years. The Bayesian framework is a common approach for fusing low-resolution HS and HRHS images. Such methods typically establish the posterior distribution of the desired HRHS image based on prior knowledge and observation models. Matrix factorization is also a common approach for HS-MS image fusion. Matrix factorization-based fusion methods typically first unfold the target HR-HS image into a matrix and then decompose it into a basis matrix and a coefficient matrix, where the basis matrix and the coefficient matrix are extracted from the low-resolution HS and HR-MS images, respectively. While matrix factorization-based methods have achieved impressive performance in HS and MS fusion, they still suffer from inherent drawbacks. All matrix factorization methods require unfolding the three-dimensional data structure into a matrix, which can destroy the spatial structure of the data, making it difficult to fully exploit the spatial-spectral correlations of the HS image. Compared to low-rank matrix factorization, low-rank tensor-based methods can better maintain and exploit the spatial structure of HS images and have therefore been applied to some HS image restoration problems. To preserve the inherent spatial structure of HS images, unlike traditional matrix factorization-based methods, tensor decomposition-based methods model the high spatial-spectral correlations in HR-HS images as a third-order tensor. Despite achieving good performance, these approaches fail to fully exploit other valid prior knowledge in HS images. Over the past decade, deep learning-based methods have achieved impressive performance in many computer vision tasks and have been introduced to address the fusion of HS and MS. While these methods offer promising performance, they typically require large amounts of training data, which is often unrealistic for HS image restoration. Summary of the Invention

[0005] In response to the above technical deficiencies, the present invention aims to provide a method based on weighted non-local low TT rank tensor decomposition and spectral unmixing to deal with the problem of hyperspectral and multispectral fusion and achieve spectral image super-resolution. In order to fully utilize the high spatial-spectral correlation and non-local self-similarity of hyper- and multispectral images, low TT rank tensor decomposition is used to approximate the recovery of block tensor data and mine the non-local self-similarity contained in the block tensor data; group sparsity is used as a regularization to describe the spatial-spectral continuity of hyperspectral images; linear spectral decomposition based on non-convex regularization is used as an effective spectral regularization to reduce spectral distortion. Based on the above prior knowledge, a unified optimization model is proposed to describe the problem of hyperspectral and multispectral image fusion, and an efficient algorithm using alternating direction multiplication operators to solve the model is designed. Compared with existing new work, this method has achieved better results in hyperspectral and multispectral fusion problems, and has better reconstruction performance and better robustness in hyperspectral image super-resolution.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] The high-order tensor spectral image super-resolution reconstruction method based on spectral information fusion performs the following steps on the basis of the prior initial model, and then repeatedly iterates and optimizes the solution to obtain a high-order tensor spectral image super-resolution reconstruction model based on spectral information fusion, thereby realizing high-order tensor spectral image super-resolution reconstruction, including:

[0008] Step 1: Add a weighted low TT rank high-order tensor regularization term to exploit the high spatial-spectral correlation and non-local self-similarity of hyper-multispectral images and mine the non-local self-similarity contained in the block tensor data;

[0009] Step 2: Add a weighted group sparse regularization term to describe the spatial-spectral continuity of the hyperspectral image;

[0010] Step 3: Add a weighted spectral unmixing regularization term and use the linear spectral decomposition of weighted non-convex regularization as spectral regularization to reduce spectral distortion and achieve high-multispectral information fusion.

[0011] The model is used to fuse hyperspectral images and multispectral images, perform super-resolution reconstruction of spectral images, and effectively improve the super-resolution performance of spectral images.

[0012] The prior super-resolution reconstruction model is a traditional prior modeling with a prior information regularization term, which is denoted as:

[0013]

[0014] in, is the spatial domain fidelity term, is the spectrum domain fidelity term, is the observed low-resolution HSI tensor data, It is the high-resolution MSI tensor data obtained by the multispectral imaging sensor in the same scene. is the high-resolution HSI tensor data to be restored, B is the sampling operator, S is the blur operator, is the degradation operator in the spectral domain, is prior information.

[0015] The high-order tensor spectral image super-resolution reconstruction model based on spectral information fusion is:

[0016]

[0017] in, is the weighted low TT rank high-order tensor regularization term, is the weighted group sparsity as a regularization term, is the weighted spectral unmixing regularization term for hyperspectral and multispectral, λ ris the low-rank weight factor, λ t is the group sparse weight factor, is the weight factor of the image to be restored, is the weight factor of the input image, W r is the low-rank weight matrix, W t is the group sparse weight matrix, is the weight matrix of the image to be restored, is the weight matrix of the input image, U1 is the richness of the image to be restored, and U2 is the richness of the input image.

[0018] The weighted low TT rank tensor term and the non-local self-similarity are achieved by dividing the image into a set of image blocks, finding a set of blocks that are most similar to the selected blocks and stacking them to form 4D non-local self-similar blocks.

[0019] The weighted group sparsity term is used to describe the smoothness of spectral images in different spectral bands and to mine the row spatial structure differences and shared sparsity structures of image blocks in each spectral band.

[0020] The weighted spectral unmixing adopts non-convex regularized linear spectral decomposition as spectral regularization to reduce spectral distortion.

[0021] During the iterative optimization and solving process, when the error between two adjacent image restoration result data is within a threshold range, it is determined that the image reconstruction up to the current round meets the convergence requirement, and the iteration is stopped.

[0022] The error function is:

[0023]

[0024] Where K is the number of iterations, is the high-resolution HSI tensor data to be restored, For the observed low-resolution HSI tensor data, when the error function is within the set threshold, the algorithm converges and stops timing.

[0025] The iterative optimization solution adopts the ADMM algorithm.

[0026] The present invention has the following beneficial effects and advantages:

[0027] 1. The method of the present invention uses weighted low TT rank tensor decomposition to approximately recover block tensor data and mine the non-local self-similarity contained in the block tensor data.

[0028] 2. The method of the present invention adopts a weighted group sparse regularization term to mine the row spatial structure differences and shared sparsity structures of image blocks in each spectral segment.

[0029] 3. The method of the present invention adopts non-convex regularized linear spectral decomposition as an effective spectral regularization term to reduce spectral distortion.

[0030] 4. The method of the present invention is superior to existing algorithms in terms of hyperspectral and multispectral fusion, and has better super-resolution reconstruction performance and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is the overall framework diagram of this method;

[0032] Figure 2 This is the 38th spectral segment of the high-resolution hyperspectral image reconstructed by different methods on the Pavia University dataset and the corresponding error map.

[0033] Figure 3 This is the 38th spectral segment of the high-resolution hyperspectral image reconstructed by different methods on the Washington DC Mall dataset and the corresponding error map. DETAILED DESCRIPTION

[0034] The present invention is further described in detail below with reference to the embodiments.

[0035] In this paper, we propose a unified optimization model to describe the hyperspectral and multispectral image fusion problem and design an efficient algorithm to solve the model using the alternating direction multiplication operator.

[0036] In order to improve the spatial resolution of hyperspectral images, this paper proposes an effective fusion method for hyperspectral images and multispectral images by using low-rank tensor analysis and spectral unmixing ideas.

[0037] A high-order tensor spectral image super-resolution reconstruction method based on spectral information fusion consists of four steps: a super-resolution step to ensure the fidelity of observed data with the original data; a weighted low-TT rank tensor decomposition to approximate the recovery of block tensor data and exploit the non-local self-similarity inherent in the block tensor data; a weighted group sparsity regularization to describe the spatial-spectral continuity of the HS image; and a linear spectral decomposition based on weighted non-convex regularization as an effective spectral regularization to reduce spectral distortion. The proposed model solves the problem of hyperspectral and multispectral fusion, enabling super-resolution reconstruction of spectral images and effectively improving the performance and robustness of spectral image super-resolution. Hyperspectral and multispectral image data are used as tensor inputs to the model, and observation data are generated by setting different sampling rates and blur operators. After each image reconstruction, the error between the recovered data is calculated twice. The convergence of the image reconstruction is determined by whether the error function meets a threshold requirement.

[0038] The error function is

[0039]

[0040] Where K is the number of iterations, To recover data, For observation data.

[0041] 1. Problem Modeling

[0042] The high-resolution HSI to be restored is tensor Where W, H, and S represent the width, height, and spectral mode of the high-resolution HSI tensor data, respectively. The observed low-resolution HSI tensor data is recorded as The spectral bands are consistent with high-resolution HSI, but the space is downsampled. Represents the high-resolution MSI obtained by the multispectral imaging sensor in the same scene. The mathematical relationship between the above three is described as

[0043]

[0044] in The representative spatial domain degradation operators include the downsampling operator B, the blur operator S, and the spectral domain degradation operator Right now:

[0045]

[0046] Because the above problem is an underdetermined problem, the solution of formula (2) is not unique. The prior information of is embedded into the model as a regular term to constrain the solution space, which is recorded as:

[0047]

[0048] in, is the spatial domain fidelity term, is the spectrum domain fidelity term, is the observed low-resolution HSI tensor mode 3 data, It is the high-resolution MSI tensor mode 3 data obtained by the multispectral imaging sensor in the same scene. is the high-resolution HSI tensor mode 3 data to be restored, B is the sampling operator, S is the blur operator, is the degradation operator in the spectral domain, is prior information.

[0049] So far, we have obtained the model of spectral image super-resolution reconstruction problem.

[0050] 1.1. Weighted Low TT Rank High-Order Tensor Regularization

[0051] In recent years, low-rank priors have shown their effectiveness and superiority in image processing tasks. Assume: represents the low-rank prior of the image data. Then the problem model can be written as

[0052]

[0053] Among them, λ r is the low-rank weight factor.

[0054] The non-local similarity of HSI describes the fact that each 3D block of HSI has many similar blocks in the nearby space. Usually, in order to exploit the self-similarity inherent in the image, HSI is first segmented into overlapping 3D blocks, and then the non-local similar blocks are clustered using the k-means algorithm. Non-local similar blocks simultaneously exploit the spectral and spatial low-rank properties of HSI. Furthermore, the low-rank tensor super-resolution reconstruction model based on non-local blocks can be written as

[0055]

[0056] Where, k is the order of non-local similarity blocks, It is an operator that takes similar blocks in the spectral image space domain.

[0057] For high-order tensor data processing, traditional tensor decomposition is limited in performance in tensor approximation due to computational NP-hardness and pattern imbalance. This method is based on the TT decomposition with better pattern balance. For a given tensor Its weighted low TT rank norm can be defined as:

[0058]

[0059] Where W r is a low-rank weight matrix, Is a diagonal matrix whose elements are n-order matrices The singular values ​​of α n is the weight of each n-order matrix, defined as in

[0060] Based on this weighted low TT rank norm, the relaxed form of formula (5) can be expressed as

[0061]

[0062] 1.2. Weighted Group Sparse Regularization

[0063] In addition to spatial spectral correlation and non-local self-similarity, It also shows the continuity of the spatial spectrum, which can be represented by the total variation (TV) regularization term. The TV regularization term is widely used to explore the spatial piecewise smooth structure to solve the HS image restoration task. Considering that the HS image also has a strong local smooth structure along its spectral mode, 3D-TV is usually used to simulate However, the image along the spectral pattern usually shows a strong shared structure sparsity. In terms of describing the smoothness of spectral images in different spectral bands, mining the spatial structure differences of image blocks in each spectral band and the shared sparsity structure, the group sparsity regularization term is more reasonable and effective than the traditional TV regularization term. For a given tensor Its weighted group sparse regularization term is

[0064]

[0065] Among them, (i,j,:) is the pixel position, is the weight in the x direction, is the weight in the y direction, and D is two differential operators, ie, D x and D y , calculate the difference along the width and height of the space.

[0066] Utilizing the weighted group sparsity in the spatial domain, the proposed model formula (7) can be rewritten as

[0067]

[0068] Among them, λ t is the group sparse weight factor, W t is the group sparse weight matrix.

[0069] 1.3. Weighted spectral unmixing regularization term

[0070] The above model (8) only considers spatial domain resolution enhancement. This processing scheme is prone to spectral distortion. Spectral unmixing has become an important spectral regularization method to reduce spectral distortion. The sparse spectral unmixing of a given matrix X can be expressed as

[0071]

[0072] Among them, E is the end member set, λ is the weight factor, and U is the image richness.

[0073] Its non-convex relaxation formula based on the weighted MCP penalty function can be written as

[0074]

[0075] Among them, W is the weight matrix.

[0076] By utilizing the above-mentioned prior regularization terms in the spatial domain and spectral domain, we propose a high-order tensor spectral image super-resolution reconstruction model based on spectral information fusion, namely

[0077]

[0078] in, is the spatial domain fidelity term, is the spectrum domain fidelity term, is the weighted low TT rank high-order tensor regularization term, is the weighted group sparsity as a regularization term, is the weighted spectral unmixing regularization term for hyperspectral and multispectral, λ r is the low-rank weight factor, λ t is the group sparse weight factor, is the weight factor of the image to be restored, is the weight factor of the input image, W r is the low-rank weight matrix, W t is the group sparse weight matrix is the weight matrix of the image to be restored, is the weight matrix of the input image, U1 is the richness of the image to be restored, and U2 is the richness of the input image.

[0079] 2. Algorithm solution

[0080] In order to solve the coupling problem in the variable solution process, auxiliary variables are introduced to the corresponding variables.

[0081]

[0082] Furthermore, we have its Lagrangian function form

[0083]

[0084] Update the variables in sequence.

[0085] renew

[0086]

[0087] This problem is a strongly convex problem and its solution can be obtained by forcing its derivative. is zero, that is

[0088]

[0089] in

[0090]

[0091] C2=BS(BS) T

[0092]

[0093] Update Z n :

[0094]

[0095] Closed-form solution

[0096]

[0097] renew

[0098]

[0099] make The closed-form solution for the fiber is calculated by the following formula

[0100]

[0101] renew

[0102]

[0103] Linear system solving

[0104]

[0105] renew

[0106]

[0107] Closed-form solution

[0108]

[0109] Update U1, U2:

[0110]

[0111] Closed-form solution

[0112]

[0113] The same logic applies

[0114]

[0115] Closed-form solution

[0116]

[0117] Update V1, V2:

[0118]

[0119] Closed-form solution

[0120]

[0121] The same logic applies

[0122]

[0123] Closed-form solution

[0124]

[0125] Update Lagrange multipliers

[0126]

[0127]

[0128] This method was experimented on the public hyperspectral image datasets of Pavia University and Washington DC Mall, and compared with representative methods, such as Figure 2 and Figure 3 As shown in Figure 3, the hyperspectral image reconstructed by our method is more detailed. The blue error image of our method is bluer and smoother, indicating that our reconstruction error is smaller.

[0129] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims

1. A high-order tensor spectral image super-resolution reconstruction method based on spectral information fusion, characterized in that: Based on the prior initial model, the following steps are performed, and then iterative optimization is performed to obtain a high-order tensor spectral image super-resolution reconstruction model based on spectral information fusion, thereby realizing high-order tensor spectral image super-resolution reconstruction, including: Step 1: Add a weighted low TT rank high-order tensor regularization term to exploit the high spatial-spectral correlation and non-local self-similarity of hyper-multispectral images and mine the non-local self-similarity contained in the block tensor data; this is done by segmenting the image into a set of image blocks, finding a set of blocks that are most similar to the selected block, and stacking them to form a 4D non-local self-similar block; Step 2: Add a weighted group sparse regularization term to describe the spatial-spectral continuity of the hyperspectral image; Step 3: Add a weighted spectral unmixing regularization term and use the weighted non-convex regularized linear spectral decomposition as spectral regularization to reduce spectral distortion and achieve high-multispectral information fusion; The high-order tensor spectral image super-resolution reconstruction model based on spectral information fusion is: in, is the weighted low TT rank high-order tensor regularization term, is the weighted group sparse regularization term, is the weighted spectral unmixing regularization term for hyperspectral and multispectral, λ r is the low-rank weight factor, λ t is the group sparse weight factor, is the weight factor of the image to be restored, is the weight factor of the input image, W r is the low-rank weight matrix, W t is the group sparse weight matrix, is the weight matrix of the image to be restored, is the weight matrix of the input image, U1 is the richness of the image to be restored, and U2 is the richness of the input image.

2. The high-order tensor spectral image super-resolution reconstruction method based on spectral information fusion according to claim 1 is characterized in that: The high-order tensor spectral image super-resolution reconstruction model based on spectral information fusion is used for the fusion of hyperspectral images and multispectral images, and performs super-resolution reconstruction on spectral images, thereby effectively improving the super-resolution performance of spectral images.

3. The high-order tensor spectral image super-resolution reconstruction method based on spectral information fusion according to claim 1 is characterized in that: The a priori initial model is a traditional a priori modeling with a priori information regularization term, which is denoted as: in, is the spatial domain fidelity term, is the spectrum domain fidelity term, is the observed low-resolution HSI tensor data, It is the high-resolution MSI tensor data obtained by the multispectral imaging sensor in the same scene. is the high-resolution HSI tensor data to be restored, B is the sampling operator, S is the blur operator, is the degradation operator in the spectral domain, is the prior information.

4. The high-order tensor spectral image super-resolution reconstruction method based on spectral information fusion according to claim 1 is characterized in that: The weighted group sparse regularization term is used to describe the smoothness of spectral images in different spectral bands and to mine the differences in row spatial structures and shared sparsity structures of image blocks in each spectral band.

5. The high-order tensor spectral image super-resolution reconstruction method based on spectral information fusion according to claim 1 is characterized in that: The weighted spectral unmixing regularization term adopts non-convex regularized linear spectral decomposition as spectral regularization to reduce spectral distortion.

6. The high-order tensor spectral image super-resolution reconstruction method based on spectral information fusion according to claim 1 is characterized in that: During the iterative optimization and solving process, when the error between two adjacent image restoration result data is within a threshold range, it is determined that the image reconstruction up to the current round meets the convergence requirement, and the iteration is stopped.

7. The high-order tensor spectral image super-resolution reconstruction method based on spectral information fusion according to claim 6 is characterized in that: The error function is: Where K is the number of iterations, is the high-resolution HSI tensor data to be restored, For the observed low-resolution HSI tensor data, when the error function is within the set threshold, the algorithm converges and stops timing.

8. The high-order tensor spectral image super-resolution reconstruction method based on spectral information fusion according to claim 1 is characterized in that: The iterative optimization solution adopts the ADMM algorithm.

Images