A Hyperspectral Video Reconstruction Method Based on Multi-Task Blind Compressive Sensing

Abstract

The present invention discloses a hyperspectral video reconstruction method based on multi-task blind compressed sensing, comprising the following steps: 1) inputting L frames of compressed measurement #imgabs0# in P frames of compressed hyperspectral video, where P≥L, and L>1, N x , N y N is the length and width of the spatial dimension of the hyperspectral video frame to be restored. s is the number of channels in the spectral dimension; 2) L compressed measurements all extract overlapping image blocks of size d×d; 3) In the multi-task non-parametric Bayesian model, the posterior probability formula of the dictionary and dictionary-related parameters is derived through the relationship between the conjugate prior and the likelihood function, and Gibbs sampling updates the dictionary atoms and dictionary correlation coefficients in turn; 4) The reconstructed image blocks are represented by a sparse combination of dictionary atoms to reconstruct the L frames of hyperspectral video obtained in this iteration. After multiple iterations of steps 3‑4, the last reconstruction result is output. 5) Based on the idea of sliding windows, after multiple iterations of steps 1‑4, the reconstruction results of all frames in the hyperspectral video are obtained, and the repeated reconstruction results of the frames are processed by the mean, and finally P frames of hyperspectral video are obtained. The present invention does not need to learn dictionary atoms from an external database first, and is suitable for the problem of reconstruction from scratch when the existing training data set is not available.

Description

Technical Field

[0001] The present invention belongs to the field of compressed hyperspectral video reconstruction, and particularly relates to a hyperspectral video reconstruction method based on multi-task blind compressive sensing. Background Art

[0002] Research shows that the spectral characteristics of hyperspectral imaging can improve the performance of computer vision tasks such as tracking, recognition, classification, and segmentation. Currently, hyperspectral imaging technology has been widely applied in fields such as biomedical imaging, remote sensing, and astronomy. A hyperspectral image has n channels in the third dimension, i.e., the spectral dimension, and each channel stores the information of the image in the corresponding spectral band. Therefore, a hyperspectral image captures three-dimensional data, i.e., a two-dimensional vector array, which includes the spectral information at each spatial position. A hyperspectral video adds a time dimension based on the hyperspectral image. Based on the compressive sensing theory, compressed hyperspectral video reconstruction extends snapshot spectral imaging to the time domain. Inspired by the compressive sensing theory, using the inversion algorithm of compressive sensing to reconstruct hyperspectral video has become a hot research issue currently.

[0003] The traditional Nyquist sampling theorem states that if one wants the sampled digital signal to completely retain the information in the original signal, the sampling frequency must be greater than twice the highest frequency in the signal. However, in 2004, scientists such as E.J. Candes, J. Romberg, T. Tao, and D.L. Donoho proved that if a signal is sparse, then it can be reconstructed and recovered from sampling points far lower than the requirements of the Nyquist sampling theorem. In 2007, several scientists formally proposed the concept of compressive sensing. The central idea of compressive sensing is to reconstruct and recover a high-dimensional signal from a low-dimensional signal on the basis that the signal is compressible. The compressive sensing theory can greatly reduce the sampling frequency of the signal. Currently, compressive sensing has been widely applied in various fields, such as data encryption, medical imaging, spectral imaging, and video interpolation.

[0004] In previous studies, analytical sparsification methods such as wavelet bases and discrete cosine transform (DCT) have been used quite a lot; in recent years, scientists have focused on the problem of sparse representation of signals through redundant dictionaries with stronger expressive ability learned from data, and have proved that replacing basis functions with overcomplete dictionaries is very effective in signal sparse representation. Compared with traditional compressive sensing, in the framework of blind compressive sensing, the orthogonal basis is unknown, that is, both the sparsification matrix and the sparsity coefficient are learned during the training process. This means that in the training process, the blind compressive sensing framework needs to estimate both the orthogonal basis and the orthogonal basis coefficient simultaneously.

[0005] It can be seen that there is a serious data loss problem in the blind compressive sensing framework. Multi-task learning can expand the training set by effectively exchanging information between tasks. By sharing key information between tasks, multi-task learning enables multiple related tasks to be jointly learned. The application scenarios of multi-task learning are very rich, such as sharing hidden nodes in neural networks, sharing priors in hierarchical Bayesian models, sharing common structures in predictor spaces, etc. In the reconstruction of compressed hyperspectral videos, multi-task learning is introduced to simultaneously reconstruct L frames in the compressed hyperspectral video into L frames in the hyperspectral video, and the problem of data loss is compensated by sharing key information between multiple tasks. This is a new attempt. Summary of the Invention

[0006] To solve the above technical problems, the present invention provides a hyperspectral video reconstruction method based on multi-task blind compressive sensing. Assume that there are P frames in the compressed hyperspectral video. First, take L frames from the compressed hyperspectral video, where P≥L>1; divide each of the L compressed measurements into N overlapping blocks; then use a multi-task non-parametric Bayesian dictionary learning model to simultaneously estimate the dictionary and the dictionary correlation coefficient for the L groups of input data, and use Gibbs sampling to estimate the parameters; obtain the reconstruction results of the L frames after multiple iterations; based on the idea of a "sliding window", for the P-frame compressed hyperspectral video, reconstruct L frames in each iteration, the moving step of the sliding window is 1, and the above steps need to be iterated P-L+1 times in total, and the repeated reconstruction results of the frames are processed by the mean value to finally obtain the reconstruction results of the P-frame compressed hyperspectral video.

[0007] The technical solution adopted by the present invention to solve its technical problems is: a hyperspectral video reconstruction method based on multi-task blind compressive sensing, and its specific operation steps are as follows:

[0008] Step 1) Obtain L frames from the P-frame compressed hyperspectral video, where P≥L>1, and the compressive sensing formula for reconstructing the hyperspectral video is expressed as {y t} t＝1，L =A{x t} t＝1，L +{∈ t} t＝1，L , where t represents the number of frames, represents the compressed measurement obtained by using a CASSI (Coded Aperture Snapshot Spectral Imager) fast camera; A represents the observation matrix, and based on the compression principle of the CASSI fast camera, the observation matrix A represents a binary coded aperture; {x t} t＝1，L represents the reconstructed hyperspectral video with dispersion added to each frame, that is, the result of each frame in the reconstructed hyperspectral video moving channel by channel along the spectral dimension; N x ​and N y are the length and width of the spatial dimension of the reconstructed hyperspectral video frame respectively, and N s is the number of channels in the spectral dimension; {∈ t} t＝1，L represents the reconstruction noise;

[0009] Step 2) Extract N overlapping image patches of size d×d from each of the L compressed measurements and reshape them into column vectors of size d 2 ×1, where d represents the size of the image patch, m = d 2 , and input the L sets of image patches into the multi-task nonparametric Bayesian dictionary learning model;

[0010] Step 3) In the multi-task nonparametric Bayesian dictionary learning model, assume that the dictionary follows a multivariate Gaussian distribution with mean 0 and covariance related to the spatial and spectral bands where the pixels are located, M = N s d 2 and m << M; the dictionary weight coefficient s ti follows a multivariate Gaussian distribution with precision , follows a Gamma distribution; the dictionary sparse coefficient z tik follows a Bernoulli distribution with precision π tk , π tk follows a Beta distribution; the reconstruction noise ∈ ti follows a multivariate Gaussian distribution with precision , follows a Gamma distribution; the multi-task nonparametric Bayesian dictionary learning model realizes collaborative work among tasks by sharing dictionaries and hyperparameters among tasks, and the inference of dictionary correlation coefficients in each task is carried out independently; the posterior probability formula of each parameter is derived through the relationship between the conjugate prior and the likelihood function, and Gibbs sampling is used to sequentially update all the parameters to be estimated, and the parameters to be estimated are the dictionary D, the dictionary weight coefficient s ti , the dictionary sparse coefficient z tik , the precision of the dictionary weight coefficient the precision π of the dictionary sparse coefficient tk , the noise precision

[0011] Step 4) Reconstruct the image patches from the dictionary D, the dictionary weight coefficient s ti , and the dictionary sparse coefficient z tik obtained by Gibbs sampling. The synthesis formula of the image patch is x ti = D(s ti ·z ti), · represents the Hadamard product; restore the image block and average the overlap of the graphic block, and then move the reconstruction result in the opposite direction along the spectral dimension channel by channel to obtain L frames of reconstructed hyperspectral video; wherein the reverse movement refers to the movement in the opposite direction of the dispersion process in step 1); after multiple iterations of steps 3)-4), the last reconstruction result is output;

[0012] Step 5) Based on the idea of ​​sliding window, for P-frame compressed hyperspectral video, L-frame hyperspectral video can be reconstructed in each round of iteration from step 1) to step 4). After multiple iterations of steps 1)-4), the reconstruction results of all frames in the compressed hyperspectral video are obtained, and the repeated reconstruction results of the frames are averaged to finally obtain a complete hyperspectral video.

[0013] Compared with the prior art, the present invention adopts the above technical solution and has the following beneficial effects:

[0014] This technical solution is mainly used for the problem of de novo reconstruction when the existing training data set is unavailable. In the framework of blind compressed sensing, it is necessary to estimate the dictionary atoms and dictionary correlation coefficients at the same time, which has a serious data missing problem. Multi-task learning is used to make up for the data missing problem in the blind compressed sensing framework. In this technical solution, dictionary atoms and hyperparameters are shared between multiple tasks to achieve the effect of expanding the training set. During the training process, a common dictionary is learned, and data from all tasks contribute to the inference of common dictionary atoms, realizing the sharing of key information between tasks. For a single task, the inference of the dictionary coefficients of the task is carried out independently, allowing variability between different frames and adapting to the inherent temporal redundancy of video sequences. The parameter inference process adopts a non-parametric Bayesian estimation strategy, which can self-learn the number of dictionary atoms and sparsity regularization parameters involved in reconstruction based on the measured data; it can be seen that the non-parametric Bayesian estimation strategy has the advantages of reliable generalization, no parameter adjustment and automatic determination of model complexity. This technical solution does not require first training from an external database to obtain a complete dictionary, but instead learns the dictionary in situ based on the current compressed measurement and expands the training set through multi-task learning. It is suitable for compressed hyperspectral video reconstruction problems where existing datasets are not available for training. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is the overall flow chart of the method of the present invention.

[0016] Figure 2 It is a sampling flow chart of the method of the present invention.

[0017] Figure 3 It is a probability model diagram of the method of the present invention.

[0018] Figure 4 It is a grayscale representation of the compressed measurement and reconstruction results of the method of the present invention. DETAILED DESCRIPTION

[0019] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings.

[0020] The hyperspectral video reconstruction method based on multi-task blind compressive sensing of the present invention includes the following steps:

[0021] Step 1) Observation model

[0022] Obtain L frames in the P-frame compressed hyperspectral video, where P≥L>1. The compressive sensing formula for reconstructing the hyperspectral video is expressed as {y t} t＝1，L =A{x t} t＝1，L +{∈ t} t＝1，L , where t represents the number of frames, represents the compressive measurement obtained using a CASSI (Coded Aperture Snapshot Spectral Imager) fast camera; A represents the observation matrix. Based on the compression principle of the CASSI fast camera, the observation matrix A represents a binary coded aperture; {x t} t＝1，L represents the hyperspectral video to be reconstructed with a dispersion process added to each frame, that is, the result of each frame in the reconstructed hyperspectral video moving channel by channel along the spectral dimension; N x and N y are the length and width of the spatial dimension of the reconstructed hyperspectral video frame respectively, and N s is the number of channels in the spectral dimension; {∈ t} t＝1，L represents the reconstruction noise.

[0023] Step 2) includes the following steps:

[0024] Extract N overlapping image patches of size d×d from each of the L compressive measurements and reshape them into column vectors of size d 2 ×1. The scanning step is step, 1≤step≤d, that is, the number of image patches extracted in each group N is equal to Input the L groups of image patches into the multi-task non-parametric Bayesian dictionary learning model; where d represents the size of the image patch, and m = d 2 .

[0025] Step 3) Core model - multi-task non-parametric Bayesian dictionary learning model

[0026] Input the L groups of data in Step 2). According to the compressive sensing formula y ti =A i D(sti ·z ti ) + ∈ ti , the image block synthesis formula is x ti = D(s ti ·z ti ), A i represents the binary coded aperture at the corresponding position of the image block x ti , s ti represents the dictionary weight coefficient, z ti represents the dictionary sparse coefficient, ∈ ti represents the reconstruction noise, and · represents the Hadamard product. In the framework of blind compressive sensing, the dictionary D and the dictionary correlation coefficient are estimated simultaneously from L compressive measurements. For a single task, the blind compressive sensing non-parametric Bayesian dictionary learning model - BPFA (Beta Process Factor Analysis) is used. BPFA uses the beta process and the Bernoulli process as the priors for non-parametric Bayesian dictionary learning, representing image blocks as sparse combinations of dictionary elements, and approximating the posterior probability of parameters through Gibbs sampling. In the non-parametric Bayesian dictionary learning model for multi-task blind compressive sensing, the Gibbs sampling formula of the parameters is generalized to the time domain for the reconstruction of compressed hyperspectral videos.

[0027] Step 3-1) The dictionary atoms d in the dictionary k follow a multivariate Gaussian distribution: d k ~N(0, Ω), where M = N s d 2 and m << M, and Ω is the covariance matrix related to the spatial position and spectral band in the image block where the pixel is located. The specific calculation method of Ω is as follows:

[0028] j, j′ represent any two pixels in the image block , represents the spatial position of pixel j in the image block, λ j represents the spectral position where pixel j is located; the distance between pixels j and j′ is calculated according to the formula, and the distance calculation formula is The formula for calculating Ω(j, j′) of pixels j and j′ is Ω(j, j′) = exp[-l(j, j′) / 2σ 2 .

[0029] The sparse coefficient z of the dictionary tik follows a Bernoulli distribution: z tik ~Bernoulli(π tk ), and the precision π tk follows a Beta distribution: The weight coefficient s of the dictionary ti follows a multivariate Gaussian distribution: where \(I\) K denotes the \(K\times K\) identity matrix, and the precision follows a Gamma distribution: The reconstruction noise \(\in\) ti follows a multivariate Gaussian distribution with precision : where \(I\) m denotes the \(m\times m\) identity matrix, and the precision follows a Gamma distribution:

[0030] Step 3 - 2) The posterior probability density formula of the non - parametric Bayesian dictionary learning model for multi - task blind compressive sensing is:

[0031]

[0032]

[0033] Based on the prior distribution that the parameters follow and the posterior probability density formula of the model, derive the formula for the posterior probability in the Gibbs sampling of the non - parametric Bayesian dictionary learning model for multi - task blind compressive sensing.

[0034] Sample \(d\) k : Based on the relationship between the prior distribution of the parameter \(d\) k and the likelihood function, derive the posterior probability formula for the dictionary atoms: Denoted as where the covariance matrix calculation formula is: The mean calculation formula is The \(y\) in the mean formula (i，t，-k) is specifically defined as \(y\) (i，t，-k) =\(y\) ti - A i D(s ti \(\cdot z\) ti ) + A i s tik z tik d k .

[0035] Sample \(z\) tik : Based on the relationship between the prior distribution of the parameter \(z\) tik and the likelihood function, derive the posterior probability formula for the dictionary sparse coefficients: where p 0 = 1 - \(\pi\) tk .

[0036] Sample \(s\) tik : Based on the relationship between the prior distribution of the parameter \(s\) tik and the likelihood function, derive the posterior probability formula for the dictionary weight coefficients: \(p(s\)tik |-)~N(s tik |μ stik ,σ stik ), the covariance calculation formula is The mean calculation formula is

[0037] Sampling π tk : Based on the relationship between the conjugate prior of parameter π tk and the likelihood function, the precision π tk of the dictionary sparse coefficient is deduced, and the posterior probability formula is

[0038]

[0039] Sampling Based on the relationship between the conjugate prior of parameter and the likelihood function, the precision of the dictionary weight coefficient is deduced, and the posterior probability formula is

[0040]

[0041] Sampling Based on the relationship between the conjugate prior of parameter and the likelihood function, the precision of the reconstruction noise is deduced, and the posterior probability formula is

[0042]

[0043] Step 4) includes the following steps:

[0044] The image block is reconstructed from the dictionary, dictionary sparse coefficient, and dictionary weight coefficient obtained by Gibbs sampling. The synthesis formula of the image block is x ti = D(s ti ·z ti ); The image block is restored and the overlapping part of the graphic block is smoothed by mean, and then the reconstruction result is moved backward channel by channel along the spectral dimension to obtain the L-frame reconstructed hyperspectral video; where the backward movement refers to the movement in the opposite direction to the dispersion process in step 1). After multiple iterations of step 3) and step 4), the reconstruction result of the last time is output.

[0045] Step 5) includes the following steps:

[0046] Based on the idea of "sliding window", for the P-frame compressed hyperspectral video, L frames in the hyperspectral video are reconstructed simultaneously each time from step 1) to step 4) The moving step size of the sliding window is 1. Steps 1) to 4) are iterated P - L + 1 times, and the repeated reconstruction results of the frames are averaged to finally obtain the P-frame hyperspectral video. In the experiment of this method, L = 5, a0 = 10 6 , b 0 = 1, c 0 = d 0 = e 0 = f 0 = 10 3 , β = 1, σ = 10, K = 32.

[0047] The accompanying drawings list some images as the results reconstructed by the hyperspectral video reconstruction method of compressive measurement and multi-task blind compressive sensing. The content described in the embodiments of this specification is only an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms stated in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art according to the inventive concept of the present invention.

Claims

1. A hyperspectral video reconstruction method based on multi-task blind compressive sensing, comprising the following steps: Step 1) Obtain the L-frame in the P-frame compressed hyperspectral video, where P ≥ L > 1. The compressive sensing formula for reconstructing the hyperspectral video is expressed as {y t} t＝1，L = A{x t} t＝1，L + {∈ t} t＝1，L , where t represents the frame number, / represents the compressive measurement obtained using a CASSI (Coded Aperture Snapshot Spectral Imager) fast camera; A represents the observation matrix. Based on the compression principle of the CASSI fast camera, the observation matrix A represents a binary coded aperture; {x t} t＝1，L represents the reconstructed hyperspectral video with dispersion added to each frame, that is, the result of each frame in the reconstructed hyperspectral video moving channel by channel along the spectral dimension; N x and N y are the length and width of the spatial dimension of the reconstructed hyperspectral video frame respectively, and N s is the number of channels in the spectral dimension; {∈ t} t＝1，L represents the reconstruction noise. Step 2) Extract N overlapping image patches of size d×d from each of the L compressive measurements and reshape them into column vectors of size d×1 where d represents the size of the image patches, and m = d 2 ×1; Input the L sets of image patches into the multi-task non-parametric Bayesian dictionary learning model; where d represents the size of the image patches, m = d 2 ; Input the L sets of image patches into the multi-task non-parametric Bayesian dictionary learning model; Step 3) In the multi-task non-parametric Bayesian dictionary learning model, assume that the dictionary obeys a multivariate Gaussian distribution with a mean of 0 and a covariance related to the spatial and spectral bands where the pixels are located, M = N s d 2 and m << M; the dictionary weight coefficient s ti obeys a multivariate Gaussian distribution with a precision of , obeys a Gamma distribution; the dictionary sparse coefficient z tik obeys a Bernoulli distribution with a precision of π tk , π tk obeys a Beta distribution; the reconstruction noise ∈ ti obeys a multivariate Gaussian distribution with a precision of , obeys a Gamma distribution; the multi-task non-parametric Bayesian dictionary learning model realizes collaborative work among tasks by sharing dictionaries and hyperparameters among tasks, and the inference of dictionary correlation coefficients in each task is carried out independently; the posterior probability formula of each parameter is derived through the relationship between the conjugate prior and the likelihood function, and Gibbs sampling is used to update all the parameters to be estimated in turn. The parameters to be estimated are the dictionary D, the dictionary weight coefficient s ti , the dictionary sparse coefficient z tik , the precision of the dictionary weight coefficient the precision π of the dictionary sparse coefficient tk , and the noise precision Step 4) The dictionary D and dictionary weight coefficient s obtained by Gibbs sampling ti , and the dictionary sparse coefficient z tik Reconstruct the image blocks, restore the image blocks and perform mean smoothing on the overlapping parts of the graphic blocks, and then move the reconstruction results backward channel by channel along the spectral dimension to obtain L frames of reconstructed hyperspectral video; where the backward movement refers to moving in the opposite direction to the dispersion process in step 1); After multiple iterations of steps 3)-4), output the reconstruction result of the last time; Step 5): Based on the idea of a sliding window, for the P-frame compressed hyperspectral video, L frames of hyperspectral video can be reconstructed in each round of iteration from Steps 1)-4). After multiple iterations of Steps 1)-4), the reconstruction results of all frames in the compressed hyperspectral video are obtained, and the repeated reconstruction results of the frames are averaged to finally obtain a complete hyperspectral video.

2. A hyperspectral video reconstruction method based on multi-task blind compressive sensing as described in Claim 1, characterized in that: Step 2) comprises the following steps: L-frame compressed measurements in the P-frame compressed hyperspectral video obtained by the CASSI fast camera are input where P ≥ L > 1, the size of the initialized sub-block is d×d, the scanning step is step, 1 ≤ step ≤ d, and L groups of compressed measurements are all used to extract overlapping small image blocks and reshape them into column vectors of size d 2 ×1 m = d 2 ; for each of the L groups, the number N of image blocks in each group is equal to Input to the next step.

3. A hyperspectral video reconstruction method based on multi-task blind compressive sensing as described in Claim 1, characterized in that: Step 3) comprises the following steps: 3-1) In the multi-task non-parametric dictionary learning model, the compressive sensing formula is y ti = A i D(s ti ·z ti ) + ∈ ti , and the image patch synthesis formula is x ti = D(s ti ·z ti ). A i represents the binary coded aperture at the corresponding position of the image patch x ti , s ti represents the dictionary weight coefficient, z ti represents the dictionary sparse coefficient, and ∈ ti represents the reconstruction noise, and · represents the Hadamard product; the prior distribution that the parameters follow needs to be specified first; the dictionary atoms d in the dictionary k follow a multivariate Gaussian distribution: where M = N s d 2 and m << M, and Ω is the covariance matrix related to the spatial position and spectral band in the image patch where the pixel is located; The sparse coefficient z of the dictionary tik follows a Bernoulli distribution: z tik ~Bernoulli(π tk ), and the precision π tk follows a Beta distribution: The weight coefficient s of the dictionary ti follows a multivariate Gaussian distribution: where I K denotes the K×K identity matrix, and the precision follows a Gamma distribution: The reconstruction noise ∈ ti follows a multivariate Gaussian distribution with precision : where I m denotes the m×m identity matrix, and the precision follows a Gamma distribution: 3-2) The posterior probability density formula of the non-parametric Bayesian dictionary learning model for multi-task blind compressive sensing is: Based on the relationship between the prior distribution and the likelihood function, the posterior distribution of the parameters can be derived, and the parameters d k , z tik , s tik , π tk , are sampled. Gibbs sampling updates the value of the current parameter by iteratively using the latest values of the other parameters.

4. A hyperspectral video reconstruction method based on multi-task blind compressive sensing as described in Claim 1, characterized in that: Step 4) comprises the following steps: According to the image block synthesis formula x ti = D(s ti ·z ti ) to obtain the reconstructed image block, restore it to the original position of the original image, take the mean value for smoothing at the overlapping parts of the blocks, and then move the reconstruction result backward channel by channel along the spectral dimension to obtain the L-frame hyperspectral video reconstructed in this iteration; the backward movement refers to the movement in the direction opposite to the dispersion process in step 1), and output the reconstruction result of the last round of iteration.

5. A hyperspectral video reconstruction method based on multi-task blind compressive sensing as described in Claim 1, characterized in that: Step 5) comprises the following steps: Based on the idea of "sliding window", for P-frame compressed hyperspectral video, in each iteration from step 1) - 4), L frames of the hyperspectral video are reconstructed simultaneously. If the moving step size of the sliding window is 1, then a total of P + L - 1 iterations are required from step 1) - 4), and the repeated reconstruction results of the frames are averaged to finally obtain the P-frame hyperspectral video.

Images