Unsupervised hyperspectral image reconstruction method and device, electronic equipment and storage medium

Abstract

The application provides a kind of unsupervised hyperspectral image reconstruction method, device, electronic equipment and storage medium, belong to the technical field of image recovery, in this method, two-stage learning framework effectively combines supervised model (i.e. auxiliary network) and unsupervised model (i.e. multi-branch deep image prior network), so that the non-local feature and local feature (i.e. internal prior information) of the second compressed image sample can be extracted simultaneously;In addition, the hyperspectral image reconstruction problem based on the multi-branch deep image prior network is constructed, because the ranks of each branch deep image prior network are different, different modalities of the second compressed image sample can be learned, and finally the information of these different modalities is fused, so that the information of each modality is fully interacted, so that the high-dimensional information of the second compressed image sample is better extracted, and the reconstructed hyperspectral image is more robust to noise, and the overfitting phenomenon in learning can be effectively avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recovery, and in particular to a method and device for unsupervised hyperspectral image reconstruction, electronic equipment and storage medium. BACKGROUND

[0002] Compared with traditional three-channel images, hyperspectral images cover more abundant spectral information, usually have dozens or even hundreds of continuous spectral bands, and the interval between adjacent bands is usually between several nanometers and tens of nanometers, so it is widely used in remote sensing, biomedical research, anomaly detection and other fields.

[0003] The way to capture hyperspectral images is roughly divided into two kinds, namely linear scanning and coding aperture hyperspectral imaging system. The linear scanning is to sample a matrix at each time node, and then complete the scanning of the entire hyperspectral image in a period of time. For example, the push-broom camera is to complete imaging by mapping the spectral snapshot onto the spatial sensor. The coding aperture hyperspectral imaging is to modulate the hyperspectral image by using the coding aperture, and then to shift the spectrum of the coded image by the dispersive prism, and finally to collect the image on the imaging device. Because the coding aperture imaging requires low bandwidth and fast acquisition speed, the mainstream hyperspectral acquisition system currently mainly adopts the coding aperture imaging scheme.

[0004] The coding aperture imaging system compresses a three-dimensional hyperspectral image into a two-dimensional image, so a reconstruction algorithm is needed to reconstruct the compressed image into a hyperspectral image. The mainstream hyperspectral reconstruction algorithm can be divided into three categories: model-based iterative algorithm, supervised learning algorithm and unsupervised learning algorithm.

[0005] Model-based iterative algorithms usually utilize a regularizer to solve a highly underdetermined inverse problem iteratively, and the selection of the regularizer depends on the prior information of the image. For example, the rank-minimization spectral image reconstruction algorithm (Decompress SCI, DESCI) based on the alternating direction method of multipliers (ADMM) fuses the non-local self-similarity of the hyperspectral image as prior information into the iterative algorithm. The two-step iterative shrinkage algorithm processes the image through two-step iteration, and optimizes the reconstruction result through soft thresholding and the like. Some researchers introduce the total variation (TV) prior into the generalized alternating projection algorithm, and first perform gradient descent on the original signal, and then project it into the expected image space. Some works are based on low-rank matrix approximation, and mine the spectral correlation inside the hyperspectral image. In addition, since the hyperspectral image can be regarded as a three-order tensor, in recent years, some researchers have introduced low-rank tensor decomposition into this task. For example, some works combine Tucker decomposition and weighted total variation prior, and at the same time, constrain the hyperspectral image to satisfy low-rank and sparsity. In the past two years, some researchers have introduced a CP low-rank constraint, first classify the hyperspectral image blocks according to their spectral characteristics, construct a four-order tensor, and then use the CP low-rank constraint to constrain the tensor to obtain a clear hyperspectral image. Although the model-based iterative algorithm has interpretability, the prior conditions used by these methods are determined empirically, and have limitations in many scenarios.

[0006] Supervised learning algorithms require a large amount of data for training, and the training time is relatively long, but the inference speed after training is generally within 1s. Some researchers use convolutional neural networks (CNNs) to learn the nonlinear mapping between two-dimensional imaging in the coded aperture imaging system and the original hyperspectral image, and achieve good reconstruction performance. In addition, in order to overcome the problem that the convolutional network can only extract local features, another part of the researchers introduces an attention mechanism into the model to better mine the spectral and spatial correlation of the hyperspectral image. In order to overcome the shortcomings of poor interpretability and slow convergence speed of end-to-end learning, people further proposed a deep unfolding method, which unfolds the iterative algorithm into a multi-stage network, thus combining the advantages of model-based and learning-based methods.

[0007] Supervised learning methods require large hyperspectral datasets, and the network needs to be retrained when different encoding designs are used or when the dataset size or number of spectral channels is changed. In this case, if a model is trained using a single encoding pattern, it can only be applied to the imaging system corresponding to that encoding and cannot reconstruct images of other imaging systems. Therefore, researchers have adopted unsupervised learning methods. Early researchers proposed a plug-and-play approach that uses pre-trained deep denoising convolutional neural network weights as constraints to regularize the solution space of hyperspectral images. Other research, inspired by Generative Adversarial Networks (GANs), has proposed a conditional generative model based on convolutional networks. Some researchers have incorporated deep image priors into plug-and-play algorithm frameworks to develop unsupervised networks. In the past two years, researchers have discovered that tensor quantum space decomposition can be used to decompose hyperspectral images into spectral basis and spatial coefficients, and these two components can be learned separately within a deep image prior framework. Although previous studies have achieved some success using tensor quantum space decomposition, the rank of the decomposition is subjectively selected, and the optimal decomposition rank varies for different data distributions. Searching for the optimal rank using a grid search method is extremely time-consuming. Furthermore, overfitting is a severe problem in deep image prior networks. While some techniques, such as early stopping, can alleviate this issue, the criteria for early stopping are difficult to determine, and the timing of training termination is also difficult to control.

[0008] In summary, traditional unsupervised hyperspectral image reconstruction methods (such as Zhang's quantum space decomposition method) use a subjectively selected single rank for decomposition, and suffer from severe overfitting problems in deep image prior networks, resulting in poor performance of the reconstructed hyperspectral images. Summary of the Invention

[0009] In view of this, the purpose of the present invention is to provide an unsupervised hyperspectral image reconstruction method, apparatus, electronic device and storage medium to alleviate the technical problem that the rank of the decomposition of traditional unsupervised hyperspectral image reconstruction methods is a subjectively selected single rank, and there is a serious overfitting problem in the depth image prior network, resulting in poor performance of the reconstructed hyperspectral image.

[0010] In a first aspect, embodiments of the present invention provide an unsupervised hyperspectral image reconstruction method, comprising:

[0011] Obtain a training dataset and a test dataset, wherein the training dataset includes: hyperspectral image samples and a first compressed image sample corresponding to the hyperspectral image samples, and the test dataset includes: a second compressed image sample;

[0012] The original auxiliary network is trained by using the training data set, and an auxiliary network is obtained, wherein the original auxiliary network is a residual transformer network.

[0013] The encoding of the coded aperture imaging system and the second compressed image sample are inferred by using the auxiliary network, and an auxiliary image is obtained.

[0014] A hyperspectral image reconstruction problem based on a multi-branch deep image prior network is constructed based on the encoding, the second compressed image sample and the auxiliary image, and the hyperspectral image reconstruction problem based on the multi-branch deep image prior network is iteratively solved by using an alternating direction multiplier method, and a reconstructed hyperspectral image corresponding to the second compressed image sample is obtained, wherein each branch deep image prior network in the multi-branch deep image prior network is a tensor subspace decomposition model with different ranks.

[0015] In a second aspect, the embodiments of the present application further provide a device for unsupervised hyperspectral image reconstruction, comprising:

[0016] An acquisition unit is configured to acquire a training data set and a test data set, wherein the training data set comprises a hyperspectral image sample and a first compressed image sample corresponding to the hyperspectral image sample, and the test data set comprises a second compressed image sample.

[0017] A training unit is configured to train an original auxiliary network by using the training data set, and obtain an auxiliary network, wherein the original auxiliary network is a residual transformer network.

[0018] An inference unit is configured to infer the encoding of the coded aperture imaging system and the second compressed image sample by using the auxiliary network, and obtain an auxiliary image.

[0019] A construction and iterative solving unit is configured to construct a hyperspectral image reconstruction problem based on a multi-branch deep image prior network based on the encoding, the second compressed image sample and the auxiliary image, and iteratively solve the hyperspectral image reconstruction problem based on the multi-branch deep image prior network by using an alternating direction multiplier method, and obtain a reconstructed hyperspectral image corresponding to the second compressed image sample, wherein each branch deep image prior network in the multi-branch deep image prior network is a tensor subspace decomposition model with different ranks.

[0020] In a third aspect, the embodiments of the present application further provide an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of any one of the first aspect when executing the computer program.

[0021] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores machine executable instructions, when the machine executable instructions are invoked and run by a processor, the machine executable instructions cause the processor to execute the method of any one of the first aspect.

[0022] In the embodiments of the present application, a method for reconstructing an unsupervised hyperspectral image is provided, including: obtaining a training data set and a test data set, wherein the training data set includes: a hyperspectral image sample and a first compressed image sample corresponding to the hyperspectral image sample, and the test data set includes: a second compressed image sample; training an original auxiliary network by using the training data set to obtain an auxiliary network, wherein the original auxiliary network is a residual transformer network; inferring the encoding of a coded aperture imaging system and the second compressed image sample by using the auxiliary network to obtain an auxiliary image; constructing a hyperspectral image reconstruction problem based on a multi-branch deep image prior network based on the encoding, the second compressed image sample and the auxiliary image, and iteratively solving the hyperspectral image reconstruction problem based on the multi-branch deep image prior network by using an alternating direction multiplier method to obtain a reconstructed hyperspectral image corresponding to the second compressed image sample, wherein each branch deep image prior network in the multi-branch deep image prior network is a tensor subspace decomposition model with different ranks. As can be seen from the above description, in the method for reconstructing an unsupervised hyperspectral image, an auxiliary network is first trained to mine the non-local features (i.e. external data features) of the second compressed image sample, that is, the auxiliary image, which is used as additional information in the iterative solving process of the hyperspectral image reconstruction problem based on the multi-branch deep image prior network, and the learning of the multi-branch deep image prior network, the two-stage learning framework effectively combines the supervised model (i.e. the auxiliary network) and the unsupervised model (i.e. the multi-branch deep image prior network), so that the non-local features and the local features (i.e. internal prior information) of the second compressed image sample can be extracted at the same time; in addition, the hyperspectral image reconstruction problem based on the multi-branch deep image prior network is constructed, and since the ranks of the branch deep image prior networks are different, different modalities of the second compressed image sample can be learned, rather than a single modality in the traditional technology, and finally the information of these different modalities is fused, so that the information of each modality is fully interacted, thereby better extracting the high-dimensional information of the second compressed image sample. Compared with the single-branch method, the reconstructed hyperspectral image is more robust to noise, and the overfitting phenomenon in learning can be effectively avoided, the quality of the reconstructed hyperspectral image is good, and the technical problem of the traditional unsupervised hyperspectral image reconstruction method that the decomposed rank is a single rank selected subjectively and there is a serious overfitting problem in the deep image prior network, and the performance of the finally reconstructed hyperspectral image is poor, is solved. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to make the technical solutions in the specific embodiments or prior art of the present application clearer, the drawings needed in the specific embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0024] Figure 1 A flow chart of an unsupervised hyperspectral image reconstruction method provided for an embodiment of the present application is shown in the figure.

[0025] Figure 2 A schematic diagram of three-dimensional tensor subspace decomposition provided for an embodiment of the present application is shown in the figure.

[0026] Figure 3 A schematic diagram of multi-branch deep image prior network learning provided for an embodiment of the present application is shown in the figure.

[0027] Figure 4 A schematic diagram of hyperspectral reconstruction based on ensemble learning provided for an embodiment of the present application is shown in the figure.

[0028] Figure 5 A schematic diagram of a hybrid self-attention decoding module provided for an embodiment of the present application is shown in the figure.

[0029] Figure 6 A schematic diagram of an unsupervised hyperspectral image reconstruction device provided for an embodiment of the present application is shown in the figure.

[0030] Figure 7 A schematic diagram of an electronic device provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0031] The technical solutions of the present application will be described below in conjunction with embodiments, obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0032] The rank of the decomposition of the traditional unsupervised hyperspectral image reconstruction method is a single rank selected subjectively, and there is a serious overfitting problem in the deep image prior network, and the performance of the finally reconstructed hyperspectral image is poor.

[0033] Based on this, in the unsupervised hyperspectral image reconstruction method of the application, a auxiliary network is first trained to mine the non-local features (that is, external data features) of the second compressed image sample, that is, an auxiliary image, and the auxiliary image is used as additional information in the iterative solving process of the hyperspectral image reconstruction problem based on the multi-branch deep image prior network, and the learning of the auxiliary multi-branch deep image prior network, the above two-stage learning framework effectively combines the supervised model (that is, the auxiliary network) and the unsupervised model (that is, the multi-branch deep image prior network), so that the non-local features and the local features (that is, the internal prior information) of the second compressed image sample can be extracted at the same time; in addition, the hyperspectral image reconstruction problem based on the multi-branch deep image prior network is constructed, and different modalities of the second compressed image sample can be learned due to the different ranks of the branch deep image prior networks, instead of the single modality in the traditional technology, and finally the information of different modalities is fused, so that the information of each modality is fully interacted, thereby better extracting the high-dimensional information of the second compressed image sample. Compared with the single-branch method, the reconstructed hyperspectral image is more robust to noise, and the overfitting phenomenon in learning can be effectively avoided, and the quality of the reconstructed hyperspectral image is good.

[0034] In order to facilitate the understanding of the embodiment, first, a kind of unsupervised hyperspectral image reconstruction method disclosed in the embodiment of the application is introduced in detail.

[0035] Embodiment one:

[0036] According to the embodiment of the application, an embodiment of a kind of unsupervised hyperspectral image reconstruction method is provided, it should be noted that the steps shown in the flowchart of the drawing can be executed in computer system such as a group of computer executable instructions, and although logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in different order from here.

[0037] Figure 1 It is the flowchart of a kind of unsupervised hyperspectral image reconstruction method according to the embodiment of the application, as Figure 1 As shown in the figure, the method comprises the following steps:

[0038] Step S102, obtain training data set and test data set, wherein the training data set includes: hyperspectral image sample and the first compressed image sample corresponding to the hyperspectral image sample, and the test data set includes: second compressed image sample;

[0039] In the embodiment of the application, the hyperspectral images in CAVE, ICVL hyperspectral data set are used after preprocessing as training data set, and the hyperspectral images in KAIST hyperspectral data set are used after preprocessing as test data set.

[0040] Specifically, the CAVE contains 32 indoor hyperspectral images with a size of 512*512*31, and the ICVL contains 200 outdoor hyperspectral images with a size of 1392*1300*31. The present application randomly selects the hyperspectral images in the CAVE and ICVL data sets, and randomly crops them to have a spatial size of 256*256 and select 28 spectrums therefrom to obtain a hyperspectral image sample with a size of 256*256*28, i.e. a hyperspectral image sample in the training data set. Then, the pixel values of the hyperspectral image sample are normalized to the interval [0, 1], and the normalized hyperspectral image sample is compressed by the coded aperture imaging system to obtain a first compressed image sample corresponding to the hyperspectral image sample, the first compressed image sample having a size of 256*283, and thus obtaining a training data set comprising the hyperspectral image sample and the first compressed image sample corresponding to the hyperspectral image sample.

[0041] The KAIST data set contains 30 hyperspectral images with a size of 2704*3376*31, and the spectral band is between 420nm and 720nm. The present application randomly selects 10 hyperspectral images of scenes as the test data set after preprocessing. Specifically, each of the 10 hyperspectral images of scenes is randomly cropped to have a size of 256*256*28, and then the pixel values of the cropped hyperspectral images are normalized to obtain normalized hyperspectral images, and the normalized hyperspectral images are compressed by the coded aperture imaging system to obtain second compressed image samples, the second compressed image samples having a size of 256*283, and thus obtaining a test data set comprising the second compressed image samples.

[0042] Step S104, training the original auxiliary network using the training data set to obtain an auxiliary network, wherein the original auxiliary network is a residual transformer network;

[0043] Specifically, the original auxiliary network is a residual network based on a transformer architecture (Transformer), i.e. a residual transformer network (Residual Transformer), and the trained auxiliary network can mine non-local features (i.e. external data features) of the image.

[0044] The auxiliary network uses a self-attention mechanism to effectively extract non-local features in an image. In order to make full use of the multi-class hyperspectral dataset, the auxiliary network is trained on a training dataset (obtained based on the CAVE and ICVL datasets), and when the average peak signal-to-noise ratio of the auxiliary network on a validation set (also obtained based on the CAVE and ICVL datasets) tends to converge, the training is stopped and the auxiliary network weight is saved (in the present application, the training is performed on an Nvidia GeForce RTX3080 GPU, and the training is ended when the iteration reaches 3000 times).

[0045] It should be noted that in subsequent applications, the auxiliary network no longer needs to be trained, and only one training of this step is required.

[0046] In step S106, the auxiliary network is used to infer the encoding of the coded aperture imaging system and the second compressed image sample to obtain an auxiliary image.

[0047] Specifically, the encoding is , represents a matrix of one row, column. The second compressed image sample is (compressed and imaged by the coded aperture imaging system on the hyperspectral image in the KAIST dataset), wherein is the spectral number of the hyperspectral image , and is the spatial resolution, and the encoding and the second compressed image sample are taken as inputs of the auxiliary network, and an auxiliary image is output, which is represented as .

[0048] The auxiliary image is subsequently integrated into the training of the multi-branch depth image prior network (specifically, the is added to In the loss function during the training of the multi-branch depth image prior network, to further improve the reconstruction quality of the image. Compared with the traditional depth image prior network, the two-stage learning (the first stage is the training process of the auxiliary network, and the second stage is the training process of the multi-branch depth image prior network) fuses the data features (i.e. external data features) of the hyperspectral data sets such as CAVE and ICVL, so that the subsequent multi-branch depth image prior network can extract non-local information in the image, making up for the deficiency of the traditional depth image prior network that can only extract local features (because the traditional depth image prior network does not have the process of the preceding auxiliary network, and there is no concept of auxiliary image, the auxiliary image is used as additional information for the training of the multi-branch depth image prior network in the present application).

[0049] In step S108, a hyperspectral image reconstruction problem based on the multi-branch depth image prior network is constructed based on the encoding, the second compressed image sample and the auxiliary image, and the hyperspectral image reconstruction problem based on the multi-branch depth image prior network is iteratively solved by using the alternating direction method of multipliers to obtain the reconstructed hyperspectral image corresponding to the second compressed image sample, wherein each branch depth image prior network in the multi-branch depth image prior network is a tensor subspace decomposition model with different ranks.

[0050] The rank is the decomposition rank of the subspace, and the tensor subspace decomposition model includes a spatial coefficient decoder and a spectral basis decoder.

[0051] The process will be described in detail below, and will not be repeated here.

[0052] In the embodiment of the present application, an unsupervised hyperspectral image reconstruction method is provided, comprising: obtaining a training data set and a test data set, wherein the training data set comprises: a hyperspectral image sample and a first compressed image sample corresponding to the hyperspectral image sample, and the test data set comprises: a second compressed image sample; training an original auxiliary network by using the training data set to obtain an auxiliary network, wherein the original auxiliary network is a residual transformer network; inferring the encoding of a coded aperture imaging system and the second compressed image sample by using the auxiliary network to obtain an auxiliary image; constructing a hyperspectral image reconstruction problem based on a multi-branch deep image prior network based on the encoding, the second compressed image sample and the auxiliary image, and iteratively solving the hyperspectral image reconstruction problem based on the multi-branch deep image prior network by using an alternating direction multiplier method to obtain a reconstructed hyperspectral image corresponding to the second compressed image sample, wherein each branch deep image prior network in the multi-branch deep image prior network is a tensor subspace decomposition model with different ranks. As can be seen from the above description, in the unsupervised hyperspectral image reconstruction method of the present application, an auxiliary network is first trained to mine the non-local features (i.e. external data features) of the second compressed image sample, i.e. the auxiliary image, which is used as additional information in the iterative solving process of the hyperspectral image reconstruction problem based on the multi-branch deep image prior network. The above two-stage learning framework effectively combines the supervised model (i.e. the auxiliary network) and the unsupervised model (i.e. the multi-branch deep image prior network), so that the non-local features and the local features (i.e. internal prior information) of the second compressed image sample can be extracted at the same time. In addition, the hyperspectral image reconstruction problem based on the multi-branch deep image prior network is constructed, and different modalities of the second compressed image sample can be learned due to the different ranks of each branch deep image prior network, rather than a single modality in the traditional technology. Finally, the information of these different modalities is fused to make the information of each modality fully interact, so that the high-dimensional information of the second compressed image sample can be better extracted. Compared with the single-branch method, the reconstructed hyperspectral image of the present application is more robust to noise, and the overfitting phenomenon in learning can be effectively avoided. The quality of the reconstructed hyperspectral image is good, and the technical problem of the traditional unsupervised hyperspectral image reconstruction method, in which the decomposition rank is a single rank selected subjectively and there is a serious overfitting problem in the deep image prior network, is solved, and the performance of the finally reconstructed hyperspectral image is poor.

[0053] The above describes the unsupervised hyperspectral image reconstruction method of the present application briefly, and the specific contents involved therein are described in detail below.

[0054] In an optional embodiment of the present application, a hyperspectral image reconstruction problem based on a multi-branch deep image prior network is constructed based on the encoding, the second compressed image sample and the auxiliary image, and specifically includes:

[0055] The hyperspectral image reconstruction problem based on the multi-branch deep image prior network includes: wherein, , , , denotes a hyperparameter, and denotes a degradation constraint term, denotes a data consistency term, denotes an image prior constraint term imposed on , denotes an external data constraint term imposed on , denotes the second compressed image sample, denotes the hyperspectral image to be reconstructed, denotes a degradation operator, denotes an output of the multi-branch deep image prior network, denotes an auxiliary variable, denotes values of the respective input variables when the objective function takes a global minimum, , , denotes the auxiliary image, , , denotes the optimal , , , denotes a three-dimensional total variation constraint, which is used to ensure that the output of the deep image prior network has local smoothness in 3 dimensions (2 spatial dimensions and 1 spectral dimension).

[0056] Specifically, the hyperspectral image reconstruction process of the coded aperture imaging can be regarded as an inverse process of the imaging of the coded aperture imaging system (i.e., the coded aperture imaging system). Let the coded aperture imaging, i.e., the second compressed image sample, be , and the hyperspectral image to be reconstructed be . The optical compression process of the coded aperture imaging system can be represented by a degradation operator , which represents a linear mapping from a third-order hyperspectral image to its two-dimensional imaging (compressed image), i.e., . In the conventional technology, after the second compressed image sample , the encoding and the degradation operator are given, the original hyperspectral image is reconstructed. As a kind of unsupervised learning method, the deep image prior can use a single neural network to learn the implicit regularization term of the image (i.e. use a neural network to implicitly represent an image), so that the hyperspectral image can be recovered from the degraded second compressed image sample without using additional training data. The hyperspectral image reconstruction problem based on the deep image prior network can be expressed as follows:

[0057]

[0058] wherein, is the output of the deep image prior network, and the above hyperspectral image reconstruction problem is a constrained optimization problem. In order to convert it into an unconstrained optimization problem, an auxiliary variable is introduced. The unconstrained optimization problem (i.e. the hyperspectral image reconstruction problem based on the deep image prior network) is as follows:

[0059]

[0060] wherein, is a hyperparameter. At the same time, considering the internal prior of the image output by the multi-branch deep image prior network and the data external prior, the image (the output of the multi-branch deep image prior network) is added with the image prior constraint term and the external data constraint term. In addition, considering that the reconstructed image (the output of the multi-branch deep image prior network) is consistent with the second compressed image sample after degradation, in the unconstrained optimization problem, the term (the degradation process of ) is additionally introduced. The converted unconstrained optimization problem (i.e. the hyperspectral image reconstruction problem based on the multi-branch deep image prior network) can be expressed as follows:

[0061] wherein, , , , all represent hyperparameters, for balancing the contribution degree of each term, and represent the degradation constraint term, i.e. the reconstructed and are subjected to the degradation process of the coded aperture imaging, represents the data consistency term, ensuring the consistency of and , represents the image prior constraint term imposed on , i.e. making more consistent with the distribution of natural images, such as local smoothness, represents the applied external data constraint term, such that the output of the auxiliary network is consistent with the constraint solution space (i.e. through the constraint solution space), such that the generated image fuses the features of the external data, represents the second compressed image sample, represents the hyperspectral image to be reconstructed, represents the degradation operator, represents the output of the multi-branch deep image prior network, represents the auxiliary variable, and argmin represents the values of the individual input variables when the objective function takes the global minimum, , i.e. further constrains the solution space with the auxiliary image such that it utilizes the external data information (i.e. the features of the external data), represents the auxiliary image, represents the third-order total variation smoothing constraint, such that the generated hyperspectral image satisfies the smoothness in space and guarantees the similarity of adjacent spectra, and the definition of the third-order total variation constraint is as follows: .

[0062] On this basis, the above-converted unconstrained optimization problem (i.e. the hyperspectral image reconstruction problem based on the multi-branch deep image prior network) can be decoupled into three sub-problems by using the alternating direction multiplier method, and the variables , and are iterated in turn, so as to optimize the objective function (i.e. the converted unconstrained optimization problem (i.e. the hyperspectral image reconstruction problem based on the multi-branch deep image prior network)) to the global optimal point.

[0063] In an optional embodiment of the present application, the hyperspectral image reconstruction problem based on the multi-branch deep image prior network is iteratively solved by using the alternating direction multiplier method, and specifically includes the following steps:

[0064] (1) iteratively solving to obtain the formula of , wherein represents the hyperspectral image to be reconstructed;

[0065] Specifically, in this step, the variables and are fixed, and the variable is optimized. The optimization sub-problem of this step (iteratively solving The hyperspectral image reconstruction problem based on the multi-branch deep image prior network when iteratively solving can be expressed as:

[0066]

[0067] The above problem is a quadratic optimization problem, and there is a closed-form solution. By taking the first-order derivative of the above problem ( ), the closed-form solution of can be obtained.

[0068] The vector form of in the k+1th iteration (i.e., the expression of ) can be expressed as: where denotes the identity matrix, denotes the encoding of the coded aperture imaging system, which is related to the degradation operator , and is generally a matrix that follows a Bernoulli distribution. denotes the vector form of , denotes tensor vectorization. In addition, since is a diagonal matrix, it can be further expanded to reduce computational complexity, so the expression of the above is further optimized as:

[0069]

[0070] where, denotes the value of in the k+1th iteration, denotes the element-wise division of two matrices, denotes taking the diagonal elements of a matrix and forming a vector.

[0071] (2) Iteratively solve to obtain the expression of , where, denotes the output of the multi-branch deep image prior network;

[0072] Specifically, in this step, the variables and are fixed, and the optimization variable is optimized. The optimization sub-problem of this step (the hyperspectral image reconstruction problem based on the multi-branch deep image prior network when iteratively solving ) can be expressed as: , and in this optimization problem (i.e., the hyperspectral image reconstruction problem based on the multi-branch deep image prior network when iteratively solving Multi-branch deep image prior networks can be used for learning.

[0073] Zhang's quantum space decomposition network (each branch of the deep image prior network) can process the output (output image) of the multi-branch deep image prior network. Decomposed into spatial coefficients and spectral basis, such as Figure 2 As shown, Subspace decomposition can be represented as:

[0074] ;

[0075] in, Represents the outer product of a matrix and a vector. For the spatial coefficient tensor, Represents the R-th spatial coefficient. For the spectral basis matrix, Represents the R-th spectral basis. Let be the rank of the subspace decomposition. For tensor modulus triple product, The Each element can be represented as Therefore, a hyperspectral image can be represented as a modulo-3 product of spatial coefficients and spectral basis. Some previous methods based on depth image priors required manually setting the rank. The spatial coefficients and spectral basis are fitted separately. However, when the spectral distributions of hyperspectral images in different datasets vary significantly, the manually set rank may not be optimal. Furthermore, searching for the optimal rank using a grid search method consumes substantial computational resources. Therefore, this invention proposes a multi-branch deep image prior network, using multiple branch deep image prior networks for learning. Each branch deep image prior network performs tensor quantum space decomposition with a different rank, and then the results of each branch are averaged and fused. Assume J deep image prior networks are used. The rank of the j-th branch deep image prior network is... Its output for ,in, and This represents random Gaussian white noise input to the prior network of the depth image in the j-th branch. Let represent the rank of the depth image prior network in the j-th branch. Let's represent a 3rd-order tensor whose dimensions are of size . , , , Represent a OK A matrix of columns, This represents the spatial coefficient decoder in the depth image prior network of the j-th branch. denotes the spectral basis decoder in the j-th branch deep image prior network, respectively, for generating and , denotes the weight of the spatial coefficient decoder in the j-th branch deep image prior network, denotes the weight of the spectral basis decoder in the j-th branch deep image prior network. The advantage of using multi-branch ensemble learning is that it can avoid the overfitting problem of traditional deep prior networks. Overfitting is caused by the mismatch between the complexity of the model and the complexity of the data features: a complex network is prone to fitting noise, while a too simple network may not effectively fit the details and textures of the hyperspectral image. However, using multi-branch subnetwork (i.e., multi-branch deep image prior network) learning can reduce the bias in learning as much as possible, even if the output of a single subnetwork is not ideal, a robust reconstructed image can still be obtained by ensemble (i.e., using multi-branch deep image prior network to learn multiple subspace decomposition models of the hyperspectral image, and integrating the reconstruction results of multiple branches to reduce the bias in network learning, to generate a more robust reconstruction result), as shown in Figure 3 .

[0076] As J branch deep image prior networks are used for learning, each branch deep image prior network uses a different tensor subspace decomposition model to learn different modalities of the hyperspectral image. The loss function of the j-th branch deep image prior network can be represented as: wherein, , denotes the output of the j-th branch deep image prior network, denotes the tensor mode-3 product, denotes the rank of the j-th branch deep image prior network, and for different branch deep image prior networks, the corresponding rank (i.e., decomposition rank) is different.

[0077] J branch subnetworks (i.e., branch deep image prior networks) are trained simultaneously, and when each branch deep image prior network converges, the output of each branch deep image prior network is obtained, and the J outputs are integrated to finally generate .

[0078] wherein, , denotes the historical information in the iteration process, denotes the value of in the k+1 iteration, denotes the value of in the k+1 iteration, denotes the weight. In addition, the variance of the random Gaussian white noise input to each branch depth image prior network is set to 0.1. Due to the parallel computing capability of GPU, multiple branch sub-networks are rapidly speeded up when back propagation is performed.

[0079] (3) The iterative solution of is obtained, and the formula of is obtained, wherein denotes an auxiliary variable.

[0080] Specifically, in this step, the fixed variables and are optimized, and the optimization variable is optimized. The optimization sub-problem of this step (iterative solution of ) can be represented as follows: The gradient of which is obtained as follows: , denotes the value of in the k+1th iteration, denotes the value of in the kth iteration, denotes the value of in the k+1th iteration, denotes the value of in the k+1th iteration.

[0081] By iteratively rotating , and , the hyperspectral image reconstruction problem based on the multi-branch depth image prior network can be optimized to the global optimum. When the loss function of each branch depth image prior network is stable (i.e., convergent), the training is stopped, and the optimal , and are obtained.

[0082] Setting of learning parameters: In the training of the multi-branch depth image prior network, the ADAM optimizer is used to optimize the loss function, and the learning rate is 0.001. Considering the performance of the multi-branch depth image prior network and the overhead of the training time, the number of iterations of the alternating direction multiplier method is set to 65. In the training of the multi-branch depth image prior network, the warm-up strategy is selected, and in the first iteration, the training rounds of the multi-branch depth image prior network are set to 2000, and the training rounds of the second to 65th iterations are set to 800.

[0083] In an optional embodiment of the present invention, each branch of the deep image prior network includes: a spatial coefficient decoder and a spectral basis decoder, wherein the spatial coefficient decoder includes: a hybrid self-attention decoding module, a first bilinear upsampling layer and a first projection layer; the hybrid self-attention decoding module includes: a plurality of cascaded first convolutional blocks, a channel semantic extraction module connected to the last of the plurality of cascaded first convolutional blocks and a spatial semantic extraction module connected to the last of the last first convolutional blocks; the spectral basis decoder includes: a second convolutional block, a second bilinear upsampling layer and a second projection layer.

[0084] Specifically, each branch of the deep image prior network needs to learn its own spatial coefficients and spectral basis. The spatial coefficients are a third-order tensor. Using convolutional networks can only extract the local information of this tensor, while the non-local information it contains cannot be effectively extracted. To efficiently extract the three dimensions of the spatial coefficients, such as... Figure 4 As shown, the spatial coefficient decoder proposed in this invention includes: multiple hybrid self-attention decoding modules, multiple first bilinear upsampling layers, and a first projection layer (…). Figure 4 The upsampling at the corresponding positions of the spatial coefficients constitutes the first bilinear upsampling layer, and the projection layer is the first projection layer. (See reference.) Figure 5 The hybrid self-attention decoding module includes: multiple concatenated first convolutional blocks, a channel semantic extraction module connected to the last of the concatenated first convolutional blocks, and a spatial semantic extraction module connected to the last first convolutional block. The decoder network corresponding to the spatial coefficients (i.e., the spatial coefficient decoder) has a 4-layer structure, plus a hybrid self-attention decoding module and a first projection layer. Each layer consists of a hybrid self-attention decoding module and a first bilinear upsampling layer, with channel numbers of 32, 64, 96, and 128, respectively. The hybrid self-attention decoding module is used to better extract the spatial and spectral features of the image. In implementation, multi-scale features first pass through multiple concatenated first convolutional blocks in the hybrid self-attention decoding module to further extract local information. Then, the channel semantic extraction module and the spatial semantic extraction module use channel and spatial attention mechanisms, respectively, to further extract high-frequency information. Finally, the average of these two modules (channel semantic extraction module and spatial semantic extraction module) is used as the output. The spectral basis is a matrix, which is learned using a four-layer structure plus a second convolutional block and a second projection layer. Each layer consists of a second convolutional block and a second bilinear upsampling layer. Figure 4 The convolutional block corresponding to the mid-spectral basis is the second convolutional block, the upsampling layer is the second bilinear upsampling layer, and the projection layer is the second projection layer. The number of channels in these four layers are 32, 64, 96, and 128, respectively.

[0085] In an optional embodiment of the present application, the first compressed image sample and the second compressed image sample are obtained by an encoding aperture imaging system.

[0086] It should be noted that the unsupervised hyperspectral image reconstruction method of the present application can be applied to various encoding aperture hyperspectral imaging systems, such as SD-CASSI systems and DD-CASSI systems. In addition, the reconstruction algorithm proposed in the present application can be migrated to other similar image inverse problems, such as the reconstruction of computer tomography images, the reconstruction of ECG electrocardiograms, the reconstruction of synthetic aperture radar images, etc., to provide a complete solution for high-performance image recovery in complex scenarios.

[0087] The method of the present application adopts a two-stage learning framework (auxiliary network and multi-branch deep image prior network), simultaneously excavates internal prior information of the image (multi-branch deep image prior network) and external data features (auxiliary network). At the same time, in order to avoid the overfitting problem in the deep image prior method and the grid search problem of tensor rank, the present application proposes an ensemble learning method, i.e. a bagging algorithm to learn multiple branch tensor subspace decomposition models (i.e. multi-branch deep image prior network), and fuse the reconstruction results of these multiple branches to reduce the bias in network learning. In addition, the present application introduces a lightweight self-attention module, i.e. a hybrid self-attention decoding module, for each branch subnetwork to simultaneously extract multi-scale spatial and spectral information of the hyperspectral image. The multi-branch deep image prior network is embedded into the optimization framework of the alternating direction multiplier method for iterative solution to obtain a clear hyperspectral image. The model proposed in the present application can be deployed in various encoding aperture imaging systems (because it is unsupervised, only the encoding, compressed image and auxiliary image output by the auxiliary network are needed in the subsequent application process, and the compressed image corresponding to the reconstructed hyperspectral image can be iteratively solved), which can reconstruct clear hyperspectral image details without providing additional training data, and various reconstruction indicators all exceed other learning methods.

[0088] The method of the present application has the following beneficial effects:

[0089] The present application proposes a two-stage learning framework, which first pre-trains a transformer-based model (i.e. auxiliary network) to extract feature information in the data, and then inputs the compressed image (i.e. second compressed image sample) of the encoding aperture hyperspectral imaging system into the auxiliary network to obtain an auxiliary image, which is used as additional information to assist the learning of the multi-branch deep image prior network. The two-stage learning framework of the present application effectively combines unsupervised models and supervised models, so that non-local features and local features of the hyperspectral image can be extracted at the same time.

[0090] The application provides a multi-branch deep image prior network.

[0091] The application provides a light self-attention module, namely a hybrid self-attention decoding module, to better extract spatial and spectral features of a hyperspectral image.

[0092] The application provides an unsupervised hyperspectral image reconstruction method based on the multi-branch deep image prior.

[0093] The application is applied to 10 scenes of the KAIST data set to verify the effectiveness of the model.

[0094] Table 1

[0095]

[0096] Table 2

[0097]

[0098] Embodiment two:

[0099] The embodiment of the present application also provides a non-supervised hyperspectral image reconstruction device, which is mainly used for executing the non-supervised hyperspectral image reconstruction method provided in the embodiment one of the present application.

[0100] Figure 6 Fig. 1 is a schematic diagram of a non-supervised hyperspectral image reconstruction device according to the embodiment of the present application, as shown in the figure, the device mainly comprises an acquisition unit 10, a training unit 20, an inference unit 30, a construction and iterative solution unit 40, wherein: Figure 6

[0101] The acquisition unit is used for acquiring a training data set and a test data set, wherein the training data set comprises a hyperspectral image sample and a first compressed image sample corresponding to the hyperspectral image sample, and the test data set comprises a second compressed image sample;

[0102] The training unit is used for training an original auxiliary network by using the training data set to obtain an auxiliary network, wherein the original auxiliary network is a residual transformer network;

[0103] The inference unit is used for inferring the coding of the coded aperture imaging system and the second compressed image sample by using the auxiliary network to obtain an auxiliary image;

[0104] The construction and iterative solution unit is used for constructing a hyperspectral image reconstruction problem based on a multi-branch deep image prior network based on the coding, the second compressed image sample and the auxiliary image, and iteratively solving the hyperspectral image reconstruction problem based on the multi-branch deep image prior network by using an alternating direction multiplier method to obtain a reconstructed hyperspectral image corresponding to the second compressed image sample, wherein each branch deep image prior network in the multi-branch deep image prior network is a tensor subspace decomposition model with different ranks.

[0105] ​In the embodiment of the present application, an unsupervised hyperspectral image reconstruction device is provided, comprising: obtaining a training data set and a test data set, wherein the training data set comprises: a hyperspectral image sample and a first compressed image sample corresponding to the hyperspectral image sample, and the test data set comprises: a second compressed image sample; training an original auxiliary network by using the training data set to obtain an auxiliary network, wherein the original auxiliary network is a residual transformer network; inferring the encoding of the coded aperture imaging system and the second compressed image sample by using the auxiliary network to obtain an auxiliary image; constructing a hyperspectral image reconstruction problem based on a multi-branch deep image prior network based on the encoding, the second compressed image sample and the auxiliary image, and iteratively solving the hyperspectral image reconstruction problem based on the multi-branch deep image prior network by using an alternating direction multiplier method to obtain a reconstructed hyperspectral image corresponding to the second compressed image sample, wherein each branch deep image prior network in the multi-branch deep image prior network is a tensor subspace decomposition model with different ranks. As can be seen from the above description, in the unsupervised hyperspectral image reconstruction device of the present application, an auxiliary network is first trained to mine the non-local features (i.e. external data features) of the second compressed image sample, i.e. the auxiliary image, which is used as additional information in the iterative solving process of the hyperspectral image reconstruction problem based on the multi-branch deep image prior network. The above two-stage learning framework effectively combines the supervised model (i.e. the auxiliary network) and the unsupervised model (i.e. the multi-branch deep image prior network), so that the non-local features and the local features (i.e. internal prior information) of the second compressed image sample can be extracted at the same time. In addition, the hyperspectral image reconstruction problem based on the multi-branch deep image prior network is constructed, and different modalities of the second compressed image sample can be learned due to the different ranks of each branch deep image prior network, rather than a single modality in the traditional technology. Finally, the information of these different modalities is fused to make the information of each modality fully interact, so that the high-dimensional information of the second compressed image sample can be better extracted. Compared with the single-branch method, the reconstructed hyperspectral image of the present application is more robust to noise, and the overfitting phenomenon in learning can be effectively avoided. The quality of the reconstructed hyperspectral image is good, and the technical problem of the traditional unsupervised hyperspectral image reconstruction method is solved, i.e. the decomposition rank is a single rank selected subjectively, and there is a serious overfitting problem in the deep image prior network, and the performance of the finally reconstructed hyperspectral image is poor.

[0106] Optionally, the constructing and iteratively solving unit is further configured to: the hyperspectral image reconstruction problem based on the multi-branch deep image prior network comprises: wherein, , , , denotes a hyper-parameter, and denotes a degenerate constraint term, denotes a data consistency term, denotes an image prior constraint term imposed on , denotes an external data constraint term imposed on , denotes a second compressed image sample, denotes a hyperspectral image to be reconstructed, denotes a degenerate operator, denotes an output of the multi-branch deep image prior network, denotes an auxiliary variable, denotes values of each input variable when the objective function takes a global minimum, , , denotes an auxiliary image, , , denote optimal , , , denotes a three-dimensional total variation constraint.

[0107] Optionally, the constructing and iteratively solving unit is further configured to: iteratively solve to obtain an equation of , wherein denotes a hyperspectral image to be reconstructed; iteratively solve to obtain an equation of , wherein denotes an output of the multi-branch deep image prior network; iteratively solve to obtain an equation of , wherein denotes an auxiliary variable.

[0108] Optionally, when iteratively solving , the hyperspectral image reconstruction problem of the multi-branch deep image prior network can be represented as , and a corresponding equation of is , wherein denotes a degenerate constraint term, denotes a data consistency term, denotes a second compressed image sample, denotes a hyperspectral image to be reconstructed, denotes a degenerate operator, denotes the output of the multi-branch deep image prior network, denotes an auxiliary variable, denotes the value of in the k+1th iteration, , denotes the value of in the k+1th iteration, denotes tensor vectorization, denotes the value of in the k+1th iteration, denotes the value of in the k+1th iteration, denotes encoding, denotes denotes the vector form of denotes element-wise division of two matrices, denotes taking the diagonal elements of a matrix and composing a vector, denotes a hyperparameter; when solving iteratively, the hyperspectral image reconstruction problem based on the multi-branch deep image prior network can be represented as , the solution of is learned using the multi-branch deep image prior network, wherein the loss function of the jth branch deep image prior network is , denotes a degradation constraint term, denotes a data consistency term, denotes an image prior constraint term imposed on , denotes an external data constraint term imposed on , , , , denotes a hyperparameter, denotes a second compressed image sample, denotes a hyperspectral image to be reconstructed, denotes a degradation operator, denotes the output of the multi-branch deep image prior network, denotes an auxiliary variable, , , denotes an auxiliary image, , denotes the output of the jth branch deep image prior network, denotes a spatial coefficient decoder in the jth branch deep image prior network, denotes a tensor mod three product, denotes a spectral basis decoder in the jth branch deep image prior network, denote the weights of the spatial coefficient decoder in the j-th branch depth image prior network, denote the weights of the spectral basis decoder in the j-th branch depth image prior network, and denote the random Gaussian white noise input to the j-th branch depth image prior network, denote the rank of the j-th branch depth image prior network, denote a 3-order tensor with each dimension of size , , , denote a row column matrix, and J branch depth image prior networks are trained simultaneously, and the output of each branch depth image prior network is obtained according to the output of each branch depth image prior network after the convergence of each branch depth image prior network , denote the value of in the k+1 iteration, , denote the weights, denote the value of in the k+1 iteration, denote the value of in the k+1 iteration; when the iterative solution is performed on , the hyperspectral image reconstruction problem based on the multi-branch depth image prior network can be represented as , the corresponding formula of is , wherein, denote the data consistency term, denote the hyperspectral image to be reconstructed, denote the output of the multi-branch depth image prior network, denote the auxiliary variable, denote the value of in the k+1 iteration, denote the value of in the k iteration, denote the value of in the k+1 iteration, denote the value of in the k+1 iteration.

[0109] ​Optionally, each branch depth image prior network comprises a spatial coefficient decoder and a spectral basis decoder, wherein the spatial coefficient decoder comprises a hybrid self-attention decoding module, a first bilinear up-sampling layer and a first projection layer; the hybrid self-attention decoding module comprises a plurality of first convolutional blocks connected in series, a channel semantic extraction module connected with a last first convolutional block in the plurality of first convolutional blocks connected in series, and a spatial semantic extraction module connected with the last first convolutional block; and the spectral basis decoder comprises a second convolutional block, a second bilinear up-sampling layer and a second projection layer.

[0110] Optionally, the first compressed image sample and the second compressed image sample are obtained by an encoding aperture imaging system.

[0111] The device provided by the embodiment of the present application has the same implementation principle and technical effects as the foregoing method embodiment, and for brevity of description, the part not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiment.

[0112] As shown in Figure 7 The embodiment of the present application provides an electronic device 600, which comprises a processor 601, a memory 602 and a bus, the memory 602 stores machine readable instructions executable by the processor 601, when the electronic device is running, the processor 601 and the memory 602 communicate through the bus, and the processor 601 executes the machine readable instructions to perform the steps of the above-mentioned unsupervised hyperspectral image reconstruction method.

[0113] Specifically, the above-mentioned memory 602 and processor 601 can be general memory and processor, which are not specifically limited here, when the processor 601 runs the computer program stored in the memory 602, the above-mentioned unsupervised hyperspectral image reconstruction method can be executed.

[0114] The processor 601 can be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware or the instruction in the form of software in the processor 601. The processor 601 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602, and combines the hardware to complete the steps of the above method.

[0115] Corresponding to the above unsupervised hyperspectral image reconstruction method, the embodiment of the present application also provides a computer readable storage medium, the computer readable storage medium stores machine executable instructions, when the processor calls and runs the computer executable instructions, the computer executable instructions make the processor run the steps of the above unsupervised hyperspectral image reconstruction method.

[0116] The unsupervised hyperspectral image reconstruction device provided by the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided by the embodiment of the present application has the same implementation principle and technical effect as the foregoing method embodiments. For the sake of brevity, the part of the device embodiment not mentioned in the foregoing method embodiments can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can be referred to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0117] In the embodiments of the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. The embodiments described above are merely specific implementation manners of the present application, and for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the among different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electric, mechanical or other forms.

[0118] For another example, the flowcharts and block diagrams in the drawings show the possible implementation architectures, functions and operations of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders from that shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0119] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0120] In addition, each functional unit in the embodiments of the present application can be integrated into one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated into one unit.

[0121] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for making an electronic device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the unsupervised hyperspectral image reconstruction method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0122] It should be noted that: similar labels and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings, in addition, the terms "first", "second", "third" and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0123] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them, the protection scope of the present application is not limited thereto, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art within the technical scope disclosed by the present application can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features thereof; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application. All should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for unsupervised reconstruction of hyperspectral images, characterized in that, The method comprises the following steps: obtaining a training data set and a test data set, wherein the training data set comprises hyperspectral image samples and first compressed image samples corresponding to the hyperspectral image samples, and the test data set comprises second compressed image samples; training an original auxiliary network by using the training data set to obtain an auxiliary network, wherein the original auxiliary network is a residual transformer network; performing inference on the encoding of a coded aperture imaging system and the second compressed image samples by using the auxiliary network to obtain an auxiliary image; constructing a hyperspectral image reconstruction problem based on a multi-branch deep image prior network based on the encoding, the second compressed image samples and the auxiliary image, and iteratively solving the hyperspectral image reconstruction problem based on the multi-branch deep image prior network by using an alternating direction multiplier method to obtain a reconstructed hyperspectral image corresponding to the second compressed image samples, wherein each branch deep image prior network in the multi-branch deep image prior network is a tensor subspace decomposition model with different ranks.

2. The method of claim 1, wherein, The method comprises the following steps: The hyperspectral image reconstruction problem based on the multi-branch deep image prior network comprises: wherein, , , , represents a hyperparameter, and represents a degradation constraint term, represents a data consistency term, represents an image prior constraint term applied to , represents an external data constraint term applied to , represents the second compressed image sample, represents a hyperspectral image to be reconstructed, represents a degradation operator, represents an output of the multi-branch deep image prior network, represents an auxiliary variable, represents values of respective input variables when the objective function takes a global minimum value, , , represents the auxiliary image, , , respectively represent optimal , , , represents a three-dimensional total variation constraint.

3. The method of claim 2, wherein, iteratively solving the hyperspectral image reconstruction problem based on the multi-branch deep image prior network by using an alternating direction multiplier method, comprising: To solved iteratively, resulting in equation, where denotes the hyperspectral image to be reconstructed; To solving iteratively, we obtain equation, where denotes the output of the multi-branch depth image prior network. right By performing iterative solutions, we obtain The formula, in which, This represents an auxiliary variable.

4. The method of claim 3, wherein right When performing iterative solutions, the hyperspectral image reconstruction problem based on a multi-branch deep image prior network can be expressed as follows: Corresponding The formula is ,in, Indicates a degradation constraint term. Indicates data consistency items, This represents the second compressed image sample. This indicates the hyperspectral image that needs to be reconstructed. Indicates the degenerate operator, This represents the output of the multi-branch depth image prior network. Represents auxiliary variables. In the (k+1)th iteration The vector form, , In the (k+1)th iteration The value, This represents tensor vectorization. In the (k+1)th iteration The value, In the (k+1)th iteration The value, This indicates the encoding. express The vector form, This indicates that the two matrices are divided element by element. This means extracting the diagonal elements of a matrix and forming a vector. Indicates hyperparameters; right When performing iterative solutions, the hyperspectral image reconstruction problem based on a multi-branch deep image prior network can be expressed as follows: Corresponding The solution is learned using the multi-branch depth image prior network, where the loss function of the j-th branch depth image prior network is... , Indicates a degradation constraint term. Indicates data consistency items, Indicates to The applied image prior constraints, Indicates to External data constraints imposed, , , , Indicates hyperparameters, This represents the second compressed image sample. This indicates the hyperspectral image that needs to be reconstructed. Indicates the degenerate operator, This represents the output of the multi-branch depth image prior network. Represents auxiliary variables. , , This refers to the auxiliary image. , This represents the output of the depth image prior network in the j-th branch. This represents the spatial coefficient decoder in the depth image prior network of the j-th branch. Represents the product of tensor modulo 3. This represents the spectral basis decoder in the depth image prior network of the j-th branch. This represents the weight of the spatial coefficient decoder in the depth image prior network of the j-th branch. This represents the weights of the spectral basis decoder in the depth image prior network of the j-th branch. and This represents random Gaussian white noise input to the prior network of the depth image in the j-th branch. Let represent the rank of the depth image prior network in the j-th branch. Let's represent a 3rd-order tensor whose dimensions are of size . , , , Represent a OK The matrix of columns is used to train J branch depth image prior networks simultaneously, and the output of each branch depth image prior network is obtained after the branch depth image prior network converges. formula , denotes the value of in the k+1th iteration, , denotes the weight, denotes the value of in the k+1th iteration, denotes the value of in the k+1th iteration; right When performing iterative solutions, the hyperspectral image reconstruction problem based on a multi-branch deep image prior network can be expressed as follows: Corresponding The formula is ,in, Indicates data consistency items, This indicates the hyperspectral image that needs to be reconstructed. This represents the output of the multi-branch depth image prior network. Represents auxiliary variables. In the (k+1)th iteration The value, In the k-th iteration The value, In the (k+1)th iteration The value, In the (k+1)th iteration The value of .

5. The method of claim 1, wherein, each branch deep image prior network comprises a spatial coefficient decoder and a spectral basis decoder, wherein the spatial coefficient decoder comprises a hybrid self-attention decoding module, a first bilinear upsampling layer and a first projection layer; the hybrid self-attention decoding module comprises a plurality of first convolutional blocks connected in series, a channel semantic extraction module connected to the last first convolutional block in the plurality of first convolutional blocks, and a spatial semantic extraction module connected to the last first convolutional block. The spectral basis decoder comprises a second convolutional block, a second bilinear upsampling layer and a second projection layer.

6. The method of claim 1, wherein, The first compressed image samples and the second compressed image samples are obtained by the coded aperture imaging system.

7. An apparatus for unsupervised reconstruction of hyperspectral images, characterized in that, The method comprises the following steps: an obtaining unit configured to obtain a training data set and a test data set, wherein the training data set comprises hyperspectral image samples and first compressed image samples corresponding to the hyperspectral image samples, and the test data set comprises second compressed image samples; a training unit configured to train an original auxiliary network by using the training data set to obtain an auxiliary network, wherein the original auxiliary network is a residual transformer network; an inference unit configured to perform inference on the encoding of a coded aperture imaging system and the second compressed image samples by using the auxiliary network to obtain an auxiliary image; an obtaining unit configured to obtain a training data set and a test data set, wherein the training data set comprises hyperspectral image samples and first compressed image samples corresponding to the hyperspectral image samples, and the test data set comprises second compressed image samples; The constructing and iterative solving unit is configured to construct a hyperspectral image reconstruction problem based on a multi-branch deep image prior network based on the encoding, the second compressed image sample and the auxiliary image, and iteratively solve the hyperspectral image reconstruction problem based on the multi-branch deep image prior network by using an alternating direction method of multipliers to obtain a reconstructed hyperspectral image corresponding to the second compressed image sample, wherein each branch deep image prior network in the multi-branch deep image prior network is a tensor subspace decomposition model with different ranks.

8. The apparatus of claim 7, wherein, The constructing and iterative solving unit is further configured to: The hyperspectral image reconstruction problem based on a multi-branch deep image prior network includes: ,in, , , , Indicates hyperparameters, and This represents a degradation constraint term. Indicates data consistency items, Indicates to The applied image prior constraints, Indicates to External data constraints imposed, This represents the second compressed image sample. This indicates the hyperspectral image that needs to be reconstructed. Indicates the degenerate operator, This represents the output of the multi-branch depth image prior network. Represents auxiliary variables. This represents the values ​​of each input variable when the objective function reaches its global minimum. , , This refers to the auxiliary image. , , These represent the optimal solutions in the hyperspectral image reconstruction problem. , , , This represents a three-dimensional total variation constraint.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores machine executable instructions which, when invoked and run by a processor, cause the processor to perform the method of any one of claims 1 to 6.

Images