Hyperspectral image compression and restoration method and device thereof

Abstract

The application relates to the technical field of image processing, and provides a hyperspectral image compression and recovery method and device thereof, the method comprising the following steps: obtaining compressed two-dimensional detection values by using a dual-camera spectral imaging system; the compressed two-dimensional detection values are obtained by spatially modulating incident light through a preset coded aperture mask; inputting the compressed two-dimensional detection values into an initial recovery model to obtain an initial recovery result; the initial recovery model adopts a deep neural network architecture and is used for extracting image features from the compressed two-dimensional detection values to obtain the initial recovery result; inputting the initial recovery result into an optimization model based on an alternating direction multiplier method to obtain a target recovery image; wherein the optimization model introduces non-local low rank, total variation regularization and a depth image priori. The initial recovery result is optimized based on multiple priori information, the authenticity and accuracy of the initial recovery result are significantly improved, and a high-quality image recovery effect is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a hyperspectral image compression and restoration method and device thereof. BACKGROUND

[0002] As a cutting-edge optical detection method, hyperspectral imaging technology has important application value in remote sensing detection, precision agriculture, environmental monitoring, industrial sorting and medical diagnosis. Compared with traditional RGB three-channel imaging methods, hyperspectral imaging systems can obtain the complete continuous spectral characteristics of the measured object, and have significant technical advantages in material composition analysis and subtle feature recognition. Especially in target detection and classification tasks, the rich spectral-spatial joint features contained in the hyperspectral data cube can effectively improve the ability to distinguish similar substances in complex scenes.

[0003] However, there are still problems to be solved in hyperspectral imaging technology, mainly in the aspects of poor restored image quality and excessive sensitivity to detection noise. At present, although a variety of hyperspectral image compression and restoration methods have been proposed, in terms of simultaneously maintaining spectral dimension accuracy and spatial structure integrity, the existing technology still cannot meet the needs of high-precision applications. SUMMARY

[0004] The present application provides a hyperspectral image compression and restoration method and device, which can accurately restore complete information of high-quality three-dimensional hyperspectral data cube from two-dimensional signals, and realize hyperspectral image reconstruction.

[0005] The present application provides a hyperspectral image compression and restoration method, comprising: obtaining compressed two-dimensional detection values by using a dual-camera spectral imaging system; wherein the compressed two-dimensional detection values are obtained by spatially modulating incident light by a pre-set coded aperture mask; inputting the compressed two-dimensional detection values into an initial restoration model to obtain an initial restoration result output by the initial restoration model; wherein the initial restoration model adopts a deep neural network architecture, and is used to extract image features from the compressed two-dimensional detection values to obtain the initial restoration result; inputting the initial restoration result into an optimization model based on an alternating direction multiplier method to obtain a target restoration image output by the optimization model; wherein the optimization model introduces non-local low-rankness, total variation regularization and depth image priori, and the objective function of the optimization model is:

[0006] ;

[0007] wherein, represents an estimated value of the original signal of the measured object , represents an estimated value of the non-local similar tensor group corresponding to the non-local similar tensor group to be solved, and respectively represent a first detection value and a second detection value in compressed two-dimensional detection values, represents a measurement matrix corresponding to the first detection value, represents a measurement matrix corresponding to the second detection value, represents M nearest neighbor non-local similar tensor groups centered on , represents an original three-dimensional data cube , i represents the first tensor block after k-nearest neighbor operation, represents a regularization term of non-local low rank, represents a depth image prior regularization term, represents a total variation regularization term, , , , and respectively represent different regularization coefficients.

[0008] According to the high-spectral image compression and recovery method provided by the application, the optimization model is used to: based on the initial recovery result, the K-nearest neighbor algorithm is used to obtain the non-local similar tensor group; based on the non-local similar tensor group, the low rank tensor group is determined by using the re-weighted kernel norm minimization; and the low rank tensor group is iteratively solved in the framework of the alternating direction multiplier method to obtain the optimization result as the target recovery image.

[0009] According to the high-spectral image compression and recovery method provided by the application, the double-camera type spectral imaging system is used to obtain the compressed two-dimensional detection value, which comprises: the double-camera type spectral imaging system, on one light path, a preset coded aperture mask is placed in the light path to modulate the incident light, and the information of the three-dimensional spectral data cube is compressed to the detector to obtain the first detection value; and on the other light path, the second detection value is obtained by directly imaging on the detector.

[0010] According to the high-spectral image compression and recovery method provided by the application, the initial recovery model is a self-supervised neural network for realizing depth image prior, and the self-supervised neural network adopts U-net network, residual network or Transformer network.

[0011] According to the high-spectral image compression and recovery method provided by the application, the U-net network comprises: a convolutional encoder module for extracting image features from the compressed two-dimensional detection value, comprising a plurality of convolutional layers, batch normalization layers and activation functions; a down-sampling module for reducing the spatial dimension of the feature map, comprising a convolutional layer and a maximum pooling layer; an up-sampling module for recovering the spatial resolution of the feature map, comprising a convolutional layer and a convolutional transpose layer; and an output layer for outputting the initial recovery result, comprising a convolutional layer and an activation function.

[0012] The hyperspectral image compression and recovery method provided by the application further comprises: processing the compressed two-dimensional detection value through deep integrated image prior information.

[0013] The application further provides a hyperspectral image compression and recovery device, comprising: a detection value acquisition module, configured to obtain a compressed two-dimensional detection value by using a dual-camera spectral imaging system; wherein the compressed two-dimensional detection value is obtained by spatially modulating incident light through a preset coded aperture mask; an initial recovery module, configured to input the compressed two-dimensional detection value into an initial recovery model to obtain an initial recovery result output by the initial recovery model; wherein the initial recovery model adopts a deep neural network architecture and is configured to extract image features from the compressed two-dimensional detection value to obtain the initial recovery result; and an optimized recovery module, configured to input the initial recovery result into an optimization model based on an alternating direction multiplier method to obtain a target recovery image output by the optimization model; wherein the optimization model introduces non-local low-rankness, total variation regularization and deep image prior, and the objective function of the optimization model is as follows:

[0014] ;

[0015] wherein, represents an estimated value of an original signal of a measured object, represents an estimated value of a non-local similar tensor group corresponding to a non-local similar tensor group to be solved, and respectively represent a first detection value and a second detection value in the compressed two-dimensional detection value, represents a measurement matrix corresponding to the first detection value, represents a measurement matrix corresponding to the second detection value, represents M nearest neighbor non-local similar tensor groups centered on represents an original three-dimensional data cube represents the k-th tensor block after k-nearest neighbor operation, represents a regularization term of non-local low-rankness, represents a deep image prior regularization term, represents a total variation regularization term, i , , , and respectively represent different regularization coefficients.

[0016] ​​​​The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the hyperspectral image compression and recovery method according to any one of the above when executing the program.

[0017] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the hyperspectral image compression and recovery method according to any one of the above.

[0018] The application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the hyperspectral image compression and recovery method according to any one of the above.

[0019] The hyperspectral image compression and recovery method and device provided by the application comprise the following steps: obtaining compressed two-dimensional detection values by using a dual-camera spectral imaging system; the compressed two-dimensional detection values are obtained by spatially modulating incident light by using a preset coded aperture mask; inputting the compressed two-dimensional detection values into an initial recovery model to obtain an initial recovery result; the initial recovery model adopts a deep neural network architecture and is used for extracting image features from the compressed two-dimensional detection values to obtain the initial recovery result; inputting the initial recovery result into an optimization model based on an alternating direction multiplier method to obtain a target recovery image; and the optimization model introduces non-local low-rankness, total variation regularization and depth image priori. Through the above method, the multi-priori fusion scheme introduced by the application optimizes the initial recovery result based on multiple priori information by introducing non-local low-rankness, total variation regularization and depth image priori, significantly improves the authenticity and accuracy of the initial recovery result, and realizes high-quality image recovery effect. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0021] Figure 1 is a flowchart of the hyperspectral image compression and recovery method provided by the embodiment of the present application.

[0022] Figure 2 is an optical path schematic diagram provided by the embodiment of the present application.

[0023] Figure 3 is a schematic diagram of the light modulation process provided by the embodiment of the present application.

[0024] Figure 4Fig. 1 is a structural schematic diagram of a hyperspectral image compression and recovery device provided by an embodiment of the present application.

[0025] Figure 5 Fig. 2 is a physical structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0027] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0028] Hyperspectral imaging technology includes various implementation means, such as traditional scanning imaging technology and coded aperture snapshot spectral imaging technology based on compressive sensing theory.

[0029] Compared with traditional scanning imaging technology, coded aperture snapshot spectral imaging technology based on compressive sensing theory compresses a three-dimensional spectral data cube into a two-dimensional signal for detection through front-end optical coding, and decodes in the rear end using a recovery algorithm, which can realize snapshot imaging and greatly reduce the detection time.

[0030] At present, hyperspectral image reconstruction methods mainly fall into two categories: one is a method based on traditional algorithms, including least square method, dictionary learning and regularization constraint and other technical means; the other is a method based on deep learning, such as an end-to-end reconstruction model based on convolutional neural network, high-fidelity image generation using generative adversarial network, feature enhancement network combined with attention mechanism and long-range dependence modeling based on Transformer architecture.

[0031] Traditional algorithms have certain mathematical theoretical basis, but there are many problems in processing hyperspectral data of complex scenes. For example, traditional algorithms are sensitive to detection noise, resulting in low quality of restored images. In addition, deep learning methods provide new solutions in the field of hyperspectral image reconstruction, but also have some limitations, such as poor model generalization ability, lack of interpretability, and large consumption of computing resources.

[0032] To solve the above problems, a method combining deep learning and traditional algorithms can be used to take full advantage of both. However, this combined method still faces some problems in hyperspectral image restoration, mainly as follows:

[0033] 1. Optimization framework is complex: the method of combining traditional optimization algorithms with deep learning models results in a multi-level, multi-objective optimization framework. Compared with the single technology route method, the optimization objective function, solution strategy and convergence analysis are all qualitatively complicated.

[0034] 2. Image feature fusion is difficult: this method usually embeds a deep learning module as a black box feature extractor into a traditional optimization framework, or only uses traditional algorithms as a pre-processing / post-processing step for deep networks. It fails to build a joint representation model that truly reflects the low-rank nature, spectral continuity and spatial-spectral structure relationship of hyperspectral data. Especially when dealing with spectral dimension information, existing hybrid methods have difficulty in balancing the relationship between physical meaning preservation and deep feature learning, resulting in deviation in spectral accuracy of the restored results, which cannot meet the application requirements of high-precision spectral analysis.

[0035] In summary, related technologies still have deficiencies in hyperspectral image restoration. The embodiments of the present application provide a hyperspectral image compression and restoration method, which introduces a multi-prior architecture based on traditional algorithms and deep learning to achieve higher precision restoration effect. Specifically, non-local low-rank and deep image prior based on deep learning can be used to capture the non-local similarity and local similarity of hyperspectral images respectively, and TV (Total Variation) regularization in the spectral dimension is added as a spectral continuity prior. Combined with the ADMM (Alternating Direction Method of Multipliers) framework for iterative solution, a high-quality restored hyperspectral image is obtained.

[0036] Please refer to Figure 1 , Figure 1 is a flowchart of the hyperspectral image compression and restoration method provided by the embodiments of the present application. In this embodiment, the hyperspectral image compression and restoration method can include steps S110 to S130, and each step is as follows:

[0037] S110: obtaining compressed two-dimensional detection values by using a dual-camera spectral imaging system; wherein the compressed two-dimensional detection values are obtained by spatial modulation of incident light by a preset coded aperture mask.

[0038] In the dual-camera spectral imaging system, one light path can be obtained by spatial and spectral modulation of incident light by the preset coded aperture mask and the dispersive element, and the other light path can be obtained by direct imaging on the detector plane.

[0039] In this embodiment, the dual-camera spectral imaging system can be used to obtain compressed two-dimensional detection values. This process uses a preset spectral coded aperture mask to spatially modulate incident light, thereby obtaining compressed two-dimensional detection values containing spectral information and spatial information.

[0040] The dual-camera spectral imaging system can obtain spectral information and spatial information from different angles or different spectral bands to obtain hyperspectral image data, laying a foundation for subsequent high-quality image restoration. Compared with a single-camera system, the dual-camera spectral imaging system can provide more comprehensive and rich raw data, which helps to improve the accuracy and quality of image restoration.

[0041] The coded aperture is an optical element for spatial modulation of incident light, including but not limited to digital micromirror array, liquid crystal on silicon, and quartz mask. Specifically, the coded aperture transmits / reflects incident light through a specific pattern into the subsequent optical system or absorbs / reflects it so that it does not enter the subsequent optical system, thereby performing spatial modulation, and combining with the dispersive element to achieve the final spatial and spectral modulation.

[0042] In some embodiments, obtaining compressed two-dimensional detection values by using a dual-camera spectral imaging system includes: a dual-camera spectral imaging system, on one light path, placing a preset coded aperture mask in the light path to modulate incident light, compressing information of a three-dimensional spectral data cube onto a detector to obtain first detection values, and on the other light path, directly imaging on the detector to obtain second detection values.

[0043] In this embodiment, the processed compressed two-dimensional measurement data has a clear physical acquisition path, i.e., spatial modulation of incident light by a specifically designed spectral coded aperture mask, followed by controlled displacement in the spectral dimension using a precision disperser, and finally obtaining compressed measurement values by integration operation.

[0044] S120: inputting the compressed two-dimensional detection values into an initial restoration model to obtain an initial restoration result output by the initial restoration model; wherein the initial restoration model adopts a deep neural network architecture for extracting image features from the compressed two-dimensional detection values to obtain the initial restoration result.

[0045] In this step, the compressed two-dimensional detection value is input into an initial restoration model based on a deep neural network architecture. The deep neural network has strong feature extraction and learning ability, which can automatically mine the key features of the image from the compressed two-dimensional detection value and convert them into a preliminary image restoration result. The initial restoration model can quickly generate an initial restoration image that is close to the original image.

[0046] In this step, the initial restoration result is input into an optimization model based on the alternating direction multiplier method (ADMM), and a target restoration image output by the optimization model is obtained. The optimization model introduces non-local low-rankness, total variation regularization, and depth image prior.

[0047] In this step, the initial restoration result is input into an optimization model based on the alternating direction multiplier method (ADMM), and a target restoration image output by the optimization model is obtained. The optimization model introduces non-local low-rankness, total variation regularization, and depth image prior.

[0048] The ADMM method can decompose a complex optimization problem into multiple relatively simple sub-problems, and gradually approach the optimal solution by alternately solving these sub-problems. It has high efficiency and stability in dealing with image restoration problems with multiple constraints and prior information.

[0049] Non-local low-rankness is a prior characteristic of images, which means that there are a large number of non-local similar structures in images, and these similar structures exhibit low-rank characteristics under certain transformation domain. This embodiment uses the non-local low-rankness prior to mine the repeated patterns and structural information in the image during the image restoration process, thereby improving the quality and accuracy of the restored image.

[0050] Total variation regularization constrains the gradient information of the image, so that the restored image reduces noise and artifacts while preserving edge information. The basic idea is to assume that the gradient distribution of the image is relatively sparse, i.e. the gradient of most regions in the image is small, while the gradient of the edge region is large. Through total variation regularization, the edge details of the image can be enhanced during the optimization process, improving the clarity and visual effect of the image.

[0051] The depth image prior is prior information based on image features and statistical rules learned by a deep learning model. The depth image prior reflects the internal characteristics and structural information of an image in a deep neural network representation, such as local features, global features, and correlations between different features. In image restoration, the depth image prior can be used to better guide the restoration process, making the restored image more consistent with the statistical rules of natural images and improving the quality and authenticity of the restoration.

[0052] Referring to Figure 2 , Figure 2 is a light path schematic diagram provided by an embodiment of the present application.

[0053] The light path diagram includes an object 210, a beam splitter 220, an objective lens 230, a coded aperture 240, a relay lens 250, a dispersive element 260, a first detector 270, and a second detector 280.

[0054] The coded aperture 240 is the preset coded aperture mask described above, and the first detector 270 and the second detector 280 are used to obtain first detection values and second detection values, respectively.

[0055] Exemplarily, when using CASSI (Compressive Spectral Snapshot Imaging) to obtain compressed two-dimensional detection values, the detection model can be represented as follows:

[0056] ;

[0057] wherein, represents the original signal of the measured object, and represent the CASSI branch and the grayscale branch detection values, respectively, and represent the measurement matrices of the CASSI branch and the grayscale branch, respectively, and represent the detection noises of the CASSI branch and the grayscale branch.

[0058] Referring to Figure 3 , Figure 3 is a schematic diagram of a light modulation process provided by an embodiment of the present application.

[0059] Specifically, the three-dimensional spectral data cube 310 obtained from the above object is processed by the coded aperture 320 to obtain a spatially encoded three-dimensional spectral data cube 330, and then processed by the dispersive element 340 to obtain a spatially encoded and dispersed three-dimensional spectral data cube 350. After being collected by the detector, the detection result 360 is obtained.

[0060] The target function can be specifically as follows after introducing the non-local low-rank property, total variation regularization and depth image prior in the optimization model:

[0061] ;

[0062] wherein, represents an estimated value of an original signal of a measured object, represents an estimated value of a non-local similar tensor group corresponding to a non-local similar tensor group to be solved, and respectively represent a first detection value and a second detection value in compressed two-dimensional detection values, represents a measurement matrix corresponding to the first detection value, represents a measurement matrix corresponding to the second detection value, represents M nearest neighbor non-local similar tensor groups centered on represents the kth tensor block after k-nearest neighbor operation of the original three-dimensional data cube represents a regularization term of the non-local low-rank property, represents a depth image prior regularization term, represents a total variation regularization term, , i , , and respectively represent different regularization coefficients. wherein,

[0063] In order to optimize the above formula, the target function can be decomposed into two sub-problems, a sub-problem and an x sub-problem.

[0064] For the sub-problem, the formula to be solved is as follows:

[0065] ;

[0066] wherein, represents a reweighted kernel norm of , represents unfolded in the third dimension, and k and m respectively represent the row number and column number of .

[0067] represents a reweighting coefficient.

[0068] represents a singular value of , represents a kernel norm.​​​​

[0069] It should be noted that in the formula expressions mentioned in the embodiments of the present invention, bold lowercase letters can represent vectors, bold uppercase letters can represent matrices, cursive letters can represent tensors, lowercase unbold letters can represent scalars, ^ on the letters represents estimated values, and (3) in the upper right corner represents expansion along the third dimension.

[0070] Based on this, The solution can be approximated using a global soft thresholding algorithm:

[0071] ;

[0072] ;

[0073] .

[0074] in, yes The singular value decomposition of the Singular Value Decomposition, this formula can explain The meaning of these three matrices, T Represents transpose. yes A matrix expanded along the third dimension, Represents the calculated The estimated singular values. σ ij It is Σ i The j-th singular value, epsilon It is a normal number.

[0075] The subproblem f can be represented as:

[0076] ;

[0077] To achieve better optimization, the ADMM framework can be used to introduce auxiliary variables. Lagrange multipliers Balance coefficient eta The optimization problem is transformed into:

[0078]

[0079] Where Θ represents the network parameters of U-net, Representative and For random vectors of the same input size, this formula can be used as the loss function for backpropagation to optimize the algorithm. Representative and Random vectors of the same size are used as input to the network, and the network can be optimized by backpropagation of the back gradient using Equation (1) as the loss function.

[0080] the value of at the tth iteration, Again, using the ADMM framework, introduce auxiliary variables and Lagrange multipliers , balance the coefficients lambda , and convert the optimization problem into two quadratic optimization problems, whose solutions can be expressed as

[0081]

[0082]

[0083]

[0084] where is the value of at the tth iteration, The solution problem of is a quadratic optimization problem, which can be obtained by

[0085]

[0086] where are the matrix forms of , respectively, is the selection tensor of the tth block, i is the transpose of the selection tensor of the tth block, is the expansion of i , and . At this time, in order to solve , again use the ADMM framework to introduce auxiliary variables and Lagrange multipliers , balance the coefficients , and convert the optimization problem into two quadratic optimization problems, whose solutions can be expressed as mu

[0087]

[0088]

[0089]

[0090] where , , are intermediate variables, is the value of at the tth iteration, and can be solved element by element, and m represents and ​​​​​​​​​​​​​The solution for the m-th element is:

[0091] ;

[0092] ;

[0093] The solution can be obtained using the following formula:

[0094] .

[0095] To quantitatively analyze the reconstruction effect, the Spectral Angle Mapper (SAM), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index Measure (SSIM) of the image were calculated, and their definitions are as follows:

[0096] ;

[0097] ;

[0098] .

[0099] in, N Represents the total number of pixels. C Represents the number of spectral channels. and Representing the original spectral data cube and the reconstructed spectral data cube respectively. i The intensity of each wavelength, where MAX represents the maximum value in the original spectral data cube. and Representing the original data cube and the reconstructed spectral data cube respectively. i The average of each wavelength, Representing the original spectral data cube and the reconstructed spectral data cube i The covariance of each wavelength, and Representing the original spectral data cube and the reconstructed spectral data cube respectively. i The variance of each wavelength, c 1 and c 2 represents a constant.

[0100] In some embodiments, the optimization model is configured to: obtain a non-local similar tensor group based on the initial restoration result by using a K-nearest neighbor algorithm; determine a low-rank tensor group based on the non-local similar tensor group by using a reweighted nuclear norm minimization; and obtain an optimization result as a target restoration image by iteratively solving in a framework of an alternating direction method of multipliers based on the low-rank tensor group.

[0101] In this embodiment, the optimization model first uses the K-nearest neighbor algorithm to capture the non-local similarity in the hyperspectral image, that is, by analyzing each tensor block in the initial restoration result, it finds similar structures in the spatial or spectral dimensions, and groups these similar structures into a non-local similar tensor group. Non-local similarity is based on the repeated patterns and textures in the hyperspectral image, which may have similar features in different regions or spectral bands of the image. Through the capture of this similarity, the model can understand the structure of the image globally.

[0102] The optimization model processes the non-local similar tensor group through the reweighted nuclear norm minimization technique to determine a low-rank tensor group. The reweighted nuclear norm minimization can effectively highlight the main, low-rank features while suppressing noise and unimportant information. The low-rank tensor group represents the most important features and structures in the image, which not only contains spatial information but also spectral information.

[0103] Further, the optimization model iteratively solves in the framework of the alternating direction method of multipliers (ADMM) to obtain an optimization result as a target restoration image. ADMM is an efficient optimization algorithm that gradually approaches the optimal solution by alternately optimizing subproblems and updating multipliers. In this case, the ADMM framework can effectively process the low-rank tensor group, further exploiting the potential information in the image while maintaining the continuity of the spectral dimension and the integrity of the spatial dimension. This iterative solving process not only improves the accuracy of image restoration, but also enhances the authenticity and visual quality of the image.

[0104] The above process captures and processes non-local similarity and low-rank features, enabling the optimization model to effectively reduce noise, enhance image details and edge information, and achieve high-precision image restoration. The final optimization result is the target restoration image, which has significantly improved quality in both spectral and spatial dimensions.

[0105] In some embodiments, the initial restoration model is a self-supervised neural network for implementing depth image prior, and the self-supervised neural network adopts a U-net network, a residual network or a Transformer network.

[0106] In this embodiment, the self-supervised neural network for implementing depth image prior includes but is not limited to a U-net network, a residual network, a Transformer network or other combinations.

[0107] In some embodiments, the U-net network comprises: a convolutional encoder module for extracting image features from the compressed two-dimensional detection values, comprising a plurality of convolutional layers, batch normalization layers and activation functions; a down-sampling module for reducing the spatial dimension of the feature map, comprising a convolutional layer and a maximum pooling layer; an up-sampling module for restoring the spatial resolution of the feature map, comprising a convolutional layer and a convolutional transpose layer; and an output layer for outputting the initial restoration result, comprising a convolutional layer and an activation function.

[0108] The convolutional encoder module is configured to extract low-level image features from the compressed two-dimensional detection values, and comprises a plurality of convolutional layers, batch normalization layers and activation functions.

[0109] The down-sampling module is configured to gradually reduce the spatial dimension of the feature image, effectively capturing multi-scale global context information and abstract features of the hyperspectral image, and comprises a convolutional layer and a maximum pooling layer.

[0110] The up-sampling module is configured to gradually restore the spatial resolution of the feature map by using techniques such as transposed convolution, and comprises a convolutional layer and a convolutional transpose layer.

[0111] The output layer is configured to map the high-dimensional features extracted by the previous hidden layer to a dimensional space consistent with the target hyperspectral data, and comprises a convolutional layer and an activation function.

[0112] It should be noted that the self-supervised neural network implementing the depth image prior can be used in both the initial restoration model and the optimization model, for example, the U-net network. The structures of the U-net networks corresponding to the two models can be the same, but the model parameters and input parameters are different.

[0113] In some embodiments, the hyperspectral image compression and restoration method further comprises: processing the compressed two-dimensional detection values by using the depth-integrated image prior information to reduce the noise introduced in the compression sampling process.

[0114] In the compression sampling process, due to the modulation of the encoding aperture mask on the incident light and the physical limitations of the detector, noise will be introduced when the compressed two-dimensional detection values are obtained. Therefore, in the image restoration process, the depth-integrated image prior information can be used to reduce the noise. Optionally, the noise filtering can be set in the initial restoration model and / or the optimization model.

[0115] The above, the embodiment of the application proposes an innovative hyperspectral image restoration technical scheme, the core of which is to construct a multi-prior hybrid architecture, organically integrating the theoretical guarantee of traditional algorithms and the representation ability of deep learning. Specifically, the application uses the non-local low-rank principle to accurately capture the non-local similarity characteristics of hyperspectral data, while introducing a depth image prior network to extract image local structure features, and introducing a spectral dimension total variation regularization term to constrain spectral continuity. The above multiple priors are optimized by an improved alternating direction multiplier method optimization framework, realizing high-quality hyperspectral image restoration and effectively solving the key problems in the prior art.

[0116] The above embodiments can be freely combined without conflict. For example, a multi-prior fusion hyperspectral image compression and restoration method can be implemented, comprising:

[0117] The compressed two-dimensional detection value is obtained by using a dual-camera coded aperture snapshot spectral imaging system; the initial restoration result is obtained by using a U-net-based deep image prior self-supervised neural network, and then the initial result is input into the optimization step under the ADMM framework for iterative optimization to finally output the optimization result.

[0118] Above, an organic fusion strategy of multiple complementary prior knowledge is adopted: the non-local low-rank tensor prior effectively captures the long-distance spatial feature association by exploring the similarity structure between hyperspectral data blocks; the deep image prior adaptively learns complex spatial textures and edge details with the help of a carefully designed U-Net network architecture; and the total variation regularization term focuses on maintaining the smooth transition characteristics in the spectral dimension and suppressing non-physical noise interference. This multi-scale collaborative optimization mechanism of spatial dimension and spectral dimension, local information and global structure significantly improves the authenticity and accuracy of the reconstruction result, overcoming the inherent limitations of single prior method.

[0119] To verify the effectiveness of the method of the application, the following two methods are used for comparison: 1. only using deep image prior; 2. using deep image prior and non-local similarity.

[0120] The results show that when only using deep image prior for restoration, the restoration result can only obtain rough spatial structure information, and the accuracy of spectral information restoration is very low; when using the method combining deep image prior and non-local similarity, the spatial structure is better reconstructed, but due to the noise existing in the experiment, the spectral continuity of the reconstruction result is poor. Compared with the above, the method proposed in the embodiment of the application not only obtains better restoration result in the spatial dimension, but also better preserves high-frequency information, and effectively ensures the spectral continuity by suppressing noise interference in the spectral information restoration.

[0121] Therefore, compared with the related art, the application has the advantages that:

[0122] 1. The multi-prior fusion scheme provided by the application effectively improves the accuracy of hyperspectral image restoration by introducing non-local low rank, total variation regularization and depth image prior, and has better image reconstruction effect.

[0123] 2. The global correlation structure in the hyperspectral image is effectively captured, the local processing limitation in the related art is overcome, the inherent redundancy feature of the hyperspectral data in the spatial-spectral dimension is fully utilized, and there is a solid mathematical theoretical basis, which provides reliable convergence guarantee, and the image restoration under the sparse sampling condition shows excellent anti-noise ability.

[0124] 3. Without large-scale external training data, the overfitting and generalization problems are avoided, the complex spatial-spectral structure features of the hyperspectral image are adaptively learned, high-quality reconstruction results are provided through end-to-end optimization, non-linear interference and complex noise patterns are effectively dealt with, and strong robustness is shown in actual application.

[0125] 4. The continuity constraint between spectral channels is accurately maintained, which conforms to the physical characteristics, effectively suppresses artifacts and noise while preserving important edge information, has clear physical interpretation, enhances the reliability of the model, has moderate computational complexity, and is convenient for engineering implementation.

[0126] 5. Multi-scale joint optimization of local-global and spatial-spectral dimensions is realized, the reconstruction quality under low signal-to-noise ratio conditions is significantly improved, stronger generalization ability is possessed, and the model is suitable for multiple acquisition conditions and application scenarios, the synergistic complementary effect is obvious, and the inherent defects of a single prior method are overcome.

[0127] On the other hand, the embodiment of the application also provides a hyperspectral image compression and restoration device, and the hyperspectral image compression and restoration device provided by the application is described below, and the hyperspectral image compression and restoration device described below can be correspondingly referred to the hyperspectral image compression and restoration method described above.

[0128] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of the hyperspectral image compression and restoration device provided by the embodiment of the application. In the embodiment, the hyperspectral image compression and restoration device can include a detection value acquisition module 410, an initial restoration module 420 and an optimized restoration module 430.

[0129] The detection value acquisition module 410 is used for obtaining compressed two-dimensional detection values by using a dual-camera spectral imaging system; wherein the compressed two-dimensional detection values are obtained by spatially modulating incident light by a pre-set coded aperture mask;

[0130] The initial recovery module 420 is used to input the compressed two-dimensional probe values ​​into the initial recovery model to obtain the initial recovery result output by the initial recovery model; wherein the initial recovery model adopts a deep neural network architecture to extract image features from the compressed two-dimensional probe values ​​to obtain the initial recovery result;

[0131] The optimization and restoration module 430 is used to input the initial restoration result into the optimization model based on the alternating direction multiplier method to obtain the target restored image output by the optimization model. The optimization model incorporates nonlocal low-rank property, total variational regularization, and depth image prior. The objective function of the optimization model is:

[0132] ;

[0133] in, The raw signal representing the object being measured The estimated value, This represents the set of nonlocal similarity tensors that need to be solved. The estimated value, and These represent the first and second probe values ​​in the compressed two-dimensional probe values, respectively. This represents the measurement matrix corresponding to the first detected value. This represents the measurement matrix corresponding to the second detected value. Indicated by For the M nearest nonlocal similar tensors centered at the center, Represents the original three-dimensional data cube After the k nearest neighbor operation, the first i Tensor blocks, The regularization term represents nonlocal low-rank property. This represents the prior regularization term for the depth image. Denotes the total variation regularization term. , , , and These represent different regularization coefficients.

[0134] In some embodiments, the optimization model can be used to: obtain a set of non-local similar tensors using the K-nearest neighbor algorithm based on the initial restoration result; determine a set of low-rank tensors by minimizing the reweighted nuclear norm based on the set of non-local similar tensors; and perform iterative solution based on the set of low-rank tensors within the framework of the alternating direction multiplier method to obtain the optimization result as the target restored image.

[0135] In some embodiments, the probe value acquisition module 410 can be specifically configured to: in a dual-camera spectral imaging system, on one light path, modulate the incident light by placing a preset coded aperture mask in the light path, compress the information of a three-dimensional spectral data cube onto a detector to obtain a first probe value, and on another light path, directly image onto a detector to obtain a second probe value.

[0136] In some embodiments, the initial restoration model is a self-supervised neural network for implementing a depth image prior, and the self-supervised neural network adopts a U-net network, a residual network or a Transformer network.

[0137] In some embodiments, the U-net network comprises: a convolutional encoder module configured to extract image features from the compressed two-dimensional probe value, comprising a plurality of convolutional layers, batch normalization layers and activation functions; a down-sampling module configured to reduce the spatial dimension of the feature map, comprising a convolutional layer and a maximum pooling layer; an up-sampling module configured to restore the spatial resolution of the feature map, comprising a convolutional layer and a convolutional transpose layer; and an output layer configured to output the initial restoration result, comprising a convolutional layer and an activation function.

[0138] In some embodiments, the hyperspectral image compression and restoration apparatus further comprises a noise filtering module, which can be specifically configured to: process the compressed two-dimensional probe value by depth-integrated image prior information.

[0139] On the other hand, the embodiments of the present application also provide an electronic device, please refer to Figure 5 , Figure 5 is the physical structure diagram of the electronic device provided by the embodiments of the present application, as Figure 5 shown, the electronic device can include a memory 520, a processor 510 and a computer program stored on the memory 520 and executable on the processor 510. The processor 510 can implement the hyperspectral image compression and restoration method when executing the program, which can include:

[0140] obtaining a compressed two-dimensional probe value by using a dual-camera spectral imaging system; wherein the compressed two-dimensional probe value is obtained by spatially modulating incident light by a preset coded aperture mask; inputting the compressed two-dimensional probe value into an initial restoration model to obtain an initial restoration result output by the initial restoration model; wherein the initial restoration model adopts a deep neural network architecture, and is configured to extract image features from the compressed two-dimensional probe value to obtain the initial restoration result; inputting the initial restoration result into an optimization model based on an alternating direction multiplier method to obtain a target restoration image output by the optimization model; wherein the optimization model introduces non-local low-rankness, total variation regularization and depth image prior, and the objective function of the optimization model is:

[0141] ;

[0142] wherein, represents the original signal of the measured object , represents the estimated value of the non-local similarity tensor group corresponding to the non-local similarity tensor group to be solved , and respectively represent the first detection value and the second detection value in the compressed two-dimensional detection value, represents the measurement matrix corresponding to the first detection value, represents the measurement matrix corresponding to the second detection value, represents the M nearest neighbor non-local similarity tensor group centered on , represents the original three-dimensional data cube , i the kth tensor block after k-nearest neighbor operation, represents the regularization term of non-local low rank, represents the depth image prior regularization term, represents the total variation regularization term, , , , and respectively represent different regularization coefficients.

[0143] Optionally, the electronic device can further include a communication bus 530 and a communication interface 540, wherein the processor 510, the communication interface 540, and the memory 520 can complete mutual communication through the communication bus 530. The processor 510 can call the computer program in the memory 520 to execute the hyperspectral image compression and recovery method provided by each method.

[0144] In addition, the logic instructions in the memory 520 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0145] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the hyperspectral image compression and recovery method provided by the above-mentioned methods, the steps and principles of which have been described in detail in the above-mentioned methods and will not be repeated here.

[0146] In yet another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the hyperspectral image compression and recovery method provided by the above-mentioned methods, the steps and principles of which have been described in detail in the above-mentioned methods and will not be repeated here.

[0147] The non-transitory computer readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor memory (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid state disk (SSD), etc.).

[0148] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0149] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0150] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A hyperspectral image compression and restoration method, characterized by, The method comprises the following steps: obtaining compressed two-dimensional detection values by using a dual-camera spectral imaging system; wherein the compressed two-dimensional detection values are obtained by spatially modulating incident light through a preset coded aperture mask; inputting the compressed two-dimensional detection values into an initial recovery model to obtain an initial recovery result output by the initial recovery model; wherein the initial recovery model adopts a deep neural network architecture and is used to extract image features from the compressed two-dimensional detection values to obtain the initial recovery result; inputting the initial recovery result into an optimization model based on an alternating direction multiplier method to obtain a target recovery image output by the optimization model; wherein the optimization model introduces non-local low-rankness, total variation regularization and deep image prior, and the objective function of the optimization model is: ； wherein, represents the original signal of the measured object , represents the estimated value of the non-local similarity tensor group corresponding to the non-local similarity tensor group needed to be solved , and respectively represent the first detection value and the second detection value in the compressed two-dimensional detection value, represents the measurement matrix corresponding to the first detection value, represents the measurement matrix corresponding to the second detection value, represents the M nearest non-local similarity tensor groups centered on , represents the original three-dimensional data cube the k-th tensor block after the k-nearest neighbor operation, i , represents the regularization term of non-local low rank, represents the depth image prior regularization term, represents the total variation regularization term, , , , and respectively represent different regularization coefficients. 2.The hyperspectral image compression and restoration method of claim 1, wherein, the optimization model is used to: based on the initial recovery result, a non-local similar tensor group is obtained by using a K-nearest neighbor algorithm; based on the non-local similar tensor group, a low-rank tensor group is determined by using a re-weighted kernel norm minimization; based on the low-rank tensor group, an optimization result is obtained by iterative solving in the framework of the alternating direction multiplier method, and the optimization result is used as the target recovery image. 3.The hyperspectral image compression and restoration method of claim 1, wherein, The method of obtaining compressed two-dimensional detection values by using a dual-camera spectral imaging system comprises the following steps: in the dual-camera spectral imaging system, on one light path, a preset coded aperture mask is placed in the light path to modulate the incident light, and the information of a three-dimensional spectral data cube is compressed onto a detector to obtain the first detection value; on another light path, the second detection value is obtained by directly imaging on the detector. 4.The hyperspectral image compression and restoration method of claim 1, wherein, The initial recovery model is a self-supervised neural network for realizing deep image prior, and the self-supervised neural network adopts a U-net network, a residual network or a Transformer network. 5.The hyperspectral image compression and restoration method of claim 4, wherein, The U-net network comprises: a convolutional encoder module for extracting image features from the compressed two-dimensional detection values, comprising a plurality of convolutional layers, batch normalization layers and activation functions; a down-sampling module for reducing the spatial dimension of a feature map, comprising a convolutional layer and a maximum pooling layer; an up-sampling module for restoring the spatial resolution of the feature map, comprising a convolutional layer and a convolutional transpose layer; an output layer for outputting the initial recovery result, comprising a convolutional layer and an activation function.

6. The hyperspectral image compression and restoration method according to any one of claims 1 to 5, characterized in that, The method further comprises the following steps: processing the compressed two-dimensional detection values by using deep integrated image prior information.

7. A hyperspectral image compression and restoration apparatus, characterized by comprising: The method comprises the following steps: a detection value acquisition module is configured to obtain compressed two-dimensional detection values by using a dual-camera spectral imaging system; wherein the compressed two-dimensional detection values are obtained by spatially modulating incident light through a preset coded aperture mask; an initial recovery module is configured to input the compressed two-dimensional detection values into an initial recovery model to obtain an initial recovery result output by the initial recovery model; wherein the initial recovery model adopts a deep neural network architecture and is used to extract image features from the compressed two-dimensional detection values to obtain the initial recovery result; an optimization recovery module is configured to input the initial recovery result into an optimization model based on an alternating direction multiplier method to obtain a target recovery image output by the optimization model. The optimization model introduces non-local low rank, total variation regularization and depth image prior, and a target function of the optimization model is as follows: ； wherein, represents the original signal of the measured object an estimate of represents an estimate of the non-local similarity tensor group corresponding to the non-local similarity tensor group and respectively represent a first and a second measurement value of the compressed two-dimensional measurement values, represents a measurement matrix corresponding to the first measurement value, represents a measurement matrix corresponding to the second measurement value, represents the M nearest non-local similarity tensor groups centered at , represents an original three-dimensional data cube the k-th tensor block after k-nearest neighbor operation, i represents a regularization term of non-local low-rankness, represents a depth image prior regularization term, represents a total variation regularization term, , , , , and respectively represent different regularization coefficients.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the hyperspectral image compression and recovery method of any one of claims 1 to 6 when executing the computer program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the hyperspectral image compression and recovery method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the hyperspectral image compression and recovery method of any one of claims 1 to 6. The computer program is executed by the processor to implement the hyperspectral image compression and recovery method of any one of claims 1 to 6.

Images