Hyperspectral image compression and recovery method and device

Abstract

The invention relates to the technical field of image processing, and provides a hyperspectral image compression and recovery method and device, and the method comprises the steps: obtaining a compressed two-dimensional detection value through a dual-camera spectral imaging system; the compressed two-dimensional detection value is obtained by spatially modulating the incident light through a preset coding aperture mask; inputting the compressed two-dimensional detection value into an initial recovery model to obtain an initial recovery result; the initial recovery model adopts a deep neural network architecture and is used for extracting image features from the compressed two-dimensional detection value to obtain an initial recovery result; inputting the initial recovery result into an optimization model based on an alternating direction multiplier method to obtain a target recovery image; wherein non-local low rank, total variation regularization and depth image prior are introduced into the optimization model. According to the method, the initial recovery result is optimized based on multiple prior information, the authenticity and accuracy of the initial recovery result are remarkably improved, and the high-quality image recovery effect is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a hyperspectral image compression and restoration method and device thereof. BACKGROUND

[0002] As a cutting-edge optical detection method, hyperspectral imaging technology has important application value in remote sensing detection, precision agriculture, environmental monitoring, industrial sorting and medical diagnosis. Compared with traditional RGB three-channel imaging methods, hyperspectral imaging systems can obtain the complete continuous spectral characteristics of the measured object, and have significant technical advantages in material composition analysis and fine feature recognition. Especially in target detection and classification tasks, the rich spectral-spatial joint features contained in the hyperspectral data cube can effectively improve the ability to distinguish similar substances in complex scenes.

[0003] However, there are still problems to be solved in hyperspectral imaging technology, mainly in the aspects of poor restored image quality and excessive sensitivity to detection noise. At present, although a variety of hyperspectral image compression and restoration methods have been proposed, in terms of simultaneously maintaining the accuracy of the spectral dimension and the integrity of the spatial structure, the existing technology still cannot meet the needs of high-precision applications. SUMMARY

[0004] The present application provides a hyperspectral image compression and restoration method and device, which can accurately restore complete information of high-quality three-dimensional hyperspectral data cube from two-dimensional signals, and realize hyperspectral image reconstruction.

[0005] The present application provides a hyperspectral image compression and restoration method, comprising: obtaining compressed two-dimensional detection values by using a dual-camera spectral imaging system; wherein the compressed two-dimensional detection values are obtained by spatially modulating incident light by a pre-set coded aperture mask; inputting the compressed two-dimensional detection values into an initial restoration model to obtain an initial restoration result output by the initial restoration model; wherein the initial restoration model adopts a deep neural network architecture, and is used to extract image features from the compressed two-dimensional detection values to obtain the initial restoration result; inputting the initial restoration result into an optimization model based on an alternating direction multiplier method to obtain a target restoration image output by the optimization model; wherein the optimization model introduces non-local low-rankness, total variation regularization and depth image priori, and the objective function of the optimization model is: ; wherein, represents an estimated value of the original signal of the measured object , represents an estimated value of the non-local similar tensor group corresponding to the non-local similar tensor group to be solved, and respectively represent a first detection value and a second detection value in the compressed two-dimensional detection values, denotes a measurement matrix corresponding to the first probe value, denotes a measurement matrix corresponding to the second probe value, denotes M nearest neighbor non-local similar tensors centered at , denotes an original three-dimensional data cube denotes the kth tensor block after k-nearest neighbor operation, i denotes a non-local low-rank regularization term, denotes a depth image prior regularization term, denotes a total variation regularization term, , , , , and denote different regularization coefficients, respectively.

[0006] According to the high-spectral image compression and recovery method provided by the application, the optimization model is used for: based on the initial recovery result, using the K nearest neighbor algorithm to obtain a non-local similar tensor group; based on the non-local similar tensor group, using the re-weighted kernel norm minimization to determine a low-rank tensor group; based on the low-rank tensor group, iteratively solving in the framework of the alternating direction multiplier method to obtain an optimization result as a target recovery image.

[0007] According to the high-spectral image compression and recovery method provided by the application, the double-camera type spectral imaging system is used to obtain compressed two-dimensional probe values, including: the double-camera type spectral imaging system, on one light path, by placing a preset coded aperture mask in the light path, modulating the incident light, compressing the information of the three-dimensional spectral data cube to the detector to obtain the first probe value, and on the other light path, directly imaging on the detector to obtain the second probe value.

[0008] According to the high-spectral image compression and recovery method provided by the application, the initial recovery model is a self-supervised neural network for realizing depth image prior, and the self-supervised neural network adopts a U-net network, a residual network or a Transformer network.

[0009] According to the high-spectral image compression and recovery method provided by the application, the U-net network includes: a convolutional encoder module for extracting image features from the compressed two-dimensional probe values, including multiple convolutional layers, batch normalization layers and activation functions; a down-sampling module for reducing the spatial dimension of the feature map, including a convolutional layer and a maximum pooling layer; an up-sampling module for recovering the spatial resolution of the feature map, including a convolutional layer and a convolutional transpose layer; and an output layer for outputting the initial recovery result, including a convolutional layer and an activation function.

[0010] The hyperspectral image compression and recovery method provided by the application further comprises: processing the compressed two-dimensional detection value through depth-integrated image prior information.

[0011] The application further provides a hyperspectral image compression and recovery device, comprising: a detection value acquisition module, configured to obtain a compressed two-dimensional detection value by using a dual-camera spectral imaging system; wherein the compressed two-dimensional detection value is obtained by spatially modulating incident light through a preset coded aperture mask; an initial recovery module, configured to input the compressed two-dimensional detection value into an initial recovery model to obtain an initial recovery result output by the initial recovery model; wherein the initial recovery model adopts a deep neural network architecture and is configured to extract image features from the compressed two-dimensional detection value to obtain the initial recovery result; and an optimized recovery module, configured to input the initial recovery result into an optimization model based on an alternating direction multiplier method to obtain a target recovery image output by the optimization model; wherein the optimization model introduces non-local low-rankness, total variation regularization and deep image prior, and the objective function of the optimization model is as follows: ; wherein, represents an estimated value of an original signal of a measured object, represents an estimated value of a non-local similar tensor group corresponding to a non-local similar tensor group to be solved, represents an estimated value of a non-local similar tensor group corresponding to a non-local similar tensor group to be solved, and and represent a first detection value and a second detection value in the compressed two-dimensional detection value, respectively, represents a measurement matrix corresponding to the first detection value, represents a measurement matrix corresponding to the second detection value, represents M nearest neighbor non-local similar tensor groups centered on , represents an original three-dimensional data cube , i represents the kth tensor block after k-nearest neighbor operation, represents a regularization term of non-local low-rankness, represents a deep image prior regularization term, represents a total variation regularization term, , , , and represent different regularization coefficients, respectively.

[0012] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the hyperspectral image compression and recovery method according to any one of the above-mentioned methods when executing the program.

[0013] The application further provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the hyperspectral image compression and recovery method according to any one of the above.

[0014] The application further provides a computer program product comprising a computer program, which, when executed by a processor, implements the hyperspectral image compression and recovery method according to any one of the above.

[0015] The application provides a hyperspectral image compression and recovery method and device, which comprises: obtaining compressed two-dimensional detection values by using a dual-camera spectral imaging system; the compressed two-dimensional detection values are obtained by spatially modulating incident light by using a preset coded aperture mask; inputting the compressed two-dimensional detection values into an initial recovery model to obtain an initial recovery result; the initial recovery model adopts a deep neural network architecture and is used to extract image features from the compressed two-dimensional detection values to obtain the initial recovery result; inputting the initial recovery result into an optimization model based on an alternating direction multiplier method to obtain a target recovery image; and the optimization model introduces non-local low-rankness, total variation regularization and depth image priori. Through the above method, the multi-prior fusion scheme introduced by the application optimizes the initial recovery result based on multiple prior information by introducing non-local low-rankness, total variation regularization and depth image priori, significantly improves the authenticity and accuracy of the initial recovery result, and achieves high-quality image recovery effect. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0017] Figure 1 is a flowchart of the hyperspectral image compression and recovery method provided by the embodiment of the present application.

[0018] Figure 2 is a schematic diagram of an optical path provided by the embodiment of the present application.

[0019] Figure 3 is a schematic diagram of the light modulation process provided by the embodiment of the present application.

[0020] Figure 4 is a structural schematic diagram of the hyperspectral image compression and recovery device provided by the embodiment of the present application.

[0021] Figure 5 is a schematic diagram of the physical structure of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below with reference to the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0023] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0024] Hyperspectral imaging technology includes various implementation means, such as traditional scanning imaging technology and coded aperture snapshot spectral imaging technology based on compressed sensing theory.

[0025] Compared with traditional scanning imaging technology, the coded aperture snapshot spectral imaging technology based on compressed sensing theory compresses the three-dimensional spectral data cube into a two-dimensional signal for detection through front-end optical coding, and decodes in the rear-end using a recovery algorithm, which can realize snapshot imaging and greatly reduce the detection time.

[0026] At present, hyperspectral image reconstruction methods mainly include two categories: one is based on traditional algorithms, including least square method, dictionary learning and regularization constraint, etc. The other is based on deep learning methods, such as end-to-end reconstruction model based on convolutional neural network, high-fidelity image generation using generative adversarial network, feature enhancement network combined with attention mechanism and long-range dependence modeling based on Transformer architecture.

[0027] Traditional algorithms have certain mathematical theoretical basis, but there are many problems in processing complex scene hyperspectral data. For example, traditional algorithms are sensitive to detection noise, resulting in low quality of recovered images. In addition, deep learning methods provide new solutions in the field of hyperspectral image reconstruction, but also have some limitations, such as poor model generalization ability, lack of interpretability, large consumption of computing resources, etc.

[0028] To solve the above problems, a method combining deep learning and traditional algorithms can be used to take full advantage of both. However, this combined method still faces some problems in hyperspectral image restoration, mainly manifested as: 1. Optimization framework complexity: The method of combining traditional optimization algorithms with deep learning models results in a multi-level, multi-objective optimization framework. Compared to a single technical approach, the optimization objective function, solution strategy, and convergence analysis are all significantly more complex.

[0029] 2. Difficulty in image feature fusion: This method usually embeds a deep learning module as a black box feature extractor into a traditional optimization framework, or only uses traditional algorithms as a pre-processing / post-processing step for a deep network. It fails to build a joint representation model that truly reflects the low-rank nature, spectral continuity, and spatial-spectral structure relationship of hyperspectral data. In particular, when dealing with spectral dimension information, existing hybrid methods have difficulty balancing the relationship between physical meaning preservation and deep feature learning, resulting in deviations in spectral accuracy of the restoration results and failing to meet the application requirements of high-precision spectral analysis.

[0030] In summary, related technologies still have deficiencies in hyperspectral image restoration. The present embodiment provides a hyperspectral image compression and restoration method that introduces a multi-prior architecture based on traditional algorithms and deep learning to achieve higher precision restoration results. Specifically, non-local low-rank and deep image priors based on deep learning can be used to capture the non-local similarity and local similarity of hyperspectral images, respectively. A spectral dimension TV (Total Variation) regularization is added as a spectral continuity prior, and an ADMM (Alternating Direction Method of Multipliers) framework is used for iterative solving to obtain high-quality restored hyperspectral images.

[0031] Please refer to Figure 1 , Figure 1 is a flowchart of the hyperspectral image compression and restoration method provided by the present embodiment. In this embodiment, the hyperspectral image compression and restoration method can include steps S110 to S130, which are as follows: S110: Obtain compressed two-dimensional detection values using a dual-camera spectral imaging system; wherein the compressed two-dimensional detection values are obtained by spatially modulating incident light through a pre-set coded aperture mask.

[0032] In the dual-camera spectral imaging system, one light path can be obtained by spatially and spectrally modulating incident light through a pre-set coded aperture mask and a dispersive element, and the other light path can be directly imaged on a detector plane.

[0033] In this embodiment, a dual-camera spectral imaging system can be used to obtain compressed two-dimensional detection values. This process uses a pre-set spectral encoding aperture mask to spatially modulate the incident light, thereby obtaining compressed two-dimensional detection values containing spectral information and spatial information.

[0034] The dual-camera spectral imaging system can obtain spectral information and spatial information from different angles or different spectral bands to obtain hyperspectral image data, laying the foundation for subsequent high-quality image restoration. Compared with a single-camera system, the dual-camera spectral imaging system can provide more comprehensive and rich raw data, which helps to improve the accuracy and quality of image restoration.

[0035] The encoding aperture is an optical element used to spatially modulate incident light, including but not limited to digital micromirror array, liquid crystal on silicon, and quartz mask. Specifically, the encoding aperture transmits / reflects incident light through a specific pattern into the subsequent optical system or absorbs / reflects it so that it does not enter the subsequent optical system, thereby performing spatial modulation, and combining with a dispersive element to achieve final spatial and spectral modulation.

[0036] In some embodiments, obtaining compressed two-dimensional detection values using a dual-camera spectral imaging system includes: a dual-camera spectral imaging system, on one optical path, modulating incident light by placing a pre-set encoding aperture mask in the optical path, compressing information of a three-dimensional spectral data cube onto a detector to obtain a first detection value, and on another optical path, directly imaging onto a detector to obtain a second detection value.

[0037] In this embodiment, the compressed two-dimensional measurement data processed has a clear physical acquisition path, i.e., spatial modulation of incident light by a specifically designed spectral encoding aperture mask, followed by controlled displacement in the spectral dimension using a precision disperser, and finally obtaining compressed measurement values through integration operation.

[0038] S120: inputting the compressed two-dimensional detection values into an initial restoration model to obtain an initial restoration result output by the initial restoration model; wherein the initial restoration model adopts a deep neural network architecture for extracting image features from the compressed two-dimensional detection values to obtain the initial restoration result.

[0039] In this step, the compressed two-dimensional detection values are input into the initial restoration model based on the deep neural network architecture. The deep neural network has strong feature extraction and learning ability, which can automatically mine the key features of the image from the compressed two-dimensional detection values and convert them into a preliminary image restoration result. The initial restoration model can quickly generate an initial restoration image that is close to the original image.

[0040] S130: input the initial restoration result into an optimization model based on an alternating direction method of multipliers to obtain a target restoration image output by the optimization model; wherein the optimization model introduces non-local low-rankness, total variation regularization, and depth image priori.

[0041] In this step, the initial restoration result is input into an optimization model based on an alternating direction method of multipliers (ADMM) to further optimize a high-quality target restoration image. In the optimization process, prior information such as non-local low-rankness, total variation regularization, and depth image priori is introduced to fully utilize the advantages of these prior information, finely adjust and optimize the initial restoration result, improve the accuracy and quality of image restoration, and make the final target restoration image closer to the real scene.

[0042] The ADMM method can decompose a complex optimization problem into multiple relatively simple sub-problems and gradually approach the optimal solution by alternately solving these sub-problems. It has high efficiency and stability in dealing with image restoration problems with multiple constraints and prior information.

[0043] Non-local low-rankness is a prior characteristic of an image, which means that there are a large number of non-local similar structures in the image, and these similar structures exhibit low-rank characteristics under certain transformation domain. The use of non-local low-rankness prior in this embodiment can extract repeated patterns and structural information from the image during image restoration, thereby improving the quality and accuracy of the restored image.

[0044] Total variation regularization constrains the gradient information of the image, so that the restored image reduces noise and artifacts while preserving edge information. The basic idea is to assume that the gradient distribution of the image is relatively sparse, i.e., the gradient of most regions in the image is small, while the gradient of the edge region is large. Through total variation regularization, the edge details of the image can be enhanced during optimization, improving the clarity and visual effect of the image.

[0045] Depth image priori is prior information based on image features and statistical rules learned by a deep learning model. Depth image priori reflects the internal characteristics and structural information of the image under the representation of a deep neural network, such as local features, global features, and the correlation between different features, etc. In image restoration, depth image priori can better guide the restoration process, making the restored image more consistent with the statistical rules of natural images, and improving the quality and authenticity of the restoration.

[0046] Please refer to Figure 2 , Figure 2 is a light path schematic diagram provided by the embodiment of the application.

[0047] The object 210, the beam splitter 220, the objective lens 230, the coded aperture 240, the relay lens 250, the dispersive element 260, the first detector 270 and the second detector 280 are included in the optical path diagram.

[0048] The coded aperture 240 is a preset coded aperture mask, and the first detector 270 and the second detector 280 are used to obtain first detection values and second detection values, respectively.

[0049] Exemplarily, when the compressed two-dimensional detection values are obtained by using CASSI (Compressive Spectral Snapshot Imaging), the detection model can be expressed as follows: ; Wherein, represents the original signal of the measured object, and represent the CASSI branch and the gray branch detection values, respectively, and represent the measurement matrix of the CASSI branch and the gray branch, respectively, and represent the detection noise of the CASSI branch and the gray branch.

[0050] Please refer to Figure 3 , Figure 3 is a schematic diagram of the light modulation process provided by the embodiment of the application.

[0051] Specifically, the three-dimensional spectral data cube 310 obtained from the above object is processed by the coded aperture 320 to obtain a spatially encoded three-dimensional spectral data cube 330, and then processed by the dispersive element 340 to obtain a spatially encoded and dispersed three-dimensional spectral data cube 350, and after being collected by the detector, the detection result 360 is obtained.

[0052] After introducing the non-local low-rank property, the total variation regularization and the depth image priori in the optimization model, the target function can be specifically: ; Wherein, represents the original signal of the measured object , represents the estimated value of the non-local similar tensor group to be solved, and represent the first detection value and the second detection value in the compressed two-dimensional detection value, respectively, represents the measurement matrix corresponding to the first detection value, represents a measurement matrix corresponding to the second probe value, represents a center M nearest neighbor non-local similar tensor group, represents an original three-dimensional data cube represents the first i tensor block after k nearest neighbor operation, represents a non-local low-rank regularization term, represents a depth image prior regularization term, represents a total variation regularization term, , , , and respectively represent different regularization coefficients.

[0053] In order to optimize the above formula, the objective function can be decomposed into two sub-problems, sub-problem and x sub-problem.

[0054] For sub-problem, the formula to be solved is: ; wherein, represents a reweighted kernel norm of , represents expansion in the third dimension, and k and m respectively represent the row and column numbers of .

[0055] represents a reweighting coefficient.

[0056] represents a singular value of , represents a kernel norm.

[0057] It should be noted that in the formula expression mentioned in the embodiments of the present application, the bold lowercase letters can represent vectors, the bold uppercase letters can represent matrices, the script letters can represent tensors, the non-bold lowercase letters can represent scalars, the ^ above the letters represents an estimated value, and the (3) in the upper right corner represents expansion in the third dimension.

[0058] On this basis, can be approximately solved by a global soft threshold algorithm: ; ; .

[0059] wherein, is ​The singular value decomposition of The meaning of these three matrices is, T represents transpose, yes The matrix expanded along the third dimension, Represents the calculated The estimated singular values ​​of σ ij is Σ i The jth singular value, epsilon is a positive constant.

[0060] The f subproblem can be expressed as: ; In order to better optimize, we can use the ADMM framework to introduce auxiliary variables , Lagrange multiplier , balance coefficient eta , transform the optimization problem into:

[0061] Among them, Θ represents the network parameters of U-net, Representatives and Random vectors of the same size can be optimized by using this formula as the loss function and performing reverse gradient backpropagation. Representatives and Random vectors of the same size are used as the input of the network, and the optimization is performed by back-propagation of the gradient using formula (1) as the loss function.

[0062] Represents the tth iteration To further solve , again using the ADMM framework, introducing auxiliary variables and Lagrange multipliers , balance coefficient lambda , transform the optimization problem into: ; ; .

[0063] in Represents the tth iteration value, The solution of is a quadratic optimization problem, which can be obtained as follows: ; in They are The matrix form of Representativei a selected tensor of a sub-block, a selected tensor of a sub-block, i a transpose of a selected tensor of a sub-block, representing the expansion of to , and . At this time, in order to solve , the ADMM framework is used again to introduce auxiliary variables and Lagrange multipliers , balance coefficients mu , and convert the optimization problem into two quadratic optimization problems, the solutions of which can be expressed as: ; ; ; wherein, , , is an intermediate variable, represents the value of in the tthiteration, and can be solved element by element, m represents the mth element in and , and the solution is: ; ; can be solved by: .

[0064] In order to quantitatively analyze the reconstruction effect, the spectral angle mapping (SAM), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM) of the image are calculated, which are defined as follows: ; ; .

[0065] wherein, N represents the total number of pixels, C represents the number of spectral channels, and represent the mth element in the original spectral data cube and the reconstructed spectral data cube, respectively iMAX represents the maximum value in the original spectral data cube at the i-th wavelength, and represent the mean value of the original spectral data cube and the reconstructed spectral data cube at the i-th wavelength, i respectively, represent the covariance of the original spectral data cube and the reconstructed spectral data cube at the i-th wavelength, i and represent the variance of the original spectral data cube and the reconstructed spectral data cube at the i-th wavelength, i c 1 and c 2 represent constants.

[0066] In some embodiments, the optimization model is configured to: obtain a non-local similar tensor group by using a K-nearest neighbor algorithm based on the initial restoration result; determine a low-rank tensor group by using a reweighted nuclear norm minimization based on the non-local similar tensor group; and obtain an optimization result as a target restoration image by iteratively solving in a framework of an alternating direction method of multipliers based on the low-rank tensor group.

[0067] In this embodiment, the optimization model first uses the K-nearest neighbor algorithm to capture the non-local similarity in the hyperspectral image, that is, by analyzing each tensor block in the initial restoration result, it finds similar structures in the spatial or spectral dimensions, and groups these similar structures into a non-local similar tensor group. Non-local similarity is based on the repeated patterns and textures in the hyperspectral image, which may have similar features in different regions of the image or spectral bands. Through the capture of this similarity, the model can understand the structure of the image globally.

[0068] The optimization model processes the non-local similar tensor group through the reweighted nuclear norm minimization technique to determine a low-rank tensor group. The reweighted nuclear norm minimization can effectively highlight the main, low-rank features while suppressing noise and unimportant information. The low-rank tensor group represents the most important features and structures in the image, which not only contains spatial information but also contains spectral information.

[0069] Further, the optimization model iteratively solves in the framework of the alternating direction method of multipliers (ADMM) to obtain an optimization result as a target restoration image. ADMM is an efficient optimization algorithm that gradually approaches the optimal solution by alternately optimizing subproblems and updating multipliers. In this case, the ADMM framework can effectively process the low-rank tensor group, further exploiting the potential information in the image while maintaining the continuity of the spectral dimension and the integrity of the spatial dimension. This iterative solving process not only improves the accuracy of image restoration, but also enhances the authenticity and visual quality of the image.

[0070] ​​The above process captures and processes the non-local similarity and low-rank characteristics, so that the optimization model can effectively reduce noise, enhance image details and edge information, and achieve high-precision image restoration results. The final optimization result is the target restoration image, and the quality of the image is significantly improved in the spectral and spatial dimensions.

[0071] In some embodiments, the initial restoration model is a self-supervised neural network for implementing depth image prior, which adopts a U-net network, a residual network or a Transformer network.

[0072] In this embodiment, the self-supervised neural network for implementing depth image prior includes but is not limited to a U-net network, a residual network, a Transformer network or other combinations.

[0073] In some embodiments, the U-net network includes: a convolutional encoder module for extracting image features from compressed two-dimensional detection values, including multiple convolutional layers, batch normalization layers and activation functions; a down-sampling module for reducing the spatial dimension of the feature map, including a convolutional layer and a maximum pooling layer; an up-sampling module for restoring the spatial resolution of the feature map, including a convolutional layer and a convolutional transpose layer; and an output layer for outputting the initial restoration result, including a convolutional layer and an activation function.

[0074] The convolutional encoder module is configured to extract low-level image features from the compressed two-dimensional detection values, and includes multiple convolutional layers, batch normalization layers and activation functions.

[0075] The down-sampling module is configured to gradually reduce the spatial dimension of the feature image, effectively capturing multi-scale global context information and abstract features of the hyperspectral image, and includes a convolutional layer and a maximum pooling layer.

[0076] The up-sampling module gradually restores the spatial resolution of the feature map through techniques such as transposed convolution, and includes a convolutional layer and a convolutional transpose layer.

[0077] The output layer maps the high-dimensional features extracted by the previous hidden layer to the dimension space consistent with the target hyperspectral data, and includes a convolutional layer and an activation function.

[0078] It should be noted that the self-supervised neural network for implementing depth image prior, such as the U-net network, can be used in both the initial restoration model and the optimization model. The structures of the U-net networks corresponding to the two models can be the same, but the model parameters and input parameters are different.

[0079] In some embodiments, the hyperspectral image compression and restoration method further comprises: processing the compressed two-dimensional detection value by deep integrated image prior information to reduce the noise introduced in the compression sampling process.

[0080] In the compression sampling process, due to the modulation of the incident light by the coded aperture mask and the physical limitations of the detector, noise will be introduced when obtaining the compressed two-dimensional detection value. Therefore, in the image restoration process, deep integrated image prior information can be used to reduce the noise effect. Optionally, noise filtering can be set in the initial restoration model and / or the optimization model.

[0081] The above, the embodiment of the application proposes an innovative hyperspectral image restoration technical solution, the core of which is to construct a multi-prior hybrid architecture, which organically integrates the theoretical guarantee of traditional algorithms and the representation ability of deep learning. Specifically, the application uses the non-local low-rank principle to accurately capture the non-local similarity characteristics of hyperspectral data, while introducing a deep image prior network to extract image local structure features, and introducing a spectral dimension total variation regularization term to constrain spectral continuity. The above multiple priors are optimized by an improved alternating direction multiplier method optimization framework, realizing high-quality hyperspectral image restoration and effectively solving the key problems in the prior art.

[0082] The above embodiments can be freely combined without conflict. For example, a multi-prior fusion hyperspectral image compression and restoration method can be implemented, comprising: The compressed two-dimensional detection value is obtained using a dual-camera coded aperture snapshot spectral imaging system; an initial restoration result is obtained using a deep image prior self-supervised neural network based on U-net, and then the initial result is input into the optimization step under the ADMM framework for iterative optimization to finally output an optimized result.

[0083] The above uses an organic fusion strategy of multiple complementary prior knowledge: the non-local low-rank tensor prior effectively captures the long-distance spatial feature association by exploring the similarity structure between hyperspectral data blocks; the deep image prior adaptively learns complex spatial textures and edge details with the help of a carefully designed U-Net network architecture; the total variation regularization term focuses on maintaining the smooth transition characteristics in the spectral dimension and suppressing non-physical noise interference. This multi-scale collaborative optimization mechanism of spatial dimension and spectral dimension, local information and global structure significantly improves the authenticity and accuracy of the reconstruction result, overcoming the inherent limitations of single prior method.

[0084] To verify the effectiveness of the method of the application, it is compared with the following two methods: 1. only using deep image prior; 2. using deep image prior and non-local similarity.

[0085] The results show that when only using the depth image prior to restore, the restoration result can only obtain rough spatial structure information, and the spectral information recovery accuracy is low; when using the depth image prior and the non-local similarity combination method, the spatial structure is better reconstructed, but due to the noise existing in the experiment, the spectral continuity of the reconstruction result is poor. Compared with the related art, the method provided in the embodiment of the application not only obtains better restoration result in the spatial dimension, but also better retains the high-frequency information, suppresses the noise interference in the spectral information recovery, and effectively ensures the spectral continuity.

[0086] Therefore, compared with the related art, the application has the following advantages: 1. The multi-prior fusion scheme provided in the application effectively improves the accuracy of hyperspectral image restoration by introducing non-local low rank, total variation regularization and depth image prior, and has better image reconstruction effect.

[0087] 2. The global correlation structure in the hyperspectral image is effectively captured, the local processing limitation in the related art is overcome, the inherent redundancy feature of the hyperspectral data in the spatial-spectral dimension is fully utilized, and the method has a solid mathematical theoretical basis and provides reliable convergence guarantee. The method shows excellent anti-noise ability for image restoration under sparse sampling conditions.

[0088] 3. Without large-scale external training data, the method avoids overfitting and generalization problems, adaptively learns the complex spatial-spectral structure characteristics of the hyperspectral image, provides high-quality reconstruction results through end-to-end optimization, effectively deals with nonlinear interference and complex noise patterns, and shows strong robustness in actual application.

[0089] 4. The continuity constraint between spectral channels is accurately maintained, which conforms to the physical characteristics, effectively suppresses artifacts and noise while retaining important edge information, has clear physical interpretation, enhances the reliability of the model, has moderate computational complexity, and is convenient for engineering implementation.

[0090] 5. The multi-scale joint optimization of local-global and spatial-spectral dimensions is realized, the reconstruction quality under low signal-to-noise ratio conditions is significantly improved, the method has stronger generalization ability, is suitable for multiple acquisition conditions and application scenarios, and has obvious synergistic complementary effect, and overcomes the inherent defects of a single prior method.

[0091] On the other hand, the embodiment of the application also provides a hyperspectral image compression and restoration device, and the hyperspectral image compression and restoration device provided by the application is described below. The hyperspectral image compression and restoration device described below can be correspondingly referred to the hyperspectral image compression and restoration method described above.

[0092] Please refer to Figure 4 , Figure 4Fig. 1 is a structural schematic diagram of a hyperspectral image compression and restoration device provided by an embodiment of the present application. In this embodiment, the hyperspectral image compression and restoration device can include a detection value acquisition module 410, an initial restoration module 420, and an optimized restoration module 430.

[0093] The detection value acquisition module 410 is configured to obtain compressed two-dimensional detection values by using a dual-camera spectral imaging system; wherein the compressed two-dimensional detection values are obtained by spatially modulating incident light by using a preset coded aperture mask. The initial restoration module 420 is configured to input the compressed two-dimensional detection values into an initial restoration model to obtain an initial restoration result output by the initial restoration model; wherein the initial restoration model adopts a deep neural network architecture, and is configured to extract image features from the compressed two-dimensional detection values to obtain the initial restoration result. The optimized restoration module 430 is configured to input the initial restoration result into an optimization model based on an alternating direction multiplier method to obtain a target restoration image output by the optimization model; wherein the optimization model introduces non-local low-rankness, total variation regularization, and deep image prior, and a target function of the optimization model is as follows: wherein, represents an estimated value of an original signal of a measured object, represents an estimated value of a non-local similar tensor group corresponding to a non-local similar tensor group to be solved, represents an estimated value of a non-local similar tensor group corresponding to a non-local similar tensor group to be solved, and and respectively represent a first detection value and a second detection value in the compressed two-dimensional detection values, represents a measurement matrix corresponding to the first detection value, represents a measurement matrix corresponding to the second detection value, represents M nearest neighbor non-local similar tensor groups centered on , represents an original three-dimensional data cube after k-nearest neighbor operation, i represents the i-th tensor block, represents a regularization term of non-local low-rankness, represents a deep image prior regularization term, represents a total variation regularization term, , , , and respectively represent different regularization coefficients.

[0094] ​In some embodiments, the optimization model can be specifically used for: based on the initial recovery result, acquiring a non-local similar tensor group by using a K-nearest neighbor algorithm; determining a low-rank tensor group by using a re-weighted kernel norm minimization based on the non-local similar tensor group; and obtaining an optimization result as a target recovery image by iteratively solving in a framework of an alternating direction multiplier method based on the low-rank tensor group.

[0095] In some embodiments, the probe value acquisition module 410 can be specifically used for: in a dual-camera spectral imaging system, on one light path, by placing a preset coded aperture mask in the light path, modulating the incident light, compressing the information of a three-dimensional spectral data cube onto a detector, to obtain a first probe value; and on another light path, directly imaging onto a detector to obtain a second probe value.

[0096] In some embodiments, the initial recovery model is a self-supervised neural network for implementing a depth image prior, and the self-supervised neural network adopts a U-net network, a residual network or a Transformer network.

[0097] In some embodiments, the U-net network comprises: a convolutional encoder module for extracting image features from compressed two-dimensional probe values, comprising a plurality of convolutional layers, batch normalization layers and activation functions; a down-sampling module for reducing the spatial dimension of the feature map, comprising a convolutional layer and a maximum pooling layer; an up-sampling module for restoring the spatial resolution of the feature map, comprising a convolutional layer and a convolutional transpose layer; and an output layer for outputting the initial recovery result, comprising a convolutional layer and an activation function.

[0098] In some embodiments, the hyperspectral image compression and recovery apparatus further comprises a noise filtering module, which can be specifically used for: processing the compressed two-dimensional probe values by using depth-integrated image prior information.

[0099] On the other hand, the embodiments of the present application also provide an electronic device, please refer to Figure 5 , Figure 5 is a schematic diagram of the physical structure of the electronic device provided by the embodiments of the present application, as Figure 5 shown, the electronic device can include a memory 520, a processor 510, and a computer program stored on the memory 520 and executable on the processor 510. The processor 510 can implement the hyperspectral image compression and recovery method when executing the program, which can include: A dual-camera spectral imaging system is used to obtain compressed two-dimensional detection values. The compressed two-dimensional detection values ​​are obtained by spatially modulating the incident light through a preset coded aperture mask. The compressed two-dimensional detection values ​​are input into an initial recovery model to obtain an initial recovery result output by the initial recovery model. The initial recovery model uses a deep neural network architecture to extract image features from the compressed two-dimensional detection values ​​to obtain an initial recovery result. The initial recovery result is input into an optimization model based on the alternating direction multiplier method to obtain a target restored image output by the optimization model. The optimization model introduces non-local low rank, total variation regularization, and deep image prior. The objective function of the optimization model is: ; in, Represents the original signal of the object being measured The estimated value of Represents the corresponding non-local similarity tensor group that needs to be solved The estimated value of and represent the first detection value and the second detection value in the compressed two-dimensional detection value, respectively, represents the measurement matrix corresponding to the first detection value, represents the measurement matrix corresponding to the second detection value, Indicates is the center M nearest non-local similar tensor group, Represents the original 3D data cube After the k nearest neighbor operation i Tensor blocks, The regularization term representing the non-local low rank, represents the deep image prior regularization term, represents the total variation regularization term, 、 、 、 and They represent different regularization coefficients respectively.

[0100] Optionally, the electronic device may further include a communication bus 530 and a communication interface 540, wherein the processor 510, the communication interface 540, and the memory 520 communicate with each other via the communication bus 530. The processor 510 may call a computer program in the memory 520 to execute the hyperspectral image compression and restoration methods provided by the above methods.

[0101] In addition, the logic instructions in the memory 520 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0102] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the hyperspectral image compression and recovery method provided by the above-mentioned methods, the steps and principles of which have been described in detail in the above-mentioned methods and will not be repeated here.

[0103] In yet another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the hyperspectral image compression and recovery method provided by the above-mentioned methods, the steps and principles of which have been described in detail in the above-mentioned methods and will not be repeated here.

[0104] The non-transitory computer readable storage medium can be any available medium or data storage device accessible by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor memory (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid state disk (SSD), etc.).

[0105] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0106] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0107] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A hyperspectral image compression and restoration method, characterized by, The method comprises the following steps: obtaining compressed two-dimensional detection values by using a dual-camera spectral imaging system; wherein the compressed two-dimensional detection values are obtained by spatially modulating incident light through a preset coded aperture mask; inputting the compressed two-dimensional detection values into an initial recovery model to obtain an initial recovery result output by the initial recovery model; wherein the initial recovery model adopts a deep neural network architecture and is used to extract image features from the compressed two-dimensional detection values to obtain the initial recovery result; inputting the initial recovery result into an optimization model based on an alternating direction multiplier method to obtain a target recovery image output by the optimization model; wherein the optimization model introduces non-local low-rankness, total variation regularization and deep image prior, and the objective function of the optimization model is: ； wherein, represents the original signal of the measured object , represents the estimated value of the non-local similarity tensor group corresponding to the non-local similarity tensor group needed to be solved , and respectively represent the first detection value and the second detection value in the compressed two-dimensional detection value, represents the measurement matrix corresponding to the first detection value, represents the measurement matrix corresponding to the second detection value, represents the M nearest non-local similarity tensor groups centered on , represents the original three-dimensional data cube the k-th tensor block after k-nearest neighbor operation, i , represents the regularization term of non-local low rank, represents the depth image prior regularization term, represents the total variation regularization term, , , , and respectively represent different regularization coefficients. 2.The hyperspectral image compression and restoration method of claim 1, wherein, the optimization model is used to: based on the initial recovery result, a non-local similar tensor group is obtained by using a K-nearest neighbor algorithm; based on the non-local similar tensor group, a low-rank tensor group is determined by using a re-weighted kernel norm minimization; based on the low-rank tensor group, an optimization result is obtained by iterative solving in the framework of the alternating direction multiplier method, and the optimization result is used as the target recovery image. 3.The hyperspectral image compression and restoration method of claim 1, wherein, The method of obtaining compressed two-dimensional detection values by using a dual-camera spectral imaging system comprises the following steps: in the dual-camera spectral imaging system, on one light path, a preset coded aperture mask is placed in the light path to modulate the incident light, and the information of a three-dimensional spectral data cube is compressed onto a detector to obtain the first detection value; on another light path, the second detection value is obtained by directly imaging on the detector. 4.The hyperspectral image compression and restoration method of claim 1, wherein, The initial recovery model is a self-supervised neural network for realizing deep image prior, and the self-supervised neural network adopts a U-net network, a residual network or a Transformer network. 5.The hyperspectral image compression and restoration method of claim 4, wherein, The U-net network comprises: a convolutional encoder module for extracting image features from the compressed two-dimensional detection values, comprising a plurality of convolutional layers, batch normalization layers and activation functions; a down-sampling module for reducing the spatial dimension of a feature map, comprising a convolutional layer and a maximum pooling layer; an up-sampling module for restoring the spatial resolution of the feature map, comprising a convolutional layer and a convolutional transpose layer; an output layer for outputting the initial recovery result, comprising a convolutional layer and an activation function.

6. The hyperspectral image compression and restoration method according to any one of claims 1 to 5, characterized in that, The method further comprises the following steps: processing the compressed two-dimensional detection values by using deep integrated image prior information.

7. A hyperspectral image compression and restoration apparatus, characterized by comprising: The method comprises the following steps: a detection value acquisition module is configured to obtain compressed two-dimensional detection values by using a dual-camera spectral imaging system; wherein the compressed two-dimensional detection values are obtained by spatially modulating incident light through a preset coded aperture mask; an initial recovery module is configured to input the compressed two-dimensional detection values into an initial recovery model to obtain an initial recovery result output by the initial recovery model; wherein the initial recovery model adopts a deep neural network architecture and is used to extract image features from the compressed two-dimensional detection values to obtain the initial recovery result; an optimization recovery module is configured to input the initial recovery result into an optimization model based on an alternating direction multiplier method to obtain a target recovery image output by the optimization model. The optimization model introduces non-local low rank, total variation regularization and depth image prior, and a target function of the optimization model is as follows: ； wherein, represents the original signal of the measured object an estimate of represents an estimate of the non-local similarity tensor group corresponding to the non-local similarity tensor group and respectively represent a first and a second measurement value of the compressed two-dimensional measurement values, represents a measurement matrix corresponding to the first measurement value, represents a measurement matrix corresponding to the second measurement value, represents the M nearest non-local similarity tensor groups centered at , represents an original three-dimensional data cube the k-th tensor block after k-nearest neighbor operation, i represents a regularization term of non-local low-rankness, represents a depth image prior regularization term, represents a total variation regularization term, , , , , and respectively represent different regularization coefficients.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the hyperspectral image compression and recovery method of any one of claims 1 to 6 when executing the computer program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the hyperspectral image compression and recovery method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the hyperspectral image compression and recovery method of any one of claims 1 to 6. The computer program is executed by the processor to implement the hyperspectral image compression and recovery method of any one of claims 1 to 6.

Images