Self-adaptive depth prior compressed sensing spectrum reconstruction method and system

Abstract

The invention discloses an adaptive depth prior compressed sensing spectrum reconstruction method and system, and the method comprises the steps: generating a compressed observation image through a forward process of a CASSI system, carrying out the modeling of a compressed observation problem of the CASSI system, and decomposing a to-be-solved optimization problem into two sub-problems: data fidelity and noise reduction; euclidean linear projection is carried out on a two-dimensional compressed observation image generated by a CASSI system, and a pre-training adaptive noise level estimation network is used to carry out dynamic estimation on the noise level of a data item in each iteration process; noise reduction is carried out on a projection result by using a pre-training adaptive noise reduction network, and sliding calculation is carried out on the spatial dimension and the channel dimension of the dimension-raised data by using three-dimensional convolution; and detecting whether the output result is optimal or not by using a self-adaptive iteration stop detector until the self-adaptive iteration stop detector judges that the current reconstruction result is smaller than the tolerance, exiting the loop, and outputting an optimal reconstruction result.

Description

Background Art

[0002] The spectral information of the real world is very rich. Since the birth of photography technology, capturing colors and obtaining spectral information has become a very important task. When a large number of bands need to be collected, traditional spectral imaging systems use bandpass filters or push-scanning methods to collect each band in turn. As the number of sampled bands increases, these methods will have the defects of low speed or low resolution. With the advancement of computational photography, a snapshot compressed spectral reconstruction technology (Coded Aperture Snapshot Spectral Imaging, CASSI) based on compressed sensing theory has emerged. The spectral images of each band are randomly sampled and superimposed together. The acquisition end only needs to obtain a two-dimensional compressed observation image to reconstruct the original spectral data cube. The advantage of the CASSI system is that the acquisition system is simple, the hardware cost is low, and the data compression rate is high. However, in fact, this imaging method transfers the hardware pressure of the data acquisition end to the data processing capability of the receiving end. The quality of the reconstruction algorithm directly determines the quality of the imaging. Therefore, the reconstruction algorithm design of the CASSI system has become a crucial issue.

[0003] The current CASSI reconstruction algorithms mainly include:

[0004] 1. Iterative convex optimization algorithm: Use manually designed priors, such as sparsity, total variation (TV), non-local similarity, etc. to constrain the ill-conditioned inverse problem to be solved to the ideal signal space to achieve the solution. However, these priors cannot fully describe the characteristic information of the ideal signal, so the solution accuracy is not high and the computational complexity is high.

[0005] 2. End-to-end (E2E) deep learning method: using a data-driven approach, a large amount of two-dimensional observations and real spectral data are used to train an end-to-end neural network, usually with coding templates and two-dimensional observation data as input, and directly output three-dimensional reconstruction results. This type of method can show extremely high reconstruction accuracy on specific systems, but usually can only run under specific resolutions and coding templates, lacking flexibility and interpretability.

[0006] 3. The Deep Unfolding Network (DUN)-based method: Instead of using a single neural network, a cascade of multi-stage small networks and linear projections are used to alternately complete the reconstruction. Since the scale of each network is smaller and the responsibility is no longer the complete reconstruction task but the denoising process of different noise levels, the interpretability and generalization are improved compared to the end-to-end network, but it is still associated with the encoding template, and the number of parameters is large, and the training stability is poor.

[0007] 4. Plug and Play (PnP)-based deep prior reconstruction method: combined with an iterative convex optimization algorithm, a pre-trained denoising network is used as a prior for iterative reconstruction. Due to the use of a pre-trained denoising network, it can be applied to different coding templates and compressed sensing spectral imaging systems without training, and has strong robustness and flexibility. However, since it only uses one denoising network to complete the iterative denoising process, it places extremely high demands on the adaptability of the pre-trained denoising network to input data with different noise levels.

[0008] The plug-and-play deep prior method can achieve a balance between flexibility, efficiency and reconstruction quality, but the current technical solutions in this field still have some defects.

[0009] In terms of deep priors, the denoising network lacks high-quality adaptive denoising capabilities for spectral data. First, existing denoising networks are usually designed using convolutional neural networks or adding local self-attention mechanisms. The local receptive field defects of the convolution mechanism limit its full utilization of the spatial-spectral long-range correlation of the spectrum, and the denoising quality needs to be improved. Secondly, some existing denoising networks rely on manually specified noise level inputs. The manually specified noise level values ​​during iterations often have a certain gap with the actual values, and blind denoising cannot be achieved and is not flexible enough. Finally, it is difficult for existing denoising networks to adapt to inputs with different numbers of bands. The solutions that can achieve different numbers of bands only use traversal denoising for a single spectral channel using its adjacent channels, and the reconstruction quality and efficiency are poor.

[0010] In terms of algorithm framework, there is a lack of reasonable adaptive iteration stopping strategy. Current technical solutions rarely mention methods to determine the best time to stop iterations. They simply set the maximum number of iterations or stop when the output difference before and after iteration is less than a certain threshold. However, input images with different characteristics often require different numbers of iterations to achieve the optimal reconstruction result, and the accuracy increase trends are different. Therefore, simply setting a fixed number of times or relying only on the difference before and after iterations makes the algorithm lack the adaptability of the number of iterations for different input data. Summary of the invention

[0011] The technical problem to be solved by the present invention is to provide an adaptive deep prior compressed sensing spectral reconstruction method and system in response to the deficiencies in the above-mentioned prior art, which is used to solve the technical problems of poor algorithm flexibility and reconstruction accuracy caused by the inability to apply a single plug-and-play deep prior algorithm to input data with different numbers of bands and different noise levels and the inability to dynamically determine the number of iterations.

[0012] The present invention adopts the following technical solutions:

[0013] A method for adaptive depth prior compressed sensing spectrum reconstruction includes the following steps:

[0014] The compressed observation image is generated through the forward process of the CASSI system, and then the compressed observation problem of the CASSI system is modeled, and the optimization problem to be solved is decomposed into a priori terms and data fidelity terms;

[0015] S2, performing Euclidean linear projection on the data fidelity item obtained in step S1 to obtain a projection result;

[0016] S3, using a pre-trained adaptive noise level estimation network to perform noise level estimation on the projection result obtained in step S2, and outputting a noise level estimation map for each channel;

[0017] S4, using a pre-trained adaptive denoising network to denoise the projection result obtained in step S2 and the noise level estimation map obtained in step S3, using three-dimensional convolution to slide calculations on the spatial dimension and channel dimension of the data after dimensionality increase, adaptively calculating input data of any number of bands and resolution, and obtaining the neural network estimation result under a single iteration of the priori sub-problem in step S1;

[0018] S5. If the output result of step S4 is not optimal, return to step S2 to iterate until the current reconstruction result is less than the tolerance, exit the loop, and output the optimal reconstruction result.

[0019] Preferably, in step S1, the CASSI system compression observation problem is modeled, and the optimization problem to be solved is decomposed into a priori terms and data fidelity terms as follows:

[0020]

[0021] Among them, x is the vectorized input spectral data, y is the vectorized two-dimensional compressed observation image, H is the perception matrix of the CASSI system, τ is the weight coefficient for adjusting the two, and v is the auxiliary variable.

[0022] Preferably, in step S2, Euclidean linear projection is performed as follows:

[0023]

[0024] Among them, (y-Hv (k) ) is the current observation value y and the predicted output Hv (k) The error between (k) is the denoised spectral image of the kth iteration, is the transposed matrix of the perception matrix, It is the outer product matrix of the perception matrix and the transposed perception matrix.

[0025] Preferably, in step S3, the noise level estimation network includes two parallel paths, and the noise level estimation network is trained by adding Gaussian noise with random noise levels to multiple spectral data sets to form training data. The trained noise level estimation network can estimate the noise level of the input noise spectral data.

[0026] Preferably, the first path uses a series of three-dimensional convolution operations to extract spatial noise features from the input data;

[0027] The second path first performs a three-dimensional convolution to transform the features, then uses global average pooling to compress the spatial dimension, uses a one-dimensional convolution in the spectral dimension to construct the modulation weights of the spectral channel, and finally multiplies the weights with the feature map of the first path to obtain the final (k+1) Noise level graphs of the same size Represents x (k+1) The noise level at each pixel location on each channel in .

[0028] Preferably, in step S4, a pre-trained adaptive denoising network is used to denoise the projection result:

[0029] v (k+1) =D μ (x (k+1) )

[0030] Among them, D μ (·) is the noise reducer working under the noise level μ, v (k+1) is the spectral image after noise reduction.

[0031] Preferably, the denoiser working at the noise level μ is implemented by a pre-trained two-dimensional convolutional Transformer denoising network, and the initial input data is the x output in step S2. (k+1) and the output of step S3 The concatenated X finally outputs the denoised spectral image v (k+1) ;

[0032] The input feature map size F first undergoes parallel three-dimensional convolution of the spectral dimension and the spatial dimension, and the output F' is obtained by fusing the two-dimensional feature map;

[0033] The feature map after the 3D convolution module will be divided into two paths. One path is input into the spectral self-attention module to further extract the spectral dimension information, and the other path is added to the output of the spectral self-attention module to realize the residual learning of attention.

[0034] The spectral self-attention module performs global average pooling on the input feature map F in the spatial dimension and reduces the dimension to F'; then F' undergoes a specific form of linear mapping to obtain the transformed key value K and query Q, and the two are matrix multiplied using Softmax to obtain the attention weight map AttentionMap;

[0035] The other way is to pass the feature map F through the linear layer W in D dimension V Transform to get the mapped value V, perform matrix multiplication on the attention weight map AttentionMap and the last two dimensions of V, and then map it through the learnable linear layer to get the final output result of spectral attention enhancement;

[0036] The output of the spectral self-attention module is further input into the self-modulation feedforward neural network for information integration and nonlinear mapping; the input feature map F is first adjusted by the linear layer to double the number of channels to obtain F'; F' is divided into two paths, one path passes through two linear layers and a GeLU activation function to obtain the output F"1, and the other path slices the last dimension of the input tensor and splits it into two paths. The modulation coefficient of its own element is calculated by the activation function and multiplied element by element, and finally the self-modulation parameter F"2 is obtained. The output result of the self-modulation feedforward neural network is F out =F”1+F”2, which serves as the denoising output of the entire Transformer module.

[0037] Preferably, the encoder part consists of one layer of three-dimensional convolution and five Transformer blocks, wherein the spatial resolution of the input and output feature maps of the first, second, fourth, and sixth modules are consistent, and the third and fifth modules perform convolution downsampling operations to capture multi-scale features;

[0038] The decoder part consists of 5 Transformer blocks and 1 layer of 3D convolution, forming a symmetrical structure with the encoder. The 1st, 2nd, 4th, and 6th modules do not change the spatial resolution of the input, and the 3rd and 5th modules use cubic interpolation to achieve upsampling.

[0039] Preferably, in step S5, an adaptive iterative stop detector is used to detect whether the output result is optimal, and the adaptive iterative detector uses a sliding variance to monitor the reconstruction result within a window:

[0040]

[0041] Where W is the window length, t represents the current time, SWV(t) is, x t+w is the w-th output result from time t, x t+i is the i-th output result from time t.

[0042] In a second aspect, an embodiment of the present invention provides an adaptive depth prior compressed sensing spectral reconstruction system, comprising:

[0043] The construction module generates compressed observation images through the forward process of the CASSI system, then models the compressed observation problem of the CASSI system, and decomposes the optimization problem to be solved into a priori terms and data fidelity terms;

[0044] The projection module performs Euclidean linear projection on the data fidelity item to obtain the projection result;

[0045] The estimation module uses a pre-trained adaptive noise level estimation network to estimate the noise level of the projection result and outputs the noise level estimation map of each channel;

[0046] The denoising module uses a pre-trained adaptive denoising network to denoise the projection results obtained by the projection module and the noise level estimation map obtained by the estimation module. It uses three-dimensional convolution to slide calculations on the spatial dimension and channel dimension of the data after dimensionality increase, and adaptively calculates input data of any number of bands and resolution to obtain the neural network estimation result under a single iteration of the priori sub-problem.

[0047] Output module, if the output result of the denoising module is not optimal, return to the projection module for iteration until the current reconstruction result is less than the tolerance, exit the loop, and output the optimal reconstruction result.

[0048] In a third aspect, a computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned adaptive depth prior compressed sensing spectral reconstruction method when executing the computer program.

[0049] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned adaptive depth prior compressed sensing spectral reconstruction method.

[0050] In a fifth aspect, a chip comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned adaptive depth prior compressed sensing spectral reconstruction method when executing the computer program.

[0051] In a sixth aspect, an embodiment of the present invention provides an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-mentioned adaptive depth prior compressed sensing spectral reconstruction method.

[0052] Compared with the prior art, the present invention has at least the following beneficial effects:

[0053] An adaptive deep prior compressed sensing spectral reconstruction method uses three-dimensional convolution on the original data, uses a pre-trained adaptive noise level estimation network and a pre-trained adaptive noise reduction network, and realizes adaptive and high-quality reconstruction of different image resolutions, spectral band numbers and noise levels. The number of iterations does not require manual intervention and is automatically controlled by an iteration stop detector. It can fully adapt to the differences between different input data and accurately and efficiently control the algorithm to output the best reconstruction result. It has the following advantages:

[0054] First, the noise level of the input data is estimated by pre-training the noise level estimation network, which avoids the previous practice of manually setting a fixed value. It can help the denoising module match the correct noise level and has stronger robustness and accuracy.

[0055] Second: The input data is denoised by pre-training an adaptive denoising network, and the Transformer architecture is combined with three-dimensional convolution to achieve efficient and high-quality denoising with adaptive channel number and resolution, overcoming the defects of previous methods that the number of image channels must be fixed or channels can only be traversed sequentially;

[0056] Third: The convergence process is dynamically monitored using the sliding variance method through an adaptive iterative stop detector, and linear time complexity is achieved through the optimization of the calculation method. Compared with the simple method of directly setting the maximum number of iterations or judging based on the changes between two iterations, it has stronger robustness and adaptability.

[0057] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0058] In summary, the present invention realizes efficient adaptive reconstruction of different image resolutions, number of spectral bands and noise levels through an adaptive deep prior compressed sensing spectral reconstruction method, combined with a pre-trained noise level estimation network and an adaptive noise reduction network, which can significantly improve the robustness and accuracy of the algorithm. The use of three-dimensional convolution and Transformer architecture for noise reduction breaks through the limitations on the number of image channels and resolution in previous methods, and can greatly improve the flexibility, computational efficiency and noise reduction quality of the pre-trained model. An innovative adaptive iterative stop detector is introduced, the convergence process is dynamically monitored through the sliding variance method, and the linear time complexity is optimized to achieve accurate and efficient control of the output, adapt to the differences in different input data, and avoid the limitations of the fixed number of iterations in traditional methods. This method can be directly applied to various types of spectral reconstruction systems based on compressed sensing without modification, and has extremely high flexibility and practicality.

[0059] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is the overall algorithm flow chart of the present invention;

[0061] Figure 2 This is the structure diagram of the pre-trained adaptive denoising network;

[0062] Figure 3 The structure diagram of the pre-trained noise level estimation network;

[0063] Figure 4 A schematic diagram of a computer device provided by an embodiment of the present invention;

[0064] Figure 5 The block diagram of a chip provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0067] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0068] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0069] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0070] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0071] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0072] The present invention provides an adaptive depth prior compressed sensing spectral reconstruction method, which realizes efficient and high-quality image reconstruction by adaptively adjusting image resolution, number of bands, coding template, iterative degradation level and dynamically determining the conditions for stopping iteration. The adaptability of the method takes into account both reconstruction speed and image quality, ensuring that the optimal performance can be achieved in different reconstruction scenarios.

[0073] See also Figure 1 The present invention provides an adaptive depth prior compressed sensing spectrum reconstruction method, comprising the following steps:

[0074] S1. Generate a compressed observation image through the forward process of the CASSI system, then model the compressed observation problem of the CASSI system, and decompose the optimization problem to be solved into two sub-problems: data fidelity and noise reduction;

[0075] The mathematical model of the CASSI system observation process is:

[0076] y=Hx

[0077] Among them, x is the vectorized input spectral data, y is the vectorized two-dimensional compressed observation image, and H is the perception matrix of the CASSI system.

[0078] The optimization problem to be solved is:

[0079]

[0080] in, is the data fidelity term, which is used to minimize the error between the estimated value and the true value. R(x) is the prior term, and τ is the weight coefficient that adjusts the two.

[0081] In order to facilitate the solution, an auxiliary variable v is introduced, and the generalized alternating projection algorithm is used to rewrite equation (1) as follows:

[0082]

[0083] Formula (2) is the CASSI measurement solution model established by the method of the present invention. By fixing one of x and v to optimize the other, the priori term and the data fidelity term are optimized and solved respectively.

[0084] S2, perform Euclidean linear projection on the two-dimensional compressed observation image generated by the input CASSI system to reduce the error between the predicted value and the true value and increase the data fidelity;

[0085] The solution process of linear projection is: given v (k) , then for x (k+1) The updating process has a closed-form solution, which can be regarded as v on the linear manifold y = Hx. (k) Euclidean projection of :

[0086]

[0087] Among them, (y-Hv (k) ) represents the current observation value y and the predicted output Hv (k) The error between It indicates the direction in the original signal space that can best reduce the error between the current estimate and the actual observation, and then projects the error and superimposes it on the predicted output to make the predicted value closer to the true value.

[0088] S3, using a pre-trained adaptive noise level estimation network to dynamically estimate the noise level of the data item in each iteration process to achieve adaptive adjustment of hyperparameters;

[0089] The output x of step S2 is obtained by pre-training the adaptive noise level estimation network (k+1) Perform noise level estimation, output the noise level estimation map of each channel, and guide the denoising network to match the denoising intensity;

[0090] The adaptive noise level estimation network consists of two parallel pathways. The first pathway uses a series of three-dimensional convolution operations to extract spatial noise features from the input data.

[0091] The second path first performs a three-dimensional convolution to transform the features, then uses global average pooling to compress the spatial dimension, uses a one-dimensional convolution in the spectral dimension to construct the modulation weights of the spectral channel, and finally multiplies the weights with the feature map of the first path to obtain the final (k+1) Noise level graphs of the same size Represents x (k+1) The noise level at each pixel position on each channel in ;

[0092] By adding Gaussian noise with random noise levels to multiple spectral data sets to form training data pairs for training, the network can directly estimate the noise level of the input noisy spectral data. Since the training process is independent of the spatial resolution of the input data, the number of spectral channels and the forward process of the CASSI system, the flexibility to apply to different systems is guaranteed.

[0093] S4, using a pre-trained adaptive denoising network to denoise the projection result of step S2, using a three-dimensional convolution to slide calculation on the spatial dimension and channel dimension of the data after dimensionality increase, and being able to adapt to input data of any number of bands and resolution;

[0094] Use the pre-trained adaptive denoising network to denoise the projection result in step S2:

[0095] v (k+1) =D μ (x (k+1) ) (4)

[0096] Among them, D μ (·) represents a denoiser operating at a noise level μ.

[0097] See also Figure 3 The denoiser working at the noise level μ is implemented by a pre-trained two-dimensional convolutional Transformer denoising network, and the initial input data is the output of step S2 and the output of step S3 Connected Finally, the spectral image after noise reduction is output

[0098] The multi-scale structure of the encoder-decoder using jump connections. The encoder part consists of a layer of 3D convolution and 5 Transformer blocks. The input and output feature maps of the 1st, 2nd, 4th, and 6th modules have the same spatial resolution, while the 3rd and 5th modules perform convolution downsampling operations to capture multi-scale features.

[0099] The decoder part also consists of 5 Transformer blocks and 1 layer of 3D convolution, forming a symmetrical structure with the encoder. The 1st, 2nd, 4th, and 6th modules do not change the spatial resolution of the input. The 3rd and 5th modules use cubic interpolation to achieve upsampling and restore image details.

[0100] Each Transformer follows the same structure, and the input feature map size is Therefore, all convolution and attention operations will not change the number of spectral bands C, but only affect the number of channels D;

[0101] F first undergoes parallel three-dimensional convolution in the spectral dimension and the spatial dimension, and the output is obtained by fusing the two-dimensional feature map

[0102] The two-dimensional three-dimensional convolution is independent of the spatial resolution and the number of spectral channels, and only depends on the number of input and output convolution kernels D, thus giving the model the ability to adapt to different wavelengths and resolutions. Through the training of a large number of natural spectral images, the inherent spectral laws of natural scenes are modeled, thereby achieving denoising of spectral data of any band and wavelength.

[0103] In addition, since the three-dimensional convolution is used to obtain the feature map to obtain the positional association between pixels, there is no need to add position encoding like the traditional Transformer. Position encoding depends on the specific resolution of the input data, and using convolution instead of position encoding can achieve the adaptability of the model to spatial resolution.

[0104] The measurement characteristics of the CASSI system retain more information in the spatial dimension, and the information loss in the spectral dimension is very serious, so it is necessary to strengthen the information acquisition ability in the spectral dimension. Therefore, the feature map after the three-dimensional convolution module will be divided into two paths. One path is input into the spectral self-attention module (Spectral Self Attention, SSA) to further extract the spectral dimension information, and the other path is added to the output of the spectral self-attention to realize the residual learning of attention, which can stabilize the training process when the model is deepened.

[0105] In order to focus on the changes in the spectral dimension, the SSA module first transforms the input feature map Perform global average pooling in the spatial dimension and reduce the dimension to Then F' undergoes a specific form of linear mapping to obtain the transformed key value and query The two are matrix multiplied and Softmax is used to obtain the attention weight map:

[0106]

[0107] The other way is to pass the feature map F through the linear layer W in D dimensionV Transform to get the mapped value Will Perform matrix multiplication with the last two dimensions of V, and then map it through a learnable linear layer to obtain the final output result of spectral attention enhancement.

[0108] The output of the SSA module will continue to be input into the Self-Modulated Feed-Forward Network for information integration and nonlinear mapping; the input feature map First, the number of channels is doubled by adjusting the linear layer to obtain F' is divided into two paths, one path passes through two linear layers and a GeLU activation function to get the output The other path slices the last dimension of the input tensor, splits it into two paths, calculates the modulation coefficient of its own elements through the activation function and multiplies them element by element, and finally obtains the self-modulation parameter The output of the self-modulating feedforward neural network is F out =F”1+F”2, which is also the denoising output of the entire Transformer module.

[0109] After a series of Transformer modules of different scales, the input noisy data cube completes a round of denoising process. The algorithm will iterate between equations (3) and (4), and each iteration will obtain clearer and lower-noise data.

[0110] S5. Use the adaptive iterative stop detector to detect whether the output result is optimal. If not, return to the projection step to iterate until the adaptive iterative stop detector determines that the current reconstruction result is less than the tolerance, then exit the loop and output the optimal reconstruction result.

[0111] The adaptive iteration detector monitors the algorithm running process and terminates the program in time. When the algorithm reaches the optimal reconstruction result, continuing to iterate will bring negative effects, so the adaptive iteration stop detector needs to track the reconstruction result to stop iteration when it reaches the optimal PSNR. The adaptive iteration detector uses sliding window variance (SWV) to monitor the reconstruction results within a window:

[0112]

[0113] Among them, W is the window length, t represents the current time, and x is the reconstructed output of a certain iteration.

[0114] At the beginning of the iteration, the higher noise level and algorithm processing will cause the difference in the iteration output to be larger, and the sliding variance will also drop rapidly, gradually decreasing as x changes. It will gradually approach x, and the difference between the variance in the current window and the variance in the next window becomes very small. When it remains basically unchanged after P iterations, it is considered that the optimal reconstruction result is achieved and the algorithm loop is exited.

[0115] In addition, the calculation complexity of formula (7) is optimized and formula (7) is rewritten as:

[0116]

[0117] Formula (8) is actually composed of x t+w and x t+i When the number of iterations t is updated, only the previous item can be removed and the new item can be added, which can achieve linear time complexity O(W). The method of the present invention comprehensively considers the previous P+W reconstruction results and has linear time complexity. It has better robustness to the fluctuation of the reconstruction results, can efficiently track the changes in the reconstruction results, and achieve the output of the optimal results.

[0118] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0119] In another embodiment of the present invention, a system for compressing sensing spectrum reconstruction with adaptive depth prior is provided. The system can be used to implement the above-mentioned method for compressing sensing spectrum reconstruction with adaptive depth prior. Specifically, the system for compressing sensing spectrum reconstruction with adaptive depth prior includes a construction module, a projection module, an estimation module, a noise reduction module and an output module.

[0120] Among them, the construction module generates a compressed observation image through the forward process of the CASSI system, and then models the compressed observation problem of the CASSI system, decomposing the optimization problem to be solved into a priori terms and data fidelity terms;

[0121] The projection module performs Euclidean linear projection on the data fidelity item to obtain the projection result;

[0122] The estimation module uses a pre-trained adaptive noise level estimation network to estimate the noise level of the projection result and outputs the noise level estimation map of each channel;

[0123] The denoising module uses a pre-trained adaptive denoising network to denoise the projection results obtained by the projection module and the noise level estimation map obtained by the estimation module. It uses three-dimensional convolution to slide calculations on the spatial dimension and channel dimension of the data after dimensionality increase, and adaptively calculates input data of any number of bands and resolution to obtain the neural network estimation result under a single iteration of the priori sub-problem.

[0124] Output module, if the output result of the denoising module is not optimal, return to the projection module for iteration until the current reconstruction result is less than the tolerance, exit the loop, and output the optimal reconstruction result.

[0125] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, graphics processors (GPU), tensor processors (TPU), digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding functions; the processor described in the embodiment of the present invention can be used for the operation of the adaptive depth prior compressed sensing spectral reconstruction method, including:

[0126] A compressed observation image is generated through the forward process of the CASSI system, and then the compressed observation problem of the CASSI system is modeled, and the optimization problem to be solved is decomposed into a priori terms and data fidelity terms; the obtained data fidelity terms are subjected to Euclidean linear projection to obtain projection results; the noise level of the obtained projection results is estimated using a pre-trained adaptive noise level estimation network, and the noise level estimation map of each channel is output; the obtained projection results and noise level estimation map are denoised using a pre-trained adaptive denoising network, and three-dimensional convolution is used to slide calculations on the spatial dimension and channel dimension of the data after dimensionality increase, and input data of any number of bands and resolution are adaptively calculated to obtain the neural network estimation result under a single iteration of the priori term sub-problem; if the output result does not reach the optimal, return to iterate until the current reconstruction result is less than the tolerance, exit the loop, and output the optimal reconstruction result.

[0127] See also Figure 4 , the terminal device is a computer device, and the computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the method for adaptive depth prior compressed sensing spectrum reconstruction in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the adaptive depth prior compressed sensing spectrum reconstruction system in the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0128] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 4 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0129] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, graphics processing units (GPU), tensor processing units (TPU), digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0130] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0131] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0132] See also Figure 5 The terminal device 600 is an electronic device, and the electronic device is in the form of a general computing device. The components of the electronic device may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0133] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0134] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0135] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0136] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0137] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0138] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (Non-VolatileMemory), such as at least one disk memory.

[0139] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the adaptive depth prior compressed sensing spectrum reconstruction method in the above embodiment; the processor may load and execute the following steps:

[0140] A compressed observation image is generated through the forward process of the CASSI system, and then the compressed observation problem of the CASSI system is modeled, and the optimization problem to be solved is decomposed into a priori terms and data fidelity terms; the obtained data fidelity terms are subjected to Euclidean linear projection to obtain projection results; the noise level of the obtained projection results is estimated using a pre-trained adaptive noise level estimation network, and the noise level estimation map of each channel is output; the obtained projection results and noise level estimation map are denoised using a pre-trained adaptive denoising network, and three-dimensional convolution is used to slide calculations on the spatial dimension and channel dimension of the data after dimensionality increase, and input data of any number of bands and resolution are adaptively calculated to obtain the neural network estimation result under a single iteration of the priori term sub-problem; if the output result does not reach the optimal, return to iterate until the current reconstruction result is less than the tolerance, exit the loop, and output the optimal reconstruction result.

[0141] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0142] Example

[0143] See also Figure 1 , taking a specific CASSI system measurement and reconstruction process as an example, the calculation process of the entire reconstruction method is described as follows:

[0144] 1. CASSI observation process and equation solving modeling

[0145] Assume that the coding template of SD-CASSI system is A 3D spectral data cube with C spectral channels The modulation result after the coding template is:

[0146] χ'(:,:,λ)=χ(:,:,λ)⊙M (9)

[0147] Where ⊙ represents element-by-element multiplication, and χ(:,:,c) represents the image of the λth band (1≤λ≤C) in χ. The modulated spectral image χ' passes through the dispersion element and is c Assuming that the dispersion direction is W and the relative displacement between adjacent bands is d, the spectral data after dispersion is:

[0148] χ”(u,v,λ b )=χ'(h,w+d(λ b -λ c ),λ b ) (10)

[0149] in, The data of d(C-1) columns are expanded in the W direction, λ b represents the bth band, and (u,v) represents the coordinates of the sensor plane. Finally, the sensor plane is integrated in the spectral dimension, that is, the images of all bands are superimposed to obtain a two-dimensional compressed observation.

[0150]

[0151] where Z is the sensor observation noise with the same dimension as Y.

[0152] For convenience, we assume that the inter-spectral shift d is equal to 1, and the vec(·) operator represents the operation of flattening a multidimensional tensor into a one-dimensional vector, Then the perception matrix H of the SD-CASSI system is written as:

[0153]

[0154] in,

[0155]

[0156] in, is the diagonal matrix after the encoding template M is vectorized, and is a zero matrix because A λi There is a relative displacement between them, and it is necessary to λi The zero padding above and below aligns the dimension with the final measured value y; thus, the vectorized model of the SD-CASSI observation process is:

[0157] y=Hx+z (14)

[0158] in,

[0159] Since the number of rows in H is less than the number of columns, directly solving y = Hx will have infinite solutions. However, it is known that the signal x to be solved has certain a priori characteristics, such as sparsity and structural characteristics. Therefore, for such an ill-posed inverse problem, the iterative optimization algorithm will add a priori term to the objective function to describe the constraints. The characteristics that should be possessed Constrained to the ideal signal space, the problem to be optimized is:

[0160]

[0161] in, is the data fidelity term, which is used to minimize the error between the estimated value and the true value. R(x) is the prior term, and τ is the weight coefficient that adjusts the two.

[0162] By introducing the auxiliary variable v, equation (15) is rewritten as:

[0163]

[0164] 2. Data Fidelity Linear Projection

[0165] By introducing the penalty factor ρ, the augmented Lagrangian function of equation (16) can be written as:

[0166]

[0167] Given v, let Then we have:

[0168]

[0169] Substituting equation (18) into Hx=y, we obtain:

[0170]

[0171] Substituting ρ into equation (18), we can get the expression for updating x given v:

[0172]

[0173] This process can be seen as passing through v k Update x by Euclidean projection on the linear manifold y = Hx (k+1) In addition, by adjusting the linear transformation in equation (20) and incorporating y into the iteration process, the convergence speed of the algorithm can be accelerated:

[0174]

[0175] y k+1 =y k +(y k -Hv k ) (twenty two)

[0176] Given x k+1 Solving for v k+1 The subproblem can be viewed as k+1 Whether the denoising network can flexibly adapt to input data with a wide range of varying noise levels is a key issue in algorithm optimization. In addition, the above projection process is a vectorized representation of the mathematical description. Due to the special properties of HHT as a diagonal matrix, the update process of equations (21) and (22) in actual calculation only requires matrix element-by-element operations, which can be directly completed using a two-dimensional matrix without the need for vectorized calculation.

[0177] 3. Pre-trained noise level estimation network

[0178] The task of the pre-trained noise level estimation network is to estimate the noise level at each position in each band of the noisy spectral image, thereby assisting the denoising network to achieve a more accurate denoising process. k+1 , first upgrade it to The three-dimensional convolution on the four-dimensional tensor can adapt to different H, W and C, only affecting the final convolution feature dimension, thereby achieving adaptability to the input data size and number of channels.

[0179] See also Figure 2 The network structure of the noise level estimation network is a dual-path design, one of which uses two sets of 3×3×3 three-dimensional convolutions with a step size of 1 and a padding of 1 and a GeLU activation function to extract features between channels and output a feature map with the same size as the input data. The other path first uses a 3×3×3, step size 1, and padding 1 three-dimensional convolution layer to map the original data to the feature space, and then uses global average pooling to compress the spatial dimension of the feature map to obtain a vector of size 1×1×C×1. Then, a set of 1×3, step size 1, and padding 1 one-dimensional convolution and GeLU activation function are performed on the vector to obtain the weight coefficient of the spectral dimension, which is used to adjust the overall difference in noise levels between different spectral channels. Finally, the feature map F1 is multiplied by the weight coefficient to obtain the noise level estimation map of each channel of the input data.

[0180] Since the model is adaptable to channels and resolutions and is independent of the CASSI system, it can be jointly trained in a variety of large spectral datasets. Large-scale training data can be generated by adding Gaussian noise with random noise levels to the original images. After sufficient training, it can be directly applied to this method without the need for retraining for a specific system.

[0181] 4. Pre-trained adaptive denoising network

[0182] Given a noisy hyperspectral data x of size H×W×C k+1 and the noise level estimate from step 3 The input data to be denoised is constructed by concatenation Where H, W and C are not fixed, and the network supports the denoising process of hyperspectral images with any resolution and any number of bands. The denoising network is jointly pre-trained on multiple spectral data sets, and denoising can be performed directly without training when applied. The following is a detailed description of the two-dimensional convolutional Transformer denoising network in the embodiment.

[0183] See also Figure 3 The entire noise reduction network consists of 12 processing modules connected in series and skipped. The encoder and decoder each contain 6 components, forming a symmetrical structure. In the encoder part, the input data First, a layer of three-dimensional convolution is used for feature transformation, and then three layers of downsampling and Transformer blocks are used to extract feature information of different scales. After each downsampling, the size of the input spectral image is halved and the number of feature maps is doubled. Batch normalization layers are used between each block to enhance training stability. Each Transformer block will use three-dimensional convolution in spectral and spatial dimensions and a spectrally guided self-attention module to model the structure and feature information of the data cube. The bottom-level Transformer will obtain the latent space encoding of the denoised spectral data.

[0184] The decoder part will reconstruct the spectral data cube with noise removed from the latent space encoding, which can be regarded as the inverse process of the encoder. It is processed by the Transformer block with 3 upsampling and fused with the encoder output of the corresponding layer. The fusion of each layer adds more details to the reconstruction result. Finally, the projection output of the 3D convolution is fused with the original cube to realize residual learning.

[0185] Input Data The specific processing steps are as follows:

[0186] 1) Input 3D convolution module No. 1. The 3D convolution module consists of 2 layers of 3D convolution layers in series. The first layer has 2 input channels and 16 output channels. It uses a 3×3 convolution kernel, a step size of 1, and a padding of 1. The second layer has 16 input channels and 16 output channels. It uses a 3×3×3 (depth x height x width) convolution kernel, a step size of 1, and a padding of 1. The final output feature map is

[0187] 2) Feature map F1 is input into the No. 2 two-dimensional convolutional Transformer module, which consists of three parts connected in series:

[0188] 2.1) The first part is a two-dimensional convolution layer, which performs three-dimensional convolution operations in the spatial dimension and the spectral dimension. The spatial convolution is a single layer of three-dimensional convolution, with 16 input channels and 16 output channels, using a 3×1×1 convolution kernel, a step size of 1, and padding of 1×0×0; the spectral dimension is composed of two layers of three-dimensional convolution + GeLU activation function, with 16 input channels and 16 output channels, using a 1×3×3 convolution kernel, a step size of 1, and padding of 0×1×1. Finally, the sum of the spectral dimension and spatial dimension feature maps is output, denoted as This part keeps the number of channels unchanged and only uses two-dimensional convolution to extract features.

[0189] 2.2) The second part is the spectral dimension attention layer, the input First, 3D batch normalization is performed to make the input data have zero mean and unit variance, and then it is divided into two paths. One path is not processed and superimposed on the output of the spectral dimension attention to form residual learning. 2-1 We need to calculate the AttentionMap, then multiply V by the AttentionMap and perform a linear transformation to get the output features. 2-1 By global average pooling, only the spectral dimension is retained to obtain the feature map To maintain adaptability, instead of using a learnable linear layer to calculate Q and K separately, we use The correlation between rows and columns in the matrix is ​​calculated as AttentionMap in the form of, and then the learnable linear layer W1 is used to obtain The final output feature map Among them, W2 is also a learnable linear layer.

[0190] 2.3) The third part is the self-adjusting feedforward neural network, the input First, use the linear transformation layer to double the number of channels and get Then it is divided into two processing paths. The first path is restored to the channel through the GeLU activation function and the linear transformation layer. The other way is to calculate the self-modulation coefficient. The number of channels of F' is divided into two. One part is passed through the Sigmoid function and then the Hadamard product is performed with the other part to obtain the self-modulation coefficient. Finally, we get the feature map of output No. 2

[0191] 3) Feature map F2 is input into the No. 3 downsampling Transformer module, which consists of 3 parts. The first part is a layer of 3D downsampling convolution, with 16 input channels and 32 output channels. It uses a 3×3×3 convolution kernel, a step size of 1×2×2, and a padding of 1. The feature map output by this convolution layer is That is, the spatial resolution is downsampled to 1 / 2 of the original, and the number of channels is doubled. The second and third parts have the same structure as the corresponding parts of module 2, but they work on the downsampled feature map. Therefore, the feature map output by module 3 is

[0192] 4) Modules 4, 5, 6, and 7 receive the output of the previous module as input, and the dual-dimensional and downsampling Transformer modules are performed alternately. The processing process is the same as that of modules 2 and 3.

[0193] 5) The decoding process starts from module 7. The difference between the decoding process and the encoding module is:

[0194] Fusion of results at the same level, for example, the outputs of modules 5 and 7 are added as the input of module 8, the outputs of modules 4 and 8 are added as the input of module 9, and so on;

[0195] Use the bicubic interpolation algorithm to achieve super-resolution of feature maps and restore them to the same level of resolution before downsampling;

[0196] The number of channels is reduced to half after upsampling, and restored to the number before the number of channels at the same level is doubled.

[0197] 6) After the decoding operation of the 6 modules of the decoder, module 12 finally outputs the noise residual learned by the network The final denoised output Y is the sum of the original input X and the noise residual D, completing the denoising.

[0198] 5. Iteration Stop Detector

[0199] The adaptive iteration stop detector maintains a reconstruction result queue of length W, the sum S of all elements in the queue, and the sum S of the squares of all elements 2 The latest result Y of each iteration i will be inserted into the end of the queue and the oldest data at the head of the queue will be removed. For each new window, 2 Subtract the data removed from the window and add the newly added data; then use the new S and S 2 Calculate the sliding variance:

[0200]

[0201] The detector also has a SWV of length P = {v1,v2,...,v P} queue, each updated sliding variance will update the queue synchronously. When the sliding variance in the queue satisfies Where μ is the error tolerance, then the algorithm loop is exited and the optimal reconstruction result is output; otherwise, it jumps to equation (21) and continues the loop.

[0202] In summary, the present invention provides an adaptive deep prior compressed sensing spectral reconstruction method and system, which utilizes a wide range of spectral data sets for pre-training, is independent of the coding template and system forward process of a specific CASSI system, and realizes efficient and high-quality noise reduction on input data of arbitrary resolution and number of channels by means of data dimensionality upgrading, and has a noise level estimation and adaptive iteration stop detection mechanism, which minimizes the human intervention factors of the algorithm, can be "plug and play" without training and is applicable to various CASSI spectral imaging systems, especially to reconstruction systems with different changes in the number of bands, has extremely strong flexibility and adaptability, can effectively reduce the computational burden of training algorithms for different systems, and improves the versatility of the reconstruction algorithm.

[0203] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0204] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0205] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0206] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0207] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0208] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0209] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0210] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0211] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0212] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0213] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A method for adaptive depth prior compressed sensing spectrum reconstruction, characterized in that: The following steps are involved: S1. Generate a compressed observation image through the forward process of the CASSI system, then model the compressed observation problem of the CASSI system, and decompose the optimization problem to be solved into a priori terms and data fidelity terms; S2, performing Euclidean linear projection on the data fidelity item obtained in step S1 to obtain a projection result; S3, using a pre-trained adaptive noise level estimation network to perform noise level estimation on the projection result obtained in step S2, and outputting a noise level estimation map for each channel; S4, using a pre-trained adaptive denoising network to denoise the projection result obtained in step S2 and the noise level estimation map obtained in step S3, using three-dimensional convolution to slide calculations on the spatial dimension and channel dimension of the data after dimensionality increase, adaptively calculating input data of any number of bands and resolution, and obtaining the neural network estimation result under a single iteration of the priori sub-problem in step S1; S5. If the output result of step S4 is not optimal, return to step S2 to iterate until the current reconstruction result is less than the tolerance, exit the loop, and output the optimal reconstruction result.

2. The method for adaptive depth prior compressed sensing spectrum reconstruction according to claim 1, characterized in that: In step S1, the CASSI system compression observation problem is modeled, and the optimization problem to be solved is decomposed into a priori terms and data fidelity terms as follows: Among them, x is the vectorized input spectral data, y is the vectorized two-dimensional compressed observation image, H is the perception matrix of the CASSI system, τ is the weight coefficient for adjusting the two, and v is the auxiliary variable.

3. The method for adaptive depth prior compressed sensing spectrum reconstruction according to claim 1, characterized in that: In step S2, Euclidean linear projection is performed as follows: Among them, (y-Hv (k) ) is the current observation value y and the predicted output Hv (k) The error between (k) is the denoised spectral image of the kth iteration, is the transposed matrix of the perception matrix, It is the outer product matrix of the perception matrix and the transposed perception matrix.

4. The adaptive depth prior compressed sensing spectrum reconstruction method according to claim 1, characterized in that: In step S3, the noise level estimation network includes two parallel paths. The noise level estimation network is trained by adding Gaussian noise with random noise levels to multiple spectral data sets to form training data. The trained noise level estimation network can estimate the noise level of the input noise spectral data.

5. The method for adaptive depth prior compressed sensing spectrum reconstruction according to claim 4, characterized in that: The first path uses a series of 3D convolution operations to extract spatial noise features from the input data; The second path first performs a three-dimensional convolution to transform the features, then uses global average pooling to compress the spatial dimension, uses a one-dimensional convolution in the spectral dimension to construct the modulation weights of the spectral channel, and finally multiplies the weights with the feature map of the first path to obtain the final (k+1) Noise level graphs of the same size Represents x (k+1) The noise level at each pixel location on each channel in .

6. The method for adaptive depth prior compressed sensing spectrum reconstruction according to claim 1, characterized in that: In step S4, the projection result is denoised using a pre-trained adaptive denoising network: v (k+1) =D μ (x (k+1) ) Among them, D μ (·) is the noise reducer working under the noise level μ, v (k+1) is the spectral image after noise reduction.

7. The method for adaptive depth prior compressed sensing spectrum reconstruction according to claim 6, characterized in that: The denoiser working at the noise level μ is implemented by a pre-trained two-dimensional convolutional Transformer denoising network, and the initial input data is the output of step S2 x (k+1) and the output of step S3 The concatenated X finally outputs the denoised spectral image v (k+1) ; The input feature map size F first undergoes parallel three-dimensional convolution of the spectral dimension and the spatial dimension, and the output F' is obtained by fusing the two-dimensional feature map; The feature map after the 3D convolution module will be divided into two paths. One path is input into the spectral self-attention module to further extract the spectral dimension information, and the other path is added to the output of the spectral self-attention module to realize the residual learning of attention. The spectral self-attention module performs global average pooling on the input feature map F in the spatial dimension and reduces the dimension to F'; then F' undergoes a specific form of linear mapping to obtain the transformed key value K and query Q, and the two are matrix multiplied using Softmax to obtain the attention weight map AttentionMap; The other way is to pass the feature map F through the linear layer W in D dimension V Transform to get the mapped value V, perform matrix multiplication on the attention weight map AttentionMap and the last two dimensions of V, and then map it through the learnable linear layer to get the final output result of spectral attention enhancement; The output of the spectral self-attention module is further input into the self-modulating feedforward neural network for information integration and nonlinear mapping; the input feature map F is first adjusted by the linear layer to double the number of channels to obtain F'; F' is divided into two paths. One path passes through two linear layers and a GeLU activation function to obtain the output F"1. The other path slices the last dimension of the input tensor and splits it into two paths. The modulation coefficient of its own elements is calculated by the activation function and multiplied element by element. Finally, the self-modulation parameter F"2 is obtained. The output result of the self-modulation feedforward neural network is F out =F”1+F”2, which serves as the denoising output of the entire Transformer module.

8. The adaptive depth prior compressed sensing spectrum reconstruction method according to claim 7, characterized in that: The encoder part consists of 1 layer of 3D convolution and 5 Transformer blocks. The spatial resolution of the input and output feature maps of modules 1, 2, 4, and 6 are consistent. Modules 3 and 5 perform convolution downsampling operations to capture multi-scale features. The decoder part consists of 5 Transformer blocks and 1 layer of 3D convolution, forming a symmetrical structure with the encoder. The 1st, 2nd, 4th, and 6th modules do not change the spatial resolution of the input, and the 3rd and 5th modules use cubic interpolation to achieve upsampling.

9. The method for adaptive depth prior compressed sensing spectrum reconstruction according to claim 1, characterized in that: In step S5, an adaptive iterative stop detector is used to detect whether the output result is optimal. The adaptive iterative detector uses a sliding variance to monitor the reconstruction result within a window: Where W is the window length, t represents the current time, SWV(t) is, x t+w is the w-th output result from time t, x t+i is the i-th output result from time t.

10. An adaptive depth prior compressed sensing spectral reconstruction system, characterized in that: include: The construction module generates compressed observation images through the forward process of the CASSI system, then models the compressed observation problem of the CASSI system, and decomposes the optimization problem to be solved into a priori terms and data fidelity terms; The projection module performs Euclidean linear projection on the data fidelity item to obtain the projection result; The estimation module uses a pre-trained adaptive noise level estimation network to estimate the noise level of the projection result and outputs the noise level estimation map of each channel; The denoising module uses a pre-trained adaptive denoising network to denoise the projection results obtained by the projection module and the noise level estimation map obtained by the estimation module. It uses three-dimensional convolution to slide calculations on the spatial dimension and channel dimension of the data after dimensionality increase, and adaptively calculates input data of any number of bands and resolution to obtain the neural network estimation result under a single iteration of the priori sub-problem. Output module, if the output result of the denoising module is not optimal, return to the projection module for iteration until the current reconstruction result is less than the tolerance, exit the loop, and output the optimal reconstruction result.

Images