An adaptive deep prior-based compressive sensing spectral reconstruction method and system

Abstract

The application discloses a kind of compressed sensing spectrum reconstruction methods and systems of adaptive depth prior, and the compressed observation image is generated by CASSI system forward process, then the compressed observation problem of CASSI system is modeled, and the optimization problem to be solved is decomposed into two sub-problems of data fidelity and noise reduction;The two-dimensional compressed observation image generated by CASSI system is projected in Euclidean linear, a pre-training adaptive noise level estimation network is used, and the noise level of data item in each iteration process is dynamically estimated;The projection result is denoised using a pre-training adaptive denoising network, and three-dimensional convolution is used to slide calculation in the spatial dimension and channel dimension of the data after dimensionality increase;Whether the output result reaches the optimum is detected using adaptive iteration stop detector, until adaptive iteration stop detector judges that the current reconstruction result is less than tolerance, exit loop, and output optimal reconstruction result.

Description

Technical Field

[0001] The present invention belongs to the technical field, and in particular relates to a method and system for adaptive depth prior compressed sensing spectral reconstruction. Background Art

[0002] The real world is rich in spectral information. Capturing color and thereby obtaining spectral information has been a crucial task since the advent of photography. When capturing a large number of bands, traditional spectral imaging systems use bandpass filters or push-scanning to sequentially capture each band. As the number of sampled bands increases, these methods suffer from slow speed and low resolution. Advances in computational photography have led to the emergence of Coded Aperture Snapshot Spectral Imaging (CASSI), a technique based on compressed sensing theory. Spectral images of each band are randomly sampled and then superimposed. The acquisition end only obtains a single two-dimensional compressed observation image to reconstruct the original spectral data cube. The advantages of CASSI lie in its simple acquisition system, low hardware cost, and high data compression rate. However, this imaging method shifts the hardware burden from the data acquisition end to the data processing capabilities of the receiving end. The quality of the image is directly determined by the quality of the reconstruction algorithm, making the design of the reconstruction algorithm for CASSI systems a crucial issue.

[0003] The current CASSI reconstruction algorithms mainly include:

[0004] 1. Iterative convex optimization algorithms: These use handcrafted priors, such as sparsity, total variation (TV), and nonlocal similarity, to constrain the ill-posed inverse problem to an ideal signal space. However, these priors cannot fully describe the characteristic information of an ideal signal, resulting in low accuracy and high computational complexity.

[0005] 2. End-to-end (E2E) deep learning methods: Using a data-driven approach, an end-to-end neural network is trained using a large amount of 2D observations and real spectral data. Typically, these methods take a coding template and 2D observation data as input and directly output a 3D reconstruction. While these methods can achieve extremely high reconstruction accuracy on specific systems, they typically operate only at specific resolutions and coding templates, lacking flexibility and interpretability.

[0006] 3. Deep Unfolding Network (DUN)-based approach: Instead of using a single neural network, this approach uses a cascade of multiple small networks and alternating linear projections to perform reconstruction. Because each network is smaller and no longer responsible for the complete reconstruction task, but rather for noise reduction at varying noise levels, it improves interpretability and generalization compared to end-to-end networks. However, it is still tied to the encoding template, has a large number of parameters, and suffers from poor training stability.

[0007] 4. Plug and Play (PnP)-based deep prior reconstruction method: Combined with an iterative convex optimization algorithm, this method uses a pretrained denoising network as a prior for iterative reconstruction. Due to the use of a pretrained denoising network, this method can be applied to different encoding templates and compressed sensing spectral imaging systems without training, demonstrating strong robustness and flexibility. However, since it only uses a single denoising network to perform the iterative denoising process, it places extremely high demands on the pretrained denoising network's adaptability to input data with varying noise levels.

[0008] The plug-and-play deep prior method can achieve a balance between flexibility, efficiency and reconstruction quality, but the current technical solutions in this field still have some defects.

[0009] In terms of deep priors, the denoising network lacks high-quality adaptive denoising capabilities for spectral data. First, existing denoising networks are usually designed using convolutional neural networks or adding local self-attention mechanisms. The local receptive field defects of the convolution mechanism limit its full utilization of the spatial-spectral long-range correlation of the spectrum, and the denoising quality needs to be improved. Secondly, some existing denoising networks rely on manually specified noise level inputs. The manually specified noise level values ​​during iteration often have a certain gap with the actual values, which makes blind denoising impossible and not flexible enough. Finally, it is difficult for existing denoising networks to adapt to inputs with different numbers of bands. The solutions that can achieve different numbers of bands only use traversal denoising for a single spectral channel using its adjacent channels, resulting in poor reconstruction quality and efficiency.

[0010] Regarding the algorithmic framework, there is a lack of a reasonable adaptive iteration stopping strategy. Current technical solutions rarely address methods for determining the optimal stopping point, simply setting a maximum number of iterations or stopping when the output difference before and after an iteration is less than a certain threshold. However, input images with different characteristics often require different numbers of iterations to achieve the optimal reconstruction result, and the accuracy improvement trends vary. Therefore, simply setting a fixed number of iterations or relying solely on the difference before and after iterations makes the algorithm lack adaptability to different input data. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to address the deficiencies in the above-mentioned existing technologies and provide a method and system for compressed sensing spectral reconstruction with adaptive deep prior, which is used to solve the technical problems of poor algorithm flexibility and reconstruction accuracy caused by the inability of a single plug-and-play deep prior algorithm to apply input data with different numbers of bands and different noise levels, and the inability to dynamically determine the number of iterations.

[0012] The present invention adopts the following technical solutions:

[0013] A method for compressed sensing spectrum reconstruction based on adaptive depth prior includes the following steps:

[0014] The compressed observation image is generated through the forward process of the CASSI system. Then the compressed observation problem of the CASSI system is modeled and the optimization problem to be solved is decomposed into a priori terms and data fidelity terms.

[0015] S2. Perform Euclidean linear projection on the data fidelity item obtained in step S1 to obtain a projection result;

[0016] S3. Use a pre-trained adaptive noise level estimation network to perform noise level estimation on the projection result obtained in step S2, and output a noise level estimation map for each channel;

[0017] S4. Use a pre-trained adaptive denoising network to denoise the projection result obtained in step S2 and the noise level estimation map obtained in step S3. Use three-dimensional convolution to perform sliding calculations on the spatial dimension and channel dimension of the dimensionalized data, adaptively calculate input data of any number of bands and resolution, and obtain the neural network estimation result under a single iteration of the prior term subproblem in step S1.

[0018] S5. If the output result of step S4 is not optimal, return to step S2 and iterate until the current reconstruction result is less than the tolerance, exit the loop, and output the optimal reconstruction result.

[0019] Preferably, in step S1, the CASSI system compression observation problem is modeled, and the optimization problem to be solved is decomposed into a priori terms and data fidelity terms as follows:

[0020]

[0021] Among them, x is the vectorized input spectral data, y is the vectorized two-dimensional compressed observation image, H is the perception matrix of the CASSI system, τ is the weight coefficient for adjusting the two, and v is the auxiliary variable.

[0022] Preferably, in step S2, Euclidean linear projection is performed as follows:

[0023]

[0024] Among them, (y-Hv (k) ) is the current observation value y and the predicted output Hv (k) The error between (k) is the denoised spectral image of the kth iteration, is the transposed matrix of the perception matrix, It is the outer product matrix of the perception matrix and the transposed perception matrix.

[0025] Preferably, in step S3, the noise level estimation network includes two parallel paths, and the noise level estimation network is trained by adding Gaussian noise of random noise level to multiple spectral data sets to form training data. The trained noise level estimation network can estimate the noise level of the input noise spectral data.

[0026] Preferably, the first path uses a series of three-dimensional convolution operations to extract spatial noise features from the input data;

[0027] The second path first performs three-dimensional convolution for feature transformation, then uses global average pooling to compress the spatial dimension, uses one-dimensional convolution in the spectral dimension to construct the modulation weight of the spectral channel, and finally multiplies the weight with the feature map of the first path to obtain the final (k+1) Noise level graphs of the same size Represents x (k+1) The noise level at each pixel location on each channel in .

[0028] Preferably, in step S4, the projection result is denoised using a pre-trained adaptive denoising network:

[0029] v (k+1) =D μ (x (k+1) )

[0030] Among them, D μ (·) is the noise reducer working under the noise level μ, v (k+1) is the spectral image after noise reduction.

[0031] Preferably, the denoiser working at the noise level μ is implemented by a pre-trained two-dimensional convolutional Transformer denoising network, and the initial input data is the x output in step S2. (k+1) and the output of step S3 The connected X finally outputs the denoised spectral image v (k+1) ;

[0032] The input feature map size F first undergoes parallel three-dimensional convolution of the spectral dimension and the spatial dimension, and the output F' is obtained by fusing the two-dimensional feature map;

[0033] After the 3D convolution module, the feature map is divided into two paths. One path is input into the spectral self-attention module to further extract the spectral dimension information, and the other path is added to the output of the spectral self-attention module to realize the residual learning of attention.

[0034] The spectral self-attention module performs global average pooling on the input feature map F in the spatial dimension and reduces the dimensionality to F'. Then, F' undergoes a specific form of linear mapping to obtain the transformed key value K and query Q. The two are matrix multiplied using Softmax to obtain the attention weight map AttentionMap.

[0035] The other way is to pass the feature map F through the linear layer W in D dimension V Transform to obtain the mapped value V, perform matrix multiplication on the attention weight map AttentionMap and the last two dimensions of V, and then map it through the learnable linear layer to obtain the final output result of spectral attention enhancement;

[0036] The output of the spectral self-attention module is further input into the self-modulation feedforward neural network for information integration and nonlinear mapping; the input feature map F is first adjusted by the linear layer to double the number of channels to obtain F'; F' is divided into two paths, one path passes through two linear layers and a GeLU activation function to obtain the output F"1, and the other path slices the last dimension of the input tensor and splits it into two paths. The modulation coefficient of its own element is calculated by the activation function and multiplied element by element to finally obtain the self-modulation parameter F"2. The output result of the self-modulation feedforward neural network is F out =F”1+F”2, which serves as the denoising output of the entire Transformer module.

[0037] Preferably, the encoder part consists of one layer of three-dimensional convolution and five Transformer blocks, where the spatial resolution of the input and output feature maps of modules 1, 2, 4, and 6 are consistent, and modules 3 and 5 perform convolution downsampling operations to capture multi-scale features;

[0038] The decoder part consists of 5 Transformer blocks and 1 layer of 3D convolution, forming a symmetrical structure with the encoder. The 1st, 2nd, 4th, and 6th modules do not change the spatial resolution of the input, and the 3rd and 5th modules use cubic interpolation to achieve upsampling.

[0039] Preferably, in step S5, an adaptive iterative stop detector is used to detect whether the output result reaches the optimality, and the adaptive iterative detector uses a sliding variance to monitor the reconstruction result within a window:

[0040]

[0041] Where W is the window length, t represents the current time, SWV(t) is, x t+wis the w-th output result from time t, x t+i is the output result of the i-th time starting from time t.

[0042] In a second aspect, an embodiment of the present invention provides an adaptive depth prior compressed sensing spectral reconstruction system, comprising:

[0043] The construction module generates compressed observation images through the forward process of the CASSI system, then models the compressed observation problem of the CASSI system and decomposes the optimization problem to be solved into a priori terms and data fidelity terms;

[0044] The projection module performs Euclidean linear projection on the data fidelity item to obtain the projection result;

[0045] The estimation module uses a pre-trained adaptive noise level estimation network to estimate the noise level of the projection results and outputs a noise level estimation map for each channel;

[0046] The denoising module uses a pre-trained adaptive denoising network to denoise the projection results obtained by the projection module and the noise level estimation map obtained by the estimation module. It uses three-dimensional convolution to slide calculations on the spatial and channel dimensions of the upscaled data, adaptively calculating input data of any number of bands and resolution, and obtaining the neural network estimation result under a single iteration of the prior term subproblem;

[0047] Output module, if the output result of the denoising module is not optimal, return to the projection module for iteration until the current reconstruction result is less than the tolerance, exit the loop, and output the optimal reconstruction result.

[0048] In a third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned adaptive depth prior compressed sensing spectral reconstruction method when executing the computer program.

[0049] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned adaptive depth prior compressed sensing spectral reconstruction method.

[0050] In a fifth aspect, a chip comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned adaptive depth prior compressed sensing spectral reconstruction method when executing the computer program.

[0051] In a sixth aspect, an embodiment of the present invention provides an electronic device, comprising a computer program, which, when executed by the electronic device, implements the steps of the above-mentioned adaptive depth prior compressed sensing spectral reconstruction method.

[0052] Compared with the prior art, the present invention has at least the following beneficial effects:

[0053] A method for compressed sensing spectral reconstruction based on adaptive depth priors uses three-dimensional convolution on raw data, along with a pre-trained adaptive noise level estimation network and a pre-trained adaptive noise reduction network. This method achieves adaptive, high-quality reconstruction for varying image resolutions, numbers of spectral bands, and noise levels. The number of iterations is automatically controlled by an iteration stop detector without manual intervention, fully adapting to differences between input data and accurately and efficiently controlling the algorithm to output the optimal reconstruction result. The method has the following advantages:

[0054] First, the noise level of the input data is estimated by pre-training the noise level estimation network, avoiding the previous practice of manually setting a fixed value. This can help the denoising module match the correct noise level, and has greater robustness and accuracy.

[0055] Second, we use a pre-trained adaptive denoising network to denoise the input data. We use the Transformer architecture combined with 3D convolution to achieve efficient and high-quality denoising that adapts to the number of channels and resolution. This overcomes the drawbacks of previous approaches that require a fixed number of image channels or can only traverse channels sequentially.

[0056] Third: The convergence process is dynamically monitored using the sliding variance method through an adaptive iterative stop detector, and linear time complexity is achieved through optimization of the calculation method. Compared with the simple method of directly setting the maximum number of iterations or judging based on the changes between two iterations, it has stronger robustness and adaptability.

[0057] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0058] In summary, the present invention realizes efficient adaptive reconstruction of different image resolutions, numbers of spectral bands and noise levels through an adaptive deep prior compressed sensing spectral reconstruction method, combined with a pre-trained noise level estimation network and an adaptive noise reduction network, which can significantly improve the robustness and accuracy of the algorithm. The use of three-dimensional convolution and Transformer architecture for noise reduction breaks through the limitations of the number of image channels and resolution in previous methods, and can greatly improve the flexibility, computational efficiency and noise reduction quality of the pre-trained model. The innovative introduction of an adaptive iterative stop detector dynamically monitors the convergence process through the sliding variance method, and optimizes the calculation to achieve linear time complexity, thereby achieving accurate and efficient control of the output, adapting to the differences in different input data, and avoiding the limitations of the fixed number of iterations in traditional methods. This method can be directly applied to various types of compressed sensing-based spectral reconstruction systems without modification, and has extremely high flexibility and practicality.

[0059] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a flow chart of the overall algorithm of the present invention;

[0061] Figure 2 This is the structure diagram of the pre-trained adaptive denoising network;

[0062] Figure 3 Figure 1. Network structure diagram for pre-training noise level estimation.

[0063] Figure 4 A schematic diagram of a computer device provided in accordance with an embodiment of the present invention;

[0064] Figure 5 The block diagram of a chip provided according to one embodiment of the present invention is shown. DETAILED DESCRIPTION

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0066] In the description of the present invention, it is to be understood that the terms “include” and “comprise” indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0067] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0068] It should be further understood that the term "and / or" as used in the present specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0069] It should be understood that although the terms "first," "second," and "third" may be used to describe preset ranges in embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are merely used to distinguish one preset range from another. For example, without departing from the scope of embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0070] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0071] The accompanying drawings illustrate various schematic diagrams of structures according to embodiments disclosed herein. These figures are not drawn to scale; for clarity, some details are exaggerated and some details may be omitted. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate in practice due to manufacturing tolerances or technical limitations. Those skilled in the art may design regions / layers with different shapes, sizes, and relative positions as needed.

[0072] This paper provides an adaptive depth prior-based compressed sensing spectral reconstruction method. By adapting image resolution, number of bands, coding templates, iterative degradation levels, and dynamically determining the conditions for stopping iterations, this method achieves efficient and high-quality image reconstruction. The method's adaptability balances reconstruction speed and image quality, ensuring optimal performance in various reconstruction scenarios.

[0073] See also Figure 1 The application discloses a compressed sensing spectrum reconstruction method based on an adaptive deep prior, and comprises the following steps.

[0074] S1, a compressed observation image is generated through a CASSI system forward process, and then a CASSI system compressed observation problem is modeled, and an optimization problem to be solved is decomposed into two sub-problems of data fidelity and noise reduction;

[0075] A mathematical model of a CASSI system observation process is as follows:

[0076] y=Hx

[0077] Wherein, x is vectorized input spectrum data, y is a vectorized two-dimensional compressed observation image, and H is a sensing matrix of the CASSI system.

[0078] The optimization problem to be solved is as follows:

[0079]

[0080] Wherein, is a data fidelity term for minimizing the error between an estimated value and an actual value, R(x) is a prior term, and tau is a weight coefficient for adjusting the two.

[0081] In order to facilitate solving, an auxiliary variable v is introduced, and formula (1) is rewritten by using a generalized alternating projection algorithm as follows:

[0082]

[0083] Formula (2) is a CASSI measurement solving model established by the method, and one of x and v is fixed to optimize the other, and the prior term and the data fidelity term are respectively optimized and solved.

[0084] S2, for the input two-dimensional compressed observation image generated by the CASSI system, a Euclidean linear projection is performed, the error between a predicted value and an actual value is reduced, and the data fidelity is increased.

[0085] The solving process of the linear projection is that, given v (k) , the updating process of x (k+1) has a closed-form solution, which is regarded as a Euclidean projection of v (k) on a linear manifold y=Hx:

[0086]

[0087] Wherein, (y-Hv (k) ) represents the error between the current observation value y and the predicted output Hv (k) , and the matrix It indicates the direction in the original signal space that can best reduce the error between the current estimate and the actual observation, and then projects the error and superimposes it on the predicted output to make the predicted value closer to the true value.

[0088] S3. Use a pre-trained adaptive noise level estimation network to dynamically estimate the noise level of the data item during each iteration to achieve adaptive adjustment of hyperparameters;

[0089] The output x of step S2 is obtained by pre-training the adaptive noise level estimation network (k+1) Perform noise level estimation, output the noise level estimation map of each channel, and guide the denoising network to match the denoising intensity;

[0090] The adaptive noise level estimation network consists of two parallel pathways. The first pathway uses a series of three-dimensional convolution operations to extract spatial noise features from the input data.

[0091] The second path first performs three-dimensional convolution for feature transformation, then uses global average pooling to compress the spatial dimension, uses one-dimensional convolution in the spectral dimension to construct the modulation weight of the spectral channel, and finally multiplies the weight with the feature map of the first path to obtain the final (k+1) Noise level graphs of the same size Represents x (k+1) The noise level at each pixel position on each channel in ;

[0092] By adding Gaussian noise with random noise levels to multiple spectral datasets to form training data pairs for training, the network can directly estimate the noise level of the input noisy spectral data. Since the training process is independent of the spatial resolution of the input data, the number of spectral channels, and the forward process of the CASSI system, it ensures flexibility in adapting to different systems.

[0093] S4. Use a pre-trained adaptive denoising network to denoise the projection result of step S2, using a three-dimensional convolution to slide calculations on the spatial and channel dimensions of the upscaled data, which can adapt to input data of any number of bands and resolution;

[0094] Use the pre-trained adaptive denoising network to denoise the projection result in step S2:

[0095] v (k+1) =D μ (x (k+1) ) (4)

[0096] Among them, D μ (·) represents the denoiser operating at the noise level μ.

[0097] See also Figure 3The denoiser working at the noise level μ is implemented by a pre-trained two-dimensional convolutional Transformer denoising network, and the initial input data is the output of step S2. and the output of step S3 connected Finally, the spectral image after noise reduction is output

[0098] A multi-scale encoder-decoder architecture using skip connections. The encoder consists of one layer of 3D convolution and five Transformer blocks. The input and output feature maps of modules 1, 2, 4, and 6 have the same spatial resolution, while modules 3 and 5 perform convolutional downsampling to capture multi-scale features.

[0099] The decoder part also consists of 5 Transformer blocks and 1 layer of 3D convolution, forming a symmetrical structure with the encoder. Its 1st, 2nd, 4th, and 6th modules do not change the spatial resolution of the input. The 3rd and 5th modules use cubic interpolation to achieve upsampling and restore image details.

[0100] Each Transformer follows the same structure, and the input feature map size is Therefore, all convolution and attention operations will not change the number of spectral bands C, but only affect the number of channels D;

[0101] F first undergoes parallel three-dimensional convolution of the spectral dimension and the spatial dimension, and the output is obtained by fusing the two-dimensional feature map

[0102] The two-dimensional three-dimensional convolution is independent of spatial resolution and the number of spectral channels, relying solely on the number of convolution kernels D in the input and output. This gives the model the ability to adapt to different wavelengths and resolutions. By training on a large number of natural spectral images, the inherent spectral patterns of natural scenes are modeled, enabling denoising of spectral data of arbitrary bands and wavelengths.

[0103] Furthermore, because 3D convolution is used to generate feature maps that capture the positional associations between pixels, there's no need for positional encoding, as in traditional Transformers. Positional encoding depends on the resolution of the input data, and using convolution instead of positional encoding allows the model to adapt to spatial resolution.

[0104] The CASSI system's measurement characteristics retain more information in the spatial dimension, but severely lose information in the spectral dimension, necessitating enhanced information acquisition capabilities in this dimension. Therefore, the feature map after the 3D convolution module is split into two paths: one path is fed into the spectral self-attention module (SSA) to further extract spectral information, while the other path is added to the output of the SSA module to implement residual learning of attention, which stabilizes the training process as the model deepens.

[0105] In order to focus on the changes in spectral dimensions, the SSA module first transforms the input feature map Perform global average pooling in the spatial dimension and reduce the dimension to Then F' undergoes a specific form of linear mapping to obtain the transformed key value and query The two are matrix multiplied and Softmax is used to obtain the attention weight map:

[0106]

[0107] The other way is to pass the feature map F through the linear layer W in D dimension V Transform to get the mapped value Will Perform matrix multiplication with the last two dimensions of V, and then map it through a learnable linear layer to obtain the final output result of spectral attention enhancement.

[0108] The output of the SSA module will continue to input the Self-Modulated Feed-Forward Network for information integration and nonlinear mapping; the input feature map First, the number of channels is doubled by adjusting the linear layer to obtain F' is divided into two paths, one path passes through two linear layers and a GeLU activation function to get the output The other path slices the last dimension of the input tensor, splits it into two paths, calculates the modulation coefficient of its own elements through the activation function and multiplies them element by element, and finally obtains the self-modulation parameter The output of the self-modulating feedforward neural network is F out =F”1+F”2, which is also the denoising output of the entire Transformer module.

[0109] After a series of Transformer modules of different scales, the input noisy data cube completes a round of denoising process. The algorithm will iterate between equations (3) and (4), and each iteration will obtain clearer and lower-noise data.

[0110] S5. Use the adaptive iterative stop detector to detect whether the output result is optimal. If not, return to the projection step to iterate until the adaptive iterative stop detector determines that the current reconstruction result is less than the tolerance, then exit the loop and output the optimal reconstruction result.

[0111] The adaptive iteration detector monitors the algorithm's progress and terminates the program when appropriate. When the algorithm reaches the optimal reconstruction result, continuing to iterate will have negative consequences. Therefore, the adaptive iteration stop detector tracks the time when the reconstruction result reaches the optimal PSNR and stops the iteration. The adaptive iteration detector uses sliding window variance (SWV) to monitor the reconstruction results within a window:

[0112]

[0113] Among them, W is the window length, t represents the current time, and x is the reconstructed output of a certain iteration.

[0114] At the beginning of the iteration, the higher noise level and algorithm processing will cause the difference in the iterative output to be larger, and the sliding variance will also drop rapidly, and gradually decrease as x changes. It will gradually approach x, and the difference between the variance in the current window and the variance in the next window becomes very small. When they remain basically unchanged after P iterations, it is considered that the optimal reconstruction result is achieved and the algorithm loop is exited.

[0115] In addition, the computational complexity of formula (7) is optimized and rewritten as:

[0116]

[0117] Formula (8) is actually composed of x t+w and x t+i The method is composed of the sum of the squares of . When the number of iterations t is updated, only the previous item can be removed and the new item can be added, achieving linear time complexity O(W). The method of the present invention comprehensively considers the results of the previous P+W reconstructions and has linear time complexity. It is more robust to fluctuations in the reconstruction results and can efficiently track changes in the reconstruction results to achieve the output of the optimal result.

[0118] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Accordingly, various aspects of the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as "circuits," "modules," or "platforms."

[0119] In still another embodiment of the present application, an adaptive depth prior compressive sensing spectrum reconstruction system is provided, which can be used to implement the adaptive depth prior compressive sensing spectrum reconstruction method described above. Specifically, the adaptive depth prior compressive sensing spectrum reconstruction system comprises a construction module, a projection module, an estimation module, a denoising module, and an output module.

[0120] The construction module generates a compressed observation image through a CASSI system forward process, and then models a CASSI system compressed observation problem, decomposing an optimization problem to be solved into a prior term and a data fidelity term.

[0121] The projection module performs a Euclidean linear projection on the data fidelity term to obtain a projection result.

[0122] The estimation module uses a pre-trained adaptive noise level estimation network to estimate the noise level of the projection result, and outputs a noise level estimation image of each channel.

[0123] The denoising module uses a pre-trained adaptive denoising network to denoise the projection result obtained by the projection module and the noise level estimation image obtained by the estimation module. A three-dimensional convolution is used to slide and calculate in the spatial dimension and the channel dimension of the upgraded data, so as to adaptively calculate input data of any waveband number and resolution, and obtain a neural network estimation result under a single iteration of the prior term sub-problem.

[0124] The output module returns to the projection module for iteration if the output result of the denoising module does not reach the optimum, until the current reconstruction result is less than the tolerance, the loop is exited, and the optimal reconstruction result is output.

[0125] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, graphics processing units (GPU), tensor processing units (TPU), digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the adaptive depth prior compressed sensing spectral reconstruction method, including:

[0126] A compressed observation image is generated through the forward process of the CASSI system, and then the compressed observation problem of the CASSI system is modeled, and the optimization problem to be solved is decomposed into a priori terms and data fidelity terms; the obtained data fidelity terms are subjected to Euclidean linear projection to obtain projection results; the noise level of the obtained projection results is estimated using a pre-trained adaptive noise level estimation network, and the noise level estimation map of each channel is output; the obtained projection results and noise level estimation map are denoised using a pre-trained adaptive denoising network, and three-dimensional convolution is used to slide calculations on the spatial dimension and channel dimension of the data after dimensionality increase, and input data of any number of bands and resolution are adaptively calculated to obtain the neural network estimation result under a single iteration of the priori term sub-problem; if the output result does not reach the optimal, return to iterate until the current reconstruction result is less than the tolerance, exit the loop, and output the optimal reconstruction result.

[0127] See also Figure 4The terminal device is a computer device. The computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the adaptive depth prior compressed sensing spectral reconstruction method of the embodiment. To avoid repetition, the details are not described here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the adaptive depth prior compressed sensing spectral reconstruction system of the embodiment. To avoid repetition, the details are not described here.

[0128] The computer device 60 may be a desktop computer, a notebook computer, a PDA, a cloud server, or other computing devices. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. It will be understood by those skilled in the art that Figure 4 This is merely an example of the computer device 60 and does not constitute a limitation of the computer device 60 . The computer device 60 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0129] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0130] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 60.

[0131] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is about to be output.

[0132] See also Figure 5 The terminal device 600 is an electronic device that is implemented as a general-purpose computing device. The components of the electronic device may include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), and a display unit 640.

[0133] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0134] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0135] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0136] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0137] The electronic device 600 can also communicate with one or more external devices 700 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 via the bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0138] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (Non-Volatile Memory), such as at least one disk memory.

[0139] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the adaptive depth prior compressed sensing spectral reconstruction method in the above embodiment; the processor may load and execute the following steps:

[0140] A compressed observation image is generated through the forward process of the CASSI system, and then the compressed observation problem of the CASSI system is modeled, and the optimization problem to be solved is decomposed into a priori terms and data fidelity terms; the obtained data fidelity terms are subjected to Euclidean linear projection to obtain projection results; the noise level of the obtained projection results is estimated using a pre-trained adaptive noise level estimation network, and the noise level estimation map of each channel is output; the obtained projection results and noise level estimation map are denoised using a pre-trained adaptive denoising network, and three-dimensional convolution is used to slide calculations on the spatial dimension and channel dimension of the data after dimensionality increase, and input data of any number of bands and resolution are adaptively calculated to obtain the neural network estimation result under a single iteration of the priori term sub-problem; if the output result does not reach the optimal, return to iterate until the current reconstruction result is less than the tolerance, exit the loop, and output the optimal reconstruction result.

[0141] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0142] Example

[0143] See also Figure 1 , taking a specific CASSI system measurement and reconstruction process as an example, the calculation process of the entire reconstruction method is described as follows:

[0144] 1. CASSI observation process and equation modeling

[0145] Assume that the coding template of SD-CASSI system is A three-dimensional spectral data cube with C spectral channel data The modulation result after the coding template is:

[0146] χ'(:,:,λ)=χ(:,:,λ)⊙M (9)

[0147] Where ⊙ represents element-by-element multiplication, and χ(:,:,c) represents the image of the λth band (1≤λ≤C) in χ. The modulated spectral image χ' passes through the dispersion element and is cAssuming that the dispersion direction is W and the relative displacement between adjacent bands is d, the spectral data after dispersion is:

[0148] χ”(u,v,λ b )=χ'(h,w+d(λ b -λ c ),λ b ) (10)

[0149] in, The data of d(C-1) columns are expanded in the W direction, λ b represents the bth band, (u, v) represents the coordinates of the sensor plane. Finally, the sensor plane is integrated in the spectral dimension, that is, the images of all bands are superimposed to obtain a two-dimensional compressed observation.

[0150]

[0151] where Z is the sensor observation noise with the same dimension as Y.

[0152] For convenience, let the inter-spectral shift d be 1, and the vec(·) operator represents the operation of flattening a multi-dimensional tensor into a one-dimensional vector, so Then the perception matrix H of the SD-CASSI system is written as:

[0153]

[0154] in,

[0155]

[0156] in, is the diagonal matrix after vectorization of the encoding template M, and is a zero matrix because A λi There is a relative displacement between them, and it is necessary to λi The zero padding lines are aligned with the dimension of the final measurement value y; the vectorized model of the SD-CASSI observation process is:

[0157] y=Hx+z (14)

[0158] in,

[0159] Since the number of rows of H is less than the number of columns, directly solving y = Hx will have infinite solutions. However, it is known that the signal x to be solved has certain prior characteristics, such as sparsity and structural characteristics. Therefore, for such an ill-posed inverse problem, the iterative optimization algorithm will add a priori term to the objective function to describe the constraints. should have the characteristics to Constrained to the ideal signal space, the problem to be optimized is:

[0160]

[0161] in, It is a data fidelity term used to minimize the error between the estimated value and the true value, R(x) is a priori term, and τ is the weight coefficient that adjusts the two.

[0162] By introducing the auxiliary variable v, Equation (15) is rewritten as:

[0163]

[0164] 2. Data Fidelity Linear Projection

[0165] By introducing the penalty factor ρ, the augmented Lagrangian function of Equation (16) can be written as:

[0166]

[0167] Given v, let Then we have:

[0168]

[0169] Substituting equation (18) into Hx=y yields:

[0170]

[0171] Substituting ρ into equation (18) yields the expression for updating x given v:

[0172]

[0173] This process can be seen as a k Update x by Euclidean projection on the linear manifold y=Hx (k+1) In addition, by adjusting the linear transformation in Equation (20) and incorporating y into the iterative process, the convergence speed of the algorithm can be accelerated:

[0174]

[0175] y k+1 =y k +(y k -Hv k ) (twenty two)

[0176] Given x k+1 Solve for v k+1 The subproblem can be viewed as the problem of k+1Whether the denoising network can flexibly adapt to input data with widely varying noise levels is a key issue in algorithm optimization. Furthermore, the above projection process is a vectorized representation of the mathematical description. Due to the special properties of the HHT diagonal matrix, the actual calculation of the update process of Equations (21) and (22) only requires matrix element-by-element operations and can be completed directly using two-dimensional matrices without the need for vectorized calculations.

[0177] 3. Pre-trained noise level estimation network

[0178] The task of the pre-trained noise level estimation network is to estimate the noise level at each position in each band of the noisy spectral image, thereby assisting the denoising network to achieve a more accurate denoising process. Given a noisy hyperspectral data x of size H×W×C k+1 , first upgrade it to The three-dimensional convolution on the four-dimensional tensor can adapt to different H, W and C, only affecting the final convolution feature dimension, thereby achieving adaptability to the input data size and number of channels.

[0179] See also Figure 2 The network structure of the noise level estimation network is a dual-path design, in which one path uses two sets of 3×3×3 three-dimensional convolutions with a step size of 1 and a padding of 1 and GeLU activation functions to extract features between channels and output a feature map with the same size as the input data. The other path first uses a 3×3×3, step size 1, and padding 1 three-dimensional convolution layer to map the original data to the feature space, and then uses global average pooling to compress the spatial dimension of the feature map to obtain a vector of size 1×1×C×1. Then, a set of 1×3, step size 1, and padding 1 one-dimensional convolution and GeLU activation function are performed on the vector to obtain the weight coefficient of the spectral dimension, which is used to adjust the overall difference in noise levels between different spectral channels. Finally, the feature map F1 is multiplied by the weight coefficient to obtain the noise level estimation map of each channel of the input data.

[0180] Because the model is adaptable to channels and resolutions and is independent of the CASSI system, it can be jointly trained on multiple large spectral datasets. By adding Gaussian noise with random noise levels to the original images, large-scale training data can be generated. After sufficient training, it can be directly applied to this method without the need for retraining for a specific system.

[0181] 4. Pre-trained adaptive denoising network

[0182] Given a noisy hyperspectral data x of size H×W×C k+1 and the noise level estimate from step 3 The input data to be denoised is composed of concatenated structures Where H, W, and C are not fixed, the network supports denoising hyperspectral images of any resolution and any number of bands. The denoising network is jointly pre-trained on multiple spectral datasets, allowing for direct denoising without prior training. The following describes the two-dimensional convolutional Transformer denoising network used in this embodiment in detail.

[0183] See also Figure 3 The entire noise reduction network consists of 12 processing modules connected in series and skipped. The encoder and decoder each contain 6 components, forming a symmetrical structure. In the encoder part, the input data First, a layer of 3D convolution transforms the features. Then, three layers of downsampling and Transformer blocks are used to extract feature information at different scales. Each downsampling step halves the size of the input spectral image and doubles the number of feature maps. Batch normalization layers are used between each block to enhance training stability. Each Transformer block utilizes 3D convolution in the spectral and spatial dimensions, along with a spectrally guided self-attention module, to model the structure and features of the data cube. The bottom-level Transformer generates a latent spatial encoding of the denoised spectral data.

[0184] The decoder reconstructs a denoised spectral data cube from the Caintic space encoding. This can be seen as the inverse process of the encoder. It is processed by three upsampling Transformer blocks and fuses the encoder outputs at the corresponding levels. Each fusion layer adds more detail to the reconstruction result. Finally, the projected output of the three-dimensional convolution is fused with the original cube to achieve residual learning.

[0185] Input Data The specific processing steps are as follows:

[0186] 1) Input 3D convolution module No. 1. The 3D convolution module consists of two layers of 3D convolution layers in series. The first layer has 2 input channels and 16 output channels, using a 3×3 convolution kernel, a stride of 1, and a padding of 1. The second layer has 16 input channels and 16 output channels, using a 3×3×3 (depth x height x width) convolution kernel, a stride of 1, and a padding of 1. The final output feature map is

[0187] 2) Feature map F1 is fed into the second two-dimensional convolutional Transformer module, which consists of three parts connected in series:

[0188] 2.1) The first part is a two-dimensional convolution layer, which performs three-dimensional convolution operations in the spatial dimension and spectral dimension respectively. The spatial convolution is a single layer of three-dimensional convolution with 16 input channels and 16 output channels, using a 3×1×1 convolution kernel, a stride of 1, and padding of 1×0×0. The spectral dimension is composed of two layers of three-dimensional convolution + GeLU activation function, with 16 input channels and 16 output channels, using a 1×3×3 convolution kernel, a stride of 1, and padding of 0×1×1. Finally, the sum of the spectral dimension and spatial dimension feature maps is output, denoted as This part keeps the number of channels unchanged and only uses two-dimensional convolution to extract features.

[0189] 2.2) The second part is the spectral dimension attention layer, the input First, 3D batch normalization is performed to make the input data have zero mean and unit variance, and then it is divided into two paths. One path is not processed and superimposed on the output of the spectral dimension attention to form residual learning. 2-1 It is necessary to calculate the AttentionMap, then use V to multiply the AttentionMap and then perform a linear transformation to obtain the output features. First, F 2-1 By global average pooling, only the spectral dimension is retained to obtain the feature map To maintain adaptability, instead of using learnable linear layers to calculate Q and K separately, we use The correlation between rows and columns in the matrix is ​​calculated as AttentionMap in the form of , and then the learnable linear layer W1 is used to obtain The final output feature map Among them, W2 is also a learnable linear layer.

[0190] 2.3) The third part is the self-tuning feedforward neural network, the input First, use the linear transformation layer to double the number of channels and get Then it is divided into two processing paths. The first path is restored to the channel through the GeLU activation function and linear transformation layer. The other way is to calculate the self-modulation coefficient. The number of channels of F' is divided into two. One part is passed through the Sigmoid function and then the Hadamard product is performed with the other part to obtain the self-modulation coefficient. Finally, the feature map of output No. 2 is obtained

[0191] 3) Feature map F2 is input to the No. 3 downsampling Transformer module, which consists of three parts. The first part is a layer of 3D downsampling convolution with 16 input channels and 32 output channels. It uses a 3×3×3 convolution kernel, a step size of 1×2×2, and a padding of 1. The feature map output by this convolution layer is That is, the spatial resolution is downsampled to 1 / 2 of the original, and the number of channels is doubled. The second and third parts have the same structure as the corresponding parts of module 2, but they work on the downsampled feature map. Therefore, the feature map output by module 3 is

[0192] 4) Modules 4, 5, 6, and 7 receive the output of the previous module as input, and the dual-dimensional and downsampling Transformer modules are performed alternately. The processing process is the same as that of modules 2 and 3.

[0193] 5) The decoding process begins from module 7. The difference between the decoding process and the encoding module is:

[0194] Fusion of results at the same level, for example, the sum of the outputs of modules 5 and 7 is used as the input of module 8, the sum of the outputs of modules 4 and 8 is used as the input of module 9, and so on;

[0195] Use the bicubic interpolation algorithm to achieve super-resolution of the feature map and restore it to the same level of resolution before downsampling;

[0196] After upsampling, the number of channels is reduced to half of the original number, restoring the number of channels at the same level before doubling.

[0197] 6) After the decoding operation of the 6 modules of the decoder, module 12 finally outputs the noise residual learned by the network The final denoised output Y is the sum of the original input X and the noise residual D, completing the denoising.

[0198] 5. Iteration Stop Detector

[0199] The adaptive iteration stop detector maintains a reconstruction result queue of length W, the sum S of all elements in the queue, and the sum S of the squares of all elements 2 The latest result Y of each iteration i The data will be inserted into the end of the queue and the earliest data will be removed. 2 Subtract the data that has been moved out of the window and add the newly added data; then use the new S and S 2 Calculate the sliding variance:

[0200]

[0201] The detector also has a SWV of length P = {v1,v2,...,v P} queue, each updated sliding variance will update the queue synchronously, when the sliding variance in the queue meets Where μ is the error tolerance, then exit the algorithm loop and output the best reconstruction result, otherwise jump to formula (21) to continue the loop.

[0202] In summary, the present invention provides an adaptive deep prior compressed sensing spectral reconstruction method and system, which utilizes a wide range of spectral data sets for pre-training, is independent of the encoding template and system forward process of a specific CASSI system, and achieves efficient and high-quality noise reduction on input data of arbitrary resolution and number of channels by data dimensionality upgrading. It also has a noise level estimation and adaptive iteration stop detection mechanism, which minimizes the human intervention factor of the algorithm. It can be "plug and play" without training and is applicable to various CASSI spectral imaging systems, especially for reconstruction systems with different numbers of bands. It has extremely strong flexibility and adaptability, can effectively reduce the computational burden of training algorithms for different systems, and improves the versatility of the reconstruction algorithm.

[0203] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0204] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0205] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0206] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms.

[0207] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0208] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0209] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0210] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0211] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0212] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0213] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A method for compressed sensing spectrum reconstruction based on adaptive depth prior, characterized in that: The following steps are involved: S1. Generate a compressed observation image through the forward process of the CASSI system, then model the compressed observation problem of the CASSI system and decompose the optimization problem to be solved into a priori terms and data fidelity terms; S2. Perform Euclidean linear projection on the data fidelity item obtained in step S1 to obtain a projection result; S3. Use a pre-trained adaptive noise level estimation network to perform noise level estimation on the projection result obtained in step S2, and output a noise level estimation map for each channel; S4. Use the pre-trained adaptive denoising network to denoise the projection result obtained in step S2 and the noise level estimation map obtained in step S3. Use three-dimensional convolution to slide calculations on the spatial dimension and channel dimension of the dimensionalized data, adaptively calculate the input data of any number of bands and resolution, obtain the neural network estimation result under a single iteration of the prior term sub-problem in step S1, and use the pre-trained adaptive denoising network to denoise the projection result: in, is the noise level The denoiser works below, is the spectral image after noise reduction; Noise level The denoiser working below is implemented by a pre-trained two-dimensional convolutional Transformer denoising network, and the initial input data is the output of step S2. and the output of step S3 connected , and finally output the denoised spectral image ; Input feature map size First, parallel three-dimensional convolution of the spectral dimension and the spatial dimension is performed, and the output is obtained by fusing the two-dimensional feature map. ; After the 3D convolution module, the feature map is divided into two paths. One path is input into the spectral self-attention module to further extract the spectral dimension information, and the other path is added to the output of the spectral self-attention module to realize the residual learning of attention. The spectral self-attention module takes the input feature map Perform global average pooling in the spatial dimension and reduce the dimension to ;Then After a specific form of linear mapping, the transformed key value is obtained and query , the two perform matrix multiplication and use Softmax to get the attention weight map ; Another way is to convert the feature map exist Dimensionally passed through a linear layer Transform to get the mapped value , the attention weight map and The last two dimensions of are matrix multiplied, and then mapped by a learnable linear layer to obtain the final output of spectral attention enhancement; The output of the spectral self-attention module is further input into the self-modulating feedforward neural network for information integration and nonlinear mapping; the input feature map First, the number of channels is doubled by adjusting the linear layer to obtain ; It is divided into two paths. One path passes through two linear layers and a GeLU activation function to get the output , the other way slices the last dimension of the input tensor, splits it into two paths, calculates the modulation coefficient of its own elements through the activation function and multiplies them element by element, and finally obtains the self-modulation parameter , the output of the self-modulating feedforward neural network is , as the denoising output of the entire Transformer module; S5. If the output result of step S4 is not optimal, return to step S2 and iterate until the current reconstruction result is less than the tolerance. Then exit the loop and output the optimal reconstruction result. Use the adaptive iterative stop detector to detect whether the output result is optimal. The adaptive iterative detector uses sliding variance to monitor the reconstruction result within a window: in, is the window length, Indicates the current moment, for The sliding variance at the moment, for From the moment The output result of this time is for From the moment The output results of times.

2. The adaptive depth prior compressed sensing spectrum reconstruction method according to claim 1, characterized in that: In step S1, the CASSI system compression observation problem is modeled, and the optimization problem to be solved is decomposed into a priori terms and data fidelity terms as follows: in, is the vectorized input spectral data, To vectorize the two-dimensional compressed observation image, is the perception matrix of the CASSI system, To adjust the weight coefficients of the two, is an auxiliary variable.

3. The adaptive depth prior compressed sensing spectrum reconstruction method according to claim 1, characterized in that: In step S2, Euclidean linear projection is performed as follows: in, is the current observation value With the predicted output The error between is the denoised spectral image of the kth iteration, is the transposed matrix of the perception matrix, It is the outer product matrix of the perception matrix and the transposed perception matrix.

4. The adaptive depth prior compressed sensing spectrum reconstruction method according to claim 1, characterized in that: In step S3, the noise level estimation network includes two parallel paths. The noise level estimation network is trained by adding Gaussian noise with random noise levels to multiple spectral data sets to form training data. The trained noise level estimation network can estimate the noise level of the input noise spectral data.

5. The adaptive depth prior compressed sensing spectrum reconstruction method according to claim 4, characterized in that: The first path uses a series of three-dimensional convolution operations to extract spatial noise features from the input data; The second path first performs three-dimensional convolution to transform the features, then uses global average pooling to compress the spatial dimension, uses one-dimensional convolution in the spectral dimension to construct the modulation weight of the spectral channel, and finally multiplies the weight with the feature map of the first path to obtain the final Noise level graphs of the same size ,express The noise level at each pixel location on each channel in .

6. The adaptive depth prior compressed sensing spectrum reconstruction method according to claim 1, characterized in that: In step S4, the encoder part consists of one layer of 3D convolution and five Transformer blocks. The spatial resolution of the input and output feature maps of modules 1, 2, 4, and 6 are consistent. Modules 3 and 5 perform convolution downsampling operations to capture multi-scale features. The decoder part consists of 5 Transformer blocks and 1 layer of 3D convolution, forming a symmetrical structure with the encoder. The 1st, 2nd, 4th, and 6th modules do not change the spatial resolution of the input, and the 3rd and 5th modules use cubic interpolation to achieve upsampling.

7. A compressed sensing spectral reconstruction system with adaptive depth prior, characterized by: include: The construction module generates compressed observation images through the forward process of the CASSI system, then models the compressed observation problem of the CASSI system and decomposes the optimization problem to be solved into a priori terms and data fidelity terms; The projection module performs Euclidean linear projection on the data fidelity item to obtain the projection result; The estimation module uses a pre-trained adaptive noise level estimation network to estimate the noise level of the projection results and outputs a noise level estimation map for each channel; The denoising module uses a pre-trained adaptive denoising network to denoise the projection results obtained by the projection module and the noise level estimation map obtained by the estimation module. It uses three-dimensional convolution to slide calculations on the spatial and channel dimensions of the upscaled data, adaptively calculates input data of any number of bands and resolution, obtains the neural network estimation result under a single iteration of the prior term sub-problem, and uses the pre-trained adaptive denoising network to denoise the projection results: in, is the noise level The denoiser works below, is the spectral image after noise reduction; Noise level The denoiser working below is implemented by a pre-trained two-dimensional convolutional Transformer denoising network, and the initial input data is the output of step S2. and the output of step S3 connected , and finally output the denoised spectral image ; Input feature map size First, parallel three-dimensional convolution of the spectral dimension and the spatial dimension is performed, and the output is obtained by fusing the two-dimensional feature map. ; After the 3D convolution module, the feature map is divided into two paths. One path is input into the spectral self-attention module to further extract the spectral dimension information, and the other path is added to the output of the spectral self-attention module to realize the residual learning of attention. The spectral self-attention module takes the input feature map Perform global average pooling in the spatial dimension and reduce the dimension to ;Then After a specific form of linear mapping, the transformed key value is obtained and query , the two perform matrix multiplication and use Softmax to get the attention weight map ; Another way is to convert the feature map exist Dimensionally passed through a linear layer Transform to get the mapped value , the attention weight map and The last two dimensions of are matrix multiplied, and then mapped by a learnable linear layer to obtain the final output of spectral attention enhancement; The output of the spectral self-attention module is further input into the self-modulating feedforward neural network for information integration and nonlinear mapping; the input feature map First, the number of channels is doubled by adjusting the linear layer to obtain ; It is divided into two paths. One path passes through two linear layers and a GeLU activation function to get the output , the other way slices the last dimension of the input tensor, splits it into two paths, calculates the modulation coefficient of its own elements through the activation function and multiplies them element by element, and finally obtains the self-modulation parameter , the output of the self-modulating feedforward neural network is , as the denoising output of the entire Transformer module; In the output module, if the output result of the denoising module is not optimal, the projector module is returned to iterate until the current reconstruction result is less than the tolerance, the loop is exited, and the optimal reconstruction result is output. The adaptive iterative stop detector is used to detect whether the output result is optimal. The adaptive iterative detector uses sliding variance to monitor the reconstruction result within a window: in, is the window length, Indicates the current moment, for The sliding variance at the moment, for From the moment The output result of this time is for From the moment The output results of times.

Images