A deep learning-based computational continuous spectrum imaging method and system for reconstructing arbitrary bands

Abstract

This invention provides a computational continuous spectral imaging method and system based on deep learning for reconstructing arbitrary spectral bands. It relates to the fields of computational spectral imaging, spectral image reconstruction, and signal processing. The method includes acquiring hyperspectral images and compressed measurement data, constructing a training dataset, building a deep unrolling framework, training the deep unrolling framework based on the training dataset to obtain a weight file for the trained framework, and performing inference operations on the compressed measurement data based on the trained framework to obtain a continuous spectral image. In application, based on the continuous spectral image obtained from arbitrary spectral bands or by combining with an external model, the method outputs the spectral image required for reconstruction, and performs related downstream tasks based on the reconstructed spectral image. This invention changes the goal of the reconstruction task by storing the spectral image in the form of a set of continuous equations, enabling the network output to quickly obtain the spectrum and its higher-order derivatives at any location, thus improving flexibility in downstream applications.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computational spectral imaging, reconstruction algorithm of spectral image and signal processing, and in particular to a method and system for reconstructing continuous spectral imaging based on deep learning. BACKGROUND

[0002] Spectral images have a wide range of applications in many fields such as agriculture and remote sensing, which reflect the absorption curve of the composition of the measured scene in the light source. By analyzing the absorption curve of the spectral image, the composition change of the scene can be deduced.

[0003] Traditional spectral measurement methods rely on scanning methods, including sweeping, pushing and filtering. These methods require multiple exposures to obtain a complete spectral cube, which is not suitable for capturing dynamic scenes. In recent years, with the development of deep learning and compressed sensing theory, computational spectral imaging has been proposed and developed rapidly. Coded aperture snapshot spectral imaging technology (Coded Aperture Snapshot Spectral Imaging, CASSI), computed tomography spectral imaging technology (Computed-tomography Imaging Spectrometer, CTIS) and RGB image direct reconstruction spectral image technology have emerged. These methods solve a ill-posed problem and reconstruct the real spectral image through deep learning method. Existing methods can well reconstruct sparse multispectral images and hyperspectral images, but it is difficult to further reconstruct more dense spectral images. Some methods combine super-resolution tasks to improve the number of spectral channels of the reconstructed spectral image, but these methods improve the resolution of the reconstructed image by combining learned implicit features. Since super-resolution is performed in the entire spectral dimension, it needs to learn all compressed sensing reconstructed spectral images, which consumes a lot of computing resources, and the network cannot be expanded once it is trained. The above multispectral image reconstruction, hyperspectral image reconstruction and super-resolution hyperspectral reconstruction belong to discrete spectral image reconstruction, and cannot obtain the required spectral band of the scene in the downstream task. In summary, it is necessary to design a method and system for reconstructing continuous spectral imaging based on deep learning. SUMMARY

[0004] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide a method and system for reconstructing continuous spectral imaging based on deep learning.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] This invention provides a computational continuous spectrum imaging method based on deep learning capable of reconstructing arbitrary bands, comprising:

[0007] Pre-training phase:

[0008] Step 1: Acquire hyperspectral images and their corresponding compressed measurement data to construct a training dataset for hyperspectral images;

[0009] Step 2: Construct a deep unfolding framework;

[0010] Step 3: Train the deep unfolding framework based on the training dataset of hyperspectral images to obtain the weight file of the trained deep unfolding framework;

[0011] Usage phase:

[0012] Step 4: Perform inference operations on the compressed measurement data based on the trained depth unfolding framework to obtain a continuous spectral image;

[0013] Step 5: Based on the continuous spectral image obtained by combining arbitrary spectral bands or external models, output the spectral image required for reconstruction, and perform related downstream tasks based on the reconstructed spectral image.

[0014] Preferably, in step 1, acquiring hyperspectral images and their corresponding compressed measurement data to construct a training dataset for hyperspectral images specifically involves:

[0015] Acquire hyperspectral images and their corresponding compressed measurement data, and construct a training dataset of hyperspectral images, where each compressed measurement data and its corresponding hyperspectral image constitute a training sample.

[0016] Preferably, in step 2, the deep unfolding framework is constructed as follows:

[0017] The mathematical formula for defining the task is:

[0018] Y = ΦHX + G (1)

[0019] In the formula, Y is the measured value of the coded and compressed spectral image, X is the real spectral image, H is the downsampling operation in the spectral dimension, Φ is the compressed sensing matrix, and G is noise;

[0020] Based on its design depth, the framework is as follows:

[0021] (2)

[0022] In the formula, For the spectral image to be reconstructed without considering downsampling, For the network after k+1 iterations, the previous round The output results after learning For compressed sensing matrix, For k iterations, the The output results after learning After k+1 iterations The output results after learning It is a data fidelity item. It is a priori. and It's a hyperparameter. It is about Auxiliary variables, It is to let and The penalty parameter for proximity to the same fixed point. It is a continuity fidelity item. To solve the pre-designed a priori formula for Z after k+1 iterations, a deep learning-based deep unfolding network is designed according to the iterative solution steps, as follows:

[0023] (3)

[0024] In the formula, and It's a hyperparameter. for The array formed for The array formed , Hyperparameters are used for k+1 iterations. This is a hyperparameter estimation module. P (·)and D (·) represents the deep learning modules designed for the deep unfolding framework, namely the gradient descent module and the denoising module, respectively. E (·) is the parameter estimation module for continuous functions. This is a deep learning module based on fidelity term design within a deep expansion framework. The input is the parameter enclosed in parentheses, and the output is... , This is a deep learning module designed for the denoising term in a deep unfolding framework. A deep learning module designed for parameter estimation of continuous functions. For the spectral band, To uniformly sample bands in an array with sequence number n, where n is a positive integer, It is a discretized module, the network passes through m After several iterations, the final output is a continuous spectral function. .

[0025] Preferably, in step 3, the depth unfolding framework is trained based on the training dataset of hyperspectral images to obtain the trained depth unfolding framework, specifically as follows:

[0026] A depth unfolding framework is constructed by selecting several samples from the training dataset as input, and the spectral image reconstructed by the model is generated.

[0027] The reconstructed continuous spectral image is discretized into a hyperspectral image and compared with the corresponding real hyperspectral image. The optimal solution is found pixel by pixel using gradient descent based on the constraint function, while updating the weight parameters in the model.

[0028] The process is iterated until the reconstruction results meet the conditions. Then, training is stopped and the network structure and model weight files are saved to obtain the trained deep expansion framework.

[0029] Preferably, in step 4, inference operations are performed on the compressed measurement data based on the trained depth unfolding framework to obtain a continuous spectral image. Specifically, inference operations are performed on the compressed measurement data based on the trained depth unfolding framework to obtain a continuous spectral image, wherein the discrete spectral image data stored directly in matrix form is converted into a set of coefficients of a continuous function stored on a pixel-by-pixel basis.

[0030] Preferably, in step 5, based on the continuous spectral image obtained by combining arbitrary spectral bands or external models, the desired reconstructed spectral image is output, and related downstream tasks are performed based on the reconstructed spectral image, specifically:

[0031] A spectral image is obtained by performing matrix multiplication between the numerical values ​​or sets of arbitrary spectral bands and the output of a continuous spectral image network, i.e.:

[0032] (4)

[0033] (5)

[0034] The output of the continuous spectrum image network is fused with features or outputs from other models to output a spectral image, as follows:

[0035] (6)

[0036] (7)

[0037] In the formula, For activation function, For weight values, For constant terms, Encoding feature vectors for other models, The computational results of a multilayer perceptron encoding feature vectors for other models.

[0038] This invention also provides a computational continuous spectrum imaging system based on deep learning capable of reconstructing arbitrary bands, applied to the aforementioned computational continuous spectrum imaging method based on deep learning capable of reconstructing arbitrary bands, comprising:

[0039] The data acquisition and processing unit is used to acquire hyperspectral images and their corresponding compressed measurement data, and to construct a training dataset for hyperspectral images.

[0040] The deep unfolding framework building unit, connected to the data acquisition and processing unit, is used to build the deep unfolding framework. The framework features three sequentially executed modules: a denoising module, a parameter estimation module based on continuous functions, and a discretization module.

[0041] The model training unit is connected to the data acquisition and processing unit and the deep unfolding framework construction unit, respectively, and is used to train the deep unfolding framework;

[0042] The inference unit, connected to the model training unit, is used to obtain a continuously expressed spectral image based on the trained deep unfolding framework.

[0043] The downstream task processing unit, connected to the inference unit, is used to fuse the continuous spectral image with the numerical value or set of arbitrary spectral bands, or the output of the external model, to output the spectral image required for reconstruction, and to perform downstream tasks such as classification, segmentation or target tracking.

[0044] Preferably, the system further includes a data storage unit connected to the inference unit for storing continuous spectral images.

[0045] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] This invention provides a computational continuous spectrum imaging method and system based on deep learning for reconstructing arbitrary spectral bands. The method includes acquiring hyperspectral images and their corresponding compressed measurement data, constructing a training dataset for the hyperspectral images, constructing a deep unwrapping framework, training the deep unwrapping framework based on the training dataset for the hyperspectral images to obtain the trained deep unwrapping framework, performing inference operations on the compressed measurement data based on the trained deep unwrapping framework to obtain a continuous spectrum image, outputting the required reconstructed spectral image based on the continuous spectrum image obtained by combining arbitrary spectral bands or external models, and performing related downstream tasks based on the reconstructed spectral image. This invention achieves both denoising and spectral curve estimation by changing the purpose of the reconstruction task. It enables continuous spectral imaging of arbitrary bands using only a single network, resulting in lower computational cost and faster reconstruction speed. In contrast, existing methods such as multispectral image reconstruction, hyperspectral image reconstruction, and super-resolution hyperspectral image reconstruction, which are all discrete spectral image reconstruction methods, cannot achieve spectral imaging of arbitrary bands, are superior. The proposed method learns spectral curves pixel by pixel, enabling continuous spectral imaging of arbitrary bands. Furthermore, compared to traditional fitting methods based on deep learning network outputs, it achieves reconstruction speed by an order of magnitude. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the method flow of an embodiment of the present invention;

[0049] Figure 2 A schematic diagram illustrating the construction of the deep expansion framework;

[0050] Figure 3 This is a diagram illustrating the usage phase.

[0051] Figure 4 This is a schematic diagram illustrating the learning process of the reconstruction method on a pixel-by-pixel basis in the spectral dimension.

[0052] Figure 5 This is a schematic diagram of the data storage format for spectral images;

[0053] Figure 6 This is a schematic diagram of the output of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] The purpose of this invention is to provide a computational continuous spectral imaging method and system based on deep learning that can reconstruct arbitrary bands. By changing the purpose of the reconstruction task, it simultaneously achieves denoising and spectral curve estimation. Continuous spectral imaging of arbitrary bands can be achieved using only a single network, resulting in lower computational costs and faster reconstruction speeds. In contrast, existing methods such as multispectral image reconstruction, hyperspectral image reconstruction, and super-resolution hyperspectral image reconstruction, which are all discrete spectral image reconstruction methods, cannot achieve spectral imaging of arbitrary bands. The proposed method learns spectral curves pixel-by-pixel, enabling continuous spectral imaging of arbitrary bands. Furthermore, compared to traditional fitting methods based on deep learning network outputs, it achieves reconstruction speeds that are orders of magnitude faster.

[0056] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] like Figure 1 As shown, this invention provides a computational continuous spectrum imaging method based on deep learning that can reconstruct arbitrary bands, comprising:

[0058] This method includes a pre-training phase and an application phase. In the pre-training phase, hyperspectral images and their corresponding compressed measurement data are first acquired, and a training dataset for the hyperspectral images is constructed. Each compressed measurement data point and its corresponding hyperspectral image constitute a training sample. Several samples are selected from the training sample set and input into the constructed depth unrolling framework to generate the reconstructed spectral image. Then, the reconstructed continuous spectral image is discretized into the hyperspectral image, and compared with the real hyperspectral image. Pixel-by-pixel, gradient descent is used to find the optimal solution based on the constraint function, while simultaneously updating the weight parameters in the model. This process is iterated until the reconstruction result meets the conditions, at which point training stops, and the network structure and model weight files are saved.

[0059] During the usage phase, the model and weight files stored during the training phase are used to perform inference operations on the collected compressed measurement data to obtain and store continuous spectral images. In downstream tasks, the required arbitrary band spectral images, multispectral images, or hyperspectral images are reconstructed from the continuous spectral images output by the network, based on the numerical values ​​or sets of arbitrary spectral bands, features or outputs of natural language models or other models. These spectral images are then stored and used for tasks such as classification, segmentation, 3D reconstruction, or target tracking. To evaluate the reconstruction quality of the system, corresponding real spectral images are acquired simultaneously with the acquisition of arbitrary band spectral images, multispectral images, or hyperspectral images, and the system is evaluated using evaluation metrics.

[0060] They are introduced separately:

[0061] Pre-training phase:

[0062] Step 1: Acquire hyperspectral images and their corresponding compressed measurement data to construct a training dataset for hyperspectral images;

[0063] Step 2: Construct a deep unfolding framework;

[0064] Step 3: Train the depth unfolding framework based on the training dataset of hyperspectral images to obtain the trained depth unfolding framework;

[0065] Usage phase:

[0066] Step 4: Perform inference operations on the compressed measurement data based on the trained depth unfolding framework to obtain a continuous spectral image;

[0067] Step 5: Based on the continuous spectral image obtained by combining arbitrary spectral bands or external models, output the spectral image required for reconstruction, and perform related downstream tasks based on the reconstructed spectral image.

[0068] like Figure 2 As shown, in step 1, hyperspectral images and their corresponding compressed measurement data are acquired to construct a training dataset for hyperspectral images, specifically as follows:

[0069] Acquire hyperspectral images and their corresponding compressed measurement data, and construct a training dataset of hyperspectral images, where each compressed measurement data and its corresponding hyperspectral image constitute a training sample.

[0070] In step 2, the deep unfolding framework is constructed, specifically as follows:

[0071] The mathematical formula for defining the task is:

[0072] Y = ΦHX + G (1)

[0073] In the formula, Y is the measured value of the coded and compressed spectral image, X is the real spectral image, H is the downsampling operation in the spectral dimension, Φ is the compressed sensing matrix, and G is noise;

[0074] The framework needs to simultaneously address two irreversible reconstruction tasks: compressed sensing and downsampling. Based on this task, the depth-expansion framework is designed as follows:

[0075] (2)

[0076] In the formula, For the spectral image to be reconstructed without considering downsampling, For the network after k+1 iterations, the previous round The output results after learning For compressed sensing matrix, In response to Data fidelity item, For k iterations, the The output results after learning For k+1 iterations The output results after learning and It's a hyperparameter. For continuity fidelity items, among which For the prior formula designed for Z after k+1 iterations, It is a priori. It is about Auxiliary variables, It is to let and The penalty parameter for proximity to the same fixed point. It is a continuity-fidelity term. Based on the iterative solution steps, a deep learning-based network is designed as follows:

[0077] (3)

[0078] In the formula, for The array formed for The array formed Use hyperparameters for k+1 iterations. This is a deep learning module based on fidelity term design within a deep expansion framework. The input is the parameter enclosed in parentheses, and the output is... , This is a deep learning module designed for the denoising term in a deep unfolding framework. A deep learning module designed for parameter estimation of continuous functions. For the spectral band, This is to uniformly sample the band with sequence number n in the array, where n is a positive integer; and It's a hyperparameter. This is a hyperparameter estimation module. P (·)and D (·) represents the deep learning modules designed for the deep unfolding framework, namely the gradient descent module and the denoising module, respectively. E (·) is the parameter estimation module for continuous functions. It is a discretized module, the network passes through m After several iterations, the final output is a continuous spectral function. ;

[0079] To elaborate further, the computational spectral imaging task has been upgraded to a computational continuous spectral imaging task, which requires solving the following simultaneously: and Two questions were raised, and a depth-expansion framework based on formula (3) was designed according to the new task. The method is as follows:

[0080] First, the hyperparameters required by the network are estimated, and the learned network modules are... Subsequently, a deep learning module was designed using optimization methods. P (·)and D (·), alternating iterative gradient descent and denoising are achieved. This method can also be simplified to a deep learning module based on optical path invertibility. The method is described as follows: In the formula, For the network after k iterations, the previous round The output results after learning To perform operations using compressed sensing matrices based on the reversible property of optical paths The encoding process is then performed, followed by the use of the designed parameter estimation module for continuous functions. Estimate the continuous spectral curves of all pixels, and then utilize them in an iterative framework. What I learned By pressing the sampled spectral bands, a discretized spectral image is obtained and fed into the next iteration. Finally, after all iterations are complete, the network outputs the optimal reconstructed continuous spectral function. The weights;

[0081] The reconstruction method requires the use of a parameter estimation module. Estimate the continuous spectrum curve of the pixel This method extracts global features in the spectral dimension through a deep learning module. By learning the continuity relationships of discrete points, it estimates explicit functions or sets of functions per pixel in the spectral dimension. The characteristics of these functions or sets of functions are:

[0082] Function types include spline functions, radial basis functions, polynomial functions, machine learning functions, piecewise polynomial functions, and polynomial functions. The spline function class includes quadratic splines, cubic splines, quartic splines, and B-spline curves; the radial basis function class includes multi-quadratic kernel radial basis functions, inverse multi-quadratic kernel radial basis functions, and thin-plate kernel radial basis functions; the polynomial function class includes linear polynomials and Emilt polynomials; the machine learning class includes K-nearest neighbor smoothing functions, decision tree smoothing functions, and random forest smoothing functions; the piecewise polynomial function class includes piecewise linear polynomial functions, piecewise quadratic polynomial functions, and piecewise cubic polynomial functions; and the polynomial function class includes quadratic polynomial functions and cubic polynomial functions. The specific formulas are shown below:

[0083] (4)

[0084] (5)

[0085] (6)

[0086] (7)

[0087] (8)

[0088] (9)

[0089] (10)

[0090] (11)

[0091] (12)

[0092] (13)

[0093] (14)

[0094] (15)

[0095] (16)

[0096] (17)

[0097] (18)

[0098] (19)

[0099] (20)

[0100] In the formula, Let be the pixel-by-pixel spectral curve of a real continuous spectral image with respect to pixel x, where x is the spectral range covered by the compressed measurement data. For any spectral band on the spectrum, formula (4) is a linear function, where , The coefficients of the linear function are given by equation (5), equation (6) is a quadratic spline, equation (7) is a cubic spline, and equation (8) is a B-spline curve. , , , and The coefficients of the spline function are... Let p be the basis functions of the spline. For p-1 degree spline basis functions, The node vector is formed by Let p be a control point. For zero-order spline basis functions, For node vectors, For the i-th item in the node vector, For the (i+1)th item in the node vector, Let x be the (i+p+1)th term in the node vector; formula (9) is a piecewise quadratic polynomial function, x i It is the i-th term in any spectral band of the compressed measurement data covering the spectral range λ, and formula (10) is a piecewise cubic polynomial function, where n The number of bands in the sampled hyperspectral image. i The position of the spectral values ​​of the sampled spectral sequence; Formula (11) is the quadratic kernel radial basis function. For the constant term and the center of the kernel function, For the multiplication of the constant term and the center of the kernel function, formula (12) is the inverse quadratic kernel radial basis function, and formula (13) is the thin-plate kernel radial basis function, where The coefficients of the kernel function, The distance between the spectral band sequences is the Euclidean distance; in formula (14) For Emiltial polynomial functions, To operate on x using Emil's polynomial function; in formula (15) It is a K-nearest neighbor smoothing function. To utilize the K-nearest neighbor smoothing function to operate on x; in formula (6) For decision tree smoothing function, To utilize the decision tree smoothing function to operate on x; in formula (17) For random forest smoothing function, To utilize the random forest smoothing function to operate on x; in formula (18) It is a piecewise linear smoothing function. To utilize a piecewise linear smoothing function to operate on x; formula (19) is a quadratic polynomial function, It is a quadratic polynomial function. To operate on x using a quadratic polynomial function, formula (20) is a cubic polynomial function, where It is a least squares function. , , , The coefficients of the polynomial function are... Operate x using polynomial functions.

[0101] Based on parameter estimation module The coefficients of a continuous function are learned from a set of sparse discrete points, and the process is as follows: The modules for deep learning methods are designed as follows:

[0102] like Figure 4 As shown, deep learning-based parameter estimation module methods include: multilayer perceptron (MLP), convolutional neural network (CNN), recurrent neural network (RNN), attention mechanism network (Attention), state space model (Mamba), and long short-term memory network (LSTM) and their combinations;

[0103] The optimal solution of the network is found using gradient descent based on the constraint function. The designed constraint function is as follows:

[0104] The depth unrolling network is constrained across the entire spectral range of the measurement. First, the data fidelity term is constrained. To impose constraints, To utilize the parameter estimation module Estimate the continuous spectral curve of the pixel, where n is the defined number of samples, for any... Secondly, since the spectral curve of each pixel should remain continuous across the entire spectral dimension, its higher-order derivatives must be consistent. Data fidelity constraints and higher-order derivative consistency constraints can improve the reconstruction accuracy of continuous spectra. The specific formula is as follows:

[0105] (twenty one)

[0106] Total loss. The loss of the 0th term (the loss of the 0th term). The loss of item 1 (Loss of item 1). The loss for item 2 (Loss of item 2). For the true value of the real hyperspectral image, For hyperparameters, This represents the predicted value of the hyperspectral image at sample n+1. Let be the true value of the hyperspectral image at sample n+1, where is the value of the true value at sample n+1. From point to The vector, From point to The vector, From point to The vector, It is a set of discrete points that each pixel represents along the spectral dimension. It is the number of channels per pixel in the spectral dimension. and These are the hyperparameters of the network.

[0107] In step 3, the deep unfolding framework is trained based on the training dataset of hyperspectral images to obtain the trained deep unfolding framework, specifically as follows:

[0108] A depth unfolding framework is constructed by selecting several samples from the training dataset as input, and the spectral image reconstructed by the model is generated.

[0109] The reconstructed continuous spectral image is discretized into a hyperspectral image and compared with the real hyperspectral image. The optimal solution is found pixel by pixel using gradient descent based on the constraint function, while updating the weight parameters in the model.

[0110] The process is iterated until the reconstruction results meet the conditions. Then, training is stopped and the network structure and model weight files are saved to obtain the trained deep expansion framework.

[0111] In step 4, inference operations are performed on the compressed measurement data based on the trained depth unfolding framework to obtain a continuous spectral image. Specifically, the continuous spectral image is obtained by performing inference operations on the compressed measurement data based on the trained depth unfolding framework. This involves converting the discrete spectral image data, which is directly stored in matrix form, into a set of coefficients for a continuous function stored pixel by pixel. The detailed structure of the data includes the horizontal and vertical coordinates in the pixel coordinate system space. , ), and a set of parameters [parameter1, parameter2, ...] describing the spectral curve of that pixel, where each parameter consists of multiple equation coefficients [coefficient1, coefficient2, ...], where the coefficients are determined by one or more function types selected from the above functions, therefore, as Figure 5 As shown, the data storage format for spectral images is as follows: ,in W and H are the length and height of the image, respectively. For length x i Width is y j The value of the first term of the coefficient 'a' for each pixel, and so on for subsequent terms; for example... Figure 3 As shown, in step 5, based on the continuous spectral image obtained by combining arbitrary spectral bands or external models, the desired reconstructed spectral image is output. Based on this reconstructed spectral image, related downstream tasks are performed. Specifically, the spectral image is obtained by performing matrix multiplication between the numerical values ​​or sets of arbitrary spectral bands and the output of the continuous spectral image network.

[0112] (twenty two)

[0113] (twenty three)

[0114] The output of the continuous spectrum image network is fused with features or outputs from other models to output a spectral image, as follows:

[0115] (twenty four)

[0116] (25)

[0117] In the formula, For activation function, For weight values, For constant terms, Encoding feature vectors for other models, The computational results of a multilayer perceptron encoding feature vectors for other models.

[0118] To assess the reconstruction quality, while acquiring spectral images, multispectral images, or hyperspectral images of any band, corresponding real spectral images are also collected. The system is then evaluated using assessment metrics, including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Spectral Angle Matching (SAM). The formula for calculating PSNR is as follows:

[0119] (26)

[0120] In the formula, This indicates the maximum possible value of an image pixel (usually 255 or 1.0). Indicates mean square error;

[0121] The SSIM calculation formula is as follows:

[0122] (27)

[0123] In the formula, and These are the reconstructed hyperspectral image and the corresponding real hyperspectral image, respectively. and These are images and The average brightness. and These are images and The luminance variance. It is an image and The brightness covariance between them. and These are two constants that can be calculated stably.

[0124] The formula for calculating SAM is as follows:

[0125] (28)

[0126] This invention also provides a computational continuous spectrum imaging system based on deep learning capable of reconstructing arbitrary bands, applied to the aforementioned computational continuous spectrum imaging method based on deep learning capable of reconstructing arbitrary bands, comprising:

[0127] The data acquisition and processing unit is used to acquire hyperspectral images and their corresponding compressed measurement data, and to construct a training dataset for hyperspectral images.

[0128] The deep unfolding framework building unit, connected to the data acquisition and processing unit, is used to build the deep unfolding framework. The framework features three sequentially executed modules: a denoising module, a parameter estimation module based on continuous functions, and a discretization module.

[0129] The model training unit is connected to the data acquisition and processing unit and the deep unfolding framework construction unit, respectively, and is used to train the deep unfolding framework;

[0130] The inference unit, connected to the model training unit, is used to obtain a continuously expressed spectral image based on the trained deep unfolding framework.

[0131] The downstream task processing unit, connected to the inference unit, is used to fuse the continuous spectral image with the numerical value or set of arbitrary spectral bands, or the output of the external model, to output the spectral image required for reconstruction, and to perform downstream tasks such as classification, segmentation or target tracking.

[0132] The system also includes a data storage unit connected to the inference unit for storing continuous spectral images.

[0133] like Figure 6 As shown, the network output implemented by the method proposed in this invention is a storage format of a continuous spectral function, which has more flexible applications, and its function expression can interact with various forms of data.

[0134] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0135] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for computing a continuous spectrum image based on deep learning and reconstructing an arbitrary waveband, characterized in that, Comprise: Pre-training stage: Step 1: obtain hyperspectral image and its corresponding compressed measurement data, construct the training data set of hyperspectral image; Step 2: construct deep unfolding framework; Specifically: The mathematical formula of the task is defined as: Y = ΦHX + G (1) In the formula, Y is the measurement value of the coded compressed spectral image, X is the real spectral image, H is the down sampling operation in the spectral dimension, Phi is the compressed sensing matrix, and G is the noise; According to its design deep unfolding framework is: （2） In the formula, For the spectral image to be reconstructed without considering downsampling, For the network after k+1 iterations, the previous round The output results after learning For compressed sensing matrix, For k iterations, the The output results after learning After k+1 iterations The output results after learning It is a data fidelity item. It is a priori. and It's a hyperparameter. It is about Auxiliary variables, It is to let and The penalty parameter for proximity to the same fixed point. It is a continuity fidelity item. To solve the pre-designed a priori formula for Z after k+1 iterations, a deep learning-based deep unfolding network is designed according to the iterative solution steps, as follows: （3） wherein and are hyperparameters, is an array consisting of, is an array consisting of, , is the hyperparameter used for k+1th iteration, is the hyperparameter estimation module, P (·) and D (·) are deep learning modules designed for the deep unfolding framework, representing the gradient descent module and the denoising module respectively, E (·) is the parameter estimation module for continuous functions, is the deep learning module designed based on the fidelity term in the deep unfolding framework, the input is the parameter in the bracket, and the output is , is the deep learning module designed based on the denoising term in the deep unfolding framework, is the deep learning module designed for parameter estimation of continuous functions, is the spectral band, is the band with sequence number n in the array uniformly sampled on the band, and n is a positive integer, is the discretization module, the network goes through m iteration stages, and finally outputs the continuous spectral function ; Step 3: based on the training data set of hyperspectral image, the deep unfolding framework is trained, and the weight file of the trained deep unfolding framework is obtained; Use stage: Step 4: based on the trained deep unfolding framework, the inference operation is carried out on the compressed measurement data, and the continuous spectral image is obtained; Step 5: the obtained continuous spectral image is combined with any spectral band or the obtained continuous spectral image is combined with external model, and the reconstructed spectral image is output, and the related downstream task is carried out based on the reconstructed spectral image.

2. The method of claim 1, wherein, In step 1, the hyperspectral image and its corresponding compressed measurement data are obtained, and the training data set of hyperspectral image is constructed, specifically: The hyperspectral image and its corresponding compressed measurement data are obtained, and the training data set of hyperspectral image is constructed, wherein each compressed measurement data and the corresponding hyperspectral image constitute a training sample.

3. The method of claim 1, wherein, In step 3, based on the training data set of hyperspectral image, the deep unfolding framework is trained, and the trained deep unfolding framework is obtained, specifically: Select several samples from the training data set to input the constructed deep unfolding framework, generate the continuous spectral image reconstructed by the model; Disperse the reconstructed continuous spectral image to the hyperspectral image, and compare it with the real corresponding hyperspectral image, find the optimal solution according to the constraint function using gradient descent, and update the weight parameters in the model; Iterate until the reconstruction result meets the condition, stop training and save the network structure and model weight file, and obtain the trained deep unfolding framework.

4. The method of claim 3, wherein, In step 4, based on the trained deep unfolding framework, the inference operation is carried out on the compressed measurement data, and the continuous spectral image is obtained, specifically: based on the trained deep unfolding framework, the inference operation is carried out on the compressed measurement data, and the continuous spectral image is obtained, wherein the discrete spectral image data stored in the form of matrix is converted into the coefficient set of continuous function stored in each pixel.

5. The method of claim 4, wherein, In step 5, the obtained continuous spectral image is combined with any spectral band or the obtained continuous spectral image is combined with external model, and the reconstructed spectral image is output, and the related downstream task is carried out based on the reconstructed spectral image, specifically: According to the numerical value or set of any spectral band and the output of the continuous spectral image network, the spectral image is obtained by matrix multiplication, that is: （4） （5） Fuse the output of the continuous spectral image network and the features or output of other models, and output the spectral image, which is: （6） （7） wherein, is an activation function, is a weight value, is a constant term, is a feature vector encoding of other models, is a computation result of a multi-layer perception of the feature vector encoding of other models.

6. A deep learning-based computationally continuous spectral imaging system capable of reconstructing arbitrary wavebands, applied to the deep learning-based computationally continuous spectral imaging method of any one of claims 1-5, characterized in that, Comprise: Data acquisition and processing unit, used for obtaining hyperspectral image and its corresponding compressed measurement data, and constructing the training data set of hyperspectral image; The deep unfolding framework construction unit is connected with the data acquisition and processing unit and is used for constructing a deep unfolding framework, and the framework features include three sequentially executed modules: a denoising module, a parameter estimation module based on a continuous function, and a discretization module; The model training unit is connected with the data acquisition and processing unit and the deep unfolding framework construction unit respectively and is used for training the deep unfolding framework; The inference operation unit is connected with the model training unit and is used for obtaining a continuous expression of a spectral image based on the trained deep unfolding framework; The downstream task processing unit is connected with the inference operation unit and is used for performing fusion processing on the continuous spectral image according to a numerical value or a set of any spectral band or in combination with an output of an external model, outputting a reconstructed spectral image, and performing a classification, segmentation or target tracking downstream task.

7. The system of claim 6, wherein, The system further includes a data storage unit connected with the inference operation unit and used for storing the continuous spectral image.

Images