Ultrafast imaging method based on snapshot compression reconstruction and deep neural network

Abstract

The invention discloses an ultrafast imaging method based on snapshot compression imaging and a deep neural network. The method comprises the following specific steps: synthesizing a simulation data set; collecting two-dimensional compressed data of an ultrafast laser pulse for detecting an ultrafast scene; constructing an end-to-end compression reconstruction network based on low-rank decomposition and a channel attention mechanism; training the end-to-end compression reconstruction network; reconstructing the two-dimensional compressed data into three-dimensional hyperspectral video data by using the trained reconstruction network; a progressive deep neural network is constructed and trained; and converting the three-dimensional hyperspectral video data into four-dimensional hyperspectral video data of which each frame contains hyperspectral information by using the trained progressive deep neural network. According to the ultrafast imaging method provided by the invention, in the aspect of three-dimensional hyperspectral video reconstruction, compared with a current leading snapshot compression reconstruction technology, the peak signal-to-noise ratio (PSNR) is about 1dB higher; and an excellent effect can also be realized in the aspect of expanding and generating a four-dimensional hyperspectral video.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an ultrafast imaging method based on snapshot compression reconstruction technology and a deep neural network. Background Art

[0002] Ultrafast imaging technology plays a vital role in analyzing transient processes. The technology has the ability to capture phenomena that occur and change rapidly in a very short time, providing a unique perspective for studying and analyzing these fast-changing processes. However, traditional camera systems, especially those using complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) sensors, have inherent limitations in capturing rapidly changing light signals. These sensors cannot achieve extremely high levels of electronic response speed, which directly limits their ability to accurately capture rapidly changing light signals in a very short time.

[0003] In addition, there is a bottleneck in the design of the readout circuits of traditional cameras. These circuits cannot process and store image data at speeds of femtoseconds (1 femtosecond equals 10^-15 seconds) or picoseconds (1 picosecond equals 10^-12 seconds). Due to this speed limitation, traditional cameras have difficulty achieving ultra-high temporal resolution, that is, they cannot capture continuous image frames in extremely short time intervals. This is a huge obstacle for processes that occur on the picosecond or femtosecond time scale. For example, in chemical reactions, biological processes, or materials science, many key events occur in such a short period of time, and traditional cameras cannot provide sufficient resolution to record these processes in detail.

[0004] Snapshot compression imaging technology, as an emerging imaging method, can capture high-dimensional information, such as spectral, temporal or spatial data, in a single exposure. The basic principle of this technology is to compress the high-dimensional spectral data cube into two-dimensional measurement data through an imaging sensor, and then use a compression reconstruction algorithm to restore the required information. The advantage of snapshot compression imaging technology is that it does not require multiple exposures, so it is particularly suitable for capturing dynamic or transient phenomena. It has been widely used in research work in the fields of biological imaging, remote sensing detection, and ultrafast physical processes.

[0005] In the field of ultrafast imaging, the Compressed Ultrafast Spectroscopy Time (CUST) photography system, which is an experimental work that introduces the concept of snapshot compression imaging, compresses time or spectral information into a single two-dimensional image through spatial encoding, and reconstructs multiple time- or spectrally resolved images using the compressed sensing algorithm TwIST. The reconstructed image is not limited by the speed of the imaging sensor, thus achieving extremely high temporal resolution and spectral resolution. Nevertheless, this work only verifies the feasibility of the idea, and the imaging quality of ultrafast phenomena still needs to be further improved.

[0006] In recent years, artificial neural networks have developed into deep learning. Deep learning adopts the concept of hierarchical abstraction to obtain high-level concepts by learning low-level concepts. This hierarchical structure is usually built through a layer-by-layer greedy training algorithm to extract features useful for machine learning. Deep learning is a series of algorithms that aims to perform high-level abstraction on data using processing layers composed of complex structures or multiple nonlinear transformations. With its powerful expressive power, deep learning has achieved excellent results in various machine learning tasks and has also shown the potential to surpass other methods in compression and reconstruction. Summary of the invention

[0007] In order to optimize the imaging quality of the above-mentioned existing ultrafast imaging system, the purpose of the present invention is to propose an ultrafast imaging method based on snapshot compression reconstruction technology and deep neural network.

[0008] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] An ultrafast imaging method based on snapshot compression imaging and deep neural network comprises the following steps:

[0010] S1, synthetically simulated four-dimensional hyperspectral video dataset;

[0011] S2, collects 2D compressed data of ultrafast laser pulses used to detect ultrafast scenes;

[0012] S3, builds an end-to-end compression and reconstruction network based on a deep unfolding architecture;

[0013] S4, training the compression reconstruction network using the data set synthesized in step S1;

[0014] S5, using the trained compression reconstruction network to reconstruct the two-dimensional compressed data collected in step S2 into three-dimensional hyperspectral video data;

[0015] S6, constructs a progressive deep learning network based on the U-Net architecture to extract spectral features from hyperspectral images and generate corresponding new hyperspectral images;

[0016] S7, training the progressive deep learning network using the data set synthesized in step S1;

[0017] S8, using the three-dimensional hyperspectral video data obtained in step S5 and the trained progressive deep learning network, generating four-dimensional hyperspectral video data, each frame of which is a hyperspectral image.

[0018] In view of the fact that the compression reconstruction problem is essentially a pathological problem, and deep learning technology has shown significant superiority in the field of compression reconstruction, the present invention combines the deep unfolding framework in deep learning and the encoder-decoder network structure to propose a novel and effective ultrafast imaging method. The method uses a synthetic simulation data set to train the proposed compression reconstruction network and progressive deep neural network, and can restore and reconstruct three-dimensional hyperspectral video data from two-dimensional compressed measurement data. Compared with other existing advanced snapshot compression reconstruction technologies, the method of the present invention has better performance in both subjective and objective evaluations; at the same time, it also has excellent results in expanding and generating four-dimensional hyperspectral videos based on the reconstructed three-dimensional hyperspectral video data. The method of the present invention has far-reaching significance for promoting the development of the field of ultrafast imaging, and shows great potential and broad prospects beyond traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly explain the specific embodiments of the present invention or the technical solutions in the prior art, the drawings involved in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other possible drawings can be deduced based on these drawings without creative work.

[0020] Figure 1 It is the overall flow chart of the method of the present invention.

[0021] Figure 2 The present invention is a flowchart for generating three-dimensional hyperspectral video data in the method of the present invention.

[0022] Figure 3 Schematic diagram of a snapshot compression imaging system in an embodiment of the present invention.

[0023] Figure 4 It is a simplified schematic diagram of a four-dimensional hyperspectral data set synthesized in an embodiment of the present invention.

[0024] Figure 5 It is a network structure diagram of a reconstructed network based on deep expansion in an embodiment of the present invention.

[0025] Figure 6 4 is a structural diagram of a high-order tensor decomposition module in an embodiment of the present invention.

[0026] Figure 7 4 is a structural diagram of a residual tensor learning module in an embodiment of the present invention.

[0027] Figure 8 4 is a structural diagram of a low-rank feature fusion module in an embodiment of the present invention.

[0028] Fig. 94 is a structural diagram of a residual dense connection module in an embodiment of the present invention.

[0029] Fig.10 2 is a network structure diagram of a progressive deep neural network based on U-net in an embodiment of the present invention.

[0030] Fig.11 Schematic diagram of a method for generating four-dimensional hyperspectral video data based on three-dimensional hyperspectral video data in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present invention more clear, the implementation method of the present invention will be further described in detail below with reference to the accompanying drawings.

[0032] In this embodiment, a method for ultrafast imaging (such as imaging with a time resolution of less than 1 picosecond) based on snapshot compression reconstruction and deep neural network is provided, and the steps are as follows:

[0033] S1, synthesizes simulation datasets for training, validation, and testing;

[0034] S2, combining pulse shaping technology, spectral encoding methods, spatial modulation means, dispersion processing and time-space integration technology to achieve the collection of two-dimensional compressed data of ultrafast laser pulses for detecting ultrafast scenes;

[0035] S3, based on the deep unfolding framework, builds an end-to-end compression and reconstruction network combining low-rank decomposition and channel attention mechanism;

[0036] S4, training end-to-end compression and reconstruction network;

[0037] S5, reconstructing the two-dimensional compressed data collected in step S2 into three-dimensional hyperspectral video data using the trained reconstruction network;

[0038] S6, building a progressive deep neural network based on U-Net;

[0039] S7, training progressive deep neural networks;

[0040] S8, based on three-dimensional hyperspectral video data and progressive deep neural network, generates four-dimensional hyperspectral video data containing hyperspectral information in each frame.

[0041] In step S1, the selection of the data set has a great impact on the training of the entire neural network. This embodiment uses the DAVIS2017 data set, which is a public video data set provided by the official of a video segmentation image competition. Select a number of continuous RGB format video frames from the data set as input data, and input them into a neural network model (such as MST++) for generating hyperspectral images from RGB images. After being processed by the model, a four-dimensional hyperspectral video data set is obtained as a simulation data set. The four dimensions represent spectral channels, time, height and width, respectively, and are intended to simulate the data collection results of ultra-fast phenomena in the real world. Figure 4 A schematic diagram of a scene in which both the spectral channel and the time dimension in a dataset are 4 is given.

[0042] In step S2, the specific implementation flowchart is as follows Figure 2 As shown. Specifically, the ultrafast laser pulses used to detect ultrafast scenes need to be shaped and stretched first. The purpose of this process is to effectively encode time information in subsequent steps. When the pulse detects ultrafast phenomena, hyperspectral data I(x, y, λ) is obtained. By matching the different wavelengths of the ultrafast pulses with time, a three-dimensional hyperspectral video data I(x, y, t) containing space, time and hyperspectral information can be obtained.

[0043] Then, the coded aperture is used to spatially modulate the 3D hyperspectral video data, and additional spatial information is introduced through a specific coding mode to introduce coding for the subsequent 2D measurement process. The coding mode here is designed in such a way that each pixel value is randomly 0 or 1, ensuring that the same coding mode is used for training and verification. Where H represents the height dimension, W represents the width dimension, C represents the channel dimension, and X is the data cube of the three-dimensional hyperspectral input to the snapshot compression imaging system. The schematic diagram of this system is shown in Figure 3 As shown in the figure, the upper part is the imaging principle of the system, and the lower part is the data cube of the three-dimensional hyperspectral input snapshot compression imaging system. The schematic diagram of this system is shown in Figure 3 As shown in the figure, the upper part is the imaging principle of the system, and the lower part is the optical implementation of the system. represents a two-dimensional entity mask, where Represents the real number domain. The three-dimensional hyperspectral data cube must first be modulated by the mask. This embodiment uses a coded grating or a spatial light modulator to implement this step. For any channel m, this process can be expressed by the following expression:

[0044] X′(:,:,m)=M⊙X(:,:,m) (1)

[0045] Where X′ represents the modulated data block, X(:,:,m) represents the mth channel of the input data, and ⊙ represents element-by-element multiplication.

[0046] The hyperspectral data cube is modulated, dispersed using a dispersive prism, and then captured using a camera, such as Figure 3 As shown in the upper part, in this process, the spectral data cube with spatial shift between channels after modulation has been summed element by element between channels to obtain the two-dimensional measurement data Y containing time information:

[0047]

[0048] Here (u, v) represents the coordinates on the camera detection plane, d i Indicates the number of pixels offset by the i-th channel in the dispersion process (in this embodiment, d i The value of is 2i-2), It indicates the noise in the measurement process. It is the collected two-dimensional measurement data. Definition is the imaging matrix, where D m It is a diagonal matrix whose diagonal elements are all vectorized coded apertures M(:,:,m), where m is a positive integer between 1 and n. By converting the other three variables Y, X′ and N in equation (2) into y, x and n by matrix vectorization (i.e. concatenating each column of the matrix into a vector), equation (2) can be converted into the following form:

[0049] y=Φx+n (3)

[0050] Combined with equation (3), the reconstruction problem can be expressed as solving the reconstruction result x given the two-dimensional measurement value y and the imaging matrix Φ. In order to solve the ill-posed problem of equation (3), in the presence of noise interference, additional regularization is usually required to achieve accurate and stable solutions. The estimated value of the variable x can be derived by solving equation (4): This solves the compression and reconstruction problem:

[0051]

[0052] Among them, Ω(x) is a regularization term used to limit the range of solutions, and λ is the parameter of the regularization term. The present invention uses the generalized alternating projection algorithm to solve this problem. The principle of this algorithm is to decompose this optimization problem into two sub-problems by introducing an auxiliary variable v with the same dimension as the variable x for iterative solution. In the kth stage of the iterative process (where k = 1...K), K is the maximum number of iterations, and the first sub-problem is to calculate the auxiliary variable v in the kth stage. (k) On Linear Manifolds y = Euclidean projection on Φx to update x in the (k+1)th stage (k+1) :

[0053] x (k+1) =v (k) +Φ T (ΦΦ T ) -1 (y-Φv k ) (5)

[0054] The goal of the second sub-problem is to train an effective denoiser To make x (k+1) Closer to the target result, use this denoiser to update x (k+1) To get v (k+1) :

[0055]

[0056] Therefore, in step S3, the present invention proposes an end-to-end compression reconstruction network based on a deep expansion framework and combining low-rank decomposition and channel attention mechanism based on equations (5) and (6). The specific structure is as follows: Figure 5As shown, K modules (K=9 in this embodiment) that implement the generalized alternating projection algorithm are connected in series, corresponding to the K stages in the iterative solution method of step S5. Formula (5) shows that the two-dimensional compressed measurement data and the mask used for spatial modulation in the measurement process are both inputs of the compression reconstruction network, and the denoiser used in formula (6) is the LRUnet denoising network designed by the present invention. The denoising network is also an encoder-decoder structure, and three high-order tensor decomposition modules are used in the encoding stage to enhance the shallow feature extraction capability of the network. Before the data is input into the high-order tensor decomposition module, the mapping layer of the "convolution layer-activation function-convolution layer" structure (hereinafter referred to as "double-layer convolution module") is first used to perform feature mapping on the input degraded video data. The kernel size of the convolution layer in all double-layer convolution modules in the entire LRUnet denoising network is set to 3×3. In order to further expand the receptive field of the network, a downsampling operation is introduced after each high-order tensor decomposition module (except the last module). In the decoding stage, two upsampling modules and three double-layer convolution modules are used to gradually restore the data to its original resolution and effectively suppress noise interference in the process. The last high-order tensor decomposition module of the encoder is connected to the first upsampling module of the decoder. At the same time, a residual dense connection module is designed for the part that performs jump connection between the encoding and decoding stages. By fusing deep and shallow features, efficient fusion of multi-level features and full utilization of global information are achieved. The output of each upsampling is spliced ​​with the output of the corresponding residual dense connection module and then input into the double-layer convolution module to obtain the output result for the next upsampling. Finally, the output result of the penultimate double-layer convolution module is passed through another double-layer convolution module and then residually connected with the input to generate the reconstructed image. Next, the design method of the high-order tensor decomposition module and the residual dense connection module will be described in detail.

[0057] Figure 8 The architecture of the low-rank feature fusion module is demonstrated. This module extracts features by using the structure of "global average pooling + convolution layer + sigmoid activation function" in the channel, height and width dimensions to generate three one-dimensional vectors, and then re-aggregates these three vectors using the Kronecker product to obtain the output features. Figure 7 The architecture of the residual tensor learning module is shown. The module adopts a block residual structure to capture the features of different frequencies in the image. The input of the first residual tensor learning module is the input of the high-order tensor decomposition module, and the input of each subsequent residual tensor learning module is the output of the previous module. Specifically, a low-rank fusion module is first used to generate a rank-1 tensor Ox from the input features, and then a difference operation is performed with the input features. The difference result is added to the output of another low-rank fusion module (here the residual structure is reflected), so as to obtain the output of the residual tensor learning module. Figure 6As shown in the figure, the architecture of the high-order tensor decomposition module includes r residual tensor learning modules, whose function is to convert the input features into r rank-1 tensors. After the splicing operation, these tensors pass through a 3×3 convolution layer to extract the three-dimensional attention map. Subsequently, the attention map is Hadamard-producted with the input features, and the result is added to the original input features. Finally, the output features are obtained through a convolution layer with a 3×3 convolution kernel and an activation function.

[0058] The structure of the residual dense connection module is as follows Fig. 9 As shown in the figure, it is mainly composed of two parallel branches. The first branch contains three 3×3 convolutional layers, which are specially used for deep feature extraction. Input feature F in Is the input of the first convolutional layer; input feature F in After being concatenated with the output F1 of the first convolutional layer, it is used as the output of the second convolutional layer; input feature F in , the output F1 of the first convolutional layer and the output F2 of the second convolutional layer are concatenated and used as the input of the third convolutional layer. The outputs F1, F2 and F3 of the three convolutional layers are concatenated and input into a 1×1 convolutional layer to further fuse these deep features, thereby generating a richer feature representation. The second branch is mainly dedicated to the extraction of global contextual information. The input data of this module is first processed by the global pooling layer, and then passes through two fully connected layers in sequence, and the ReLU activation function is applied after each layer, and finally the processing result of this branch is output. The sum of the output results of these two branches is the final output of the entire module.

[0059] In step S4, the Adam adaptive gradient optimization algorithm is used to train the end-to-end compression reconstruction network. The loss function is defined as the mean square error between the reconstruction result output by the network and the true value. When training and verifying the network, the mask is a randomly generated matrix of size 256×256 containing only 0 and 1; the images in the four-dimensional hyperspectral video dataset synthesized in step S1 are randomly cropped into image blocks with a length and width of 256 and a channel number less than 31 (because the maximum number of channels for generating hyperspectral images from RGB images is 31), so as to Figure 4 Taking the simplified schematic diagram in as an example, a four-dimensional hyperspectral video dataset with 4 spectral channels and 4 time dimensions is shown in the figure. Four images with the same spectral channels and time axis coordinates (images on the diagonal passing through the origin) are selected and stitched in sequence into an image block with 4 channels. Adjusting the number of channels only requires modifying the number of diagonal stitched images during the operation. In this embodiment, the number of channels is selected as 8. The network is trained end-to-end using these image blocks. After the loss function Loss value reaches a convergence state, the training model is saved as an end-to-end compression reconstruction network.

[0060] In step S5, the data set synthesized in step S1 is processed according to the process in step S2 to obtain compressed two-dimensional measurement data, which is sent as input to the trained end-to-end compression reconstruction network together with the mask, and the three-dimensional hyperspectral video data I (x, y, t) is output. The reconstruction level of the reconstruction network proposed in step S3 performs well in terms of peak signal-to-noise ratio (PSNR), which is about 1 dB higher than other advanced snapshot compression reconstruction methods. The results show that the method proposed in the present invention can more effectively maintain the details of the original signal during the image reconstruction process, and the theoretical basis that the structural information in the noisy image is mainly contained in the low-rank components, while the noise components are mainly distributed in the high-rank parts is reasonable.

[0061] In step S6, the present invention designs a progressive deep neural network based on U-net, which inputs a pair of images and a task number and can output a pair of images. Its structure is as follows Fig.10 As shown. First, a preliminary feature mapping is performed through a 3×3 convolution layer to obtain an input feature with 32 channels. The core structure of each layer of the progressive deep neural network is the standard U-net architecture, which performs two downsampling and upsampling operations in both the encoding and decoding stages. U-nets at all levels in the network share the same set of parameters. Each level of U-net receives the concatenated data of the output and input features of the previous layer of U-net as input; for the first layer of U-net, the input is the concatenation of two identical input features to meet the network's specific requirements for the input data size. In addition, the role of the network's other input task number is to determine which layer of U-net output is selected as the final output. The output of the U-net at the selected level will be feature mapped through a 3×3 convolution layer, and its number of channels will be remapped to 2 as the final output of the network. The output feature mapping convolution layer parameters corresponding to each task number are different.

[0062] In step S7, the Adam adaptive gradient optimization algorithm is used to train the progressive deep neural network. The loss function is defined as the mean square error between the network output result and the true value. Image pairs are selected from the four-dimensional hyperspectral video dataset synthesized in step S1. Figure 4 For example, let the horizontal axis t represent time and the vertical axis λ represent the wavelength of the spectrum. i ,λ j ) indicates that at t i The corresponding spectral wavelength at the moment is λ jAn image. Any two frames can be selected as the images of the two vertices of a diagonal of a square and spliced ​​as the input of the network, that is, two images are selected, and the absolute value of the difference in their time axis coordinates is equal to the absolute value of the difference in their spectral wavelength axis coordinates. The two vertices on the other diagonal of the square are used as the true values ​​corresponding to the network output results. In each training process, the task number is defined as the absolute value of the difference in the coordinates of the two input images on the time axis. The method of selecting the training set in the form of an expression is: the input image is the coordinate (t i ,λ j ) and (t m ,λ n ), where i, j, m and n are all positive integers greater than 0 and less than the maximum value of the time axis coordinate, and satisfy |im|=|jn| and are not 0. The value of this equation is defined as the task number, and the true value corresponding to the output image is the coordinate (t i ,λ n ) and (t m ,λ j ). Only the parameters of the output convolution layer corresponding to the task number of the current input data will be updated. The task numbers of all training data in the same batch are equal until the loss function converges, and then the training model is saved.

[0063] In step S8, the three-dimensional hyperspectral video data I(x, y, t) reconstructed in step S5 needs to be expanded into a four-dimensional hyperspectral video I(x, y, t, λ) containing hyperspectral information in each frame. The expansion method is as follows: Fig.11 As shown in the diagram here, it is assumed that the size of the reconstructed three-dimensional hyperspectral video data is 4×256×256, and the size of the four-dimensional hyperspectral video to be obtained by expansion is 4×4×256×256. Fig.11 The horizontal axis t of this coordinate system represents time, and the vertical axis λ represents the wavelength of the spectrum. i ,λ j ) means t i At this moment, the corresponding spectral wavelength is λ j According to the rules followed in formulating the training set of the reconstruction network in step S4, in this coordinate system, the four images of the reconstructed 3D hyperspectral video data with a size of 4×256×256 should be located on the diagonal line with the same time axis coordinate and spectral channel coordinate (that is, the image coordinate (t i ,λ j ) satisfies i=j), such as Fig.11 Shown in the upper left corner. Fig.11The operation in the first row in indicates that the images at (t1, λ1) and (t3, λ3) are selected for splicing and then sent to the progressive deep neural network trained in step S7 as input. According to the definition of the task number in step S7, the task number at this time is |1-3|=2. At this time, the images of the two channels in the output result of the network represent the images at (t1, λ3) and (t3, λ1) respectively. After performing such operations on the other 5 pairs of different images on the diagonal with the same horizontal and vertical coordinate numbers, a four-dimensional hyperspectral video data with a size of 4×4×256×256 can be completed, and each frame of the video is a multi-channel hyperspectral image. It can be easily calculated that a three-dimensional hyperspectral video data set with m spectral channels needs to perform 0.5*m(m-1) such operations, which can be expanded to a four-dimensional hyperspectral video data with m spectral channels and m time channels. The average peak signal-to-noise ratio (PSNR) calculated between the final output four-dimensional hyperspectral video data and the true value can reach up to 39 dB. It can be seen that the effect of the expansion method of the present invention is relatively excellent.

Claims

1. An ultrafast imaging method based on snapshot compression imaging and deep neural network, characterized in that: The steps include: S1, synthetically simulated four-dimensional hyperspectral video dataset; S2, collects 2D compressed data of ultrafast laser pulses used to detect ultrafast scenes; S3, builds an end-to-end compression and reconstruction network based on a deep unfolding architecture; S4, training the compression reconstruction network using the data set synthesized in step S1; S5, using the trained compression reconstruction network to reconstruct the two-dimensional compressed data collected in step S2 into three-dimensional hyperspectral video data; S6, constructs a progressive deep learning network based on the U-Net architecture to extract spectral features from hyperspectral images and generate corresponding new hyperspectral images; S7, training the progressive deep learning network using the data set synthesized in step S1; S8, using the three-dimensional hyperspectral video data obtained in step S5 and the trained progressive deep learning network, generating four-dimensional hyperspectral video data, each frame of which is a hyperspectral image.

2. The ultrafast imaging method based on snapshot compression imaging and deep neural network according to claim 1, characterized in that: In step S1, the synthetic simulated four-dimensional hyperspectral video dataset specifically includes: selecting a number of continuous RGB format video frames from the video dataset as input data, inputting them into a neural network model for generating hyperspectral images from RGB images, and obtaining a four-dimensional hyperspectral video dataset after being processed by the model, wherein the four dimensions represent spectral channel, time, height and width, respectively.

3. The ultrafast imaging method based on snapshot compression imaging and deep neural network according to claim 1, characterized in that: In step S2, the collecting of two-dimensional compressed data of ultrafast laser pulses for detecting ultrafast scenes specifically includes: S21, shaping and stretching the ultrafast laser pulses used to detect ultrafast scenes to obtain three-dimensional hyperspectral video data; S22, spatially modulating three-dimensional hyperspectral video data using coded aperture; S23, dispersing the modulated data using a dispersive prism; S24, using an acquisition device to capture two-dimensional compressed data.

4. The ultrafast imaging method based on snapshot compression imaging and deep neural network according to claim 1, characterized in that: In step S3, the compression reconstruction network includes a plurality of modules for implementing the generalized alternating projection algorithm connected in series, each module including a denoising network, the denoising network including an encoder and a decoder, the encoder including a double-layer convolution module, three high-order tensor decomposition modules and two down-sampling modules, the high-order tensor decomposition module is used to convert the input features into a tensor with a rank of 1; the decoder includes two up-sampling modules and three double-layer convolution modules, and a residual dense connection module is provided between the encoder and the decoder for jump connection.

5. The ultrafast imaging method based on snapshot compression imaging and deep neural network according to claim 4, characterized in that: The high-order tensor decomposition module includes r residual tensor learning modules, the input of the first residual tensor learning module is the input of the high-order tensor decomposition module, and the input of each subsequent residual tensor learning module is the output of the previous module; the residual tensor learning module adopts a block residual structure to capture the features of different frequencies in the image.

6. The ultrafast imaging method based on snapshot compression imaging and deep neural network according to claim 5, characterized in that: The residual tensor learning module includes two low-rank fusion modules. First, the first low-rank fusion module is used to generate a tensor with a rank of 1 from the input features, and then a difference operation is performed with the input features. The difference result is added to the output of the second low-rank fusion module to obtain the output of the residual tensor learning module.

7. The ultrafast imaging method based on snapshot compression imaging and deep neural network according to claim 6, characterized in that: The low-rank fusion module adopts the structure of global average pooling + convolution layer + Sigmoid activation function.

8. The ultrafast imaging method based on snapshot compression imaging and deep neural network according to claim 4, characterized in that: The residual dense connection module includes two parallel branches: the first branch contains three 3×3 convolutional layers for extracting deep features, where the output F1 of the first convolutional layer is correlated with the input feature F in After concatenation, it is used as the output of the second convolutional layer; input feature F in The output F1 of the first convolutional layer and the output F2 of the second convolutional layer are concatenated and used as the input of the third convolutional layer. The outputs F1, F2 and F3 of the three convolutional layers are concatenated and input into a 1×1 convolutional layer to further fuse these deep features. The second branch is used to extract global context information. Its input data is first processed by the global pooling layer, and then passes through two fully connected layers in sequence. The ReLU activation function is applied after each layer, and then the processing result of the second branch is output. The output results of the two branches are added together to obtain the final output of the entire residual dense connection module.

9. The ultrafast imaging method based on snapshot compression imaging and deep neural network according to claim 1, characterized in that: In step S6, the input of the progressive deep learning network is an image pair and a task number, and the core structure of each layer is a U-net architecture, which performs two downsampling and upsampling operations in both the encoding and decoding stages; U-nets of all levels in the progressive deep learning network share the same parameter set, and each layer of U-net receives the concatenated data of the output and input features of the previous layer of U-net as input; for the first layer of U-net, the input is the concatenation of two identical input features; The task number is the absolute value of the coordinate difference between the two input images on the time axis. The progressive deep learning network selects the output of the corresponding layer U-net according to the input task number, and performs feature mapping through the convolution layer as the final output. The output feature mapping convolution layer parameters corresponding to each task number are different.

10. The ultrafast imaging method based on snapshot compression imaging and deep neural network according to claim 1, characterized in that: In step S7, two frames are arbitrarily selected from the data set synthesized in step S1 as images of two vertices of a diagonal line of a square and spliced ​​as the input of the progressive deep learning network, that is, two images are selected whose absolute value of the difference in time axis coordinates is equal to the absolute value of the difference in spectral wavelength axis coordinates, and the two vertices on the other diagonal line of the square are used as the true values ​​corresponding to the network output result.

Images