A video snapshot compressed imaging reconstruction method and system

Abstract

This application relates to a video snapshot compressed imaging reconstruction method and system. The method includes inputting a video frame sequence and its time-varying mask set into a measurement model to obtain an initial estimate; constructing a reconstruction network, including a feature extraction module, a gated residual network module, and a video reconstruction module; the feature extraction module includes two three-dimensional convolutional layers, each connected to an activation function, and extracts initial features from the initial estimate; inputting the initial features into the gated residual network module, outputting reconstructed information features; and fusing the reconstructed information features through the video reconstruction module, performing upsampling and detail refinement to reconstruct the video sequence. This invention effectively suppresses ghosting and flickering, improves the stability of long-term reconstruction, and provides an effective solution for high compression ratio, ultra-high resolution, and long-sequence SCI reconstruction, with almost no increase in parameters and computing power.

Description

Technical Field

[0001] This application relates to the field of video snapshot compressed imaging reconstruction technology, and in particular to a video snapshot compressed imaging reconstruction method and system. Background Technology

[0002] Video snapshot compressed imaging (Video SCI) is an imaging technique that acquires information from multiple frames of video through a single exposure, such as... Figure 1 As shown, the core idea is to encode multiple consecutive video frames using a time-varying mask during exposure and integrate them onto a two-dimensional sensor to form a compressed measurement. A reconstruction network based on a reconstruction algorithm then reconstructs the high-speed video sequence from the compressed measurement. Many mature SCI systems have emerged, but numerous problems and challenges remain regarding the reconstruction algorithm. Traditional model-optimized reconstruction algorithms, such as GAP-TV (Generalized Alternating Projection Total Variation) within the Generalized Alternating Projection (GAP) framework, offer advantages like strong interpretability and robustness to different noise levels. Another approach utilizes Gaussian Mixture Models (GMMs) to model spatiotemporal patches, improving texture and motion detail recovery. While these artificially pre-defined model optimizations offer some interpretability, they often require numerous iterations in high-resolution and long-sequence scenarios, resulting in long inference times and sensitivity to initialization and parameter tuning.

[0003] To balance physical consistency and data-driven capabilities, the Plug-and-Play (PnP) framework has been introduced into the field of video snapshot compression imaging. This typically involves inserting a pre-trained network into a loop to replace traditional regularization terms, while simultaneously using denoising algorithms to transform the image reconstruction problem into a denoising problem. For example, FFDNet is a high-performance denoising network in GAP iterations, and FastDVDNet is a high-performance denoising network in ADMM iterations, achieving a better quality-speed tradeoff than pure optimization methods under ultra-high definition and complex noise conditions. However, the PnP framework still relies on iterative inference; computational and memory overhead increases with sequence length and resolution, and convergence and parameter selection require experience.

[0004] Currently, with the development of deep learning and end-to-end models, many deep learning-based models have been proposed, driving the large-scale application of video snapshot compression imaging reconstruction algorithms. For example, RevSCI significantly reduces training memory usage by using reversible 3D convolutional networks with multiple sets of reversible structures, and for the first time, it integrates demosaicing and reconstruction end-to-end to directly recover color video from Bayer measurements. BIRNEAT uses bidirectional recurrent neural networks and adversarial training for two-stage frame-by-frame inference, improving temporal consistency. DUN-3DNet uses a deep unfolding method to perform iterative networking, achieving a trade-off between interpretability and efficiency, significantly improving reconstruction accuracy and throughput, but further optimization is still needed in terms of cross-device generalization and memory / latency. The Transformer series further enhances the modeling capabilities for long-term dependencies and large scenes. STFormer decouples and merges attention spatial / temporal branches, systematically utilizing intra-frame and inter-frame correlations; EfficientSCI uses spatiotemporal factorization and dense connection structures in residual blocks to balance high resolution and high compression ratio.

[0005] While the aforementioned deep learning-based methods have made significant progress in video snapshot compression imaging, the complexity of standard spatiotemporal self-attention mechanisms is quadratic with sequence length. Therefore, increasing the temporal span or spatial resolution significantly increases memory footprint and latency, ultimately limiting scalability. Secondly, at high compression ratios, the combined effects of spatial masking and temporal aliasing propagate noise, and traditional convolutional encoders often produce redundant inter-channel representations, thus suppressing edges and fine textures. Thirdly, in cases of complex motion or early mismatch, indiscriminately using densely connected temporal modules amplifies mixing errors over time, reduces temporal consistency, and leads to ghosting and flickering. Summary of the Invention

[0006] Based on this, the purpose of this invention is to address the above-mentioned technical problems by providing a video snapshot compressed imaging reconstruction method and system that can simultaneously improve reconstruction quality and temporal stability under high compression ratios and complex motion scenarios, significantly reduce the number of parameters, improve memory usage and inference latency, and meet the scalable real-time reconstruction and engineering deployment requirements of ultra-high definition and long sequences.

[0007] To achieve the above-mentioned objectives, the first aspect of this application provides a video snapshot compressed imaging reconstruction method, comprising:

[0008] Obtain the video frame sequence and its corresponding time-varying mask set, and input them into the video snapshot compressed imaging forward measurement model to obtain the initial estimate;

[0009] Construct a reconstruction network, including a feature extraction module, a gated residual network module, and a video reconstruction module;

[0010] The initial estimate is input into the feature extraction module of the reconstruction network. The feature extraction module includes two three-dimensional convolutional layers, each followed by an activation function. The feature extraction module extracts initial features from the initial estimate.

[0011] The initial features are input into the gated residual network module, which outputs reconstructed information features.

[0012] The video reconstruction module fuses the reconstructed information features, performs upsampling through subpixel convolution, and refines details using 3D convolution to reconstruct the video sequence.

[0013] Preferably, the gated residual network module includes five ResLNet blocks, wherein a three-dimensional convolutional layer and an activation function are provided between the second and third ResLNet blocks, and two three-dimensional convolutional layers are provided between the fourth and fifth ResLNet blocks. The output of the first three-dimensional convolutional layer in the two three-dimensional convolutional layers is concatenated with the output of the second ResLNet block by channel before the second three-dimensional convolution is performed.

[0014] Each ResLNet block transmits information through short-hop connections or long-hop connections, and models spatiotemporal correlations using grouping, frame-level cross-gating, and temporal linear attention mechanisms. The steps for each ResLNet block to model spatiotemporal correlations using grouping, frame-level cross-gating, and temporal linear attention mechanisms include:

[0015] The input features of the ResLNet block are uniformly divided along the channel dimension. S Group;

[0016] The first group is directly subjected to spatiotemporal factorization to obtain the output. Y 1;

[0017] For the i-th group, where i {2,..., S}, perform frame-level cross-gating operation, and set the output of the previous group to... Y i-1 Input with the current group X i Perform adaptive fusion to obtain fused features Z i ;

[0018] The fusion features Z i Spatiotemporal factorization is performed to obtain the output. Y i ;

[0019] Output of all groups { Y 1,..., Y SThe features are concatenated along the channel dimension, compressed by 1×1×1 convolution, and then residually connected to the input features of the ResLNet block to obtain the output of the ResLNet block.

[0020] Preferably, the frame-level cross-gating operation specifically includes:

[0021] For the output of the previous group respectively Y i-1 With the current input X i Perform global average pooling to obtain the corresponding pooled feature vector. and :

[0022]

[0023]

[0024] in, This indicates that global average pooling is performed on each frame in both the channel and spatial dimensions. H Indicates altitude, W Indicates width, H × W That is, spatial dimension. C Indicates the channel dimension;

[0025] The pooled feature vectors are concatenated along the channel dimension, and a one-dimensional temporal convolution is applied to the concatenated features for smoothing.

[0026]

[0027] in, Indicates the features after splicing; For the time dimension, i.e., the number of frames T One-dimensional convolution is used for smoothing gated sequences. k Indicates the kernel size; This indicates splicing by channel dimension;

[0028] After passing through the Sigmoid activation function, the frame-by-frame gated weight vector is obtained. :

[0029]

[0030] in, The Sigmoid activation function maps real numbers to (0,1);

[0031] Frame-level cross-gating is constructed using the aforementioned gating weight vector. :

[0032]

[0033] in, Represents a gated function; ⊙ indicates element-wise multiplication and automatic broadcasting by dimension. , = C / S Indicates the number of channels in each group;

[0034] Frame-level cross-gating is used to adaptively harmonize historical and current information, and output fused features. Z i .

[0035] Preferably, the spatiotemporal factorization process is implemented through a spatiotemporal factorization block ELFormer, which includes three branches that are set in parallel and fused sequentially:

[0036] Spatial channel fusion branch is used for local spatial feature extraction and channel recalibration;

[0037] A time-linear attention branch is used to model global temporal dependencies with near-linear complexity.

[0038] The Swish-gated time feedforward network branch is used to perform steady-state gated fusion of the outputs of the first two branches.

[0039] Preferably, the spatial channel fusion branch performs the following operations:

[0040] Input features of the spatiotemporally factorized block ELFormer Channel reshaping is performed to obtain the reshaped input features. ;

[0041] For input features Two layers of 2D convolutions are applied to each frame to reconstruct spatial details. The two layers of 2D convolutions are connected by an activation function to obtain features. ;

[0042] Subsequently, a lightweight channel attention mechanism was used to analyze the features. Perform channel recalibration, specifically as follows:

[0043] Global average pooling is performed on the input features after reconstructing spatial details, and for each frame... Each channel The intensity statistics for each channel of each frame are obtained by averaging the pixel values. :

[0044]

[0045] Intensity statistics Apply one-dimensional convolution, convolution kernel length k Determined based on the number of channels:

[0046]

[0047] in, and is a constant and ;

[0048] Attention is generated through one-dimensional convolution along the channel. :

[0049]

[0050] in, This represents one-dimensional convolution;

[0051] The channel weights are obtained by passing the output of the one-dimensional convolution through the Sigmoid activation function. :

[0052]

[0053] Extend the channel weights to the spatial dimension and combine them with the features. Element-wise multiplication is used to recalibrate the channels, resulting in... ;

[0054] Finally, Perform channel reshaping to obtain the reshaped output features. .

[0055] Preferably, the time-linear attention branch performs the following operations:

[0056] Attention is applied along the time axis to block the spatiotemporal factors of the ELFormer's input features. Channel reshaping yields:

[0057]

[0058] Will Grouped by spatial location, the temporal feature sequence of each spatial location is obtained, the first... The time series of each spatial location is denoted as ;

[0059] Through trainable linear projection matrix Perform linear projection on the temporal feature sequence of each spatial location, and plot the data of each frame. Feature mapping to Dimensional queries, key-value spaces yield the following:

[0060]

[0061] in, For query sequence, For key sequences, For value sequences, Value dimension;

[0062] right and Applying nonnegative eigenmap function ,get and ,in ;

[0063] Time Index definition:

[0064]

[0065] Where M is a hyperparameter of time-scale intensity, used to set the phase step size for weighted cosine or sine.

[0066] Frame-by-frame scaling is applied to the mapped query and key sequences using cosine and sine-based proximity biases:

[0067]

[0068] in, These represent the mapped query matrix. The Row by coefficient or Perform element-wise scaling and at time Inject the corresponding cosine or sine weights; These represent the mapped key matrices respectively. The Row by coefficient or Perform element-wise scaling and at time Inject the corresponding cosine or sine weights; Indicates element-wise multiplication;

[0069] The output at each time t is calculated using a linear attention mechanism. :

[0070]

[0071] in, The length is A vector of all 1s; It is the numerical stability constant; Indicates transpose;

[0072] Finally, for all pixels The output is obtained after parallel computation. And then By reshaping the channel, the output of the time-linear attention branch can be obtained. .

[0073] Preferably, the Swish-gated time-feedforward network branch performs the following operations:

[0074] The input features of the Swish-gated time feedforward network branch are analyzed. After pointwise convolution and activation, it is divided into value branches and gated branches along the channel dimension;

[0075] Perform 3D convolution and activation on the value branch to obtain the output Y1;

[0076] For the gated branch, it is multiplied element-wise with Y1, which has undergone one-dimensional temporal convolution and gated processing with the Swish activation function SiLU, and then three-dimensional convolution and activation are performed to obtain the output Y2.

[0077] Y1 and Y2 are spliced ​​together by channel, and then activated and... After pointwise convolution, a residual connection is made with the input features of the Swish-gated time feedforward network branch to obtain the final output of the Swish-gated time feedforward network branch.

[0078] To achieve the purpose of the invention, a second aspect of this application provides a video snapshot compressed imaging reconstruction system, which applies the video snapshot compressed imaging reconstruction method described above. The system includes:

[0079] The preprocessing module is used to obtain the video frame sequence and its corresponding time-varying mask set, and input it into the video snapshot compressed imaging forward measurement model to obtain the initial estimate;

[0080] The reconstruction network module includes a feature extraction module, a gated residual network module, and a video reconstruction module;

[0081] The feature extraction module is used to extract initial features based on the initial estimate, including two three-dimensional convolutional layers, each of which is followed by an activation function.

[0082] The gated residual network module is used to extract reconstructed information features and includes five ResLNet blocks. A three-dimensional convolutional layer and activation function are set between the second and third ResLNet blocks, and two three-dimensional convolutional layers are set between the fourth and fifth ResLNet blocks. The output of the first three-dimensional convolutional layer in the two three-dimensional convolutional layers is concatenated with the output of the second ResLNet block by channel before the second three-dimensional convolution is performed. Each ResLNet block transmits information through short-skip connections or long-skip connections, and uses group processing, frame-level cross-gating, and temporal linear attention mechanisms to model spatiotemporal correlation.

[0083] The video reconstruction module is used to fuse reconstruction information features, perform upsampling through subpixel convolution, and apply three-dimensional convolution to refine details and reconstruct the video sequence;

[0084] The output module is used to output the reconstructed video sequence.

[0085] Preferably, the ResLNet block includes:

[0086] Grouping units are used to group input features according to the channel dimension.

[0087] Gating unit, used to calculate frame-level cross-gating;

[0088] Spatiotemporal factorization unit, used to perform spatiotemporal factorization processing on each set of features;

[0089] The fusion unit is used to concatenate the outputs of each group and perform a 1×1×1 convolution, and finally perform a residual connection with the input features of the ResLNet block to obtain the output of the ResLNet block.

[0090] Preferably, the spatiotemporal factorization unit includes an ELFormer block, and each ELFormer block has three branches, including:

[0091] Spatial channel fusion branch is used for local spatial feature extraction and channel recalibration;

[0092] A time-linear attention branch is used to model global temporal dependencies with near-linear complexity.

[0093] The Swish-gated time feedforward network branch is used to perform steady-state gated fusion of the outputs of the first two branches.

[0094] Compared with the prior art, the beneficial effects of this invention are:

[0095] This invention introduces frame-by-frame gating and lightweight temporal smoothing at the fusion of historical and current information between groups through frame-level cross-gating of ResLNet blocks. This effectively suppresses ghosting and flickering and improves the stability of long-term reconstruction without increasing parameters or computing power. Compared with indiscriminate dense temporal fusion or single-path stacking, this invention reduces the risk of error amplification and temporal jitter at the structural level. By using a temporal linear attention mechanism, kernelized linear attention in the temporal dimension reduces the quadratic time or memory overhead of standard self-attention with sequence length to a near-linear level, avoiding the explicit construction of a T×T scoring matrix. This reduces both computational load and storage consumption in long-sequence SCI reconstruction. Attached Figure Description

[0096] Figure 1 This is a schematic diagram of the existing video snapshot compression imaging (SCI) process;

[0097] Figure 2 This is a schematic diagram illustrating the steps of a video snapshot compression imaging reconstruction method in an embodiment;

[0098] Figure 3 This is a schematic diagram illustrating the process of reconstructing a video sequence using a reconstruction network in an embodiment.

[0099] Figure 4 This is a schematic diagram of the gated residual network block in the embodiment;

[0100] Figure 5 This is a schematic diagram of frame-level cross-gating in the embodiment;

[0101] Figure 6 This is a schematic diagram of the spacetime factor block ELFormer in the embodiment;

[0102] Figure 7 This is a schematic diagram of the ECA3D block of the spatial channel fusion branch in the embodiment;

[0103] Figure 8 This is a schematic diagram of the Swish-gated time feedforward network (SFFN) branch in the embodiment.

[0104] Figure 9 The images show the visualization results of several reconstruction methods, including the LiteSCI method, in the embodiments.

[0105] Figure 10 The images show the visual reconstruction results of several reconstruction methods, including the LiteSCI method, in a simulated scene, as illustrated in the embodiments. Detailed Implementation

[0106] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention. The following embodiments are used to illustrate the invention but are not intended to limit its scope.

[0107] Example 1

[0108] Embodiment 1 of this application provides a video snapshot compressed imaging reconstruction method, such as... Figure 2 As shown, it includes the following steps:

[0109] S1: Obtain the video frame sequence and its corresponding time-varying mask set, and input them into the video snapshot compressed imaging forward measurement model to obtain the initial estimate;

[0110] S2: Construct the reconstruction network, including a feature extraction module and a gated residual network module ( Figure 2The ResLNetModule and the video reconstruction module;

[0111] S3: Input the initial estimate into the feature extraction module of the reconstruction network. The feature extraction module includes two three-dimensional convolutional layers, each of which is followed by an activation function. The feature extraction module extracts initial features from the initial estimate.

[0112] S4: Input the initial features into the gated residual network module and output the reconstructed information features;

[0113] S5: The video reconstruction module fuses the reconstructed information features, performs upsampling through subpixel convolution, and refines details using three-dimensional convolution to reconstruct the video sequence.

[0114] The specific steps for S1-S5 are as follows:

[0115] The video frame sequence and its corresponding time-varying mask set are taken as input. According to the SCI forward imaging measurement model, the input is assumed to be continuous video frames: Time-varying mask sequences (sets) and noise: The output is:

[0116]

[0117] in This indicates element-wise multiplication; the output of the SCI forward imaging measurement model is a compressed snapshot measurement. That is, an initial estimate consistent with the perception process. Then... Each video frame was recovered by rebuilding the network.

[0118] A flowchart for recovering or reconstructing a video sequence by rebuilding the network is shown below. Figure 3 As shown, the initial estimate obtained through the measurement model is... The input is fed into the feature extraction module. In this embodiment, the feature extraction module consists of two 3D convolutional layers with kernel sizes of 3×3×3 and 1×3×3, and each 3D convolutional layer is followed by a LeakyReLU activation function.

[0119] Furthermore, the reconstruction network includes a feature extraction module, a gated residual network module, and a video reconstruction module. The gated residual network module comprises five ResLNet blocks. Figure 2 The ResLNet blocks 1-5 are used in the ResLNet architecture. A 3D convolutional layer and activation function are placed between the second and third ResLNet blocks; two 3D convolutional layers are placed between the fourth and fifth ResLNet blocks. The output of the first 3D convolutional layer is connected to the second ResLNet block (…). Figure 2The output of ResLNetBlock2 is concatenated by channel before a second 3D convolution is performed; each ResLNet block transmits information through short-hop or long-hop connections, and spatiotemporal correlation is modeled using group processing, frame-level cross-gating and temporal linear attention mechanisms.

[0120] The structure of the gated residual network block, i.e., the ResLNet block, is as follows: Figure 4 As shown in the ResLNet Block. (Using...) This indicates the input to the ResLNet block. Under the paradigm of "grouping-processing-fusion," it is first processed along the channel dimension. Divided into equal parts In this embodiment, the groups are divided into 4 groups. Figure 4 middle S =4:

[0121]

[0122] Each group ; Indicates the number of channels in each group; It is the set of real numbers; T Indicates time; H Indicates altitude; W Indicates width; C Indicates a channel; Indicates grouping;

[0123] Group 1 is directly fed into the spatiotemporal factorization block. (ELFormer is the sub-network defined in this application: including intra-frame 2D convolution / channel recalibration, temporal linear attention, and gated feedforward; the output of ELFormer is in the same form as the input.) Spatiotemporal factorization is performed to obtain the output. ;

[0124] For the Group, construct frame-level cross-gating:

[0125]

[0126] Frame-level cross-gating, i.e., the Gate module, is as follows Figure 5 As shown, the gating weights , ⊙ represents element-wise multiplication, where:

[0127]

[0128]

[0129] in, Indicates the features after splicing; This indicates that global average pooling is performed on each frame in both the channel and spatial dimensions, and its output is... ; For the time dimension, i.e., the number of frames. T The length of the core is k One-dimensional convolution with a value of 3 is used to smooth gated sequences; This indicates splicing by channel dimension; The sigmoid function maps real numbers to (0,1) as weights; ⊙ represents the Hadamard (element-wise) multiplication and automatic broadcasting by dimension. ; = C / S This indicates the number of channels in each group.

[0130] Frame-level cross-gating is used to adaptively harmonize historical and current information, and output fused features. Z i .

[0131] Will Spatiotemporal factorization processing is performed: (Input and output are in the same form), Assembled by channel After Pointwise convolution performs channel blending and mixes with the input. The output is obtained by adding the residuals. .

[0132] Global average pooling is used to pool the values ​​of each frame. Feature compression is performed as a single-value statistic (an average value per channel / per frame); Convolution only changes the channel combination and has little effect on the spatial / temporal resolution; 3D and 1D convolutions represent local weighting on the spatiotemporal volume and the pure time axis, respectively.

[0133] By adaptively harmonizing historical and current information, this frame-level cross-gating design significantly suppresses ghosting and flickering with minimal overhead and improves long-term stability.

[0134] like Figure 6 As shown, the spatiotemporal factorization block ELFormer includes three branches that are set up in parallel and fused sequentially:

[0135] Spatial channel fusion branch is used for local spatial feature extraction and channel recalibration;

[0136] A time-linear attention branch is used to model global temporal dependencies with near-linear complexity.

[0137] The Swish-gated time feedforward network branch is used to perform steady-state gated fusion of the outputs of the first two branches.

[0138] Spatial Channel Fusion Branch SCFB (input is The output is (Input and output are of the same form) such as Figure 6 As shown in SCFB, its overall process includes:

[0139] Input features of the spatiotemporally factorized block ELFormer Channel reshaping is performed to obtain the reshaped SCFB input features. .

[0140] Reshaped input features Two layers per frame The 2D convolutions are connected by a LeakyReLU activation function. The two 2D convolutions only perform convolutions in the spatial dimension to enhance edges and textures and reconstruct spatial details. The processed data then enters the ECA3D module, whose structure is as follows: Figure 7 As shown, the ECA3D module first performs global average pooling to obtain the intensity statistics for each channel per frame. (Meaning: For each frame) Each channel (Average of pixels across dimensions). Intensity statistics. Attention is generated using one-dimensional convolution along the channel: The kernel length of a one-dimensional convolution Through adaptive kernel function confirm:

[0141]

[0142] in, .

[0143] The channel weights are obtained by passing the output of the one-dimensional convolution through the Sigmoid activation function. :

[0144]

[0145] Channel weights Extending to the spatial dimension and with features Element-wise multiplication is used to recalibrate the channels, resulting in... .

[0146] in, For the Sigmoid function; Represents matrix multiplication; This represents element-wise multiplication. Channel recalibration uses adaptive 1D convolutional kernels to model inter-channel correlations, suppressing redundancy and preserving detail with almost no added overhead.

[0147] Finally, Perform channel reshaping to obtain the reshaped output features. As the output of the spatial channel fusion branch.

[0148] Time-linear attention branch (TLAB) such as Figure 6 As shown, TLAB only performs attention along the time axis. The overall process of TLAB includes:

[0149] Spatiotemporally factorized block ELFormer input features Channel reshaping was performed to obtain :

[0150]

[0151] Will Grouped by spatial location, the temporal feature sequence of each spatial location is obtained; where, the... The time series of each spatial location is denoted as Representing pixels exist Channel feature trajectories on the frame. This is achieved through a trainable linear projection matrix. Perform linear projection on the temporal feature sequence of each spatial location, and plot the data of each frame. Feature mapping to Dimensional queries, key-value spaces yield the following:

[0152]

[0153] in, This is a query sequence used to match each frame. Perform similarity calculations to generate attention weights; A key sequence, used with The pairing determines the degree of attention given to each frame; Let be a sequence of values, where For value dimension.

[0154] right and Applying nonnegative eigenmap function ,get and , where d s =d; The non-negative feature mapping function uses a kernel method to change the computational complexity to linear while ensuring that the attention weights are non-negative.

[0155] To introduce nearest-neighbor bias to ensure stability, time indexing definition:

[0156]

[0157] Where M is a hyperparameter of time-scale intensity, used to set the phase step size for weighted cosine or sine.

[0158] Frame-by-frame scaling is applied to the mapped query and key sequences using cosine and sine-based proximity biases:

[0159]

[0160] in, These represent the mapped query matrix. The Row by coefficient or Perform element-wise scaling and at time Inject the corresponding cosine or sine weights; These represent the mapped key matrices respectively. The Row by coefficient or Perform element-wise scaling and at time Inject the corresponding cosine or sine weights; Indicates element-wise multiplication;

[0161] The output at each time t is calculated using a linear attention mechanism. :

[0162]

[0163] in, The length is A vector of all 1s; It is the numerical stability constant; This indicates transpose.

[0164] Finally, for all pixels The output is obtained after parallel computation. And then By reshaping the channel, the output of the time-linear attention branch can be obtained. The time-linear attention branch uses standardized attention (...). Combined with cosine weighting Modeling global dependencies in the time dimension preserves both locality and numerical stability, while reducing complexity from... dropped to near It can adapt to long sequences and high-resolution SCI.

[0165] The outputs of SCFB and TLAB are matrix-added and then combined with the input features of the spatiotemporally factored block ELFormer. After matrix addition, the input of the Swish-gated time feedforward network (SFFN) branch is obtained. .

[0166] SFFN input Represents a T-frame with a spatial size of And it has the characteristic of having C channels. Swish-gated time-feedforward network (SFFN) branch, such as... Figure 8 As shown. The overall process of SFFN is as follows:

[0167] First, the input of SFFN conduct Pointwise convolution and activation yield:

[0168]

[0169] in, for Weights of pointwise convolution; Choose the LeakyReLU activation function.

[0170] After that Divided into two branches along the channel dimension, the value branch and the value branch. and gated branches .

[0171] value branch Performing dimensionality-upgrading and nonlinear calculations (i.e., performing 3D convolution and activation) yields... :

[0172]

[0173] in, for The weights of the 3D convolution.

[0174] Gated branches first Make along the time axis One-dimensional temporal convolution is performed and activated using the SiLU activation function, then combined with... Multiplication and subsequent passage 3D convolution and through After the function is activated, the result is obtained :

[0175]

[0176] in, for The weights of the 3D convolution; For along the time axis The weights of the one-dimensional temporal convolution; Element-wise multiplication; Choose the SiLU activation function: ,in This is the Sigmoid function used to smooth out nonlinearities.

[0177] Will and Obtained by splicing along the channels After Pointwise convolution followed by activation using the LeakyReLU function and then applied to the input Perform residual summation (residual summation in) Figure 8 (Not shown in the image), output is obtained. :

[0178]

[0179] Among them, the output Shape and Input same.

[0180] The Swish-gated temporal feedforward network branch replaces the basic feedforward network with a lightweight structure of channel grouping and feature reuse. It adds 1D temporal convolution to the gated signal for smoothing and uses SiLU activation to form multiplicative gating, thereby performing "amplification-suppression" adaptive recalibration of edges, fine lines and motion trajectories. It can stably fuse the outputs of the SCFB and TLAB branches and reduce temporal jitter.

[0181] Each group in each ResLNet block After spatiotemporal factorization processing through frame-level cross-gating and / or the spatiotemporal factorization block ELFormer, the result is... S Group (in this embodiment 1 and) Figure 4 middle S The output of (=4) is: Y1, Y2, Y3 and Y4.

[0182] Y1, Y2, Y3, and Y4 are then subjected to a 1×1×1 3D convolution and then combined with the input of the ResLNet block. The output of the ResLNet block is obtained by summing the residuals. .

[0183] Output of the last ResLNet block With gated residual network module ( Figure 2 The input of the ResLNet Module is concatenated by channel to obtain the reconstructed information features.

[0184] Finally, the reconstruction information features are fused through the reconstruction head in the video reconstruction module, upsampling is performed through subpixel convolution, and 1×1×1 and 3×3×3 three-dimensional convolutions are applied to refine the details, ultimately reconstructing and generating a video sequence.

[0185] Example 2

[0186] This application, in embodiment 2, based on embodiment 1, provides a video snapshot compression imaging reconstruction system, the system comprising:

[0187] The preprocessing module is used to acquire video frame sequences and time-varying mask sets, and input them into the video snapshot compressed imaging forward measurement model to obtain initial estimates;

[0188] The reconstruction network module includes a feature extraction module, a gated residual network module, and a video reconstruction module;

[0189] The feature extraction module is used to extract initial features based on the initial estimate, including two three-dimensional convolutional layers, each of which is followed by an activation function.

[0190] The gated residual network module is used to extract reconstruction information features, including five ResLNet blocks ( Figure 2 The ResLNet blocks 1-5 are used in the ResLNet architecture. A 3D convolutional layer and activation function are placed between the second and third ResLNet blocks; two 3D convolutional layers are placed between the fourth and fifth ResLNet blocks. The output of the first 3D convolutional layer is connected to the second ResLNet block (…). Figure 2 The output of ResLNet Block 2 is concatenated by channel before a second 3D convolution is performed; each ResLNet block transmits information through short-skip or long-skip connections, and uses group processing, frame-level cross-gating and temporal linear attention mechanisms to model spatiotemporal correlation.

[0191] The video reconstruction module is used to fuse reconstruction information features, perform upsampling through subpixel convolution, and apply three-dimensional convolution to refine details and reconstruct the video sequence;

[0192] The output module is used to output the reconstructed video sequence.

[0193] In this embodiment 2, the ResLNet block includes:

[0194] Grouping units are used to group input features according to the channel dimension.

[0195] Gating unit, used to calculate frame-level cross-gating;

[0196] Spatiotemporal factorization unit, used to perform spatiotemporal factorization processing on each set of features;

[0197] The fusion unit is used to concatenate the outputs of each group and perform a 1×1×1 convolution, and finally perform a residual connection with the input features of the ResLNet block to obtain the output of the ResLNet block.

[0198] In this embodiment 2, the spatiotemporal factorization unit includes an ELFormer block, and each ELFormer block has three branches, including:

[0199] Spatial channel fusion branch is used for local spatial feature extraction and channel recalibration;

[0200] A time-linear attention branch is used to model global temporal dependencies with near-linear complexity.

[0201] The Swish-gated time feedforward network branch is used to perform steady-state gated fusion of the outputs of the first two branches.

[0202] Example 3

[0203] This embodiment 3, based on embodiments 1 and 2, compares the LiteSCI method of this application with state-of-the-art video snapshot compression imaging (SCI) reconstruction methods on simulated and real datasets. For the simulated benchmark tests, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are reported as the main evaluation metrics.

[0204] LiteSCI was trained using the DAVIS2017 dataset, which provides 90 different scenes with resolutions of 480×894 and 1080×1920. Three classes of simulation benchmarks were evaluated in a uniform setting: the first class consisted of grayscale simulation videos (Basketball, Runner, Drop, Traffic, Aerial, Vehicle), each in 256×256×8 format, ordered as height, width, and frame; the second class consisted of color simulation videos (Beauty...). Figure 10 (Not listed in the text), Bosphorus, Jockey, Runner, ShakeNDry, Traffic), each video is in the format of (512×512×3×8), in the order of height, width, channels, frames; the third category is ablation experiments.

[0205] All models were implemented in PyTorch and trained on four NVIDIA A100 Tensor Core GPUs. Standard data augmentation methods, including random cropping, scaling, and horizontal flipping, were applied to DAVIS2017. The Adam optimizer was used during training with an initial learning rate of 0.0001. After 300 epochs, the learning rate was decayed to 0.00001, and training continued for another 40 epochs until convergence. After training, the PSNR and SSIM values ​​from simulated benchmark tests were reported. These values ​​were calculated frame-by-frame and averaged over frames and sequences. The comparison results are shown in Table 1.

[0206] Table 1 Comparison of Reconstruction Results of Simulated Grayscale Datasets

[0207]

[0208] On simulated grayscale datasets, the proposed method was benchmarked against representative model-based methods (GAP-TV, PnP-FFDNet, PnP-FastDVDnet, and DeSCI) and deep learning-based methods (BIRNAT, RevSCI, DUN-3DUnet, ELP-Unfolding, STFormer, EfficientSCI, and EfficientSCI++). Table 1 lists the average PSNR and SSIM values ​​of different reconstruction methods on six benchmark grayscale datasets, as well as the average reconstruction time per measurement. Figure 9 Visualizations of several state-of-the-art reconstruction methods are presented. It can be observed that the LiteSCI method proposed in this application significantly improves both reconstruction quality and real-time performance compared to previous state-of-the-art methods. Visualizations of the reconstructed video show that the LiteSCI method can recover more details and edge information. Specifically, the LiteSCI method achieves an average PSNR of 36.48 dB and an SSIM of 0.975. Compared to previous end-to-end deep learning methods STFormer-B and EfficientSCI-B (the best published results), the method in this application also performs well in terms of reconstruction runtime, achieving a good balance between reconstruction quality and performance. The LiteSCI model proposed in this application achieves a reconstruction quality higher than 36.5 dB and a runtime within 200 milliseconds. Although BIRNAT is faster than LiteSCI, its reconstruction quality is poor, with an average PSNR below 34 dB, more than 2 dB lower than LiteSCI.

[0209] LiteSCI performs best on the Traffic and Crash datasets, both of which contain long-distance motion and frequent occlusion. On the Traffic dataset, LiteSCI achieves a PSNR of 32.32 dB and an SSIM of 0.968; on the Crash dataset, it achieves a PSNR of 31.15 dB and an SSIM of 0.971, slightly outperforming the EfficientSCI-B and EfficientSCI-B++ algorithms. The temporal linear attention branch enhances long-range consistency and reduces flickering and ghosting in fast scenes (e.g., vehicle markers in the Crash dataset); while the channel attention method based on channel recalibration of the spatiotemporally factorized block ELFormer in this application enhances fine textures and edges (e.g., license plates and leaves), resulting in clearer details. On the Basketball dataset, its jersey numbers are clear, matching the best benchmark models; on the Aerial dataset, it retains comparable quality while better preserving fine branches. Qualitative frames validated these observations, showing clearer insertion points and fewer artifacts in images with strong motion and rich high-frequency content.

[0210] Table 2 Comparison of reconstruction results of simulated color datasets

[0211]

[0212] To evaluate generalization beyond grayscale reconstruction, the network was extended to color SCI and evaluated on six RGB benchmarks, where the third axis represents the RGB channels. A compression ratio of B=8 was used, similar to the grayscale setting. Compressed Bayer measurements were acquired via a camera equipped with Bayer mosaic; four Bayer measurements were obtained for each 32-frame color segment. Since the model is resolution-independent, a small-scale training strategy was employed to accelerate optimization and reduce memory consumption. Considering the input size and masking limitations of DUN-3DUnet and RevSCI (which makes end-to-end training at 512×512 particularly resource-intensive), color data comparisons were limited to representative iterative methods (GAP-TV, DeSCI), end-to-end baseline methods (BIRNAT-color), plug-and-play methods (PnP-FFDNet, PnP-FastDVDnet), and deep learning-based methods (STFormer). For completeness, the PnP baseline model was instantiated in both grayscale and color formats, depending on whether a grayscale or color denoiser was used. Reconstruction results of different algorithms, such as Figure 10As shown in Table 2, LiteSCI achieved the best average accuracy and lowest single-measurement run time across six medium-scale color benchmarks, with a PSNR of 37.65 dB, an SSIM of 0.971, and a single-measurement run time of 0.87 seconds. Compared to EfficientSCI-B, this equates to a 0.16 dB improvement and a 43% reduction in run time; compared to STFormer-B, this equates to a 0.33 dB improvement and a 67% reduction in run time.

[0213] Figure 10 The presentation showcases the visual reconstruction results for some simulated scenes. In the Bosphorus scene, LiteSCI preserved the sharpness of the boat edges and water ripples. In the Jockey scene, LiteSCI resolved the saddle markings and limb outlines. In the Runner scene, LiteSCI sharpened shoe details and motion boundaries. In the ShakeNDry scene, LiteSCI preserved fine dust particles without resulting in a patchy smoothing. In the Traffic scene, LiteSCI reconstructed lane lines and vehicle features, and reduced ringing and ghosting. These advantages stem from the long temporal consistency provided by the temporal linear attention branch, and the fine detail enhancements resulting from channel-based recalibration.

[0214] Ablation experiment:

[0215] To verify the impact of the proposed ResLNet block and ELFormer module on reconstruction quality, as well as the memory usage of different reconstruction algorithms, ablation experiments were conducted. The results are shown in Tables 3 to 6, which not only compare the reconstruction quality of different models but also analyze the model parameters and floating-point operations (FLOPs). All experiments were performed on six grayscale benchmark datasets.

[0216] Table 3 Ablation Experiment Results of ResLNet Blocks

[0217]

[0218] ResLNet Blocks: Ablation experiments were conducted on ResLNet blocks to evaluate frame-by-frame gating and temporal smoothing. As shown in Table 3, all models were fixed in four groups within the residual blocks, with 256 input channels, and the ELFormer remained unchanged. The metrics were the average PSNR and SSIM values ​​across six grayscale benchmark tests, with a B-value of 8. The gating mechanism delivered significant performance improvements with negligible overhead: replacing simple summation with frame-by-frame gating improved PSNR from 36.12 dB to 36.38 dB and SSIM from 0.972 to 0.974, while keeping parameters and FLOPs essentially unchanged. Gating suppressed inter-group error accumulation and reduced short-tail artifacts under occlusion and fast motion. Adding lightweight temporal smoothing to the gating further improves accuracy to 36.50 dB PSNR and 0.975 SSIM, resulting in more stable frame-by-frame weights and less flicker; the increase in computation is very small, with approximately 0.1% more parameters and approximately 0.2% more floating-point operations. Increasing the smoothing kernel to 5 yields diminishing returns, as the accuracy is comparable to or slightly lower than 3 kernels, while computational cost increases, indicating that smaller kernels achieve a better balance between noise suppression and detail fidelity. Overall, the recommended ResLNet configuration is 3 gates plus a kernel size, which consistently improves PSNR and SSIM and significantly reduces ghosting artifacts under challenging spatiotemporal conditions.

[0219] Table 4 Ablation Evaluation Results of ELFormer Module

[0220]

[0221] Table 5 Ablation Evaluation Results Based on Feedforward Design

[0222]

[0223] ELFormer separates the spatial branch and temporal attention mechanisms while maintaining the feedforward design, and reports the average PSNR and SSIM of six grayscale benchmarks at a compression ratio of B=8. As shown in Table 5, the baseline model with the spatial convolutional branch and standard temporal self-attention mechanism achieves PSNR and SSIM of 36.44 dB and 0.975, respectively. Replacing the spatial branch with the proposed spatial-channel fusion branch combining lightweight convolution and ECA3D improves the PSNR to 36.48 dB at the same SSIM value, confirming that the channel recalibration in this application can improve detail recovery at a negligible cost. Under the original spatial branch, replacing the temporal module with the proposed temporal linear attention branch reduces time and memory complexity while maintaining an accuracy of 36.41 dB and 0.974. Using both components simultaneously yields the best results, with PSNR and SSIM of 36.50 dB and 0.975, respectively, indicating that spatial-channel fusion and linear temporal attention are complementary and can jointly improve reconstruction quality. Furthermore, as can be seen from Table 5, compared with CFFN and GFFN, the SFFN proposed in this application brings gains of approximately 0.91 dB and 0.06 dB to the PSNR value, respectively.

[0224] Table 6. Comparison of model parameters (Params) and computational complexity (FLOPs) on six grayscale benchmark datasets.

[0225]

[0226] Table 6 compares the number of parameters, computational cost, and reconstruction quality in six grayscale benchmarks. This application's LiteSCI achieves the best overall accuracy, with a PSNR of 36.50 dB and an SSIM of 0.975, while maintaining a moderate computational cost of only 980 GFLOPs and a measurement time of 0.20 seconds per measurement. Compared to the latest Transformer baseline model STFormer-B, LiteSCI improves PSNR by 0.16 dB, reduces FLOPs by approximately 68%, and shortens inference time by approximately 62%. Compared to EfficientSCI-B, LiteSCI reduces FLOPs by approximately 31%, shortens inference time by 44%, and achieves comparable or slightly higher accuracy; compared to EfficientSCI-B++, LiteSCI reduces FLOPs by approximately 8%, shortens inference time by 29%, and slightly improves PSNR while maintaining a similar SSIM. Compared to computationally intensive 3D-CNN or unfolded baseline methods (such as DUN-3DUnet and ELP-Unfolding), LiteSCI achieves a 1.24 dB and 1.09 dB improvement in PSNR, respectively, while reducing computation by 75%. The number of parameters is also kept to a minimum: LiteSCI has 8.20M, while STFormer-B has 19.48M, DUN-3DUnet has 61.91M, and ELP-Unfolding has 565.73M. In summary, LiteSCI achieves the highest accuracy among the listed methods while significantly reducing computation and latency, thus improving the expected accuracy-efficiency balance in the evaluation.

[0227] In summary, the present invention provides a video snapshot compression imaging method and system, which can achieve the following beneficial effects:

[0228] 1. Significantly Improved Scalability for Long Sequences: This invention employs kernelized linear attention (TLAB) in the temporal dimension, reducing the quadratic time or memory overhead of standard self-attention with sequence length to a near-linear level. This avoids explicitly constructing a T×T scoring matrix, thereby simultaneously reducing computational and storage requirements in long sequence SCI reconstruction, improving throughput while maintaining numerical stability and expressiveness. Unlike existing standard attention, which exhibits quadratic complexity in the temporal dimension (limiting the scalability of long videos), this invention achieves locality enhancement and robust aggregation through non-negative feature mapping and cosine nearest neighbor weighting, making it more suitable for deployment in high compression ratio and ultra-high-definition scenarios.

[0229] 2. Enhanced temporal stability and anti-artifact capability: The frame-level cross-gating proposed in this application introduces frame-by-frame gating and lightweight temporal smoothing at the fusion of historical and current information between groups. Without increasing parameters and computing power, it effectively suppresses ghosting and flicker, and improves the stability of long-term reconstruction. Compared with indiscriminate dense temporal fusion or single-path stacking, this invention reduces the risk of error amplification and temporal jitter at the structural level.

[0230] 3. Channel Redundancy Suppression and Detail Quality Enhancement: The spatial channel fusion branch employs an ECA3D module for lightweight channel recalibration. Through global average pooling, adaptive kernel-length 1D convolutions, and Sigmoid weights, replaying is performed, which significantly improves the response to critical details such as edges and textures while reducing channel redundancy, thus achieving higher reconstruction quality with the same computational budget. Compared to schemes relying solely on conventional convolutions or manual channel compression, this invention achieves superior channel selectivity with lower complexity.

[0231] 4. Comprehensive advantages in the trade-off between quality and efficiency: Based on the combination of linear temporal aggregation, cross-gated steady-state fusion, and lightweight channel recalibration described above, this invention achieves a competitive or even advanced level of accuracy and efficiency across six grayscale references and six color references. This is reflected in achieving higher or comparable reconstruction quality and more robust temporal performance with similar or lower computing power and memory.

[0232] In summary, this invention, through system improvements in time complexity, temporal stability, and channel selectivity, achieves quantifiable and interpretable enhancements in efficiency, memory usage, anti-artifact capability, and imaging quality compared to existing technologies, providing a superior engineering scalable solution for high compression ratio, ultra-high resolution, and long sequence SCI reconstruction.

Claims

1. A video snapshot compressive imaging reconstruction method, characterized in that, The method comprises the following steps: obtaining a video frame sequence and a corresponding time-varying mask set, and inputting the video frame sequence and the corresponding time-varying mask set into a video snapshot compression imaging forward measurement model to obtain an initial estimation; constructing a reconstruction network, including a feature extraction module, a gated residual network module and a video reconstruction module; inputting the initial estimation into the feature extraction module of the reconstruction network, the feature extraction module including two three-dimensional convolution layers, each three-dimensional convolution layer being connected with an activation function, and extracting initial features from the initial estimation through the feature extraction module; inputting the initial features into the gated residual network module to output reconstruction information features; fusing the reconstruction information features through the video reconstruction module, performing sub-pixel convolution for upsampling, performing three-dimensional convolution for detail refinement, and reconstructing a video sequence; the gated residual network module includes five ResLNet blocks, a three-dimensional convolution layer and an activation function are arranged between the second and third ResLNet blocks, two three-dimensional convolution layers are arranged between the fourth and fifth ResLNet blocks, and the output of the first three-dimensional convolution layer in the two three-dimensional convolution layers is spliced with the output of the second ResLNet block in the channel and then subjected to the second three-dimensional convolution; each ResLNet block performs information transmission through short or long jump connections, and models the spatial and temporal correlation by adopting grouped processing, frame-level cross gating and time linear attention mechanism, and each ResLNet block comprises the following steps of performing the grouped processing, frame-level cross gating and time linear attention mechanism modeling spatial and temporal correlation: The input features of the ResLNet block are evenly divided into S groups in the channel dimension. The first group is directly subjected to a spatiotemporal factorization to obtain the output Y 1; For the i-th group, where i {2,..., S}, a frame-level cross-gating operation is performed to adaptively fuse the output of the previous group Y i-1 with the input of the current group X i to obtain the fused feature Z i ; performing the fusion feature Z i spatio-temporal factorization to obtain an output Y i ; Output of all groups { Y 1,..., Y S The features are concatenated along the channel dimension, compressed by 1×1×1 convolution, and then residually connected to the input features of the ResLNet block to obtain the output of the ResLNet block. the spatial and temporal factorization processing is realized through a spatial and temporal factorization block ELFormer, and the spatial and temporal factorization block ELFormer includes three branches arranged in parallel and sequentially fused: a spatial channel fusion branch for performing local spatial feature extraction and channel re-labeling; a time linear attention branch for modeling global time domain dependence with near-linear complexity; a Swish-gated time feedforward network branch for stably gating and fusing the outputs of the previous two branches.

2. The method of claim 1, wherein, The frame-level cross gating operation specifically comprises: respectively on the output of the previous group Y i-1 with the current input X i global average pooling to obtain the corresponding pooling feature vector and : wherein, denotes a global average pooling over the channel and spatial dimensions for each frame, H denotes a height, W denotes a width, H x W i.e. spatial dimensions, C denotes a channel dimension; splicing the pooled feature vectors along the channel dimension, and performing one-dimensional time convolution on the spliced features for smoothing processing: wherein, represents the concatenated features; is a one-dimensional convolution along the time dimension, i.e. the number of frames T for smoothing the gating sequence, k represents the size of the convolution kernel; represents concatenation along the channel dimension; After the sigmoid activation function, the frame-by-frame gating weight vector is obtained : wherein sigmoid is a Sigmoid activation function mapping real numbers to (0,1); With the gating weight vector, a frame-level cross-gating is constructed : wherein, denotes a gating function; and , = C / S， denotes the number of channels per group; The frame-level cross-gating is used to adaptively reconcile the historical information and the current information, and output the fusion features Z i .

3. The method of claim 2, wherein, the spatial channel fusion branch performs the following operations: Input features for spatio-temporal factorized block elformer Channel reshaping is performed to obtain reshaped input features ; applying a double layer two-dimensional convolution to the input features to reconstruct spatial details, the double layer two-dimensional convolutions being connected by an activation function to obtain features ; Subsequently, a lightweight channel attention mechanism was used to analyze the features. Perform channel recalibration, specifically as follows: The input features after the reconstruction of spatial details are globally average-pooled, and the pixels of each channel of each frame are averaged to obtain the intensity statistics of each channel of each frame. :​​ On intensity statistics Apply one-dimensional convolution, kernel length k Determined according to the number of channels: wherein and is a constant and denotes taking the nearest odd integer; Generating attention by one-dimensional convolution along a channel : wherein denotes a one-dimensional convolution; The output of the one-dimensional convolution is passed through a sigmoid activation function to obtain the channel weights : The channel weights are extended to spatial dimensions and combined with features element-wise multiplication, implementing channel re-scaling, resulting in ; Finally, the channel reshaping is performed to obtain reshaped output features .

4. The method of claim 2, wherein, the time linear attention branch performs the following operations: Attention is made along the time axis, and the input features of the spatiotemporal factorization block ELFormer Channel reshaping is performed to obtain: Will Grouping by spatial position, get the time feature sequence of each spatial position, the time sequence of the i-th spatial position is recorded as ; i = 1, 2, …, Ns ; By a trainable linear projection matrix Linearly project the time feature sequence of each spatial bin Map each frame's features to A d-dimensional query, key, value space yields: A d-dimensional query, key, value space yields: wherein, is a query sequence, is a key sequence, is a value sequence, is a value dimension; to and applying a non-negative feature mapping function , resulting in and where d s =d; Time index Definition: wherein M is a hyperparameter of time scale intensity, used to set a phase step for reweighting the cosine or sine; the mapped query sequence and the key sequence are introduced into a cosine and sine-based proximity bias for frame-by-frame scaling: in, These represent the mapped query matrix. The Row by coefficient or Perform element-wise scaling and at time Inject the corresponding cosine or sine weights; These represent the mapped key matrices respectively. The Row by coefficient or Perform element-wise scaling and at time Inject the corresponding cosine or sine weights; Indicates element-wise multiplication; The output at each time instant t is computed using a linear attention mechanism : wherein is an all-one vector of length ; is a numerical stability constant; denotes transpose; Finally, the output of all pixels Parallel computing is performed to obtain the output , and then The output of the timeline attention branch is obtained by restoring through channel reshaping .

5. The method of claim 2, wherein, the Swish-gated time feedforward network branch performs the following operations: The input features of the Swish-gated time feedforward network branch are After point-wise convolution and activation, it is equally divided along the channel dimension into a value branch and a gating branch; performing three-dimensional convolution and activation on the value branch to obtain an output Y1; for the gating branch, the Y1 subjected to one-dimensional time convolution and gating processing through a Swish activation function SiLU is multiplied element by element, and then three-dimensional convolution and activation are performed to obtain an output Y2; Y1 and Y2 are spliced by channel, and then activated and After point-by-point convolution, the residual connection is performed with the input features of the Swish-gated temporal feedforward network branch to obtain the final output of the Swish-gated temporal feedforward network branch.

6. A video snapshot compressed imaging reconstruction system for implementing a video snapshot compressed imaging reconstruction method according to any one of claims 1 to 5, characterized in that, The system comprises: a preprocessing module for obtaining a video frame sequence and a time-varying mask set, and inputting the video frame sequence and the time-varying mask set into a video snapshot compression imaging forward measurement model to obtain an initial estimation; a reconstruction network module including a feature extraction module, a gated residual network module and a video reconstruction module; The feature extraction module is configured to extract initial features according to the initial estimation, and includes two three-dimensional convolution layers, each of which is followed by an activation function; The gated residual network module is configured to extract reconstruction information features, and includes five ResLNet blocks, wherein a three-dimensional convolution layer and an activation function are arranged between the second and third ResLNet blocks, two three-dimensional convolution layers are arranged between the fourth and fifth ResLNet blocks, the output of the first three-dimensional convolution layer in the two three-dimensional convolution layers is spliced with the output of the second ResLNet block in the channel, and then the second three-dimensional convolution is performed; each ResLNet block transmits information through short or long jump connections, and models the space-time correlation by using group processing, frame-level cross gating, and time linear attention mechanism; The video reconstruction module is configured to fuse the reconstruction information features, perform sub-pixel convolution for up-sampling, and apply three-dimensional convolution to refine details, so as to reconstruct a video sequence; An output module is configured to output the reconstructed video sequence.

7. The system of claim 6, wherein, The ResLNet block includes: A grouping unit configured to group the input features in the channel dimension; A gating unit configured to calculate frame-level cross gating; A space-time factorization unit configured to perform space-time factorization processing on each group of features; A fusion unit configured to splice the outputs of the groups and perform 1×1×1 convolution, and finally perform residual connection with the input features of the ResLNet block to obtain the output of the ResLNet block.

8. The system of claim 7, wherein, The space-time factorization unit includes an ELFormer block, and each ELFormer block has three branches, including: A spatial channel fusion branch configured to extract local spatial features and re-label channels; A time linear attention branch configured to model global time domain dependence with near-linear complexity; A Swish-gated time feedforward network branch configured to stably gate and fuse the outputs of the first two branches.

Images