End-to-end snapshot compression computer vision method based on pseudo-random mask array

Abstract

The invention discloses an end-to-end snapshot compression computer vision method based on a pseudo-random mask array. The method comprises the following steps of: performing single exposure on a video sequence subjected to space-time light intensity modulation by using a two-dimensional image sensor and a pseudo-random binary mask to obtain a corresponding two-dimensional (2D) compression measurement value; a video reconstruction is taken as a proxy task, a trained compression denoising auto-encoder model is utilized to extract potential spatial-temporal feature representations from the 2D compression measurement values, and the compression denoising auto-encoder model comprises a shared encoder and at least one task-specific decoder. The shared encoder is configured to extract the potential spatio-temporal feature representation from the 2D compressed measurements; and inputting the potential spatio-temporal feature representation into the decoder to obtain a dynamic result of the downstream computer vision task. According to the method, the image reconstruction complexity and the calculation cost are remarkably reduced, the excellent performance can still be kept especially in an extremely low illumination scene, and the privacy protection characteristic is achieved.

Description

End-to-end snapshot compression computer vision method based on pseudo-random mask array Technical Field

[0001] This invention relates to the fields of computational imaging and computer vision, and more specifically, to an end-to-end snapshot compression computer vision method based on a pseudo-random mask array. Background Technology

[0002] Traditional computer vision techniques have been widely validated in video and image processing (such as video classification, object detection and tracking, and monocular depth estimation). For example, frame-based computer vision paradigms are typically used in the RGB space, acquiring high-resolution spatial and temporal samples through synchronous exposure and progressive scanning at fixed time intervals using image sensors such as CMOS active pixel sensors. This approach first converts the light signal into analog voltage using a pixel array, then quantizes it into 12-14 bit raw RAW data via an ADC (analog-to-digital converter). Subsequently, to compensate for fixed-mode noise caused by the non-uniformity of the CMOS process, black level correction and dark current compensation are performed. Next, a demosaic algorithm (such as bilinear or guided filtering-based algorithms) is used for CFA (color filter array) interpolation to reconstruct the RGB three-channel image, supplemented by ISP pipeline processing such as lens shading correction, white balance, and gamma correction, finally outputting an 8-bit compressed standard dynamic range image. While this full-frame exposure and synchronous readout mechanism can provide dense texture details, it involves high pixel sampling throughput (e.g., 1080p@60fps requires processing 124MB / s of raw data). It requires multi-level caching and a dedicated ISP (Image Signal Processing) unit for enhancement operations such as denoising, de-mosaicing, and color space conversion, resulting in high power consumption (typical smartphone ISP power consumption is about 400~800mW) and high latency (end-to-end processing latency is usually more than 30ms), making it difficult to meet the real-time requirements of low-power, low-latency scenarios (such as AR / VR headsets or micro drones).

[0003] In recent years, the development of SCI (Snapshot Compressed Imaging) technology has provided a promising alternative to traditional imaging. In SCI, dynamic optical modulation achieves temporal aliasing coding by assigning a unique coded aperture (modulation mask) to each frame. These masks are rapidly refreshed during a single exposure of the detector using a high-speed SLM (Spatial Light Modulator). Therefore, 2D measurements can be obtained using the same exposure cycle with compressed sampling of multiple modulated frames at super-temporal resolution without changing the original imaging sensor frame rate. Compared to costly high-speed cameras without optical coding, this significantly reduces the data bandwidth and power consumption of the sensor readout, and the coding based on compressed sensing mathematics allows for high-fidelity reconstruction of the original video frames through algorithms. However, SCI still faces fundamental hardware hurdles. SCI systems typically require large-size, frame-level spatial modulation masks to satisfy RIP (Restricted Isometry Property). One approach to implementing the mask is to load it from external memory and transmit the mask pattern to the spatial light modulation device before each exposure. Obviously, this approach introduces significant latency, power consumption, and system complexity at higher resolutions. A more efficient alternative is to directly control the CMOS pixel state via dedicated external signal lines. However, since CMOS control lines are typically 0.05–1 micrometer wide, roughly the same width as the pixel cell spacing, each pixel can only accommodate a maximum of about 10 such lines due to physical limitations. This imposes a strict upper limit on the mask size, making the efficiency-performance trade-off of large-size dynamic masks in practical sensors quite challenging.

[0004] The prevalent low-light and low signal-to-noise ratio imaging conditions in the real world pose a challenge to APS and SCI-based vision systems, yet are crucial in scenarios such as autonomous driving and robotics. These conditions generate complex noise patterns (typically modeled as Poisson-Gaussian noise). In dynamic scenes, signal strength fluctuates over time, and Poisson noise becomes non-stationary and time-dependent, deviating from the traditional memoryless assumption. Existing denoising neural networks rarely consider this temporal dependence, thus achieving robust computer vision under extremely low light conditions remains an open challenge. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an end-to-end snapshot compression computer vision method based on a pseudo-random mask array. The method includes the following steps: using a two-dimensional image sensor, a video sequence subjected to spatiotemporal intensity modulation is subjected to a single exposure using a pseudo-random binary mask to obtain the corresponding 2D compressed measurement value; using video reconstruction as a proxy task, a trained compression-denoising autoencoder model is used to extract latent spatiotemporal feature representations from the 2D compressed measurement values. The compression-denoising autoencoder model includes a shared encoder and at least one task-specific decoder. The shared encoder is used to extract the latent spatiotemporal feature representations from the 2D compressed measurement values; the latent spatiotemporal feature representations are input into the decoder to obtain the dynamic results of the downstream computer vision task.

[0006] Compared with the prior art, the advantages of the present invention are that it can directly perform computer vision tasks without image reconstruction by using snapshot compressed sensing technology. The main advantages are as follows: 1) The present invention avoids the failure problem of computer vision methods based on traditional ISP pipelines in low-light and low signal-to-noise ratio sound scenes by directly extracting spatiotemporal features in the compressed measurement domain. When APC=20, the edge detection ODS can reach 0.690, which is significantly better than the highest 0.474 of the traditional method.

[0007] 2) This invention uses a fixed, such as 8×8 pseudo-random binary mask, which eliminates the need for large-scale dynamic masks or external memory, significantly reducing the manufacturing complexity and power consumption of the sensor, and is suitable for mobile and embedded devices.

[0008] 3) This invention utilizes a self-supervised pre-training strategy, which requires only a large number of unlabeled compressed measurements to learn a general noise-robust representation, thus eliminating the dependence on expensive RGB-truth pairs.

[0009] 4) This invention compresses the number of parameters to the extreme through rate-constrained training, with an inference time of only 2.60 ms / frame, which is 1 / 25 of the existing best depth estimation model Unidepth, making it suitable for real-time applications.

[0010] 5) This invention introduces a task-specific lightweight decoder that directly maps compressed features to edge, depth, or semantic segmentation results without reconstructing the original video frames, achieving end-to-end cross-modal output. The shared encoder + lightweight decoder architecture supports plug-and-play task switching, and the same model achieves or surpasses the current two-stage baseline model level in the three major tasks of edge detection, depth estimation, and semantic segmentation.

[0011] 6) Since the high-fidelity RGB frames are not reconstructed throughout the process, this invention naturally meets the needs of privacy-sensitive scenarios, such as medical, security, and vehicle cabin monitoring applications.

[0012] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0014] Figure 1 is a flowchart of an end-to-end snapshot compression computer vision method based on an 8×8 pseudo-random mask according to an embodiment of the present invention; Figure 2 is an overall framework diagram of the end-to-end snapshot compression computer vision method based on an 8×8 pseudo-random mask according to an embodiment of the present invention; Figure 3 is a schematic diagram comparing the edge detection results of the compressed denoising autoencoder model and DexiNed-f according to an embodiment of the present invention when APC≤20; Figure 4 is a schematic diagram comparing the semantic segmentation results of the compressed denoising autoencoder model, SegFormer, and the two-stage method DIC+SegFormer according to an embodiment of the present invention on the Cityscapes dataset; Figure 5 is a schematic diagram comparing the edge detection results of the compressed denoising autoencoder modality and edge detection model and the corresponding two-stage methods according to an embodiment of the present invention on the SCI grayscale benchmark dataset; Figure 6 is a schematic diagram comparing the depth estimation results of the compressed denoising autoencoder model, monocular depth estimation model, and the corresponding two-stage methods according to an embodiment of the present invention on the KITTI dataset; Figure 7 is a schematic diagram of the edge detection effect of the compressed denoising autoencoder model on real SPAD512S ultra-low illumination data according to an embodiment of the present invention. Detailed Implementation

[0015] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0016] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0017] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0018] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0019] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0020] For ease of understanding, the existing technology will be described first by way of example.

[0021] In existing technologies, Snapshot Compressive Imaging (SCI) systems generally adopt a two-stage process of "reconstruction first, recognition later". First, high-dimensional video sequences are compressed into a single two-dimensional measurement frame using compressed sensing theory. Then, iterative optimization or deep neural networks are used to restore the measurement frame into multiple original video frames. Finally, downstream tasks such as object detection, depth estimation, and semantic segmentation are performed on the reconstructed video. The core link is described below.

[0022] (i) Optical Encoding Stage: During the exposure cycle, the high-speed digital micromirror device (DMD) is encoded according to a frame-level mask sequence. For video frames Two-dimensional spatial modulation is performed, and then all modulated frames are accumulated into a single measurement frame. Where Cr is the compression ratio and N is the noise. To meet the conditions of compressed sensing RIP, the mask size must be equal to the image size (H×W), resulting in huge DMD loading bandwidth and high system cost.

[0023] (II) Video Reconstruction Stage: Existing technologies in this stage generally follow the paradigm of "solving the inverse problem of compressed sensing," with the core objective being to recover the original video sequence from a single measurement frame Y. The specific implementations fall into two main categories: model-based iterative optimization methods and deep learning-based reconstruction networks.

[0024] For model-based iterative optimization methods, early works such as GAP-TV embedded the total variational (TV) prior into a generalized alternating projection framework, achieving reconstruction through iterative threshold shrinkage. DeSCI further introduced nonlocal low-rank constraints, utilizing the low-rank characteristics of similar block groups to suppress noise, but required a large number of iterations. PnP-FFDNet embedded a plug-and-play denoiser into the ADMM iteration, improving quality, but GPU memory usage increased quadratically with resolution. These methods fail in low-light scenes (APC ≤ 20) due to noise amplification, causing the TV / LR prior to fail.

[0025] For deep learning-based reconstruction networks, DUN-3DUnet employs a deep unfolding architecture, deeply unfolding the iterative steps; the spatiotemporal Transformer module models long-range dependencies through spatiotemporal self-attention; EfficientSCI introduces hierarchical residual connections to accelerate 4K video inference, but requires separate model training for each compression ratio. All deep methods require training on large-scale synthetic data (DAVIS2017), resulting in a significant decrease in generalization on real low-light data. In addition to the aforementioned general reconstruction frameworks, recent research has proposed system-level innovations specifically for low-light scenes. However, this method models the noise of low-light compression measurements as Poisson noise with a fixed mean, lacking a reasonable description of the object properties of shot noise. Furthermore, the key points of this method in the reconstruction stage can be summarized as an adaptive progressive coding strategy. Unlike the traditional method of loading the entire frame-level mask all at once, the authors design the mask sequence as "progressive," that is, first projecting a coarse mask with low spatial frequencies and high energy transmission, and then gradually projecting a fine mask with high spatial frequencies and low energy transmission. This strategy significantly suppresses low-light shot noise by increasing the system's luminous flux in the illuminance range of 2 lux–60 lux by adjusting the mask energy distribution in real time. Although recent work has greatly improved reconstruction quality under low illumination, it still essentially belongs to the "reconstruction-then-recognition" paradigm. Although the mask is progressive, its overall size is still the same as the image, and the bandwidth and latency bottlenecks of loading the mask still exist. Therefore, existing low-light SCI reconstruction techniques still cannot meet the application requirements of cross-modal, ultra-low illumination, and millisecond-level response.

[0026] In summary, regardless of model-based iterative optimization methods or deep learning methods, the output of the existing reconstruction stage is still a complete video frame, requiring downstream tasks to process full-resolution pixels (e.g., 1920×1080×Cr), with computational complexity proportional to the number of frames. When Cr=8, the subsequent detection network needs to process 8× pixels, creating a "data explosion" bottleneck. Furthermore, reconstruction errors accumulate non-linearly in low-light, high-noise scenarios, making downstream task accuracy sensitive to reconstruction quality, resulting in an "error amplification" effect. This invention completely skips this stage when performing computer vision tasks, directly extracting task features in the compressed measurement domain, thus thoroughly solving the above problems.

[0027] (iii) In the downstream recognition stage, the reconstructed video sequence is input into an existing RGB network (such as DexiNed, Unidepth, SegFormer) to complete the visual task. Since reconstruction errors, motion blur, and low-light noise are amplified in the secondary processing, the overall accuracy is heavily dependent on the reconstruction quality, forming a "error accumulation" bottleneck.

[0028] Recent studies have attempted to cascade reconstruction and recognition networks to reduce error propagation through feature sharing or loss weighting. However, these methods still require complete video frames as intermediate representations and cannot eliminate the memory and latency overhead of the reconstruction stage.

[0029] In summary, existing technologies have the following drawbacks: High hardware complexity: Traditional frame-level masks require storing a mask matrix with the same size as the frame, which places stringent requirements on the latency, bandwidth, and power consumption of the spatial light modulator; Poor low-light robustness: The signal-to-noise ratio of the reconstruction network drops sharply under low photon (APC≤20) conditions, causing downstream tasks to fail; Insufficient real-time performance: The two-stage serial processing not only has the accumulation of reconstruction errors, but also brings a double delay of "reconstruction delay + recognition delay", which cannot meet the millisecond-level requirements of autonomous driving, real-time monitoring and other scenarios.

[0030] To address the shortcomings of the existing technologies, this invention proposes an end-to-end snapshot compression computer vision method based on a pseudo-random mask array. By using a fixed small-size mask (e.g., 8×8), an asymmetric compression denoising autoencoder, a task-specific decoder, and unified noise training and rate constraint training, it completes low-light, low-signal-to-noise ratio feature extraction in the single-exposure compression measurement domain, completely eliminating video reconstruction in downstream tasks and realizing an end-to-end paradigm of "compression equals recognition".

[0031] In general, as shown in Figures 1 and 2, the provided end-to-end snapshot compression computer vision method based on 8×8 pseudo-random mask includes the following steps: S1, snapshot compression sensing: high-speed video sequences are compressed into a single two-dimensional snapshot measurement by spatiotemporal light intensity modulation.

[0032] For example, using a two-dimensional image sensor, a single exposure is performed on a high-speed video sequence that has undergone spatiotemporal light intensity modulation using an 8×8 pseudo-random binary mask to obtain the raw data of a 2D compressed measurement value. The spatiotemporal light intensity modulation can be achieved by placing a set of time-varying modulation masks in front of the sensor; S2, Spatiotemporal representation pre-training: a Compressed Denoising Autoencoder (CompDAE) model is directly pre-trained from the raw compressed data and robust spatiotemporal features are extracted from the model encoder.

[0033] For example, a Poisson-Gaussian composite noise model is introduced during a single sampling exposure to construct a real sensor noise distribution. Then, 2D compressed measurements containing composite noise are used as input. A CompDAE (Compression Denoising Autoencoder) model is used to extract a compact latent spatiotemporal feature representation directly from the 2D compressed measurements by using video reconstruction as a proxy task.

[0034] S3. Downstream task generalization: Use pre-trained features to complete various downstream computer vision tasks such as edge detection and depth estimation.

[0035] For example, by inputting latent spatiotemporal feature representations into a task-specific decoding module model, high-speed dynamic results for other downstream computer vision tasks can be directly output using a lightweight task-specific decoder, without the need to explicitly reconstruct the original video frames throughout the fine-tuning stage and model inference process.

[0036] In one embodiment, the snapshot compressed sensing in step S1 specifically includes: mask generation: repeatedly tiling an 8×8 pseudo-random binary matrix in space through Kronecker product to form a frame-level mask consistent with the sensor resolution. When loading the pseudo-random mask, only 8×8×Cr size mask data needs to be transmitted; exposure modulation: within a single exposure cycle, the mask sequence is switched in chronological order to achieve spatial-temporal encoding of T-frame video; Poisson-Gaussian noise injection: simulating the composite measurement noise of a real sensor.

[0037] In one embodiment, the spatiotemporal representation pre-training in step S2 specifically includes: Network structure: CompDAE consists of a shared encoder and a lightweight decoder, wherein the shared encoder, except for the preprocessing module and the noise level estimation module which are not defined using neural network layers, includes 3D convolutional layers and (MN) spatiotemporal Transformer modules, and the decoder uses only N spatiotemporal Transformer modules and 3D inverted convolutional layers; Agent task: The self-supervised objective is to reconstruct the original video frame, and the loss function is the pixel-wise MSE loss function; Noise adaptation: Gaussian noise level map is added as input and the average photon count (APC) is estimated based on the 2D compressed measurement value. After being concatenated with the coarse video estimate predicted by the preprocessing module, it is embedded into the encoder to achieve robust training across illumination and signal-to-noise ratio.

[0038] In one embodiment, the downstream task generalization of step S3 specifically includes: task decoder replacement: the edge detection head uses 1×1 convolution + sigmoid to output a single-channel probability map; the depth estimation head uses inverted convolution upsampling + 1×1 convolution to output a single-channel depth; the semantic segmentation head uses PPM module + 1×1 convolution + bilinear interpolation upsampling + Softmax to output class probability; partial fine-tuning: freeze the weights of the shared encoder network and train only the task decoder; cross-task switching: the same latent feature is switched between tasks in real time through different decoders.

[0039] Specifically, the solution of the present invention is further described according to the following examples: 1. Architecture Overview The present invention aims to completely revolutionize the processing flow of computational imaging and computer vision in order to address the fundamental challenges faced by traditional imaging systems in extreme scenarios such as high hardware complexity, high power consumption and latency, and sharp performance reduction when the signal-to-noise ratio is low.

[0040] The core of this invention comprises three organically integrated components: a hardware-friendly snapshot compression front-end, a spatiotemporal Transformer-based shared encoder, and a lightweight task-specific decoder. This invention overturns the traditional two-stage paradigm of "reconstruction first, recognition later" commonly followed in the field of snapshot compression imaging (SCI). Its workflow is as follows: First, a fixed, small-sized (8×8) pseudo-random binary mask is used at the sensor front-end to efficiently modulate the spatiotemporal light intensity of high-speed scenes, thereby compressing multiple consecutive video frames into a single two-dimensional measurement map within a single exposure. Subsequently, a powerful shared encoder directly learns and extracts robust and universal spatiotemporal feature representations from the raw compressed measurement data containing real physical noise (Poisson-Gaussian noise). Finally, depending on the specific application requirements, this feature is input into different plug-and-play lightweight decoders to directly output high-level visual information such as edge maps, depth maps, or semantic segmentation results in an end-to-end manner, without reconstructing any high-fidelity original video frames.

[0041] Furthermore, to ensure the model's superior generalization ability and deployment efficiency in complex and ever-changing real-world environments, this invention innovatively combines a unified noise-adaptive training strategy with a rate-constrained optimization strategy. The former enables a single model to adapt to a wide dynamic range from extremely low light to normal lighting, while the latter generates highly sparse and compressible compact models, making them ideal for deployment on edge devices. Through this integrated design, this invention achieves a novel computational vision architecture that is hardware-simplified, computationally efficient, low-light robust, and inherently privacy-preserving, truly realizing "visual intelligence without images."

[0042] 2. Snapshot Compression Front-end: This invention constructs an optical coding front-end that is physically easy to implement, cost-effective, and meets the performance requirements of compression sensing, aiming to overcome the most significant hardware implementation bottleneck in traditional SCI systems.

[0043] (1) Hardware-friendly mask design Traditional SCI systems typically require the use of dynamic masks of the same size as the sensor resolution (i.e., frame-level) in order to satisfy the restricted isometric property (RIP) in compressed sensing theory. This requires reading huge mask data from external memory and loading it onto a spatial light modulator (such as DMD) before each exposure. This process introduces huge data bandwidth, power consumption and system latency, becoming a major obstacle to large-scale commercial deployment.

[0044] To address this issue, this invention proposes a hardware-simplified masking scheme. Instead of frame-level masking, this scheme uses a set of extremely small (e.g., 8×8 pixels) pseudo-random binary sub-masks. Where T is the number of compressed frames (Cr), i.e., the number of compressed video frames in a single measurement. In practical applications, these submasks are periodically copied and tiled in space through Kronecker product operations, thereby dynamically generating a matrix that corresponds to the sensor resolution. A consistent full-size modulation mask Mt. Its mathematical expression is: (1) Among them, J is a matrix of all ones. Represents Kronecker.

[0045] This design requires storing and processing only a very small amount of mask data (8×8×Cr) throughout the imaging process, reducing the data volume by several orders of magnitude compared to traditional methods. This significantly reduces storage and bandwidth requirements, making it possible to directly integrate mask control logic into the CMOS image sensor chip, thereby achieving a low-power, low-latency on-chip computer vision system. The probability (i.e., density) of non-zero elements in the mask. ) is an adjustable hyperparameter that affects light throughput and coding diversity.

[0046] (2) Forward Imaging Model This invention uses Coordinated Aperture Compressed Time Imaging (CACTI) as its physical model. Within a single exposure time of a camera, the high-speed video sequence to be acquired... Each frame in They will all be masked by their corresponding full-size mask. Pixel-by-pixel intensity modulation is performed. The intensity of all modulated frames is physically accumulated (integrated) on the sensor's photosensitive unit, ultimately forming a two-dimensional compressed measurement map. This noise-free, ideal forward imaging process can be mathematically represented as: (2) Where ⊙ represents the Hadamard product (i.e., element-wise product).

[0047] (3) Realistic Noise Modeling In the real world, especially under low light conditions, sensor noise is a critical factor that cannot be ignored. In order to learn noise-robust characteristics, this invention accurately simulates the physical sources of noise in training data generation and model design. Unlike simply assuming additive white Gaussian noise, this invention adopts a Poisson-Gaussian mixture noise model, which can more realistically reflect the noise characteristics of the original sensor data.

[0048] The Poisson-Gaussian mixed noise model includes two main types of noise: Poisson noise and Gaussian noise. Poisson noise, also known as shot noise, originates from the quantum randomness of photons arriving at the sensor, and its intensity is related to the signal itself (i.e., light intensity); the stronger the light, the greater the noise. Gaussian noise mainly includes readout noise and thermal noise, and its intensity is independent of the signal.

[0049] In one embodiment, the measurement process incorporating real noise is modeled as follows: (3) Where T represents the total number of high-speed video frames whose single exposure time is compressed. This represents the Poisson noise component in the t-th frame during a single exposure. This represents the additive white Gaussian noise component of a single-exposure snapshot. It is the t-th raw frame of the high-speed video. It is the mask corresponding to the modulation of the t-th frame.

[0050] The probability distribution of each noise component follows the following: and .

[0051] Here It is a gain factor. and These represent the Poisson and normal distributions, respectively. It's worth noting that in dynamic scenarios, due to the changing signal strength over time, Poisson noise is actually time-varying and non-stationary, and its statistical characteristics are correlated over time. The end-to-end learning framework of this invention can implicitly learn and handle this complex time-correlated noise.

[0052] 3. The CompDAE network model is the core computational module of this invention. It is a deep neural network specifically designed to process the compressed, noisy measurement data described above. Its goal is not to reconstruct images, but to directly extract high-level semantic features for downstream visual tasks.

[0053] (1) Overall Architecture Concept CompDAE is essentially a compression and denoising autoencoder. During the pre-training phase, CompDAE learns a mapping from a highly compressed and noisy measurement domain to a clean and complete spatiotemporal signal domain. The CompDAE architecture adopts an efficient asymmetric encoder-decoder architecture.

[0054] Heavy Encoder: The encoder is the core of the model, with a complex structure and a large number of parameters. The heavy encoder's responsibility is to handle the difficult inverse problem: decoupling and extracting high-dimensional, clear latent spatiotemporal features from aliased, compressed, and noisy two-dimensional inputs.

[0055] Lightweight Decoder: The structure of the decoder is very simple and has a small number of parameters. The task of the lightweight decoder is relatively easy: to recover task-specific output (such as reconstructed video frames or segmentation maps) from the clean, structured latent features provided by the encoder.

[0056] This asymmetric design greatly improves the efficiency of training and fine-tuning. During the fine-tuning phase, only the decoder with a small number of parameters needs to be trained, while the encoder, which accounts for the majority of the model's parameters, remains frozen.

[0057] (2) Detailed Explanation of the Shared Encoder: The encoder uses a powerful spatiotemporal Transformer as its core unit and integrates other modules designed specifically for processing noise data in the compressed domain. Initial Estimation and Noise Adaptive Input: Before entering the Transformer body, the input data passes through a preprocessing module. This module first performs preliminary calculations on the input measurements and mask to obtain a rough estimate of the original signal. More importantly, this invention introduces a unified noise adaptive training strategy. This strategy constructs a multi-channel input tensor that explicitly separates the signal from the noise, guiding the network to learn noise-perceived features. This input tensor is composed of three parts: the aforementioned initial signal rough estimate; a graph representing the Gaussian noise level, where all pixel values ​​are equal to the known noise standard deviation σ; and a graph representing the Poisson noise level, whose value is inferred from the average photon count from the measured intensity. Specifically, the Gaussian noise level graph is: the Gaussian noise standard deviation coefficient estimated pixel by pixel in the video. Since the Gaussian noise in the measurement noise is signal-independent noise, zero-mean Gaussian white noise is specifically used. The Poisson noise level diagram is as follows: Due to quantum effects before photons reach the sensor during measurement, and the number of photons reaching the sensor unit within a single exposure time following a Poisson distribution, which is directly related to the mean of the original signal, the APC (Average Photon Count) is used to characterize the Poisson noise level. In this way, a single CompDAE model can intelligently adapt to various illumination and signal-to-noise ratio conditions, ranging from extremely low light (APC=1) to normal illumination (APC~60), without retraining.

[0058] Spatiotemporal Token Generation Module: The preprocessed multichannel tensor is fed into a 3D convolutional layer. 3D convolution can effectively capture local spatiotemporal neighborhood information. The feature vector at each spatial location in the feature map output by this convolutional layer is regarded as a "spatiotemporal token", which encapsulates the local features of that spatiotemporal region.

[0059] Spatio-Temporal Transformer Module: The generated spatio-temporal token sequence is then fed into a sequence stacked by multiple (e.g., M - N) spatio-temporal Transformer modules. The core of the spatio-temporal Transformer module is its efficient spatio-temporal self-attention mechanism, which can capture the long-range dependencies between features simultaneously in the spatial and temporal dimensions, thus globally modeling the dynamic changes and spatial structure of the video. After being processed by all spatio-temporal Transformer modules, the output is the final, highly condensed latent spatio-temporal feature representation.

[0060] (3) Task-Specific Decoder The design of the decoder follows the principles of light weight and flexibility.

[0061] General Structure: The basic structure of the decoder consists of, for example, a few (N, where N < M / 2) spatio-temporal Transformer modules and a final task prediction head (usually a convolutional layer or a fully connected layer).

[0062] Role Transformation: In the pre-training stage, the task of the decoder is to receive the latent features output by the encoder and upsample and reconstruct them into the original, clean multi-frame video sequence. In the fine-tuning stage, this reconstruction head is removed and replaced by a brand-new prediction head designed for a specific downstream task.

[0063] The task-specific decoder can be adapted to various types of tasks. For example, for the edge detection task, the prediction head can be a simple 3D transposed convolutional layer, outputting a single-channel edge probability map. For the monocular depth estimation task, the decoder can include a spatio-temporal Transformer block and a 3D transposed convolutional layer, outputting a single-channel depth map. For the semantic segmentation task, the decoder can be directly replaced by a mature segmentation head, such as the Pyramid Scene Parsing Network (PPM), which is attached after the fixed encoder for training. This "encoder sharing, decoder plug-and-play" mode makes CompDAE a highly scalable and general computer vision platform.

[0064] 4. Training and Optimization Strategies To enable the CompDAE model to achieve the best performance and highest efficiency, a two-stage training framework is designed and advanced rate-constrained optimization techniques are introduced.

[0065] (1) Two-stage training framework The first stage is self-supervised pre-training, which is the stage where the model learns general knowledge. Its core idea is to use massive amounts of video data that do not require manual annotation (such as the DAVIS2017 dataset) to train the model. The training goal (or “surrogate task”) is to enable the CompDAE network to learn to reconstruct the original, clean video frames as accurately as possible from the generated compressed measurements containing Poisson-Gaussian noise. In this process, the shared encoder is forced to learn how to separate useful spatiotemporal structures from the messy information, thereby gaining a deep understanding of the video content and dynamic changes. This stage usually involves a long training time, such as 80 epochs.

[0066] The second stage is task-specific partial fine-tuning. After obtaining a powerful general encoder through pre-training, it is adapted to a specific downstream task, such as edge detection or depth estimation. This stage employs an efficient "partial fine-tuning" strategy: freezing the encoder: the weights of the pre-trained shared encoder, which accounts for the majority of the model's parameters, are completely fixed, preventing it from participating in gradient updates during fine-tuning; training the decoder: only the newly added, lightweight, task-specific decoder is trained. Due to the small number of decoder parameters, this process is very fast and requires only a relatively small amount of labeled data (e.g., 40 epochs for edge detection fine-tuning, 50 epochs for depth estimation fine-tuning).

[0067] This two-stage training strategy greatly saves training time and computational resources, effectively avoids overfitting on small task datasets, and allows the same pre-trained encoder to be reused for a variety of different visual tasks, achieving high model reusability.

[0068] (2) Rate-Constrained Training for Model Compression In order to enable the model to be efficiently deployed on edge devices with limited computing and storage resources (such as drones and mobile phones), an advanced model compression technique, namely rate-constrained training (RCT), is further introduced during the training process.

[0069] The core of rate-constrained training is to add an additional regularization term to the standard task loss function (such as the mean squared error loss used for reconstruction). This term aims to penalize the parameter complexity of the model. The overall optimization objective becomes: (4) Among them, the rate regularization term The definition draws on the Exp-Golomb coding principle in information theory, and its form is: (5) Here, N is the total number of model parameters. It is a single parameter. , , These are preset hyperparameters. By minimizing this regularization term, the optimizer tends to push many parameters toward zero, resulting in a very sparse and highly compressible distribution of the final model parameters.

[0070] Experiments revealed that imposing such strong constraints throughout the entire pre-training phase may have a slight negative impact on the final accuracy of downstream tasks. To achieve the optimal balance between model compression ratio and task performance, this invention also proposes and employs a "Half-BackSlash Strategy": activation rate regularization is applied only in the first half of the pre-training phase (e.g., the first 40 out of 80 epochs), and turned off in the second half, allowing the model to focus on optimizing task accuracy. This constraint is completely ignored during the fine-tuning phase. Experiments demonstrate that this strategy can achieve extremely high model compression ratios (measured by the Effective Columbus Compression Ratio, EGCR) while maintaining state-of-the-art performance on downstream tasks with almost no loss.

[0071] In summary, the training process of the encoder and decoder models includes: Data preparation stage: Using publicly available video datasets from multiple scenes, and combining the basic principles of single-snapshot exposure and the Poisson-Gaussian signal-noise model, data pairs of 2D compressed measurements containing Poisson-Gaussian noise with arbitrary light intensity and video frame sequences are generated. For other downstream tasks, such as edge detection, monocular depth estimation, and semantic segmentation, triplet data of 2D compressed measurements containing Poisson-Gaussian noise with arbitrary light intensity, video frame sequences, and task labels are generated; Pre-training stage: The shared encoder and decoder are jointly optimized through a self-supervised learning framework and rate-constrained (parameter compression) training to reconstruct the original clean video frame sequence from the compressed measurements containing measurement noise; Fine-tuning stage: The parameters of the shared encoder model obtained in the pre-training stage are fixed, and only the task-specific decoder model is fine-tuned to adapt to the specific downstream computer vision task. In the fine-tuning stage, rate-constrained training is no longer applied. The input data is propagated in RAW compressed measurement domain form, avoiding the information loss associated with converting traditional high-speed imaging to RGB images and the additional high-power ISP (Image Signal Processing) pipeline. The 2D compressed measurement domain image is equivalent to a blurred noisy image to the human eye, thus achieving privacy protection. The output results are processed in a low-dimensional semantic form for specific visual tasks, which is both intuitive and avoids the privacy leakage that may be caused by reconstructing high-fidelity RGB frames.

[0072] This invention can be applied to various types of computer vision tasks. The application process can be summarized as follows: using a two-dimensional image sensor, a video sequence with spatiotemporally emphasized light is subjected to a single exposure using a pseudo-random binary mask to obtain the corresponding 2D compressed measurement value; using video reconstruction as a proxy task, a trained compression-denoising autoencoder model is used to extract latent spatiotemporal feature representations from the 2D compressed measurement values. The compression-denoising autoencoder model includes a shared encoder and at least one task-specific decoder. The shared encoder is used to extract the latent spatiotemporal feature representations from the 2D compressed measurement values; the latent spatiotemporal feature representations are input into the decoder to obtain the dynamic results of downstream computer vision tasks. Downstream computer vision tasks include, but are not limited to, edge detection, monocular depth estimation, and semantic segmentation.

[0073] To further verify the effectiveness of this invention, all models were developed and trained using the PyTorch framework on a single NVIDIA A100 (80GB) GPU. The optimizer used was Adam with an initial learning rate of 0.0001. To improve the model's generalization ability, data augmentation techniques such as random scaling and cropping, and random horizontal flipping were employed during training. The default compression sampling settings were: compression ratio Cr=8, and video frame spatial resolution of 128×128 pixels.

[0074] To comprehensively and rigorously evaluate the performance of the proposed CompDAE method, a series of detailed experiments were conducted, covering various computer vision tasks, challenging scenarios, and in-depth comparisons with state-of-the-art technologies. Figure 3 shows a comparison of edge detection results between CompDAE and DexiNed-f when APC ≤ 20; Figure 4 shows a comparison of semantic segmentation results of CompDAE, SegFormer, and the two-stage method DIC+SegFormer on the Cityscapes dataset; Figure 5 shows a comparison of edge detection results of the CompDAE modality and edge detection models and their corresponding two-stage methods on the SCI grayscale benchmark dataset; Figure 6 shows a comparison of depth estimation results of CompDAE, monocular depth estimation models, and their corresponding two-stage methods on the KITTI dataset; Figure 7 shows the edge detection performance of CompDAE on real SPAD512S ultra-low illumination data.

[0075] (1) Experimental setup Dataset: Pre-training data: Self-supervised pre-training was performed using the large-scale video segmentation dataset DAVIS2017. This dataset contains diverse scenes and motions, providing a rich data foundation for the model to learn general features.

[0076] Edge detection: Due to the lack of publicly available video edge detection benchmarks, six grayscale video datasets widely used in SCI research (Kobe, Traffic, Runner, etc.) were used, and the results generated by the state-of-the-art edge detection model DexiNed on high-definition original videos were adopted as high-quality ground truth labels.

[0077] Monocular depth estimation: The KITTI dataset from the field of autonomous driving is used. Utilizing the sparse LiDAR point cloud and corresponding RGB images provided by the dataset, a dense depth map is generated using depth completion techniques as the ground truth for both training and testing.

[0078] Semantic segmentation: The Cityscapes dataset was used, which provides fine-grained, pixel-level semantic annotations for urban scenes.

[0079] Real-world low-light data: To verify the effectiveness of the method in real-world physical environments, a SPAD512S single-photon avalanche diode camera was used to acquire video data in extremely dark conditions. This camera is capable of photon counting imaging, and its output data is the gold standard for verifying low-light performance.

[0080] Evaluation metrics: Edge detection: Standard Optimal Dataset Scale (ODS) and Optimal Image Scale (OIS) F1-scores are used. The higher the score, the better the edge detection result.

[0081] Depth estimation: Employs multiple standard metrics, including: absolute relative error (AbsRel), root mean square error (RMSE), logarithmic error (log10), and accuracy threshold (…). ).

[0082] Semantic segmentation: using average intersection-union ratio (mIoU) and pixel accuracy (PA).

[0083] Baseline Comparison: Traditional image-based methods: directly applying state-of-the-art (SOTA) models from various task domains to single-frame images extracted from videos. These include Canny, RCF, BDCN, DexiNed (for edge detection); MiDaS v3.1, Unidepth, Metric3D, and LightedDepth (for depth estimation).

[0084] Two-stage SCI pipeline: First, an advanced SCI video reconstruction algorithm (DIC) designed for low light is used to reconstruct compressed measurements into video frames. Then, the reconstructed video frames are input into the above-mentioned SOTA task model (e.g., forming combinations such as DIC+DexiNed, DIC+Unidepth, etc.).

[0085] (2) Core performance evaluation of robustness under ultra-low illumination conditions: Qualitative comparison: As shown in Figure 3, under conditions of extremely low average photon count (APC), the traditional SOTA edge detector DexiNed completely fails when processing noisy images, and its output is almost meaningless noise. In contrast, the CompDAE of this invention can still extract clear and coherent object edge structures directly from a single compressed measurement value even under such poor signal-to-noise ratio, demonstrating amazing noise resistance.

[0086] Table 1: Edge detection results on the grayscale benchmark video dataset

[0087] Quantitative Comparison (Edge Detection): In the comparison in Table 1, a striking result is that, under low-light conditions with APC=20, the CompDAE of this invention (ODS=0.672) not only far outperforms traditional methods running on equally noisy images, but also significantly outperforms all baseline methods, including DexiNed, running on pristine, clean, and noise-free images. This demonstrates the fundamental advantages of the end-to-end paradigm of this invention in utilizing spatiotemporal information and joint denoising-feature extraction.

[0088] Table 2: Monocular Depth Estimation Results on the KITTI Dataset

[0089] Quantitative Comparison (Depth Estimation): As shown in Table 2, with APC=20, CompDAE significantly outperforms state-of-the-art depth estimation algorithms such as Unidepth and MiDaS v3.1 across all evaluation metrics. More notably, while achieving superior performance, CompDAE has only 1 / 15 the number of model parameters as Unidepth, and its single-frame inference time is more than 25 times faster (2.60ms vs 63.90ms), fully demonstrating its significant efficiency advantage.

[0090] To demonstrate that the end-to-end paradigm of "direct identification from the compressed domain" in this invention is superior to the traditional two-stage paradigm of "reconstruction first, identification later", a comprehensive comparison was conducted.

[0091] Table 3: Edge detection results under different lighting conditions

[0092] Table 4: Monocular Depth Estimation Results (APC=20)

[0093] Edge Detection: As shown in Table 3, at all low illumination levels (APC=5, 10, 20) tested, CompDAE's ODS and OIS scores significantly outperformed all "DIC reconstruction + edge detector" combinations. For example, at APC=20, CompDAE (ODS=0.670) outperformed the best-performing two-stage combination DIC+DexiNed (ODS=0.546) by more than 0.12.

[0094] Depth estimation: The same trend was observed in the depth estimation task, see Table 4. CompDAE's end-to-end model performance comprehensively outperformed all two-stage combinations.

[0095] Experimental results show that the intermediate video reconstruction step is an information bottleneck, which can not only introduce artifacts and errors but also lose subtle information useful for downstream tasks in the original compressed measurements. The end-to-end method of this invention bypasses this step, enabling more efficient use of the original sensor data and thus achieving superior performance.

[0096] Furthermore, all experiments used the same pre-trained encoder portion of a unified model trained using a Half-BackSlash strategy. Experimental results show that by simply changing different lightweight decoders, this general encoder can successfully drive a variety of distinct visual tasks, such as edge detection, depth estimation, and semantic segmentation, achieving state-of-the-art (SOTA) performance in all of them, demonstrating that the latent features it learns are highly general and information-rich.

[0097] Meanwhile, as can be seen in Table 3, the performance of this unified model decreased very gradually as APC decreased from 20 to 10 and then to 5, demonstrating its strong robustness and generalization ability across different lighting conditions.

[0098] (3) Ablation Study In order to verify the rationality of the various design choices in the model, a series of ablation experiments were conducted. Except for Table 8, the results below are assumed to be without Backslash rate constraint training and uniform noise adaptive training, and the training of the model is assumed to be 20 cycles.

[0099] Table 5: Impact of Decoder Depth

[0100] Impact of Decoder Depth: The impact of the number of spatiotemporal Transformer modules (N) in the decoder on performance during the fine-tuning phase was investigated. As shown in Table 5, edge detection performance (ODS / OIS score) continuously improves as N increases from 0 to 2; however, a slight performance decrease occurs when N increases to 3. This indicates that a relatively lightweight decoder (N=2) is the most efficient choice, and deeper decoders do not provide additional benefits but instead increase computational costs.

[0101] Table 6: Impact of Snapshot Compression Frame Count

[0102] Compression Ratio (Cr) Impact: See Table 6 for the performance of training and testing models at different compression ratios (Cr=4, 8, 16). The results show that Cr=8 achieves the best performance balance when the training and testing Cr values ​​are consistent. Notably, when the training and testing Cr values ​​are mismatched (e.g., testing a model trained with Cr=8 on data with Cr=16), performance drops sharply, indicating that the features learned by the model are closely related to the specific compression coding method.

[0103] Table 7: The Influence of Non-zero Density in Mask Array

[0104] Effect of non-zero density in the mask: As shown in Table 7, we tested the proportion of non-zero elements in an 8×8 mask. ( The impact of adjusting (=0.3, 0.4, 0.5) on performance. The results show that adjusting within this range... The impact on the final edge detection performance is minimal, proving that the system of the present invention is insensitive to the specific pattern of the mask and has good robustness.

[0105] Table 8: The Influence of Different Training Strategies Under Different Illumination Conditions

[0106] Training Strategy Comparison: See Table 8 for a detailed comparison of the effects of different training strategies. Key conclusions include: 1) The "unified model" (trained on a wide range of data from 1 to 60 APCs) shows almost no performance loss across all APC test points compared to the "specific model" (trained only on a specific APC), demonstrating strong generalization ability; 2) Applying the full BackSlash strategy slightly impairs edge detection accuracy; 3) The proposed "Half-BackSlash" strategy is the optimal choice, achieving a considerable model compression ratio (EGCR of approximately 50%) while maintaining almost the same high performance as without compression.

[0107] (4) Real-world data verification: To ultimately verify the ability of this invention to meet the challenges of the real physical world without simulation data, CompDAE was deployed on real photon counting videos captured by a SPAD512S camera. As shown in Figure 7, even in real-world scenes with extremely poor lighting conditions (illuminance as low as 0.1 lux) and complex and variable physical noise, the model of this invention can still robustly detect the clear outline of moving vehicles. This successful practical application strongly demonstrates the practical value of the method of this invention and successfully bridges the gap between theoretical simulation and real-world application.

[0108] In summary, this invention achieves a significant breakthrough in the fields of computational imaging and computer vision through integrated system design and algorithmic innovation. Compared with existing technologies, it has the following main advantages: 1) Innovative compressed vision paradigm: This invention pioneers the "image-free visual intelligence" paradigm, completely abandoning the lengthy and inefficient "reconstruction-recognition" pipeline in traditional SCI. By performing end-to-end high-level visual task inference directly in the original compressed measurement domain, it fundamentally eliminates the problems of information loss, artifact introduction, error accumulation, and increased latency caused by intermediate reconstruction steps.

[0109] 2) Hardware Simplicity and Feasibility: By employing a fixed, small-size (8×8) pseudo-random mask design, this invention overcomes the core bottleneck of traditional SCI technology, which requires large-size dynamic masks at the frame level and is difficult to implement in hardware. This design minimizes the system's requirements for storage, bandwidth, and power consumption, paving the way for low-cost, low-power on-chip intelligent vision sensors.

[0110] 3) Excellent low-light robustness: This invention uses accurate Poisson-Gaussian physical noise modeling, combined with a unified noise adaptive training strategy, to enable the model to maintain excellent performance in ultra-low light environments with extremely low signal-to-noise ratios (APC≤20). Its performance far surpasses all traditional state-of-the-art methods based on full-frame image processing, solving a major pain point in this field.

[0111] 4) High-efficiency model and computation: The combination of rate-constrained training inspired by BackSlash (especially the Half-BackSlash strategy) and an asymmetric encoder-decoder architecture enables the model of this invention to have top-notch performance while having a very small model size and extremely fast inference speed (2.60ms / frame), which fully meets the needs of real-time edge computing applications.

[0112] 5) Cross-task versatility and flexibility: The modular design, combining a powerful shared encoder with a "plug-and-play" lightweight decoder, makes CompDAE a highly flexible and general-purpose vision platform. The same pre-trained model can be seamlessly switched, efficiently solving various vision tasks such as edge detection, depth estimation, and semantic segmentation.

[0113] 6) Inherent privacy protection: Since the system operates on aliased compressed measurement data that is invisible to the human visual system throughout the entire processing flow, and never generates or stores any identifiable high-fidelity RGB video frames, the present invention naturally possesses strong privacy protection characteristics, making it very suitable for application scenarios that are highly sensitive to privacy, such as public surveillance, medical imaging, and smart homes.

[0114] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0115] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0116] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0118] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. An end-to-end snapshot compression computer vision method based on pseudo-random mask array, comprising the following steps: Using a two-dimensional image sensor, a single exposure is performed on a video sequence that has undergone spatiotemporal light intensity modulation using a pseudo-random binary mask to obtain the corresponding 2D compressed measurement value. Using video reconstruction as a proxy task, a trained compression denoising autoencoder model is used to extract latent spatiotemporal feature representations from the 2D compressed measurements. The compression denoising autoencoder model includes a shared encoder and at least one task-specific decoder. The shared encoder is used to extract the latent spatiotemporal feature representations from the 2D compressed measurements. The latent spatiotemporal feature representation is input into the decoder to obtain dynamic results for downstream computer vision tasks.

2. The method according to claim 1, characterized in that, The modulation mask is generated by spatially repeating a binary submask with a size smaller than the video frame resolution. The submask has a size of m × m, where m is a positive integer less than or equal to 32.

3. The method according to claim 1, characterized in that, The downstream computer vision tasks include edge detection, monocular depth estimation, or semantic segmentation, and the pseudo-random binary mask is an 8×8 pseudo-random binary mask array.

4. The method according to claim 1, characterized in that, The compression-denoising autoencoder model is trained according to the following steps: using publicly available video datasets from multiple scenes, combining the basic principle of single snapshot exposure and the Poisson-Gaussian composite noise model, data pairs of 2D compressed measurements containing Poisson-Gaussian noise and video frame sequences with various light intensities are generated. A training set is constructed based on the set downstream task, reflecting the correspondence between 2D compressed measurements containing Poisson-Gaussian noise, video frame sequences, and task labels. The compression-denoising autoencoder model is pre-trained using the training set. During the pre-training stage, the shared encoder and the decoder are jointly trained through self-supervised learning and rate-constrained training. The parameters of the shared encoder obtained during the fixed pre-training phase are used to fine-tune the decoder only to adapt it to the set downstream computer vision task, and rate constraint training is no longer applied during the fine-tuning process.

5. The method according to claim 4, characterized in that, The Poisson-Gaussian composite noise model is modeled as follows: Among them, the Poisson noise component Gaussian noise component ， This represents the Poisson noise component in the t-th frame during a single exposure. This represents the additive white Gaussian noise component of a single-exposure snapshot. It is the t-th raw frame of the high-speed video. It is the mask corresponding to the modulation of the t-th frame.

6. The method according to claim 4, characterized in that, The overall loss function used in the pre-training of the compression-denoising autoencoder model is set as follows: in, It is a standard task loss function. It is a regularization term. The rate regularization term is set as follows: Where N is the total number of parameters in the compression-denoising autoencoder model. It is a single parameter. ， ， These are preset hyperparameters.

7. The method according to claim 6, characterized in that, The rate regularization term It only takes effect during the first 50% of the pre-training phase.

8. The method according to claim 1, characterized in that, The compression and denoising autoencoder model adopts an asymmetric architecture design, and the number of parameters and computational complexity of the shared encoder are higher than those of the decoder.

9. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images