High-efficiency and high-quality image reconstruction method and system for pulse camera

Abstract

A pulse camera-oriented high-efficiency and high-quality image reconstruction method and system belong to the technical field of high-speed imaging, and comprise the following steps: obtaining explicit pulse representation by an explicit pulse representation method based on internal statistics, obtaining implicit pulse representation by an implicit pulse representation method based on a neural network, and obtaining an implicit pulse representation by an implicit pulse representation method based on neural network; the pulse embedding features with more characterization force are obtained after the pulse embedding features are mixed; setting the pulse embedding features as bottom features of a feature pyramid to construct a multi-layer feature pyramid, and sequentially extracting multi-granularity feature representations from a plurality of continuous pulse substreams; symmetrically capturing interdependence between the intermediate features and the time context features by using a multi-head cross attention mechanism of a top pyramid, and obtaining interaction features; motion-intensity interaction is carried out progressively from coarse granularity to fine granularity, multiple groups of motion fields are estimated in a bottom pyramid, and joint refinement of space-time complementary information is realized. According to the invention, the precision of image reconstruction of the pulse camera is improved, and the demand of the model for computing resources is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of high-speed imaging technology, and in particular relates to a high-efficiency and high-quality image reconstruction method and system for pulse cameras. Background Art

[0002] High-speed imaging technology plays an increasingly critical role in cutting-edge fields such as autonomous driving, industrial monitoring, and robotics, thanks to its ability to accurately capture continuous changes in light intensity in a scene. These fields place extremely stringent requirements on real-time and accuracy, emphasizing that the system must be able to respond to dynamic changes quickly and accurately. However, traditional digital cameras face severe challenges in capturing high-speed moving objects. Not only do they need to rely on expensive professional sensors, but they also inevitably need to make a trade-off between frame rate and cost, which greatly hinders the popularization and further development of high-speed imaging technology.

[0003] In this context, the emergence of neuromorphic cameras has brought revolutionary breakthroughs in the field of high-speed imaging. In particular, neuromorphic visual sensors such as event cameras and pulse cameras, with their outstanding characteristics such as high temporal resolution, high dynamic range and low latency, make true high-speed imaging possible. Among them, the pulse camera developed by Professor Huang Tiejun's team at Peking University achieved a sampling frequency of up to 40,000 Hz by simulating the sampling mechanism of the biological visual system. Unlike event cameras that only record relative changes in light intensity, pulse cameras can encode absolute light intensity, and their pulse emission rate is proportional to the brightness of the scene. This unique capability makes pulse cameras show great potential in pixel-level tasks such as image reconstruction, depth estimation, semantic segmentation and optical flow estimation. However, the dense binary time series information recorded by pulse cameras has brought unprecedented challenges to traditional frame-based visual algorithms.

[0004] To address this problem, researchers have proposed a variety of pulse camera image reconstruction methods in recent years. Traditional methods mainly use temporal statistical characteristics and simulate the physiological mechanism of biological vision, but often need to make trade-offs when dealing with noise and motion blur, resulting in image reconstruction quality that is difficult to meet actual needs. With the rapid development of deep learning technology, pulse camera image reconstruction methods based on deep learning have emerged. With the powerful feature representation capabilities of the end-to-end learning framework, this type of method can more effectively mine potential spatiotemporal clues from the pulse stream, providing new ideas and possibilities for high-quality reconstruction of dynamic scenes. However, current deep learning-based methods usually adopt a two-stage paradigm of first estimating motion and then reconstructing the image when designing the network architecture. This is not only computationally inefficient, but also not conducive to further improving the accuracy of the reconstructed image. Artifacts are prone to appear at the edges of motion and local details, which brings bottlenecks to the system deployment of ultra-high-speed pulse cameras in real environments.

[0005] In summary, facing the urgent demand for high-speed imaging in cutting-edge fields such as autonomous driving, industrial monitoring and robotics, pulse camera image reconstruction, as a cornerstone task, has very important research significance.

[0006] Prior art 1: Spk2ImgNet: Learning to Reconstruct Dynamic Scene from Continuous Spike Stream, its main technical route is: based on end-to-end deep learning, a pulse camera image reconstruction network, called Spk2ImgNet, is designed for the first time. k , Spk2ImgNet inputs a set of k The continuous pulse frame S k ={S k+i},i=±1,±2,…, to exploit temporal correlations to generate high-quality reconstructions The overall network consists of three components: a spatially adaptive light inference subnetwork, a feature extraction subnetwork, and a deformable alignment-based image reconstruction subnetwork.

[0007] Prior art 2: Learning Temporal-Ordered Representation for Spike Streams Based on Discrete Wavelet Transforms, the main technical route is: In order to learn better pulse representation, it is proposed to use wavelet transform in time-frequency space to mine the temporal robust feature representation of pulses. A wavelet-guided pulse enhancement paradigm, referred to as WGSE, was developed, which includes three consecutive steps: multi-level wavelet transform, a learnable module based on CNN (Convolutional Neural Network) and inverse wavelet transform. Subsequently, the pulse representation is decoded into the target image through an additional image reconstruction module.

[0008] The shortcomings of the prior art are as follows:

[0009] (1) Traditional pulse camera image reconstruction methods utilize temporal statistical characteristics or simulate human physiological mechanisms. However, noise and motion blur inevitably bring about a difficult trade-off problem in the process of dynamic scene reconstruction, which often leads to poor image reconstruction quality.

[0010] (2) In the task of pulse camera image reconstruction, motion estimation and intensity recovery are essentially a "chicken and egg" problem: more accurate motion modeling will lead to better intermediate frame reconstruction, and vice versa. Existing deep learning-based technologies all adopt a two-stage paradigm of first estimating motion and then reconstructing images, which hinders the interaction and joint optimization of information in time and space, and is not conducive to improving the quality of pulse camera image reconstruction.

[0011] (3) The commonly used two-stage network design scheme makes the existing CNN-based models inevitably have high model complexity, low reasoning efficiency, and large computational complexity, which is not conducive to efficient deployment in practical applications. Summary of the invention

[0012] The purpose of the present invention is to provide a high-efficiency and high-quality image reconstruction method and system for pulse cameras. The present invention aims to break through the limitations of the existing technology and innovatively realize spatiotemporal joint learning in a single-stage paradigm, thereby significantly improving the accuracy and efficiency of image reconstruction from pulses, and making substantial contributions to technological progress in related fields. Through this innovative method, it is expected to promote the widespread deployment of high-speed imaging technology in practical applications and lay the foundation for the development of future intelligent systems.

[0013] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0014] The present invention provides a high-efficiency and high-quality image reconstruction method for a pulse camera, comprising the following steps:

[0015] Step 1: Mixed pulse embedding representation;

[0016] An explicit pulse representation method based on internal statistics is used to obtain an explicit pulse representation, and an implicit pulse representation method based on a neural network is used to obtain an implicit pulse representation. The explicit pulse representation and the implicit pulse representation are connected and mixed to obtain a pulse embedding feature.

[0017] Step 2: Hierarchical feature encoding;

[0018] The pulse embedding features are set as the bottom features of the feature pyramid, a multi-layer feature pyramid is constructed, and multi-granularity feature representations are extracted from multiple continuous pulse sub-streams in sequence;

[0019] Step 3: Symmetrical interactive attention block;

[0020] The multi-head cross-attention mechanism of the top-level pyramid is used to symmetrically capture the interdependencies between intermediate features and temporal context features and obtain interactive features.

[0021] Step 4: Spatiotemporal interactive decoding;

[0022] Motion-intensity interaction is performed progressively from coarse-grained to fine-grained to achieve the joint refinement of spatiotemporal complementary information. At the same time, a multi-motion field estimation block is constructed in the bottom pyramid to further enhance the spatiotemporal interaction capability of the model by estimating multiple sets of motion fields, thereby reconstructing a higher quality intensity image.

[0023] Furthermore, the specific implementation process of step one is:

[0024] For three consecutive pulse sub-streams of input Based on different time windows, multiple explicit pulse representations are obtained using the traditional playback-based image texture reconstruction method; at the same time, the pulse substreams are Input into the ResNet block for implicit modeling to obtain the corresponding implicit pulse representation; the obtained features are connected in the channel dimension and then mixed by 2D convolution to generate pulse embedding features

[0025] Furthermore, the specific implementation process of step 2 is as follows:

[0026] Construct an L-layer feature pyramid. In each layer of the feature pyramid, use a 3×3 two-dimensional convolution with a stride of 2 to perform feature downsampling, and then output the result through a residual block. A PReLU activation function is attached after each 3×3 two-dimensional convolution. The number of feature channels in the l-layer pyramid is C l ,The obtained multi-granularity feature representation promotes the subsequent coarse-to-fine inter-frame and intra-frame interaction learning.

[0027] Furthermore, the specific implementation process of step three is:

[0028] At the top pyramid level, the intermediate features As query vector query, temporal context features and They are used as key vector key / value vector value respectively to ensure symmetric interaction with query vector query; each component, namely query vector query, key vector key and value vector value, is linearly projected by application layer normalization, and the intermediate features are projected into two query vectors, namely and Temporal context features and are projected into two key vectors, namely and and two value vectors, namely and Then, attention-based bilateral correlations are calculated; finally, multi-head query vector-key vector feature interactions are performed in the channel rather than the spatial dimension to improve computational efficiency, and the final interaction features are obtained by aggregating local and non-local contextual information.

[0029] Furthermore, the calculation formula of the attention-based bilateral correlation is:

[0030]

[0031] in, represents the attention between the intermediate features and the temporal context features, α i represents a learnable scaling parameter used to control the size of the dot product, Represents the key vector The transpose of .

[0032] Furthermore, the interactive feature Its mathematical expression is:

[0033]

[0034] Among them, H and W represent the height and width of the image respectively, L represents the number of layers of the pyramid, and C L represents the feature dimension of the top pyramid, represents the attention between the intermediate feature and its previous feature, Indicates the attention between the intermediate feature and its subsequent feature, [ ] indicates the connection operation between channels, and Conv indicates the point-by-point convolution layer; the internal intermediate feature Preserved through skip connections.

[0035] Furthermore, the specific implementation process of step 4 is as follows:

[0036] ① Movement-intensity interaction block;

[0037] In the top pyramid, the interaction feature χ L This is fed into the motion-intensity interaction block, which simultaneously estimates the motion field and synthesize intermediate intensity frames using a dense network block At the L-1 level of the feature pyramid, the motion field and Upsampled to inversely transform temporal context features Thus, the spatiotemporal information around timestamps t0 and t2 is aligned to the intermediate timestamp t1 to complete the inter-frame feature alignment; the bilinear upsampled intermediate intensity frame The corresponding intermediate features After the connection, a two-dimensional convolution is performed to synthesize the upsampled frame and intermediate features Implement intra-frame feature filtering; further introduce a dense network block to merge the complementary advantages of feature warping and frame synthesis; realize information sharing and mutual collaboration of spatiotemporal features by progressively refining the motion-intensity interaction block from the Lth layer to the 1st layer of the pyramid;

[0038] ②Multi-motion field estimation block;

[0039] Inspired by the current popular video interpolation method, the output channels are enlarged in the bottom pyramid to estimate the motion field of group G This will further enhance the model's spatial and temporal interaction capabilities.

[0040] Furthermore, in step ①, during the inter-frame feature alignment process, the mathematical expression of feature distortion is:

[0041]

[0042] in, represents the distorted feature candidate at the L-1 layer of the feature pyramid, w represents the feature distortion operation, and ↑ represents the upsampling variable;

[0043] The mathematical expression of the intra-frame feature filtering is:

[0044]

[0045] in, Represents the intermediate features synthesized at the L-1 layer of the feature pyramid, Conv2D represents the 2D convolution operation, represents the intermediate feature, represents the upsampled frame;

[0046] The mathematical expression for combining the complementary advantages of feature warping and frame synthesis through a dense network block is:

[0047]

[0048] in, represents the image frame reconstructed at the L-1 layer of the feature pyramid, and Both represent the motion field between the middle moment of the feature pyramid L-1 layer and the moments before and after it. DenseBlock represents a dense network block. and Both represent the motion field of the upsampled feature pyramid layer L.

[0049] Furthermore, it also includes the step of network training, and its specific implementation process is as follows:

[0050] Step S1: Obtain training data set;

[0051] The synthetic SREDS dataset is used for model training. The SREDS dataset is divided into 240 training scenes and 30 test scenes. Each scene contains 24 consecutive frames and there is a corresponding pulse stream around the center of each frame. The spatial resolution is 1280×720. During the training process, each scene is cropped to 96×96 without overlapping, resulting in a total of 21,840 image patches.

[0052] Step S2: basic settings;

[0053] Step S2.1: The time length of the input pulse sub-stream is 60; the number of layers of the pyramid is set to 5;

[0054] Step S2.2: In the hybrid pulse embedding representation module, for each pulse substream, a 5-channel TFP-based explicit representation with a scaling step of 4 and an 11-channel ResNet-based implicit representation are constructed, so the number of feature channels of the feature pyramid is 16, 24, 32, 64 and 96 respectively; finally, 3 sets of motion fields are estimated in the bottom pyramid;

[0055] Step S3: model training;

[0056] Step S3.1: Using reconstruction loss Perceived loss and multi-scale consistency loss As the total loss function To train the model, that is:

[0057]

[0058] Reconstruction losses Measure the l1 distance between the predicted image and the true value image, that is:

[0059]

[0060] Among them, H and W represent the height and width of the image respectively. represents the estimated intermediate image corresponding to the intermediate time t1, Represents the true intermediate image;

[0061] Introducing perceptual loss to alleviate the blurring effect and preserve more details, namely:

[0062]

[0063] Among them, φ vgg represents the feature extractor of the pre-trained VGG network. In addition, a multi-scale consistency loss is proposed Multi-scale intermediate intensity frames synthesized from the 2nd to the Lth level pyramid are forced Keep consistent with the true value image; the e1 distance of the multi-scale consistency loss is expressed as:

[0064]

[0065] Step S3.2: training data enhancement;

[0066] During the training phase, the input pulse signal and the true value image are randomly flipped vertically and randomly rotated by a certain angle to perform data augmentation;

[0067] Step S3.3: The code relies on the PyTorch framework for implementation. The model is trained using the Adam optimizer and is trained and tested on a single NVIDIA RTX 3090 GPU.

[0068] The present invention provides a high-efficiency and high-quality image reconstruction system for pulse cameras, which is used to implement the high-efficiency and high-quality image reconstruction method for pulse cameras. The system mainly includes:

[0069] A hybrid pulse embedding representation module is used to obtain multiple explicit pulse representations based on an explicit pulse representation method based on internal statistics, and to obtain implicit pulse representations based on an implicit pulse representation method based on a neural network, and to obtain pulse embedding features after connecting and mixing multiple explicit pulse representations and implicit pulse representations;

[0070] Hierarchical feature encoder, used to obtain pyramid features of different granularities;

[0071] Symmetrical interactive attention block, which is used to utilize the multi-head cross attention mechanism of the top-level pyramid to symmetrically capture the interdependencies between intermediate features and temporal context features and obtain interactive features;

[0072] A spatiotemporal interaction decoder is used to progressively perform motion-intensity interaction from coarse-grained to fine-grained to achieve joint refinement of spatiotemporal complementary information.

[0073] Compared with the prior art, the present invention has the following advantages:

[0074] (1) The single-stage spatiotemporal interactive network architecture proposed in the present invention improves the image reconstruction quality of pulse cameras by jointly learning complementary motion and intensity information. The existing two-stage architecture limits the interaction and joint optimization of information in time and space, while the single-stage spatiotemporal interactive network architecture proposed in the present invention can extract more representative features by integrating motion information and absolute light intensity in time series data. This joint learning strategy not only improves the accuracy of reconstruction, but also reduces the model's demand for computing resources, making it more efficient.

[0075] (2) The hybrid pulse embedding representation module designed by the present invention combines explicit and implicit representation methods to better combine the strong interpretability of explicit representation with the strong expressive power of implicit representation. Among them, the explicit representation provides a relatively stable feature basis, while the implicit representation further mines the potential patterns of features through CNN, enhancing the adaptability to complex scenes. This hybrid pulse embedding representation design effectively combines the advantages of explicit and implicit pulse representations, not only injecting stronger spatiotemporal feature expression capabilities into the pulse sub-stream, but also significantly improving the robustness of the overall system, so as to achieve more robust feature learning under different motion conditions.

[0076] (3) The present invention adopts a hierarchical pyramid feature encoder and a spatiotemporal interactive decoder to realize the progressive refinement and collaborative optimization of spatiotemporal features from coarse to fine, thereby achieving the purpose of improving the accuracy of image reconstruction while reducing the computational cost.

[0077] (4) The symmetrical interactive attention block designed by the present invention effectively enhances the integration of spatiotemporal features by capturing the bidirectional correlation between the intermediate frame and its temporal context features. This symmetrical design ensures that the attention mechanism can pay equal attention to the feature correlation of different contexts, which helps to better align inter-frame and intra-frame features during the reconstruction process, and also helps to reduce the complexity of the model.

[0078] (5) The present invention enhances the bilateral correlation modeling and context detail compensation of the pulse signal through the symmetrical interactive attention block and the multi-motion field estimation block, ensuring that the reconstruction quality can still meet the requirements in the case of motion blur and occlusion.

[0079] (6) The present invention realizes inter-frame feature alignment and intra-frame feature filtering through a joint motion-intensity learning architecture, thereby improving the image reconstruction quality and computational efficiency of the pulse camera in complex dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 A flowchart of a high-efficiency and high-quality image reconstruction method for a pulse camera provided by the present invention.

[0081] Figure 2 The architecture diagram of the symmetric interactive attention block.

[0082] Figure 3 Schematic diagram of the motion-intensity interaction block.

[0083] Figure 4 Architecture diagram for the multi-motion field estimation block.

[0084] Figure 5 Qualitative comparison results on the synthetic dataset “SREDS”.

[0085] Figure 6 Qualitative comparison results on the real dataset "momVidarReal2021".

[0086] Figure 7 Qualitative comparison results on the real dataset “recVidarReal2019”.

[0087] Figure 8 The figure is a qualitative image reconstruction comparison result based on the real pulse data collected by the present invention.

[0088] Fig. 9 It is the high frame rate video reconstruction result based on the real pulse data collected by the present invention. DETAILED DESCRIPTION

[0089] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0090] In a first aspect, the present invention provides a high-efficiency and high-quality image reconstruction system for pulse cameras.

[0091] See also Figure 1 To illustrate, the present invention is a high-efficiency and high-quality image reconstruction system for pulse cameras, through which the mutual cooperation and mutual promotion of temporal motion estimation and spatial intensity recovery can be achieved. It mainly uses an efficient single-stage spatiotemporal interaction network to perform pulse camera image reconstruction tasks. The system mainly includes the following modules: a hybrid pulse embedding representation module, a hierarchical feature encoder, a symmetric interactive attention block and a spatiotemporal interaction decoder.

[0092] Specifically, the present invention first designs a simple but effective hybrid pulse embedding representation module, which combines the common advantages of explicit pulse representation based on internal statistics (with better certainty and interpretability) and implicit pulse representation based on neural network (with stronger expressive power); then the hybrid pulse embedding representation obtained by the hybrid pulse embedding representation module is input into the hierarchical feature encoder to obtain pyramid features of different granularities; then, the present invention also proposes a spatiotemporal interaction decoder, which gradually performs motion-intensity interaction from coarse granularity to fine granularity to achieve joint refinement of spatiotemporal complementary information. In particular, inter-frame feature alignment and intra-frame feature filtering can be performed simultaneously to achieve the purpose of mutual promotion. The former, i.e., inter-frame feature alignment, mainly focuses on temporal motion clues to complete distortion-based feature alignment, while the latter, i.e., intra-frame feature filtering, gradually maintains purer image features through synthesis. In addition, a symmetric interactive attention block is integrated in the top pyramid, and a multi-motion field estimation block is introduced in the bottom pyramid to further enhance the spatiotemporal interaction capability of the overall network.

[0093] In a second aspect, the present invention provides a high-efficiency and high-quality image reconstruction method for pulse cameras.

[0094] See also Figure 1 To illustrate, the present invention provides a high-efficiency and high-quality image reconstruction method for pulse cameras, and its specific implementation process is as follows:

[0095] (1) Hybrid pulse embedding representation;

[0096] For three consecutive pulse sub-streams of input The present invention uses a hybrid pulse embedding representation module. First, multiple explicit pulse representations are obtained based on different time windows using the traditional TFP (Texture from Playback) method. This is because short windows can provide better details but bring noise, while long windows can suppress noise but easily introduce blur. At the same time, the pulse substream is also The input is fed into the ResNet (Residual Network) block for implicit modeling to obtain multiple implicit pulse representations. Finally, these obtained features are concatenated in the channel dimension and then a 2D convolution is performed to generate the pulse embedding feature.

[0097] (2) Hierarchical feature encoding;

[0098] After obtaining the pulse embedding features corresponding to three consecutive pulse sub-streams and Afterwards, they are set as the bottom features of the feature pyramid and On this basis, a hierarchical feature encoder is designed to construct an L-layer feature pyramid to extract multi-granularity feature representations from the pulse substream and Note that the network parameters are shared among the three consecutive spike substreams. At each level of the feature pyramid, a 3×3 2D convolution with a stride of 2 is used for feature downsampling, followed by a residual block. In addition, a PReLU activation function is attached after each 3×3 2D convolution. The number of feature channels at the lth level of the pyramid is C l Next, the multi-granularity feature representation and Coarse-to-fine inter-frame and intra-frame interactive learning will be promoted.

[0099] (3) Symmetrical interactive attention blocks;

[0100] ① The symmetrical interactive attention block uses the multi-head cross attention mechanism of the top pyramid to symmetrically capture the intermediate features and temporal context features In particular, the intermediate features As the query vector query, and the temporal context feature and They are used as key vector key / value vector value respectively to ensure symmetrical interaction with query vector query. The query vector query is used to determine the relevance of the current input in the multi-head cross attention mechanism when allocating attention; the key vector key is used to match the query vector in the multi-head cross attention mechanism to calculate the attention weight; the value vector value is used in the multi-head cross attention mechanism to perform weighted summation based on the calculated attention weights and output the final context representation.

[0101] ② If Figure 2 As shown in Figure 1, each component (i.e., query vector query, key vector key, and value vector value) is linearly projected by applying layer normalization. In this way, the intermediate features are projected into two query vectors, namely and At the same time, temporal context features and are projected into two key vectors, namely and and two value vectors, namely and Then the attention-based bilateral correlation is calculated. The attention-based bilateral correlation can be calculated symmetrically as follows:

[0102]

[0103] in, represents the attention between the intermediate features and the temporal context features, α i represents a learnable scaling parameter used to control the size of the dot product, Represents the key vector The multi-head query vector-key vector feature interaction is performed on the channel rather than the spatial dimension, which can effectively improve the computational efficiency because the linear complexity replaces the quadratic complexity. By aggregating local and non-local context information, the final interaction feature It can be expressed as:

[0104]

[0105] Among them, H and W represent the height and width of the image respectively, L represents the number of layers of the pyramid, and C L represents the feature dimension of the top pyramid, represents the attention between the intermediate feature and its previous feature, represents the attention between the intermediate feature and its subsequent feature, [ ] represents the connection operation between channels, and Conv is a point-by-point convolution layer. Note that the intrinsic intermediate feature Preserved through skip connections.

[0106] (4) Spatiotemporal interactive decoding;

[0107] By cascading multiple motion-intensity interaction blocks from coarse-grained to fine-grained, intermediate image frames can be decoded progressively. In addition, in the bottom pyramid, the present invention also proposes predicting multiple sets of motion fields to aggregate more comprehensive temporal contextual complementary information, which helps to further improve the image reconstruction quality.

[0108] ① Movement-intensity interaction block;

[0109] like Figure 3 As shown, in the top pyramid, the interaction feature χ L This is fed into the motion-intensity interaction block, which simultaneously estimates the motion field and synthesize intermediate intensity frames using a dense network block Then, at the L-1 level of the feature pyramid, the motion field and Upsampled to inversely transform temporal context features Thus, the spatiotemporal information around timestamps t0 and t2 is aligned to the intermediate timestamp t1. This process is called inter-frame feature alignment, where the mathematical expression of feature distortion is:

[0110]

[0111] in, represents the distorted feature candidates at level L-1 of the feature pyramid, Represents the feature warping operation, and ↑ represents the upsampling variable. At the same time, bilinear upsampling of the intermediate intensity frame Then compare it with the corresponding intermediate feature concatenate, followed by a 2D convolution to reduce the impact of pulse fluctuations. This process essentially synthesizes upsampled frames and intermediate features The intra-frame feature filtering is implemented, and the mathematical expression of the process is:

[0112]

[0113] in, Represents the intermediate features synthesized at the L-1 layer of the feature pyramid, Conv2D represents the 2D convolution operation, represents the intermediate feature, Represents an upsampled frame.

[0114] Note that motion-based feature warping is more effective in handling significant pixel shifts, but is less robust under occlusion. Conversely, frame synthesis-based methods show better robustness to occlusion and inconsistent brightness, but may affect image quality in short-time pulse substreams. To this end, a dense network block is further introduced to merge the complementary advantages of feature warping and frame synthesis, namely:

[0115]

[0116] in, represents the image frame reconstructed at the L-1 layer of the feature pyramid, and represents the motion field between the middle moment of the feature pyramid L-1 layer and the moments before and after it, DenseBlock represents a dense network block, and Both represent the motion field of the upsampled feature pyramid layer L.

[0117] Therefore, by progressively refining the motion-intensity interaction blocks from the Lth level to the 1st level of the pyramid, information sharing and mutual collaboration of spatiotemporal features can be achieved.

[0118] ②Multi-motion field estimation block;

[0119] Multiple motion field estimation has been shown to be an effective strategy to improve reconstruction quality in video interpolation tasks. We simply upscale the output channels in the bottom pyramid to estimate G sets of motion fields. Therefore, G sets of warped feature candidates can improve feature diversity at full resolution, e.g. Figure 4 This helps to compensate for the additional details when a single set of motion fields has local inaccuracies, thereby enhancing spatiotemporal interactions.

[0120] (5) In addition, the present invention uses synthetic data to train each model in the single-stage spatiotemporal interaction network, and the specific training process is as follows:

[0121] Step S1: Obtain training data set;

[0122] The present invention uses a synthetic SREDS dataset for model training, which is synthesized based on the REDS dataset. The SREDS dataset is divided into 240 training scenes and 30 test scenes. Each scene contains 24 consecutive frames, and there is a corresponding pulse stream around the center of each frame, where N=20 and the spatial resolution is 1280×720. During the training process, each scene is cropped to 96×96 without overlapping, resulting in a total of 21,840 image blocks.

[0123] Step S2: basic settings;

[0124] Step S2.1: The time length of the input pulse substream is 60, that is, N = 20. The number of pyramid layers is set to 5, that is, L = 5. Reconstruction loss Perceived loss and multi-scale consistency loss Used together to train the model.

[0125] Step S2.2: In the hybrid spike embedding representation module, for each spike substream, a 5-channel TFP-based explicit representation with a scaling step of 4 and an 11-channel ResNet-based implicit representation are constructed. Therefore, the number of feature channels of the feature pyramid is 16, 24, 32, 64, and 96, respectively. In addition, 3 sets of motion fields are estimated in the bottom pyramid, i.e., G = 3.

[0126] Step S3: model training;

[0127] Step S3.1: Using reconstruction loss Perceived loss and multi-scale consistency loss As the total loss function To train the model, that is:

[0128]

[0129] Reconstruction losses Measure the l1 distance between the predicted image and the true value image, that is:

[0130]

[0131] Among them, H and W represent the height and width of the image respectively. represents the estimated intermediate image corresponding to the intermediate time t1, Represents the true intermediate image.

[0132] We also introduce perceptual loss to alleviate the blurring effect and preserve more details, namely:

[0133]

[0134] Among them, φ vgg Represents the feature extractor of the pre-trained VGG (Visual Geometry Group) network. In addition, a multi-scale consistency loss is proposed Multi-scale intermediate intensity frames synthesized from the 2nd to the Lth level pyramid are forced Keep consistent with the true value image. The l1 distance of multi-scale consistency loss can be expressed as:

[0135]

[0136] Step S3.2: training data augmentation;

[0137] In the process of training the model, in order to improve the generalization ability and performance of the model, data augmentation technology is used. Specifically, during the training stage, the input pulse signal and true value image are randomly flipped vertically, and they are also randomly rotated by a certain angle, such as 90 degrees, 180 degrees or 270 degrees, to increase the diversity and richness of the training data.

[0138] Step S3.3: The code relies on the PyTorch framework for implementation. The model is trained using the Adam optimizer for 150 epochs with a batch size of 8. The initial learning rate is 0.0001 and decays to 0.7 times the previous rate every 50 epochs. The model is trained and tested on a single NVIDIA RTX 3090 GPU.

[0139] In order to fully test the potential of the single-stage spatiotemporal interaction network, the performance of the model is evaluated using real pulse data collected by the latest generation of megapixel pulse cameras. At the same time, as a comparative example, the performance of the model is simultaneously evaluated using the publicly available "momVidarReal2021" and "recVidarReal2019" datasets based on the first generation of 100,000-pixel pulse cameras. Among them, the megapixel pulse camera has a resolution of 1000 (H) × 1000 (W), a frame rate of 40,000fps, a pixel size of 17μm × 17μm, a data output rate of 40Gbps, and a chip process of 110nm FSI process. The continuous pulse substream captured by the megapixel pulse camera is divided into multiple segments in the time dimension, and then input into the trained neural network to obtain the final high-quality image reconstruction result. 100,000-pixel pulse camera, resolution 400(H)×250(W), frame rate 20,000fps, pixel size 20μm×20μm, data output rate 2Gbps, chip technology adopts 110nm FSI technology, including high-speed motion scenes of objects and cameras, the continuous pulse sub-stream captured by the 100,000-pixel pulse camera is divided into multiple segments in the time dimension, and then input into the trained neural network to obtain the final high-quality image reconstruction result.

[0140] The qualitative comparison results obtained by the present invention are as follows Figures 5 to 9 As shown. Figures 5 to 9It can be observed that the images reconstructed by the method of the present invention are more pleasing in visual perception and have higher realism, especially at the edges of fast-moving objects. For example, in cases involving pedestrians, footballs, roulette wheels, and high-speed trains, the method of the present invention achieves sharper edges, less noise, and fewer blur and aliasing artifacts. In addition, the moment of a water splash explosion and a high-speed rotating fan can also be faithfully recorded and reproduced. It is worth noting that the method of the present invention also ensures efficient reconstruction of image frames, which further highlights its potential in practical applications.

[0141] The extensive experimental results of the above-mentioned synthetic and actual captured data show that the method of the present invention has significant advantages over current methods in terms of model complexity and reconstruction accuracy. Compared with traditional methods and methods based on pulse neural networks, the reconstruction accuracy on synthetic and real data sets is much better than the current most advanced methods, and the image reconstruction quality has been greatly improved. Compared with the CNN-based pulse camera image reconstruction method, the method of the present invention is an average of 11 times faster than Spk2ImgNet (CVPR 2021) and 5 times faster than WGSE (AAAI 2023), and the PSNR of the reconstructed image is improved by more than 1dB, which greatly enhances the deployment and use capabilities of ultra-high-speed pulse cameras in real application scenarios.

[0142] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A high-efficiency and high-quality image reconstruction method for pulse cameras, characterized in that: The following steps are involved: Step 1: Mixed pulse embedding representation; An explicit pulse representation method based on internal statistics is used to obtain an explicit pulse representation, and an implicit pulse representation method based on a neural network is used to obtain an implicit pulse representation. The explicit pulse representation and the implicit pulse representation are connected and mixed to obtain a pulse embedding feature. Step 2: Hierarchical feature encoding; The pulse embedding features are set as the bottom features of the feature pyramid, a multi-layer feature pyramid is constructed, and multi-granularity feature representations are extracted from multiple continuous pulse sub-streams in sequence; Step 3: Symmetrical interactive attention block; The multi-head cross-attention mechanism of the top-level pyramid is used to symmetrically capture the interdependencies between intermediate features and temporal context features and obtain interactive features. Step 4: Spatiotemporal interactive decoding; Motion-intensity interaction is performed progressively from coarse-grained to fine-grained to achieve the joint refinement of spatiotemporal complementary information. At the same time, a multi-motion field estimation block is constructed in the bottom pyramid to further enhance the spatiotemporal interaction capability of the model by estimating multiple sets of motion fields, thereby reconstructing a higher quality intensity image.

2. A high-efficiency and high-quality image reconstruction method for pulse cameras according to claim 1, characterized in that: The specific implementation process of step one is: For three consecutive pulse sub-streams of input Based on different time windows, multiple explicit pulse representations are obtained using the traditional playback-based image texture reconstruction method; at the same time, the pulse substreams are Input into the ResNet block for implicit modeling to obtain the corresponding implicit pulse representation; the obtained features are connected in the channel dimension and then mixed by 2D convolution to generate pulse embedding features 3. The high-efficiency and high-quality image reconstruction method for pulse cameras according to claim 1, characterized in that: The specific implementation process of step 2 is as follows: Construct an L-layer feature pyramid. In each layer of the feature pyramid, use a 3×3 two-dimensional convolution with a stride of 2 to perform feature downsampling, and then output the result through a residual block. A PReLU activation function is attached after each 3×3 two-dimensional convolution. The number of feature channels in the l-layer pyramid is C l ,The obtained multi-granularity feature representation promotes the subsequent coarse-to-fine inter-frame and intra-frame interaction learning.

4. The high-efficiency and high-quality image reconstruction method for pulse cameras according to claim 1, characterized in that: The specific implementation process of step three is as follows: At the top pyramid level, the intermediate features As query vector query, temporal context features and They are used as key vector key / value vector value respectively to ensure symmetric interaction with query vector query; each component, namely query vector query, key vector key and value vector value, is linearly projected by application layer normalization, and the intermediate features are projected into two query vectors, namely and Temporal context features and are projected into two key vectors, namely and and two value vectors, namely and Then, attention-based bilateral correlations are calculated; finally, multi-head query vector-key vector feature interactions are performed in the channel rather than the spatial dimension to improve computational efficiency, and the final interaction features are obtained by aggregating local and non-local contextual information.

5. The high-efficiency and high-quality image reconstruction method for pulse cameras according to claim 4, characterized in that: The calculation formula of the attention-based bilateral correlation is: in, represents the attention between the intermediate features and the temporal context features, α i represents a learnable scaling parameter used to control the size of the dot product, Represents the key vector The transpose of .

6. The high-efficiency and high-quality image reconstruction method for pulse cameras according to claim 4, characterized in that: The interactive features Its mathematical expression is: Among them, H and W represent the height and width of the image respectively, L represents the number of layers of the pyramid, and C L represents the feature dimension of the top pyramid, represents the attention between the intermediate feature and its previous feature, Indicates the attention between the intermediate feature and its next feature, [] indicates the connection operation between channels, and Conv indicates the point-by-point convolution layer; the internal intermediate feature Preserved through skip connections.

7. The high-efficiency and high-quality image reconstruction method for pulse cameras according to claim 1, characterized in that: The specific implementation process of step 4 is as follows: ① Movement-intensity interaction block; In the top pyramid, the interaction feature χ L This is fed into the motion-intensity interaction block, which simultaneously estimates the motion field and synthesize intermediate intensity frames using a dense network block At the L-1 level of the feature pyramid, the motion field and Upsampled to inversely transform temporal context features Thus, the spatiotemporal information around timestamps t0 and t2 is registered to the middle timestamp t1 to complete the inter-frame feature alignment; Bilinearly upsample the intermediate intensity frames The corresponding intermediate features After the connection, a two-dimensional convolution is performed to synthesize the upsampled frame and intermediate features Implement intra-frame feature filtering; further introduce a dense network block to merge the complementary advantages of feature warping and frame synthesis; realize information sharing and mutual collaboration of spatiotemporal features by progressively refining the motion-intensity interaction block from the Lth layer to the 1st layer of the pyramid; ②Multi-motion field estimation block; Inspired by the current popular video interpolation method, the output channels are enlarged in the bottom pyramid to estimate the motion field of group G This will further enhance the model's spatial and temporal interaction capabilities.

8. The high-efficiency and high-quality image reconstruction method for pulse cameras according to claim 7, characterized in that: In step ①, during the inter-frame feature alignment process, the mathematical expression of feature distortion is: in, represents the distorted feature candidates at level L-1 of the feature pyramid, represents the feature distortion operation, ↑ represents the upsampling variable; The mathematical expression of the intra-frame feature filtering is: in, Represents the intermediate features synthesized at the L-1 layer of the feature pyramid, Conv2D represents the 2D convolution operation, represents the intermediate feature, represents the upsampled frame; The mathematical expression for combining the complementary advantages of feature warping and frame synthesis through a dense network block is: in, represents the image frame reconstructed at the L-1 layer of the feature pyramid, and Both represent the motion field between the middle moment of the feature pyramid L-1 layer and the moments before and after it. DenseBlock represents a dense network block. and Both represent the motion field of the upsampled feature pyramid layer L.

9. The high-efficiency and high-quality image reconstruction method for pulse cameras according to claim 1, characterized in that: It also includes the steps of network training, and its specific implementation process is: Step S1: Obtain training data set; The synthetic SREDS dataset is used for model training. The SREDS dataset is divided into 240 training scenes and 30 test scenes. Each scene contains 24 consecutive frames and there is a corresponding pulse stream around the center of each frame. The spatial resolution is 1280×720. During the training process, each scene is cropped to 96×96 without overlapping, resulting in a total of 21,840 image patches. Step S2: basic settings; Step S2.1: The time length of the input pulse sub-stream is 60; the number of layers of the pyramid is set to 5; Step S2.2: In the hybrid pulse embedding representation module, for each pulse substream, a 5-channel TFP-based explicit representation with a scaling step of 4 and an 11-channel ResNet-based implicit representation are constructed, so the number of feature channels of the feature pyramid is 16, 24, 32, 64 and 96 respectively; finally, 3 sets of motion fields are estimated in the bottom pyramid; Step S3: model training; Step S3.1: Using reconstruction loss Perceived loss and multi-scale consistency loss As the total loss function To train the model, that is: Reconstruction losses Measure the l1 distance between the predicted image and the real image, that is: Among them, H and W represent the height and width of the image respectively. represents the estimated intermediate image corresponding to the intermediate time t1, Represents the true intermediate image; Introducing perceptual loss to alleviate the blurring effect and preserve more details, namely: Among them, φ vgg represents the feature extractor of the pre-trained VGG network. In addition, a multi-scale consistency loss is proposed Multi-scale intermediate intensity frames synthesized from the 2nd to the Lth level pyramid are forced Keep consistent with the true value image; the l1 distance of multi-scale consistency loss is expressed as: Step S3.2: training data augmentation; During the training phase, the input pulse signal and the true value image are randomly flipped vertically and randomly rotated by a certain angle to perform data augmentation; Step S3.3: The code relies on the PyTorch framework for implementation. The model is trained using the Adam optimizer and is trained and tested on a single NVIDIA RTX 3090 GPU.

10. A high-efficiency and high-quality image reconstruction system for pulse cameras, characterized in that: The system is used to implement a high-efficiency and high-quality image reconstruction method for a pulse camera as described in any one of claims 1 to 9, comprising: A hybrid pulse embedding representation module is used to obtain multiple explicit pulse representations based on an explicit pulse representation method based on internal statistics, and to obtain implicit pulse representations based on an implicit pulse representation method based on a neural network, and to obtain pulse embedding features after connecting and mixing multiple explicit pulse representations and implicit pulse representations; Hierarchical feature encoder, used to obtain pyramid features of different granularities; Symmetrical interactive attention block, which is used to utilize the multi-head cross attention mechanism of the top-level pyramid to symmetrically capture the interdependencies between intermediate features and temporal context features and obtain interactive features; A spatiotemporal interaction decoder is used to progressively perform motion-intensity interaction from coarse-grained to fine-grained to achieve joint refinement of spatiotemporal complementary information.

Images