Video reconstruction method and system based on a pulsed camera and a rolling shutter color camera

Abstract

The present application relates to a kind of video reconstruction methods based on the fusion of impulse camera and rolling shutter color camera.The method comprises: collecting the impulse stream data and rolling shutter color image of the same target scene;Virtual rolling shutter pulse frame time alignment with rolling shutter color image is constructed by impulse stream data, and virtual global shutter pulse frame corresponding to each target reconstruction time is constructed by impulse stream data;Virtual global shutter pulse frame and virtual rolling shutter pulse frame are input into optical flow estimation module, and the initial global shutter color image corresponding to each target reconstruction time is obtained;Each initial global shutter color image and corresponding virtual global shutter pulse frame are input into reconstruction module, and the high frame rate color video sequence of target scene is obtained.In the present application, rolling shutter color image provides multi-bit luminance information, and impulse stream provides supplementary dynamic timing details in extremely bright and extremely dark change scene, so that the high frame rate color video sequence obtained has good high dynamic range.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a video reconstruction method and system based on a pulse camera and a rolling shutter color camera. Background Technology

[0002] With the development of applications such as high-speed visual perception, robot perception, intelligent driving, industrial inspection, and scientific imaging, imaging systems not only need to acquire spatial texture information of the scene, but also need to stably output high-quality video under conditions of high-speed motion, drastic changes in brightness, occlusion, and complex dynamics. Traditional frame cameras typically perform integral imaging of the entire image through a fixed exposure time. Although they can directly output multi-bit images with color information, they are prone to motion blur in high-speed motion scenes. Furthermore, due to limitations in sensor readout speed and exposure mechanisms, it is difficult to simultaneously achieve both HFR (High Frame Rate) and HDR (High Dynamic Range).

[0003] In recent years, pulse cameras inspired by biological visual mechanisms have gradually become an important direction in the field of high-speed vision. In this type of sensor, each pixel independently accumulates incident photons, and when the accumulated voltage reaches a preset threshold, it outputs a 1-bit pulse signal. Therefore, it has an extremely high time sampling rate, typically reaching the 20,000Hz to 40,000Hz range. Furthermore, since it does not rely on a fixed exposure time for imaging, it theoretically possesses HDR imaging capabilities. This gives it a significant advantage in high-speed motion scenes, scenes with both strong and weak light, and HDR video reconstruction tasks.

[0004] However, the output of a pulse camera is not a multi-bit image in the conventional sense, but a continuous 1-bit pulse stream. To adapt to traditional computer vision tasks, image reconstruction usually needs to be performed on a pulse stream in the time dimension, that is, to reconstruct the image at a certain moment by accumulating the pulses in the time domain within a fixed time window. When the accumulation window is too short, although high temporal resolution can be maintained, the dynamic range and signal-to-noise ratio of the reconstructed image will be significantly limited. When the accumulation window is increased, complex alignment of the neural network is required to eliminate motion blur, which is not conducive to the reconstruction of HFR video. This creates a fundamental contradiction between HDR and HFR. At the same time, the 1-bit pulse stream lacks color information, so the reconstruction result is limited to grayscale images.

[0005] To address the lack of color information and sparse texture representation in pulse data, related technologies have attempted to incorporate RGB camera-assisted reconstruction. For example, by combining a pulse camera with an RGB camera, pulse data provides high temporal resolution motion information, while RGB images provide color and structural texture information, thus enabling color video reconstruction. However, if a conventional rolling shutter color camera is used, the inherent line-by-line exposure distortion of RS (Rolling Shutter) images must be addressed. Some existing methods use only the RS image itself for correction, or utilize event data to assist in RS-to-GS (Global Shutter) image restoration, but these methods still easily suffer from insufficient correction, residual structural distortion, missing textures, or inter-frame flickering. Summary of the Invention

[0006] To address or partially address the problems existing in related technologies, this application provides a video reconstruction method and system based on a pulse camera and a rolling shutter color camera.

[0007] This invention provides a color video reconstruction method based on the fusion of a pulse camera and a rolling shutter color camera. The method includes: Step S1, acquiring pulse stream data and rolling shutter color images of the same target scene; wherein, the pulse stream data is acquired by a pulse camera, and the rolling shutter color images are acquired by a rolling shutter color camera; Step S2, based on the line-by-line exposure timing of the rolling shutter color images, constructing virtual rolling shutter pulse frames time-aligned with the rolling shutter color images from the pulse stream data, configuring multiple target reconstruction times, and constructing virtual global shutter pulse frames corresponding to each target reconstruction time from the pulse stream data; Step S3, converting the timing data... The matched virtual global shutter pulse frame and the virtual rolling shutter pulse frame are input into the optical flow estimation module to obtain the optical flow field between the global shutter angle and the rolling shutter color image at each target reconstruction time. The rolling shutter color image is then subjected to inverse distortion processing based on the optical flow field to obtain the initial global shutter color image at each target reconstruction time. In step S4, each frame of the initial global shutter color image and the corresponding virtual global shutter pulse frame are input into the reconstruction module to perform image feature and pulse feature fusion, output the global shutter color image at each target reconstruction time, and stitch them together to obtain a color video sequence of the target scene.

[0008] Furthermore, in the above-mentioned color video reconstruction method, in step S2, constructing the virtual global shutter pulse frame includes: performing temporal accumulation of pulse events within a preset time window near the target reconstruction time to generate a virtual global shutter pulse frame corresponding to the target reconstruction time.

[0009] Furthermore, in the above-mentioned color video reconstruction method, step S2, constructing the virtual rolling shutter pulse frame includes: extracting corresponding row pulse data from multiple virtual global shutter pulse frames row by row according to the exposure time corresponding to each image row of the rolling shutter color image and combining them to form a virtual rolling shutter pulse frame that is time-aligned with the rolling shutter color image.

[0010] Furthermore, in the above-mentioned color video reconstruction method, step S3, which involves performing reverse distortion processing on the rolling shutter color image based on the optical flow field to obtain the initial global shutter color image corresponding to each target reconstruction time, includes: estimating the pixel displacement relationship between the global shutter angle at the target reconstruction time and the rolling shutter color image based on the virtual global shutter pulse frame and the virtual rolling shutter pulse frame; and performing reverse distortion on the rolling shutter color image based on the pixel displacement relationship to obtain the initial global shutter color image.

[0011] Furthermore, in the above-mentioned color video reconstruction method, the reverse distortion processing adopts an interpolation sampling method, which maps the pixels in the rolling shutter color image to the positions corresponding to the target reconstruction time according to the optical flow field, so as to correct the image geometric distortion caused by progressive exposure.

[0012] Furthermore, in the above-mentioned color video reconstruction method, before inputting the virtual global shutter pulse frame into the reconstruction module, the virtual global shutter pulse frame is subjected to temporal accumulation processing with multiple time window lengths to obtain pulse representations at multiple time scales, and the pulse representations at multiple time scales are spliced ​​together in the channel dimension and input into the reconstruction module as pulse features.

[0013] Furthermore, in the above-mentioned color video reconstruction method, the reconstruction module includes an image coding branch, a pulse coding branch, and a cross-modal feature fusion unit; the image coding branch is used to extract multi-scale image features of the initial global shutter color image; the pulse coding branch is used to extract multi-scale pulse features of the pulse feature input; and the cross-modal feature fusion unit is used to fuse the multi-scale image features and the multi-scale pulse features to obtain fused features.

[0014] Furthermore, in the above-mentioned color video reconstruction method, the cross-modal feature fusion unit includes: a first cross-attention branch, used for feature interaction with image features as queries and impulse features as keys and values; a second cross-attention branch, used for feature interaction with impulse features as queries and image features as keys and values; and a gated fusion subunit, used for weighted fusion of the outputs of the first cross-attention branch and the second cross-attention branch, and residual connection with the original image features to obtain the fused features.

[0015] Furthermore, in the above-mentioned color video reconstruction method, the reconstruction module further includes a temporal consistency modeling unit and a decoding reconstruction unit; the temporal consistency modeling unit is used to receive the fusion features of the current target reconstruction time and the hidden state of the previous target reconstruction time, and output the temporal enhancement features of the current target reconstruction time; the decoding reconstruction unit is used to output residual results according to the temporal enhancement features; the residual results are added to the initial global shutter color image to obtain the global shutter color image.

[0016] On the other hand, the present invention also provides a color video reconstruction system based on the fusion of a pulse camera and a rolling shutter color camera. The system includes: a pulse camera for acquiring pulse stream data of a target scene; a rolling shutter color camera for acquiring a rolling shutter color image of the target scene; a beam splitter for enabling the pulse camera and the rolling shutter color camera to share the same field of view of the target scene; a synchronization and calibration module for performing time synchronization and spatial calibration on the pulse stream data and the rolling shutter color image; a processor; and a memory storing executable code, which, when executed by the processor, causes the processor to perform any of the methods described above.

[0017] The present invention has the following technical effects: First, in this invention, the RS color image provides multi-bit brightness information, and the pulse stream provides supplementary dynamic temporal details in extremely bright and dark changing scenes, so that the resulting high frame rate color video sequence has good high dynamic range performance.

[0018] Secondly, by constructing virtual rolling shutter pulse frames and virtual global shutter pulse frames, this invention establishes a temporal relationship between the pulse domain and the RS progressive exposure domain, thereby enhancing the spatiotemporal correspondence at the physical imaging mechanism level. Through pulse-guided GS2RS optical flow estimation, the RS color image is unfolded, effectively reducing rolling shutter distortion.

[0019] Third, by fusing pulse information and image information in the subsequent reconstruction module, color expression, texture details and high-speed motion information can be taken into account, thereby outputting higher quality color video.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0021] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0022] Figure 1 This is a schematic flowchart illustrating a video reconstruction method based on a pulse camera and a rolling shutter color camera, as shown in an embodiment of this application. Figure 2 This is a schematic diagram of pulse optical flow unfolding in the first stage of training, as shown in an embodiment of this application; Figure 3 This is a diagram illustrating the image refinement network structure for the second stage of training, as shown in an embodiment of this application. Figure 4 This is a network structure diagram of the core module that needs to be trained in the second stage, as shown in an embodiment of the present invention. Detailed Implementation

[0023] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0024] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0025] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0026] To enable those skilled in the art to better understand the technical solution of the present invention, the specific embodiments of the present invention will be described in detail below in conjunction with the order of the claims. It should be understood that the following embodiments are only for explaining the present invention and do not constitute a limitation on the scope of protection of the present invention. Various equivalent substitutions and modifications can be made by those skilled in the art without departing from the concept of the present invention, and these should all fall within the scope of protection of the present invention.

[0027] See Figure 1 , Figure 1 This is a flowchart of a color video reconstruction method based on the fusion of a pulse camera and a rolling shutter color camera, provided in an embodiment of the present invention. As shown in the figure, the method includes the following steps: In step S1, pulse stream data and rolling shutter color images of the same target scene are acquired. In specific implementation, pulse stream data can be acquired using a pulse camera, and rolling shutter color images can be acquired using a rolling shutter color camera.

[0028] In this step, a hybrid acquisition system is first constructed. This system includes at least one pulse camera and at least one rolling shutter color camera. Preferably, the two types of cameras share the same field of view via a beam splitter to reduce parallax; simultaneously, external triggering, clock synchronization modules, or software timestamp alignment strategies can be combined to ensure that the pulse stream data and the RS color image correspond on the time axis. Spatially, intrinsic and extrinsic parameters can be calibrated using a calibration board, mapping the data from the two imaging channels to a unified coordinate system. In practical applications, the pulse camera can be a standard 1-bit pulse imaging device, and the rolling shutter color camera can be an industrial camera, as long as it can output progressively exposed color images.

[0029] Step S2: Based on the line-by-line exposure sequence of the rolling shutter color image, construct a virtual rolling shutter pulse frame that is time-aligned with the rolling shutter color image using pulse stream data, configure multiple target reconstruction times, and construct a virtual global shutter pulse frame corresponding to each target reconstruction time using pulse stream data.

[0030] In this step, since the exposure times of each row of the RS image are different, directly matching the accumulated pulse frames within a fixed time window with the entire RS image would cause a significant time mismatch. Therefore, this embodiment matches pulse information near the corresponding exposure time for each image row in the RS image according to the row-by-row exposure sequence, and stitches them together row by row to obtain a virtual rolling shutter pulse frame that is time-aligned with the RS image.

[0031] In this embodiment, multiple target reconstruction times are preset within the same RS readout period. Each target reconstruction time constructs a corresponding virtual global shutter pulse frame through the pulse stream near that time. This provides a timing basis for subsequent expansion from the RS domain to any target GS time.

[0032] Step S3: Input the time-matched virtual global shutter pulse frame and virtual rolling shutter pulse frame into the optical flow estimation module to obtain the optical flow field between the global shutter angle and the rolling shutter color image at each target reconstruction time. Then, perform reverse distortion processing on the rolling shutter color image according to the optical flow field to obtain the initial global shutter color image at each target reconstruction time.

[0033] In this step, the constructed virtual GS pulse frame and virtual RS pulse frame are input into the optical flow estimation module. The optical flow estimation module can employ existing optical flow networks based on pulses or high temporal resolution data, or it can employ a specially trained spatiotemporal motion estimation network. For example, in a preferred embodiment, a feature extraction structure of HiST-Sflow (Hierarchical Spatial-Temporal Spike Optical Flow) can be used, fine-tuned with a U-shaped network. The output of the optical flow estimation module is the pixel displacement field of the target GS time relative to the RS image. Based on this displacement field, the original RS color image is then reverse-distorted to obtain the initial global shutter color image corresponding to the target GS time. Because this process introduces high temporal resolution motion constraints of the pulse flow, it can more accurately correct geometric distortions caused by line-by-line exposure compared to unfolding only the RS image itself.

[0034] Step S4: Input the initial global shutter color image of each frame and the corresponding virtual global shutter pulse frame into the reconstruction module, perform image feature and pulse feature fusion, output the global shutter color image corresponding to each target reconstruction time, and stitch and integrate to obtain a high frame rate and high dynamic range color video sequence of the target scene.

[0035] In this embodiment, high frame rate means that the output video frame rate is higher than the original acquisition frame rate of the rolling shutter color camera; high dynamic range means that the output video can retain richer details in bright and dark areas compared to the result based only on short-time pulse accumulation or only on the unfolding of the rolling shutter image.

[0036] While optical flow reverse warping can complete basic geometric unfolding, artifacts, blurring, or loss of color details may still remain in occluded areas, weak texture areas, and areas with complex non-rigid motion. Therefore, this invention inputs the initial GS color image and the corresponding virtual GS pulse frame from each target time step in step S3 into the reconstruction module. Through deep fusion of image features and pulse features, more accurate texture, structure, and temporal consistency are restored. Finally, the reconstruction results from multiple target times are stitched together in chronological order to obtain a high frame rate, high dynamic range color video sequence.

[0037] This embodiment outputs global shutter color images corresponding to the reconstruction time of each target, and stitches them together to obtain a high frame rate color video sequence of the target scene. The rolling shutter color image provides multi-bit brightness information, and the pulse stream data provides high temporal resolution dynamic detail information to improve the dynamic range performance of the color video sequence.

[0038] This embodiment has the following technical effects: First, in this embodiment, the RS color image provides multi-bit brightness information, and the pulse stream provides supplementary dynamic temporal details in extremely bright and dark changing scenes, so that the resulting high frame rate color video sequence has good high dynamic range performance.

[0039] Secondly, this embodiment establishes a temporal relationship between the pulse domain and the RS progressive exposure domain by constructing virtual rolling shutter pulse frames and virtual global shutter pulse frames, which enhances the spatiotemporal correspondence at the physical imaging mechanism level. By using pulse-guided GS2RS optical flow estimation to unfold the RS color image, rolling shutter distortion can be effectively reduced.

[0040] Third, by fusing pulse information and image information in the subsequent reconstruction module, color expression, texture details and high-speed motion information can be taken into account, thereby outputting higher quality color video.

[0041] In one embodiment of the present invention, in step S2, constructing a virtual global shutter pulse frame includes: performing temporal accumulation of pulse events within a preset time window near the target reconstruction time to generate a virtual global shutter pulse frame corresponding to the target reconstruction time.

[0042] The pulse camera outputs a continuous binary pulse stream, typically represented as a sequence of pulse events triggered over time at pixel locations. Since pulse events at a single moment are too sparse to be directly used as input to subsequent networks, it is necessary to aggregate pulse events near a target moment in the time dimension. In this embodiment of the invention, for each target reconstruction moment, a preset time window is set with that moment as the center or offset reference. Pulse events falling within this time window are accumulated in the time domain to obtain the corresponding virtual global shutter pulse frame.

[0043] Specifically, the pulse camera accumulates electrical signals by capturing photons, and reads out signal 1 when the electrical signal reaches a threshold. Assuming the pixel position is... , Always in position The accumulated voltage is Voltage threshold is Then the pulse signal is:

[0044] In the above formula: This represents the pulse signal at time t.

[0045] By accumulating pulse signals within a time window This yields a pulse substream, which is then used as a virtual GS pulse frame. :

[0046] In the above formula, τ is the minimum temporal sampling interval of the pulse camera. t is the coefficient, N is the total number of sampling points within the time window, and t+kτ is the number of time-series sampling points centered at the target time t, with N / 2 points before and after it.

[0047] The above describes the scene's temporal activity under a unified time reference near a single target moment. Therefore, unlike the RS frames with progressive exposure, it corresponds to an approximate representation of the global shutter angle at a unified moment.

[0048] In this embodiment, the preset time window length can be adjusted based on factors such as scene motion speed, pulse signal density, expected reconstruction frame rate, and noise level. For example, in high-speed motion scenes, a shorter time window can be used to retain stronger temporal resolution; in weak texture or low-light scenes, the window length can be appropriately increased to improve the stability of the pulse response. In specific implementation, the time window can be determined according to actual needs.

[0049] This embodiment forms a virtual GS pulse frame by temporally accumulating pulse events near the target reconstruction time. Since the generated virtual GS pulse frame has a unified time reference, it provides a foundation for subsequently establishing a spatiotemporal mapping between the target GS time and the RS image. Simultaneously, compared to directly using the original pulse stream, temporal accumulation improves signal stability and processability, and reduces the uncertainty caused by single-event sparsity. Therefore, it can improve the efficiency of optical flow estimation and subsequent reconstruction modules in utilizing pulse information, enabling more effective characterization of the target's motion and texture changes at that time.

[0050] In some specific implementations, step S2, constructing a virtual rolling shutter pulse frame, may include: extracting corresponding row pulse data from multiple virtual global shutter pulse frames line by line according to the exposure time corresponding to each image row of the rolling shutter color image and combining them to form a virtual rolling shutter pulse frame that is time-aligned with the rolling shutter color image.

[0051] In this embodiment, based on the line-by-line exposure sequence of the rolling shutter color image, pulse data matching the exposure time of each image line is selected from multiple virtual global shutter pulse frames corresponding to different target times, and then stitched together line by line to finally form a virtual rolling shutter pulse frame that corresponds to the rolling shutter color image in time.

[0052] It is understood that during the imaging process, a rolling shutter color camera exposes the image sequentially from row 1 to row H, where H represents the total number of rows and r is the row index of row r. Because rolling shutter color images have a row-by-row temporal sequence characteristic, if only a pulse frame constructed at a single, uniform moment is matched with the entire rolling shutter image, significant temporal mismatch will occur. Especially when the target is moving at high speed, the camera is rotating rapidly, or the scene exhibits complex non-rigid changes, this mismatch will disrupt the correspondence between the pulse domain and the image domain, thereby affecting the accuracy of subsequent optical flow estimation and geometric unfolding. Therefore, this embodiment of the invention employs a row-by-row temporal matching framing mechanism to construct a virtual rolling shutter pulse frame.

[0053] In this embodiment, within the readout period corresponding to a rolling shutter color image, multiple target reconstruction times ty1, ty2, ..., tyK are pre-configured, and multiple virtual global shutter pulse frames are constructed around each target reconstruction time in the manner described above. Each virtual global shutter pulse frame represents the pulse response of the same scene under different unified time references. Subsequently, for each image row in the rolling shutter color image, based on the exposure time of that row... The pulse data corresponding to the row is extracted from the virtual global shutter pulse frame that best matches in time, and all image rows are combined in their original spatial order to obtain the final virtual rolling shutter pulse frame.

[0054] Specifically, a sliding window approach can be used to construct virtual RS pulse frames in an aligned manner. :

[0055] In the above formula: t is the reference time corresponding to the complete exposure cycle of a rolling shutter color image. The virtual GS pulse frame is obtained by pulse flow integration at exposure time th for the h-th row pixel of the RS camera. Indicates sampling virtual GS pulse frames The OK, Indicates the first The moment of exposure ， H represents the total number of pixel rows in the RS image.

[0056] In other words, the virtual rolling shutter pulse frame constructed in this embodiment of the invention is essentially a row-level time-stitched pulse representation: its r-th row originates from the r-th row pulse data in the virtual global shutter pulse frame whose exposure time is closest to that of the r-th row. Since each row comes from a moment that matches its actual exposure time, the virtual rolling shutter pulse frame is closer to the formation process of a real rolling shutter image in terms of time structure.

[0057] In other embodiments, for a given image row, corresponding row data can be extracted from two or more temporally adjacent virtual global shutter pulse frames, and weighted and fused according to the time distance to obtain the pulse representation of that row, forming more refined row-level pulse data. As long as it essentially still involves selecting or combining corresponding row data from multiple virtual global shutter pulse frames according to the exposure times of each row of the rolling shutter image to form rolling shutter timing-corresponding pulse frames, it is acceptable.

[0058] It should be noted that, to accommodate different readout modes of rolling shutter cameras, the exposure time corresponding to each image row in this embodiment can also be understood as the exposure time corresponding to each spatial sampling unit. For example, in some special sensors, the rolling shutter may perform exposure by column scanning, block scanning, or other sequences. In this case, the corresponding region data can be extracted and combined from multiple virtual global shutter pulse frames by column or block accordingly.

[0059] In practice, the number of target reconstruction moments can be determined based on the actual situation. A larger number results in a smaller interval between adjacent target reconstruction moments, leading to denser time sampling of the virtual global shutter pulse frames, thus more accurately approximating the rolling shutter line-by-line exposure process. If the number is relatively small, computational and storage consumption can be reduced.

[0060] In this embodiment, the virtual rolling shutter pulse frame has the same or corresponding spatial dimensions as the original rolling shutter color image and shares the same line-by-line temporal structure in its formation mechanism. Therefore, this virtual rolling shutter pulse frame is not simply a pulse accumulation map, but an auxiliary representation that is highly consistent with the rolling shutter color image in the temporal dimension. When it is subsequently input into the optical flow estimation module along with the virtual global shutter pulse frame corresponding to the target reconstruction time, the transformation relationship from the line-by-line exposure spatiotemporal domain to the global shutter domain at a unified time can be learned more effectively.

[0061] This invention, through row-by-row matching of exposure times, extracts corresponding row data from multiple virtual global shutter pulse frames, making the resulting virtual rolling shutter pulse frames more consistent with the generation process of real rolling shutter images in terms of time structure, thereby significantly reducing the cross-modal time mismatch problem.

[0062] In some implementations, step S3 involves performing reverse distortion processing on the rolling shutter color image based on the optical flow field to obtain the initial global shutter color image corresponding to each target reconstruction time. This includes: estimating the pixel displacement relationship between the global shutter angle at the target reconstruction time and the rolling shutter color image based on the virtual global shutter pulse frame and the virtual rolling shutter pulse frame; and performing reverse mapping on the rolling shutter color image based on the pixel displacement relationship to obtain the initial global shutter color image.

[0063] In this embodiment of the invention, the virtual global shutter pulse frame represents the scene temporal response at a uniform time near the reconstruction time of a certain target; the virtual rolling shutter pulse frame corresponds to the line-by-line time stitching pulse formed during the line-by-line exposure of the RS image. The two reflect the changing relationship of the same scene under different imaging spatiotemporal coordinates. Therefore, by using these two types of virtual pulse frames, the pixel displacement relationship of the target GS viewpoint relative to the current RS color image can be estimated.

[0064] The so-called pixel displacement relationship refers to the motion field mapped from the position of each pixel in the target GS image back to the sampling position in the RS image, or vice versa, from the RS pixel to the target GS coordinate system. In specific implementations, a two-dimensional displacement vector field can be output by the optical flow estimation module, with each pixel corresponding to a horizontal displacement and a vertical displacement. This displacement field can represent the optical flow from the global shutter view to the rolling shutter image, or it can represent the optical flow from the rolling shutter to the global shutter view; the only difference in implementation involves the twist direction and the way the sampling coordinates are expressed. This embodiment of the invention preferably uses a displacement field expression that facilitates backsampling to reduce the hole problem caused by forward scattering.

[0065] After obtaining the pixel displacement relationship, inverse warping is performed on the RS color image. Inverse warping refers to using the pixel grid of the target GS image as a reference, finding the corresponding sampling point in the RS color image using the displacement relationship, and reading the color value of that sampling point as the initial value of the target pixel. Since the color information in the RS image is complete and consists of multi-bit values, geometric correction can be achieved while preserving color and brightness information through inverse warping. If the sampling position obtained after mapping is not at an integer pixel, bilinear interpolation, bicubic interpolation, or other continuous sampling methods can be used to obtain the color value.

[0066] In a preferred embodiment, the optical flow estimation module employs a deep neural network with virtual GS pulse frames and virtual RS pulse frames as inputs. The network first extracts multi-scale features from the two frames, then estimates the displacement field through a correlation volume or hierarchical spatiotemporal fusion structure, and finally outputs an optical flow field consistent with the target reconstruction resolution through upsampling and thinning branches. Furthermore, the optical flow estimation module can be indirectly optimized by supervising the reconstruction error between the distorted GS image and the training ground value.

[0067] In this embodiment, since the displacement estimation is based on virtual GS pulse frames and virtual RS pulse frames, it can explicitly utilize the high temporal resolution information of the pulse stream to compensate for the temporal distortion caused by RS line-by-line exposure. Color and brightness information is directly recovered from the RS color image through inverse warping, allowing for the acquisition of the initial GS color image with minimal resampling error. Compared to directly generating the target frame using only an interpolation network, this embodiment better preserves the details in the original color image.

[0068] In some implementations, the reverse distortion process employs an interpolation sampling method to map pixels in the rolling shutter color image to the positions corresponding to the target reconstruction time based on the optical flow field, in order to correct the image geometric distortion caused by progressive exposure.

[0069] Since the estimated optical flow field is typically subpixel accurate, the mapped position generally does not fall on the discrete integer pixel grid of the RS image. Therefore, an interpolation sampling strategy is required when mapping the RS color image to the target time. Preferably, bilinear interpolation can be used. The principle is to select four neighboring pixels centered at the sampling location in the RS image and calculate the color value by distance weighting. For color images, interpolation can be performed independently for the R, G, and B channels. To obtain a smoother visual effect, higher-order methods such as bicubic interpolation can also be used.

[0070] In this embodiment, mapping pixels in the rolling shutter color image to the positions corresponding to the target reconstruction time is not a simple overall translation, but rather spatial resampling based on the local displacement of each pixel in the optical flow field. Since RS distortion essentially originates from line-by-line exposure, meaning different lines of the image correspond to different sampling times, when there is movement in the scene, the geometric positions of each line will experience inconsistent temporal misalignment. This embodiment corrects this spatiotemporal misalignment through pixel-level optical flow field correction, restoring structures that were originally bent, tilted, or stretched in the RS image to a relatively consistent geometric shape at the target GS time.

[0071] To further enhance the stability of the distortion results, in some embodiments, an effective region mask can be generated for the inverse distortion result, filling in areas exceeding the boundary and using a subsequent reconstruction network to complete areas with low visibility. It should be noted that this embodiment does not require the complete elimination of all artifacts in this step, but rather uses interpolation sampling to correct basic geometric distortion, providing a higher-quality initial image for subsequent reconstruction modules.

[0072] This embodiment achieves pixel-level correction of geometric distortion in RS images by employing an inverse distortion method using interpolation sampling. Since the mapping position can achieve sub-pixel accuracy, compared to a coarse method that only uses integer pixel shifts, it can reduce jagged edges, breaks, and misalignments, improving the continuity and smoothness of the initial GS image. Simultaneously, this step preserves the original color information in the RS color image, giving the initial GS image good refining potential and laying the foundation for the final video reconstruction quality.

[0073] In some implementations, before inputting the virtual global shutter pulse frame into the reconstruction module, the virtual global shutter pulse frame is subjected to time-domain accumulation processing with multiple time window lengths to obtain pulse representations at multiple time scales, and the pulse representations at multiple time scales are spliced ​​together in the channel dimension as pulse features input into the reconstruction module.

[0074] In this embodiment, multi-scale feature extraction can be achieved through two coding branches, extracting multi-scale image features from the initial GS image and multi-scale pulse features from the pulse temporal features. This multi-scale approach allows the network to simultaneously utilize high-resolution detail information and low-resolution global semantic information, adapting to the dual requirements of large displacement motion and fine-grained texture restoration.

[0075] While virtual GS pulse frames constructed with a single time window length can represent temporal activities at a specific time scale, the motion frequencies and texture change patterns are not consistent across different scenes. Shorter windows are better suited for preserving fast-moving edges and high-frequency temporal details, but the signal is sparser; longer windows are better for accumulating stable structural information and improving the signal-to-noise ratio, but the temporal blur is stronger. To simultaneously utilize the advantages of different time scales, this embodiment introduces multi-window temporal domain accumulation processing before the virtual GS pulse frame enters the reconstruction module.

[0076] Specifically, multiple time windows of different lengths can be set, such as short, medium, and long windows, to accumulate the pulse stream around the same target reconstruction time, generating multiple pulse representations. These pulse representations at different time scales are then concatenated along the channel dimension to form a multi-scale pulse feature tensor. This feature tensor can be used as input to subsequent pulse coding branches, enabling the network to simultaneously perceive rapidly changing details and relatively stable structural information.

[0077] For example, in one embodiment, three or four window lengths can be used. The window scales can be set according to arithmetic, geometric, or empirical values. For instance, shorter windows are used to extract sharp motion boundaries, medium windows are used to characterize the motion trend of the subject, and longer windows are used to enhance the stability of the background and weak texture areas. The resulting multi-scale pulse representations can be normalized before input to ensure that the cumulative intensity under different windows is within a comparable range.

[0078] It should be noted that, in actual implementation, the number and length of windows can be determined according to the actual situation, and this embodiment does not impose any limitations on them.

[0079] Channel-dimensional concatenation in this embodiment is a preferred method because it maintains the same spatial dimensions, facilitating direct processing by subsequent convolutional or Transformer coding networks. Besides concatenation, in some other embodiments, the channels can be encoded separately first and then fused.

[0080] This embodiment constructs multi-timescale pulse representations and performs channel stitching, enabling the reconstruction module to simultaneously acquire motion and texture information under different time windows. This allows for the retention of rapid dynamic changes using short time windows and the supplementation of weak texture and stable structural information using long time windows, thereby alleviating the contradiction between temporal resolution and signal-to-noise ratio in single-scale pulse representation. This helps to improve the detail recovery quality and reconstruction robustness in complex dynamic scenes.

[0081] In some implementations, the reconstruction module includes an image coding branch, a pulse coding branch, and a cross-modal feature fusion unit. The image coding branch extracts multi-scale image features from the initial global shutter color image; the pulse coding branch extracts multi-scale pulse features from the pulse feature input; and the cross-modal feature fusion unit fuses the multi-scale image features and multi-scale pulse features to obtain fused features.

[0082] As mentioned earlier, although the initial GS color image has undergone geometric unfolding, it may still suffer from artifacts and detail loss due to optical flow estimation errors, occlusion, and incomplete sampling. While pulse features contain rich motion and temporal high-frequency information, they lack direct color representation. Therefore, this embodiment employs a dual-branch coding and cross-modal fusion structure to leverage the advantages of both types of data simultaneously.

[0083] The image coding branch is used to extract multi-scale image features from the initial global shutter color image. Preferably, a convolutional encoder, residual encoder, U-shaped encoder, or lightweight Transformer encoder can be used. The multi-scale features typically include high-resolution shallow detail features and low-resolution deep semantic features to simultaneously take into account edge texture and global structure.

[0084] The pulse coding branch is used to extract multi-scale pulse features from pulse features concatenated across multiple time scales. Since pulse features reflect temporal variations and high-frequency textures, the pulse coding branch can focus on extracting dynamic boundaries, rapid motion cues, and local temporal patterns. In one embodiment, the network depth and channel design of the pulse coding branch can correspond to those of the image coding branch to enable subsequent feature fusion at the same scale.

[0085] The cross-modal feature fusion unit establishes information interaction between image features and impulse features. Unlike simple stitching, this embodiment uses fusion processing, specifically a learnable mechanism, to make the color and main structure in the image features complement the motion and texture gain in the impulse features, resulting in fused features more suitable for the reconstruction task. This fused feature can then be input into the temporal modeling and decoding modules to recover the final high-quality frame.

[0086] In practice, the fusion process can be convolutional fusion, attention fusion, gating fusion, or a combination of the above fusion methods.

[0087] This embodiment achieves the division of labor and effective fusion of two types of heterogeneous data by setting up an image coding branch, a pulse coding branch, and a cross-modal feature fusion unit. The image branch is responsible for color and structure, the pulse branch is responsible for dynamics and high-frequency information, and the fusion unit establishes a complementary relationship between the two. Therefore, this embodiment can more fully recover details, improve color performance, and enhance stability in complex motion scenes.

[0088] Furthermore, the cross-modal feature fusion unit includes: a first cross-attention branch, used for feature interaction with image features as queries and impulse features as keys and values; a second cross-attention branch, used for feature interaction with impulse features as queries and image features as keys and values; and a gated fusion subunit, used to perform weighted fusion of the outputs of the first and second cross-attention branches and perform residual connection with the original image features to obtain fused features. Before inputting the two cross-attention branches, image features and multi-scale impulse temporal features can be mapped to the same channel dimension through 1×1 convolutions to complete modal feature dimension alignment.

[0089] This embodiment employs a bidirectional cross-attention and gating fusion mechanism. In heterogeneous modality fusion, simply concatenating image features and impulse features often leads to problems such as information redundancy, increased conflict, or modality dominance imbalance. To address these issues, this embodiment of the invention employs a bidirectional cross-attention structure.

[0090] In the first cross-attention branch, image features are used as queries, and impulse features are used as keys and values ​​to extract fine texture information beneficial to the image from the impulses. This means that for each spatial location or feature token in the image features, the dynamic response and high-frequency structural information related to it are actively retrieved from the impulse features. Through this process, information in the impulse features that helps correct image motion errors and enhance texture boundaries is introduced into the image feature stream.

[0091] In the second cross-attention branch, impulse features are used as queries, and image features are used as keys and values ​​to extract color and structural information beneficial to the impulse from the image. This means that impulse features actively acquire color, structural contours, and subject context information from image features to compensate for the inherent lack of color and dense static texture in impulse signals. Thus, neither modality is passively concatenated, but rather a deep interaction is achieved through mutual querying and enhancement.

[0092] After obtaining the cross-attention outputs from both directions, a gated fusion subunit is introduced. Preferably, the gated fusion subunit first concatenates the two outputs along the channel dimension, and then generates two spatial-channel weight maps in the 0-1 range through a convolutional layer and a sigmoid activation function. The sum of the values ​​at the same spatial location in the two weight maps is approximately 1, corresponding to the contribution coefficients of the two branches, respectively. Subsequently, the outputs of the two branches are weighted and summed pixel-by-pixel, and residual connections are made with the original image features to obtain the final fused features. The reason for using residual connections is that the initial image features already contain relatively complete color and main structure information. Preserving this basis is beneficial for stabilizing the reconstruction process and avoiding excessive modal perturbations after fusion.

[0093] In one embodiment, the attention module can be set separately in multi-scale feature layers; or it can be set only in mid-to-high semantic layers to reduce computational cost. The gated fusion weights can be pixel-wise weights, channel-wise weights, or a combination of both. The cross-modal feature fusion unit can be placed between the encoder and decoder of the reconstruction network SURF-Net, and the output fused features are fed into the residual decoding branch to complete color image reconstruction.

[0094] This embodiment employs bidirectional cross-attention. On one hand, image features, aided by impulse features, achieve stronger compensation for dynamic boundaries, texture errors, and temporal details. On the other hand, impulse features, aided by image features, acquire color and subject structure features. Subsequently, a gated fusion subunit adaptively adjusts the contributions of the two information streams, and residual connections maintain image substrate stability. This reduces information conflicts caused by simple fusion, improves cross-modal fusion quality, and further enhances texture restoration and reconstruction accuracy.

[0095] In some embodiments, the reconstruction module further includes a temporal consistency modeling unit and a decoding reconstruction unit; the temporal consistency modeling unit is used to receive the fusion features of the current target reconstruction time and the hidden state of the previous target reconstruction time, and output the temporal enhancement features of the current target reconstruction time; the decoding reconstruction unit is used to output the residual result according to the temporal enhancement features; and the residual result is added to the initial global shutter color image to obtain the global shutter color image.

[0096] First, the temporal consistency modeling unit receives the fusion features of the current target reconstruction moment and the hidden state of the previous target reconstruction moment. Since the goal of this invention is not single-frame image enhancement but rather the generation of high frame-rate video sequences, there is a natural strong temporal correlation between adjacent frames. If each frame is reconstructed independently, even with high spatial quality, brightness jitter, local texture flicker, or temporal instability of details can easily occur during video playback. Therefore, this invention introduces a temporal consistency modeling unit to explicitly utilize the contextual information of the previous frame and even earlier frames to constrain the reconstruction of the current frame.

[0097] In a preferred embodiment, the temporal consistency modeling unit can be a Convolutional Gated Recurrent Unit (ConvGRU). This unit takes the current fused features and the hidden state from the previous time step as input, updates them through gating, outputs the temporal enhancement features for the current time step, and updates the hidden state for use in the next time step. Compared to ordinary RNNs, ConvGRU is better suited to preserving spatial structure information, thus making it suitable for video reconstruction tasks. Besides ConvGRU, ConvLSTM (Convolutional Long Short-Term Memory), a Transformer, or other temporal modeling structures with hidden state propagation capabilities can also be used.

[0098] Secondly, the decoding and reconstruction unit outputs a residual result based on the temporal enhancement features. Preferably, the decoding and reconstruction unit can adopt a lightweight U-shaped structure, including an upsampling unit, a channel adjustment convolution (e.g., a 1×1 convolution unit), channel attention, and skip connections. The upsampling unit is used to restore resolution; the 1×1 convolution is used for channel adjustment; the channel attention unit is used to enhance the response of key channels; and the skip connections are used to supplement shallow detail information. Finally, the features are mapped to the image space through an output convolution layer and the residual is added to the initial GS image to obtain the final refined GS image. The role of the decoding and reconstruction unit is to gradually restore the low-resolution, high-semantic features after encoding and temporal modeling to the original image resolution and predict a residual image. The residual result here can be understood as the content that needs to be compensated relative to the initial GS color image, such as the amount of detail enhancement, artifact correction, and local color correction.

[0099] Finally, the residual result is added pixel-by-pixel to the initial GS color image to obtain the final global shutter color image. The reason for using residual learning is that the initial GS image already possesses good basic structure and color information, so the network does not need to generate an entire frame image from scratch, but only needs to learn correction terms. This reduces the learning difficulty, speeds up convergence, and reduces over-smoothing or color drift problems.

[0100] The following is combined Figures 2 to 4 The embodiments of the present invention will be described in more detail below.

[0101] See Figure 2 The figure illustrates the rolling shutter imaging mechanism, the global shutter imaging mechanism, and how to use virtual pulse frames to expand the rolling shutter image to the global shutter moment of any target.

[0102] The left side of the image illustrates the exposure difference between rolling shutter and global shutter. Rolling shutter cameras do not expose the entire image at the same moment; instead, they expose each row sequentially from top to bottom, resulting in different exposure times for each row. When there is movement in the scene or the camera, different rows in the same frame actually record the scene state at different times, easily leading to rolling shutter distortions such as tilting, stretching, and bending. In contrast, in a global shutter image, all rows are exposed simultaneously at the same target time, and the entire image corresponds to a unified time reference, thus eliminating the geometric misalignment caused by row-by-row exposure.

[0103] The right side of the figure illustrates the core workflow of this method. First, based on the line-by-line exposure sequence of the real rolling shutter color image, virtual rolling shutter pulse frames are obtained by sampling and combining lines from the pulse stream, aligning these virtual pulse frames temporally with the real rolling shutter color image. Simultaneously, for any target global shutter moment to be reconstructed, temporal accumulation is performed from the pulse stream near that moment to construct the corresponding virtual global shutter pulse frame. In this way, a unified temporal correspondence is established between the real color image and the pulse data.

[0104] Subsequently, the virtual rolling shutter pulse frame and the virtual global shutter pulse frame are input into the optical flow estimation module to estimate the pixel motion relationship between them. Because the pulse camera has extremely high temporal resolution, the constructed virtual pulse frame can provide dense motion information, thus more accurately describing the displacement relationship of each pixel in the rolling shutter image from the progressive exposure state to the target global shutter moment. Finally, based on the estimated optical flow field, the real rolling shutter color image is subjected to inverse distortion processing, unfolded to the specified target moment, and the corresponding initial global shutter color image is obtained. The purpose of this process is to correct the rolling shutter distortion using the high-speed motion information provided by the pulse flow, while preserving the color and texture information provided by the color camera, thereby providing the basic input for subsequent high frame rate color video reconstruction.

[0105] See Figure 3 The figure illustrates the overall structure of the SURF-Net (Spatial Uniform Refinement Feature Network) reconstruction module and its continuous frame processing method. The input to this reconstruction module includes the initial global shutter color images from the previous and current time steps, as well as the corresponding virtual global shutter pulse frames. The initial global shutter color image, obtained from the optical flow unwrapping of the previous stage, already possesses basic color and structural information, but may still contain issues such as local artifacts, blurred details, occlusion errors, and inter-frame flicker. Therefore, the role of SURF-Net is to further refine and reconstruct based on this initial result.

[0106] The left side of the diagram represents the temporal processing between consecutive frames. For each target time step, the network not only processes the image and pulse information of the current frame but also receives the temporal feature state passed down from the previous time step. This avoids the temporal inconsistency problem caused by independent reconstruction of each frame, making the texture changes between adjacent frames more continuous and reducing flickering during video playback.

[0107] The right-hand area of ​​the figure represents the main network structure of SURF-Net. First, the virtual global shutter pulse frames undergo multi-time window TFP temporal accumulation processing to form pulse representations containing information at different time scales. Shorter time windows are beneficial for preserving high-speed motion details, while longer time windows are beneficial for improving the stability and signal-to-noise ratio of the pulse representations. Then, the network extracts multi-scale features through image coding and pulse coding branches: the image branch mainly extracts color, brightness, and spatial texture information, while the pulse branch mainly extracts high-speed motion, edge changes, and temporal dynamic information.

[0108] Subsequently, the multi-scale features extracted from the two branches are fed into the BiCAB (Bidirectional Cross-Attention Block) bidirectional cross-attention fusion module. This module establishes a cross-modal interaction relationship between image features and impulse features, enabling image information to use impulse information to correct motion regions and detail regions, while impulse information can also obtain more stable structural constraints with the help of image information. The fused features are then input into the ConvGRU temporal modeling module. ConvGRU receives the fused features at the current time step and the hidden state passed from the previous time step to model the temporal consistency between consecutive frames, thereby enhancing the stability of the video sequence.

[0109] Finally, the network gradually recovers the spatial resolution through the decoder and outputs the residual result of the current frame. This residual result is added to the input initial global shutter color image to obtain the final refined global shutter color image. The small structure on the right side of the figure further illustrates the composition of the encoder and upsampling module. The encoding part includes downsampling convolution and residual blocks, while the upsampling module includes upsampling, 1×1 convolution, and an SE channel attention module. Overall, this network achieves detail enhancement, artifact correction, and inter-frame stabilization of the initial unfolded image through joint modeling of "image color texture information + pulse high-speed temporal information + continuous frame temporal constraints".

[0110] This figure illustrates the specific structure of the BiCAB module, demonstrating how bidirectional cross-modal fusion is performed between image features and impulse features. Taking intermediate-scale features as an example, the input includes image features obtained from the image coding branch and impulse features obtained from the impulse coding branch. Image features contain color, brightness, and spatial texture information, while impulse features contain high-speed motion, edge variations, and temporal dynamics information; the two are complementary.

[0111] The BiCAB module employs a bidirectional cross-attention mechanism for feature interaction. The first branch uses image features as the query and impulse features as the key and value, calculating the corrected image features guided by impulse information through attention calculation. This branch utilizes the high-speed motion and detail changes in the impulse data to supplement unreliable areas in the image features caused by shutter distortion, motion blur, or occlusion. The second branch uses impulse features as the query and image features as the key and value, obtaining impulse features corrected by image information. This branch utilizes the structural, texture, and brightness information in the color image to apply spatial semantic constraints to the impulse features, ensuring that the impulse features not only express motion changes but also better serve color video reconstruction.

[0112] The right side of the figure illustrates the specific operations of attention computation. Before entering the attention computation stage, the input features undergo LayerNorm (Layer Normalization), convolutional mapping, and window partitioning to generate queries, keys, and values. The window operation divides the entire feature map into local windows, performing attention computation within each window, thereby reducing computational complexity and enhancing fine-grained feature interactions within local regions. After attention computation is complete, window merging and feature mapping are performed to obtain the cross-modal enhanced feature result.

[0113] After obtaining the cross-attention outputs from both directions, BiCAB does not simply add them together, but further introduces a gating fusion mechanism. This gating module generates adaptive weights through 1×1 convolutions and sigmoid activation, which are used to control the contribution ratio of the image-guided branch and the impulse-guided branch to the final result, respectively. In other words, the network can automatically determine whether to rely more on image information or impulse information based on the content of different regions. For example, in regions with clear colors and textures, more image features can be utilized; in regions with high-speed motion, edge misalignment, or missing details, more impulse features can be utilized. Finally, the fused result is residually connected with the original image features to obtain the final cross-modal fusion feature.

[0114] The function of this module is to achieve full information exchange between image modalities and impulse modalities through bidirectional attention, and then adaptively select the contribution of different modalities through a gating mechanism, thereby taking into account both the texture expression capability of color images and the high-speed dynamic perception capability of impulse data, and providing more reliable and richer fusion features for subsequent decoding and reconstruction.

[0115] The training method for the optical flow estimation module and the reconstruction module in the color video reconstruction method of this invention includes: constructing synthetic training data; performing a first-stage training on the optical flow estimation module; and performing a second-stage training on the reconstruction module.

[0116] In this embodiment, a separate training method is employed because, under actual acquisition conditions, it is difficult to simultaneously obtain strictly time-aligned rolling shutter color images, pulse stream data, and high-quality global shutter ground truth images corresponding to the target reconstruction time. Especially for high frame rate video reconstruction tasks, directly relying on real acquisition data for fully supervised training often suffers from synchronization errors, annotation difficulties, and equipment limitations. Therefore, this embodiment constructs synthetic training data with ground truth supervision to provide controllable and repeatable training samples for each module.

[0117] The construction of synthetic training data may include: generating a rolling shutter color image sequence based on a high frame rate global shutter video sequence, simulating the line-by-line exposure mechanism of a rolling shutter camera, and generating corresponding pulse stream data based on the high frame rate global shutter video sequence. Thus, rolling shutter input, pulse input, and the ground truth global shutter image at the corresponding target time can be obtained simultaneously in the same scene, facilitating subsequent supervision of each training stage.

[0118] In the first stage of training the optical flow estimation module, virtual global shutter pulse frames corresponding to the target reconstruction time and virtual rolling shutter pulse frames time-aligned with the rolling shutter color image are input into the optical flow estimation module. The optical flow estimation module outputs the optical flow field between the global shutter viewpoint and the rolling shutter viewpoint. Subsequently, the rolling shutter color image is subjected to inverse warping based on the optical flow field to obtain the initial global shutter color image corresponding to the target time. The error between this initial global shutter color image and the global shutter ground truth image is used as a supervision signal to train the optical flow estimation module.

[0119] It is understandable that, since it is usually difficult to directly obtain accurate pixel-by-pixel true optical flow fields from synthetic training data, this embodiment preferably uses an "image reconstruction result supervision" approach to indirectly train the optical flow estimation module. That is, it does not require the optical flow output by the network to be completely consistent with a certain predefined optical flow, but rather uses the proximity of the distorted initial global shutter image to the ground truth image to inversely constrain the optical flow estimation module. This approach avoids the problem of difficult annotation of true optical flow, while still allowing the network to learn effective displacement relationships applicable to rolling shutter unfolding tasks.

[0120] In the second stage of training for the reconstruction module, the initial global shutter color image output from the first stage and the virtual global shutter pulse frame corresponding to the target time are input into the reconstruction module to output a refined global shutter color image. The reconstruction module is then trained based on the error between this refined global shutter color image and the ground truth global shutter image. Since the first stage has already completed basic geometric correction, the second stage mainly learns to compensate for and correct for detail artifacts, texture loss, local blurring, and temporal instability.

[0121] This embodiment divides the training process into the two stages mentioned above, thereby relatively decoupling the optical flow estimation task and the image refinement task in terms of training objectives. The first stage focuses on learning the geometric unfolding relationship between the rolling shutter image and the target global shutter moment, while the second stage focuses on learning cross-modal information fusion and temporal consistency enhancement. This reduces the optimization difficulty of end-to-end direct joint training, improves the stability of model training, and helps to enhance the spatial quality and temporal continuity of the final output video.

[0122] Furthermore, in specific implementations, after the first stage of training is completed, the parameters of the optical flow estimation module can be fixed, and only the second-stage reconstruction module can be trained; alternatively, the first-stage module can be jointly fine-tuned in the second stage. Any training approach that first establishes an initial global shutter color image and then performs refined reconstruction falls within the scope of this invention.

[0123] In a preferred embodiment, constructing synthetic training data includes: adjusting a high frame rate global shutter video sequence to a preset resolution and a preset frame rate; generating a rolling shutter color image sequence by sequentially sampling according to the image rows based on the exposure time and readout time of the rolling shutter camera; and generating high temporal resolution pulse stream data based on the high frame rate global shutter video sequence using a pulse camera simulator.

[0124] In this embodiment, a high frame rate global shutter video sequence can be used as a source of ground truth training data. Since the global shutter video synchronously exposes the entire image at the same time, each frame can serve as a reference ground truth image for the reconstruction of a specific target. To make the training data more closely resemble subsequent practical application scenarios, the original high frame rate global shutter video sequence can first be adjusted to a preset resolution and frame rate. For example, spatial scaling and temporal resampling can be performed on the video sequence based on the training memory, model input size, and target output frame rate requirements.

[0125] Based on this, a rolling shutter color image sequence is generated by simulating the imaging mechanism of a rolling shutter camera. It can be understood that a rolling shutter camera does not expose the entire image at a single moment, but rather exposes different image rows sequentially according to a certain time order. Therefore, this embodiment samples adjacent time-series images in a high frame rate global shutter video sequence row by row by setting the exposure time and readout time, generating a color image that conforms to the imaging rules of a rolling shutter. Specifically, for the h-th row in the rolling shutter image, the image row data closest to that moment can be selected from the high frame rate global shutter video according to its corresponding exposure time, or interpolation sampling can be performed based on adjacent time-series images to obtain the pixel value of that row. After combining all image rows in spatial order, a rolling shutter color image sequence is formed.

[0126] Simultaneously, a pulse camera simulator is used to generate high temporal resolution pulse stream data based on a high frame rate global shutter video sequence. The pulse camera simulator can simulate and output a pulse signal sequence at the corresponding pixel location based on the differences in brightness changes between images at adjacent time points, combined with preset pulse trigger thresholds, integration methods, and time sampling intervals. Therefore, the generated pulse stream data and the simulated rolling shutter image share a common scene source on the time axis, facilitating the construction of subsequent virtual rolling shutter pulse frames and virtual global shutter pulse frames.

[0127] In one specific embodiment, the Gev-RS dataset can be used as the basic data source. This dataset includes multiple sets of indoor and outdoor dynamic scenes with high frame rate global shutter video sequences. The original resolution can be adjusted to 800×500 and the frame rate to 5000 FPS. Then, a 50 FPS rolling shutter color image sequence is simulated and generated based on the line-by-line exposure principle of a rolling shutter camera. Further, the images are scaled to a size matching the pulse processing resolution, for example, 400×250, and 20,000 Hz pulse stream data is generated based on the scaled image sequence. It should be noted that the above resolution and frame rate parameters are merely preferred examples, and the present invention does not limit them.

[0128] This embodiment constructs synthetic training data based on high frame rate global shutter video. On the one hand, it can provide accurate global shutter ground truth supervision for rolling shutter image unfolding and video reconstruction. On the other hand, the simulated rolling shutter color image and pulse flow data have a strictly controllable time correspondence, avoiding the synchronization error problem in real multi-sensor acquisition. Therefore, it helps to improve the quality of training data and the model training effect.

[0129] In a preferred embodiment, the second-stage training uses a weighted loss function to optimize the reconstruction module. The weighted loss function includes pixel reconstruction loss, structural similarity loss, perceptual loss, and temporal consistency loss.

[0130] In this embodiment, the goal of the second-stage reconstruction module is not only to restore the numerical accuracy of a single frame image, but also to simultaneously consider the integrity of image structure, subjective visual quality, and temporal stability of the video sequence. Therefore, if only a single pixel-level loss is used for optimization, the output result may be close to the true value in the sense of average error, but it may still be insufficient in terms of detail texture, edge structure, or stability of consecutive frames. Based on this, this embodiment adopts a weighted combination of multiple losses to jointly constrain the reconstruction module.

[0131] The pixel reconstruction loss constrains the approximation of the reconstructed image to the ground truth image at the pixel level. Preferably, Charbonnier loss can be used as the pixel reconstruction loss. This loss can be seen as a smooth approximation of the L1 loss, maintaining relative robustness to outliers while providing stable gradients during training. In practice, L1 loss, L2 loss, or other pixel-level losses suitable for image reconstruction tasks can also be used instead.

[0132] Structural similarity loss is used to constrain the consistency between the reconstructed image and the ground truth image in terms of brightness, contrast, and structural distribution. Preferably, a corresponding loss term can be constructed based on the Structural Similarity Index Measure (SSIM), for example, by subtracting the SSIM value from 1 as the loss, to encourage the network output to better match the structural information perceived by the human eye. This loss is particularly helpful in improving the recovery quality of edge regions, texture regions, and local structural regions.

[0133] Perceptual loss is used to constrain the proximity between the reconstructed image and the ground truth image in the high-level semantic feature space. Preferably, LPIPS (Learned Perceptual Image Patch Similarity) loss can be used as the perceptual loss. LPIPS encodes features of two images through a pre-trained feature extraction network and measures their differences in the deep feature space, thus effectively improving the subjective visual quality of the reconstruction result, reducing over-smoothing, and improving texture detail representation. In addition to LPIPS, other perceptual loss forms based on the feature differences of pre-trained networks can also be used.

[0134] Temporal consistency loss is used to constrain the temporal continuity between the output results of adjacent target reconstruction moments, thereby reducing brightness jitter, texture flicker, and local temporal instability in the video. In one specific embodiment, the temporal consistency loss can be constructed based on the difference relationship between reconstructed frames at adjacent moments, that is, constraining the change trend of the current reconstructed frame to be consistent with the change trend of the corresponding ground truth frame; alternatively, the reconstruction result at the previous moment can be motion-aligned and then constrained with the reconstruction result at the current moment. Any loss term that is essentially used to suppress unreasonable temporal fluctuations between adjacent frames can be applied.

[0135] The aforementioned loss terms can be weighted and summed to form the total loss function used in the second stage of training. That is, the total loss can be expressed as the sum of the pixel reconstruction loss, structural similarity loss, perceptual loss, and temporal consistency loss multiplied by their respective weight coefficients. Each weight coefficient can be set and adjusted according to the characteristics of the training data, the reconstruction objective, and the numerical range of different loss terms. For example, an experimental combination of weights can be selected to achieve a better balance between pixel error, structural quality, and video stability in the reconstruction result. It should be noted that the specific values ​​of each weight coefficient can be determined based on the actual situation, and this invention does not impose any limitations on them.

[0136] In a preferred embodiment, during training, samples are input sequentially according to the video time order of the same scene, enabling the temporal consistency modeling unit in the reconstruction module to fully utilize the information of preceding frames. When training samples switch from one scene to another, the hidden state is reinitialized to avoid cross-scene error propagation. In this way, the temporal consistency loss can work in conjunction with the temporal modeling mechanism within the network to jointly improve the coherence of the output video.

[0137] This embodiment introduces multiple weighted loss functions in the second stage of training, which simultaneously constrains the pixel accuracy, structural similarity, subjective perceived quality, and temporal continuity of the reconstruction results. As a result, the trained reconstruction module not only outputs clearer and more stable single-frame images, but also exhibits better visual smoothness and consistency during continuous video playback, making it more suitable for high frame rate video reconstruction tasks.

[0138] During training, to ensure the temporal modeling unit functions effectively, samples are preferably input sequentially according to the video sequence. When switching between different scenes, the hidden state is reset to avoid cross-scene error propagation. The loss function can consist of pixel loss, structural similarity loss, perceptual loss, and time-by-time consistency loss.

[0139] This embodiment introduces a temporal consistency modeling unit, ensuring that the reconstruction of the current frame depends not only on the current input but also on historical hidden states, thereby enhancing video-level continuity and reducing inter-frame jitter and flicker. By adding the residual output by the decoding and reconstruction unit to the initial GS image, the main structure of the original unfolded result can be preserved while focusing on correcting details and artifacts, improving reconstruction efficiency and stability, and outputting a smoother, more coherent, and more detailed high frame rate color video sequence.

[0140] The training method in this embodiment will be described in more detail below.

[0141] In terms of data construction, a combination of synthetic data and real-world collected data can be used.

[0142] For synthetic data, RS images can be generated first by simulating high frame rate and high quality GS video sequences, and then a high time resolution pulse stream can be generated by a pulse simulator based on the image sequence.

[0143] This implementation uses the Gev-RS dataset as the basic data source. This dataset contains various dynamic scenes both indoors and outdoors, covering camera movement, object movement, occlusion, and high-speed motion. It features high frame rate (HFR) and high GS image quality. The dataset contains 29 sequences with an original resolution of 1280×720 and a frame rate of 5700 frames per second. To adapt to task requirements, its resolution and frame rate were subsequently adjusted. First, the high-quality GS video sequences were adjusted to 800×500 resolution and 5000 FPS. Then, following the imaging principle of an RS camera, RS image simulation was performed on the GS sequences. This involved setting the exposure and readout times and sampling row-by-row to generate RS image sequences, with the frame rate controlled at 50 FPS. Further, the image resolution was scaled to 400×250 to match the pulse data processing resolution. Then, using a pulse camera simulator, 20,000 Hz pulse data was generated from the scaled image sequences. This method facilitates obtaining high-quality ground truth GS values ​​for supervised training.

[0144] In addition to synthetic data, this embodiment also constructed real-world scene data to verify the effectiveness of the method in practical application environments. The real-world data acquisition system consisted of an industrial camera and a pulse camera working in conjunction with a beam splitter for synchronous acquisition. The industrial camera, model MV-CU060-10UC, was capable of outputting 50 FPS RS color images. Twenty target reconstruction times could be set within each RS readout cycle, thus unfolding each RS image into 20 frames of GS color images, ultimately resulting in a 1000 FPS video output. A beam splitter was placed in front of both cameras to allow the two types of sensors to share the same scene field of view and to align the RS image data and pulse stream data in time and space. A total of 10 sets of real-world scene data were acquired.

[0145] For training, a two-stage training strategy can be adopted. The first stage trains the optical flow unwrapping module, optimizing the optical flow estimation network by supervising the error between the initial GS image after inverse warping and the ground truth GS image. The second stage fixes or combines the first-stage module with the reconstruction module, enabling it to reconstruct the final high-quality GS image based on the initial GS image and virtual GS pulse frames. Training losses can include Charbonnier loss, SSIM loss, LPIPS loss, and temporal consistency loss. The training platform can be based on the PyTorch framework and completed on a GPU.

[0146] Phase 1 Training: Pulsed Optical Flow-Guided RS Image Unfolding

[0147] In the first stage, the pulse optical flow estimation module HiST-SFlow is fine-tuned and trained. The fine-tuning method uses the feature extraction module of HiST-SFlow. Then, a U-Net-type network is built to estimate the GS2RS optical flow, directly inversely twisting the RS image to obtain the GS image, and supervising the output GS image. Since a real optical flow field is lacking, this method is chosen to supervise the twisted GS image.

[0148] Phase Two Training: Network Reconstruction Training

[0149] In the second stage, the initial GS image output from the first stage is used as one of the inputs, combined with the virtual GS pulse frames corresponding to the target time, to perform end-to-end training on the reconstruction network. During training, samples are input sequentially according to the video sequence of the same scene to ensure that ConvGRU fully models temporal consistency. To avoid error propagation of update states between different scenes, the cyclic update state is reinitialized when switching scenes.

[0150] The loss function is composed of a weighted average of multiple loss functions to balance pixel reconstruction quality, structural similarity, perceptual quality, and temporal consistency. This embodiment employs the following loss function combination approach: the main losses include Charbonnier loss, SSIM structural similarity loss, and LPIPS loss, while also incorporating a time-by-time temporal consistency constraint loss.

[0151] The model is implemented using the PyTorch framework and trained for 30 epochs on an NVIDIA GeForce RTX 4090 graphics processor. The learning rate linearly decayed from 1×10⁻⁴ to 1×10⁻⁶. Before training, color images were randomly cropped to 448×320, and pulse data were spatially aligned and cropped to 224×160 to reduce memory usage and increase the diversity of training samples.

[0152] The acquired real-world scene data includes RS images and pulse data, and is also processed in a two-stage inference process. Our RS image sequence has a frame rate of 50 FPS. We sample pulse data to obtain virtual RS pulse frames by aligning the exposure times of each row. Simultaneously, we sequentially select 20 times within the RS image reading cycle to sample 20 virtual GS pulse frames. The RS images are unfolded using pulsed optical flow guidance in the first stage, and each RS image is interpolated into 20 GS images. The interpolated results, along with the corresponding virtual GS pulse frames, are input into the reconstruction network in the second stage, finally outputting a 1000 FPS color HDR video.

[0153] The embodiments of the present invention can not only realize high frame rate color video reconstruction, but also improve reconstruction quality, reduce geometric distortion residue in complex dynamic scenes, and enhance overall visual stability, thus solving the problem that pulse data and RS color images are difficult to align directly in related technologies.

[0154] This invention also provides a color video reconstruction system based on the fusion of a pulse camera and a rolling shutter color camera. The system includes: a pulse camera for acquiring pulse stream data of a target scene; a rolling shutter color camera for acquiring rolling shutter color images of the target scene; a beam splitter for enabling the pulse camera and the rolling shutter color camera to share the same field of view of the target scene; a synchronization and calibration module for performing time synchronization and spatial calibration on the pulse stream data and the rolling shutter color images; a processor; and a memory storing executable code, which, when executed by the processor, causes the processor to execute any of the methods described above.

[0155] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A color video reconstruction method based on the fusion of a pulse camera and a rolling shutter color camera, characterized in that, include: Step S1: Acquire pulse stream data and rolling shutter color images of the same target scene; wherein, the pulse stream data is acquired by a pulse camera, and the rolling shutter color images are acquired by a rolling shutter color camera; Step S2: Based on the line-by-line exposure sequence of the rolling shutter color image, construct a virtual rolling shutter pulse frame that is time-aligned with the rolling shutter color image using the pulse stream data, configure multiple target reconstruction times, and construct a virtual global shutter pulse frame corresponding to each target reconstruction time using the pulse stream data. Step S3: Input the time-matched virtual global shutter pulse frame and the virtual rolling shutter pulse frame into the optical flow estimation module to obtain the optical flow field between the global shutter angle and the rolling shutter color image corresponding to each target reconstruction time. Then, perform reverse distortion processing on the rolling shutter color image according to the optical flow field to obtain the initial global shutter color image corresponding to each target reconstruction time. Step S4: Input the initial global shutter color image of each frame and the corresponding virtual global shutter pulse frame into the reconstruction module, perform image feature and pulse feature fusion, output the global shutter color image corresponding to each target reconstruction time, and stitch and integrate them to obtain the color video sequence of the target scene.

2. The color video reconstruction method according to claim 1, characterized in that, In step S2, constructing the virtual global shutter pulse frame includes: Pulse events within a preset time window near the target reconstruction time are accumulated in the time domain to generate a virtual global shutter pulse frame corresponding to the target reconstruction time.

3. The color video reconstruction method according to claim 2, characterized in that, In step S2, constructing the virtual rolling shutter pulse frame includes: According to the exposure time corresponding to each image row of the rolling shutter color image, the corresponding row pulse data is extracted from multiple virtual global shutter pulse frames and combined to form a virtual rolling shutter pulse frame that is time-aligned with the rolling shutter color image.

4. The color video reconstruction method according to claim 3, characterized in that, In step S3, the step of performing reverse distortion processing on the rolling shutter color image based on the optical flow field to obtain the initial global shutter color image corresponding to each target reconstruction time includes: Based on the virtual global shutter pulse frame and the virtual rolling shutter pulse frame, estimate the pixel displacement relationship between the global shutter angle at the target reconstruction time and the rolling shutter color image; Based on the pixel displacement relationship, the rolling shutter color image is reverse-distorted to obtain the initial global shutter color image.

5. The color video reconstruction method according to claim 4, characterized in that, The reverse distortion processing employs an interpolation sampling method, which maps the pixels in the rolling shutter color image to the positions corresponding to the target reconstruction time based on the optical flow field, in order to correct the image geometric distortion caused by progressive exposure.

6. The color video reconstruction method according to claim 5, characterized in that, Before inputting the virtual global shutter pulse frame into the reconstruction module, the virtual global shutter pulse frame is subjected to time-domain accumulation processing with multiple time window lengths to obtain pulse representations at multiple time scales. The pulse representations at multiple time scales are then spliced ​​together in the channel dimension and input into the reconstruction module as pulse features.

7. The color video reconstruction method according to claim 6, characterized in that, The reconstruction module includes an image coding branch, a pulse coding branch, and a cross-modal feature fusion unit; The image coding branch is used to extract multi-scale image features from the initial global shutter color image; The pulse coding branch is used to extract the multi-scale pulse features of the pulse feature input; The cross-modal feature fusion unit is used to fuse the multi-scale image features and the multi-scale pulse features to obtain fused features.

8. The color video reconstruction method according to claim 7, characterized in that, The cross-modal feature fusion unit includes: The first cross-attention branch is used to perform feature interaction using image features as queries and impulse features as keys and values. The second cross-attention branch is used to perform feature interaction using impulse features as queries and image features as keys and values; The gated fusion subunit is used to perform weighted fusion of the outputs of the first cross-attention branch and the second cross-attention branch, and perform residual connection with the original image features to obtain the fused features.

9. The color video reconstruction method according to claim 7 or 8, characterized in that, The reconstruction module also includes a time-consistency modeling unit and a decoding reconstruction unit; The temporal consistency modeling unit is used to receive the fusion features of the current target reconstruction time and the hidden state of the previous target reconstruction time, and output the temporal enhancement features of the current target reconstruction time. The decoding and reconstruction unit is used to output residual results based on the temporal enhancement features; The residual result is added to the initial global shutter color image to obtain the global shutter color image.

10. A color video reconstruction system based on the fusion of a pulse camera and a rolling shutter color camera, characterized in that, include: A pulse camera is used to acquire pulse stream data of a target scene. A rolling shutter color camera is used to capture rolling shutter color images of the target scene; A beam splitter is used to enable the pulse camera and the rolling shutter color camera to share the same field of view of the target scene; The synchronization and calibration module is used to perform time synchronization and spatial calibration on the pulse stream data and the rolling shutter color image; A processor and a memory, wherein executable code is stored in the memory, and when the executable code is executed by the processor, the processor performs the method as described in any one of claims 1-9.

Images