A multimodal video interpolation method based on event camera reconstruction reference

Abstract

The present invention discloses a multimodal video interpolation method based on event camera reconstruction reference. First, an event stream is generated from a high-speed video through an event generation model and aligned with the frame on a time scale to ensure that video interpolation frames can be performed at any time during the training process. The event stream is processed into voxels and event attention masks, and the voxels are input into a reference frame reconstruction network to generate a reference frame. The reference frame processed by the event attention mask strategy and the video frame are input into a synthesis network based on a reference frame for bidirectional alignment, thereby synthesizing a coarse interpolation frame. Finally, the coarse interpolation frame, event voxels and two video key frames are input into an interpolation frame refinement network to generate a clear interpolation frame. The present invention utilizes event-based references to directly synthesize intermediate frames, avoiding the necessity of explicit motion estimation, thereby greatly enhancing the ability to handle motion occlusion.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a multimodal video interpolation method based on event camera reconstruction reference. Background Art

[0002] With the rapid evolution of display technology, high-refresh-rate display devices have become increasingly popular. Simultaneously, the demand for high-frame-rate video is also growing. Consequently, video frame interpolation (VFI) has garnered widespread attention. It not only overcomes the temporal resolution limitations of cameras at a low cost but also has applications in a variety of fields, including slow-motion video generation, video compression, and video frame prediction.

[0003] Event-based Video Frame Interpolation (E-VFI) technology has recently driven the development of the VFI field. Event cameras, with their high temporal resolution, can effectively bridge the information gap between consecutive video frames. Although most advanced E-VFI methods still follow the traditional VFI paradigm, which is based on motion estimation between consecutive frames and generates intermediate frames through a warping and refinement process, this method is highly dependent on the quality and consistency of keyframes, making it challenging to handle extreme real-world scenarios such as missing moving objects and severe occlusions.

[0004] While the introduction of event data into frame interpolation has made progress, resulting in more accurate motion estimation and improved interpolated frame quality, occlusion in VFI tasks, particularly the handling of severe occlusions and interpolated scenes that are invisible in keyframes, remains a challenge. These difficulties stem from the unknown nature of inter-frame information. Both traditional synthesis-based and motion-based methods rely heavily on the information provided by keyframes, such as scene content and temporal correlation. Therefore, how to more effectively utilize event data to assist interpolation tasks in overcoming these difficulties is an urgent issue. Summary of the Invention

[0005] Based on the above analysis, this paper proposes a new method for directly synthesizing interpolated frames based on inter-frame references reconstructed from event data. While current E-VFI works also utilize inter-frame cues from events, they usually treat this information as an enhancement to keyframes, such as assisting keyframe motion estimation. In contrast, our method reverses the roles of events and keyframes. We use keyframes as auxiliary information to complete the event-based reference, avoiding the challenges of fitting nonlinear motion and the occlusion problem introduced by the warping operation. At the same time, by utilizing event data as reference, we avoid the reliance on additional keyframes and long-term temporal information commonly adopted in traditional synthesis-based methods.

[0006] The present invention discloses a multimodal video frame insertion method based on event camera reconstruction reference, which specifically includes the following steps:

[0007] Step 1: Construct a high-frame-rate video and event stream dataset. Use the event generator ESIM to generate a synthetic dataset. Use an event camera and a high-frame-rate optical camera to capture a real dataset aligned in time and space. Use two images with multiple skipped frames, any one of the skipped frames, the timestamp corresponding to the frame, and the event stream within that period as a set of input data.

[0008] Step 2: For each group of input data, the event stream is divided according to the time of the intermediate frame of the selected skipped frame, processed into the network input format, and the event focus area mask is generated in the process.

[0009] To facilitate subsequent reference generation;

[0010] Step 3: Build and train network models. These include a reference frame reconstruction network, a reference frame-based synthesis network, and a frame refinement network. All network models are learnable, and trained models are used to process event streams and video frames, enabling high-precision image interpolation. Three network models based on convolutional neural networks and attention mechanisms are constructed, and data is fed into the network models for end-to-end training.

[0011] Step 4: Insert frame generation and input video and event stream into the trained network model to generate clearer video with higher frame rate;

[0012] The details of each step are as follows:

[0013] In step 1, the event generator ESIM is used to generate a synthetic event dataset. A real dataset is then created using a single event camera and a single high-speed optical camera to capture spatiotemporally aligned video. Data processing then uses two images with multiple skipped frames, a random frame from the skipped frames, the timestamp corresponding to that frame, and the event stream within that time period as input data.

[0014] A set of input data in step 1 is:

[0015] I i , I i+n , τ

[0016] Among them, I i and I i+n are the i-th and i+n-th frames of the video, where n∈[2,8], is any frame between the i-th and i+n-th frames of the video, where r∈[1, n). This frame is used as the frame to be inserted, and its corresponding timestamp is normalized to τ according to the value of n. The specific operation is as follows:

[0017] τ=r / n

[0018] It is the event stream within the exposure time from the i-th frame to the i+n-th frame of the video. The specific format is Where t is the timestamp of the event generation time, x(t)∈[1,W], y(t)∈[1,H] is the generation position of the event at time t, where W and H represent the maximum length of the row and column respectively. p(t)∈+1,-1 represents the polarity of the event generated at time t, +1 represents the brightness increase by a threshold, and -1 represents the brightness decrease by a threshold. For convenience of representation, a set of data is used as an example, marked as I0, I1, E 0→1 , τ.

[0019] The event stream processing involved in step 2 is divided into two parts: first, the event stream is divided according to The corresponding timestamps divide the event stream into E 0→τ and E τ→1 Two parts, corresponding to I0 frame to The event flow of the frame is the same as the The event stream is from frame to frame I1. Voxelization is then performed to discretize the time dimension into B consecutive time bins, and then the events are linearly integrated into the 3D space-time voxel grid (E∈R B×H×W ), where the integration for a particular time bin (k) can be formulated as:

[0020]

[0021] where t0 and Respectively represent the start time and end time of the integration event flow, N e Represents the number of event data, k range is [0, B-1]. The divided event streams after voxelization are represented as V 0→τ and V τ→1 , and serves as the input of the subsequent network.

[0022] At the same time, it is also necessary to generate an event attention mask based on the specific pixel location of the event during the entire motion process. First, an all-zero matrix is ​​created based on the frame size. All inter-frame events are counted at the corresponding pixel locations. If an event occurred at that pixel location, 1 is added to that location. Finally, the matrix is ​​mapped. If the value at a pixel location is greater than or equal to 1, indicating an event, it is mapped to 1. The mask is then generated using the proposed event-aware reconstruction strategy. This strategy involves applying a customized erosion operation to eliminate the influence of low event confidence regions during the subsequent reconstruction process. Next, an n×n kernel is defined and the number of surrounding pixels whose values ​​differ from the central value is counted. If this number exceeds a set threshold δ, the value of the central pixel is inverted. Next, a dilation operation is used to connect the sparse event points in the processed mask into paths, maintaining a certain degree of semantic consistency in the subsequent reconstructed image.

[0023] The reference frame reconstruction network described in step 3 includes a convolution layer, an average pooling layer, a bilinear interpolation layer, and a LeakyReLU activation function; the reference frame-based synthesis network includes a convolution layer, a LeakyReLU activation function, and a maximum pooling layer; and the frame refinement network includes a convolution layer, a normalization layer, and a gated forward pass module;

[0024] The data processing flow after the network is constructed is as follows: the processed event voxel V 0→τ and V τ→1 Input the reference frame reconstruction network to obtain the reference frame reconstructed from the event stream The reference frame The real frame to be inserted is processed using the event attention mask, and the two are supervised after processing. Input the synthesis network based on the reference frame with frames I0 and I1 to obtain the coarse insertion frame Finally, the rough insertion frame Bidirectional event stream V 0→τ and V T→1 Frames I0 and I1 are input into the frame refinement network for detail optimization, and the reconstruction result Iτ is finally obtained.

[0025] Our method introduces a novel pure synthesis framework for the E-VFI task, facilitating the generation of interpolated frames by aligning keyframes with event-based references. We incorporate an event-aware reconstruction strategy tailored to event properties, ensuring that the event-based reference preserves structural cues in high-confidence regions while mitigating the adverse effects of sparsity and noise prevalent in event data on interpolation. Furthermore, we propose E-PCD, a specialized bidirectional alignment module, to synthesize interpolated frames while preserving structural information in the reference. Our method addresses the occlusion challenge in the VFI task by directly generating event-based references for the frames to be interpolated. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of the entire algorithm process proposed by the present invention.

[0027] Figure 2 This is the reference frame reconstruction network used in the present invention.

[0028] Figure 3 This is the reference frame-based synthesis network proposed in the present invention.

[0029] Figure 4 This is a diagram of the network structure refined by the insertion frame used in the present invention.

[0030] Figure 5 The figure shows the comparison results with the current advanced interpolation algorithm on the synthetic dataset SNU-FILM (extreme scene).

[0031] Figure 6 The figure shows the comparison results with the current advanced event-based interpolation algorithm on the real datasets HQF and HSERGB. DETAILED DESCRIPTION

[0032] The purpose of the present invention is to propose a multimodal video interpolation method based on event camera reconstruction reference. The present invention is designed specifically for the characteristics of event data and performs video interpolation on a trained deep learning model.

[0033] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0034] The workflow of the present invention is divided into a training phase and a prediction phase. In the training phase, by simulating multiple random timestamp interpolation tasks, the learnable parameters of the reference frame reconstruction network, the reference frame-based synthesis network, and the frame refinement network are trained, allowing the model to learn sufficient transferable meta-knowledge from a large number of interpolated frame samples. In the prediction phase, based on the trained model parameters, interpolation is performed on low-frame-rate videos in different scenarios.

[0035] It is difficult to obtain a spatiotemporally aligned event stream-frame dataset. Currently, there are only a few real datasets open source, and the scenarios are relatively simple. Referring to related advanced work, it is feasible to first use a large-scale synthetic dataset for pre-training and then fine-tune it on a real dataset. In this invention, the event generator ESIM is first used in Vim eoA large simulated event dataset was generated using 90k and GOPRO cameras. This dataset is spatiotemporally aligned but lacks the inherent noise of real scenes. The proposed method is first pre-trained on this simulated dataset and then fine-tuned on an open-source real-world dataset. To enhance the generalization ability of the trained model, a series of data augmentation techniques are used:

[0036] (1) Random flip: Perform horizontal and vertical flips randomly according to probability.

[0037] (2) Random rotation: Rotate the image clockwise or counterclockwise by 90, 180, or 270 degrees randomly according to probability.

[0038] (3) Random cropping: Randomly crop the region of interest of the image. In the experiment, the size is set to 256 pixels in length and width.

[0039] The above data augmentation techniques are applied to both image frames and event streams. At the same time, since event stream data does not have a frame structure, it is first voxelized before the corresponding operations are performed.

[0040] A set of input data in the invention is represented as:

[0041] I i , I i+n , τ

[0042] Among them, I i and I i+n are the i-th and i+n-th frames of the video, where n∈[2,8], is any frame between the i-th and i+n-th frames of the video, where r∈[1, n). This frame is used as the frame to be inserted, and its corresponding timestamp is normalized to τ according to the value of n. The specific operation is as follows:

[0043] τ=r / n

[0044] It is the event stream within the exposure time from the i-th frame to the i+n-th frame of the video. The specific format is Where t is the timestamp of the event generation time, x(t)∈[1,W], y(t)∈[1,H] is the generation position of the event at time t, where W and H represent the maximum length of the row and column respectively. p(t)∈+1,-1 represents the polarity of the event generated at time t, +1 represents the brightness increase by a threshold, and -1 represents the brightness decrease by a threshold. For convenience of representation, a set of data is used as an example, marked as I0, I1, E 0→1 , τ.

[0045] In order to process the event stream into a format that can be input into the network to predict event references, the event stream is first divided into The corresponding timestamps divide the event stream into E 0→τ and E τ→1 Two parts, corresponding to I0 frame to The event flow of the frame is the same as the The event stream is from frame to frame I1. Voxelization is then performed to discretize the time dimension into B consecutive time bins, and then the events are linearly integrated into the 3D space-time voxel grid (E∈R B×H×W ), where the integration for a particular time bin (k) can be formulated as:

[0046]

[0047] where t0 and Respectively represent the start time and end time of the integration event flow, N e Represents the number of event data, k range is [0, B-1]. The divided event streams after voxelization are represented as V 0→τ and V τ→1 , and serves as the input of the subsequent network.

[0048] At the same time, it is also necessary to generate an event attention mask based on the specific pixel location where the event occurred during the entire motion process. First, create an all-0 matrix according to the frame size, and superimpose the number of all inter-frame events at the corresponding pixel location. If an event occurs at the pixel location, the location is added with 1. Finally, the matrix is ​​mapped. If the value at a pixel location is greater than or equal to 1, that is, an event occurs, it is mapped to 1. Then, the proposed event-aware reconstruction strategy is used to process and generate a mask. We apply a customized erosion operation to the mapped matrix to eliminate the influence of low event confidence regions during subsequent reconstruction. We then define an n×n kernel and count the number of pixels around it whose values ​​differ from the central value. If this number exceeds a set threshold δ, the value of the central pixel is inverted. Next, we use a dilation operation to connect the sparse event points in the processed mask into paths, maintaining a certain degree of semantic consistency in the subsequent reconstructed image.

[0049] After the event flow is pixelated and masked, Figure 1 、 2, 3, 4 construct the algorithm network of the present invention, including a reference frame reconstruction network, a reference frame-based synthesis network, and a frame refinement network. The reference frame reconstruction network used includes a convolution layer, an average pooling layer, a bilinear interpolation layer, and a LeakyReLU activation function. The reference frame-based synthesis network includes a convolution layer, a LeakyReLU activation function, and a maximum pooling layer. The frame refinement network includes a convolution layer, a normalization layer, and a gated forward transfer module. The data processing flow of the entire algorithm is as follows: Figure 1 As shown:

[0050] Processed event voxel V 0→1 Input the reference frame reconstruction network to obtain the reference frame reconstructed from the event stream The reference frame The real frame to be inserted is processed using the event attention mask, and the two are supervised after processing. Input the synthesis network based on the reference frame with frames I0 and I1 to obtain the coarse insertion frame Finally, the rough insertion frame Bidirectional event stream V 0→τ and V τ→1 And frames I0 and I1 are input into the frame refinement network for detail optimization, and the reconstruction result I is finally obtained. τ .

[0051] Throughout the process, I τ For supervision, L1 distance and LPIPS perception loss are used as loss functions to constrain the quality of the reconstructed image. The formula is as follows:

[0052]

[0053]

[0054]

[0055] in Represents the loss calculation at three different scales. The overall constraint formula is as follows:

[0056]

[0057] where λ rec ,λ syn ,λ refine Set them to 1, 1, 1 respectively.

[0058] Attachment Figure 5 The visualization of the interpolation effect of this method on the synthetic dataset SUN-FILM (extreme scene) is shown; Figure 6The interpolation effect of this method is visualized on the real datasets HQF and HSERGB. The following interpolation algorithms are compared on the synthetic datasets:

[0059] (1) BMBC: An interpolation method based on image frames, incorporating multi-scale ideas.

[0060] (2) RIFE: An interpolation method based on image frames, which optimizes optical flow from coarse to fine.

[0061] (3) TimeReplayer: An interpolation method based on image frames and event streams, using event streams for motion estimation. (4) TimeLens: An interpolation method based on image frames and event streams, using event streams for motion estimation.

[0062] (5)A 2 OF: An interpolation method based on image frames and event streams, using image frames for optical flow estimation.

[0063] (7)CBMNet-L: An interpolation method based on image frames and event streams, which uses event streams and image frames for motion estimation.

[0064] (6) Ours: An interpolation method based on image frames and event streams, using event streams and image frames for motion estimation.

[0065] The PSNR and SSIM indicators are used for evaluation on different data sets. Their specific definitions are as follows:

[0066]

[0067]

[0068] where μ X ,μ Y represent the means of X and Y respectively, σ X ,σ Y ,σ XY They represent the variance and covariance of X and Y, respectively. Higher PSNR and SSIM values ​​indicate better reconstruction. PSNR is more of a numerical evaluation, while SSIM is more comprehensive and better represents the visual effect observed by the human eye.

[0069] Table 1 shows the test results on the synthetic dataset:

[0070] Table 1: Evaluation of interpolation effects on synthetic datasets

[0071]

[0072] Next, Table 2 shows the test results on a real dataset. This section only compares methods based on event streams and image frames:

[0073] Table 2: Evaluation of interpolation effects on real datasets

[0074]

[0075] As shown in the results of Table 1 and Table 2, the present invention shows superior performance in high-speed video frame insertion compared with other related works.

[0076] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A multimodal video interpolation method based on event camera reconstruction reference, characterized by: The following steps are involved: Step (1): Construct a high frame rate video and event stream dataset; This includes using the event generator ESIM to generate synthetic datasets, and using event cameras and high-frame-rate optical cameras to capture real datasets that are temporally and spatially aligned. Step (2): Based on the data set constructed in step (1), select two pictures that skip multiple frames, any one of the skipped frames, and the event stream within the period as a set of input data, and process them. The specific operation method of data processing in step (2) is as follows: Step (2-1), first read a set of input data as an event-image group, the set of data including two image frames and an event stream between the two image frames; Step (2-2): Process the input data obtained in step (2-1), discretize the event stream into several time bins according to the time dimension, and encode them into a 3D voxel grid as the input of the subsequent network; Step (2-3): the event voxels obtained in step (2-2) and the two image frames are subjected to a series of random rotations, RGB channel transformations, and cropping operations to achieve data enhancement. Step (3): Generate an event focus area mask based on the event stream data obtained in step (2) to facilitate subsequent reference frame generation processing; Step (4): Based on the data characteristics obtained in step (2) and step (3), a network model is constructed; the network model includes a reference frame reconstruction network, a reference frame-based synthesis network, and a frame refinement network; Step (5): Building a reference frame reconstruction network based on the network model architecture of step (4), wherein the network module includes several convolutional layers, pooling layers, and activation functions to generate a reference frame; Step (6): Building a synthetic network based on the reference frame according to the network model architecture of step (4), wherein the network module includes several convolutional layers, pooling layers, and activation functions, and aligns the key frame to the reference frame to generate a coarse insertion frame; Step (7): Build a frame refinement network based on the network model architecture of step (4), wherein the network module includes several convolutional layers, pooling layers, activation functions, and linear layers for refining the inserted frames; Step (8): The data obtained in step (2) and step (3) are input into the model obtained in step (4) for training, wherein the processed events are input into the reference frame reconstruction network and the frame refinement network, and the image data is input into the reference frame-based synthesis network for interaction; Step (9): Use the model trained in step (8) to perform multimodal video interpolation, input the paired event-RGB frame information into the model, and output the inserted frame to achieve high frame rate multimodal video interpolation.

2. The multimodal video interpolation method based on event camera reconstruction reference according to claim 1, characterized in that: The details of the data set in step (1) are as follows: Events are aligned with RGB image frames at event stamps. There is an event stream between each two images, which records the motion that occurs during the time interval between the two frame exposures.

3. The multimodal video interpolation method based on event camera reconstruction reference according to claim 1, characterized in that: The method for encoding the 3D voxel grid in step (2-2) is: Step (2-2-1) divides the time dimension of the event information into B consecutive time bins, each containing a certain number of events, and normalizes the timestamps of the event information to between 0 and B-1; Step (2-2-2): For a coordinate in a time bin obtained in step (2-2-1), perform a weighted sum of the time dimension of all events in the time bin under the coordinate. The closer the timestamp of the event is to the timestamp of the time bin, the greater the weight. Step (2-2-3): Use the method in step (2-2-2) to process the entire event information to obtain the most original event representation.

4. The multimodal video interpolation method based on event camera reconstruction reference according to claim 1, characterized in that: The method for generating the event attention area mask in step (3) is: Step (3-1) first superimposes the number of all inter-frame events at the corresponding pixel position. If an event occurs at the pixel position, the superposition matrix is ​​mapped to determine whether an event occurs at each pixel position. Step (3-2) applies a customized erosion operation to the mapped matrix to eliminate the influence of low event confidence regions in the subsequent reconstruction process; Step (3-3) uses the dilation operation to connect the sparse event points in the processed mask into paths to maintain the semantic consistency of the subsequent reconstructed image to a certain extent.

5. The multimodal video interpolation method based on event camera reconstruction reference according to claim 4, characterized in that: The construction method and processing flow of the network model constructed in step (5) are: Step (5-1), the reference frame reconstruction network is a classic UNet structure, including several convolutional layers, residual connections, average pooling layers and activation functions; Step (5-2), using the reference frame reconstruction network constructed in step (5-1) to accept the processed event voxel grid as input, thereby reconstructing a grayscale reference frame and outputting a reference frame with an untrustworthy region; In step (5-3), the reference frame obtained in step (5-2) is processed using the mask obtained in step (3-3) to reduce the influence of the untrustworthy area and serve as the input of the subsequent network.

6. The multimodal video interpolation method based on event camera reconstruction reference according to claim 1, characterized in that: The construction method and processing flow of the network model constructed in step (6) are: Step (6-1), the reference frame-based synthesis network is composed of two pyramid-structured networks and a dense residual module cascaded together, including several convolutional layers, deformable convolutions, and activation functions; Step (6-2): The pyramid structure network constructed in step (6-1) is a module with shared weights, and each time it receives the reference frame processed in step (5-3), a key frame, and the event voxel grid between the reference frame and the key frame, it performs a one-way alignment; In step (6-3), the two coarse frames aligned in the direction obtained in step (6-2) are input into the dense residual module for synthesis to obtain a coarse insertion frame.

7. The multimodal video frame insertion method based on event camera reconstruction reference according to claim 6, characterized in that: The construction method and processing flow of the network model constructed in step (7) are: Step (7-1), the frame refinement network is a pyramid-structured attention network, including several cross-attention layers, sub-attention layers, feedforward neural network layers, residual modules, and normalization layers; In step (7-2), the frame refinement network constructed in step (7-1) accepts two key frames, the inter-frame event voxel grid, and the coarse interpolation frame obtained in step (6-3) as input, performs cross-attention and sub-attention between three different features each time, and outputs interpolation frames of three different scales for supervision, focusing on details at different scales.

8. The multimodal video interpolation method based on event camera reconstruction reference according to claim 1, characterized in that: The model training process mentioned in step (8) is: Step (8-1) first inputs paired event-RGB frame data into the model for inference; Step (8-2) inputs the result obtained in step (8-1) into the loss function to calculate the loss; Step (8-3) inputs the loss obtained in step (8-2) into the AdamW optimizer for back propagation, updates the network parameters, and updates the learning rate using the cosine annealing algorithm; Step (8-4) Repeat the above steps (8-1) to (8-3), test each round a certain number of times, and save the model parameters with the best current effect; The multimodal interpolation process mentioned in step (9) is: Step (9-1) is to load the network with the optimal model parameters saved in step (8); Step (9-2) inputs the data into the network obtained in step (9-1) to obtain an insertion frame; Step (9-3) processes the pixel values ​​of the inserted frame into 0-255 for visualization.

Images