Self-supervised video deblurring and image frame insertion method based on event camera

Abstract

The invention provides a self-supervised video deblurring and image frame insertion method based on an event camera. The method comprises the steps of constructing a fuzzy video and event stream data set, preprocessing event streams, constructing a learnable double integral network and a fusion network to process the fuzzy video and the event streams, performing network training through a self-supervised framework, and reconstructing a high-frame-rate clear video. The event stream preprocessing method is established through a joint event generation model and a blurred image generation model, and it is guaranteed that potential clear image reconstruction can be conducted at any target moment inside and outside the blurred video exposure time. And inputting the preprocessed event stream and the blurred image into a learnable double integral network and a fusion network to carry out motion blurring removal and image frame insertion. And finally, constructing a self-supervision framework to train a network model by utilizing the mutual relation among the fuzzy video, the clear video and the event stream, and processing the fuzzy video and the event stream by using the trained model to reconstruct the high-frame-rate clear video. According to the method, the problems of motion blur and inter-frame information loss are solved, and good deblurring and image frame insertion effects are achieved.

Description

technical field [0001] The invention belongs to the field of image processing, in particular to realizing self-supervised video deblurring and image frame interpolation by using an event camera. Background technique [0002] In high-dynamic scenes (such as fast-moving cameras or non-linear moving objects), video shooting often faces the problem of motion blur and loss of inter-frame information, thereby reducing the overall video quality. Motion deblurring and frame interpolation, as important branches in the field of underlying visual image processing, can reconstruct clear, high-frame-rate videos from blurred, low-frame-rate videos, providing clear and smooth The video look and feel is also convenient for subsequent algorithm processing such as target detection, tracking, and 3D reconstruction, which has extremely high application value. [0003] However, due to the ambiguity of the motion direction and the erasure of the target edge information by the blurred image, it i...

AI Technical Summary

Problems solved by technology

The current deblurring and image interpolation methods based on event cameras often treat these two tasks separately, but in fact, motion blur and inter-frame information loss have a strong symbiosis in real scenes, so they need to be considered jointly

In addition, most of the current deblurring and image frame interpolation algorithms rely on supervised training on simulation datasets, and due to the inconsistent data distribution between simulation datasets and real datasets, they often have performance degradation in real scene testing problem, spawning the need for a self-supervised framework

Images