A self-supervised video deblurring and image interpolation method based on event camera

Abstract

The present invention provides a self-supervised video deblurring and image interpolation method based on an event camera, including the construction of a blurred video and event stream data set, event stream preprocessing, the construction of a learnable double integral network and a fusion network to process the blurred video and event stream, and a self-supervised framework to perform network training and reconstruct a high frame rate clear video. The event stream preprocessing method is established by combining an event generation model with a blurred image generation model, which ensures that the present invention can select any target moment within or outside the blurred video exposure time for potential clear image reconstruction. The preprocessed event stream and blurred image will be input into a learnable double integral network and a fusion network for motion blur removal and image interpolation. Finally, the relationship between the blurred video, the clear video and the event stream is used to construct a self-supervised framework to train the network model, and the trained model is used to process the blurred video and event stream to reconstruct a high frame rate clear video. The present invention solves the problems of motion blur and inter-frame information loss, and has a good deblurring and image interpolation effect.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to realizing self-supervised video deblurring and image interpolation using an event camera. Background Art

[0002] In highly dynamic scenes (such as fast-moving cameras or nonlinearly moving targets), video shooting often faces the problem of motion blur and loss of inter-frame information, which reduces the overall video quality. Motion deblurring and frame interpolation, as important branches in the field of underlying visual image processing, can reconstruct clear, high-frame-rate videos from blurry, low-frame-rate videos. While providing a clear and smooth video experience, it also facilitates subsequent algorithm processing such as target detection, tracking, and 3D reconstruction, and has extremely high application value.

[0003] However, due to the ambiguity of the motion direction and the erasure of the target edge information by the blurred image, the current frame-based motion blur removal algorithm is difficult to accurately reconstruct the potential clear image. In addition, the current frame-based interpolation method usually assumes that the motion between adjacent frames is linear, but this linear assumption is difficult to guarantee in real scenes, especially when encountering complex nonlinear motion, and therefore often leads to incorrect image interpolation results.

[0004] The event camera based on biological visual perception mechanism has become a breakthrough in solving the above problems. Unlike traditional optical cameras, event cameras only perceive the logarithmic brightness changes of the scene, and replace the traditional frame representation with asynchronous event streams. They have the characteristics of low latency, high dynamic range, low bandwidth requirements and low power consumption. Compared with traditional optical cameras, event cameras can respond to instantaneous changes in scene brightness with extremely low latency, so they can still retain the edge texture information of the target in high-speed motion scenes, and can supplement the motion information lost between frame images. Current deblurring and image interpolation methods based on event cameras often handle these two tasks separately, but in fact, motion blur and inter-frame information loss have strong symbiosis in real scenes, so they need to be considered jointly. In addition, most of the current deblurring and image interpolation algorithms rely on supervised training on simulation datasets. Due to the inconsistent data distribution between simulation datasets and real datasets, they often have performance degradation problems when tested in real scenes, which has given rise to the need for self-supervised frameworks. Summary of the invention

[0005] Based on the above analysis, the purpose of the present invention is to provide a self-supervised video deblurring and image interpolation method based on an event camera. The method utilizes the low latency advantage of the event camera, treats image interpolation and deblurring as a unified task, and designs a self-supervised training framework using the mutually restrictive relationship between blurred images, clear images and event streams, so that the algorithm can be directly trained in real blurred, low frame rate videos, and reconstruct any high frame rate, clear video.

[0006] The self-supervised video deblurring and image interpolation method based on event camera provided by the present invention comprises the following specific steps:

[0007] Step 1: Construction of blurred video and event stream dataset; Use a single event camera or a combination of an event camera and an ordinary optical camera to shoot blurred video, and complete the spatiotemporal matching of event streams and image frames.

[0008] Step 2: Event stream preprocessing: For each set of input data, the event stream is split according to the target time to be reconstructed and preprocessed into the network input format.

[0009] Step 3: Construct a network model, which includes a learnable double integral network and a fusion network. Use the constructed network model to process blurred video and event streams to achieve motion blur removal and image interpolation; construct a learnable double integral network and a fusion network based on a convolutional neural network to perform motion blur removal and image interpolation.

[0010] Step 4: Use a self-supervised framework to train the network and reconstruct high frame rate clear video; use the relationship between blurry video, clear video and event stream to build a self-supervised framework to train the network model.

[0011] Step 5: Use the trained network model to process the blurred video and event stream and reconstruct a high frame rate clear video.

[0012] The details of each step are as follows:

[0013] In step 1, a single event camera (such as DAVIS346, etc.) or a combination of an event camera and an ordinary optical camera is required to shoot blurred video. After the spatiotemporal matching of the image frames and the event streams, two adjacent blurred image frames, their corresponding event streams, and the target moment to be reconstructed are used as a set of input data.

[0014] The set of input data described in step 1 is:

[0015]

[0016] Among them, B i and B i+1are the i-th and i+1-th blurred images of the video, and their corresponding exposure times are For B i Exposure time starts at B i+1 The time period when the exposure time ends (denoted as ), where is the timestamp of its generation time, p(t)∈{+1,-1} is its polarity, x(t)∈[1,W] and y(t)∈[1,H] indicate that its generation position is the xth column and yth row of the camera imaging plane; f is the target time to be reconstructed, which is the time from B i Exposure time starts at B i+1 The time period when the exposure time ends Any time within.

[0017] The event stream preprocessing function described in step 2 for:

[0018]

[0019] Where f is the target time to be reconstructed; t r is the reference target moment; For the period [f, t r ]Event stream data triggered within; t∈[0,t r -f]} is an event stream processing operator including time displacement, where e(t) = p(t)·δ(t-τ) is the continuous representation of event data (δ(·) is the Dirac function); t∈[0,ft r i} is an event stream processing operator that includes time displacement, flipping, and polarity inversion. Assume that the input blurred image B i The exposure start and end time are t1 and t2, B i+1 The exposure start and end time are t3 and t4. Then for the input event stream After preprocessing, 4 segments of event data can be obtained As input to the subsequent network.

[0020] The learnable double integral network (LDI) described in step 3 is composed of convolutional layers and ReLU activation functions, and the fusion network (Fusion) is composed of convolutional layers, residual modules, and attention modules CBAM. The data processing flow after building the network is as follows: First, the preprocessed event data At a certain time interval Δt The event points are accumulated in each time interval to form an event frame of size 2×H×W (2 represents positive and negative polarity, H and W are the height and width of the event frame), forming an N×2×H×W tensor input into the weight-sharing LDI network, and the event flow is calculated by the following formula: Regarding the time f to be reconstructed and the blurred image B i Exposure time (exposure start time is t1, end time is t2) double integral

[0021]

[0022] The event stream can also be obtained Regarding the time f to be reconstructed and the blurred image B i+1 Exposure time (exposure start time is t3, end time is t4) double integral

[0023]

[0024] Then combine the blurred image B i , B i+1 Get the preliminary reconstruction result L i (f), L i+1 (f):

[0025]

[0026]

[0027] By weighting L i (f), L i+1 (f) Generate an additional reconstruction result

[0028]

[0029] Among them, the weight function ω(f) is defined as (assuming

[0030]

[0031] Finally, the preliminary reconstruction result L i (f), L i+1 (f) Double integral results with event flow The cascade is transformed into a tensor and sent to the fusion network for processing, outputting the final reconstruction result.

[0032] The self-supervision framework described in step 4 is:

[0033]

[0034] in is a fuzzy-event constraint; for the fuzzy-sharp constraint; is a clear-event constraint; α, β, γ are the corresponding weight scalars.

[0035] Fuzzy-Event Constraints Defined as

[0036]

[0037] The superscript represents the logarithmic domain, such as The symbol ||·||1 represents the l1 norm.

[0038] Fuzzy-Sharp Constraints Defined as

[0039]

[0040] in To generate a blurred image based on the blurred image generation theorem, the output of the network is regenerated, such as The formula is generated as

[0041]

[0042] in The output of the fusion network at the exposure time The mth potential clear image reconstructed in . Similarly, The formula for generating is

[0043]

[0044] Clear-Event Constraints Defined as

[0045]

[0046] in is the difference between two reconstructed clear images in the logarithmic domain; is the result of integrating events between time [f, t]; is the normalization operator, defined as

[0047]

[0048] Where X represents the input image, It is a pixel-by-pixel event mask, that is, the value of the pixel position with event output is 1, and the value of the pixel position without event data is 0.

[0049] Compared with the prior art, the advantages and beneficial effects of the present invention are:

[0050] The event stream preprocessing method of the present invention is established by combining the event generation model with the blurred image generation model, which ensures that the present invention can select any target moment within or outside the blurred image exposure time for potential clear image reconstruction. The preprocessed event stream and blurred image will be input into the learnable double integral network and fusion network for motion blur removal and image interpolation. Finally, the relationship between the blurred video, clear video and event stream is used to construct a self-supervised framework to train the network model. The present invention uses the low latency advantage of the event camera to solve the problems of motion blur and inter-frame information loss, and designs a self-supervised learning framework so that the network can be directly trained on a real-shot data set, and has good deblurring and image interpolation effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 For a real video, both the deblurring algorithm and the image interpolation algorithm can only reconstruct a partial image, while the present invention can reconstruct an image at any time within or outside the image exposure time.

[0052] Figure 2 This is a schematic diagram of the event stream preprocessing proposed by the present invention.

[0053] Figure 3 This is the overall flow chart of the algorithm proposed in the present invention and a schematic diagram of self-supervision constraints.

[0054] Figure 4 This is a network structure diagram of the learnable dual integrator network (LDI) and fusion network (Fusion) proposed in the present invention.

[0055] Figure 5 The figure shows the comparison results with the current motion blur removal algorithm on the simulation datasets GoPro and HQF.

[0056] Figure 6 The figure shows the comparison results with the current motion blur removal algorithm on the real dataset RBE.

[0057] Figure 7 The figure shows the comparison results with the current image interpolation algorithm on the simulation datasets GoPro and HQF.

[0058] Figure 8 The figure shows the comparison results with the current image interpolation algorithm on the real dataset RBE. DETAILED DESCRIPTION

[0059] In order to understand the present invention more clearly, the technical content of the present invention is specifically introduced below.

[0060] High-dynamic scenes, such as fast-moving objects or cameras, are captured by ordinary optical cameras and event cameras, and videos with low frame rates and motion blur are collected. Blurred image frames and event streams are matched in time and space to construct a blurred video dataset. Due to the limited scale of data captured on the spot, data enhancement methods can be used to expand samples. Deep learning is a data-driven method. The larger the training data set, the stronger the generalization ability of the trained model. However, when collecting data in practice, it is difficult to cover all scenes, and collecting data also requires a lot of cost, which leads to limited training sets in practice. If various training data can be generated based on existing data, better cost reduction can be achieved, which is the purpose of data enhancement. Although event stream data does not have a frame structure, it can still be transformed accordingly according to the pixel position of each event point to obtain an enhanced event stream. Commonly used data enhancement techniques are:

[0061] (1) Flip: Flip includes horizontal flip and vertical flip.

[0062] (2) Rotation: Rotation is clockwise or counterclockwise rotation. Note that when rotating, it is best to rotate 90-180°, otherwise there will be scale problems.

[0063] (3) Scaling: Images can be enlarged or reduced. When scaling, the enlarged image size will be larger than the original size. Most image processing architectures will crop the enlarged image to its original size.

[0064] (4) Cropping: Cropping the region of interest in the image. Usually during training, different regions are randomly cropped and then scaled back to the original size.

[0065] The event point data generated by the event camera can be expressed as {t, p(t), x(t), y(t)}, where t is the timestamp of the event point generation, p(t)∈{+1,-1} is its polarity, and x(t)∈[1,W] and y(t)∈[1,H] indicate that its generation position is the xth column and yth row of the camera imaging plane. For the convenience of representation, the event point data is converted into a continuous function

[0066] e(t)=p(t)·δ(t-τ)

[0067] Where δ(·) is the Dirac function. Each set of input data can be expressed as follows

[0068]

[0069] Among them, B i and B i+1 are the i-th and i+1-th blurred images of the video, and their corresponding exposure times are For B iExposure time starts at B i+1 The time period when the exposure time ends (denoted as ) is the event stream data collected in B; f is the target time to be reconstructed, which is the time from B i Exposure time starts at B i+1 The time period when the exposure time ends Before inputting the network, the event stream needs to be preprocessed according to the timestamp f to be reconstructed. is the event stream preprocessing function, which is defined as

[0070]

[0071] Where f is the target time to be reconstructed; t r is the reference target moment; For the period [f, t r ]Event stream data triggered within; t∈[0,t r -f]} is an event stream processing operator that includes time displacement; t∈[0,ft r ]} is an event stream processing operator that includes time shift, flip, and polarity reversal, such as Figure 2 Assume that the input blurred image B i The exposure start and end time are t1 and t2, B i+1 The exposure start and end time are t3 and t4. Then for the input event stream After preprocessing, 4 segments of event data can be obtained As subsequent network input (such as Figure 3 This event stream preprocessing operation enables the model to reconstruct The potential clear frame at any time within Image interpolation can be performed while motion blur is removed.

[0072] After the event stream is preprocessed, first Figure 4 Construct a learnable event double integral network (LDI) and a fusion network (Fusion), wherein the learnable double integral network LDI includes a convolutional layer and a ReLU activation function, and the fusion network Fusion includes a convolutional layer, a residual module, and an attention module CBAM. The subsequent data processing flow is as follows (e.g. Figure 3 shown):

[0073] (1) The preprocessed event data At a certain time interval Δt The event points are accumulated in each time interval to form an event frame of size 2×H×W (2 represents positive and negative polarity, H and W are the height and width of the event frame), forming an N×2×H×W tensor input into the weight-sharing LDI network, and the event flow is calculated by the following formula: Regarding the time f to be reconstructed and the blurred image B i Exposure time (exposure start time is t1, end time is t2) double integral

[0074]

[0075] The event stream can also be obtained Regarding the time f to be reconstructed and the blurred image B i+1 Exposure time (exposure start time is t3, end time is t4) double integral

[0076]

[0077] (2) Combined with blurred image B i , B i+1 Get the preliminary reconstruction result L i (f), L i+1 (f):

[0078]

[0079]

[0080] By weighting L i (f), L i+1 (f) Generate an additional reconstruction result

[0081]

[0082] Among them, the weight function ω(f) is defined as (assuming ):

[0083]

[0084] (3) The initial reconstruction result L i (f), L i+1 (f) Double integral results with event flow The cascade is transformed into a tensor and sent to the fusion network for processing, outputting the final reconstruction result.

[0085] Finally, the self-supervised framework is used to train the LDI network and the Fusion network. The training method is the same as the normal neural network training, that is, first take part of the data as the training set to train the network, and then use the trained network to process the video. Here, the loss function used in the training is designed in the form of self-supervision. Figure 3 shown) is:

[0086]

[0087] in is a fuzzy-event constraint; for the fuzzy-sharp constraint; To clarify the event constraints; α, β, γ are the corresponding weight scalars. According to experience, the early stage of training is set to [α, β, γ] = [512, 1, 0.1] for faster convergence, and the later stage of training is set to [α, β, γ] = [128, 1, 0.1] to learn more refined features. The various parts of the self-supervised framework are defined as follows:

[0088] (1) Fuzzy-Event Constraints Defined as

[0089]

[0090] The superscript represents the logarithmic domain, such as The symbol ||·||1 represents the l1 norm.

[0091] (2) Fuzzy-clear constraints Defined as

[0092]

[0093] in To generate a blurred image based on the blurred image generation theorem, the output of the network is regenerated, such as The formula for generating is

[0094]

[0095] in The output of the fusion network at the exposure time The mth potential clear image reconstructed in . Similarly, The formula for generating is

[0096]

[0097] (3) Clear-Event Constraints Defined as

[0098]

[0099] in is the difference between two reconstructed clear images in the logarithmic domain; is the result of integrating events between time [f, t]; is the normalization operator, defined as

[0100]

[0101] It is a pixel-by-pixel event mask, that is, the value of the pixel position with event output is 1, and the value of the pixel position without event data is 0.

[0102] Attached Figure 5 The blur removal effect of this method on the simulation datasets GoPro (upper two rows) and HQF (lower two rows) is shown; Figure 6 The motion blur removal effect of this method on the real dataset RBE is demonstrated; Figure 7 The image interpolation effect of this method on the simulation datasets GoPro (upper two rows) and HQF (lower two rows) is shown; Figure 8 The image interpolation effect of this method on the real dataset RBE is demonstrated. By comparing the following motion blur removal algorithms:

[0103] (1)LEVS: motion blur removal algorithm based on image frames.

[0104] (2) EDI: Motion blur removal algorithm based on event stream and model drive.

[0105] (3)LEDVDI: Motion blur removal algorithm based on deep learning.

[0106] (4) eSL-Net: Motion blur removal algorithm based on deep learning and sparse reconstruction.

[0107] (5)RED: Motion blur removal algorithm based on deep learning and semi-supervision.

[0108] (6) EVDI(Ours): Motion blur removal algorithm based on deep learning and self-supervision.

[0109] We measured the numerical metrics on the same dataset:

[0110] Table 1. Test results of model motion blur removal

[0111]

[0112] To measure the effect of model image interpolation, we compare the following interpolation algorithms:

[0113] (1) Jin: A fuzzy image interpolation algorithm based on image frames.

[0114] (2) BIN: Blurred image interpolation algorithm based on image frames.

[0115] (3) DAIN: A clear image interpolation algorithm based on image frames.

[0116] (4) EDI: Image interpolation algorithm based on event stream and model drive.

[0117] (5)LEDVDI: A fuzzy image interpolation algorithm based on event stream and deep learning.

[0118] (6) Time Lens: A clear image interpolation algorithm based on event stream and deep learning.

[0119] (7) EVDI(Ours): Image interpolation algorithm based on deep learning and self-supervision.

[0120] The numerical indicators under the same data set are as follows:

[0121] Table 2 Model image insertion test results

[0122]

[0123] The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as measurement criteria in the experiment. The specific definitions of the two are as follows:

[0124] PSNR = 10*log10(255 2 / mean(mean((XY). 2 )))

[0125] SSIM=[L(X,Y) a ]×[C(X,Y) b ]×[S(X,Y) c ]

[0126] in μ X and μ Y represent the means of X and Y respectively, σ X , σ Y and σ XY Represent the variance of X and Y and their covariance respectively. The higher the PSNR and SSIM values ​​are, the better the reconstruction effect is. As shown in the numerical results of Table 1 and Table 2, the present invention shows high performance in the tasks of removing motion blur and image interpolation, which verifies the effectiveness of the present invention.

[0127] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A self-supervised video deblurring and image interpolation method based on event camera, characterized in that: The following steps are involved: Step 1: Construction of blurred video and event stream datasets: Use a single event camera or a combination of an event camera and an ordinary optical camera to shoot blurred videos, and complete the spatiotemporal matching of event streams and image frames, with two adjacent blurred image frames and their corresponding event streams and the target moment to be reconstructed as a set of input data; The set of input data described in step 1 is: Among them, B i and B i+1 are the i-th and i+1-th blurred images of the video, and their corresponding exposure times are For B i Exposure time starts at B i+1 The event stream data collected during the period from the end of the exposure time, where B i Exposure time starts at B i+1 The time period when the exposure time ends is recorded as is the timestamp of its generation time, p(t)∈{+1,-1} is its polarity, x(t)∈[1,W] and y(t)∈[1,H] indicate that its generation position is the xth column and yth row of the camera imaging plane; f is the target time to be reconstructed, which is the time from B i Exposure time starts at B i+1 The time period when the exposure time ends Any time within Step 2: event stream preprocessing: for each set of input data, the event stream is segmented according to the target time to be reconstructed and preprocessed into the network input format; The event stream preprocessing function described in step 2 for: Where f is the target time to be reconstructed; t r is the reference target moment; For the period [f,t r ]Event stream data triggered within; is an event stream processing operator with time displacement, where e(t) = p(t)·δ(t-τ) is the continuous representation of event data and δ(·) is the Dirac function; It is an event stream processing operator that includes time shift, flip, and polarity reversal; Assume the input blurred image B i The exposure start and end time are t1 and t2, B i+1 The exposure start and end time are t3 and t4, then for the input event stream After preprocessing, 4 segments of event data can be obtained As input to subsequent networks; Step 3, constructing a network model, wherein the network model includes a learnable double integral network and a fusion network, and using the constructed network model to process blurred video and event streams to achieve motion blur removal and image interpolation; The learnable double integral network LDI described in step 3 includes a convolutional layer and a ReLU activation function, and the fusion network Fusion includes a convolutional layer, a residual module and an attention module CBAM; The data processing flow after the network is built is as follows: first, the preprocessed event data At a certain time interval Δt The event points are accumulated in each time interval to form an event frame of size 2×H×W, where 2 represents positive and negative polarity, H and W represent the height and width of the event frame, forming a tensor of N×2×H×W that is input into the weight-sharing LDI network and the event flow is calculated by the following formula: Regarding the time f to be reconstructed and the blurred image B i Exposure time The double integral of The exposure starts at t1 and ends at t2; The event stream can also be obtained Regarding the time f to be reconstructed and the blurred image B i+1 Exposure time The double integral of The exposure starts at t3 and ends at t4; Then combine the blurred image B i ,B i+1 Get the preliminary reconstruction result L i (f),L i+1 (f): By weighting L i (f),L i+1 (f) Generate an additional reconstruction result Among them, assuming The weight function ω(f) is defined as: Finally, the preliminary reconstruction result L i (f),L i+1 (f), Double integral results with event flow The cascade is transformed into a tensor and sent to the fusion network for processing, outputting the final reconstruction result. Step 4: Using the relationship between blurry video, clear video and event stream, a self-supervisory framework is constructed to train the network model; The self-supervision framework described in step 4 is: in is a fuzzy-event constraint; for the fuzzy-sharp constraint; is the clear-event constraint; α, β, γ are the corresponding weight scalars; Fuzzy-Event Constraints Defined as: The superscript represents the logarithmic domain, The symbol ||·||1 represents the l1 norm; Fuzzy-Sharp Constraints Defined as: in To generate a blurred image based on the blurred image generation theorem, The generating formula is: in The output of the fusion network at the exposure time The mth potential clear image reconstructed in the inner The generating formula is: Clear-Event Constraints Defined as: in is the difference between two reconstructed clear images in the logarithmic domain; is the result of integrating events between time [f, t]; is the normalization operator, defined as: Where X is the input image, is a pixel-by-pixel event mask, i.e., the value of a pixel at an event output is 1, and the value of a pixel at a pixel without event data is 0; Step 5: Use the trained network model to process the blurred video and event stream and reconstruct a high frame rate clear video.

2. The self-supervised video deblurring and image interpolation method based on event camera according to claim 1, characterized in that: Step 2 also includes performing corresponding transformations according to the pixel positions of the video image frames and the event points to obtain enhanced video and event streams, and the data enhancement techniques include: flipping, rotating, scaling and cropping.

3. The self-supervised video deblurring and image interpolation method based on event camera according to claim 1, characterized in that: The single event camera used in step 1 includes DAVIS346.

Images