An event-driven video interpolation optimization method for low-light scenes

Abstract

The application relates to an event-driven video interpolation optimization method for a low-light scene, and the method comprises the following steps: S1, a pre-training data set is acquired, a degradation event stream is generated based on an intensity-dependent random sampling low-pass filter; S2, an interpolation network is pre-trained based on the degradation event stream and high-frame-rate video data, and a pre-training model is obtained; S3, a real low-light video is acquired and is denoised, the pre-training model is fine-tuned based on the denoised real low-light video, and an interpolation model is obtained; and S4, an actual video to be interpolated is acquired, the actual video to be interpolated is input into the interpolation model, and an interpolation result is obtained. Compared with the prior art, the application has the advantages of improving the interpolation effect of a dark-light video, enhancing robustness and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video interpolation, in particular to an event-driven video interpolation optimization method for low-light scenes. BACKGROUND

[0002] Video interpolation technology plays a crucial role in video quality improvement, frame rate adjustment and slow motion effect production. Event cameras exhibit their advantages in high dynamic range and high frame rate by capturing the logarithmic luminance changes of each pixel. The introduction of events significantly improves the performance of video interpolation technology, especially when recording fast and complex motion scenes. However, event camera applications in low-light environments still face many challenges. For example, noise interference and motion smearing phenomenon, which weaken the performance of event-driven video interpolation methods, limit their robustness and universality.

[0003] Current interpolation methods include optical flow-based video interpolation methods (Super SloMo, RIFE), event-driven video interpolation methods (TimeLens, CBMNet, etc.). However, optical flow-based video interpolation techniques (RIFE, SuperSlomo) have difficulty handling fast nonlinear motion scenes. Event-driven video interpolation techniques (Time Lens, CBMNet) mainly address normal lighting interpolation and blurred image restoration issues, and cannot handle low-light scenes. Under low-light conditions, event cameras will produce smearing effects, and the smearing effect simulated by the existing event simulator (v2e) has a large deviation from the real situation. Videos taken in low-light dynamic scenes have obvious noise. The intermediate frames interpolated by the currently disclosed video interpolation techniques in low-light dynamic scenes cannot overcome noise and cannot simulate smearing effects, resulting in obvious ghosting and white edges, and even some details are distorted and deformed, severely affecting the visual effect. SUMMARY

[0004] The purpose of the present application is to better simulate the smearing effect of event cameras in low-light dynamic scenes, and to provide an event-driven video interpolation optimization method for low-light scenes to improve the interpolation effect in low-light dynamic scenes. The method converts the image to the luminance domain through inverse tone mapping, inverse gamma correction, inverse color correction and inverse white balance, then simulates events, and combines intensity-dependent random sampling low-pass filters to simulate the motion smearing phenomenon of event cameras in low light to generate simulation data. Through pre-training with simulation data, the interpolation effect is improved and the robustness is enhanced in the presence of obvious noise in the interpolated video and obvious smearing in the event stream.

[0005] The purpose of the present application can be achieved by the following technical solutions:

[0006] An event-driven video interpolation optimization method for low-light scenes, the method comprising the following steps:

[0007] S1, obtain a pre-training dataset, interpolate the pre-training dataset to obtain high-frame-rate video data, convert the high-frame-rate video data into a luminance domain signal, pass the luminance domain signal through an intensity-dependent random sampling low-pass filter to simulate the RC circuit response signal of the event camera under low light with trailing phenomenon, and then pass the RC circuit response signal through v2e simulation to add various degradations to obtain a degraded event stream;

[0008] S2, pre-train an interpolation network based on the degraded event stream and the high-frame-rate video data to obtain a pre-training model;

[0009] S3, obtain a real low-light video and denoise it, fine-tune the pre-training model based on the denoised real low-light video to obtain an interpolation model;

[0010] S4, obtain an actual video to be interpolated and an event stream, input the actual video to be interpolated and the event stream into the interpolation model to obtain an interpolation result.

[0011] Further, the RC circuit response of the intensity-dependent random sampling low-pass filter is specifically:

[0012] I t '=RαI t +(1-Rα)I t-Δt

[0013] Wherein, I t ' is the current response, i.e. the simulated event stream generated by the event camera, I t is the current received luminance domain signal, I t-Δt is the response cached in the last frame, R is a random integer sampling mask containing 0-1, and a is a weight factor.

[0014] Further, the weight factor is:

[0015] α=1-e -Δt / τ

[0016] Wherein,

[0017] τ=1 / 2πf c

[0018] f c is the cutoff frequency of the event camera.

[0019] Further, the conversion of the high-frame-rate video data into a luminance domain signal is specifically:

[0020] Inverse tone mapping, inverse gamma correction, inverse color correction and inverse white balance are performed on the high-frequency video data to obtain a luminance domain signal.

[0021] Further, the various types of degradation include readout noise, shot noise, threshold noise, hot pixels, leakage noise and dead time.

[0022] Further, the specific steps of denoising are: using a Restormer network to denoise the real low-light video to obtain a denoised real low-light video.

[0023] Further, the specific steps of fine-tuning the pre-trained model based on the denoised real low-light video to obtain the frame insertion model are:

[0024] Every other frame of the denoised real low-light video is extracted as a supervision frame, and the event stream between the frames not extracted and the continuous frames is input into the pre-trained model to fine-tune the pre-trained model, and the pre-trained model is supervised and trained based on the supervision frame during the fine-tuning process to obtain the frame insertion model.

[0025] Further, the specific steps of inputting the event stream between the frames not extracted and the continuous frames into the pre-trained model, fine-tuning the pre-trained model, and supervising and training the pre-trained model based on the supervision frame during the fine-tuning process are:

[0026] For three continuous frames of images I t-1 , I t and I t+1 in the denoised real low-light video, assuming I t is the extracted supervision frame, I t-1 and I t+1 are the frames not extracted, and the event streams E t-1,t and E t,t+1 are input into the pre-trained model to fine-tune the pre-trained model, and the pre-trained model is supervised and trained based on the supervision frame I t during the fine-tuning process to obtain the frame insertion model.

[0027] Further, the actual video to be inserted is a video denoised.

[0028] Further, the actual video to be inserted is a video obtained under the same environment as the real low-light video.

[0029] Compared with the prior art, the present application has the following beneficial effects:

[0030] (1) The present application generates a large amount of simulation data of motion smearing phenomenon existing in low light by simulating events through inverse tone mapping, inverse gamma correction, inverse color correction, inverse white balance conversion to the luminance domain, and combining intensity-dependent random sampling low-pass filter event camera, pre-trains through a large amount of simulation data, reduces the acquisition burden of training data, improves the frame insertion effect of the model in the case that the inserted frame video exists obvious noise and the event stream exists obvious smearing, and enhances the robustness.

[0031] (2) The present application fine-tunes the event-driven video frame insertion model by frame extraction of the video to obtain the supervised frame and combining the event stream corresponding to the video, and inserts the intermediate frame unknown in the time interval of two consecutive frames of the video on the fine-tuned frame insertion model. In the above process, the present application integrates fine-tuning training in the inference process of the model, so that the frame insertion model can be applied in real low-light frame insertion tasks after pre-training on simulation data, and better adapt to real low-light frame insertion tasks. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The flowchart of the present application;

[0033] Figure 2 The simulation data synthesis flowchart of the present application;

[0034] Figure 3 The comparison chart of real data, simulation data of the present application and simulation data formed by other methods, wherein Figure 3 (a) is a low-light video frame, Figure 3 (b) is a real event, Figure 3 (c) is a simulation event of v2e, Figure 3 (d) is simulation data of the present application;

[0035] Figure 4 The schematic diagram of scene-by-scene fine-tuning of the present application;

[0036] Figure 5 The frame insertion effect comparison chart of the method of the present application and other methods;

[0037] Figure 6 The schematic diagram of event smearing phenomenon in low light. DETAILED DESCRIPTION

[0038] The present application will be described in detail below in combination with the drawings and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present application, and detailed implementation and specific operation process are given, but the protection scope of the present application is not limited to the following embodiments.

[0039] In order to solve the problem that in low light, the event signal will produce obvious motion tailing phenomenon, and the degradation will usually cause obvious artifacts and detail damage in video interpolation, and affect the final video viewing experience, the present application provides an event-driven video interpolation optimization method for low light scene, and a flowchart of the method is as shown in Figure 1 The method of the present application simulates low light event tailing degradation, and can generate a large amount of simulation data for pre-training of a deep learning interpolation model. During the pre-training process, the image is converted to the brightness domain through inverse tone mapping, gamma correction, color correction, white balance, and then simulated events are generated, and the motion tailing phenomenon of the event camera in low light is simulated by using the above-mentioned intensity-dependent random sampling low-pass filter (IDS-LPF). The simulation data is generated by using the event data synthesis method, and the video interpolation model is pre-trained.

[0040] The event data synthesis (data synthesis pipeline) is as shown in Figure 2 The clear frame of the Gopro dataset is interpolated by the RIFE interpolation model to obtain a high frame rate video. The video is converted to the brightness domain by using inverse tone mapping, inverse gamma correction, inverse color correction and inverse white balance in the inverse ISP operation, the tailing effect of the event camera in low light is simulated by combining the IDS-LPF, and the event stream is generated by using v2e, and the readout noise, shot noise, threshold noise, thermal pixel noise, leakage noise events and refractory period are added in the process of generating the event stream by using v2e. The above process is the process of synthesizing the event stream by video.

[0041] During the interpolation process, the interpolation model is first fine-tuned scene by scene, and then inference is performed. The fine-tuning process is embedded in the interpolation process, which improves the generalization ability of the model to different scenes and camera settings, and the time consumption remains in the same order of magnitude as direct inference. The event-driven video interpolation model is fine-tuned by extracting supervision frames from the video and combining the event stream corresponding to the video. The unknown intermediate frame in the time interval of two consecutive frames of the video is interpolated on the fine-tuned interpolation model. The event tailing phenomenon in low light is as shown in Figure 6 In the present application, the video is pre-de-noised before low light video interpolation, so as to reduce the negative influence of noise on the interpolation process.

[0042] The application makes the simulated low-light event stream closer to the real situation by designing a new filter and a data synthesis pipeline, and better frame insertion effect can be achieved based on the pre-training of the frame insertion model on the more realistic simulated data. First, the pre-noise removal module is used to pre-noise the low-light video, solving the noise problem caused by short exposure of high frame rate video. Then, the pre-trained frame insertion model is fine-tuned and optimized scene by scene. During the fine-tuning process, the complete event stream and the video extracted frame are used as input, and the extracted frame is used as supervision to build data pairs to train the frame insertion model. The training time of scene-by-scene optimization is less than the inference time, and it can be integrated into the model inference process, so that the model pre-trained on simulation data can be generalized on real data.

[0043] The method of the application comprises the following steps:

[0044] S1, obtain a pre-training data set, perform frame insertion on the pre-training data set to obtain high frame rate video data, convert the high frame rate video data into a luminance domain signal, pass the luminance domain signal through an intensity-dependent random sampling low-pass filter to simulate the RC circuit response signal of the event camera under low light with trailing phenomenon, and then pass the RC circuit response signal through v2e simulation to add various degradations (readout noise, shot noise, threshold noise, hot pixel noise, leakage noise events and refractory period) to obtain a final degraded event stream;

[0045] S2, pre-train a frame insertion network based on the degraded event stream and the high frame rate video data to obtain a pre-training model;

[0046] S3, obtain a real low-light video and perform denoising, fine-tune the pre-training model based on the denoised real low-light video to obtain a frame insertion model;

[0047] S4, obtain an actual video to be inserted and an event stream, input the actual video to be inserted and the event stream into the frame insertion model to obtain a frame insertion result.

[0048] The simulation data of the application, i.e. the synthesis flowchart of the degraded event stream, is as follows: Figure 2The core of the present application includes the pre-training of S1 and S2 and the scene-by-scene optimization (fine-tuning) of S3. In the pre-training stage, the simulated event stream is generated based on the low-light event simulation method introduced in the present application. In the present application, a new intensity-dependent random sampling low-pass filter (IDS-LPF) is integrated into the simulation process, making the simulated dark light event tail closer to the actual shooting situation. The high-frame-rate video and its corresponding simulated event stream are used to pre-train the video interpolation network. Then, on the pre-trained model, real data is used for scene-by-scene optimization. The real data is first pre-de-noised (using the Restormer network) to obtain the de-noised video, and then the frame skipping is used as the supervision frame, which is combined with the real low-light event stream to fine-tune the pre-trained model. Finally, the fine-tuned model is used to interpolate the input low-frame-rate video with the help of events.

[0049] In the pre-training process, the pre-training data set is first interpolated by 8 times by the RIFE method to obtain higher frame rate video data. The video is converted to the luminance domain through inverse tone mapping, inverse gamma correction, inverse color correction, and inverse white balance, and then passes through a specially designed intensity-dependent random sampling low-pass filter (IDS-LPF) to simulate the response of the event camera in low light:

[0050] I t ' = RαI t + (1-Rα)I t-Δt

[0051] where I t ' is the current response, I t is the current received luminance signal, I t-Δt is the cached response of the last frame, R is a random integer sampling mask containing 0-1, and α is a weight factor, α = 1-e -Δt / τ , τ = 1 / 2πf c , f c is the cutoff frequency of the event camera. Then the event stream with degradation such as readout noise, shot noise, threshold noise, hot pixels, leakage noise, and refractory period is simulated by v2e.

[0052] Figure 3 The comparison chart of real data, simulated data of the present application, and simulated data formed by other methods. In low light, the simulated events of the present application are closer to the real events in visual effect.

[0053] The schematic diagram of scene-by-scene optimization is shown in Figure 4 For real continuous frames, the middle frame is extracted (every other frame is extracted) to fine-tune the training interpolation network. The extracted frame is the supervision signal, and the unextracted frame is the input signal. As Figure Four , It-1 and I t+1 is an input frame, E t-1,t and E t,t+1 is an input event stream. I t is an extracted intermediate frame, which is used as a target image during fine-tuning, supervising the training of the network. Since the fine-tuning time consumption is close to the inference time consumption, the fine-tuning is integrated into the inference process of the model, so that the frame insertion model can be applied in real low-light frame insertion tasks after being pre-trained on simulation data.

[0054] The frame insertion model of the present application is event-driven, requiring time-space aligned event streams and videos. Real data can be obtained by using an event-video split-light shooting system. The split-light prism in the system is used to ensure that the event camera and the color camera have the same field of view, and the image registration algorithm is further used to ensure the spatial alignment of the shooting pictures. The synchronous trigger is used to ensure the time alignment.

[0055] The present application mainly solves the problem that the existing event-driven frame insertion method performs poorly in low light (the inserted intermediate frame has obvious ghosting and white edges, and some details are even distorted and deformed). In the case that the inserted video has obvious noise and the event stream has obvious trailing, the present application improves the detail texture performance and visual effect of low-light video frame insertion, improves the frame insertion effect, and enhances the robustness.

[0056] Based on the real dataset obtained by the event-video split-light shooting system, the effectiveness and advancement of the present application in the low-light video frame insertion task are verified by experiments. The frame insertion effect comparison chart of the method of the present application and other methods is shown in Figure 5 . It can be seen that the method of the present application is superior to the current optical flow-based frame insertion methods (RIFE, SuperSloMo) and event-based frame insertion methods (Time Lens, CBMNet) in visual effect.

[0057] The above describes the preferred embodiments of the present application in detail. It should be understood that those skilled in the art can make many modifications and changes to the present application without creative labor according to the concept of the present application. Therefore, any technical solution obtained by logical analysis, reasoning or limited experiments by those skilled in the art on the basis of the prior art according to the concept of the present application shall be within the protection scope determined by the claims.

Claims

1. An event-driven video frame interpolation optimization method for low-light scenes, characterized in that, The method includes the following steps: S1. Obtain a pre-training dataset, interpolate frames in the pre-training dataset to obtain high frame rate video data, convert the high frame rate video data into a luminance domain signal, pass the luminance domain signal through an intensity-dependent random sampling low-pass filter to simulate the RC circuit response signal of the event camera under low light with a trailing phenomenon, and then add various degradations to the RC circuit response signal through V2E simulation to obtain a degradation event stream. S2. The frame interpolation network is pre-trained based on the degradation event stream and high frame rate video data to obtain a pre-trained model; S3. Obtain real low-light video and denoise it. Fine-tune the pre-trained model based on the denoised real low-light video to obtain the frame interpolation model. S4. Obtain the actual video to be interpolated and the event stream, input the actual video to be interpolated and the event stream into the interpolation model, and obtain the interpolation result; The specific RC circuit response of the intensity-dependent random sampling low-pass filter is as follows: I t '=RαI t +(1-Rα)I t-Δt Among them, I t 'I' is the current response, i.e., the simulated event stream generated by the event camera. t It is the currently received luminance domain signal, I t-Δt It is the response of the previous frame buffer, R is a random integer sample mask containing 0-1, and α is a weighting factor; The weighting factor is: α＝1-e -Δt / τ in, τ=1 / 2πf c f c It is the cutoff frequency of the event camera; The specific steps of converting the high frame rate video data into a luminance domain signal are as follows: The high frame rate video data is subjected to inverse tone mapping, inverse gamma correction, inverse color correction, and inverse white balance to obtain a luminance domain signal.

2. The event-driven video frame interpolation optimization method for low-light scenes according to claim 1, characterized in that, The various types of degradation include readout noise, shot noise, threshold noise, hot pixels, leakage noise, and refractory period.

3. The event-driven video frame interpolation optimization method for low-light scenes according to claim 1, characterized in that, The specific steps of the denoising are as follows: the real low-light video is denoised using the Restormer network to obtain the denoised real low-light video.

4. The event-driven video frame interpolation optimization method for low-light scenes according to claim 3, characterized in that, The specific steps for fine-tuning the pre-trained model based on the denoised real low-light video to obtain the frame interpolation model are as follows: For the denoised real low-light video, one frame is extracted every other frame as a supervision frame. The event stream between the unextracted frames and consecutive frames is input into the pre-trained model. The pre-trained model is fine-tuned. During the fine-tuning process, supervised training is performed based on the supervision frames to obtain the frame interpolation model.

5. The event-driven video frame interpolation optimization method for low-light scenes according to claim 4, characterized in that, The specific steps for inputting the event stream between the unextracted frames and consecutive frames into the pre-trained model, and fine-tuning the pre-trained model, including supervised training based on supervised frames during the fine-tuning process, are as follows: For three consecutive frames I in a real low-light video after denoising t-1 I t and I t+1 Assume I t For the extracted supervision frame, then I t-1 and I t+1 For the unextracted frames, the unextracted frames, and the event stream E t-1,t and E t,t+1 The input is fed into the pre-trained model, and the pre-trained model is fine-tuned. During the fine-tuning process, it is based on the supervised frame I. t Supervised training is performed to obtain the frame interpolation model.

6. The event-driven video frame interpolation optimization method for low-light scenes according to claim 1, characterized in that, The actual video to be interpolated is a denoised video.

7. The event-driven video frame interpolation optimization method for low-light scenes according to claim 6, characterized in that, The actual video to be interpolated is a video obtained under the same environment as the real low-light video.

Images