A stereoscopic event and intensity camera based video interpolation method and system

Abstract

The application provides a video interpolation method and system based on a stereo event and intensity camera, and belongs to the technical field of image processing, and comprises the following steps: collecting original image frames and original event streams; extracting any continuous image frame in the original image frames, determining an interframe event stream based on the any continuous image frame; inputting the any continuous image frame and the interframe event stream into a convolutional neural network for training, and obtaining a video interpolation model based on a loss function constructed by an interpolation result obtained through model training and a real image; and determining that a to-be-processed image frame and a to-be-processed interframe event stream are input into the video interpolation model to obtain a video interpolation result. The application comprehensively utilizes an intensity camera and an event camera based on a convolutional neural network to perform video interpolation, and realizes high-quality stereo video interpolation under the condition that two types of data are not spatially aligned.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a video frame interpolation method and system based on stereo events and intensity cameras. Background Technology

[0002] Video frame interpolation is an application of computer vision technology in the field of video enhancement. It aims to use consecutive input image frames to generate missing intermediate image frames, thereby overcoming the limitations of the low sampling frequency of traditional cameras and increasing the video frequency to obtain a smoother and more fluid viewing experience.

[0003] Event cameras, with their extremely high temporal resolution, can provide missing texture and motion information between image frames, significantly improving the performance of video frame interpolation algorithms. However, current event-based video frame interpolation methods typically rely on the assumption of pixel-level spatial alignment between the event stream and image frames. In practical applications, however, separate event and intensity cameras are often required for data acquisition, leading to parallax between the event stream and image frames. This parallax results in significant distortion and artifacts in the interpolation results, causing a substantial decrease in algorithm performance. Although event cameras such as Dynamic and Active Vision Sensors (DAVIS) exist, which can simultaneously acquire aligned event streams and image frames using a single sensor, the images captured by these cameras suffer from low resolution and low dynamic range, hindering the widespread adoption of existing methods in practical applications. Therefore, most applications capture data using stereo event and intensity cameras. Homography variations and stereo correction methods can roughly solve the spatial alignment problem of stereo cameras, but these methods are ineffective in solving alignment problems in dynamic scenes with complex motion and depth variations.

[0004] Traditional stereo camera matching algorithms rely on two simultaneously captured image frames to estimate disparity data. However, there is a serious coupling problem in stereo video frame interpolation tasks: missing intermediate image frames and corresponding event streams are needed to estimate disparity, and disparity is also needed to assist in the data alignment of the stereo camera and further guide the generation of intermediate frames. Summary of the Invention

[0005] This invention provides a video frame interpolation method and system based on stereo events and intensity cameras, which solves the defect in the prior art where data cannot be aligned during video frame interpolation, resulting in unsatisfactory processing results.

[0006] In a first aspect, the present invention provides a video frame interpolation method based on stereo events and intensity cameras, comprising:

[0007] Acquire raw image frames and raw event streams;

[0008] Extract any consecutive image frames from the original image frames, and determine the inter-frame event stream based on the arbitrary consecutive image frames;

[0009] The arbitrary continuous image frames and the inter-frame event stream are input into a convolutional neural network for training, and a video frame interpolation model is obtained by constructing a loss function based on the frame interpolation results obtained from the model training and the real images.

[0010] The image frame to be processed and the inter-frame event stream to be processed are input into the video interpolation model to obtain the video interpolation result.

[0011] According to the present invention, a video frame interpolation method based on stereo events and intensity cameras is provided, wherein the arbitrary consecutive image frames and the inter-frame event stream are input into a convolutional neural network for training, and a video frame interpolation model is obtained based on the frame interpolation results obtained from the model training and a loss function constructed from real images, comprising:

[0012] The arbitrary consecutive image frames and the inter-frame event stream are input into a bidirectional optical flow network to obtain coarse frame interpolation results and bidirectional optical flow.

[0013] The coarse frame interpolation result and the bidirectional optical flow are input into the fusion network to obtain the fused frame interpolation result;

[0014] The fused frame interpolation result is input into the refining network, and the optimized frame interpolation result is output.

[0015] Based on the optimized interpolation results and the real image, a comprehensive loss function is constructed to obtain the video interpolation model.

[0016] According to the present invention, a video frame interpolation method based on a stereo event and intensity camera is provided, wherein the arbitrary consecutive image frames and the inter-frame event stream are input into a bidirectional optical flow network to obtain a coarse frame interpolation result and a bidirectional optical flow, comprising:

[0017] Image feature information is extracted from any consecutive image frames using an image frame encoder, and event feature information is extracted from the inter-frame event stream using an event stream encoder.

[0018] The image feature information and the event feature information are fused and aligned using a feature fusion attention module to output the bidirectional optical flow;

[0019] The coarse frame interpolation result is obtained by transforming the arbitrary consecutive image frames using the bidirectional optical flow.

[0020] According to the present invention, a video frame interpolation method based on a stereo event and intensity camera is provided, wherein the coarse frame interpolation result and the bidirectional optical flow are input into a fusion network to obtain a fused frame interpolation result, comprising:

[0021] Determine two pixel-level masks corresponding to the arbitrary consecutive image frames;

[0022] The two pixel-level masks are multiplied with the bidirectional optical flow at the pixel level to obtain the fused frame interpolation result.

[0023] According to the present invention, a video frame interpolation method based on stereo event and intensity camera is provided, wherein the fused frame interpolation result is input into a refining network and the optimized frame interpolation result is output, comprising:

[0024] The fused interpolation result is input into a residual dense network to obtain the optimized interpolation result composed of the fused interpolation result and the output residual.

[0025] According to the present invention, a video frame interpolation method based on a stereo event and intensity camera is provided, wherein constructing a comprehensive loss function based on the optimized frame interpolation result and the real image includes:

[0026] An L1 norm loss function is used to construct a reconstruction loss function based on the fused interpolation result, the optimized interpolation result, and the real image.

[0027] A smoothed L1 norm loss is employed to construct an image display loss function based on feature disparity and true disparity.

[0028] An event flow loss function is constructed based on the optimized insertion results and the bidirectional optical flow, employing photometric consistency loss and optical flow smoothness loss.

[0029] The combined loss function is obtained by weighted summation of the reconstruction loss function, the image display loss function, and the event flow loss function.

[0030] According to the present invention, a video frame interpolation method based on stereo events and intensity cameras, wherein determining the image frame to be processed and the inter-frame event stream to be processed are input into the video frame interpolation model to obtain the video frame interpolation result includes:

[0031] Obtain any consecutive image frames in the image frame to be processed, and extract the inter-frame event stream between the exposure times of the image frame to be processed;

[0032] The arbitrary consecutive image frames and the inter-frame event stream are input into the video interpolation model, and the video interpolation result is output.

[0033] Secondly, the present invention also provides a video frame interpolation system based on stereo events and intensity cameras, comprising:

[0034] The acquisition module is used to acquire raw image frames and raw event streams;

[0035] The extraction module is used to extract any consecutive image frames from the original image frames and determine the inter-frame event stream based on the arbitrary consecutive image frames;

[0036] The training module is used to input the arbitrary continuous image frames and the inter-frame event stream into the convolutional neural network for training, and to obtain a video frame interpolation model based on the frame interpolation results obtained from the model training and the loss function constructed from the real images.

[0037] The processing module is used to determine the image frame to be processed and the inter-frame event stream to be processed, and input them into the video interpolation model to obtain the video interpolation result.

[0038] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video frame interpolation method based on stereo events and intensity cameras as described above.

[0039] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video frame interpolation method based on stereo events and intensity cameras as described above.

[0040] The video interpolation method and system based on stereo event and intensity cameras provided by this invention achieve high-quality stereo video interpolation under spatial misalignment conditions by comprehensively utilizing intensity cameras and event cameras based on convolutional neural networks. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is one of the flowcharts illustrating the video frame interpolation method based on stereo events and intensity cameras provided by the present invention;

[0043] Figure 2 This is a schematic diagram of the stereo camera assembly provided by the present invention;

[0044] Figure 3 This is the second flowchart illustrating the video frame interpolation method based on stereo events and intensity cameras provided by this invention.

[0045] Figure 4 This is a schematic diagram of the neural network structure provided by the present invention;

[0046] Figure 5 This is a comparison chart of frame interpolation results for the DSEC dataset provided by this invention;

[0047] Figure 6 This is a comparison chart of frame interpolation results for the MVSEC dataset provided by this invention;

[0048] Figure 7 This is a schematic diagram of the video frame interpolation system based on stereo events and intensity cameras provided by the present invention;

[0049] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] Figure 1 This is one of the flowcharts illustrating the video frame interpolation method based on stereo events and intensity cameras provided in this embodiment of the invention, such as... Figure 1 As shown, it includes:

[0052] Step 100: Acquire raw image frames and raw event stream;

[0053] Step 200: Extract any consecutive image frames from the original image frames, and determine the inter-frame event stream based on the arbitrary consecutive image frames;

[0054] Step 300: Input the arbitrary continuous image frames and the inter-frame event stream into the convolutional neural network for training, and obtain the video frame interpolation model based on the frame interpolation results obtained from the model training and the loss function constructed from the real images;

[0055] Step 400: Determine the image frame to be processed and the inter-frame event stream to be processed, and input them into the video interpolation model to obtain the video interpolation result.

[0056] This invention utilizes a stereo camera group consisting of an event camera and an intensity camera, such as... Figure 2 As shown, dual-view shooting of the target scene can be achieved, making full use of the connection between event stream data and intensity images. By combining the depth and parallax information between the two viewpoints, the video frame interpolation problem under stereo event and intensity camera settings is transformed into a fine optical flow estimation and distortion image fusion model. High-quality interpolated image results are reconstructed by solving the energy minimization problem under the loss function constraint.

[0057] It should be noted that this invention first acquires the event streams corresponding to the intensity images and image frames continuously captured by the stereo camera and the intensity camera output; then, it constructs and trains a convolutional neural network, in which a dual encoder extracts feature information from the input intensity image and event stream respectively, and uses a feature fusion attention module to extract the disparity relationship between the intensity image and the event stream, associating and fusing deep multimodal feature information to obtain bidirectional fine optical flow. The fusion network performs mask estimation on the input optical flow-distorted image, and uses pixel-level fusion to eliminate artifacts and holes. The refinement network further refines the fused image to obtain the final high-quality frame interpolation result. The high-quality frame interpolation result obtained by the frame interpolation algorithm is combined with the real high-quality image to construct a loss function, and the convolutional neural network is trained through an adaptive momentum (ADAM) iterative optimizer; finally, the corresponding event streams between the intensity image and image frames are used to obtain the reconstructed high-quality frame interpolation image through the trained frame interpolation model.

[0058] The process is as follows Figure 3 As shown, the intensity image is first acquired by the image frame encoder, the event stream is acquired by the event stream encoder, the intensity image and the event stream are then fused by the feature fusion attention module, and then processed by the optical flow decoder. After optical flow distortion is performed on the intensity image, the image is then processed by the fusion network and the refinement network in sequence to obtain the final frame interpolation result.

[0059] Specifically, firstly, image frames are acquired using an intensity camera, and simultaneously, an event stream is continuously acquired using an event camera. Then, any consecutive image frames I0,1 captured by the intensity camera are selected, where I0 represents the image frame at time 0 and I1 represents the image frame at time 1. Based on the image frame timestamps, the event stream E excited between the two frames at time 0 and time 1 is selected. 0→1 The data needs to be preprocessed. For the expected normalized interpolation time t (t∈[0,1]), the event stream is divided into E... t→0 E t→1 Two parts, of which E t→0 E represents the inverse representation of the event stream triggered from time 0 to time t. t→1 This represents the event stream triggered from time t to time 1. The two image frames and the two event streams are then input into a convolutional neural network to generate a high-quality frame interpolation result I. t Then, using the interpolation results I obtained from the network t With real image I gt A loss function is constructed, and a perfect video frame interpolation model is obtained through iterative training. Finally, the input image frames and the corresponding inter-frame event stream data are input into the trained video frame interpolation model to generate high-quality intermediate frames, thereby improving the frame rate of the input video.

[0060] This invention studies video frame interpolation technology based on stereo event and intensity cameras, associates and fuses high temporal resolution event data with parallax with low frame rate intensity images, and generates missing intermediate image frames. This has important value for the application research of event cameras in real-world scenarios.

[0061] Based on the above embodiments, step 300 includes:

[0062] The arbitrary consecutive image frames and the inter-frame event stream are input into a bidirectional optical flow network to obtain coarse frame interpolation results and bidirectional optical flow.

[0063] The coarse frame interpolation result and the bidirectional optical flow are input into the fusion network to obtain the fused frame interpolation result;

[0064] The fused frame interpolation result is input into the refining network, and the optimized frame interpolation result is output.

[0065] Based on the optimized interpolation results and the real image, a comprehensive loss function is constructed to obtain the video interpolation model.

[0066] The step of inputting the arbitrary consecutive image frames and the inter-frame event stream into a bidirectional optical flow network to obtain coarse frame interpolation results and bidirectional optical flow includes:

[0067] Image feature information is extracted from any consecutive image frames using an image frame encoder, and event feature information is extracted from the inter-frame event stream using an event stream encoder.

[0068] The image feature information and the event feature information are fused and aligned using a feature fusion attention module to output the bidirectional optical flow;

[0069] The coarse frame interpolation result is obtained by transforming the arbitrary consecutive image frames using the bidirectional optical flow.

[0070] The step of inputting the coarse frame interpolation result and the bidirectional optical flow into the fusion network to obtain the fused frame interpolation result includes:

[0071] Determine two pixel-level masks corresponding to the arbitrary consecutive image frames;

[0072] The two pixel-level masks are multiplied with the bidirectional optical flow at the pixel level to obtain the fused frame interpolation result.

[0073] The step of inputting the fused frame interpolation result into the refining network and outputting the optimized frame interpolation result includes:

[0074] The fused interpolation result is input into a residual dense network to obtain the optimized interpolation result composed of the fused interpolation result and the output residual.

[0075] The step of constructing a comprehensive loss function based on the optimized post-interpolation result and the real image includes:

[0076] An L1 norm loss function is constructed based on the fused interpolation result, the optimized interpolation result, and the real image;

[0077] A smoothed L1 norm loss is employed to construct an image display loss function based on feature disparity and true disparity.

[0078] An event flow loss function is constructed based on the optimized insertion results and the bidirectional optical flow, employing photometric consistency loss and optical flow smoothness loss.

[0079] The combined loss function is obtained by weighted summation of the reconstruction loss function, the image display loss function, and the event flow loss function.

[0080] Specifically, the neural network structure involved in this invention is as follows: Figure 4 As shown, the network comprises three sub-networks: Bi-directional FlowNet, FusionNet, and RefineNet. The Bi-directional FlowNet includes an image frame encoder, an event stream encoder, a Feature FusionAttention Module (FFAM), and an optical flow decoder. The FusionNet includes optical flow warping operations, a channel block attention module, and two masks. The RefineNet includes a residual dense network. Here, the Bi-directional FlowNet is built based on an encoder-decoder structure. We set up two encoders with non-shared weights to extract feature information from the image frame and the event stream respectively. Feature fusion attention modules are set at the outputs of each stage of the encoder to fuse and align the features extracted by the encoder.

[0081] Specifically, for features extracted by dual encoders in This represents the extracted image frame features. The extracted event stream features are first input into the Spatial Attention Module (SAM), where spatial attention is extracted using parallel convolutional kernels of different sizes. The extracted features are then input into the Feature Disparity Estimate Module (FDEM) to estimate the disparity between the two feature classes. Where Ω f Ω eLet represent the imaging planes of the intensity camera and the event camera, respectively. Through disparity estimation, spatial alignment of the two types of features is achieved using a disparity transformation function. Furthermore, the aligned features are used to estimate the bidirectional optical flow, expressed as:

[0082]

[0083] in, Represents the image frame at time t Image frame at time 0 The optical flow between them Represents the image frame at time t Image frame at time 1 The optical flow between them, FlowNet represents the bidirectional optical flow network function, and t represents the normalized representation of the target interpolation time. This represents the image frame captured by the intensity camera at time 0. This represents the image frame captured by the intensity camera at time 1. This represents the inverted representation of the event data captured by the event camera from time 0 to time t. This represents the event data captured by the event camera from time t to time 1.

[0084] After aligning the data and predicting the optical flow using a bidirectional optical flow network, the input image frames can be transformed using the optical flow to obtain two coarse frame interpolation results. However, when objects in the input image frame undergo significant motion, the interpolation results obtained through optical flow transformation often exhibit severe distortion at the object and image edges. To address this issue, this invention designs a fusion network that uses coarse interpolation results from optical flow transformation and bidirectional optical flow as input. It focuses on the sharp content within the two coarse results, predicting two pixel-level masks for subsequent interpolation result fusion, as shown below:

[0085]

[0086]

[0087] Where ⊙ represents a pixel-level multiplication operation. This represents the output of the fusion network. This represents the pixel-level mask at time 0 corresponding to the output of the fusion network. This represents the pixel-level mask at time 1 corresponding to the output of the fusion network. FusionNet represents the fusion network function used to estimate the pixel-level mask. Indicates the transmission of light through optical flow Will Image frames obtained by transforming from time 0 to time t Indicates the transmission of light through optical flow Will The image frame obtained by transforming from time 1 to time t.

[0088] The interpolation results obtained through the fusion network eliminate parallax to some extent, but blurring and imperfections still exist in the details. Therefore, this invention sets up a refinement network that uses a residual dense network to receive the fused image as input and calculates the residuals to further optimize the details of the interpolation results, expressed as:

[0089]

[0090]

[0091] in, I represents the residual of the refined network output. t This represents the optimized frame interpolation result. Represents the output of the fusion network; RefineNet represents the refined network function. This represents the inverted representation of the event data captured by the event camera from time 0 to time t. This represents the event data captured by the event camera from time t to time 1.

[0092] During the model training process of this invention, the predicted disparity bidirectional optical flow Fusion results and the final frame interpolation result I t The loss function is calculated, and it consists of the following three parts:

[0093]

[0094]

[0095]

[0096] in, Represents the reconstruction loss function. This represents the image display loss function. Represents the event stream loss function. I represents the output of the fusion network. gt Represents a real image, I t This represents the optimized frame interpolation result. I represents the error between the two predicted features, where 0 represents the image frame at time 0, and I1 represents the image frame at time 1. t Represents the image frame at normalized interpolation time t. Represents the image frame at time t Image frame at time 0 The optical flow between them Represents the image frame at time t Image frame at time 1 The optical flow between them Indicates L1 norm loss, D represents the smooth L1 norm loss. gt For true parallax data, Indicates loss of photometric uniformity. This represents the smoothness loss of the optical flow. The final loss function is a weighted sum of the above terms, expressed as:

[0097]

[0098] Where λ r , λ d and λ f The weights for the three loss functions—reconstruction loss function, image display loss function, and event flow loss function—are defined respectively. Let be the comprehensive loss function, and let the weight vector be: [λ r ,λ d ,λ f = [2, 0.01, 0.01].

[0099] Finally, the input data, which requires selecting continuous intensity image frames and event stream data between the exposure times of the image frames, is input into the trained model for frame interpolation to obtain high-quality reconstructed images.

[0100] Based on the above embodiments, the present invention has also been verified on different datasets using different algorithms, wherein... Figure 5 This is a comparison chart of the frame interpolation results of the DSEC dataset provided by this invention. From top left to bottom right, the chart shows the overlapping diagram of the two input image frames, the image frame interpolation algorithm based on image frames that considers depth information (DAIN), the real-time video frame interpolation algorithm based on intermediate stream estimation (RIFE), the video frame interpolation algorithm through residual subdivision (RRIN), the video frame interpolation algorithm based on high-quality intermediate frame estimation (SuperSlomo), the event-based video frame interpolation algorithm (Time Lens), the stereo event video frame interpolation algorithm based on this invention (SE-VFI), and the real high-quality image frame as a reference. Table 1 shows the comparison data of the frame interpolation results of the DSEC dataset.

[0101] Table 1

[0102]

[0103] Figure 6This is a comparison chart of the frame interpolation results of the MVSEC dataset provided by this invention. From top left to bottom right, the chart shows the overlapping diagram of the two input image frames, the image frame interpolation algorithm based on image frames that considers depth information (DAIN), the real-time video frame interpolation algorithm based on intermediate stream estimation (RIFE), the video frame interpolation algorithm through residual subdivision (RRIN), the video frame interpolation algorithm based on high-quality intermediate frame estimation (SuperSlomo), the event-based video frame interpolation algorithm (Time Lens), the stereo event video frame interpolation algorithm based on this invention (SE-VFI), and the real high-quality image frame as a reference. Table 2 shows the comparison of the frame interpolation results metrics of the MVSEC dataset.

[0104] Table 2

[0105]

[0106] It should be noted that in experiments with reference images, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) were used as metrics, and their specific definitions are as follows:

[0107] PSNR = 10 * log10(255) 2 / mean(mean((XY)) 2 )))

[0108] SSIM = [L(X,Y)] a ]×[C(X,Y) b ]×[S(X,Y) c ]

[0109] in `mean` is the function to calculate the mean of an array, where X and Y are any two objects being compared, and μ... X and μ Y Let σ represent the mean of X and Y, respectively. X σ Y and σ XY Let C1, C2, and C3 represent the variances and covariances of X and Y, respectively. Additionally, C1, C2, and C3 are adjustment constants in L(X,Y), C(X,Y), and S(X,Y), respectively. To avoid systematic errors caused by a denominator of 0, a, b, and c are exponents. In practical applications, the exponents a = b = c = 1, and C3 = 0.5 × C2. The higher the PSNR and SSIM values, the better the reconstruction effect.

[0110] The video frame interpolation system based on stereo events and intensity cameras provided by the present invention is described below. The video frame interpolation system based on stereo events and intensity cameras described below can be referred to in correspondence with the video frame interpolation method based on stereo events and intensity cameras described above.

[0111] Figure 7 This is a schematic diagram of the structure of a video frame interpolation system based on a stereo event and intensity camera provided in an embodiment of the present invention, as shown below. Figure 7 As shown, it includes: a data acquisition module 71, an extraction module 72, a training module 73, and a processing module 74, wherein:

[0112] The acquisition module 71 is used to acquire original image frames and original event streams; the extraction module 72 is used to extract any consecutive image frames from the original image frames and determine the inter-frame event stream based on the arbitrary consecutive image frames; the training module 73 is used to input the arbitrary consecutive image frames and the inter-frame event stream into a convolutional neural network for training, and obtain a video frame interpolation model based on the frame interpolation results obtained from the model training and the loss function constructed from the real images; the processing module 74 is used to determine the image frames to be processed and the inter-frame event stream to be processed and input them into the video frame interpolation model to obtain the video frame interpolation result.

[0113] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a video frame interpolation method based on stereo events and an intensity camera. The method includes: acquiring original image frames and an original event stream; extracting arbitrary consecutive image frames from the original image frames, and determining an inter-frame event stream based on the arbitrary consecutive image frames; inputting the arbitrary consecutive image frames and the inter-frame event stream into a convolutional neural network for training, and obtaining a video frame interpolation model based on the frame interpolation results obtained from the model training and a loss function constructed from real images; determining the image frames to be processed and the inter-frame event stream to be processed and inputting them into the video frame interpolation model to obtain the video frame interpolation result.

[0114] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video interpolation method based on stereo events and intensity cameras provided by the above methods. The method includes: acquiring original image frames and original event streams; extracting arbitrary consecutive image frames from the original image frames and determining inter-frame event streams based on the arbitrary consecutive image frames; inputting the arbitrary consecutive image frames and the inter-frame event streams into a convolutional neural network for training, and obtaining a video interpolation model based on the interpolation results obtained from the model training and a loss function constructed from real images; determining the image frames to be processed and the inter-frame event streams to be processed and inputting them into the video interpolation model to obtain video interpolation results.

[0116] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a video interpolation method based on a stereo event and intensity camera provided by the methods described above. This method includes: acquiring original image frames and an original event stream; extracting arbitrary consecutive image frames from the original image frames and determining an inter-frame event stream based on the arbitrary consecutive image frames; inputting the arbitrary consecutive image frames and the inter-frame event stream into a convolutional neural network for training, and obtaining a video interpolation model based on the interpolation results obtained from the model training and a loss function constructed from real images; determining the image frames to be processed and the inter-frame event stream to be processed and inputting them into the video interpolation model to obtain a video interpolation result.

[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video frame interpolation method based on stereo events and intensity cameras, characterized in that, include: Acquire raw image frames and raw event streams; Extract any consecutive image frames from the original image frames, and determine the inter-frame event stream based on the arbitrary consecutive image frames; The arbitrary continuous image frames and the inter-frame event stream are input into a convolutional neural network for training, and a video frame interpolation model is obtained by constructing a loss function based on the frame interpolation results obtained from the model training and the real images. The image frame to be processed and the inter-frame event stream to be processed are input into the video interpolation model to obtain the video interpolation result; The step of inputting the arbitrary consecutive image frames and the inter-frame event stream into a convolutional neural network for training, and obtaining a video frame interpolation model based on the frame interpolation results obtained from the model training and a loss function constructed from real images, includes: The arbitrary consecutive image frames and the inter-frame event stream are input into a bidirectional optical flow network to obtain coarse frame interpolation results and bidirectional optical flow. The coarse frame interpolation result and the bidirectional optical flow are input into the fusion network to obtain the fused frame interpolation result; The fused frame interpolation result is input into the refining network, and the optimized frame interpolation result is output. Based on the optimized frame interpolation results and the real image, a comprehensive loss function is constructed to obtain the video frame interpolation model; The step of inputting the arbitrary consecutive image frames and the inter-frame event stream into a bidirectional optical flow network to obtain coarse frame interpolation results and bidirectional optical flow includes: Image feature information is extracted from any consecutive image frames using an image frame encoder, and event feature information is extracted from the inter-frame event stream using an event stream encoder. The image feature information and the event feature information are fused and aligned using a feature fusion attention module to output the bidirectional optical flow; The coarse frame interpolation result is obtained by transforming the arbitrary consecutive image frames using the bidirectional optical flow. The step of inputting the coarse frame interpolation result and the bidirectional optical flow into the fusion network to obtain the fused frame interpolation result includes: Determine two pixel-level masks corresponding to the arbitrary consecutive image frames; The two pixel-level masks are multiplied with the bidirectional optical flow at the pixel level to obtain the fused frame interpolation result.

2. The video frame interpolation method based on stereo events and intensity cameras according to claim 1, characterized in that, The step of inputting the fused frame interpolation result into the refining network and outputting the optimized frame interpolation result includes: The fused interpolation result is input into a residual dense network to obtain the optimized interpolation result composed of the fused interpolation result and the output residual.

3. The video frame interpolation method based on stereo events and intensity cameras according to claim 1, characterized in that, The construction of a comprehensive loss function based on the optimized post-interpolation result and the real image includes: An L1 norm loss function is used to construct a reconstruction loss function based on the fused interpolation result, the optimized interpolation result, and the real image. A smoothed L1 norm loss is employed to construct an image display loss function based on feature disparity and true disparity. An event flow loss function is constructed based on the optimized insertion results and the bidirectional optical flow, employing photometric consistency loss and optical flow smoothness loss. The combined loss function is obtained by weighted summation of the reconstruction loss function, the image display loss function, and the event flow loss function.

4. The video frame interpolation method based on stereo events and intensity cameras according to claim 1, characterized in that, The process of determining the image frame to be processed and the inter-frame event stream to be processed, and inputting them into the video interpolation model to obtain the video interpolation result, includes: Obtain any consecutive image frames in the image frame to be processed, and extract the inter-frame event stream between the exposure times of the image frame to be processed; The arbitrary consecutive image frames and the inter-frame event stream are input into the video interpolation model, and the video interpolation result is output.

5. A video frame interpolation system based on stereo events and intensity cameras, based on the video frame interpolation method based on stereo events and intensity cameras according to any one of claims 1 to 4, characterized in that, include: The acquisition module is used to acquire raw image frames and raw event streams; The extraction module is used to extract any consecutive image frames from the original image frames and determine the inter-frame event stream based on the arbitrary consecutive image frames; The training module is used to input the arbitrary continuous image frames and the inter-frame event stream into the convolutional neural network for training, and to obtain a video frame interpolation model based on the frame interpolation results obtained from the model training and the loss function constructed from the real images. The processing module is used to determine the image frame to be processed and the inter-frame event stream to be processed, and input them into the video interpolation model to obtain the video interpolation result.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the video frame interpolation method based on stereo events and intensity cameras as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the video frame interpolation method based on stereo events and intensity cameras as described in any one of claims 1 to 4.

Images