An event camera based self-supervised high frame rate video reconstruction method and system

Abstract

The application provides an event camera-based self-supervised high-frame-rate video reconstruction method and system, which comprises the construction of a video reference frame image and an event stream data set, the preprocessing of the event stream, the construction of an event-based unified intra-frame and inter-frame learning network model to complete deblurring and frame interpolation, and a self-supervised framework to train the network and generate a clear video. According to the event stream, the reference image and the potential clear image at any time to be estimated are associated, and a unified intra-frame and inter-frame double-integral estimation module is constructed. By using the constraint relationship among the reference frame image, the potential clear image and the event stream, a self-supervised framework is proposed to train the network model, and the trained network model is used to reconstruct a high-frame-rate and high-quality clear video. The application solves the problem of inter-frame and intra-frame information loss caused by long exposure time and slow shutter speed during video shooting, and has good deblurring and frame interpolation video reconstruction effect.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a self-supervised high frame rate video reconstruction method and system based on an event camera. Background Technology

[0002] High-quality video acquisition faces several challenges when dealing with highly dynamic scenes, such as fast-moving targets or complex non-linear motion. These challenges primarily stem from the prevalent frame blur and the lack of basic target information between consecutive frames.

[0003] Currently, frame-based methods employ motion deblurring, frame interpolation, and blurred video enhancement techniques to address these issues. However, frame-based deblurring methods face significant challenges in predicting sharp underlying frames from severely blurred videos. This is primarily due to the challenges posed by motion blur and the loss of intensity texture. Furthermore, existing frame-based interpolation methods typically assume linear motion between adjacent frames, which often leads to inaccurate predictions in complex nonlinear motion scenarios, especially in real-world situations.

[0004] Recent research has revealed the advantages of event cameras in video deblurring and frame interpolation: on the one hand, the output of an event camera inherently embeds accurate motion and sharp edges because it records asynchronous event data with extremely low latency (on the microsecond scale), effectively mitigating motion blur. On the other hand, the efficiency of event cameras in capturing continuous brightness changes effectively bridges the gaps in information between consecutive frames, enabling accurate reconstruction of intermediate frames in scenes with complex nonlinear motion. However, existing methods typically treat motion deblurring and frame interpolation as separate tasks, making frame interpolation performance heavily dependent on the quality of the reference frame, often resulting in degradation when using motion-blurred reference frames. Therefore, a unified approach that considers both tasks is needed. Furthermore, existing methods are often trained on simulated datasets with real labels, and the difference between simulated and real data causes significant performance degradation in real-world scenarios, prompting a need for self-supervised learning frameworks. Summary of the Invention

[0005] This invention provides a self-supervised high frame rate video reconstruction method and system based on event cameras, which addresses the shortcomings of existing technologies for high frame rate video reconstruction.

[0006] In a first aspect, the present invention provides a self-supervised high frame rate video reconstruction method based on an event camera, comprising:

[0007] Acquire raw images and raw event streams;

[0008] The original event stream is segmented based on the target time to be reconstructed, and the segmented original event stream is preprocessed to obtain event frames;

[0009] Input the event frames corresponding to any two consecutive images into the learnable double integral model to generate event double integral coarse results;

[0010] The double-integral coarse result of the event is optimized using the exposure transfer reconstruction module to obtain the optimized reconstruction result;

[0011] Based on the adaptive parameterless fusion module, a clear reconstructed image is generated from the optimized reconstruction result and the corresponding event double integral coarse result;

[0012] A self-supervised high frame rate video reconstruction model is constructed by training a network to address the constraints between the original image, the event stream, and the reconstructed sharp image using a self-supervised framework.

[0013] The event stream to be reconstructed and the video images are input into the self-supervised high frame rate video reconstruction model, which outputs a high frame rate reconstructed video.

[0014] According to the present invention, a self-supervised high frame rate video reconstruction method based on an event camera acquires original images and original event streams, including:

[0015] The original image is acquired using a conventional camera, and the original event stream is acquired using an event camera.

[0016] Alternatively, a series of clear images can be captured using a high-speed camera within any given time period. The original event stream can be generated by processing the series of clear images using an event generation model, and the original blurred image can be obtained by averaging the values ​​of multiple series of clear images.

[0017] According to the present invention, a self-supervised high frame rate video reconstruction method based on an event camera is provided, which segments the original event stream based on the target time to be reconstructed, and preprocesses the segmented original event stream to obtain event frames, including:

[0018] Determine the time period constructed using the target time and reference time to reconstruct a clear image;

[0019] The original event stream is triggered within the time period. If the reference time is greater than or equal to the target time, a time offset operation is performed on the original event stream. Otherwise, a time offset, time flip, and polarity flip operation is performed on the original event stream to obtain the event preprocessing operator.

[0020] The event stream within the time period is divided into N event streams at equal time intervals, and the event points are accumulated within the N time intervals to form an event frame of size 2×H×W, where 2 represents positive and negative polarity, and H and W represent the height and width of the event frame, thus forming an N×2×H×W event frame.

[0021] According to the self-supervised high frame rate video reconstruction method based on an event camera provided by the present invention, the event frames corresponding to any two consecutive images are input into a learnable double integral model to generate event double integral coarse results, including:

[0022] The learnable dual integral model includes four convolutional layers, with a ReLU activation function between every two convolutional layers.

[0023] Determine the first and second event frames corresponding to two consecutive images;

[0024] The integral of a specific event from 0 to any time point is fitted by a learnable event-based double integral network;

[0025] Substituting the target time and exposure period for reconstructing the clear image into the specific event integral, we obtain the general event integral;

[0026] Based on the start time of the exposure period and the target time of the reconstructed clear image, the first weight and the second weight of the general event integral are calculated.

[0027] The event double integral rough result is obtained from the first weight, the second weight, and the event preprocessing operator.

[0028] According to the self-supervised high frame rate video reconstruction method based on an event camera provided by the present invention, the event double integral coarse result is optimized using an exposure transfer reconstruction module to obtain an optimized reconstruction result, including:

[0029] The exposure transfer reconstruction module includes 3 convolutional layers, 16 dense residual blocks, and 2 pixel reconstruction PS modules;

[0030] The image-event correlation formula is obtained by comparing the ratio of the original image to the reciprocal of the double integral coarse result of the event;

[0031] The image event correlation formula and the event double integral coarsening result are input into the exposure transfer reconstruction module to obtain an optimized and reconstructed clear image.

[0032] The optimized reconstructed clear image and the reciprocal of the event double integral coarse result are input into the exposure transfer reconstruction module to obtain the optimized reconstruction result.

[0033] According to the present invention, a self-supervised high frame rate video reconstruction method based on an event camera generates a reconstructed sharp image based on an adaptive parameterless fusion module, using the optimized reconstruction result and the corresponding event double integral coarse result. The method includes:

[0034] Using the exposure transfer reconstruction module, a first optimized reconstructed clear image and a second optimized reconstructed clear image are generated based on the target time of the reconstructed clear image.

[0035] The first optimized reconstructed clear image, the second optimized reconstructed clear image, the first event frame, and the second event frame number are input into the exposure transfer reconstruction module to obtain the reconstructed clear image.

[0036] According to the self-supervised high frame rate video reconstruction method based on an event camera provided by the present invention, a self-supervised framework is used to train a network to construct a self-supervised high frame rate video reconstruction model by training the network on the constraints between the original image, the event stream, and the reconstructed sharp image, including:

[0037] The loss function for the original image and the event stream is obtained from the L1 norm of the difference between the optimized reconstruction results of adjacent original images and adjacent original event streams.

[0038] The loss function for the reconstructed sharp image and the original image is obtained by summing the L1 norms between adjacent optimized reconstructed images and adjacent original images;

[0039] The loss function of the original image is obtained by summing the L1 norms of the differences between the adjacent predicted original input images and the adjacent original images;

[0040] The self-supervised frame loss function is obtained by weighted summing of the original image loss function with the event stream loss function, the reconstructed clear image loss function with the original image loss function, and the original image loss function with the original image loss function.

[0041] The Adam optimization algorithm is used to constrain and optimize the model using the self-supervised framework loss function. The independent adaptive learning rate is determined by calculating the first-order moment estimate and second-order moment estimate of the gradient, thus obtaining the self-supervised high frame rate video reconstruction model.

[0042] Secondly, the present invention also provides a self-supervised high frame rate video reconstruction system based on an event camera, comprising:

[0043] The acquisition module is used to acquire raw images and raw event streams;

[0044] The event stream preprocessing module is used to segment the original event stream based on the target time to be reconstructed, and to preprocess the segmented original event stream to obtain event frames;

[0045] The event stream double integration module is used to input the event frames corresponding to any two consecutive images into the learnable double integration model and generate coarse event double integration results.

[0046] The reconstruction optimization module is used to optimize the double integral coarse result of the event using the exposure transfer reconstruction module to obtain the optimized reconstruction result;

[0047] An image fusion module is used to generate a reconstructed clear image based on the optimized reconstruction result and the corresponding event double integral coarse result, using the adaptive parameterless fusion module.

[0048] The training module is used to train the network using a self-supervised framework to address the constraints between the original image, the event stream, and the reconstructed sharp image, thereby constructing a self-supervised high frame rate video reconstruction model.

[0049] The processing module is used to input the event stream to be reconstructed and the video image into the self-supervised high frame rate video reconstruction model and output the high frame rate reconstructed video.

[0050] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the self-supervised high frame rate video reconstruction method based on an event camera as described above.

[0051] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the self-supervised high frame rate video reconstruction method based on an event camera as described above.

[0052] The self-supervised high frame rate video reconstruction method and system based on event cameras provided by this invention solves the joint problem of video deblurring and video frame interpolation by utilizing the high temporal resolution of event cameras. Furthermore, a self-supervised learning framework is designed so that the network can be trained on real-world shooting datasets and achieves high-performance deblurring and frame interpolation results. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating the self-supervised high frame rate video reconstruction method based on event cameras provided by the present invention.

[0055] Figure 2 This is a schematic diagram of a specific method for event stream preprocessing provided by the present invention;

[0056] Figure 3 This is a schematic diagram of the event preprocessing module and the data flow of the LDI network provided by the present invention;

[0057] Figure 4 This is a schematic diagram of the exposure transfer and reconstruction module provided by the present invention;

[0058] Figure 5 This is a schematic diagram of the adaptive parameterless fusion module provided by the present invention;

[0059] Figure 6 This is a data flow diagram of the reconstruction module and self-supervised architecture provided by the present invention;

[0060] Figure 7 This is a comparison image of the image deblurring method provided by this invention on the simulation dataset REDS with the current image deblurring method;

[0061] Figure 8 This is a comparison chart of the results provided by this invention with the current clear image frame interpolation method on the simulated dataset ColorDVS;

[0062] Figure 9 This is a comparison diagram of the results provided by this invention on the simulation dataset ColorDVS and the current blurred image interpolation method;

[0063] Figure 10 This is a comparison diagram of the results provided by this invention on the real dataset ColorRBE and the current blurred image interpolation method;

[0064] Figure 11 This is a schematic diagram of the structure of the self-supervised high frame rate video reconstruction system based on an event camera provided by the present invention;

[0065] Figure 12 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0067] Figure 1 This is a flowchart illustrating the self-supervised high frame rate video reconstruction method based on an event camera provided in an embodiment of the present invention, as shown below. Figure 1As shown, it includes:

[0068] Step 100: Acquire raw images and raw event streams;

[0069] Step 200: Based on the target time to be reconstructed, the original event stream is segmented, and the segmented original event stream is preprocessed to obtain event frames;

[0070] Step 300: Input the event frames corresponding to any two consecutive images into the learnable double integral model to generate event double integral coarse results;

[0071] Step 400: Optimize the double integral coarse result of the event using the exposure transfer reconstruction module to obtain the optimized reconstruction result;

[0072] Step 500: Based on the adaptive parameterless fusion module, a reconstructed clear image is generated from the optimized reconstruction result and the corresponding event double integral coarse result;

[0073] Step 600: Use a self-supervised framework to train a network to address the constraints between the original image, the event stream, and the reconstructed clear image, thereby constructing a self-supervised high frame rate video reconstruction model;

[0074] Step 700: Input the event stream to be reconstructed and the video image into the self-supervised high frame rate video reconstruction model, and output the high frame rate reconstructed video.

[0075] This invention provides a self-supervised learning method for high frame rate sharp video reconstruction. The method receives a sequence of consecutive low-frame-rate blurred or sharp images F∈{B,I} and generates a sequence of sharp high-frame-rate images. The proposed method comprises three modules: a Learnable Double Integral (LDI) network, an Exposure Transfer Reconstruction (ETR) module, and an Adaptive No-Parameter Fusion (APF) module. During training, we propose a fully self-supervised learning framework that leverages the mutual constraints between blurred frames, sharp latent images, and event streams to help the network adapt to the distribution of real data without requiring real labeled images.

[0076] Specifically, this invention uses clear video captured by a high-speed camera to simulate blurred images and event streams, or directly uses images and event streams captured by traditional optical cameras and event cameras. The event stream is preprocessed: for each set of input data, the event stream is segmented according to the target time to be reconstructed and preprocessed into a multi-channel tensor as the network input, then event-based double integral inference is performed. Based on an event-based double integral model, a learnable event-based double integral convolutional neural network is constructed. The reconstruction result is then optimized by constructing an exposure transfer reconstruction module to optimize the coarse results generated by the LDI network. Multiple images are then fused by constructing an adaptive parameterless fusion module, using the confidence embedded in the LDI output to weight multiple reconstructed images and generate the final result. Finally, a self-supervised framework is used to train the network: utilizing the relationship between the original blurred image, the event stream, and the reconstructed clear image, a self-supervised constraint optimization network is constructed. In practical applications, an event camera captures the event stream, and a traditional camera captures video. After processing, the event stream yields event frames that can be input into the network. The event frames and video images are then used through the optimized network model to reconstruct a high-frame-rate clear video.

[0077] This invention solves the joint problem of video deblurring and frame interpolation by utilizing the high temporal resolution of event cameras, and designs a self-supervised learning framework that enables the network to be trained on real-world shooting datasets, achieving high-performance deblurring and frame interpolation results.

[0078] Based on the above embodiments, step 100 includes:

[0079] The original image is acquired using a conventional camera, and the original event stream is acquired using an event camera.

[0080] Alternatively, a series of clear images can be captured using a high-speed camera within any given time period. The original event stream can be generated by processing the series of clear images using an event generation model, and the original blurred image can be obtained by averaging the values ​​of multiple series of clear images.

[0081] Specifically, for real-world scenarios, images L(t) and event streams ε can be captured directly using traditional cameras and event cameras.

[0082] Blurred images and event streams can also be simulated using clear video captured by a high-speed camera, selecting a time period. A high-speed camera captures continuous images, an event generation model is used to generate an event stream, and the time average of the images is calculated to obtain a blurred image.

[0083] The simulation generation steps are as follows:

[0084] The event generation model is as follows:

[0085] log(L(t,x))-log(L(τ,x))=p·c

[0086] Where L(t,x) and L(τ,x) are the values ​​at time t and τ respectively in the image. The brightness value of the location; log(·) is the logarithmic operator; p = {+1, -1} represents the polarity of the event, +1 is positive polarity (ON), indicating that the scene brightness increases, and -1 is negative polarity (OFF), indicating that the scene brightness decreases. The above formula indicates that when the brightness change value of a certain location in the scene in the logarithmic domain exceeds the threshold c at a certain moment, the camera will output an event point of the corresponding polarity; the event stream ε represents the set of events triggered within the entire time period.

[0087] The process of calculating the time mean is as follows:

[0088]

[0089] Where F∈{B,I} represents the generated blurred or sharp image (when... ), where T is the time period. The duration of exposure in the video.

[0090] The correlation formula for image events can be obtained by combining the event generation model and the time averaging process:

[0091]

[0092] in:

[0093]

[0094] in It is a continuous representation of events, where δ(·) is the Dirac function and f is the target time for reconstructing a clear image. Two consecutive images and their corresponding event streams are selected from the data to form a set of data.

[0095] Based on the above embodiments, step 200 includes:

[0096] Determine the time period constructed using the target time and reference time to reconstruct a clear image;

[0097] The original event stream is triggered within the time period. If the reference time is greater than or equal to the target time, a time offset operation is performed on the original event stream. Otherwise, a time offset, time flip, and polarity flip operation is performed on the original event stream to obtain the event preprocessing operator.

[0098] The event stream within the time period is divided into N event streams at equal time intervals, and the event points are accumulated within the N time intervals to form an event frame of size 2×H×W, where 2 represents positive and negative polarity, and H and W represent the height and width of the event frame, thus forming an N×2×H×W event frame.

[0099] Specifically, in this embodiment of the invention, for each set of input data, the event stream is segmented according to the target time to be reconstructed and preprocessed to be converted into a multi-channel tensor as the input of the network.

[0100] Define event stream preprocessing as The specific method is as follows: for the time period [f,t] r Events triggered during the period

[0101]

[0102] Where t r For reference only; This indicates a time offset operation for the event; This represents event offset, time flip, and polarity reversal operations, such as... Figure 2 As shown.

[0103] The specific process of converting an event stream into a multi-channel tensor is as follows: each event stream is divided into N event streams according to equal time intervals, and the event points are accumulated within N time intervals to form an event frame of size 2×H×W (2 represents positive and negative polarity, and H and W represent the height and width of the event frame), ultimately forming an N×2×H×W event frame.

[0104] Based on the above embodiments, step 300 includes:

[0105] The learnable dual integral model includes four convolutional layers, with a ReLU activation function between every two convolutional layers.

[0106] Determine the first and second event frames corresponding to two consecutive images;

[0107] The integral of a specific event from 0 to any time point is fitted by a learnable event-based double integral network;

[0108] Substituting the target time and exposure period for reconstructing the clear image into the specific event integral, we obtain the general event integral;

[0109] Based on the start time of the exposure period and the target time of the reconstructed clear image, the first weight and the second weight of the general event integral are calculated.

[0110] The event double integral rough result is obtained from the first weight, the second weight, and the event preprocessing operator.

[0111] Specifically, a learnable event-based dual-integral network receives the event streams corresponding to two consecutive images. And generate two event double integrals. and Specifically, LDI networks are used to fit the following special cases:

[0112]

[0113] in ε represents the time interval from 0 to T>0; [0,T] This corresponds to the event. Let's consider one below. A more general case:

[0114]

[0115] Where t s Representative exposure period The starting time point. Let t' = tf, s' = sf, and after substitution, we can obtain:

[0116]

[0117] Where ω1=(ft) s ) / T, ω2=(t s +Tf) / T is the weight of each term, and G(·) is a general formula defined by the following equation:

[0118]

[0119] Based on the above derivation, the event double integral can be calculated by fitting the formula using LDI. When t r When -f≥0:

[0120]

[0121] When t r When -f<0:

[0122]

[0123] In conjunction with the event preprocessing operators defined in the foregoing embodiments, The expression can be rewritten as:

[0124]

[0125] This indicates that any one Both can be represented as a weighted sum of the outputs of the LDI network. This represents a special case. In implementation, the LDI network consists of four convolutional layers, with a ReLU activation function added between every two convolutional layers. The last convolutional layer directly outputs the result, such as... Figure 3 As shown.

[0126] Based on the above embodiments, step 400 includes:

[0127] The exposure transfer reconstruction module includes 3 convolutional layers, 16 dense residual blocks, and 2 pixel reconstruction PS modules;

[0128] The image-event correlation formula is obtained by comparing the ratio of the original image to the reciprocal of the double integral coarse result of the event;

[0129] The image event correlation formula and the event double integral coarsening result are input into the exposure transfer reconstruction module to obtain an optimized and reconstructed clear image.

[0130] The optimized reconstructed clear image and the reciprocal of the event double integral coarse result are input into the exposure transfer reconstruction module to obtain the optimized reconstruction result.

[0131] Specifically, embodiments of the present invention optimize the coarse results generated by the learnable event dual integral network by constructing an exposure transfer reconstruction module.

[0132] As can be seen from the correlation formula of image events, the potential sharp image L(f) can be obtained by the input F∈{B,I} and the corresponding event double integral. To obtain.

[0133] Specifically, fit the correlation formula of image events to optimize the reconstruction of clear images.

[0134]

[0135] Furthermore, the correlation formula for image events can be rewritten as follows: Where E'(·) is the reciprocal of E(·). Therefore, we can obtain:

[0136]

[0137] in The ETR network uses the reconstructed sharp image to infer the original input image for prediction. In its implementation, the ETR network consists of 3 convolutional layers, 16 dense residual blocks, and 2 pixel shuffle (PS) modules, as shown below. Figure 4 As shown, the input image F and the event double integral First, the data passes through a first PS block and a convolutional layer, then through cascaded dense residual blocks, and finally through a convolutional layer, a PS block, and another convolutional layer. The F value is directly concatenated to the input positions of the first two convolutional layers and the output position of the last convolutional layer. This process allows for the convenient construction of a cycle consistency loss, facilitating self-supervised training and improving training efficiency.

[0138] Based on the above embodiments, step 500 includes:

[0139] Using the exposure transfer reconstruction module, a first optimized reconstructed clear image and a second optimized reconstructed clear image are generated based on the target time of the reconstructed clear image.

[0140] The first optimized reconstructed clear image, the second optimized reconstructed clear image, the first event frame, and the second event frame number are input into the exposure transfer reconstruction module to obtain the reconstructed clear image.

[0141] Specifically, embodiments of the present invention utilize an exposure transfer reconstruction module, such as... Figure 5 As shown, two clear images are generated based on the virtual timestamp f:

[0142]

[0143]

[0144] An Adaptive Parameterless Fusion (APF) module is proposed, based on and the corresponding Generate the final clear image

[0145] The adaptive parameterless fusion module here is specifically:

[0146]

[0147] in Using the adaptive parameterless fusion module, it is possible to achieve fusion throughout the entire exposure period. Achieve better performance internally, without relying on any parameters that need to be learned.

[0148] Based on the above embodiments, step 600 includes:

[0149] The loss function for the original image and the event stream is obtained from the L1 norm of the difference between the optimized reconstruction results of adjacent original images and adjacent original event streams.

[0150] The loss function for the reconstructed sharp image and the original image is obtained by summing the L1 norms between adjacent optimized reconstructed images and adjacent original images;

[0151] The loss function of the original image is obtained by summing the L1 norms of the differences between the adjacent predicted original input images and the adjacent original images;

[0152] The self-supervised frame loss function is obtained by weighted summing of the original image loss function with the event stream loss function, the reconstructed clear image loss function with the original image loss function, and the original image loss function with the original image loss function.

[0153] The Adam optimization algorithm is used to constrain and optimize the model using the self-supervised framework loss function. The independent adaptive learning rate is determined by calculating the first-order moment estimate and second-order moment estimate of the gradient, thus obtaining the self-supervised high frame rate video reconstruction model.

[0154] Specifically, such as Figure 6 As shown, the self-supervised framework constructed in this embodiment of the invention is as follows:

[0155]

[0156] in For blurred image and event stream loss, For clear and blurry images, Let [α,β,γ] be the blurred image and the blurred image loss, and let [α,β,γ] be the weight vector, which can be taken as [1,0.5,1].

[0157] Blurred image and event stream loss are defined as:

[0158]

[0159] The loss between sharp and blurry images is defined as:

[0160]

[0161] Blurred image and blurred image loss are defined as follows:

[0162]

[0163] ||·||1 represents the l1 norm.

[0164] Furthermore, the Adam optimization algorithm is employed to design independent adaptive learning rates for different parameters by calculating the first and second moment estimates of the gradient, and to iteratively update the parameters of the neural network convolutional layers based on the training data. The optimized multi-layer convolutional neural network is constructed by building a multi-layer convolutional neural network based on the optimized parameters of each convolutional layer.

[0165] Finally, an event camera is used to capture the event stream, while a traditional camera captures video. The event stream is processed to obtain event frames that can be input into the network. The event frames and video images are then used to reconstruct a clear video with a high frame rate through an optimized network model.

[0166] The following comparison with several existing image deblurring methods illustrates the effectiveness and superiority of the present invention. The methods include:

[0167] 1) LEVS: A model-based deep learning and supervised image deblurring method.

[0168] 2) MotionETR: A model-based deep learning and supervised image deblurring method.

[0169] 3) MPRNet: A model-based deep learning and supervised image deblurring method.

[0170] 4) EDI: A model-based method for deblurring and interpolating images from event cameras.

[0171] 5) eSL-Net: An image deblurring method based on deep learning and sparse representation.

[0172] 6) LEDVDI: A deep learning-based supervised image deblurring method for event cameras.

[0173] 7) RED-Net: A deep learning-based self-supervised image deblurring method for event cameras.

[0174] 8) EFNet: A deep learning-based supervised image deblurring method for event cameras.

[0175] 9) EVDI: A deep learning-based self-supervised image deblurring and frame interpolation method for event cameras.

[0176] 10) EVDI++ (Ours): A deep learning-based self-supervised image deblurring and frame interpolation method for event cameras.

[0177] Table 1 shows the deblurring test results of the model on single image sequences on the REDS and ColorDVS datasets. Figure 7 The image shows a comparison of the results of this invention with the current image deblurring method on the simulation dataset REDS.

[0178] Table 1

[0179]

[0180] For video frame interpolation tasks, this invention compares the following methods:

[0181] 1) RIFE: A model-based deep learning and supervised image interpolation method.

[0182] 2) EMA: A model-based deep learning and supervised image interpolation method.

[0183] 3) UnSuperSlom: A model-based deep learning and self-supervised image interpolation method.

[0184] 4) EDI: A model-based method for deblurring and interpolating images from event cameras.

[0185] 5) TimeLens: A model-based deep learning and supervised image interpolation method.

[0186] 6) EVDI++ (Ours): A deep learning-based self-supervised image deblurring and frame interpolation method for event cameras.

[0187] Table 2 shows the test results of the model on single image sequence interpolation on the REDS and ColorDVS datasets.

[0188] Table 2

[0189]

[0190] Figure 8 This image shows a comparison of the results provided by this invention with the current clear image frame interpolation method on the simulated ColorDVS dataset. Figure 9 This image shows a comparison of the results provided by this invention with the current blurred image interpolation method on the ColorDVS simulation dataset. Figure 10 The image shows a comparison between the results provided by this invention on the real dataset ColorRBE and the current blurred image interpolation method.

[0191] Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were used as evaluation metrics in the experiment, and their specific definitions are as follows:

[0192] PSNR: First, calculate the pixel-level mean squared error (MSE) between the ground truth image and the reconstructed image.

[0193]

[0194] Where f(i,j) and g(i,j) represent the pixel values ​​at position (i,j) in the ground truth image and the reconstructed image, respectively, and W and H represent the width and height of the image, respectively.

[0195]

[0196] SSIM: Evaluates reconstruction quality from three aspects: brightness, contrast, and structure of the ground truth image and the reconstructed image. Its specific calculation method is as follows:

[0197]

[0198] Where μ x μ xσ represents the mean of the ground truth image and the reconstructed image, respectively. x , σ y σ represents the variance of the GT image and the reconstructed image. xy C1 and C2 are constants introduced to prevent the denominator from being zero, representing the covariance of the GT image and the reconstructed image.

[0199] Higher PSNR and SSIM values ​​indicate better reconstruction results. As shown in Tables 1 and 2, this invention demonstrates the highest performance in both video image deblurring and frame interpolation tasks, validating the effectiveness of the invention.

[0200] The self-supervised high frame rate video reconstruction system based on event cameras provided by the present invention will be described below. The self-supervised high frame rate video reconstruction system based on event cameras described below can be referred to in correspondence with the self-supervised high frame rate video reconstruction method based on event cameras described above.

[0201] Figure 11 This is a schematic diagram of the structure of the self-supervised high frame rate video reconstruction system based on an event camera provided in an embodiment of the present invention, as shown below. Figure 11 As shown, it includes: an acquisition module 1101, an event stream preprocessing module 1102, an event stream dual integration module 1103, a reconstruction optimization module 1104, an image fusion module 1105, a training module 1106, and a processing module 1107, wherein:

[0202] The acquisition module 1101 is used to acquire the original image and the original event stream; the event stream preprocessing module 1102 is used to segment the original event stream based on the target time to be reconstructed, and preprocess the segmented original event stream to obtain event frames; the event stream dual integration module 1103 is used to input the event frames corresponding to any two consecutive images into a learnable dual integration model to generate event dual integration coarse results; the reconstruction optimization module 1104 is used to optimize the event dual integration coarse results using an exposure transfer reconstruction module to obtain optimized reconstruction results; the image fusion module 1105 is used to generate a reconstructed clear image based on the optimized reconstruction results and the corresponding event dual integration coarse results using an adaptive parameterless fusion module; the training module 1106 is used to train a network using a self-supervised framework to perform network training on the constraints between the original image, the event stream and the reconstructed clear image to construct a self-supervised high frame rate video reconstruction model; the processing module 1107 is used to input the event stream to be reconstructed and the video image into the self-supervised high frame rate video reconstruction model and output a high frame rate reconstructed video.

[0203] Figure 12 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 12As shown, the electronic device may include: a processor 1210, a communications interface 1220, a memory 1230, and a communications bus 1240, wherein the processor 1210, the communications interface 1220, and the memory 1230 communicate with each other through the communications bus 1240. The processor 1210 can call logic instructions in the memory 1230 to execute a self-supervised high frame rate video reconstruction method based on an event camera. This method includes: acquiring original images and an original event stream; segmenting the original event stream based on the target time to be reconstructed, preprocessing the segmented original event stream to obtain event frames; inputting the event frames corresponding to any two consecutive images into a learnable dual-integral model to generate event dual-integral coarse results; optimizing the event dual-integral coarse results using an exposure transfer reconstruction module to obtain optimized reconstruction results; generating a reconstructed sharp image based on an adaptive parameterless fusion module using the optimized reconstruction results and the corresponding event dual-integral coarse results; training a network using a self-supervised framework to address the constraints between the original images, the event stream, and the reconstructed sharp image to construct a self-supervised high frame rate video reconstruction model; inputting the event stream to be reconstructed and the video images into the self-supervised high frame rate video reconstruction model to output a high frame rate reconstructed video.

[0204] Furthermore, the logical instructions in the aforementioned memory 1230 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0205] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the self-supervised high frame rate video reconstruction method based on an event camera provided by the methods described above. The method includes: acquiring original images and original event streams; segmenting the original event stream based on the target time to be reconstructed, preprocessing the segmented original event stream to obtain event frames; inputting the event frames corresponding to any two consecutive images into a learnable dual-integral model to generate event dual-integral coarse results; optimizing the event dual-integral coarse results using an exposure transfer reconstruction module to obtain optimized reconstruction results; generating a reconstructed clear image based on the optimized reconstruction results and the corresponding event dual-integral coarse results using an adaptive parameterless fusion module; training a network using a self-supervised framework to constrain the original images, the event stream, and the reconstructed clear image to construct a self-supervised high frame rate video reconstruction model; inputting the event stream to be reconstructed and the video image into the self-supervised high frame rate video reconstruction model to output a high frame rate reconstructed video.

[0206] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0207] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A self-supervised high frame rate video reconstruction method based on event cameras, characterized in that, include: Acquire raw images and raw event streams; The original event stream is segmented based on the target time to be reconstructed, and the segmented original event stream is preprocessed to obtain event frames; Inputting event frames corresponding to any two consecutive images into a learnable double-integral model generates coarse event double-integral results, including: The learnable dual integral model includes four convolutional layers, with a ReLU activation function between every two convolutional layers. Determine the first and second event frames corresponding to two consecutive images; The integral of a specific event from 0 to any time point is fitted by a learnable event-based double integral network; Substituting the target time and exposure period for reconstructing the clear image into the specific event integral, we obtain the general event integral; Based on the start time of the exposure period and the target time of the reconstructed clear image, the first weight and the second weight of the general event integral are calculated. The event double integral rough result is obtained from the first weight, the second weight, and the event preprocessing operator; The exposure transfer reconstruction module is used to optimize the double integral coarse result of the event, resulting in an optimized reconstruction result, including: The exposure transfer reconstruction module includes 3 convolutional layers, 16 dense residual blocks, and 2 pixel reconstruction PS modules; The image-event correlation formula is obtained by comparing the ratio of the original image to the reciprocal of the double integral coarse result of the event; The image event correlation formula and the event double integral coarsening result are input into the exposure transfer reconstruction module to obtain an optimized and reconstructed clear image. The optimized reconstructed sharp image and the reciprocal of the event double integral coarseness result are input into the exposure transfer reconstruction module to obtain the optimized reconstruction result; Based on the adaptive parameterless fusion module, a reconstructed clear image is generated from the optimized reconstruction result and the corresponding event double integral coarse result, including: Using the exposure transfer reconstruction module, a first optimized reconstructed clear image and a second optimized reconstructed clear image are generated based on the target time of the reconstructed clear image. The first optimized reconstructed clear image, the second optimized reconstructed clear image, the first event frame, and the second event frame number are input into the exposure transfer reconstruction module to obtain the reconstructed clear image; A self-supervised high frame rate video reconstruction model is constructed by training a network to address the constraints between the original image, the event stream, and the reconstructed sharp image using a self-supervised framework. The event stream to be reconstructed and the video images are input into the self-supervised high frame rate video reconstruction model, which outputs a high frame rate reconstructed video.

2. The self-supervised high frame rate video reconstruction method based on event cameras according to claim 1, characterized in that, Acquire raw images and raw event streams, including: The original image is acquired using a conventional camera, and the original event stream is acquired using an event camera. Alternatively, a series of clear images can be captured using a high-speed camera within any given time period. The original event stream can be generated by processing the series of clear images using an event generation model, and the original blurred image can be obtained by averaging the values ​​of multiple series of clear images.

3. The self-supervised high frame rate video reconstruction method based on event cameras according to claim 1, characterized in that, Based on the target time to be reconstructed, the original event stream is segmented, and the segmented original event stream is preprocessed to obtain event frames, including: Determine the time period constructed using the target time and reference time to reconstruct a clear image; The original event stream is triggered within the time period. If the reference time is greater than or equal to the target time, a time offset operation is performed on the original event stream. Otherwise, a time offset, time flip, and polarity flip operation is performed on the original event stream to obtain the event preprocessing operator. The event stream within the time period is divided into equal time intervals. Segment event flow, and in The event points are accumulated within a time interval to form a size of [value]. The event frame, where 2 represents positive or negative polarity, and The height and width of the event frame are represented to form The event frame.

4. The self-supervised high frame rate video reconstruction method based on event cameras according to claim 1, characterized in that, A self-supervised high frame rate video reconstruction model is constructed by training a network to address the constraints between the original image, the event stream, and the reconstructed sharp image using a self-supervised framework, including: The loss function for the original image and the event stream is obtained from the L1 norm of the difference between the optimized reconstruction results of adjacent original images and adjacent original event streams. The loss function for the reconstructed sharp image and the original image is obtained by summing the L1 norms between adjacent optimized reconstructed images and adjacent original images; The loss function of the original image is obtained by summing the L1 norms of the differences between the adjacent predicted original input images and the adjacent original images; The self-supervised frame loss function is obtained by weighted summing of the original image loss function with the event stream loss function, the reconstructed clear image loss function with the original image loss function, and the original image loss function with the original image loss function. The Adam optimization algorithm is used to constrain and optimize the model using the self-supervised framework loss function. The independent adaptive learning rate is determined by calculating the first-order moment estimate and second-order moment estimate of the gradient, thus obtaining the self-supervised high frame rate video reconstruction model.

5. A self-supervised high frame rate video reconstruction system based on an event camera, comprising the self-supervised high frame rate video reconstruction method based on an event camera as described in any one of claims 1 to 4, characterized in that, include: The acquisition module is used to acquire raw images and raw event streams; The event stream preprocessing module is used to segment the original event stream based on the target time to be reconstructed, and to preprocess the segmented original event stream to obtain event frames; The event stream double integration module is used to input the event frames corresponding to any two consecutive images into the learnable double integration model and generate coarse event double integration results. The reconstruction optimization module is used to optimize the double integral coarse result of the event using the exposure transfer reconstruction module to obtain the optimized reconstruction result; An image fusion module is used to generate a reconstructed clear image based on the optimized reconstruction result and the corresponding event double integral coarse result, using the adaptive parameterless fusion module. The training module is used to train the network using a self-supervised framework to address the constraints between the original image, the event stream, and the reconstructed sharp image, thereby constructing a self-supervised high frame rate video reconstruction model. The processing module is used to input the event stream to be reconstructed and the video image into the self-supervised high frame rate video reconstruction model and output the high frame rate reconstructed video.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the self-supervised high frame rate video reconstruction method based on an event camera as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the self-supervised high frame rate video reconstruction method based on an event camera as described in any one of claims 1 to 4.

Images