Video frame insertion method and device, computer equipment and storage medium

Abstract

The invention relates to a video frame insertion method and device, computer equipment and a storage medium. The method is applied to a double-camera system, and the double-camera system comprises a main camera and an event camera. Acquiring a to-be-interpolated frame image set acquired by the main camera at a first frame rate, event stream data output by the event camera and a reference image set acquired synchronously with the to-be-interpolated frame image set; calculating a space alignment optical flow field between the main camera and the event camera according to the to-be-interpolated frame image set and the reference image set; aligning the event stream data with the image set to be subjected to frame insertion according to the spatial alignment optical flow field to obtain aligned event data; and obtaining a target frame insertion image according to the image set to be subjected to frame insertion and the alignment event data. By adopting the method, the frame insertion accuracy can be improved in a complex dynamic scene.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video frame interpolation method, apparatus, computer device, and storage medium. Background Technology

[0002] Video frame interpolation is a method that improves video frame rate or smoothness by generating intermediate frames between original video frames. As video quality and people's demands for video continue to increase, the accuracy of generating intermediate frames becomes increasingly critical.

[0003] Traditional video frame interpolation methods typically utilize optical flow estimation or deep learning models to predict intermediate frames. This prediction method calculates the possible pixel information of the intermediate frame based on pixel changes in the input frame during those changes. Therefore, the accuracy of intermediate frame prediction depends on the accuracy of the pixel changes, i.e., the accuracy of the optical flow estimation. Thus, its performance is highly dependent on clear and continuous spatiotemporal information in the input frame. However, under conditions of high-speed motion or low light, due to image blur, motion occlusion, or missing information, traditional video frame interpolation methods struggle to accurately identify optical flow changes between preceding and following frames, leading to errors in intermediate frame interpolation. The interpolated results still exhibit artifacts, blurring, or structural distortion, severely impacting the user's viewing experience.

[0004] This shows that existing technologies still suffer from low frame interpolation accuracy in complex dynamic scenes. Summary of the Invention

[0005] Therefore, it is necessary to provide a video frame interpolation method, apparatus, computer equipment, and storage medium that can improve the accuracy of frame interpolation in complex dynamic scenes, addressing the aforementioned technical problems.

[0006] In a first aspect, this application provides a video frame interpolation method applied to a dual-camera system, the dual-camera system including a main camera and an event camera, the video frame interpolation method comprising:

[0007] The system acquires the set of frames to be interpolated captured by the main camera at the first frame rate, as well as the event stream data output by the event camera and the set of reference images acquired synchronously with the set of frames to be interpolated.

[0008] Based on the set of frames to be interpolated and the set of reference images, calculate the spatially aligned optical flow field between the main camera and the event camera;

[0009] The event stream data is aligned with the set of frames to be interpolated based on the spatially aligned optical flow field to obtain aligned event data;

[0010] The target interpolated image is obtained based on the set of images to be interpolated and the alignment event data.

[0011] In one embodiment, acquiring the set of interpolated frame images captured by the main camera at a first frame rate, and the event stream data output by the event camera and the set of reference images acquired synchronously with the set of interpolated frame images include:

[0012] The first image captured by the main camera at a first moment and the second image captured at a second moment are obtained to obtain a set of images to be interpolated; the time interval between the first moment and the second moment corresponds to the first frame rate;

[0013] The third image captured by the event camera at the first moment and the fourth image captured at the second moment are obtained to obtain a reference image set;

[0014] Acquire the event stream data output by the event camera between the first time and the second time.

[0015] In one embodiment, calculating the spatially aligned optical flow field between the main camera and the event camera based on the set of images to be interpolated and the set of reference images includes:

[0016] The set of images to be interpolated and the set of reference images are input into a pre-trained first optical flow estimation network to obtain the spatially aligned optical flow field.

[0017] In one embodiment, the first optical flow estimation network further outputs a first optical flow confidence set corresponding to the spatially aligned optical flow field; the process of obtaining the spatially aligned optical flow field further includes:

[0018] If the first optical flow confidence set does not meet the preset abnormal conditions, then the optical flow values ​​in the first optical flow confidence set with confidence levels lower than the first confidence threshold are identified to obtain the optical flow value to be optimized; and the optical flow value to be optimized is filled and smoothed according to the optical flow values ​​with confidence levels higher than the second confidence threshold within the preset range of the optical flow value to be optimized.

[0019] If the first optical flow confidence set meets the preset abnormal conditions, then frame interpolation of the set of images to be interpolated will be stopped.

[0020] In one embodiment, the step of inputting the set of frames to be interpolated and the set of reference images into a pre-trained first optical flow estimation network further includes:

[0021] The set of images to be interpolated is preprocessed to obtain a preprocessed set of images to be interpolated; the preprocessing includes grayscale conversion and size adjustment.

[0022] In one embodiment, obtaining the target interpolated frame image based on the set of frames to be interpolated and the alignment event data includes:

[0023] The set of images to be interpolated, the alignment event data, and the target interpolation time are input into a pre-trained interpolation synthesis network to obtain the target interpolated image. The interpolation synthesis network includes a second optical flow estimation network and an interpolation result synthesis network. The second optical flow estimation network is used to generate a bidirectional optical flow and a fusion mask from the target interpolation time to the set of images to be interpolated, based on the set of images to be interpolated, the alignment event data, and the target interpolation time. The interpolation result synthesis network is used to perform interpolation calculations based on the bidirectional optical flow, the fusion mask, and the set of images to be interpolated to obtain the target interpolated image.

[0024] In one embodiment, the step of performing frame interpolation calculations based on the bidirectional optical flow, the fusion mask, and the set of images to be interpolated to obtain the target interpolated image includes:

[0025] The set of images to be interpolated is reverse-mapped to the target interpolation time according to the bidirectional optical flow to obtain the mapped image;

[0026] The target interpolated image is obtained by weighted fusion of the fusion mask and the mapped image.

[0027] In one embodiment, the second optical flow estimation network further outputs a second optical flow confidence set corresponding to the bidirectional optical flow; the frame interpolation synthesis network further includes a frame interpolation feasibility determination module, which is used for:

[0028] The key points of the images in the set of images to be interpolated are identified respectively to obtain the first key point feature and the second key point feature;

[0029] Calculate the feature matching stability based on the features of the first key point and the features of the second key point;

[0030] The motion reliability is obtained by calculating the optical flow consistency based on the bidirectional optical flow.

[0031] Based on the second optical flow confidence set, the feature matching stability, and the motion confidence, the frame interpolation risk score of the set of images to be interpolated is determined;

[0032] If the frame interpolation risk score does not meet the preset conditions, then frame interpolation filtering or weight reduction processing is performed based on the risk areas in the frame interpolation risk score.

[0033] In one embodiment, the training process of the frame interpolation synthesis network includes:

[0034] Acquire a main camera image sample set captured by the main camera or a camera of the same type as the main camera at a second frame rate, as well as sample event stream data output by the event camera and a reference image sample set acquired synchronously with the main camera image sample set; the second frame rate is greater than the first frame rate;

[0035] Based on the first frame rate, frames are extracted from the main camera image sample set and the reference image sample set respectively to determine multiple sample frame-to-interpolation image sets and sample reference image sets at the first frame rate, and a subset of sample event stream data corresponding to each sample frame-to-interpolation image set is extracted from the sample event stream data.

[0036] Based on the sample images extracted during the frame extraction process, the acquisition time of the sample images, the set of sample images to be inserted, the set of sample reference images, and the subset of sample event stream data, a sample dataset is constructed.

[0037] Based on the sample dataset, a preset machine learning model is trained to obtain the frame interpolation synthesis network.

[0038] In one embodiment, constructing the sample dataset includes:

[0039] Based on the set of sample frames to be interpolated and the set of sample reference images, the sample spatial alignment optical flow field is calculated. Based on the sample spatial alignment optical flow field, the sample event stream data subset is aligned with the sample frames to be interpolated to obtain sample alignment event data.

[0040] Based on the sampled frame images and their acquisition time, the set of sample frames to be interpolated, the sample alignment event data, and the acquisition time are used as input features, and the sampled frame images are used as output features to construct a sample dataset.

[0041] Secondly, this application provides a video frame interpolation device applied to a dual-camera system, the dual-camera system including a main camera and an event camera, the video frame interpolation device comprising:

[0042] The image acquisition module is used to acquire the set of interpolated frame images captured by the main camera at the first frame rate, as well as the event stream data output by the event camera and the set of reference images acquired synchronously with the set of interpolated frame images;

[0043] The optical flow calculation module is used to calculate the spatially aligned optical flow field between the main camera and the event camera based on the set of frames to be interpolated and the set of reference images;

[0044] The event alignment module is used to align the event stream data with the set of frames to be interpolated based on the spatially aligned optical flow field to obtain aligned event data;

[0045] The image interpolation module is used to obtain the target interpolated image based on the set of images to be interpolated and the alignment event data.

[0046] Thirdly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0047] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0048] The aforementioned video frame interpolation method, apparatus, computer equipment, and storage medium acquire a set of images to be interpolated captured by the main camera at a first frame rate, as well as event stream data output by the event camera and a set of reference images acquired synchronously with the set of images to be interpolated; calculate the spatially aligned optical flow field between the main camera and the event camera based on the set of images to be interpolated and the reference image set; align the event stream data with the set of images to be interpolated based on the spatially aligned optical flow field to obtain aligned event data; obtain the target interpolated image based on the set of images to be interpolated and the aligned event data; and obtain high-fidelity image structure information provided by the main camera and the event camera. This method provides high temporal resolution dynamic events unaffected by exposure blur. Under the influence of spatially aligned optical flow fields, the two achieve cross-modal coordinate system unification, enabling the event stream data to accurately correspond to the real motion trajectory in the main camera image coordinate system. Together, they drive the interpolation algorithm to infer pixel changes in intermediate frames. At the same time, since the event data maintains a high signal-to-noise ratio under high-speed motion and low light conditions, and spatial alignment ensures its geometric consistency with the image frame, the video frame interpolation method in this embodiment can still accurately reconstruct pixel motion trajectories even in cases of occlusion, blurring, or missing information areas. Without relying on traditional optical flow estimation, it avoids artifacts and structural distortion, achieving the effect of improving frame interpolation accuracy in complex dynamic scenes. Attached Figure Description

[0049] Figure 1 This is a diagram illustrating the application environment of a video frame interpolation method in one embodiment.

[0050] Figure 2 This is a flowchart illustrating a video frame interpolation method in one embodiment;

[0051] Figure 3 This is a flowchart illustrating a video frame interpolation method in another embodiment;

[0052] Figure 4 This is a schematic diagram of the structure of a video frame interpolation model in one embodiment;

[0053] Figure 5 This is a structural block diagram of a video frame interpolation device in one embodiment;

[0054] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] The video frame interpolation method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the processor 102 communicates with both the main camera 104 and the event camera 106. The memory 108 stores the data that the processor 102 needs to process. The memory 108 can be integrated with the processor 102 in a dual-camera system. The processor 102 acquires the set of interpolated frame images captured by the main camera 104 at a first frame rate, as well as the event stream data output by the event camera 106 and a set of reference images synchronously acquired with the set of interpolated frame images; it calculates the spatial alignment optical flow field between the main camera 104 and the event camera 106 based on the set of interpolated frame images and the reference image set; it aligns the event stream data with the set of interpolated frame images based on the spatial alignment optical flow field to obtain aligned event data; and it obtains the target interpolated frame image based on the set of interpolated frame images and the aligned event data. The dual-camera system can be integrated into a server or terminal device, which can be a personal computer, laptop, smartphone, tablet, IoT device, or other device that requires video frame interpolation processing of the acquired images. For example, IoT devices can be smart TVs, smart in-vehicle devices, etc.

[0057] The main camera described in this application can be an optoelectronic imaging device that acquires image sequences at a fixed frame rate, providing high-resolution image input relative to the event camera. In this embodiment, the main camera can periodically expose the scene within its field of view using a global shutter, rolling shutter mechanism, etc., converting the light signal into a digital image in the form of a pixel matrix. For example, the main camera can be an RGB camera, a multispectral camera, etc. It is understood that the video frame interpolation method provided in this application can be applied to scenarios where hardware conditions are low, such as high resolution, insufficient power, or low sampling rate, resulting in a low frame rate for the main camera and requiring an increase in frame rate. It can also be applied to scenarios where, for cost considerations, the main camera acquires images at a lower frame rate relative to the event camera, and the frame rate is increased by combining event stream data from the event camera.

[0058] An event camera can be a vision sensor triggered by pixel-level brightness changes. By outputting event data when it detects changes in light intensity, it can capture dynamic information under conditions of high-speed motion and low light. For example, an event camera can employ a Dynamic Vision Sensor (DVS), an Asynchronous Time-based Image Sensor (ATIS), or a Dynamic and Active Vision Sensor (DAVIS).

[0059] In one embodiment, such as Figure 2 As shown, a video frame interpolation method is provided, which is applied to... Figure 1 Taking processor 102 as an example, the dual-camera system includes a main camera and an event camera. The video frame interpolation method includes the following steps:

[0060] Step S100: Obtain the set of frames to be interpolated captured by the main camera at the first frame rate, as well as the event stream data output by the event camera and the set of reference images captured synchronously with the set of frames to be interpolated.

[0061] The set of images to be interpolated can be a continuous image sequence captured by the main camera at the first frame rate, and may include at least two images to be interpolated. The video frame interpolation method in this embodiment is used to generate an intermediate frame, i.e., an interpolated image, between two images to be interpolated. It can be understood that when the set of images to be interpolated includes three or more images, adjacent images can be selected sequentially according to the event sequence to generate the intermediate frame, such as selecting the first and second images, the second and third images, etc. This embodiment will not elaborate on this.

[0062] Event stream data can be a discrete set of events indexed by time series, output by an event camera. Each event can contain data on spatial location and brightness changes, thereby describing dynamic changes in the scene. For example, event stream data can be generated in real time and continuously output by the event camera's pixel-level comparator and time encoding circuit, achieving low-latency, high-temporal-resolution dynamic information acquisition.

[0063] The reference image set can be a sequence of auxiliary images from the main camera, acquired synchronously with the image set to be interpolated, used for spatial registration to assist in establishing the spatial mapping relationship between the event camera and the main camera. In this embodiment, the reference image set can be a set of image frames acquired by the event camera at the same timestamp as the image set to be interpolated.

[0064] For example, the set of images to be interpolated, the set of reference images, and the event stream data can be acquired in real time. For instance, when applied to a terminal with an integrated dual-camera system, the processor can read and cache continuous frame data from the image output interface of the main camera and the image output interface of the event camera respectively according to the time interval corresponding to the first frame rate, and receive the event stream data in real time and cache the timestamp and coordinate information according to the time interval corresponding to the second frame rate.

[0065] In another example, the set of images to be interpolated, the set of reference images, and the event stream data can be obtained during post-processing. For example, when applied to the server side, the set of images and the event stream data can be obtained via wired transmission or remotely to achieve cloud-based frame interpolation processing of the set of images to be interpolated.

[0066] Step S200: Calculate the spatially aligned optical flow field between the main camera and the event camera based on the set of images to be interpolated and the set of reference images.

[0067] Among them, the spatially aligned optical flow field can be a vector field that characterizes the spatial transformation relationship between the pixel coordinates of the event camera and the pixel coordinates of the main camera, and can be used for spatial alignment of event data and the frame image to be interpolated.

[0068] For example, the spatially aligned optical flow field is based on feature matching between the reference image set and the frame image set to be interpolated. Through optical flow algorithms, it calculates pixel-level displacement mapping across sensors, thereby establishing a spatial correspondence between the event camera and the main camera.

[0069] Furthermore, feature point matching can be used, or a convolutional neural network model can be employed. By inputting the main camera image and the reference image respectively, a cross-modal spatial transformation field can be obtained, thereby establishing a spatial correspondence between the event camera and the main camera, providing an accurate and reliable mapping basis for the spatial reprojection of the event stream.

[0070] Step S300: Align the event stream data with the set of frames to be interpolated based on the spatially aligned optical flow field to obtain aligned event data.

[0071] The aligned event data can be event stream data aligned with the main camera coordinate system. In this embodiment, the pixel coordinates in the original event stream can be spatially transformed according to the spatially aligned optical flow field to make them consistent with the spatial position of the main camera image frame, thereby ensuring the spatial alignment of the event data and the image frame.

[0072] For example, each event can be mapped according to the spatially aligned optical flow field, offsetting the event position to the corresponding position of the main camera image grid. This can eliminate the event position deviation caused by spatial misalignment between sensors and ensure that the event data and image frames are strictly synchronized in space.

[0073] Step S400: Obtain the target interpolated image based on the set of images to be interpolated and the alignment event data.

[0074] The target interpolated image can be an intermediate image frame generated between the original frames to increase the frame rate, serving as the output of the video interpolation method. For example, the target interpolated image can be obtained by fusing texture information from the set of images to be interpolated with temporal change information from alignment event data, and then reconstructing pixel values ​​using algorithms such as neural networks and interpolation models.

[0075] Furthermore, the timestamp corresponding to the target interpolated image can be the average of adjacent times. For example, when the first image is obtained based on the first timestamp and the image corresponding to the second timestamp in the set of frames to be interpolated, the timestamp of the target interpolated image can be the average of the first timestamp and the second timestamp, or it can be set according to the interpolation algorithm.

[0076] It is understandable that multiple target interpolated images can be generated between two images to be interpolated, and the timestamps of these multiple target interpolated images can be determined by evenly dividing the difference in timestamps between the two images to be interpolated. Continuing with the example of the first and second timestamps above, when generating two target interpolated images, the timestamps of the first and second target interpolated images can be located at one-third and two-thirds of the time window between the first and second timestamps, respectively.

[0077] Based on the set of images to be interpolated and alignment event data, the target interpolated image is obtained. This can be achieved by inputting the pixel information of the images to be interpolated and the temporal change information of the alignment event data into the interpolation model to generate intermediate frames. For example, a multimodal fusion network based on U-Net can be used, inputting RGB frames from the image channel and an event accumulation map from the event channel to obtain the output interpolated frame. Alternatively, an optical flow-guided interpolator based on a physical motion model can be constructed, using the event flow to derive pixel motion trajectories and interpolate pixel values. This allows for the fusion of high-fidelity image structure and high-precision event temporal information, generating structurally complete and artifact-free intermediate frames in complex dynamic scenes.

[0078] This embodiment provides a video frame interpolation method, which involves acquiring a set of images to be interpolated captured by a main camera at a first frame rate, event stream data output by an event camera, and a set of reference images acquired synchronously with the set of images to be interpolated; calculating a spatially aligned optical flow field between the main camera and the event camera based on the set of images to be interpolated and the reference image set; aligning the event stream data with the set of images to be interpolated based on the spatially aligned optical flow field to obtain aligned event data; and obtaining a target interpolated image based on the set of images to be interpolated and the aligned event data. The main camera provides high-fidelity image structure information, while the event camera provides information unaffected by... High temporal resolution dynamic events affected by exposure blur are integrated with the spatially aligned optical flow field to achieve cross-modal coordinate system unification. This allows the event stream data to accurately correspond to the real motion trajectory in the main camera image coordinate system, jointly driving the interpolation algorithm to infer pixel changes in intermediate frames. At the same time, since the event data maintains a high signal-to-noise ratio under high-speed motion and low light conditions, and spatial alignment ensures its geometric consistency with the image frame, the video frame interpolation method in this embodiment can still accurately reconstruct pixel motion trajectories even in cases of occlusion, blurring, or missing information areas. Without relying on traditional optical flow estimation, it avoids artifacts and structural distortion, achieving the effect of improving frame interpolation accuracy in complex dynamic scenes.

[0079] In one embodiment, acquiring the set of interpolated frame images captured by the main camera at a first frame rate, and the event stream data output by the event camera and the set of reference images acquired synchronously with the set of interpolated frame images include:

[0080] The first image captured by the main camera at the first moment and the second image captured at the second moment are obtained to obtain a set of images to be interpolated; the time interval between the first moment and the second moment corresponds to the first frame rate;

[0081] The third image captured by the event camera at the first moment and the fourth image captured at the second moment are obtained to obtain a set of reference images;

[0082] Acquire the event stream data output by the event camera between the first and second moments.

[0083] The first and second images are single-frame images captured by the main camera at the first and second moments, respectively. The third and fourth images are single-frame images captured by the event camera at the first and second moments, respectively. The first moment can be the starting point of the synchronous capture triggered by the main camera and the event camera, and also represents the time boundary of the frame interpolation interval. The second moment can be the ending point of the synchronous capture triggered by the main camera and the event camera, and also represents the time boundary of the frame interpolation interval.

[0084] The time interval can be the time difference between the first moment and the second moment, i.e., the target interval where the frame is interpolated. The time interval between the first moment and the second moment corresponds to the first frame rate. It can be determined based on the first frame rate. For example, when the video frame rate is 24 frames per second, the time interval between the first moment and the second moment can be 1 / 24 of a second.

[0085] The first image captured by the main camera at the first moment and the second image captured at the second moment are obtained to obtain a set of images to be interpolated. This can be done by combining the first image and the second image in chronological order into a two-frame image sequence. For example, the first image and the second image can be stored in a preset path for further processing, or an array structure containing the first image and the second image can be constructed and marked with its timestamp.

[0086] Accordingly, the third image captured by the event camera at the first moment and the fourth image captured at the second moment are obtained to form a reference image set. This can be achieved by combining the third and fourth images into a two-frame image sequence in chronological order. For example, the third and fourth images can be stored in a preset path for further processing, or an array structure containing the third and fourth images can be constructed and marked with timestamps to provide a visual reference for the event camera at the start and end times of the interpolation interval, which is used to accurately calculate the cross-modal space aligned optical flow field.

[0087] The event stream data output by the event camera between the first and second moments can be obtained by continuously receiving the event stream data output by the event camera within the time window from the first to the second moment, thereby obtaining high temporal resolution dynamic change information within the frame interpolation interval and providing a blur-free motion trajectory prior for frame interpolation.

[0088] This embodiment provides a video frame interpolation method. It obtains a set of images to be interpolated by acquiring a first image captured by a main camera at a first moment and a second image captured at a second moment; it obtains a set of reference images by acquiring a third image captured by an event camera at a first moment and a fourth image captured at a second moment; and it acquires the event stream data output by the event camera between the first and second moments. By minimizing the amount of input data, strengthening time boundary control, and improving the synchronization accuracy between the event stream and the images, accurate pixel change reconstruction of intermediate frames can be achieved using only two frames and the intermediate event stream, thus improving the accuracy of frame interpolation in complex dynamic scenes.

[0089] In one embodiment, calculating the spatially aligned optical flow field between the main camera and the event camera, based on the set of frames to be interpolated and the set of reference images, includes:

[0090] The set of images to be interpolated and the set of reference images are input into the pre-trained first optical flow estimation network to obtain the spatially aligned optical flow field.

[0091] The pre-trained first optical flow estimation network can be an optical flow computation model trained end-to-end on a large-scale dynamic visual dataset based on a deep learning architecture. It can be used to extract high-precision, sub-pixel-level spatial motion fields from the set of images to be interpolated and the set of reference images, improving the robustness of cross-modal alignment. In this embodiment, the pre-trained first optical flow estimation network learns the nonlinear mapping relationship between image pixel changes and corresponding motion vectors. After training, the parameters are fixed for the inference stage, enabling cross-modal motion modeling between the images to be interpolated from the main camera and the reference images from the event camera. For example, the pre-trained first optical flow estimation network can be a convolutional network using a pyramid structure to enhance optical flow computation, a global motion modeling network based on a transformer mechanism, or a bidirectional optical flow network with an occlusion awareness mechanism, etc. This embodiment does not limit the specific implementation.

[0092] The set of images to be interpolated and the set of reference images are input into a pre-trained first optical flow estimation network to obtain a spatially aligned optical flow field. Alternatively, a pair of images to be interpolated and the reference images can be used as dual-channel inputs to obtain a two-dimensional motion vector field output, which is used for spatial alignment between the main camera and the event camera. This enables the generation of a high-fidelity, continuous, and high-precision spatial optical flow motion field under one or more conditions such as complex dynamics, blur, low light, and texture loss, providing a reliable geometric mapping basis for the spatial reprojection of event stream data.

[0093] This embodiment provides a video frame interpolation method that obtains a spatially aligned optical flow field by inputting a set of images to be interpolated and a set of reference images into a pre-trained first optical flow estimation network. A depth model is introduced for motion modeling, achieving a stable output of the spatial optical flow motion field in complex dynamic scenes. Under one or more conditions such as complex dynamics, blurring, low illumination, and texture loss, it can improve the accuracy of spatial reprojection of event flow and enhance cross-modal alignment robustness, thereby improving the frame interpolation accuracy in complex dynamic scenes.

[0094] In one embodiment, the first optical flow estimation network further outputs a first optical flow confidence set corresponding to the spatially aligned optical flow field; after obtaining the spatially aligned optical flow field, the method further includes:

[0095] If the first optical flow confidence set does not meet the preset anomaly condition, then the optical flow values ​​in the first optical flow confidence set with confidence levels lower than the first confidence threshold are identified to obtain the optical flow value to be optimized; based on the optical flow values ​​with confidence levels higher than the second confidence threshold within the preset range of the optical flow value to be optimized, the optical flow value to be optimized is filled and smoothed.

[0096] If the first set of optical flow confidence scores meets the preset abnormal conditions, then frame interpolation of the set of images to be interpolated will be stopped.

[0097] The first optical flow confidence set can be a set including the confidence values ​​of motion estimates for each pixel in the spatially aligned optical flow field. For example, the first optical flow estimation network can output the confidence value corresponding to the motion estimate of each pixel in the spatially aligned optical flow field while simultaneously outputting the spatially aligned optical flow field. The first optical flow confidence set can be obtained from the confidence values ​​of multiple pixel motion estimates.

[0098] Determining whether the first optical flow confidence set meets the preset anomaly conditions can be achieved by performing a global statistical analysis on the first optical flow confidence set to determine whether it conforms to the preset anomaly judgment rules.

[0099] The preset anomaly condition can be a global judgment standard used to determine whether the overall quality of the spatially aligned optical flow field is effective. For example, the preset anomaly condition can be a region proportion threshold with confidence levels below a preset confidence threshold, a distribution entropy threshold with confidence levels below a preset confidence threshold, an extreme value distribution condition with confidence levels below a preset confidence threshold, or other conditions based on confidence levels for anomaly judgment. Furthermore, the preset confidence threshold can be dynamically determined based on one or more of the first optical flow confidence set, the image to be interpolated, and the reference image. For example, the confidence levels in the first confidence set can be sorted, and the confidence level corresponding to the preset qualified proportion position can be used as the preset confidence threshold. Alternatively, the complexity of the current scene can be determined by performing sharpness calculations and contour extraction on the image to be interpolated and / or the reference image, and the confidence threshold corresponding to that complexity can be mapped as the preset confidence threshold. Other methods for dynamically determining the dynamic confidence threshold are also possible, and this embodiment does not limit this approach.

[0100] When the first set of optical flow confidence scores does not meet the preset anomaly conditions, it indicates that the current spatially aligned optical flow field has a certain degree of reliability, but there may still be cases where the accuracy of optical flow motion estimation is poor. In such cases, optical flow values ​​can be compensated for in regions with low confidence scores. It is understandable that when there are no optical flow values ​​with confidence scores lower than the first confidence threshold in the first set of optical flow confidence scores, the optimization process for optical flow values ​​can be skipped.

[0101] For example, identifying optical flow values ​​with confidence levels below a first confidence threshold in the first optical flow confidence set as optical flow values ​​to be optimized can be achieved by traversing the first optical flow confidence set and marking all optical flow vectors below the first confidence threshold as objects to be repaired. Furthermore, low-confidence regions can be filtered out through masking operations, and the boundaries of low-confidence regions can be expanded using a morphological dilation algorithm, thereby accurately locating unreliable regions in the optical flow field and isolating erroneous motion information.

[0102] In this embodiment, the first confidence threshold can be a local discrimination threshold used to distinguish whether an optical flow vector is a low-confidence outlier, and is used to filter out the optical flow values ​​to be optimized. The optical flow values ​​to be optimized can be a set of optical flow vectors identified as having a confidence level lower than the first confidence threshold, that is, an abnormal region that needs to be repaired in the spatially aligned optical flow field.

[0103] The second confidence threshold can be a set of reliable optical flow values ​​within a preset range used to filter the optical flow values ​​to be optimized. As a confidence threshold for reliable optical flow values, it ensures that the neighborhood information involved in smoothing is sufficiently reliable. In an exemplary embodiment, the second confidence threshold is greater than or equal to the first confidence threshold. Smoothing the optical flow values ​​to be optimized based on optical flow values ​​within a preset range with a confidence level higher than the second confidence threshold can be achieved by, for each optical flow value to be optimized, filtering optical flow vectors within its preset range with a confidence level higher than the second confidence threshold, and then replacing, superimposing, or weighting the original values ​​using methods such as weighted averaging or interpolation. This allows for the repair of abnormal optical flow while maintaining local motion continuity.

[0104] Conversely, when the first optical flow confidence set meets the preset anomaly condition, it indicates that the reliability of the current spatially aligned optical flow field has fallen below the allowable level. Aligning the set of images to be interpolated with the event stream data based on this spatially aligned optical flow field may result in abnormal alignment, causing the interpolated images to be inferior to expectations. Therefore, interpolation of the set of images to be interpolated can be stopped. Furthermore, subsequent interpolation calculation steps can be skipped, and the original frame or empty frame identifier can be output.

[0105] This embodiment provides a video frame interpolation method that outputs a first optical flow confidence set through a first optical flow estimation network and determines whether a preset anomaly condition is met based on this set. If the condition is not met, low-confidence optical flow values ​​are identified and filled and smoothed. If the condition is met, frame interpolation is stopped. By identifying low-confidence regions and using high-confidence information from the neighborhood for repair, and by actively terminating the process when the preset anomaly condition is met, the reliability of the optical flow field under extreme conditions can be significantly improved without relying on additional sensors or higher computing power, thereby achieving the technical effect of improving the frame interpolation accuracy in complex dynamic scenes.

[0106] In one embodiment, the method further includes inputting the set of images to be interpolated and the set of reference images into a pre-trained first optical flow estimation network, and then:

[0107] The set of images to be interpolated is preprocessed to obtain the preprocessed set of images to be interpolated.

[0108] In this embodiment, the image to be interpolated can be a color image, and the reference image can be a grayscale image. Preprocessing of the set of images to be interpolated may include grayscale conversion and resizing.

[0109] Grayscale conversion can be a linear weighted process that converts a multi-channel color image into a single-channel luminance image to eliminate color channel redundancy and enhance the dominant role of luminance variations in motion estimation. In an exemplary embodiment, grayscale conversion can perform channel-weighted summation for each pixel according to a standard luminance formula, outputting a single-channel grayscale value, making optical flow estimation more focused on motion structures dominated by the true luminance gradient.

[0110] Size adjustment can be a resampling operation that uniformly transforms the spatial resolution of the image sequence to the input size required by the first optical flow estimation network. In this embodiment, size adjustment can be performed by using bilinear interpolation, region averaging, or other methods to scale the image while maintaining the aspect ratio, through cropping or padding. This ensures that the scale of the input image is consistent with the receptive field during network training, avoiding motion vector blurring, spatial sampling distortion, or feature alignment deviation caused by resolution mismatch, thereby improving the accuracy of optical flow estimation.

[0111] The preprocessed set of images to be interpolated can be a set of images to be interpolated after grayscale conversion and size adjustment, or a set of images after the number of channels has been compressed to a single channel and the spatial resolution has been standardized.

[0112] This embodiment provides a video frame interpolation method that performs grayscale processing and resizing on the set of images to be interpolated before inputting them into the first optical flow estimation network. Grayscale processing eliminates color redundancy and noise interference in the multi-channel color images, allowing optical flow estimation to focus on the real motion structure dominated by the brightness gradient. Resizing ensures that the scale of the input matches the receptive domain during network training, avoiding motion vector blurring or sampling errors caused by resolution mismatch. This achieves the technical effect of improving the signal-to-noise ratio and structural consistency of the input data through deterministic operations without relying on additional models, thereby improving the accuracy of frame interpolation in complex dynamic scenes.

[0113] In one embodiment, the target interpolated image is obtained based on the set of images to be interpolated and alignment event data, including:

[0114] The set of images to be interpolated, alignment event data, and target interpolation time are input into a pre-trained interpolation synthesis network to obtain the target interpolated image.

[0115] The pre-trained frame interpolation synthesis network can be an end-to-end deep learning architecture consisting of two sub-networks, used to fuse image information and event stream data to generate intermediate frames. For example, the pre-trained frame interpolation synthesis network can be trained under supervision using a labeled dataset containing a sequence of main images and event stream data aligned with that sequence, with the goal of minimizing the pixel differences between the generated frames and the real intermediate frames, as well as minimizing perceptual loss, to obtain the trained frame interpolation synthesis network.

[0116] The frame interpolation synthesis network includes a second optical flow estimation network and a frame interpolation result synthesis network.

[0117] The second optical flow estimation network can be used to generate bidirectional optical flow and a fusion mask from the target interpolation time to the set of images to be interpolated, based on the set of images to be interpolated, alignment event data, and the target interpolation time. For example, the second optical flow estimation network can take the set of images to be interpolated, alignment event data, and the target interpolation time as input, extract spatiotemporal features through a multimodal feature encoder, and output forward and backward optical flow and a fusion mask through an optical flow prediction head. For example, it can employ a convolution-based optical flow estimation network, an event-image joint attention optical flow network, or an optical flow prediction network guided by temporal interpolation points.

[0118] The target interpolation time can be the precise position of the intermediate frame to be generated on the time axis, which can be used to constrain the time alignment of optical flow estimation and interpolation processes. For example, the target interpolation time can be determined by the interpolation requirements, can be the midpoint between two adjacent frames, or can be any time proportion of a specified or corresponding target frequency.

[0119] Bidirectional optical flow can be a pixel motion vector field describing the target's motion at each frame interpolation time to the preceding and following frames, containing motion information in both forward and backward directions. In an exemplary embodiment, the bidirectional optical flow is output by a second optical flow estimation network, where each pixel can correspond to two vectors, one pointing to the previous frame and the other to the following frame, thereby enabling the modeling of non-rigid and asymmetric motion.

[0120] The fusion mask can be a channel map consistent with the image resolution, where each pixel value can represent a confidence weight indicating whether the interpolation at that location should prioritize image information or event information. For example, the fusion mask can be generated in the feature space by a second optical flow estimation network using a sigmoid activation function.

[0121] The frame interpolation result synthesis network can be used to perform frame interpolation calculations based on bidirectional optical flow, a fusion mask, and a set of images to be interpolated, to obtain the target interpolated image. For example, the frame interpolation result synthesis network can obtain the target interpolated image by backsampling the preceding and following frames using bidirectional optical flow, weighting and fusing the sampling results using a fusion mask, and compensating for nonlinear motion errors through a residual correction module.

[0122] This embodiment provides a video frame interpolation method. By inputting the set of images to be interpolated, alignment event data, and the target interpolation time into a pre-trained frame interpolation synthesis network, a second optical flow estimation network generates bidirectional optical flow and a fusion mask. The interpolation result synthesis network then performs frame interpolation calculations. By introducing the pre-trained frame interpolation synthesis network, the second optical flow estimation network combines image structure and event dynamic information, and generates bidirectional optical flow and a fusion mask under the constraint of the target interpolation time. This enables refined modeling of non-rigid motion, occlusion, and blurred regions. The generated bidirectional optical flow provides a complete forward and backward motion path, and the fusion mask can dynamically allocate information weights between the image and the event. Based on this, the frame interpolation result synthesis network performs pixel-level backsampling and adaptive weighted fusion, thereby achieving complete reconstruction and recovery of motion trajectories in complex dynamic scenes. This improves the accuracy of frame interpolation in complex dynamic scenes without relying on a higher acquisition frame rate.

[0123] In one embodiment, frame interpolation calculations are performed based on bidirectional optical flow, a fusion mask, and a set of images to be interpolated, to obtain the target interpolated image, which includes:

[0124] The set of images to be interpolated is reverse-mapped to the target interpolation time based on bidirectional optical flow to obtain the mapped image;

[0125] The target interpolated image is obtained by weighted fusion of the fusion mask and the mapped image.

[0126] The mapped image can be a pixel-reconstructed image generated by back-projecting the preceding and following frames in the set of images to be interpolated to the target interpolation time according to bidirectional optical flow. For example, the mapped image may include a backward mapped image corresponding to the first frame to be interpolated and a forward mapped image corresponding to the second frame to be interpolated. In a specific embodiment, the preceding frame may be backsampled according to backward optical flow, and the following frame may be backsampled according to forward optical flow, thereby generating two intermediate projected images spatially aligned with the target time.

[0127] The target interpolated image is obtained by weighted fusion of the fusion mask and the mapped image. For example, the two mapped images can be linearly combined according to the pixel weights of the fusion mask to obtain the target interpolated image. This can achieve pixel-level adaptive fusion, which prioritizes the preservation of original image information in areas with clear textures and relies on the mapping result with stronger motion consistency in blurred or occluded areas, effectively suppressing artifacts and enhancing structural integrity.

[0128] This embodiment provides a video frame interpolation method that reverse-maps the set of images to be interpolated to the target interpolation time according to bidirectional optical flow to obtain a mapped image. Then, it performs weighted fusion based on a fusion mask and the mapped image to obtain the target interpolated image. By spatially resampling the preceding and following frame images according to their corresponding backward and forward optical flow vectors, the pixel positions are aligned to the coordinate system of the target interpolation time. Finally, the two mapped images are linearly combined according to the pixel weights of the fusion mask to obtain the target interpolated image. This method can improve the accuracy of frame interpolation in complex dynamic scenes.

[0129] In one embodiment, the frame interpolation synthesis network further includes a frame interpolation feasibility determination module, which is used to:

[0130] The key points in the set of images to be interpolated are identified separately to obtain the first key point features and the second key point features.

[0131] Calculate the feature matching stability based on the features of the first key point and the features of the second key point;

[0132] The motion reliability is obtained by calculating the optical flow consistency based on the bidirectional optical flow.

[0133] Based on the second optical flow confidence set, feature matching stability, and motion confidence, the frame interpolation risk score of the set of images to be interpolated is determined;

[0134] If the frame interpolation risk score does not meet the preset conditions, frame interpolation filtering or weight reduction will be performed based on the risk areas in the frame interpolation risk score.

[0135] Here, image key points can be local feature points with significant structural responses in the set of images to be interpolated. In this embodiment, image key points can be obtained by extracting pixel positions with high response values ​​from previous and subsequent frames through corner detection, deep learning models, etc.

[0136] The first keypoint feature can be a local descriptive feature of the keypoint in the previous frame of the image set to be interpolated. The second keypoint feature can be a local descriptive feature of the keypoint in the next frame of the image set to be interpolated.

[0137] Feature matching stability can be an indicator that measures the consistency of matching between the features of the first keypoint and the features of the second keypoint under spatial transformation, and can be used to evaluate the repeatability of local structures in the time dimension. For example, feature matching stability can be generated by calculating the similarity distance and a threshold used to evaluate the similarity distance, thereby identifying feature mismatch phenomena caused by motion blur, occlusion, changes in illumination, etc.

[0138] Motion reliability can be based on the reciprocal consistency analysis results of bidirectional optical flow, used to evaluate the rationality of forward and backward optical flow vectors under the physical motion model. In this embodiment, motion reliability can be verified by back-projecting the forward and backward optical flow, calculating whether the pixel returns to its original position after forward and backward motion; the greater the deviation, the lower the reliability. Motion reliability can include, but is not limited to, one or more of the following: high-consistency motion regions, low-consistency occlusion regions, and cyclic error accumulation regions.

[0139] Optical flow consistency calculations based on bidirectional optical flow yield motion confidence. This can be achieved by applying forward optical flow to each pixel and then mapping it backward using backward optical flow, calculating the offset error as the reciprocal of the confidence score. Furthermore, this operation can be implemented by constructing an optical flow cyclic consistency loss function, assigning low confidence scores to regions with errors exceeding a threshold. This can effectively detect motion inconsistencies caused by occlusion, non-rigid deformation, or optical flow estimation errors, thereby improving the ability to identify abnormal motion.

[0140] The second optical flow estimation network also outputs a second optical flow confidence set corresponding to the bidirectional optical flow. This second optical flow confidence set can be a set including the confidence values ​​of motion estimates for each pixel in the corresponding bidirectional optical flow. For example, the second optical flow estimation network can output the confidence value corresponding to the motion estimate of each pixel in the bidirectional optical flow while simultaneously outputting the bidirectional optical flow. The second optical flow confidence set can be obtained by using the confidence values ​​of multiple pixel motion estimates.

[0141] The frame interpolation risk score can be a pixel-level risk assessment map generated by fusing the second optical flow confidence set, feature matching stability, and motion confidence, used to identify the unreliability of frame interpolation in each region. For example, the frame interpolation risk score can be obtained by weighted summation after normalizing the second optical flow confidence set, feature matching stability, and motion confidence, or by non-linear fusion using a lightweight neural network, outputting a risk score for each pixel.

[0142] A risk region can be a set of pixels in the interpolation risk scoring results whose scores exceed a preset risk threshold. It is used to characterize the interpolation results in that region as unreliable and requiring intervention. For example, the regions with scores higher than the preset risk threshold can be extracted by thresholding or binarizing the pixels based on the interpolation risk scoring results; these are the risk regions.

[0143] Determine whether the frame interpolation risk score meets the preset conditions. For example, the preset conditions may be that the risk area in the frame interpolation risk assessment result does not reach the preset proportion, or that the entropy value of the risk area does not meet the preset entropy value threshold, or that there are no pixels in the frame interpolation risk assessment result that exceed the maximum risk threshold, etc. This embodiment will not elaborate on these conditions.

[0144] If the interpolation risk score does not meet the preset conditions, interpolation filtering or weight reduction processing is performed based on the risk areas in the interpolation risk score results. This can be done by reducing the weight of areas with risk scores exceeding a threshold or blocking their participation in interpolation during the interpolation fusion stage. For example, during the weighted fusion stage, the mask value of the corresponding risk area in the fusion mask can be lowered. Alternatively, the neighborhood mean can be used to replace the interpolation result. Or, after the mapped image is generated, hole filling can be performed on high reliability risk areas, or low-pass filtering, copying neighboring pixels, etc. can be used to replace them, thereby actively suppressing unreliable motion information from participating in interpolation, avoiding artifact propagation, and using conservative strategies to maintain structural integrity in high reliability risk areas.

[0145] Furthermore, for regions where the risk score exceeds the threshold, the input frame can be copied only, or no interpolation can be performed, or the two consecutive frames in the image to be interpolated can be weighted and fused to obtain the interpolation result for that region.

[0146] This embodiment provides a video frame interpolation method that outputs a second optical flow confidence set through a second optical flow estimation network. The frame interpolation feasibility determination module identifies key points in the image and extracts features of the first and second key points. It calculates feature matching stability and motion credibility, and fuses them to generate a frame interpolation risk score. When the score does not meet the conditions, the risk area is filtered or downweighted. This method can integrate multi-dimensional risk information such as the uncertainty of quantified optical flow estimation, the consistency of image structure matching, and the rationality of optical flow motion to implement local intervention strategies. This can effectively suppress artifact propagation and structural distortion, thereby improving the visual coherence and structural fidelity of the frame interpolation results in complex dynamic scenes.

[0147] In one embodiment, the training process of the frame interpolation synthesis network includes:

[0148] Acquire a main camera image sample set captured by the main camera or a camera of the same type at the second frame rate, as well as sample event stream data output by the event camera and a reference image sample set acquired synchronously with the main camera image sample set; the second frame rate is greater than the first frame rate;

[0149] Based on the first frame rate, frames are extracted from the main camera image sample set and the reference image sample set respectively to determine multiple sample frame interpolation image sets and sample reference image sets under the first frame rate, and a subset of sample event stream data corresponding to each sample frame interpolation image set is extracted from the sample event stream data.

[0150] Based on the sample images extracted during the frame extraction process, the acquisition time of the sample images, the set of sample images to be inserted, the set of sample reference images, and the subset of sample event stream data, a sample dataset is constructed.

[0151] Based on the sample dataset, a pre-defined machine learning model is trained to obtain a frame interpolation synthesis network.

[0152] The main camera image sample set can be a set of image samples captured by the main camera or a camera of the same type at a second frame rate, used to provide complete visual information of real high-speed motion. For example, the frame rate of the main camera image sample set can be the same as or higher than the frame rate corresponding to the set of images to be interpolated after interpolation. In an exemplary embodiment, the main camera image sample set can be used by a high frame rate imaging system to continuously capture dynamic scenes, outputting uncompressed or lightly compressed image sequences to ensure complete recording of high-speed motion trajectories and time alignment with the event data of the main camera.

[0153] The reference image sample set can be an auxiliary image sequence acquired synchronously with the main camera image sample set for spatial registration, used to assist in establishing the spatial mapping relationship between the event camera and the main camera. In this embodiment, the reference image sample set can be a set of image frames acquired by the event camera at the same timestamp as the main camera image sample set.

[0154] Sample event stream data can be a set of discrete events synchronously acquired by the event camera and time-aligned with the main camera image sample set. Each event can contain data on spatial location and brightness changes, thereby describing the dynamic changes in the scene. It can be used to characterize high temporal resolution dynamic information of pixel-level brightness changes in the scene.

[0155] The second frame rate can be a higher temporal resolution frame rate used by the main camera when acquiring sample data, relative to the first frame rate, which can capture the complete trajectory of real high-speed motion. In this embodiment, the second frame rate is higher than the first frame rate.

[0156] It is understandable that the main camera may be able to provide high frame rate image acquisition in a test environment. However, in a production environment, due to various external environmental factors such as insufficient power and insufficient channel frequency, the main camera may need to reduce the frame rate, or, for cost considerations and to cooperate with external environmental factors, remove some modules, resulting in the need to acquire images at a lower frame rate, such as the first frame rate.

[0157] Therefore, during training, frames can be extracted from the image set at the second frame rate using the first frame rate to obtain multiple sets of sample frames to be interpolated, a set of sample reference images, and the corresponding subset of sample event stream data. Among these:

[0158] The sample set of interpolated frames can be a low-frame-rate image sequence generated by extracting frames from the main camera image sample set at a first frame rate. This sequence can be used to simulate real interpolation input and, together with the event stream data, constitutes multimodal input features. In an exemplary embodiment, the sample set of interpolated frames can be formed by uniformly extracting image frames from the main camera image sample set at time intervals of the first frame rate, creating an image sequence with consistent time intervals and ensuring time synchronization with the event stream data and reference images.

[0159] The sample reference image set can be an auxiliary image sequence generated synchronously with the sample interpolation frame set. For example, the sample reference image set can be generated at the same frame extraction time points as the sample interpolation frame set.

[0160] A subset of sample event stream data can be event stream data extracted from the sample event stream data that matches the time range of the sample set of frames to be interpolated. In an exemplary embodiment, extracting a subset of sample event stream data corresponding to each set of sample frames to be interpolated can be done by extracting event stream data within a corresponding time period based on the timestamp range of the set of sample frames to be interpolated.

[0161] A sample image extracted during the frame extraction process can be a specific image frame that is not included in the main camera image sample set. It can be understood that a sample image extracted can be the actual result corresponding to the acquisition time of the extracted image, given a set of sample images to be interpolated, a set of sample reference images, and a subset of sample event stream data as input. This allows for comparison of the loss between the model output and the actual result, enabling optimization iteration.

[0162] Based on the first frame rate, frames are extracted from the main camera image sample set and the reference image sample set respectively. This can be done by uniformly extracting image frames from the high frame rate sample set at time intervals according to the first frame rate to form a low frame rate sequence, thereby constructing a low frame rate input sample that is consistent with the actual deployment environment.

[0163] The sample dataset can be a multimodal training dataset consisting of sampled frames, the acquisition time of the sampled frames, a set of sample frames to be interpolated, a set of sample reference images, and a subset of sample event stream data. In an exemplary embodiment, the sample dataset can be organized into structured sample units by aligning the above elements in a time-aligned manner. Each sample contains an input image pair, a reference image, an event subset, and a timestamp, and is used for training the multimodal frame interpolation network.

[0164] The preset machine learning model can be a deep neural network architecture with trainable parameters, which can be used to learn the mapping relationship from low frame rate images and event streams to intermediate frames. In one specific embodiment, the preset machine learning model can be a U-Net-based multimodal frame interpolation network, a Transformer-based temporal fusion network, or an optical flow-guided generative adversarial network; this embodiment is not limited to any particular model.

[0165] This embodiment provides a video frame interpolation method that acquires complete motion trajectories in a high frame rate acquisition environment and systematically extracts frames according to the first frame rate of actual deployment to construct training samples consistent with real application scenarios. This exposes the preset machine learning model to the input distribution of real deployment conditions during the training phase, forcing the network to learn motion reconstruction rules under extreme conditions such as low frame rate, high dynamics, low texture, and occlusion, rather than relying on idealized interpolation modes. This enables the trained frame interpolation synthesis network to accurately fuse complementary information from blurred images and event streams, significantly improving the spatiotemporal consistency and structural fidelity of the interpolation results in actual deployment. Without relying on hardware upgrades, it achieves the technical effect of improving the accuracy of frame interpolation in complex dynamic scenes.

[0166] In one embodiment, constructing the sample dataset includes:

[0167] Based on the set of sample frames to be interpolated and the set of sample reference images, the sample spatial alignment optical flow field is calculated. Based on the sample spatial alignment optical flow field, the sample event stream data subset is aligned with the sample frames to be interpolated to obtain the sample alignment event data.

[0168] Based on the sample images and their acquisition time, a sample dataset is constructed by using the set of sample images to be inserted, the sample alignment event data, and the acquisition time as input features, and the sample images as output features.

[0169] The sample space-aligned optical flow field can be a vector field characterizing the spatial transformation relationship between the event camera pixel coordinates and the main camera pixel coordinates, and can be used for spatial alignment of event data and the frame image to be interpolated. In an exemplary embodiment, based on feature matching between the sample reference image set and the sample frame image set to be interpolated, the pixel displacement mapping across sensors can be calculated through an optical flow estimation algorithm, and combined with linear interpolation to obtain the sample space-aligned optical flow field, so that the event flow data can be mapped to the main camera image coordinate system.

[0170] The sample alignment event data can be a subset of sample event stream data that is strictly synchronized with the coordinate system of the main camera image after being mapped by the sample spatial alignment optical flow field. The sample event stream data subset is aligned with the sample frame image to be interpolated based on the sample spatial alignment optical flow field to obtain the sample alignment event data. This can be done in the same way as in any of the above embodiments, where the event stream data is aligned with the set of frame images to be interpolated based on the spatial alignment optical flow field; this embodiment will not repeat the details.

[0171] Based on the sample images and their acquisition time, the set of images to be interpolated, the sample alignment event data, and the acquisition time are used as input features, and the sample images are used as output features to construct a sample dataset. The input and output features can be organized into structured training sample units to form end-to-end supervision pairs. This allows the construction of supervised learning samples that are completely consistent with the real deployment environment, enabling the model to directly learn the physical laws of co-reconstructing real intermediate frames from blurred images and event streams during the training phase.

[0172] This embodiment provides a video frame interpolation method that calculates the spatially aligned optical flow field of the sample space by using a set of sample images to be interpolated and a set of sample reference images. This enables precise spatial registration between the event camera and the main camera in the training samples. By using extracted sample images as output features and the set of sample images to be interpolated, sample aligned event data, and acquisition time as input features, an end-to-end supervised sample is constructed. This allows the trained frame interpolation synthesis network to have the ability to deeply model the relationship between blur and event collaboration in real dynamic scenes. This achieves the technical effect of improving the accuracy of frame interpolation in complex dynamic scenes without relying on higher frame rate acquisition hardware or reducing production costs.

[0173] To more clearly illustrate the technical solution of this application, a detailed embodiment is also provided.

[0174] In one embodiment, a video frame interpolation method is provided to address the defective interpolation results caused by inaccurate motion vector estimation in existing video frame interpolation techniques. The method inputs low-frame-rate continuous video frames captured by the main camera, low-frame-rate video frames captured by the secondary camera, and high-frame-rate event frames into a trained multi-source fusion frame interpolation network to obtain high-frame-rate interpolation results. Figure 3 As shown, this video frame interpolation method includes the following steps:

[0175] Step S1: Acquire image data and event data through the sensor assembly. The sensor assembly is a dual-camera module consisting of a high-resolution RGB main camera and a low-resolution DAVIS event camera. The DAVIS event camera can simultaneously capture high-frame-rate event pixels and low-frame-rate scene grayscale imaging results. Image data includes RGB image frames output by the RGB main camera and grayscale image frames output by the DAVIS event camera, while event data consists of high-frame-rate time frames output by the DAVIS event camera.

[0176] Step S2: Based on the image data and event data obtained in Step S1, the RGB image frames are converted to grayscale and resized. Using the optical flow estimation network V, the geometric relationship between the main camera's RGB image frames and the secondary camera's APS grayscale frames is learned in real time, and the optical flow field and optical flow confidence score are output. Anomaly detection is performed on the optical flow confidence score. If an abnormal scene is detected, no frame interpolation is performed; otherwise, optical flow post-processing is performed as if it were a normal scene.

[0177] Step S3: Based on the high frame rate event sequence obtained in step S1 and the optical flow field obtained in step S2, the event camera data collected by the secondary camera is mapped to the image space of the main camera, i.e., event frame alignment, so as to achieve spatial alignment between the event data and the main camera image.

[0178] In step S4, the optical flow estimation network G performs fusion encoding on the multimodal data based on the RGB image of the main camera obtained in step S1 and the aligned event data obtained in step S3, and combines the target time scale parameter (time index). During the fusion process, feature representations at different spatial scales are extracted.

[0179] Step S5: Based on the fusion features obtained in step S4, decode to obtain the bidirectional optical flow field and the fusion mask.

[0180] In step S6, the interpolation result synthesis network performs inverse transformation processing based on the bidirectional optical flow field obtained in step S5 and the RGB image of the main camera obtained in step S1, and performs weighted fusion on the transformed image results based on the fusion mask to generate the interpolated image at the target time.

[0181] Step S7: Based on the optical flow results obtained in step S5 and the preliminary frame interpolation results obtained in step S6, and with reference to the judgment results of the anomaly detection mechanism, generate a reliable frame interpolation image at the target time.

[0182] In the above steps, the optical flow estimation network outputs the optical flow field, obtains the bidirectional optical flow length and fusion mask, and generates the interpolated image at the target time. This can be achieved through a pre-trained video interpolation model. This pre-trained interpolation model includes multiple sub-networks. The preset optical flow network ArcFlowNet (i.e., the optical flow estimation network V) is used to output the optical flow field. The preset interpolation network ArcVFINet includes the optical flow estimation network G and the interpolation result synthesis network, used to obtain the bidirectional optical flow length and fusion mask, and generate the interpolated image at the target time.

[0183] The training process of the video frame interpolation model in this embodiment includes: using the main camera to capture high frame rate image frames, the secondary camera to capture high frame rate image frames and even higher frame rate event frames, and extracting frames from the dual-camera high frame rate image frames to obtain high-low frame rate matching image frame pairs; inputting the extracted low frame rate image frames from the main and secondary cameras and the high frame rate event frames from the secondary camera into the neural network for training, and calculating the loss function based on the model training results and the extracted high frame rate image frames to obtain the trained video frame interpolation model. Specifically, it includes the following steps:

[0184] 1. Taking frame extraction at 3-frame intervals as an example, consecutive high frame rate image frames id_0, id_1, id_2, id_3, ..., id_N from the main and secondary cameras are extracted at 3-frame intervals to obtain the input images (id_0, id_4), ..., (id_N-4, id_N), which is the set of sample frames to be interpolated, and the extracted sample images gt (id_1, id_2, id_3), ..., (id_N-3, id_N-2, id_N-1). It can be understood that when the acquisition time t=0.25, gt is id_1; when the acquisition time t=0.5, gt is id_2; and when the acquisition time t=0.75, gt is id_3. This allows for the mapping from 30FPS video to 120FPS video. Simultaneously, the timestamps of the event camera data and image data within the time period id_0 to id_4 are aligned and decompressed into a continuous multi-frame image format.

[0185] It is understood that, in addition to the 3-frame interval method, other intervals of frames can also be used, such as 1 frame interval, 2 frames interval, etc., which are not limited in this embodiment.

[0186] In one specific embodiment, multiple sets of input images can be acquired sequentially with a preset step size. For example, when the preset step size is 1, the input images can be (id_0, id_4), (id_1, id_5), (id_2, id_6), etc., and correspondingly, the frame-stripped images can be (id_1, id_2, id_3), (id_2, id_3, id_4), (id_3, id_4, id_5), etc.

[0187] 2. The input images id_0 and id_4 from the main and secondary cameras, the event data EVS, and the acquisition time t are concatenated along the channel dimension to obtain the network's input features. Since this scheme needs to run on edge devices, to improve efficiency without sacrificing accuracy, the main camera image is downsampled during the network's input stage. In this embodiment, the Space2depth algorithm is used for downsampling to achieve image downsampling without information loss. For example, the Space2depth operation can be performed on the input before network computation, converting the original B×C×H×W data into B×C... 16×H / 4×W / 4, in contrast, the Depth2space operation is performed in the output stage of the network.

[0188] 3. The multi-source fusion frame interpolation network in this embodiment includes two stages: an alignment stage and an image frame interpolation stage. In traditional schemes, data is collected using an existing HS-ERGB dataset with a dual-camera system, and the data from both cameras is aligned using a calibrated scheme. Therefore, it only achieves good alignment for distant objects or objects at a fixed distance, while the alignment effect is poor for nearby objects or objects not on the same plane. Another traditional scheme uses a beam splitter to collect data via BS-ERGB or ERF-X170FPS, eliminating the need for data alignment; however, the beam splitter is bulky and unsuitable for mobile devices. This embodiment proposes an alignment scheme with higher accuracy and wider applicability as follows:

[0189] (1) In the alignment stage, the optical flow between the main camera and the secondary camera is estimated using the preset optical flow network ArcFlowNet, or other existing optical flow networks can be used. Specifically, the id_0 images of the main camera and the secondary camera are input into ArcFlowNet to obtain the optical flow result flow_0, and the id_4 images of the main camera and the secondary camera are input into ArcFlowNet to obtain the optical flow result flow_4. The optical flow of the intermediate frames is obtained by linear interpolation. At the same time, since the optical flow scheme is prone to errors in complex scenes, the alignment result is abnormal. ArcFlowNet outputs the optical flow confidence score pixel by pixel while outputting the optical flow. For areas with low confidence, the optical flow values ​​of the neighboring high-confidence areas are used for filling and smoothing to obtain better alignment results.

[0190] To enable the technical solution of this embodiment to run efficiently on different types of devices, the pre-defined optical flow network ArcFlowNet described in this embodiment includes depthwise separable convolutions, combined with channel attention and spatial attention mechanisms, thereby achieving efficient optical flow estimation without sacrificing computational accuracy. The pre-defined optical flow network ArcFlowNet can be pre-trained on the TartanAir dataset and then jointly trained with an interpolation network.

[0191] (2) In the frame interpolation stage, a preset interpolation network, ArcVFINet, is used to estimate the optical flow and fusion weights between the target frame and the input two frames. To achieve frame interpolation at any time, unlike existing methods such as Timelens, this embodiment additionally inputs data t∈(0,1) from one channel to control the timing of the model's frame generation. The Timelens series of schemes divides the EVS data according to t, into EVS data from 0 to t and EVS data from t-1, thereby controlling the timing of the interpolation result. However, such methods require repeated division of EVS data, which leads to a large number of repetitive operations and is not suitable for running on edge devices. ArcVFINet, on the other hand, only requires input of data from a single channel, which can easily modify the time step.

[0192] Similar to ArcFlowNet, ArcVFINet can also include depthwise separable convolutions, along with attention modules. The network as a whole adopts an encoder-decoder architecture and can be pre-trained on the Inter4K dataset.

[0193] like Figure 4 The diagram shows the structure of the video frame interpolation model in this embodiment. The input image is processed through a pyramid structure, with a scale-related feature pyramid as the backbone network. Then, it passes through the optical flow estimation network ArcFlowNet, outputting bidirectional optical flow lengths (Flows) and a fusion mask. These are then encapsulated with residuals and output to ArcVFINet (FUSION) to produce the interpolated frame result. This embodiment's model structure is hardware-friendly, and combined with its training process, it can effectively improve the performance and effectiveness of the deployed model.

[0194] 4. ArcFlowNet and ArcVFINet are integrated into an end-to-end framework for joint optimization, avoiding the error accumulation and process redundancy issues of the two-stage method. For the calculation of aligned optical flow, GMFlow is used to generate pseudo-labels, which are then used to train ArcFlowNet in a weakly supervised manner during training. In the initial stage of training, ArcVFINet can be frozen, and only ArcFlowNet can be trained. After ArcFlowNet converges to a certain extent, both modules can be trained simultaneously, which can achieve faster convergence.

[0195] This embodiment provides a video frame interpolation method that can achieve the desired effect.

[0196] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0197] Based on the same inventive concept, this application also provides an apparatus for implementing the method described above. The solution provided by this apparatus is similar to the solution described in the above method; therefore, specific limitations in one or more apparatus embodiments provided below can be found in the limitations of the method described above, and will not be repeated here.

[0198] In one embodiment, such as Figure 5 As shown, this application provides a video frame interpolation device applied to a dual-camera system, the dual-camera system including a main camera and an event camera, the video frame interpolation device including:

[0199] The image acquisition module 100 is used to acquire the set of frames to be interpolated captured by the main camera at the first frame rate, as well as the event stream data output by the event camera and the set of reference images acquired synchronously with the set of frames to be interpolated.

[0200] The optical flow calculation module 200 is used to calculate the spatially aligned optical flow field between the main camera and the event camera based on the set of frames to be interpolated and the set of reference images.

[0201] The event alignment module 300 is used to align the event stream data with the set of frames to be interpolated based on the spatially aligned optical flow field to obtain aligned event data.

[0202] The image interpolation module 400 is used to obtain the target interpolated image based on the set of images to be interpolated and alignment event data.

[0203] In one embodiment, the image acquisition module 100 is further configured to:

[0204] The first image captured by the main camera at a first moment and the second image captured at a second moment are obtained to obtain a set of images to be interpolated; the time interval between the first moment and the second moment corresponds to the first frame rate;

[0205] The third image captured by the event camera at the first moment and the fourth image captured at the second moment are obtained to obtain a reference image set;

[0206] Acquire the event stream data output by the event camera between the first time and the second time.

[0207] In one embodiment, the optical flow calculation module 200 is further configured to:

[0208] The set of images to be interpolated and the set of reference images are input into a pre-trained first optical flow estimation network to obtain the spatially aligned optical flow field.

[0209] In one embodiment, the optical flow calculation module 200 is further configured to:

[0210] If the first optical flow confidence set does not meet the preset abnormal conditions, then the optical flow values ​​in the first optical flow confidence set with confidence levels lower than the first confidence threshold are identified to obtain the optical flow value to be optimized; and the optical flow value to be optimized is filled and smoothed according to the optical flow values ​​with confidence levels higher than the second confidence threshold within the preset range of the optical flow value to be optimized.

[0211] If the first optical flow confidence set meets the preset abnormal conditions, then frame interpolation of the set of images to be interpolated will be stopped.

[0212] In one embodiment, the optical flow calculation module 200 is further configured to:

[0213] The set of images to be interpolated is preprocessed to obtain a preprocessed set of images to be interpolated; the preprocessing includes grayscale conversion and size adjustment.

[0214] In one embodiment, the image frame interpolation module 400 is further configured to:

[0215] The set of images to be interpolated, the alignment event data, and the target interpolation time are input into a pre-trained interpolation synthesis network to obtain the target interpolated image. The interpolation synthesis network includes a second optical flow estimation network and an interpolation result synthesis network. The second optical flow estimation network is used to generate a bidirectional optical flow and a fusion mask from the target interpolation time to the set of images to be interpolated, based on the set of images to be interpolated, the alignment event data, and the target interpolation time. The interpolation result synthesis network is used to perform interpolation calculations based on the bidirectional optical flow, the fusion mask, and the set of images to be interpolated to obtain the target interpolated image.

[0216] In one embodiment, the image frame interpolation module 400 is further configured to:

[0217] The set of images to be interpolated is reverse-mapped to the target interpolation time according to the bidirectional optical flow to obtain the mapped image;

[0218] The target interpolated image is obtained by weighted fusion of the fusion mask and the mapped image.

[0219] In one embodiment, the second optical flow estimation network further outputs a second optical flow confidence set corresponding to the bidirectional optical flow; the frame interpolation synthesis network further includes a frame interpolation feasibility determination module, which is used for:

[0220] The key points of the images in the set of images to be interpolated are identified respectively to obtain the first key point feature and the second key point feature;

[0221] Calculate the feature matching stability based on the features of the first key point and the features of the second key point;

[0222] The motion reliability is obtained by calculating the optical flow consistency based on the bidirectional optical flow.

[0223] Based on the second optical flow confidence set, the feature matching stability, and the motion confidence, the frame interpolation risk score of the set of images to be interpolated is determined;

[0224] If the frame interpolation risk score does not meet the preset conditions, then frame interpolation filtering or weight reduction processing is performed based on the risk areas in the frame interpolation risk score.

[0225] In one embodiment, the apparatus further includes a network training module, which is further configured to:

[0226] Acquire a main camera image sample set captured by the main camera or a camera of the same type as the main camera at a second frame rate, as well as sample event stream data output by the event camera and a reference image sample set acquired synchronously with the main camera image sample set; the second frame rate is greater than the first frame rate;

[0227] Based on the first frame rate, frames are extracted from the main camera image sample set and the reference image sample set respectively to determine multiple sample frame-to-interpolation image sets and sample reference image sets at the first frame rate, and a subset of sample event stream data corresponding to each sample frame-to-interpolation image set is extracted from the sample event stream data.

[0228] Based on the sample images extracted during the frame extraction process, the acquisition time of the sample images, the set of sample images to be inserted, the set of sample reference images, and the subset of sample event stream data, a sample dataset is constructed.

[0229] Based on the sample dataset, a preset machine learning model is trained to obtain the frame interpolation synthesis network.

[0230] In one embodiment, the network training module is further configured to:

[0231] Based on the set of sample frames to be interpolated and the set of sample reference images, the sample spatial alignment optical flow field is calculated. Based on the sample spatial alignment optical flow field, the sample event stream data subset is aligned with the sample frames to be interpolated to obtain sample alignment event data.

[0232] Based on the sampled frame images and their acquisition time, the set of sample frames to be interpolated, the sample alignment event data, and the acquisition time are used as input features, and the sampled frame images are used as output features to construct a sample dataset.

[0233] Each module in the aforementioned video frame interpolation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0234] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a video frame interpolation method. The display screen can be an LCD screen or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0235] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0236] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the video frame interpolation method of any of the above embodiments:

[0237] The system acquires the set of frames to be interpolated captured by the main camera at the first frame rate, as well as the event stream data output by the event camera and the set of reference images acquired synchronously with the set of frames to be interpolated.

[0238] Based on the set of frames to be interpolated and the set of reference images, calculate the spatially aligned optical flow field between the main camera and the event camera;

[0239] The event stream data is aligned with the set of frames to be interpolated based on the spatially aligned optical flow field to obtain aligned event data;

[0240] The target interpolated image is obtained based on the set of images to be interpolated and the alignment event data.

[0241] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the video frame interpolation method of any of the above embodiments:

[0242] The system acquires the set of frames to be interpolated captured by the main camera at the first frame rate, as well as the event stream data output by the event camera and the set of reference images acquired synchronously with the set of frames to be interpolated.

[0243] Based on the set of frames to be interpolated and the set of reference images, calculate the spatially aligned optical flow field between the main camera and the event camera;

[0244] The event stream data is aligned with the set of frames to be interpolated based on the spatially aligned optical flow field to obtain aligned event data;

[0245] The target interpolated image is obtained based on the set of images to be interpolated and the alignment event data.

[0246] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0247] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0248] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0249] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video frame interpolation method, characterized in that, Applied to a dual-camera system, the dual-camera system including a main camera and an event camera, the video frame interpolation method includes: The system acquires the set of frames to be interpolated captured by the main camera at the first frame rate, as well as the event stream data output by the event camera and the set of reference images acquired synchronously with the set of frames to be interpolated. Based on the set of frames to be interpolated and the set of reference images, calculate the spatially aligned optical flow field between the main camera and the event camera; The event stream data is aligned with the set of frames to be interpolated based on the spatially aligned optical flow field to obtain aligned event data; The target interpolated image is obtained based on the set of images to be interpolated and the alignment event data.

2. The video frame interpolation method according to claim 1, characterized in that, The acquisition of the set of frames to be interpolated captured by the main camera at the first frame rate, and the event stream data output by the event camera and the set of reference images acquired synchronously with the set of frames to be interpolated, include: The first image captured by the main camera at a first moment and the second image captured at a second moment are obtained to obtain a set of images to be interpolated; the time interval between the first moment and the second moment corresponds to the first frame rate; The third image captured by the event camera at the first moment and the fourth image captured at the second moment are obtained to obtain a reference image set; Acquire the event stream data output by the event camera between the first time and the second time.

3. The video frame interpolation method according to claim 1, characterized in that, The step of calculating the spatially aligned optical flow field between the main camera and the event camera based on the set of frames to be interpolated and the set of reference images includes: The set of images to be interpolated and the set of reference images are input into a pre-trained first optical flow estimation network to obtain the spatially aligned optical flow field.

4. The video frame interpolation method according to claim 3, characterized in that, The first optical flow estimation network also outputs a first optical flow confidence set corresponding to the spatially aligned optical flow field; after obtaining the spatially aligned optical flow field, the process further includes: If the first optical flow confidence set does not meet the preset abnormal conditions, then the optical flow values ​​in the first optical flow confidence set with confidence levels lower than the first confidence threshold are identified to obtain the optical flow value to be optimized; and the optical flow value to be optimized is filled and smoothed according to the optical flow values ​​with confidence levels higher than the second confidence threshold within the preset range of the optical flow value to be optimized. If the first optical flow confidence set meets the preset abnormal conditions, then frame interpolation of the set of images to be interpolated will be stopped.

5. The video frame interpolation method according to claim 4, characterized in that, Before inputting the set of images to be interpolated and the set of reference images into the pre-trained first optical flow estimation network, the following steps are also included: The set of images to be interpolated is preprocessed to obtain a preprocessed set of images to be interpolated; the preprocessing includes grayscale conversion and size adjustment.

6. The video frame interpolation method according to claim 1, characterized in that, The step of obtaining the target interpolated frame image based on the set of frames to be interpolated and the alignment event data includes: The set of images to be interpolated, the alignment event data, and the target interpolation time are input into a pre-trained interpolation synthesis network to obtain the target interpolated image. The interpolation synthesis network includes a second optical flow estimation network and an interpolation result synthesis network. The second optical flow estimation network is used to generate a bidirectional optical flow and a fusion mask from the target interpolation time to the set of images to be interpolated, based on the set of images to be interpolated, the alignment event data, and the target interpolation time. The interpolation result synthesis network is used to perform interpolation calculations based on the bidirectional optical flow, the fusion mask, and the set of images to be interpolated to obtain the target interpolated image.

7. The video frame interpolation method according to claim 6, characterized in that, The step of performing frame interpolation calculations based on the bidirectional optical flow, the fusion mask, and the set of images to be interpolated to obtain the target interpolated image includes: The set of images to be interpolated is reverse-mapped to the target interpolation time according to the bidirectional optical flow to obtain the mapped image; The target interpolated image is obtained by weighted fusion of the fusion mask and the mapped image.

8. The video frame interpolation method according to claim 7, characterized in that, The second optical flow estimation network also outputs a second optical flow confidence set corresponding to the bidirectional optical flow; the frame interpolation synthesis network further includes a frame interpolation feasibility determination module, which is used for: The key points of the images in the set of images to be interpolated are identified respectively to obtain the first key point feature and the second key point feature; Calculate the feature matching stability based on the features of the first key point and the features of the second key point; The motion reliability is obtained by calculating the optical flow consistency based on the bidirectional optical flow. Based on the second optical flow confidence set, the feature matching stability, and the motion confidence, the frame interpolation risk score of the set of images to be interpolated is determined; If the frame interpolation risk score does not meet the preset conditions, then frame interpolation filtering or weight reduction processing is performed based on the risk areas in the frame interpolation risk score.

9. The video frame interpolation method according to claim 7, characterized in that, The training process of the frame interpolation synthesis network includes: Acquire a main camera image sample set captured by the main camera or a camera of the same type as the main camera at a second frame rate, as well as sample event stream data output by the event camera and a reference image sample set acquired synchronously with the main camera image sample set; the second frame rate is greater than the first frame rate; Based on the first frame rate, frames are extracted from the main camera image sample set and the reference image sample set respectively to determine multiple sample frame-to-interpolation image sets and sample reference image sets at the first frame rate, and a subset of sample event stream data corresponding to each sample frame-to-interpolation image set is extracted from the sample event stream data. Based on the sample images extracted during the frame extraction process, the acquisition time of the sample images, the set of sample images to be inserted, the set of sample reference images, and the subset of sample event stream data, a sample dataset is constructed. Based on the sample dataset, a preset machine learning model is trained to obtain the frame interpolation synthesis network.

10. The video frame interpolation method according to claim 9, characterized in that, The constructed sample dataset includes: Based on the set of sample frames to be interpolated and the set of sample reference images, the sample spatial alignment optical flow field is calculated. Based on the sample spatial alignment optical flow field, the sample event stream data subset is aligned with the sample frames to be interpolated to obtain sample alignment event data. Based on the sampled frame images and their acquisition time, the set of sample frames to be interpolated, the sample alignment event data, and the acquisition time are used as input features, and the sampled frame images are used as output features to construct a sample dataset.

11. A video frame interpolation device, characterized in that, Applied to a dual-camera system, the dual-camera system including a main camera and an event camera, the video frame interpolation device includes: The image acquisition module is used to acquire the set of interpolated frame images captured by the main camera at the first frame rate, as well as the event stream data output by the event camera and the set of reference images acquired synchronously with the set of interpolated frame images; The optical flow calculation module is used to calculate the spatially aligned optical flow field between the main camera and the event camera based on the set of frames to be interpolated and the set of reference images; The event alignment module is used to align the event stream data with the set of frames to be interpolated based on the spatially aligned optical flow field to obtain aligned event data; The image interpolation module is used to obtain the target interpolated image based on the set of images to be interpolated and the alignment event data.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 11.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 11.

Images