Methods, apparatus and equipment for video frame interpolation

Abstract

This disclosure provides methods, apparatus, and devices for video frame interpolation, relating to the field of multimedia processing. In the method for video frame interpolation, a first image at a first time point, a second image at a second time point, and sensing data collected by a dynamic visual sensing device are acquired. The sensing data includes dynamic event data between the first and second time points. Based on the first image, the second image, and the sensing data, at least one target image is determined, whereby the at least one target image corresponds to at least one target time point between the first and second time points. Thus, dynamic event data is used to help compensate for missing motion information in existing image data, achieving accurate prediction of intermediate images and improving image prediction performance.

Description

Technical Field

[0001] The embodiments of this disclosure primarily relate to the field of multimedia processing, and more specifically, to methods, apparatus, and devices for video frame interpolation. Background Technology

[0002] Video frame interpolation refers to predicting and inserting one or more intermediate frames between adjacent original frames to obtain a higher frame rate video. Currently, video frame interpolation technology has received considerable research attention, as it overcomes the temporal resolution limitations of recorded video and shows great potential in numerous tasks such as slow-motion generation, video editing, and virtual reality. For example, people have increasingly higher demands for the frame rate and content richness of video images on terminal devices, but the frame rate of videos shot by ordinary cameras is limited by physical mechanisms and cannot provide high frame rate videos. Therefore, video frame interpolation technology is needed to enhance video images on terminal devices. Furthermore, it is clear that mainstream movies mostly use low frame rates, which can easily lead to discontinuity and blurring in high-speed scenes when facing high refresh rates, greatly affecting the consumer experience. In such cases, dynamic video frame interpolation technology can also provide higher frame rate videos. In addition, dynamic video frame interpolation technology can also alleviate the shortage of high frame rate video content on mobile devices to some extent. Summary of the Invention

[0003] The embodiments of this disclosure provide a scheme for video frame interpolation.

[0004] In a first aspect of this disclosure, a method for video frame interpolation is provided. The method includes: acquiring a first image at a first time, a second image at a second time, and sensing data collected by a dynamic visual sensing device, the sensing data including dynamic event data between the first and second times; and determining at least one target image based on the first image, the second image, and the sensing data, wherein the at least one target image is an image corresponding to at least one target time between the first and second times.

[0005] According to embodiments of this disclosure, dynamic event data is used to help compensate for the missing motion information in existing image data. Through optical flow estimation, more nonlinear motion information can be obtained, thereby ensuring the interpolation of nonlinear motion information in complex scenes, achieving accurate prediction of intermediate images, and predicting images with better results.

[0006] In one implementation of the first aspect, at least one target image includes a first target image corresponding to a first target time between a first time and a second time. Determining at least one target image based on the first image, the second image, and sensing data includes: determining a first optical flow from the first target time to the first time based on a first portion of sensing data, wherein the first portion of sensing data includes dynamic event data between the first time and the first target time; determining a second optical flow from the first target time to the second time based on a second portion of sensing data, wherein the second portion of sensing data includes dynamic event data between the first target time and the second time; and performing frame interpolation on the first image and the second image based on the first optical flow and the second optical flow to obtain the first target image corresponding to the first target time. In this manner, the optical flow between two time points can be estimated using dynamic event data, thereby enabling frame interpolation using an optical flow method.

[0007] In another implementation of the first aspect, performing frame interpolation on the first image and the second image based on the first optical flow and the second optical flow includes: converting the first image into a first intermediate image based on the first optical flow; converting the second image into a second intermediate image based on the second optical flow; and merging the first intermediate image and the second intermediate image to obtain a first target image. In this manner, an image at a predicted intermediate time point can be obtained by transforming images from known start and end times to intermediate time points, and then used to merge them into a target image.

[0008] In another implementation of the first aspect, merging the first intermediate image and the second intermediate image to obtain the first target image includes: adjusting the first optical flow based on the first image, the first portion of sensing data, and the first intermediate image to obtain a first adjusted optical flow; converting the first image into a third intermediate image based on the first adjusted optical flow; and merging the third intermediate image and the second intermediate image to obtain the first target image. By adjusting the optical flow, more accurate motion information can be obtained, thereby achieving better image generation.

[0009] In another implementation of the first aspect, merging the third intermediate image and the second intermediate image to obtain the first target image includes: adjusting the second optical flow based on the second image, the second portion of sensing data, and the second intermediate image to obtain a second adjusted optical flow; converting the second image into a fourth intermediate image based on the second adjusted optical flow; and merging the third intermediate image and the fourth intermediate image to obtain the first target image. Similarly, adjustments can be made to another optical flow used in frame interpolation to obtain more accurate motion information for better image generation.

[0010] In another implementation of the first aspect, merging the third and fourth intermediate images to obtain the first target image includes: determining a first fusion weight for the third intermediate image and a second fusion weight for the fourth intermediate image, wherein the first fusion weight indicates the importance of corresponding pixels in the third intermediate image and the second fusion weight indicates the importance of corresponding pixels in the fourth intermediate image; and performing a weighted merging of the third and fourth intermediate images based on the first and second fusion weights to obtain the first target image. By determining the fusion weights, more important pixels can have a greater impact on the target image, resulting in a higher proportion and probability of being retained in the target image. This can further improve the accuracy of the target image.

[0011] In another implementation of the first aspect, determining the first fusion weight and the second fusion weight includes: determining the first fusion weight based on the first image, the first portion of sensor data, the first optical flow, and the first intermediate image; and determining the second fusion weight based on the second image, the second portion of sensor data, the second optical flow, and the second intermediate image. The fusion weights can also be determined using existing images, sensor data, and optical flow, thereby enabling a more accurate determination of the weights of each pixel.

[0012] In another implementation of the first aspect, the method further includes: organizing the first image, the second image, and at least one target image into a target video segment in chronological order. This allows low-frame-rate, or even static, images to be grouped into a high-frame-rate video segment, suitable for video generation needs in various application scenarios.

[0013] In another implementation of the first aspect, the first image and the second image comprise a first video frame at a first time and a second video frame at a second time within a video segment. In yet another implementation of the first aspect, the first image and the second image comprise still images acquired by a still imaging device. By utilizing dynamic event data, frame interpolation can be performed on both dynamic video segments and still images, thus expanding the applicability of frame interpolation technology.

[0014] In another implementation of the first aspect, determining at least one target image based on the first image, the second image, and the sensor data includes: applying the first image, the second image, and the sensor data to a trained video interpolation model to obtain at least one target image output by the video interpolation model. In such an implementation, automatic and accurate video interpolation can be achieved by learning and training a model using machine learning algorithms.

[0015] In a second aspect of this disclosure, a method for training a video frame interpolation model is provided. The method includes acquiring a first sample image at a first sample time, a second sample image at a second sample time, and sample sensing data, the sample sensing data including dynamic event data between the first and second sample times; applying the first sample image, the second sample image, and the sample sensing data to the video frame interpolation model to obtain a first predicted image corresponding to a target sample time, the target sample time being between the first and second sample times; using the video frame interpolation model, generating at least one of a second predicted image corresponding to the first sample time and a third predicted image corresponding to the second sample time, based on the first sample image, the second sample image, the first predicted image, and the sample sensing data; and updating the parameter values ​​of the video frame interpolation model based on at least one error: a first error between the generated second predicted image and the first sample image, and a second error between the generated third predicted image and the second sample image.

[0016] According to the example embodiment of video frame interpolation disclosed herein, the reciprocating motion information of two frames can be obtained based on event data, and three frame interpolations and two supervisions are performed to complete the cycle consistency training process. This training method can complete two supervised frame interpolations using only two frames of data, which is less than the traditional method of using three frames of data to complete one supervised frame interpolation. It uses less frame data, performs more supervisions, and has better accuracy.

[0017] In one implementation of the second aspect, generating at least one of the second and third predicted images includes at least one of the following: applying a second sample image, a first predicted image, and sample sensing data to a video interpolation model to obtain the second predicted image; and applying a first sample image, a first predicted image, and sample sensing data to a video interpolation model to obtain the third predicted image. Predicting images at different sample times can be used to provide supervision information for the video interpolation model to facilitate model training.

[0018] In another implementation of the second aspect, applying the second sample image, the first predicted image, and the sample sensing data to a video interpolation model to obtain the second predicted image includes: using the video interpolation model to perform the following operations: determining a first sample optical flow from a first sample time to a target sample time based on a first portion of the sample sensing data, wherein the first portion of the sample sensing data includes dynamic event data between the first sample time and the target sample time; determining a second sample optical flow from the first sample time to a second sample time based on the sample sensing data; and performing a frame interpolation operation on the first predicted image and the second sample image based on the first sample optical flow and the second sample optical flow to obtain the second predicted image corresponding to the first sample time. Since the sample dynamic event data can indicate the reciprocating motion information between the start and end times, it can be used for optical flow at any time point and in any direction within this time range.

[0019] In another implementation of the second aspect, applying the first sample image, the first predicted image, and the sample sensing data to a video interpolation model to obtain the third predicted image includes: using the video interpolation model to perform the following operations: determining a third sample optical flow from a second sample time to a target sample time based on a second portion of the sample sensing data, wherein the second portion of the sample sensing data includes dynamic event data between the target sample time and the second sample time; determining a fourth sample optical flow from the second sample time to the first sample time based on the sample sensing data; and performing an interpolation operation on the first predicted image and the first sample image based on the third and fourth sample optical flows to obtain the third predicted image corresponding to the second sample time. Since the sample dynamic event data can indicate reciprocating motion information between the start and end times, it can be used for optical flow at any time point and in any direction within this time range.

[0020] In a third aspect of this disclosure, an apparatus for video frame interpolation is provided. The apparatus includes: an acquisition unit configured to acquire a first image at a first time, a second image at a second time, and sensing data collected by a dynamic visual sensing device, the sensing data including dynamic event data between the first and second times; and a frame interpolation unit configured to determine at least one target image based on the first image, the second image, and the sensing data, wherein the at least one target image is an image corresponding to at least one target time between the first and second times. In practical applications, the frame interpolation unit can be used to implement the method as described in the first aspect or any possible implementation thereof. The video frame interpolation apparatus may include functional modules for implementing the method as described in the first aspect or any possible implementation thereof.

[0021] In a fourth aspect of this disclosure, an apparatus for training a video frame interpolation model is provided. The apparatus includes: a sample acquisition unit configured to acquire a first sample image at a first sample time, a second sample image at a second sample time, and sample sensing data, the sample sensing data including dynamic event data between the first and second sample times; a first frame interpolation unit configured to apply the first sample image, the second sample image, and the sample sensing data to the video frame interpolation model to obtain a first predicted image corresponding to a target sample time, the target sample time being between the first and second sample times; a second frame interpolation unit configured to use the video frame interpolation model to generate at least one of a second predicted image corresponding to the first sample time and a third predicted image corresponding to the second sample time, based on the first sample image, the second sample image, the first predicted image, and the sample sensing data; and a parameter update unit configured to update the parameter values ​​of the video frame interpolation model based on at least one of the following errors: a first error between the generated second predicted image and the first sample image, and a second error between the generated third predicted image and the second sample image. In practical applications, the apparatus for training the video frame interpolation model may include functional modules for implementing the methods described in the second aspect or any possible implementation of the second aspect.

[0022] In a fifth aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one computing unit; at least one memory coupled to the at least one computing unit and storing instructions for execution by the at least one computing unit, wherein, when executed by the at least one computing unit, the instructions cause the device to perform the method of the first aspect or any implementation thereof.

[0023] In a sixth aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one computing unit; at least one memory coupled to the at least one computing unit and storing instructions for execution by the at least one computing unit, wherein, when executed by the at least one computing unit, the instructions cause the device to perform the method of the second aspect or any implementation thereof.

[0024] In a seventh aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the method of the first aspect or any implementation thereof.

[0025] In an eighth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the method of the second aspect or any implementation thereof.

[0026] In a ninth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions, which, when executed by a processor, cause a computer to perform some or all of the steps of the method in the first aspect or any implementation thereof.

[0027] In a tenth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions, which, when executed by a processor, cause a computer to perform some or all of the steps of the method in the second aspect or any implementation thereof.

[0028] Understandably, the apparatus for video frame interpolation in the third aspect, the apparatus for training the video frame interpolation model in the fourth aspect, the electronic devices in the fifth and sixth aspects, the computer storage media in the seventh and eighth aspects, or the computer program products in the ninth and tenth aspects, all provided above, are used to implement the method provided in the first aspect. Therefore, the explanations or descriptions regarding the first aspect also apply to the second, third, fourth, and fifth aspects. Furthermore, the beneficial effects achievable by the second, third, fourth, and fifth aspects can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0029] These and other aspects of the invention will become more apparent from the following description of several embodiments. Attached Figure Description

[0030] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0031] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure can be implemented is shown;

[0032] Figure 2 A schematic diagram of an example structure of a video frame interpolation apparatus according to some embodiments of the present disclosure is shown;

[0033] Figure 3 A schematic diagram of an example structure of a video frame interpolation model according to some embodiments of the present disclosure is shown;

[0034] Figure 4 A schematic diagram of the information flow during the processing of a video frame interpolation model according to some embodiments of the present disclosure is shown;

[0035] Figure 5A schematic diagram of a system for training a video frame interpolation model according to some embodiments of the present disclosure is shown;

[0036] Figure 6 A schematic diagram illustrating an example process for training a video frame interpolation model according to some embodiments of the present disclosure is shown;

[0037] Figure 7 A schematic diagram of a three-stage frame interpolation process during training of a video frame interpolation model according to some embodiments of the present disclosure is shown;

[0038] Figure 8 A flowchart of a process for video frame interpolation according to some embodiments of the present disclosure is shown;

[0039] Figure 9 A flowchart is shown illustrating a process for training a video frame interpolation model according to some embodiments of the present disclosure;

[0040] Figure 10 A block diagram of an apparatus for video frame interpolation according to some embodiments of the present disclosure is shown;

[0041] Figure 11 A block diagram of an apparatus for training a video frame interpolation model according to some embodiments of the present disclosure is shown; and

[0042] Figure 12 A block diagram of an example device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0043] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0044] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0045] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably. A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs; it typically includes an input layer, an output layer, and one or more hidden layers between the input and output layers.

[0046] Machine learning typically comprises three phases: training, testing, and usage (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively, until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the usage phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0047] In this article, "frame" or "video frame" refers to the individual images in a video clip. "Image" and "frame" are used interchangeably in this article. Multiple consecutive images can form a dynamic video clip, where each image is considered a frame.

[0048] In video frame interpolation, most methods assume uniform motion between consecutive frames to estimate the spatial position of objects in intermediate frames. Under this assumption, objects should move at a constant speed along a straight line between consecutive frames. While simple, this approach can lead to inaccurate motion estimation and, in many real-world scenarios, incorrect intermediate frame predictions and artifacts. Recent research has attempted to address the shortcomings of the uniform motion assumption by proposing multi-frame velocity and acceleration estimation and employing a quadratic motion model for better intermediate frame prediction. However, inaccurate motion prediction still exists in cases of large and complex motion. Currently, the better interpolation methods are primarily based on warped bidirectional optical flow. However, most of these methods also simply assume continuous frames between uniform motion and linear optical flow, which may fail to approximate complex nonlinear motions in the real world. Therefore, nonlinear methods have also been proposed to solve complex nonlinear motion problems by learning higher-order acceleration information between frames. However, errors in acceleration estimation can cause motion trajectories to deviate from the true values.

[0049] In some applications, machine learning techniques are used to train video frame interpolation models to perform video frame interpolation tasks. However, a problem exists: it is difficult to collect supervisory data for video frame interpolation in real-world scenarios. In a given low frame rate video, there are no paired real frames for model supervision. For example, given the original frames at times t0 and t1 extracted from a video, the video frame interpolation task aims to predict one or more intermediate frames between t0 and t1. However, in low frame rate videos, it is impossible to obtain real intermediate frames for training the video frame interpolation model.

[0050] Most existing video frame interpolation models are trained on synthetic datasets. For example, in low frame rate videos, supervised frame interpolation lacks benchmark data. Therefore, some existing methods typically extract frames from high frame rate videos recorded by high-speed cameras, using the remaining frames and the extracted frames as model input and supervision information, respectively, for training. However, these limitations significantly increase the cost of acquiring training data, restrict the dataset size, and may lead to a gap between the source training domain and the target domain. Furthermore, there are domain differences between synthetic and real intermediate frames. Such supervised methods cannot fine-tune the target low frame rate image sequence, potentially causing performance degradation when the model is applied for actual frame interpolation in the application phase.

[0051] Some methods have established self-supervised frame interpolation methods based on cycle consistency, supervising the original input frames. These methods rely on cycle consistency, where multiple intermediate frames are predicted and then used to reconstruct intermediate input frames. However, to achieve cycle consistency, these methods must perform uniform motion over large time steps between consecutive frames, which exposes them to the same problems as methods based on the assumption of uniform motion. Cycle consistency is widely used to establish constraints without direct supervision, such as in 3D dense communication, visual relation disambiguation, and unpaired image-to-image transformation. When video frame interpolation tasks face challenges, self-supervised methods based on cycle consistency can learn behavior from arbitrary target low-frame-rate video sequences and synthesize high-frame-rate interpolations. However, to achieve cycle consistency, this method requires multiple input frames and assumes uniform motion over large time steps between consecutive frames, leading to artifacts caused by inaccurate motion prediction.

[0052] According to embodiments of this disclosure, an improved video frame interpolation scheme is proposed. Addressing the problem of traditional frame-based cameras lacking intermediate information, this scheme introduces dynamic event data to perform prediction of intermediate frames. The dynamic event data is collected by a motion sensing device. Since the motion sensing device can sense changes in the intensity of continuous optical flow, it can store rich inter-frame information, which is very useful for recovering intermediate frames and helps alleviate the difficulty of complex motion modeling in video frame interpolation. In some embodiments, the introduction of dynamic event data can also effectively help achieve accurate prediction of the video frame interpolation model in the absence of self-supervised information for video frame interpolation.

[0053] The exemplary embodiments of this disclosure will be discussed in detail below with reference to the accompanying drawings.

[0054] Example environment and example implementation of video frame interpolation

[0055] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, the environment 100 includes an imaging device 110, a dynamic visual sensing (DVS) device 120, and a video frame interpolation device 130.

[0056] Imaging device 110 may include a dynamic imaging device or a static imaging device. A dynamic imaging device can acquire dynamic image data, such as video. A static imaging device can acquire static image data, such as discrete still images. In some embodiments, imaging device 110 may include one or more cameras, webcams, etc. Imaging device 110 can acquire one or more still images, a certain length of video, or an animated GIF, etc., in a specific scene. Figure 1In the example, imaging device 110 provides images 112-1, 112-2, 112-3, ..., 112-N (collectively or individually referred to as images 112). In a video context, image 112 may also be referred to as a video frame (or simply a "frame") in the video.

[0057] DVS device 120 is used to acquire sensing data. The sensing data acquired by DVS device 120 includes dynamic event data 122. DVS device 120 may include, or be referred to as, an event camera, a dynamic vision sensor (DVS), a silicon retina, an event-based camera, or a frameless camera. DVS device 120 is a bio-inspired, event-driven, and time-based neuromorphic vision sensor. DVS device 120 can perceive the world on a principle quite different from traditional intensity cameras, recording events by asynchronously sensing dynamic changes in the brightness of each pixel, triggering events when changes exceed a certain threshold. Therefore, DVS device 120 generates and transmits data about changes in light intensity (i.e., dynamic event data), rather than a larger amount of data about the absolute intensity at each light sensor. The asynchronous event-driven processing allows the resulting dynamic event data 122 to perceive brightness changes at high resolution (e.g., microsecond resolution), and also features low power consumption and low bandwidth.

[0058] The video frame interpolation device 130 is configured to perform frame interpolation operations on image 112. The video frame interpolation device 130 may include or be implemented in any physical or virtual device with computing capabilities, such as a server, mainframe, general-purpose computer, virtual machine, terminal device, cloud computing system, etc. Embodiments of this disclosure are not limited in this respect.

[0059] During video frame interpolation, the video frame interpolation device 130 acquires two images 112 at different times and generates an intermediate image based on the two images 112 for insertion between them. This results in more images after interpolation, allowing for a higher frame rate video clip. For example, the video frame interpolation transpose 130 can output an interpolated video clip including images 112-1, 132-1, 112-2, 132-2, 112-3, ..., 112-N. In this video clip, image 132-1 is predicted and interpolated between images 112-1 and 112-2, and image 132-2 is predicted and interpolated between images 112-2 and 112-3. Although Figure 1 The example only shows predicting and inserting one image between the original two images, but in other examples, more images can be inserted in between.

[0060] Because the DVS device 120 can record intensity changes of a nearly continuous stream and store rich inter-frame information, in this embodiment, dynamic event data 122 acquired by the DVS device 120 is introduced to perform video interpolation. As will be discussed in detail below, the dynamic event data 122 can be used to achieve more accurate motion information estimation, thereby making the video interpolation results more accurate and realistic. Furthermore, the sensing data stored by the DVS device 120 is sparse pulse data, and sensing data is only output when motion is detected; therefore, the amount of dynamic event data that needs to be acquired and stored is relatively small.

[0061] In some scenarios, dynamic event data (122) can be used to assist in video frame interpolation of images acquired by ordinary imaging devices. This allows for the acquisition and storage of small amounts of low-frame-rate video / static image data, as well as sparse dynamic event data within that time period. Video frame interpolation then yields high-definition, high-frame-rate video data, thus achieving video storage and quality optimization. High-definition, high-frame-rate video data can also be applied in image information reconstruction, autonomous driving, and augmented reality (AR) / virtual reality (VR) / mixed reality (MR) imaging.

[0062] In some embodiments, the imaging device 110 and the DVS device 120 may be integrated into, for example, a terminal device, or centrally or separately installed at any data acquisition point for acquiring image data and dynamic event data from the same scene. In some embodiments, the video frame interpolation device 130 may be integrated with the imaging device 110 and the DVS device 120 in the same device, or may be located at a remote device / system. For example, the video frame interpolation device 130 may be included in a terminal device, or may be included in a remote server or cloud computing system. The embodiments disclosed herein are not limited in this respect.

[0063] The following section will discuss in detail the video frame interpolation process based on dynamic event data, with reference to some example implementations.

[0064] Figure 2 A schematic diagram of an example structure of a video frame interpolation apparatus 130 according to some embodiments of the present disclosure is shown. Figure 2 In the example embodiment, a video frame interpolation model 200 is constructed and used to perform video frame interpolation processing. The training process of the video frame interpolation model 200 will be described below with reference to the accompanying drawings. The trained video frame interpolation model 200 can, for example, be developed by... Figure 1 The video frame interpolation device 130 is used to perform video frame interpolation processing.

[0065] like Figure 2As shown, the video frame interpolation model 200 acquires image 201 at time t0 and image 202 at time t1, and also acquires sensor data collected by the DVS device, including dynamic event data 205 between times t0 and t1. It is assumed that time t1 is after time t0. For a pair of images used for frame interpolation, t0 can sometimes be referred to as the start time of the interpolation, and t1 can be referred to as the end time.

[0066] exist Figure 2 In the example embodiment, images 201 and 202 may be images selected from two different time points in a series of images 112 acquired by the imaging device 110. Images 201 and 202 may be adjacent images or images spaced at any time interval. In some embodiments, images 201 and 202 may include video frames at two different times in a video clip. In some embodiments, images 201 and 202 may also include still images acquired at different times.

[0067] exist Figure 2 In the example embodiment, dynamic event data 205 may be part or all of the sensing data acquired by the DVS device 120. Dynamic event data 205 covers at least the time range from t0 to t1. As mentioned above, dynamic event data 205 indicates changes in light intensity in a specific scene acquired within the time range from t0 to t1, which corresponds to the scene captured by images 201 and 202.

[0068] In embodiments of this disclosure, the video interpolation model 200 determines target images 250-1, 250-2 (collectively or individually referred to as target images 250) corresponding to one or more target times between t0 and t1 based on images 201, 202, and dynamic event data 205. The number of target images to be predicted depends on specific needs. For example, to obtain a higher frame rate video, more images may need to be interpolated between t0 and t1. It is understood that although multiple target images 250 are shown, in some examples, the video interpolation model 200 may determine only one target image 250, or determine more target images 250 than shown. When generating multiple target images 250, different target images 250 correspond to different times between t0 and t1, with arbitrary intervals between these times (the interval may, for example, depend on the desired frame rate).

[0069] In some embodiments, such as Figure 2As shown, the video frame interpolation device 130 may further include a video generation module 206, which is configured to organize images 201, 202 and one or more generated target images 250 in chronological order into a target video segment 208. Thus, if images 201, 202 are from source video segments, a higher frame rate video segment can be obtained by inserting the target video image 250. If images 201, 202 are still images acquired by a still imaging device, a dynamic video segment can be obtained by inserting the target video image 250.

[0070] When generating the target image 250, since the dynamic event data 205 is driven by events (e.g., motion events) and indicates changes in light intensity, it can be used to estimate the motion information of objects between any two time points within the time range t0 and t1. In some embodiments, video frame interpolation can be implemented based on optical flow. Optical flow refers to the instantaneous velocity of pixel motion of a spatially moving object on the observation imaging plane. Therefore, optical flow utilizes the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames.

[0071] In some embodiments, dynamic event data 205 can be used to estimate the optical flow between two images, and the image at an intermediate time point can be predicted based on the optical flow. In some embodiments, optical flow estimation can be implemented using machine learning algorithms. Optical flow estimation based on machine learning algorithms involves: first training an optical flow estimation network; using the trained optical flow estimation network to determine the optical flow between images at two time points; and transforming a known image at one time point based on the determined optical flow to obtain an intermediate image corresponding to the other time point. The target image to be inserted is then determined based on the obtained intermediate image.

[0072] Assume that video frame interpolation model 200 determines the target image 250 corresponding to time t between t0 and t1. t can be any interval from t0 or t1, such as τ. If multiple target images 250 are to be determined, the time t at which different target images 250 are located can have different intervals from t0 or t1. When estimating optical flow, the first portion of the sensing data (denoted as E) in the dynamic event data 205 between t0 and t1 is used. t→t0 That is, it includes dynamic event data between t0 and t to determine the optical flow from t to t0 (denoted as F). t→t0 Furthermore, based on the second part of the sensing data in the dynamic event data 205 (represented as E), t→t1 That is, it includes dynamic event data between t and t1 to determine the optical flow from t to t1 (denoted as F). t→t1 These two optical flows Ft→t0 and F t→t1 Image 201 at time t0 and image 202 at time t1 are used to transform image 201 to an intermediate image at time t and image 202 to an intermediate image at time t, respectively.

[0073] In traditional methods, due to the lack of dynamic time data, the optical flow between t0 and t1 needs to be directly calculated, and the optical flow between t and t0 and between t and t1 is determined proportionally based on the assumption of linear motion. However, in the embodiments of this disclosure, the dynamic event data has high resolution and rich motion information between two time points (t0 and t1), and therefore can be used to accurately predict the motion information of moving objects between any two time points within the range of t0 and t1, including complex nonlinear motion. Therefore, in the embodiments of this disclosure, the optical flow from t to t0 and the optical flow from t to t1 can be estimated separately through two optical flow estimations, and then used to perform transformations on the images at time t0 and t1 respectively to achieve image prediction at the intermediate time point t.

[0074] Figure 2 It also specifically illustrates an example structure of the video frame interpolation model 200 in an optical flow-based implementation. For example... Figure 2 As shown, the video frame interpolation model 200 includes an optical flow estimation network 210, a transformation module 220, a frame synthesis network 230, and optionally, a transformation and fusion module 240. In the video frame interpolation model 200, the optical flow estimation network 210 is configured to perform optical flow estimation. The other modules / networks perform image prediction at intermediate time points based on the determined optical flow. The following will refer to... Figure 3 Let's describe in more detail the specific functions of these components in the video frame interpolation model 200. Figure 3 The process for determining a single target image 250 is shown only. If multiple target images 250 between t0 and t1 need to be determined, it can be done in a similar manner.

[0075] exist Figure 3 The lower branch is used to estimate the optical flow from t to t0 and to perform subsequent processing based on the estimated optical flow. The upper branch is used to estimate the optical flow from t to t1 and to perform subsequent processing based on the estimated optical flow.

[0076] like Figure 3 As shown, the optical flow estimation network 210 can be divided into optical flow estimation network 210-1 and optical flow estimation network 210-2. Optical flow estimation network 210-1 can be configured to determine the optical flow 311 (denoted as F) from time t to t0 based on dynamic event data 205-1 between t0 and t. t→t0The optical flow estimation network 210-2 can be configured to be based on the second part of the sensing data in the dynamic event data 205 (denoted as E). t→t1 ), that is, including dynamic event data 205-2 between t and t1 to determine the optical flow 312 (denoted as F) from time t to t1. t→t1 In some embodiments, optical flow estimation networks 210-1 and 210-2 can be configured as machine learning models or neural networks, such as the FlowNet network based on dynamic event data. Optical flow estimation networks 210-1 and 210-2 can learn to estimate optical flow from dynamic event data through a training process. The training process for optical flow estimation networks 210-1 and 210-2 will be performed in conjunction with the video frame interpolation model 200, the training of which will be discussed in detail below.

[0077] Two optical flows 311 and 312 (F) t→t0 and F t→t1 These are used to transform image 201 at time t0 and image 202 at time t1, respectively. Specifically, the conversion module 220 can be configured based on optical flow 311F. t→t0 The image 201 is transformed to the intermediate image 321 at time t, and based on optical flow 312F t→t1 The image 202 is transformed to the intermediate image 322 at time t. As mentioned above, optical flow indicates the instantaneous velocity of pixel motion on the imaging plane. Therefore, if the optical flow at two time points and the image at one of the time points are known, it can be determined which pixels in the image at the other time point correspond to each pixel in the image at that time point. Accordingly, the known image (e.g., image 201 or 202) can be transformed to the corresponding image (e.g., image 321 or 322). In this optical flow-based image transformation process, the assumption of "brightness invariance" can also be used, that is, the brightness of the same target does not change when it moves between different frames. The operation of transforming images 201 or 202 based on optical flow is sometimes called an optical flow-based mapping operation or warp operation.

[0078] Suppose image 201 is represented as I t0 Image 202 is represented as I t1 Based on optical flow 311F t→t0 The intermediate image 321 obtained after transforming image 201 can be represented as g(I) t0 ,F t→t0 Based on optical flow 312F t→t1 The intermediate image 322 obtained after transforming image 202 can be represented as g(I) t1 ,F t→t1 ), where g represents the transformation operation.

[0079] Intermediate images 321 and 322 correspond to images transformed from images 201 and 202 to the target time t, respectively. In some embodiments, the target image 250 at target time t can be determined by merging intermediate images 321 and 322. In some embodiments, the target image 250 can be generated by weighted merging of intermediate images 321 and 322. The weights of intermediate images 321 and 322 can be predetermined or can be assumed to be equal by default.

[0080] In some embodiments, the optical flows 311, 312 can be further accurately determined by the frame synthesis network 230 adjusting and / or using weights when merging intermediate images, and image merging can be performed based on the adjusted optical flows and / or the given weights.

[0081] Specifically, in some embodiments, the frame synthesis network 230 can be configured to adjust the optical flow 311F based on image 201, partial sensing data between t0 and t (i.e., dynamic event data 205-1), and intermediate image 321. t→t0 To determine the adjusted optical flow 331 from t to t0. The frame synthesis network 230 can also be configured to adjust the optical flow 312F based on the image 202, partial sensing data between t and t1 (i.e., dynamic event data 205-2), and the intermediate image 322. t→t1 The optical flow 332 is adjusted to determine the time from t to t1. The adjustment of the optical flow is to make the optical flow used for image prediction more accurate. The frame synthesis network 230 can, based on the original image 201 (and image 202) and the converted intermediate image 321 (and intermediate image 322), use dynamic event data again to determine whether the motion information between the two time points needs to be adjusted to obtain a more accurate optical flow estimation result.

[0082] In some embodiments, the frame synthesis network 230 can determine the optical flow 311F. t→t0 The optical flow adjustment amount (expressed as ΔF) t→t0 ), and for optical flow 312F t→t1 The optical flow adjustment amount (expressed as ΔF) t→t1 The frame synthesis network 230 can perform frame synthesis operations on the network input to obtain the adjustment amount of the optical flow. The frame synthesis network 230 can determine adjustment management based on the optical flow adjustment amount. For example, for optical flow 311F... t→t0 The adjusted optical flow can be determined as F t→t0 +ΔF t→t0 And for optical flow 312F t→t1 The adjusted optical flow can be determined as F t→t1 +ΔF t→t1 .

[0083] In some embodiments, in addition to adjusting optical flow or as an alternative, the frame synthesis network 230 may also be configured to be based on image 201, dynamic event data 205-1, intermediate image 321, and optical flow 311F. t→t0 To determine the fusion weights 341 (represented as V) for image 201. t0 The frame synthesis network 230 can also be configured to be based on image 202, dynamic event data 205-2, intermediate image 322, and optical flow 312F. t→t1 To determine the fusion weights 342 (represented as V) for image 202. t1 These fusion weights are used for subsequent weighted merging of intermediate images obtained using the adjusted optical flow. In some examples, the fusion weights can be represented as a matrix, where each element indicates the weight of the corresponding pixel.

[0084] In some embodiments, the frame synthesis network 230 can be configured as a machine learning model or neural network that learns through a training process to determine the amount of adjustment for the current optical flow based on the input raw image, dynamic event data, optical flow, and intermediate images. The training process of the frame synthesis network 230 is performed together with the video frame interpolation model 200, the training of which will be discussed in detail below.

[0085] In some embodiments where optical flow adjustment is performed by frame synthesis network 230, transform plus fusion module 240 can be configured to perform a transform operation again based on the adjusted optical flow. Transform plus fusion module 240 can transform image 201 into another intermediate image (denoted as g(I)) based on the adjusted optical flow 331 from t to t0. t0 ,F t→t0 +ΔF t→t0 And based on the adjustment of optical flow 332 from t to t1, the image 202 is transformed into another intermediate image (represented as g(I)). t1 ,F t→t1 +ΔF t→t1 The intermediate images here correspond to the images transformed from images 201 and 202 to the target time t using adjusted optical flow, respectively. In some embodiments, the transform-and-fusion module 240 can determine the target image 250 (denoted as ) at target time t by merging these two intermediate images. ).

[0086] In some embodiments where the fusion weights are determined by the frame synthesis network 230, the transform plus fusion module 240 can be configured to perform transformation on the intermediate image g(I) based on the fusion weights. t0 ,F t→t0 +ΔF t→t0 ) and g(I t1 ,F t→t1 +ΔFt→t1 The image is then weighted and merged to obtain the target image 250. Here, the fusion weight V... t0 It can indicate the intermediate image g(I) t0 ,F t→t0 +ΔF t→t0 The importance of the corresponding pixels in the matrix is ​​determined, and the weight V is fused. t1 It can indicate the intermediate image g(I) t1 ,F t→t1 +ΔF t→t1 The fusion weight V represents the importance of the corresponding pixels in the target image. A higher fusion weight for each pixel means it is more likely to be visible in the target image. t0 and V t1 Sometimes it is also called visual weight or visual matrix.

[0087] In some embodiments, the weighting of the intermediate image based on the fusion weights can be represented as follows: Where z represents the normalization term, which equals (t1-t)V t0 +(t-t0)V t1 , ⊙ represents matrix dot product.

[0088] In some embodiments, the fusion weights used for the final intermediate images to be merged can also be determined in other ways, such as by default being equal, being other predetermined values, or being configured in other ways. Embodiments of this disclosure are not limited in this respect.

[0089] Figure 4 A schematic diagram of the information flow during processing in a video frame interpolation model 200 according to some embodiments of the present disclosure is shown. Note that in Figure 4 Example images are provided, but it's understandable that they are just examples.

[0090] like Figure 4 As shown, the input to the entire processing includes image 201 at time t0, image 202 at time t1, and sensor data between t0 and t1, including dynamic event data 205-1 between t and t0 and dynamic event data 205-2 between t and t1. The optical estimation network 210-1 estimates the optical flow 311F from t to t0. t→t0 The optical estimation network 210-2 estimates the optical flow 312F from t to t1. t→t0 Optical Flow 311F t→t0 Used by conversion module 220 to convert image 201I t0 Converted to intermediate image 321g(I t0 ,F t→t0 Optical flow 312F t→t1 Used by conversion module 220 to convert image 202It1 Converted to intermediate image 322g(I) t1 ,F t→t1 ).

[0091] Intermediate image 321g(I) t0 ,F t→t0 ), Optical Flow 311F t→t0 Dynamic event data 205-1E t→t0 And Image 201I t0 It is cascaded and used as input to frame synthesis network 230 to determine the optical flow adjustment amount 431ΔF. t→t0 and / or fusion weight 341V t0 Intermediate image 322g(I) t1 ,F t→t1 ), Optical Flow 312F t→t1 Dynamic event data 205-2E t→t1 And Image 202I t1 It is cascaded and used as input to frame synthesis network 230 to determine the optical flow adjustment amount 432ΔF. t→t1 and / or fusion weight 342V t1 Optical flow adjustment amount: 431ΔF t→t0 Added to optical flow 311F t→t0 The adjusted optical flow from t to t0 is obtained. The optical flow adjustment amount is 432ΔF. t→t1 Added to optical flow 312F t→t1 The adjusted optical flow from t to t1 is obtained.

[0092] The conversion and fusion module 240 can be adjusted based on an optical flow adjustment of 431ΔF. t→t0 And optical flow 311F t→t0 Determine the adjustment of optical flow for image 201I t0 Perform the transformation operation again, and adjust based on the optical flow by 432ΔF. t→t1 And optical flow 312F t→t1 Determine the adjustment of optical flow for image 202I t1 The transformation operation is performed again. The transformation and fusion module 240 can also utilize the fusion weight 341V. t0 And fusion weight 342V t1 The intermediate image g(I) obtained by the second transformation t0 ,F t→t0 +ΔF t→t0 ) and g(I t1 ,F t→t1 +ΔF t→t1 Weighted fusion is performed to obtain the target image 250.

[0093] According to an example embodiment of video frame interpolation disclosed herein, dynamic event data is used to help compensate for the missing motion information in existing image data. By using an optical flow estimation network, more nonlinear motion information can be obtained, thereby ensuring the interpolation of nonlinear motion information in complex scenes and achieving accurate prediction of intermediate images.

[0094] Training of video frame interpolation model

[0095] As mentioned above, the video frame interpolation model 200 needs to be trained to determine appropriate parameter values, especially the optical flow estimation networks 210-1 and 210-2 and the frame synthesis network 230, whose parameter values ​​need to be determined through model parameters. In traditional video frame interpolation methods, to train the model, frames need to be extracted from high frame rate videos, with the remaining frames and extracted frames serving as the model's input and output supervision information, respectively. However, this significantly increases the cost of acquiring training data, limits the size of the dataset, and may lead to a gap between the source training domain and the target domain. Therefore, a self-supervised method is needed. In the embodiments of this disclosure, self-supervised model training can be achieved using dynamic event data instead of direct supervision information about intermediate frames.

[0096] Figure 5 A schematic diagram of an environment 500 for training a video frame interpolation model according to some embodiments of the present disclosure is shown. In environment 500, a model training device 510 is configured to perform training on a video frame interpolation model 200 with initial parameter values. At this stage, the structure of the video frame interpolation model 200 can be determined, but the specific parameter values ​​used for processing are not optimized.

[0097] The model training device 510 is configured to acquire sample images 501 and 502, as well as sample sensing data acquired by a motion sensing device, including sample motion event data 505. Sample images 501 and 502 can be images from two different time points. Similar to images 201 and 202, sample images 501 and 502 can be images selected from a series of images acquired by an imaging device at two different time points.

[0098] Sample images 501 and 502 can be adjacent images or images spaced at any time interval. In some embodiments, sample images 501 and 502 can include video frames at two different times within a video clip. In some embodiments, sample images 501 and 502 can also include still images captured at different times. For ease of discussion later, it is assumed that the sample time of sample image 501 is denoted as t0, and the sample time of sample image 502 is denoted as t1. However, it should be understood that sample images 501 and 502 can be different from images 201 and 202, and therefore, although the corresponding times are also represented by t0 and t1, these times can be different.

[0099] The sample dynamic event data 505 covers at least the time range from t0 to t1. As mentioned above, the dynamic event data 505 indicates the light intensity changes in a specific scene acquired within the time range from t0 to t1, which corresponds to the scene captured by sample images 501 and 502.

[0100] The model training device 510 is configured to train the video interpolation model 200 using sample images 501, 502 and sample dynamic event data 505 to optimize the parameters of the video interpolation model 200. In the model training embodiments of this disclosure, it is assumed that the image sequence consists of reciprocating motion between scenes of two input images, i.e., objects in the scene repeatedly move forward and backward. Based on this, in the absence of a real target image corresponding to the target time t for interpolation, the model training device 510 can perform two or three interpolation processes using the sample dynamic event data 505 and sample images 501, 502, based on the reciprocating motion information indicated by the sample dynamic event data 505. The video interpolation model 200 then determines the predicted image corresponding to the target sample time t, the predicted image corresponding to the start time t0, and / or the predicted image corresponding to the end time t1, respectively. Then, the model training device 510 uses the predicted image corresponding to the start time t0 and the sample image 501 corresponding to the actual start time t, and / or the predicted image corresponding to the end time t1 and the sample image 502 corresponding to the actual end time t1 for supervised training, thereby forming an unsupervised training scheme.

[0101] The model training device 510 includes a self-consistency module 512, which is configured to update the parameter values ​​of the video interpolation model 200 based on the error between the interpolated predicted image and the sample images 501 and / or 502. To update or optimize the parameter values ​​to the desired state, multiple iterative update operations may be required, with each iteration updating the current parameter values ​​of the video interpolation model 200. Although Figure 5It is not shown in detail here, but multiple pairs of sample images and related sample dynamic event data are required during the iterative update process. This is a well-known operation in model training, and will not be elaborated on here.

[0102] To better understand the self-supervised model training method based on dynamic event data proposed in the embodiments of this disclosure, Figure 6 A schematic diagram illustrating an example process for training a video frame interpolation model 200 according to some embodiments of the present disclosure is shown, and Figure 7 A schematic diagram of a three-stage frame interpolation process during training of a video frame interpolation model according to some embodiments of the present disclosure is shown.

[0103] like Figure 6 As shown, in each iteration, the sample image 501 at time t0 and the sample image 502 at time t1, along with the sample dynamic event data 505, are first input into the video interpolation model 200. Based on these inputs, the video interpolation model 200 can generate a predicted image 610 (represented as...) corresponding to the target sample time t. The target sample time t has an interval τ with t0. The internal processing of the input by the video frame interpolation model 200 can be referred to the embodiment described above. During the training phase, the parameter values ​​used by the video frame interpolation model 200 for processing in each iteration can be the parameter values ​​updated in the previous iteration, and can be processed using the initial parameter values ​​initially. For example... Figure 7 As shown, it can be seen from sample image 501I t0 and sample image 502I t1 To determine the predicted image 610 corresponding to the target time t of the intermediate sample.

[0104] The obtained target sample time t corresponding to the predicted image 610, the sample image 501 at t0, and the sample dynamic event data 505 are input into the video frame interpolation model 200 to generate the predicted image 621 corresponding to t1 (represented as...). In this frame interpolation process, the image to be predicted is the image corresponding to the end time t1, and the inputs are the images corresponding to times t0 and t before t1, respectively. Therefore, in the processing of the video frame interpolation model 200, the optical flow estimation networks 210-1 and 210-2 are based on the sample dynamic event data E between t and t1, respectively. t1→t To determine the optical flow from t1 to t, and based on complete sample dynamic event data between t0 and t1 505E t1→t0 To determine the optical flow from t1 to t0. Since the sample dynamic event data 505 can indicate the reciprocating motion information between t0 and t1, it can be used to determine the optical flow at any time point and in any direction between t0 and t1.

[0105] After determining these optical flows, the video interpolation model 200 can perform an interpolation operation based on the optical flow from t1 to t, the optical flow from t1 to t0, the sample image 501, and the predicted image 610 to determine the predicted image 621 for t1. The various components in the video interpolation model 200 can perform the functions described above. Specifically, the conversion module 220 in the video interpolation model 200 can convert the predicted image 610 based on the optical flow from t1 to t0. The sample image 501I is converted into an intermediate image and then processed based on the optical flow from t1 to t0. t0 The image is converted into an intermediate image. The frame synthesis network 230 and the conversion and fusion module 240 in the video frame interpolation model 200 continue to perform optical flow adjustment and fusion weight determination based on the converted intermediate image to obtain a predicted image 621 with a given t1. like Figure 7 As shown, it can be seen from sample image 501I t0 And predicted image 610 To determine the predicted image 621 corresponding to the target time t of the intermediate sample.

[0106] The obtained target sample time t corresponding to the predicted image 610, t1 sample image 502, and sample dynamic event data 505 can be input again into the video frame interpolation model 200 to generate the predicted image 622 corresponding to t0 (represented as...). In this frame interpolation process, the image to be predicted is the image corresponding to the starting time t0, and the inputs are the images corresponding to times t and t1 after t0, respectively. Therefore, in the processing of the video frame interpolation model 200, the optical flow estimation networks 210-1 and 210-2 are based on the sample dynamic event data E between t0 and t, respectively. t0→t To determine the optical flow from t0 to t, and based on complete sample dynamic event data between t0 and t1 505E t0→t1 To determine the optical flow from t0 to t1. Since the sample dynamic event data 505 can indicate the reciprocating motion information between t0 and t1, it can be used to determine the optical flow at any time point and in any direction between t0 and t1.

[0107] After determining these optical flows, the video interpolation model 200 can perform an interpolation operation based on the optical flow from t0 to t, the optical flow from t0 to t1, the sample image 502, and the predicted image 610 to determine the predicted image 622 for t0. The various components in the video interpolation model 200 can perform the functions described above. Specifically, the conversion module 220 in the video interpolation model 200 can convert the predicted image 610 based on the optical flow from t0 to t. The sample image 502I is converted into an intermediate image and then processed based on the optical flow from t0 to t1. t1The image is converted into an intermediate image. The frame synthesis network 230 and the conversion and fusion module 240 in the video frame interpolation model 200 continue to perform optical flow adjustment and fusion weight determination based on the converted intermediate image to obtain a predicted image 622 with a determined t0. like Figure 7 As shown, it can be seen from sample image 502I t1 And predicted image 610 To determine the predicted image 622 corresponding to the target time t of the intermediate sample.

[0108] Sample image 501I t0 and sample image 502I t1 And predicted image 622 And predicted image 621 Provided to the self-consistency module 512, which is configured to determine the sample image 501I t0 With predicted image 622 The error between them, and the sample image 502I t1 With predicted image 622 The error between the values ​​is calculated, and the parameter values ​​of the video interpolation model 200 are updated based on the determined error. In some embodiments, the self-consistency module 512 can construct a loss function based on the aforementioned error. Where ||||1 represents the L1 norm. When updating parameter values, based on the loss function, the self-consistency module 512 can update the parameter values ​​to minimize or reduce the value of the loss function to achieve the convergence objective.

[0109] Various model training algorithms, such as stochastic gradient descent, can be used during parameter updates and model training. The embodiments of this disclosure are not limited in this respect. It should be understood that, although... Figure 6 and Figure 7 The example shows the prediction of two errors, but in some embodiments, only one error may be constructed.

[0110] According to the example embodiment of video frame interpolation disclosed herein, the reciprocating motion information of two frames can be obtained based on event data, and three frame interpolations and two supervisions are performed to complete the cycle-consistent training process. This training method can complete two supervised frame interpolations using only two frames of data, which is less than the traditional method of using three frames of data to complete one supervised frame interpolation. It uses less frame data, performs more supervisions, and has better accuracy. In addition, since the training process lacks real benchmark data for supervision, using the original frame data for self-supervision can better solve the problems in real-world scenarios and improve the predicament of lacking real benchmark data.

[0111] Example process

[0112] Figure 8 A flowchart of a video frame interpolation process 800 according to some embodiments of the present disclosure is shown. Process 800 may be implemented, for example, in... Figure 1 The video frame interpolation device has 130 locations. For ease of description, the following will refer to... Figure 1 To describe process 800.

[0113] In block 810, the video frame interpolation device 130 acquires a first image at a first time, a second image at a second time, and sensing data collected by a dynamic visual sensing device, the sensing data including dynamic event data between the first and second times. In block 820, the video frame interpolation device 130 determines at least one target image based on the first image, the second image, and the sensing data, the at least one target image being an image corresponding to at least one target time between the first and second times.

[0114] In some embodiments, at least one target image includes a first target image corresponding to a first target time between a first time and a second time, and determining at least one target image based on the first image, the second image, and sensing data includes: determining a first optical flow from the first target time to the first time based on a first portion of sensing data in the sensing data, wherein the first portion of sensing data includes dynamic event data between the first time and the first target time; determining a second optical flow from the first target time to the second time based on a second portion of sensing data in the sensing data, wherein the second portion of sensing data includes dynamic event data between the first target time and the second time; and performing frame interpolation on the first image and the second image based on the first optical flow and the second optical flow to obtain the first target image corresponding to the first target time.

[0115] In some embodiments, performing frame interpolation on a first image and a second image based on a first optical flow and a second optical flow includes: converting the first image into a first intermediate image based on the first optical flow; converting the second image into a second intermediate image based on the second optical flow; and merging the first intermediate image and the second intermediate image to obtain a first target image.

[0116] In some embodiments, merging a first intermediate image and a second intermediate image to obtain a first target image includes: adjusting a first optical flow based on a first image, a first portion of sensing data, and the first intermediate image to obtain a first adjusted optical flow; converting the first image into a third intermediate image based on the first adjusted optical flow; and merging the third intermediate image and the second intermediate image to obtain the first target image.

[0117] In some embodiments, merging a third intermediate image and a second intermediate image to obtain a first target image includes: adjusting a second optical flow based on a second image, second portion of sensing data, and a second intermediate image to obtain a second adjusted optical flow; converting the second image into a fourth intermediate image based on the second adjusted optical flow; and merging the third intermediate image and the fourth intermediate image to obtain the first target image.

[0118] In some embodiments, merging a third intermediate image and a fourth intermediate image to obtain a first target image includes: determining a first fusion weight for the third intermediate image and a second fusion weight for the fourth intermediate image, wherein the first fusion weight indicates the importance of a corresponding pixel in the third intermediate image and the second fusion weight indicates the importance of a corresponding pixel in the fourth intermediate image; and performing a weighted merging of the third intermediate image and the fourth intermediate image based on the first fusion weight and the second fusion weight to obtain the first target image.

[0119] In some embodiments, determining the first fusion weight and the second fusion weight includes: determining the first fusion weight based on the first image, the first portion of sensing data, the first optical flow, and the first intermediate image; and determining the second fusion weight based on the second image, the second portion of sensing data, the second optical flow, and the second intermediate image.

[0120] In some embodiments, the method further includes: organizing the first image, the second image, and at least one target image into a target video segment in chronological order.

[0121] In some embodiments, the first image and the second image include a first video frame at a first time and a second video frame at a second time in a video segment. In some embodiments, the first image and the second image include still images acquired by a still imaging device.

[0122] In some embodiments, determining at least one target image based on a first image, a second image, and sensor data includes: applying the first image, the second image, and sensor data to a trained video interpolation model to obtain at least one target image output by the video interpolation model.

[0123] Figure 9 A flowchart of a process 900 for training a video frame interpolation model according to some embodiments of the present disclosure is shown. Process 900 may be implemented, for example, in... Figure 5 The model training device is located at 510. For ease of description, the following will refer to... Figure 5 To describe process 900.

[0124] In block 910, the model training device 510 acquires a first sample image at a first sample time, a second sample image at a second sample time, and sample sensing data, the sample sensing data including dynamic event data between the first and second sample times. In block 920, the model training device 510 applies the first sample image, the second sample image, and the sample sensing data to a video interpolation model to obtain a first predicted image corresponding to a target sample time, the target sample time being between the first and second sample times. In block 930, the model training device 510 uses the video interpolation model to generate at least one of a second predicted image corresponding to the first sample time and a third predicted image corresponding to the second sample time, based on the first sample image, the second sample image, the first predicted image, and the sample sensing data. In block 940, the model training device 510 updates the parameter values ​​of the video interpolation model based on at least one of the following errors: a first error between the generated second predicted image and the first sample image, and a second error between the generated third predicted image and the second sample image.

[0125] In some embodiments, generating at least one of the second and third predicted images includes at least one of the following: applying a second sample image, a first predicted image, and sample sensing data to a video interpolation model to obtain a second predicted image; and applying a first sample image, a first predicted image, and sample sensing data to a video interpolation model to obtain a third predicted image.

[0126] In some embodiments, applying a second sample image, a first predicted image, and sample sensing data to a video interpolation model to obtain a second predicted image includes: using the video interpolation model to perform the following operations: determining a first sample optical flow from a first sample time to a target sample time based on a first portion of sample sensing data in the sample sensing data, wherein the first portion of sample sensing data includes dynamic event data between the first sample time and the target sample time; determining a second sample optical flow from the first sample time to a second sample time based on the sample sensing data; and performing an interpolation operation on the first predicted image and the second sample image based on the first sample optical flow and the second sample optical flow to obtain a second predicted image corresponding to the first sample time.

[0127] In some embodiments, applying a first sample image, a first predicted image, and sample sensing data to a video interpolation model to obtain a third predicted image includes: using the video interpolation model to perform the following operations: determining a third sample optical flow from a second sample time to a target sample time based on a second portion of sample sensing data in the sample sensing data, wherein the second portion of sample sensing data includes dynamic event data between the target sample time and the second sample time; determining a fourth sample optical flow from the second sample time to the first sample time based on the sample sensing data; and performing an interpolation operation on the first predicted image and the first sample image based on the third sample optical flow and the fourth sample optical flow to obtain a third predicted image corresponding to the second sample time.

[0128] Example devices and equipment

[0129] Figure 10 A schematic block diagram of an apparatus 1000 for video frame interpolation according to some embodiments of the present disclosure is shown. The apparatus 1000 may be implemented as or included in... Figure 1 The video frame interpolation device 130.

[0130] The device 1000 may include multiple modules for performing tasks such as Figure 8 The corresponding steps in process 800 discussed herein. For example... Figure 10 As shown, the device 1000 includes an acquisition unit 1010 configured to acquire a first image at a first time, a second image at a second time, and sensing data collected by a dynamic visual sensing device, the sensing data including dynamic event data between the first time and the second time; and a frame interpolation unit 1020 configured to determine at least one target image based on the first image, the second image, and the sensing data, the at least one target image being an image corresponding to at least one target time between the first time and the second time.

[0131] In some embodiments, at least one target image includes a first target image corresponding to a first target time between a first time and a second time. The frame interpolation unit 1020 includes: a first optical flow determination unit configured to determine a first optical flow from the first target time to the first time based on a first portion of sensing data in sensing data, wherein the first portion of sensing data includes dynamic event data between the first time and the first target time; a second optical flow determination unit configured to determine a second optical flow from the first target time to the second time based on a second portion of sensing data in sensing data, wherein the second portion of sensing data includes dynamic event data between the first target time and the second time; and an optical flow frame interpolation unit configured to perform frame interpolation operations on the first image and the second image based on the first optical flow and the second optical flow to obtain the first target image corresponding to the first target time.

[0132] In some embodiments, the optical flow interpolation unit includes: a first conversion unit configured to convert a first image into a first intermediate image based on a first optical flow; a second conversion unit configured to convert a second image into a second intermediate image based on a second optical flow; and an image merging unit configured to merge the first intermediate image and the second intermediate image to obtain a first target image.

[0133] In some embodiments, the image merging unit includes: a first adjustment unit configured to adjust a first optical flow based on a first image, a first portion of sensing data, and a first intermediate image to obtain a first adjusted optical flow; a first adjustment conversion unit configured to convert the first image into a third intermediate image based on the first adjusted optical flow; and a first adjustment merging unit configured to merge the third intermediate image and a second intermediate image to obtain a first target image.

[0134] In some embodiments, the first adjustment merging unit includes: a second adjustment unit configured to adjust a second optical flow based on a second image, second partial sensing data, and a second intermediate image to obtain a second adjusted optical flow; a second adjustment conversion unit configured to convert the second image into a fourth intermediate image based on the second adjusted optical flow; and a second adjustment merging unit configured to merge a third intermediate image and a fourth intermediate image to obtain a first target image.

[0135] In some embodiments, the second adjustment merging unit includes: a weight determination unit configured to determine a first fusion weight for a third intermediate image and a second fusion weight for a fourth intermediate image, the first fusion weight indicating the importance of corresponding pixels in the third intermediate image and the second fusion weight indicating the importance of corresponding pixels in the fourth intermediate image; and a weighted merging unit configured to perform weighted merging of the third intermediate image and the fourth intermediate image based on the first fusion weight and the second fusion weight to obtain a first target image.

[0136] In some embodiments, the weight determination unit includes: a first weight determination unit configured to determine a first fusion weight based on a first image, a first portion of sensing data, a first optical flow, and a first intermediate image; and a second weight determination unit configured to determine a second fusion weight based on a second image, a second portion of sensing data, a second optical flow, and a second intermediate image.

[0137] In some embodiments, such as Figure 10 As shown, the apparatus 1000 may further include a video generation unit 1030 configured to organize a first image, a second image, and at least one target image into a target video segment in chronological order.

[0138] In some embodiments, the first image and the second image include a first video frame at a first time and a second video frame at a second time in a video segment. In some embodiments, the first image and the second image include still images acquired by a still imaging device.

[0139] In some embodiments, the interpolation unit 1020 includes a model-based determination unit configured to apply a first image, a second image, and sensor data to a trained video interpolation model to obtain at least one target image output by the video interpolation model.

[0140] Figure 11 A schematic block diagram of an apparatus 1100 for video frame interpolation according to some embodiments of the present disclosure is shown. The apparatus 1100 may be implemented as or included in... Figure 5 The model training device is 500.

[0141] Device 1100 may include multiple modules for performing, such as Figure 8 The corresponding steps in process 800 discussed herein. For example... Figure 11 As shown, the device 1100 includes a sample acquisition unit 1110 configured to acquire a first sample image at a first sample time, a second sample image at a second sample time, and sample sensing data, wherein the sample sensing data includes dynamic event data between the first and second sample times. The device 1100 also includes a first frame interpolation unit 1120 configured to apply the first sample image, the second sample image, and the sample sensing data to a video frame interpolation model to obtain a first predicted image corresponding to a target sample time, the target sample time being between the first and second sample times; and a second frame interpolation unit 1130 configured to use the video frame interpolation model to generate at least one of a second predicted image corresponding to the first sample time and a third predicted image corresponding to the second sample time, based on the first sample image, the second sample image, the first predicted image, and the sample sensing data. Furthermore, the device 1100 includes a parameter update unit 1140 configured to update the parameter values ​​of the video frame interpolation model based on at least one of the following errors: a first error between the generated second predicted image and the first sample image, and a second error between the generated third predicted image and the second sample image.

[0142] In some embodiments, the second frame interpolation unit 1130 includes at least one of the following: a second prediction generation unit configured to apply a second sample image, a first prediction image, and sample sensing data to a video frame interpolation model to obtain a second prediction image; and a third prediction generation unit configured to apply a first sample image, a first prediction image, and sample sensing data to a video frame interpolation model to obtain a third prediction image.

[0143] In some embodiments, the second prediction generation unit is configured to: perform the following operations using a video frame interpolation model: determine a first sample optical flow from a first sample time to a target sample time based on a first portion of sample sensing data in the sample sensing data, wherein the first portion of sample sensing data includes dynamic event data between the first sample time and the target sample time; determine a second sample optical flow from the first sample time to a second sample time based on the sample sensing data; and perform frame interpolation on a first predicted image and a second sample image based on the first sample optical flow and the second sample optical flow to obtain a second predicted image corresponding to the first sample time.

[0144] In some embodiments, the third prediction generation unit is configured to: perform the following operations using a video frame interpolation model: determine a third sample optical flow from a second sample time to a target sample time based on a second portion of sample sensing data in the sample sensing data, wherein the second portion of sample sensing data includes dynamic event data between the target sample time and the second sample time; determine a fourth sample optical flow from the second sample time to a first sample time based on the sample sensing data; and perform frame interpolation on a first predicted image and a first sample image based on the third sample optical flow and the fourth sample optical flow to obtain a third predicted image corresponding to the second sample time.

[0145] Figure 12 A schematic block diagram of an example device 1200 that can be used to implement embodiments of the present disclosure is shown. Device 1200 may be implemented as or included in... Figure 1 The video frame interpolation device 130 can be implemented as or included in Figure 5 The model training device is 500.

[0146] As shown in the figure, device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to computer program instructions stored in random access memory (RAM) and / or read-only memory (ROM) 1202 or computer program instructions loaded into RAM and / or ROM 1202 from storage unit 1207. Various programs and data required for the operation of device 1200 can also be stored in RAM and / or ROM 1202. The computing unit 1201 and RAM and / or ROM 1202 are interconnected via bus 1203. Input / output (I / O) interface 1204 is also connected to bus 1203.

[0147] Multiple components in device 1200 are connected to I / O interface 1204, including: input unit 1205, such as keyboard, mouse, etc.; output unit 1206, such as various types of monitors, speakers, etc.; storage unit 1207, such as disk, optical disk, etc.; and communication unit 1208, such as network card, modem, wireless transceiver, etc. Communication unit 1208 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0148] The computing unit 1201 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as process 800 and / or process 900. For example, in some embodiments, process 800 and / or process 900 may be implemented as computer software programs tangibly contained in a computer-readable medium, such as storage unit 1207. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1200 via RAM and / or ROM and / or communication unit 1208. When the computer program is loaded into RAM and / or ROM and executed by the computing unit 1201, one or more steps of process 800 and / or process 900 described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute process 800 and / or process 900 by any other suitable means (e.g., by means of firmware).

[0149] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of this disclosure, a machine-readable medium or computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0151] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0152] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for training a video frame interpolation model, comprising: Acquire a first sample image at a first sample time, a second sample image at a second sample time, and sample sensing data, wherein the sample sensing data includes dynamic event data between the first sample time and the second sample time; The first sample image, the second sample image, and the sample sensing data are applied to the video frame interpolation model to obtain the first predicted image corresponding to the target sample time, wherein the target sample time is between the first sample time and the second sample time. Using the video frame interpolation model, at least one of the second predicted image corresponding to the first sample time and the third predicted image corresponding to the second sample time is generated based on the first sample image, the second sample image, the first predicted image and the sample sensing data. as well as The parameter values ​​of the video interpolation model are updated based on at least one of the following errors: a first error between the generated second predicted image and the first sample image, and a second error between the generated third predicted image and the second sample image. Generating at least one of the second predicted image and the third predicted image includes at least one of the following: The second sample image, the first predicted image, and the sample sensing data are applied to the video interpolation model to obtain the second predicted image. as well as The first sample image, the first predicted image, and the sample sensing data are applied to the video interpolation model to obtain the third predicted image.

2. The method according to claim 1, characterized in that, Applying the second sample image, the first predicted image, and the sample sensor data to the video interpolation model to obtain the second predicted image includes: performing the following operations using the video interpolation model. A first sample optical flow from the first sample time to the target sample time is determined based on a first portion of the sample sensing data, wherein the first portion of the sample sensing data includes dynamic event data between the first sample time and the target sample time; Based on the sample sensing data, determine the second sample optical flow from the first sample time to the second sample time; and Frame interpolation is performed on the first predicted image and the second sample image based on the first sample optical flow and the second sample optical flow to obtain the second predicted image corresponding to the first sample time.

3. The method according to claim 1, characterized in that, Applying the first sample image, the first predicted image, and the sample sensor data to the video interpolation model to obtain the third predicted image includes: performing the following operations using the video interpolation model. A third sample optical flow from the second sample time to the target sample time is determined based on the second portion of the sample sensing data, wherein the second portion of the sample sensing data includes dynamic event data between the target sample time and the second sample time; Based on the sample sensing data, determine the fourth sample optical flow from the second sample time to the first sample time; and Frame interpolation is performed on the first predicted image and the first sample image based on the third sample optical flow and the fourth sample optical flow to obtain the third predicted image corresponding to the second sample time.

4. A video frame interpolation method, comprising: Acquire a first image at a first time, a second image at a second time, and sensing data collected by a dynamic visual sensing device, wherein the sensing data includes dynamic event data between the first time and the second time. as well as The first image, the second image, and the sensor data are applied to a trained video interpolation model to obtain at least one target image, wherein the at least one target image is an image corresponding to at least one target time between the first time and the second time, and wherein the video interpolation model is obtained by the method according to any one of claims 1 to 3.

5. The method according to claim 4, characterized in that, The at least one target image includes a first target image corresponding to a first target time between the first time and the second time, and obtaining at least one target image includes performing the following operations using the video frame interpolation model. A first optical flow from the first target time to the first time is determined based on a first portion of the sensing data, wherein the first portion of the sensing data includes dynamic event data between the first time and the first target time; A second optical flow from the first target time to the second time is determined based on a second portion of the sensing data, wherein the second portion of the sensing data includes dynamic event data between the first target time and the second time; and Frame interpolation is performed on the first image and the second image based on the first optical flow and the second optical flow to obtain the first target image corresponding to the first target time.

6. The method according to claim 5, characterized in that, Performing frame interpolation operations on the first image and the second image based on the first optical flow and the second optical flow includes: performing the following operations using the video frame interpolation model. Based on the first optical flow, the first image is converted into a first intermediate image; Based on the second optical flow, the second image is converted into a second intermediate image; and The first intermediate image and the second intermediate image are merged to obtain the first target image.

7. The method according to claim 6, characterized in that, Merging the first intermediate image and the second intermediate image to obtain the first target image includes: The first optical flow is adjusted based on the first image, the first portion of sensing data, and the first intermediate image to obtain the first adjusted optical flow; Based on the first adjustment of optical flow, the first image is converted into a third intermediate image; and The third intermediate image and the second intermediate image are merged to obtain the first target image.

8. The method according to claim 7, characterized in that, Merging the third intermediate image and the second intermediate image to obtain the first target image includes performing the following operations using the video frame interpolation model. Based on the second image, the second portion of sensing data, and the second intermediate image, the second optical flow is adjusted to obtain a second adjusted optical flow; Based on the second adjustment of optical flow, the second image is converted into a fourth intermediate image; and The third intermediate image and the fourth intermediate image are merged to obtain the first target image.

9. The method according to claim 8, characterized in that, Merging the third intermediate image and the fourth intermediate image to obtain the first target image includes performing the following operations using the video frame interpolation model. Determine a first fusion weight for the third intermediate image and a second fusion weight for the fourth intermediate image, wherein the first fusion weight indicates the importance of corresponding pixels in the third intermediate image, and the second fusion weight indicates the importance of corresponding pixels in the fourth intermediate image; and The third intermediate image and the fourth intermediate image are weighted and merged based on the first fusion weight and the second fusion weight to obtain the first target image.

10. The method according to claim 9, characterized in that, Determining the first fusion weight and the second fusion weight includes performing the following operations using the video frame interpolation model: Based on the first image, the first portion of sensor data, the first optical flow, and the first intermediate image, the first fusion weight is determined; and The second fusion weight is determined based on the second image, the second portion of the sensing data, the second optical flow, and the second intermediate image.

11. The method according to any one of claims 4 to 10, characterized in that, Also includes: The first image, the second image, and the at least one target image are organized into a target video segment in chronological order.

12. The method according to any one of claims 4 to 10, characterized in that, The first image and the second image include a first video frame at the first time and a second video frame at the second time in the video clip, or The first image and the second image include static images acquired by a static imaging device.

13. An apparatus for training a video frame interpolation model, comprising: The sample acquisition unit is configured to acquire a first sample image at a first sample time, a second sample image at a second sample time, and sample sensing data, wherein the sample sensing data includes dynamic event data between the first sample time and the second sample time. The first frame interpolation unit is configured to apply the first sample image, the second sample image and the sample sensing data to the video frame interpolation model to obtain a first predicted image corresponding to a target sample time, wherein the target sample time is between the first sample time and the second sample time. The second frame interpolation unit is configured to use the video frame interpolation model to generate at least one of a second predicted image corresponding to the first sample time and a third predicted image corresponding to the second sample time, based on the first sample image, the second sample image, the first predicted image, and the sample sensing data, wherein generating at least one of the second predicted image and the third predicted image includes at least one of the following: The second sample image, the first predicted image, and the sample sensing data are applied to the video interpolation model to obtain the second predicted image; and The first sample image, the first predicted image, and the sample sensing data are applied to the video interpolation model to obtain the third predicted image; and The parameter update unit is configured to update the parameter values ​​of the video interpolation model based on at least one of the following errors: a first error between the generated second predicted image and the first sample image, and a second error between the generated third predicted image and the second sample image.

14. An electronic device, characterized in that, include: At least one computing unit; At least one memory coupled to the at least one computing unit and storing instructions for execution by the at least one computing unit, the instructions, when executed by the at least one computing unit, causing the device to perform the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 12.

16. A computer program product, characterized in that, The computer program product includes computer-executable instructions that, when executed by a processor, cause the computer to perform the method according to any one of claims 1 to 12.

Images