Method and System for Deep Learning-Based Interpolation of Video Frames

Abstract

A system and method for deep learning-based video frame interpolation are disclosed. In some embodiments, a method for interpolating video frames may include receiving a series of input video frames, the series of input video frames comprising a first higher resolution frame, a second higher resolution frame, and a lower resolution intermediate frame having an intermediate temporal position in the series between the first higher resolution frame and the second higher resolution frame. The lower resolution intermediate frame may be upscaled to create an upscaled intermediate frame, and the first higher resolution frame, the second higher resolution frame, and the upscaled intermediate frame may be concatenated to generate a concatenated frame. The concatenated frame may be input to a trained video frame interpolation (VFI) model, and the interpolated frame may be output from the trained VFI model, the interpolated frame having an intermediate temporal position in the series.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 495,066, filed on April 7, 2023, titled "METHODS AND SYSTEMS FOR DEEP - LEARNING BASED INTERPOLATION OF VIDEO FRAMES", the entire content of which is incorporated herein by reference for all purposes.

[0002] The present disclosure relates to video frame interpolation, and more particularly, to deep - learning - based interpolation.

Background Art

[0003] The use of digital video has become widespread and is still expanding in many industries and areas of endeavor. Applications such as computer vision, surveillance, and ultra - low - latency real - time video conferencing are likely to continue driving the demand for high - quality digital video processing. New technologies such as never - before - seen high - resolution videos (e.g., 8K videos), virtual reality, and augmented reality are also imposing requirements on the processing of video content. Such uses can benefit from more efficient technologies for capturing, compressing, transmitting, and / or storing digital video.

[0004] Discrete cosine transform (DCT)-based compression techniques, such as codecs, are a common method for compressing digital image data for both still and video images. For video images, compression standards have evolved primarily through increasing search options, including finer-grained motion estimation, supporting larger block size selections for estimation, and extending the existing data that can be referenced (e.g., intra-frame and multi-frame references). To achieve higher compression performance, subsequent versions require greater computational power than before. While current manual video codecs are well-designed, as the resolution and quality of digital video continue to improve, conventional video codecs are approaching the limits of what can be optimized through compression.

[0005] In DCT-based compression techniques, all data is explicitly considered through a combination of direct coding, motion estimation reference, and residual coding. In comparison, deep learning-based compression techniques aim to have one or more neural networks predict or automatically generate at least some image data within a frame, adding residual coding as needed. In other words, the goal is to skip as much explicit coding of data as possible.

[0006] As general-purpose graphics processing units (GPUs) become more readily available, deep learning (e.g., machine learning) video compression techniques are emerging as a potentially viable alternative to DCT-based video compression techniques.

[0007] However, deep learning-based video frame interpolation (VFI) techniques, which can use one or more convolutional neural networks (CNNs) to design autoencoders or one or more recurrent neural networks (RNNs) to build progressive image compression schemes, may exhibit quality issues when attempting to interpolate frames in situations of large motion between frames, nonlinear motion, and / or severe occlusion. [Overview of the project]

[0008] The inventors recognized the limitations, weaknesses, and various problems in various methods for applying deep learning techniques to video compression and established a method to mitigate these problems. This method leverages the advantages of both conventional video codecs and newer deep learning-based VFI techniques while minimizing their drawbacks. In this method, low-resolution frames ("hint frames") provide hints to the deep learning framework to improve the quality of interpolated frames. Conventional video codecs (such as H.264 / AVC or H.265 / HEVC) are used to compress two (or more) full-resolution (e.g., keyframes) and hint frames, which together are fed into a deep learning-based VFI architecture. The information in the hint frames acts as a guide for the VFI process, resulting in a significant improvement in the ability of the deep learning-based VFI architecture to correctly interpolate frames in situations of large motion between frames, nonlinear motion, and / or severe occlusion. In some embodiments, the full-resolution frames may be I-frames. In some embodiments, the full-resolution frames may be P-frames and / or B-frames.

[0009] Importantly, even when there is a relatively large temporal distance between two full-resolution frames, and / or between either full-resolution frame and a hint frame, the quality of the interpolated frames is significantly improved, even in situations of large motion between frames, nonlinear motion, and / or severe occlusion. Therefore, instead of providing a hierarchical architecture (e.g., Chao-Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl, 2018, Video Compression through Image Interpolation, CoRR abs / 1804.06919 (2018), arXiv:1804.06919 http: / / arxiv.org / abs / 1804.06919, hereafter "Wu") and explicitly using motion or residual information, the technique described herein uses low-resolution hint frames to provide additional "true" content to guide the VFI.

[0010] As described herein, this technique can significantly improve interpolation results, thereby mitigating problems caused by deformations including occlusion, large motion and / or illumination, and nonlinear motion. Therefore, this technique can significantly improve adaptive streaming of video data and provide higher compression efficiency.

[0011] Various systems and methods for deep learning-based interpolation of video frames are discussed and described herein. In some embodiments, the above-mentioned problems can be addressed by the way in which a series of input video frames are received. The series of frames may include a first frame, a second frame, and a third frame which may be temporally positioned at a predetermined temporal position between the first and second frames. The first frame may have a first resolution, the second frame may have a second resolution which in various embodiments may be the same as the first resolution, and the third frame may have a third resolution which may be lower than the first resolution (and / or second resolution). The first frame, the second frame, and the third frame may be provided to a trained VFI model which may have a deep learning architecture. Using the trained VFI model, an interpolated frame may be generated at a predetermined temporal position between the first and second frames, the interpolated frame having a first resolution. In this way, by providing hint frames with a lower resolution than the resolution of the first and / or second frames, the quality of frames interpolated from the first and second frames by the trained VFI model can be significantly improved.

[0012] In some embodiments, the problems described above can be addressed by a method for interpolating specific video frames of a series. The series may include a first frame having a first temporal position in the series, a second frame having a second temporal position in the series, and an intermediate frame having an intermediate temporal position in the series between the first and second temporal positions. The first and second video frames may have relatively higher resolutions, while the intermediate frame may have a relatively lower resolution (e.g., lower than the resolution of the first and second video frames). For interpolation, the lower-resolution intermediate frame may be upscaled to create an upscaled intermediate frame, and the first higher-resolution frame, the second higher-resolution frame, and the upscaled intermediate frame may be concatenated to generate a concatenated frame. The concatenated frame may be provided as input to a trained VFI model, which may provide an interpolated frame having an intermediate temporal position in the series and a higher resolution as output. In this way, by providing intermediate frames of relatively lower resolution at any temporal position between the first and second video frames, the quality of the interpolated frames provided as output by the trained VFI model can be significantly improved. The VFI framework described and depicted herein may incorporate any of the various deep learning-based video frame interpolation architectures.

[0013] In some embodiments, the above problem can be addressed by a system for interpolating video frames containing a trained VFI model. A first full-resolution frame, a second full-resolution frame, and a lower-resolution intermediate frame may be received, the lower-resolution intermediate frame being at a temporal position between the first full-resolution frame and the second full-resolution frame. The intermediate frame may be upscaled to create an upscaled intermediate frame having the same resolution as the first and second full-resolution frames. The first full-resolution frame, the second full-resolution frame, and the upscaled intermediate frame may be concatenated into a concatenated frame. The concatenated frame may be provided to a trained VFI model, which may output an interpolated frame at the temporal position of the intermediate frame, the interpolated frame having the same resolution as the first and second full-resolution frames. In various embodiments, the first full-resolution frame and / or the second full-resolution frame may be I-frames. In this way, by upscaling lower-resolution intermediate frames, concatenating the first full-resolution frame, the upscaled intermediate frame, and the second full-resolution frame, and providing the concatenated frame to the trained VFI model, the quality of the interpolated frames output from the trained VFI model can be significantly improved. In various embodiments, the first high-resolution frame and / or the second high-resolution frame may be I-frames.

[0014] It should be understood that the above brief description is provided to introduce a selection of concepts that will be further explained in the detailed description. It is not intended to identify any important or essential features of the claimed subject matter, and its scope will be uniquely defined by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to any implementation that solves any defect described above or in any part of this disclosure.

[0015] This patent or application file includes at least one drawing made in color. A copy of this patent or patent application publication accompanied by the color drawing(s) will be provided by the Patent Office upon request and payment of the necessary fees.

[0016] This disclosure may be better understood by reading the following description of non-limiting embodiments with reference to the accompanying drawings. [Brief explanation of the drawing]

[0017] [Figure 1A] One or more embodiments of this disclosure show a triplet of three digital video frames from a sequence of frames related to deep learning-based interpolation of video frames. [Figure 1B] This shows a portion of an intermediate frame related to the intermediate digital video frame in Figure 1A, according to one or more embodiments of the present disclosure. [Figure 2] This disclosure provides a top-level view of an architecture for deep learning-based interpolation of video frames, according to one or more embodiments of this disclosure. [Figure 3] The lower-level diagrams of architectures for deep learning-based interpolation of the video frames in Figure 2, according to one or more embodiments of the present disclosure, are shown. [Figure 4] This disclosure describes one or more embodiments of a method for deep learning-based interpolation of video frames. [Figure 5] This disclosure describes one or more embodiments of a method for deep learning-based interpolation of video frames. [Figure 6] This disclosure describes one or more embodiments of a system for deep learning-based interpolation of video frames. [Figure 7] This disclosure describes one or more embodiments of a system for deep learning-based interpolation of video frames. [Figure 8A] This disclosure describes the performance of one or more embodiments of a method and system for deep learning-based interpolation of video frames, as discussed and shown herein. [Figure 8B] Describe the performance of methods and systems for deep learning-based interpolation of video frames as discussed and shown herein, according to one or more embodiments of the present disclosure. [Figure 8C] Describe the performance of methods and systems for deep learning-based interpolation of video frames as discussed and shown herein, according to one or more embodiments of the present disclosure.

DETAILED DESCRIPTION

[0018] Deep learning-based video frame interpolation (VFI) techniques can be used to generate intermediate frames between two key frames without explicitly encoded data, except for the ending frame. Although such techniques have made great progress, the inventors have recognized that there are still many scenarios that are extremely difficult for deep learning-based VFI techniques to handle. For example, some VFI methods focus on interpolating pixel values of key frames and do not consider other information such as depth or texture. This approach tends to make the generated frames look artificial and produce inaccurate results. Some VFI methods rely heavily on flow and depth information to create accurate and realistic interpolated frames, which may not be very practical because the computational cost may be high and it may be suitable for all types of video content or devices.

[0019] Various deep learning alternatives to DCT-based coding design an autoencoder using one or more convolutional neural networks (CNNs) or construct a progressive image compression scheme using one or more recurrent neural networks (RNNs). However, deep learning-based compression methods may not have built-in error tolerance mechanisms for fault tolerance that may exist in conventional coders (such as headers and / or resynchronization markers). As a result, errors and / or impairments in the data can more likely lead to loss or errors in the information in the output.

[0020] Some techniques for applying deep learning techniques to video compression have focused on end-to-end encoders that explore the capabilities of deep neural networks to achieve high compression ratios. In some techniques, for example, as described in Wu, frame interpolation can be used to generate intermediate frames. The main limitation of such interpolation-based techniques is that the quality of the frames generated by simple interpolation degrades as the temporal distance between video key frames increases, particularly in the case of video sequences that reflect non-linear movements such as rotation or viewpoint changes. Some techniques may use ground truth motion and residual coding to clarify the interpolation.

[0021] Other techniques use interpolation of intermediate frames and attempt to identify and transmit macroblocks that do not match within a certain peak signal-to-noise ratio (PSNR) threshold for each macroblock. The drawback of such techniques can be combining both DCT coding and frame interpolation for each macroblock, effectively mixing two styles of frames. As a result, the interpolation and explicit corrections must be very well aligned, or the deep learning framework is left to resolve the inconsistencies.

[0022] Therefore, VFI technology has been proposed as a way to skip the encoding of various frames and instead generate those frames from surrounding frames, but current deep learning-based methods do not completely replace standards and / or conventional video coders such as the H.26x series of compression standards (e.g., H.264 Advanced Video Coding (AVC) standard, H.265 High Efficiency Video Coding (HEVC) standard, etc.). Furthermore, currently there is no widely accepted standard for deep learning-based video compression, which can make it difficult to guarantee interoperability and even to compare different methods.

[0023] This specification discloses various systems and methods for deep learning-based interpolation of video frames. A group of images from a digital video sequence may include a first high-resolution frame or keyframe, a second high-resolution frame or keyframe, and an intermediate frame (which may be from a temporal position between the first and second high-resolution frames), as depicted in Figures 1A and 1B. Lower-resolution versions of the intermediate frame may be created, and in some embodiments, one or more of the first high-resolution frame, the second high-resolution frame, and the intermediate frame may be compressed (e.g., by a conventional compression codec). Following potential compression, the first high-resolution frame, the lower-resolution intermediate frame, and the second I-frame may be transmitted and / or stored (e.g., by a first device) and subsequently received and / or accessed (e.g., by a second device). In various embodiments, in the methods and systems described and illustrated herein, the high-resolution frame may be a full-resolution frame such as an I-frame (which may improve the quality of interpolation), but may also be other full-resolution frames (such as B-frames and / or P-frames). In various embodiments, a first high-resolution frame, a second high-resolution frame, and an intermediate frame may be part of a sequence of high-resolution frames, and the intermediate frame may be one of several lower-resolution intermediate frames from a temporal position between the first high-resolution frame and the second high-resolution frame. In some embodiments, all high-resolution frames may be I-frames, or frames generated according to a video compression standard, and some parts of the frame may be partially derived, for example, using a recursive application of a video codec and / or VIF technology.

[0024] As depicted in Figures 2 and 3, and as addressed by the methods in Figures 4 and 5, low-resolution intermediate frames may be upscaled and / or upsampled, and within the VFI framework disclosed herein, a first high-resolution frame, an upscaled intermediate frame, and a second high-resolution frame may be concatenated (e.g., into a concatenated frame) and presented to a trained VFI model. The trained VFI model may then interpolate (e.g., predict) the intermediate frames using the first high-resolution frame, the second high-resolution frame, and the upscaled intermediate frame (including, e.g., "hints" regarding the movement and occlusion of various elements in the image). Figures 6 and 7 depict several systems that can perform these methods. Finally, Figures 8A–8C depict results based on the systems and methods disclosed herein.

[0025] Figure 1A shows a triplet of three digital video frames from a sequence of frames related to deep learning-based interpolation of video frames. The sequence of digital video frames 100 may include a first frame 110, a second frame 120, and an intermediate frame 130 that may occupy a temporal position between the first frame 110 and the second frame 120. In various embodiments, the first frame 110 and the second frame 120 may also be “keyframes” for use in digital image compression and / or digital video compression, and may then be associated with high-resolution frames (which may be I-frames, which can improve the quality of the interpolation) of the compressed sequence of digital video frames. The first frame 110, the second frame 120, and the intermediate frame 130 may be used to create a digital video frame that can be used by a trained VFI to generate an interpolation frame that can replace the intermediate frame 130. Therefore, a relatively high-quality representation of the sequence of frames associated with the first frame 110, the second frame 120, and the intermediate frame 130 can be transmitted by transmitting only the encoded versions of lower-resolution versions of the first frame 110, the second frame 120, and the intermediate frame 130.

[0026] An image triplet for use in the systems and methods disclosed herein may include a first frame 110, a second frame 120, and an intermediate frame 130. In the case of this image triplet, there are regions in each of these frames that can present challenges to other deep learning-based VFI techniques. With respect to the first region 112 of the first frame 110, the second region 122 of the second frame 120, and the intermediate region 132 of the intermediate frame 130, the subject hand in the image is largely hidden from view (e.g., obscured) in the first frame 110, but is no longer hidden from view by the time of the second frame 120. As described below, other deep learning-based VFI techniques cannot correctly generate an interpolated version of the intermediate region 132 of the intermediate frame 130, and therefore cannot generate a high-quality interpolated version of the intermediate frame 130 as a whole.

[0027] Figure 1B is a comparison of different video frame interpolations of intermediate frames related to the intermediate digital video frame in Figure 1A. As depicted in Figure 1B, region 152 (labeled "ground truth") shows the contents of intermediate region 132 in their true state. For comparison, region 154 shows the contents of intermediate region 132 after encoding (here by the H.264 codec, but other codecs may be used in various embodiments). As can be seen from the figure, the encoded contents of intermediate region 132 represented by region 154 closely match the ground truth of intermediate region 132 represented by region 152. That is, the hands of the subject in the first frame 110 and the second frame 120 are clearly represented in region 154 after encoding.

[0028] Regions 162, 164, 166, and 168 each represent the content of intermediate region 132 as interpolated (e.g., predicted) by various VFI techniques. Region 162 represents the content of intermediate region 132 as interpolated by the “Channel Attention for Frame Interpolation” (CAIN) VFI technique (Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee, 2020, Channel Attention Is All You Need for Video Frame Interpolation, Proceedings of the AAAI Conference on Artificial Intelligence 34, 07 (Apr. 2020), 10663-10671, https: / / doi.org / 10.1609 / aaai.v34i07.6693, which is incorporated herein by reference in its entirety). Region 164 shows the contents of intermediate region 132 interpolated by the “Asymmetric Bilateral Motion Estimation” (ABME) VFI technology (described in J. Park, C. Lee, and C. Kim, 2021, Asymmetric Bilateral Motion Estimation for Video Frame Interpolation, In International Conference on Computer Vision).Region 166 shows the contents of intermediate region 132 interpolated by “Frame Interpolation with Large Motion” (FILM) VFI technology (Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee, 2020, Channel Attention Is All You Need for Video Frame Interpolation, Proceedings of the AAAI Conference on Artificial Intelligence 34, 07 (Apr. 2020), 10663-10671, https: / / doi.org / 10.1609 / aaai.v34i07.6693). Region 168 shows the contents of intermediate region 132 interpolated by “eXtreme Video Frame Interpolation” (XVFI) VFI technology (H.Sim, J.Oh, and M.Kim. 2021, XVFI: eXtreme Video Frame Interpolation, In Proceedings of the IEEE International Conference on Computer Vision (ICCV)).

[0029] As can be seen from the figure, the interpolated content of the intermediate regions represented by region 162 (for CAIN technology), region 164 (for ABME technology), region 166 (for FILM technology), and region 168 (for XVFI technology) cannot accurately capture the hand movements of the underlying sequence of the video frames that are at least partially captured by intermediate region 132 because the target hands of the first frame 110 and the second frame 120 are almost completely absent in regions 162, 164, 166, and 168.

[0030] In comparison, region 170 shows the content of intermediate region 132 as interpolated by the VFI method and / or system disclosed herein. The interpolated content of intermediate region 132 represented by region 154 does not match as closely as region 170 to the ground truth of intermediate region 132 represented by region 152. However, as can be seen from the figure, the hands of the subject in the first frame 110 and the second frame 120 are captured and represented in region 170. Thus, the interpolated content of intermediate region 132 represented by region 170 is of significantly higher quality than the interpolated content of intermediate region 132 represented by regions 162, 164, 166, and 168.

[0031] Because the hand is hidden from view in the first frame 110, CAIN, ABME, FILM, and XVFI techniques cannot correctly estimate the hand's movement when interpolating the intermediate frame 130. Large inter-frame movements, nonlinear movements, and / or severe occlusion situations can occur in complex and varied ways, making it extremely difficult to make any interpolation-based technique work well across different types of image sequences. Other techniques may use context, depth, flow, and / or edges to address these issues, but such techniques lead to more processing steps and, consequently, more computation.

[0032] Therefore, there are some sequences that can be extremely difficult for deep learning-based methods to interpolate efficiently. The methods and systems disclosed herein can advantageously use a lower-resolution version of the intermediate frame to assist guide deep learning-based VFI, thereby taking advantage of a reduction in the number of bits for transmission while improving the quality of the resulting interpolated frame. Furthermore, the methods and systems disclosed herein can advantageously maintain high visual quality while further reducing the number of bits for transmission by using a standards-based (e.g., DCT-based) coding algorithm.

[0033] According to the methods and systems disclosed herein, a lower-resolution version of the intermediate frame 130 may be generated (or the resolution of the intermediate frame 130 may be optionally reduced), and a sequence of video frames including a first frame 110, a second frame 120, and the lower-resolution version of the intermediate frame 130 may be compressed (in accordance with an H.26x compression standard, such as the H.264 / AVC or H.265 / HEVC standard). The compressed sequence of video frames may be transmitted and / or stored (for example, by a first device) and then received and / or accessed (for example, by a second device).

[0034] As will be further explained below, upon reception (and / or access), an upscaled version of intermediate frame 130 may be generated (or a lower-resolution version of intermediate frame 130 may be otherwise upscaled) according to the VFI framework disclosed herein, and the first frame 110, the second frame 120, and the upscaled version of intermediate frame 130 may be concatenated and provided to a trained VFI model. The trained VFI model may then output an interpolated version of intermediate frame 130.

[0035] Figure 2 shows a top-level diagram of an architecture for deep learning-based interpolation of video frames. The deep learning-based VFI architecture 200 may include an input stage 210, an upscale stage 220, a VFI framework having a concatenation stage 230 and a VFI model stage 240, and an output stage 250.

[0036] In the input stage 210, the VFI architecture 200 may receive and / or access a first frame 212, a second frame 214, and a third frame 216 of a sequence of video frames. The first frame 212 may be the first high-resolution frame or first keyframe of the sequence, the second frame 214 may be the second I-frame or second keyframe of the sequence, and the third frame 216 may be an intermediate frame of the sequence (where there may be multiple intermediate frames, but only one intermediate frame is depicted), and may be located at a temporal position between the temporal position of the first frame 212 and the temporal position of the second frame 214. In various embodiments, the first high-resolution frame and / or the second high-resolution frame may be I-frames. Thus, the first frame 212 may be substantially the same as the first frame 110, the second frame 214 may be substantially the same as the second frame 120, and the third frame 216 may be substantially the same as the intermediate frame 130 (as described herein). In addition, as described, the third frame 216 has a lower resolution than the first frame 212 and the second frame 214. Therefore, in various embodiments, the first frame 212 and the second frame 214 may be full-resolution frames (e.g., high-resolution frames and / or keyframes), and the third frame 216 may be a significantly lower-resolution frame (e.g., a frame with a resolution significantly lower than full resolution).

[0037] In some embodiments, a third frame 216 (e.g., received and / or accessed by the input stage 210) may be a downsampled frame. Downsampling of a frame may use bicubic sampling and / or otherwise, a cubic function may be used to estimate the value between two points. In various embodiments, the third frame 216 may be downsampled using bicubic downsampling that uses a 4x4 block weighted average to determine the pixels. The bicubic algorithm may be able to more advantageously preserve detail than other downsampling methods (e.g., hand-drawn downsampling methods).

[0038] The advantage of the hint frames disclosed herein is that they may explicitly provide a low-resolution approximation of the actual frame, which may have the ability to provide a trained VFI model on the decoder side with hints about a wide variety of motion and / or context information, rather than having the VFI model guess it. Thus, hint frames may provide additional guidance to the trained VFI model regarding large nonlinear motions and rotations that would otherwise be difficult to process. This is advantageous because it may help the interpolation model, for example, to locate pixels in intermediate frames associated with the motion of the depicted object.

[0039] In various embodiments, the first frame 212, the second frame 214, and / or the third frame 216 (e.g., after downsampling) may be compressed when received and / or accessed by the input stage 210. In some embodiments, the first frame 212, the second frame 214, and the third frame 216 may be compressed according to a compression standard from the H.26x series of compression standards (e.g., the H.264 / AVC standard or the H.265 / HEVC standard).

[0040] As I-frames in the H.26x video coding standard, the first frame 212 and the second frame 214 may be of a type that provides references for intermediate frames in order to minimize and / or prevent error propagation in the context of the coding standard. The deep learning-based VFI methods and systems discussed herein may favorably redirect this property of I-frames in the context of the H26x coding standard in order to help interpolate a third frame 216 and / or other frames between the first frame 212 and the second frame 214.

[0041] In various embodiments, the coding command used to encode the first frame 212, the second frame 214, and / or the third frame 216 may use quantization parameters (QPs) instead of constant rate coefficients (CRFs). Because QPs use a fixed set of quantization levels and a limited number of bits to represent video, the use of QPs may, advantageously, result in lower file sizes and / or faster coding times. The use of QPs may also result in less complex algorithms for modeling the probability of a particular image configuration, which may, advantageously, be less computationally expensive. The use of QPs may also avoid adaptive changes to coding quantization parameters on the fly, which may, advantageously, simplify the consistent measurement of quality metrics such as PSNR. In some embodiments, the coding command may use QP parameters 5, 10, 14, 18, 23, 27, 31, and / or 40. In various embodiments, the coding command may use any QP parameter (lower QP parameters correspond to higher quality coding and more bits to represent it, supporting higher quality interpolation, while higher QP parameters correspond to lower quality coding and fewer bits to represent it, but supporting lower quality interpolation).

[0042] In the upscaling stage 220, following the reception and / or access of the first frame 212, second frame 214, and third frame 216 in the input stage 210, the third frame 216 may be upscaled to the same resolution as the first frame 212 and / or second frame 214. In some embodiments, upsampling may use pixel duplication, which can advantageously avoid reliance on the upsampling interpolation method. In some embodiments, upscaling may use parameterless bicubic resampling to resample the third frame 216 to a desired resolution (e.g., to the resolution of the first frame 212 and / or second frame 214), which can advantageously allow and / or facilitate the transmission of fewer parameters.

[0043] Within the VFI framework, the first frame 212, the upscaled third frame 216, and the second frame 214 may be concatenated in the concatenation stage 230. The concatenated frame (e.g., the concatenation of the first frame 212, the upscaled third frame 216, and the second frame 214) may be provided to the VFI model stage 240 of the VFI framework.

[0044] The VFI model in VFI model stage 240 may generate an interpolated frame 256 at the same resolution as the first frame 212 and the second frame 214 (e.g., a full-resolution frame). Thus, the interpolated frame 256 may represent the prediction made by the frame architecture 200 used when generating the third frame 216 of the input stage 210. The output stage 250 may include the first frame 212, the interpolated frame 256, and the second frame 214.

[0045] Therefore, in various embodiments, the first frame 212 and the second frame 214 may be provided in the input stage 210 for each of the sets of hint frames provided in place of the third frame 216. For each of the hint frames, the VFI architecture 200 may generate a corresponding interpolation frame in place of the interpolation frame 256.

[0046] Other VFI architectures proposed for video compression use separate VFI models, each trained for different temporal distances between the frame being interpolated and the two frames used for interpolation. In contrast, VFI architecture 200 is fixed, and the VFI model in VFI model stage 240 can be trained to interpolate frames regardless of the temporal distance between the frame being interpolated and the keyframe used for interpolation. Thus, when the VFI model in VFI model stage 240 is presented with a concatenated frame (described further below) incorporating a keyframe and intermediate frames at one of several different temporal distances from the keyframe, the VFI model can interpolate (e.g., predict) the intermediate frames based on hint frames at the same several different temporal distances from the corresponding keyframe.

[0047] Therefore, since the trained VFI model in VFI model stage 240 can interpolate based on hint frames regardless of the temporal distance associated with the hint frames, the VFI architecture 200 can favorably handle the storage and transmission of fewer parameters during the process of supporting the associated compression and decompression. Furthermore, the trained VFI model can directly interpolate frames at various temporal distances from the corresponding keyframes, thus favorably avoiding recursive interpolation of frames (which may propagate errors between frames).

[0048] Figure 3 shows a lower-level diagram of the architecture for deep learning-based interpolation of the video frames of Figure 2. The VFI architecture portion 300 may receive a first frame 322, a second frame 324, and an upscaled third frame 326 (followed by an upscale stage which may be substantially similar to, for example, an upscale stage 220). The VFI architecture portion 300 may be similar to the portion of the VFI architecture by CAIN VFI technology.

[0049] In the VFI architecture portion 300, the first frame 322, the second frame 324, and the upscaled third frame 326 may be down-shuffled, for example, according to the “PixelShuffle” technique (described in Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang, 2016, Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network, CoRR abs / 1609.05158(2016), arXiv:1609.05158) for periodic shuffling and / or rearrangement of tensor elements. A down-shuffled version of the first frame 322, a down-shuffled upscaled version of the third frame 326, and a down-shuffled version of the second frame 324 may be concatenated by means of channel-by-channel concatenation, etc. The concatenated frames may be provided to a trained VFI model 340 (which may be substantially the same as the trained VFI model of VFI model stage 240), to which various residual blocks with channel attention may be applied (e.g., according to CAIN VFI technique). Hint frames may be provided in favor of channel attention to interpolate high-quality intermediate frames such as interpolated frames 356 (e.g., by handling situations of large motion, nonlinear motion, and / or large occlusion between frames) without obtaining explicit motion information (e.g., by explicitly estimating the flow).

[0050] The first difference between the VFI architecture portion 300 and CAIN VFI technology is that, instead of simply receiving the first frame 322 and the second frame 324, the VFI architecture portion 300 also receives an upscaled third frame 326, for example, the hint frame described herein. The second difference is that, following downshuffing, the VFI architecture portion performs channel-level concatenation on the downshuffed version of the first frame 322, the downshuffed version of the upscaled third frame 326, and the downshuffed version of the second frame 324, rather than simply performing channel-level concatenation on the downshuffed version of the first frame 322 and the downshuffed version of the second frame 324. As a result of these differences, the VFI architecture portion 300 can advantageously generate interpolated frames of significantly higher quality than other VFI technologies, including CAIN VFI technology, as described herein.

[0051] Therefore, in various embodiments, a system for interpolating video frames may comprise a trained VFI model (such as the trained VFI model of model stage 240 and / or trained VFI model 340), one or more processors, and non-temporary memory having executable instructions that, when executed, can cause one or more processors to perform various operations as described below. It may receive a first high-resolution frame (such as the first frame 322), a second high-resolution frame (such as the second frame 324), and an intermediate frame of lower resolution at a temporal position between the first and second high-resolution frames. The intermediate frame may be upscaled to create an upscaled intermediate frame (such as the third frame 326) having the same resolution as the first and second high-resolution frames. The first high-resolution frame, the second high-resolution frame, and the upscaled intermediate frame may be concatenated into a concatenated frame, which may be provided to the trained VFI model. The trained VFI model may output interpolated frames (e.g., interpolated frame 356) at a temporal position, the interpolated frames having the same resolution as the first and second high-resolution frames. In various embodiments, the first and / or second high-resolution frames may be keyframes and / or full-resolution frames (e.g., I-frames).

[0052] In some embodiments, the first high-resolution frame, the second high-resolution frame, and the intermediate frame may be compressed in accordance with an H.26x video compression standard such as the H.264 / AVC or H.265 / HEVC standard. In various embodiments, the first high-resolution frame and / or the second high-resolution frame may be fully encoded frames and / or compressed (e.g., in accordance with an H.26x video compression standard). In some embodiments, the upscaling of the intermediate hint frame may be via pixel duplication. In some embodiments, the concatenation of the first high-resolution frame, the second high-resolution frame, and the upscaled intermediate frame may be performed on a channel-by-channel basis. In some embodiments, the first high-resolution frame, the second high-resolution frame, and the upscaled intermediate frame may be down-shuffled before concatenation.

[0053] Figures 4 and 5 illustrate a method for deep learning-based interpolation of video frames. As depicted in Figure 4, method 400 may include the steps of receiving input (410), providing an image to a model (420), and generating interpolated frames (430). In various embodiments, method 400 may also include the steps of down-shuffling frames (412), concatenating frames (414), and / or upscaling a third frame (416). In various embodiments, method 400 may use some or all of these steps in any order.

[0054] In 410, a series of input video frames may be received. The series of input video frames may include a first frame, a second frame, and a third frame at a predetermined temporal position between the first and second frames. The first and second frames may have a first resolution, and the third frame may have a second resolution lower than the first resolution. In 420, the first frame, the second frame, and the third frame may be provided to a trained VFI model. In 430, an interpolated frame may be generated at a predetermined temporal position between the first and second frames using the trained VFI model, the interpolated frame having a first resolution.

[0055] In some embodiments, the first frame, second frame, and third frame may be compressed. In some embodiments, the first frame, second frame, and / or third frame may be compressed in accordance with the H.26x video compression standard. The second resolution may be smaller than the first resolution, for example, to trade off bitrate and quality. In some embodiments, the second resolution may be less than or equal to 1 / 4 of the size of the first resolution along both the X and Y axes. In other embodiments, the second resolution may be 1 / 4 to 1 / 2 the size of the first resolution along both the X and Y axes. In various embodiments, the second resolution may be 1 / 4 to 1 / 6 the size of the first resolution along both the X and Y axes.

[0056] In some embodiments, in 416, the third frame may be upscaled to the first resolution before being provided to the trained VFI model. In some embodiments, the hint frame may be upscaled via pixel duplication. In some embodiments, in 414, the first, second, and third frames may be concatenated channel by channel before being provided to the trained VFI model. In some embodiments, in down-shuffling 412, the first, second, and third frames may be down-shuffled channel by channel before being concatenated channel by channel.

[0057] In some embodiments, an indicator for disabling the use of a low-resolution third frame may be received along with the first frame, the second frame, and / or the third frame. The indicator may have a first predetermined value representing the disabling of the use of the third frame and a second predetermined value representing the enabling of the use of the third frame. The provision of the first frame, the second frame, and the third frame to the trained VFI model may be based on the value of the indicator matching the second predetermined value, or alternatively, the first frame and the second frame may simply be provided to the trained VFI model if the value of the indicator matches the first predetermined value.

[0058] Method 500 may include receiving an input (510), upscaling a frame (520), concatenating frames (530), inputting a frame (540), and outputting an interpolated frame (550). In various embodiments, Method 500 may also include down-shuffling a frame 522. In various embodiments, Method 500 may use some or all of these steps in any order.

[0059] In 510, a series of input video frames may be received (for example, by a device and / or system implementing deep learning VFI techniques as disclosed herein), the series of input video frames including a first higher resolution frame having a first temporal position in the series, a second higher resolution frame having a second temporal position in the series, and a lower resolution intermediate frame having an intermediate temporal position in the series between the first and second temporal positions. In 520, the lower resolution intermediate frame may be upscaled to create an upscaled intermediate frame. In 530, the first higher resolution frame, the second higher resolution frame, and the upscaled intermediate frame may be concatenated to produce a concatenated frame. In input 540, the concatenated frame may be input to a trained VFI model. In 550, an interpolated frame may be output from the trained VFI model, the interpolated frame having an intermediate temporal position in the series.

[0060] The second resolution may be smaller than the first resolution, for example, to trade off bitrate and quality. In some embodiments, the second resolution may be less than or equal to 1 / 4 the size of the first resolution along both the X and Y axes. In other embodiments, the second resolution may be 1 / 4 to 1 / 2 the size of the first resolution along both the X and Y axes. In some embodiments, the resolution of the first higher resolution frame, the resolution of the second higher resolution frame, and the resolution of the lower resolution intermediate frame are the same. In other embodiments, one or more of the frames have a different resolution from the other frames. In some embodiments, upscaling of the lower resolution intermediate frame may be done via pixel duplication. In some embodiments, the concatenation of the first higher resolution frame, the second higher resolution frame, and the upscaled intermediate frame may be done on a channel-by-channel basis. In some embodiments, in downshruffing 522, the first higher resolution frame, the second higher resolution frame, and the upscaled intermediate frame may be downshruffed after the upscaling of the lower resolution intermediate frame.

[0061] This method may be configured for the operation of the system disclosed herein. Therefore, the same advantages applicable to the system may also apply to the method.

[0062] The descriptions of embodiments have been presented for illustrative and explanatory purposes. Preferred modifications and variations to the embodiments may be carried out in accordance with the above description or obtained by practicing the methods. For example, unless otherwise stated, one or more of the methods described may be carried out by preferred devices and / or combinations of devices. The methods may be carried out by using one or more logic devices (e.g., processors) in combination with one or more additional hardware elements such as storage devices, memory, image sensor / lens systems, optical sensors, hardware network interfaces / antennas, switches, actuators, and clock circuits to execute stored instructions. The described methods and associated operations may also be carried out in parallel and / or simultaneously in various orders in addition to the order described in this application.

[0063] Figures 6 and 7 illustrate a system for deep learning-based interpolation of video frames. Figure 6 shows a system 600 which may include a case 610, a power supply 620, an interconnect board 630, one or more processors 640, one or more memory devices 650, one or more input / output (I / O) interfaces 660, and / or one or more media drives 670.

[0064] The case 610 may be any type of enclosure for a part of the system 600. In some embodiments, the case 610 may have a configuration suitable for use as a case for a personal computer (e.g., a desktop computer). In other embodiments, the case 610 may have a configuration suitable for insertion into a rack of components (e.g., server blades).

[0065] The various processors included in Processor 640 may have one or more central processing units (CPUs) and / or microprocessors. Each CPU and / or microprocessor may have any number of processing cores, and each core may be capable of operating to process one or more threads at a time. In various embodiments, the CPU and / or microprocessor may include any of the following: a digital signal processor (DSP), a graphics processing unit (GPU), a network processor, and / or other dedicated processors, coprocessors, or units.

[0066] The memory device 650 may include a device based on any of various storage technologies. In various embodiments, the memory device 650 may include magnetic disk-based storage and / or optical disk-based storage. The memory device 650 may also include various types of random access memory (RAM)-based storage, such as dynamic RAM (DRAM)-based storage. In some embodiments, the memory device 650 may include non-volatile memory, such as flash memory or phase-change memory (PCM). The media drive 670 may also include a device based on, for example, magnetic disk-based storage technology and / or optical disk-based storage technology. When executed, the memory device 650 may store executable instructions that cause the processor 640 to perform various operations as disclosed herein (for example, with respect to Method 400, Method 500, the VFI architecture of Figures 2 and 3, and / or any of the various methods and systems disclosed herein).

[0067] The I / O interface 660 may include electronic interfaces that can operate to communicately couple to I / O devices using electronic equipment developed for various I / O technologies (e.g., bus specifications, interconnect specifications, etc.). The I / O interface may include appropriate interconnect configurations, as well as interfaces that comply with, for example, the Universal Serial Bus (USB) protocol, the Serial Advanced Technology Attachment (SATA) bus protocol, the Peripheral Component Interconnect (PCI) Express protocol, etc. Some I / O interfaces 660 may connect components within the system 600 to each other, and other I / O interfaces 660 may connect components within the system 600 to other devices outside the system 600. The I / O interface 660 may also include one or more interfaces for one or more wired internet connections (e.g., Ethernet connections) and / or wireless connections (e.g., Wi-Fi and / or cellular connections).

[0068] The power supply 620 may be capable of supplying power to the processor 640, memory device 650, I / O interface 660, and / or media drive 670. Meanwhile, the interconnection board 630 may electrically and / or electronically couple the power supply 620, processor 640, memory device 650, I / O interface 660, and / or media drive 670. The interconnection board 630 may include, for example, a motherboard or other printed circuit boards (PCBs) used to power and / or interconnect various electronic devices.

[0069] System 600 (and / or other systems and devices disclosed herein) may be configured according to the systems discussed herein. For example, System 600 may implement a VFI architecture substantially similar to the VFI architecture of Figure 2 or Figure 3, and / or implement a method substantially similar to Method 400 and / or Method 500. Thus, the same advantages described herein that apply to the VFI architectures of Figures 2 and 3, and / or Method 400 and / or Method 500, may apply to System 600.

[0070] Figure 7 shows a system 700 which may comprise a case 710, a power supply 720, one or more processors 740, one or more memory devices 750, one or more antennas 760, and / or a display screen 780.

[0071] The case 710 may be any type of enclosure for a part of the system 700. In some embodiments, the case 710 may have a form suitable for use as a case for a smart device such as a tablet or smartphone.

[0072] The various processors included in Processor 740 may have one or more CPUs and / or microprocessors. Each CPU and / or microprocessor may have any number of processing cores, and each core may be capable of operating to process one or more threads at a time. In various embodiments, the CPU and / or microprocessor may include any of the following: DSP, GPU, network processor, and / or other dedicated processors, coprocessors, or units.

[0073] The memory device 750 may include a device based on any of the various storage technologies. In various embodiments, the memory device 750 may include flash memory or non-volatile memory such as PCM. When executed, the memory device 750 may store executable instructions that cause the processor 740 to perform various operations as disclosed herein. When executed, the memory device 750 may store executable instructions that cause the processor 740 to perform various operations as disclosed herein (for example, with respect to Method 400, Method 500, the VFI architecture of Figures 2 and 3, and / or any of the various methods and systems disclosed herein).

[0074] The I / O interface 760 may include an electronic interface that can operate to communicately couple to an I / O device using electronic equipment developed for various I / O technologies (e.g., bus specifications, interconnect specifications, etc.). The I / O interface may include an appropriate interconnect configuration, as well as an interface compliant with, for example, the USB protocol. The I / O interface 760 may also include one or more interfaces for wireless connectivity (e.g., Wi-Fi and / or cellular connectivity). The power supply 720 may be operable to supply power to the processor 740, the memory device 750, the I / O interface 760, and the end / or media drive 770.

[0075] System 700 (and / or other systems and devices disclosed herein) may be configured according to the systems discussed herein. For example, System 700 may implement a VFI architecture substantially similar to the VFI architecture of Figure 2 or Figure 3, and / or implement a method substantially similar to Method 400 and / or Method 500. Thus, the same advantages discussed herein also apply to the VFI architecture of the FGS. Figures 2 and 3, and / or Method 400 and / or Method 500 may be applied to System 700.

[0076] Figures 8A–8C illustrate the performance of a method and system for deep learning-based interpolation of video frames, as described and depicted herein. The dataset 800 includes ground truth 810, coding 820 (e.g., H.26x coding), CAIN VFI 830, first VFI 842, second VFI 844, and third VFI 846. With respect to dataset 800, CAIN VFI 830, first VFI 842, second VFI 844, and third VFI 846 were trained using a training set of over 50,000 video frame triplets (e.g., three frames of digital video) with a resolution of 448 × 256. Following training, various VFI models were tested against various video frame triplets from test sets associated with the training set.

[0077] The group of pictures (GOP) size was set to 16. With a GOP size of 16, the first frame of one GOP, and the first frame of the next GOP, acted as two high-resolution frames (or keyframes, such as I-frames). Therefore, adjacent GOPs share the same boundary high-resolution frame (or keyframe). Each GOP was processed independently. Different GOP sizes (e.g., 2 to 12 GOP sizes) were tested, and the quality of the reconstructed frame results (e.g., interpolated frames) was fairly consistent. A GOP size of 12 was chosen to reduce the number of bits used.

[0078] Regarding frame interpolation, Table 1 presents data on PSNR and Structural Similarity Index (SSIM) for CAIN VFI 830 ("CAIN (Baseline)"), the first VFI 842 ("+1 / 4 Color"), the second VFI 844 ("+1 / 4 Grayscale"), and the third VFI 846 ("+1 / 6 Color"). [Table 1] In Table 1, "+1 / 4" means that the hint frame is 1 / 4 the size of the full-resolution frame in each direction (resulting in a pixel area that is 1 / 16 the size of the full-resolution pixel area). Similarly, "+1 / 6" means that the hint frame is 1 / 6 the size of the full-resolution frame in each direction (resulting in a pixel area that is 1 / 36 the size of the full-resolution pixel area).

[0079] As illustrated in Figure 8A, the use of additional hint frames can significantly improve interpolation results. The advantage of hint frames is their ability to help a trained VFI model better position objects within the frame. Filling in details is up to the interpolation model. The VFI techniques disclosed herein can maintain correct coloring of hands (although they have some difficulty interpolating fine details). Given lower resolution hint frames, video interpolation methods can be expected to benefit from improvements to assist in detail reconstruction. Even using smaller color hint frames (1 / 6 of the full-resolution dimensions in both width and height), the reconstructed frame looks perceptually much better than a frame reconstructed using larger grayscale hint frames (e.g., 1 / 4 of the full-resolution dimensions in both width and height).

[0080] Regarding video compression, Tables 2A and 2B show quantitative data on test results for the Vimeo90K test and four SNU-FILM tests (easy, moderate, difficult, and extreme). Some data pertains to the CAIN VFI, XVFI VFI, and ABME VFI technologies, as mentioned above. Some additional data pertains to the “Depth-Aware video frame Interpolation” (DAIN) VFI technique (described in Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang, 2019, Depth-Aware Video Frame Interpolation, In IEEE Conference on Computer Vision and Pattern Recognition). Other additional data pertains to the “Intermediate Feature Refine Network” (IFRNet) VFI method (Lingtong Kong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying Tai, Chengjie Wang, and Jie Yang, 2022, IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation, In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR)).

[0081] The CAIN, DAN, XVFI, ABME, and IFRNet VFI techniques use raw frames as input, and the VFI techniques disclosed herein use H.264 compression (with QP set to 23) to obtain the decompressed input frames. [Table 2A] [Table 2B]

[0082] For the SNUFILM dataset, SNU-FILM (Easy) is the easiest for midpoint frame interpolation. As shown, all VFI techniques achieved good performance in generating high-quality target frames. However, in the case of the VFI techniques disclosed herein, the use of resampled hint frames made it difficult for the model of the present invention to fully capture fine details, resulting in average performance. For the Vimeo90K test dataset, the VFI techniques disclosed herein performed slightly better than the others. For SNUFILM (Medium), where the temporal distance between the keyframe and target frame is 2, motion within the triplet is larger. The VFI techniques disclosed herein were implemented equally well with the others, with IFRNet yielding the best performance and ABME coming in second. However, for SNU-FILM (Hard) and SNU-FILM (Extreme), the large temporal distance between the keyframe and target frame (e.g., 4 and 8, respectively) occludes more area around the edges of motion, making it difficult to capture useful information from the keyframe and generate the correct target frame using other methods. The VFI technique disclosed herein can interpolate high-quality frames even when using H.264-processed keyframes as input, using guidance from low-resolution hint frames. This performance is relatively stable across the Vimeo90K and SNU-FILM (easy and moderate) datasets when motion is small to moderate.

[0083] As depicted in Figures 8B and 8C, seven videos with a 1920x1080 pixel resolution option from the Open Ultra Video Group (UVG) dataset were used to evaluate the performance of the VFI technology disclosed herein. These included, for example, the "Beauty" video 852, the "Bosphorus" video 854, the "ReadySetGo" video 856, the "ShakeNDry" video 858, the "Honeybee" video 862, the "Jockey" video 864, and the "YachtRide" video 866. Each of these videos had over 300 frames and was captured at 50 / 120fps (the first 99 frames from each video were used for testing). A video sequence from the Xiph.org Video Test Media dataset 1, for example, the "Tango" video 868, was also used. This video is a sequence with a lot of motion captured at 60fps and has 294 frames (the first 99 frames from this video were also used for testing). The final evaluation dataset will have motion categories of varying degrees, such as local motion, global motion, and congested motion.

[0084] The video was encoded with a QP size of 23 and a GOP size of 16, with each GOP being processed individually (a GOP is a group of pictures, all of which have B-frames, with only the first frame being an I-frame, and two consecutive GOPs sharing one I-frame between them). This setting was chosen because it is consistent with the nature of the VFI technology disclosed herein, allowing intermediate frames to simply use the surrounding I-frames directly for reference.

[0085] VFI technology is evaluated by the method of this invention based on the compression ratio, measured by bits per pixel (BPP), and the quality of compression as PSNR. BPP is an objective measure indicating the amount of data required to represent an image or video. Lower BPP values ​​indicate better compression efficiency, while higher PSNR values ​​indicate higher visual quality of the resulting frames. Both high-resolution keyframes and low-resolution hint frames were counted for BPP calculation because hint frames were used to guide the VFI process.

[0086] Figures 8B and 8C show the video compression performance for all eight video sequences tested, with BPP on the x-axis and PSNR on the y-axis. The tests focused on the 0.0–0.4 BPP range in our evaluation, and the bitstreams of the VFI technologies disclosed herein consist simply of decompressed keyframes and low-quality hint frames. For video compression performance, solid lines represent video compressed according to the H.264 standard, dotted lines represent the performance of the VFI technologies disclosed herein (dash-dot lines represent the performance of the hint frames themselves).

[0087] For example, in the case of video sequences with a relatively large amount of fine detail, such as Bosphorus video 854, ReadySetGo video 856, and YachtRide video 866, the VFI technology disclosed herein performs better than H.264 only in the low BPP range. This is because the VFI technology disclosed herein uses extra hint frames for guidance and may not be configured to capture fine details as well as H.264. However, in the case of video sequences with more significant motion and / or less detail, such as Beauty video 852, Honeybee video 862, and ShakeNdry video 858, the VFI technology model disclosed herein easily outperforms mere H.264 performance. This is because the VFI technology disclosed herein can utilize extra hint frames to provide more accurate predictions of video frames, resulting in better compression performance.

[0088] Overall, the VFI technology disclosed herein exhibits performance comparable to H.264 for many sequences and tends to perform better for very low bitrate applications (e.g., less than 0.05 BPP). Therefore, the VFI technology disclosed herein demonstrates promising results for low bitrate applications and could be a valuable addition to current video compression standards. The invention can be used in a variety of applications where low bitrate is a critical factor, such as video transmission over low-bandwidth networks or storage to devices with limited capacity.

[0089] Accordingly, the Disclosure provides support for a method comprising: receiving a series of input video frames, each comprising a first frame, a second frame, and a third frame located at a predetermined temporal position between the first and second frames, wherein the first frame has a first resolution, the second frame has a second resolution, and the third frame has a third resolution lower than the first and second resolutions; providing the first frame, the second frame, and the third frame to a trained video frame interpolation (VFI) model; and generating an interpolated frame at a predetermined temporal position between the first and second frames using the trained VFI model, wherein the interpolated frame has a first resolution. In a first example of the Method, the first frame, the second frame, and the third frame are compressed. In a second example of the Method, the first frame, the second frame, and the third frame are compressed in accordance with the H.26x video compression standard, optionally including the first example. A third example of this method optionally includes one or both of the first and second examples, wherein the third resolution is smaller than the size of the first resolution (e.g., less than or equal to 1 / 4 of the size of the first resolution along both the X and Y axes, or between 1 / 4 and 1 / 2 of the size of the first resolution along both the X and Y axes) to trade off bitrate and quality. A fourth example of this method optionally includes one or more of the first to third examples, or each of them, wherein the method further includes upscaling the third frame to the first resolution before providing the third frame to a trained VFI model. A fifth example of the method optionally includes one or more of the first to fourth examples, or each of them, wherein the third frame is upsampled via pixel duplication. A sixth example of this method optionally includes one or more of the first to fifth examples, further comprising concatenating the first, second, and third frames on a channel-by-channel basis before providing the first, second, and third frames to a trained VFI model.A seventh example of this method optionally includes one or more of the first to sixth examples, further comprising down-shuffling the first, second, and third frames before concatenating them channel by channel. An eighth example of this method optionally includes one or more of the first to seventh examples, further comprising receiving an indicator to disable the use of the third frame, the indicator having a first predetermined value representing the disablement of the use of the third frame and a second predetermined value representing the enablement of the use of the third frame, and providing the first, second, and third frames to a trained VFI model depends on the indicator value matching the second predetermined value.

[0090] The Disclosure also provides support for a method for interpolating video frames, the method comprising: receiving an input video frame in a series, the input video frame in a series comprising: a first higher resolution frame having a first temporal position in the series; a second higher resolution frame having a second temporal position in the series; and a lower resolution intermediate frame having an intermediate temporal position in the series between the first and second temporal positions; upscaling the lower resolution intermediate frame to create an upscaled intermediate frame; concatenating the first higher resolution frame, the second higher resolution frame, and the upscaled intermediate frame to produce a concatenated frame; inputting the concatenated frame into a trained video frame interpolation (VFI) model; and outputting an interpolated frame from the trained VFI model, the interpolated frame having an intermediate temporal position in the series. In a first example of the Method, the resolution of the lower resolution intermediate frame is less than or equal to one-quarter the size of the resolution of the first higher resolution frame along both the X and Y axes. In an alternative example of this method, the resolution of the lower-resolution intermediate frame is 1 / 4 to 1 / 2 the size of the resolution of the first higher-resolution frame along both the X and Y axes. In a second example of the method, optionally including the first example, the resolution of the first higher-resolution frame, the resolution of the second higher-resolution frame, and the resolution of the lower-resolution intermediate frame are the same. In a third example of this method, optionally including one or both of the first and second examples, the upscaling of the lower-resolution intermediate frame is by pixel duplication. In a fourth example of this method, optionally including one or more of the first to third examples, the concatenation of the first higher-resolution frame, the second higher-resolution frame, and the upscaled intermediate frame is performed on a channel-by-channel basis.A fifth example of the method optionally includes one or more of the first through fourth examples, the method further comprising down-shuffling the first higher-resolution frame, the second higher-resolution frame, and the upscaled intermediate frame after upscaling the lower-resolution intermediate frame.

[0091] This disclosure also provides support for a system for interpolating video frames, the system comprising a trained video frame interpolation (VFI) model, one or more processors, and non-temporary memory having executable instructions, the executable instructions, when executed, cause one or more processors to receive a first high-resolution frame, a second full-resolution frame, and a low-resolution intermediate frame at a temporal position between the first full-resolution and the second full-resolution; upscale the low-resolution intermediate frame to produce an upscaled intermediate frame, the resolution of which the upscaled intermediate frame is the same as the resolution of the first full-resolution and the second full-resolution; concatenate the first full-resolution, the second full-resolution and the upscaled intermediate frame into a concatenated frame; provide the concatenated frame to the trained VFI model to output an interpolated frame at a temporal position, the resolution of which the interpolated frame is the same as the resolution of the first full-resolution and the second full-resolution. In the first example of the system, the first full resolution, second full resolution, and lower resolution intermediate frames are compressed in accordance with a video compression standard selected from the H.264 Advanced Video Coding (AVC) standard and the H.265 High Efficiency Video Coding (HEVC) standard. The second example of the system optionally includes the first example, and the upscaling of intermediate hint frames is done via pixel duplication. The third example of the system optionally includes one or both of the first and second examples, and the concatenation of the first full resolution, second full resolution, and upscaled intermediate frames is done on a channel-by-channel basis. The fourth example of the system optionally includes one or more of the first to third examples, or each of them, and the executable instruction, when executed, causes one or more processors to further downshuffle the first full resolution, second full resolution, and upscaled intermediate frames before concatenating them.

[0092] The following claims specifically point to certain combinations and partial combinations that are considered novel and non-obvious. These claims may refer to an “an” element or a “first” element, or equivalents thereof. Such claims should be understood to include the incorporation of one or more such elements, and not to require or exclude two or more such elements. Other combinations and partial combinations of the disclosed features, functions, elements, and / or properties may be claimed through modifications to these claims or through the presentation of new claims in this application or related applications. Such claims, whether broader, narrower, equal to or different from the original claims, are also considered to be included within the subject matter of this disclosure.

Claims

1. It is a method, Receiving a series of input video frames comprising a first frame, a second frame, and a third frame located at a predetermined temporal position between the first frame and the second frame, wherein the first frame and the second frame have a first resolution, and the third frame has a second resolution lower than the first resolution. The first frame, the second frame, and the third frame are provided to a trained video frame interpolation (VFI) model, A method comprising generating an interpolated frame at a predetermined temporal position between the first frame and the second frame using the trained VFI model, wherein the interpolated frame has the first resolution.

2. The method according to claim 1, wherein the first frame, the second frame, and the third frame are compressed.

3. The method according to claim 2, wherein the first frame, the second frame, and the third frame are compressed in accordance with the H.26x video compression standard.

4. The method according to claim 1, wherein the size of the second resolution is smaller than the size of the first resolution along both the X and Y axes.

5. The method according to claim 4, wherein the size of the second resolution is 1 / 4 or less of the size of the first resolution along both the X and Y axes.

6. The method according to claim 4, wherein the size of the second resolution is 1 / 4 to 1 / 2 of the size of the first resolution along both the X and Y axes.

7. The method according to claim 1, further comprising upscaling the third frame to the first resolution before providing the third frame to the trained VFI model.

8. The method according to claim 7, wherein the third frame is upsampled via pixel duplication.

9. The method according to claim 1, further comprising concatenating the first frame, the second frame, and the third frame on a channel-by-channel basis before providing the first frame, the second frame, and the third frame to the trained VFI model.

10. The method according to claim 9, further comprising down-shuffling the first frame, the second frame, and the third frame before concatenating the first frame, the second frame, and the third frame on a channel-by-channel basis.

11. The further includes receiving an indicator for disabling the use of the third frame, the indicator having a first predetermined value representing disabling the use of the third frame and a second predetermined value representing enabling the use of the third frame, The method according to claim 1, wherein providing the first frame, the second frame, and the third frame to the trained VFI model depends on the value of the indicator matching a second predetermined value.

12. A method for interpolating video frames, wherein the method is Receiving a series of input video frames, the series of input video frames comprising a first higher resolution frame having a first temporal position in the series, a second higher resolution frame having a second temporal position in the series, and a lower resolution intermediate frame having an intermediate temporal position in the series between the first and second temporal positions. Upscaling intermediate frames with a lower resolution than those mentioned above to create upscaled intermediate frames, The first higher resolution frame, the second higher resolution frame, and the upscaled intermediate frame are concatenated to generate a concatenated frame, The concatenated frames are input into a trained video frame interpolation (VFI) model, A method for interpolating video frames, comprising outputting an interpolated frame from the trained VFI model, wherein the interpolated frame has the intermediate temporal position of the series.

13. A method for interpolating video frames according to claim 12, wherein the resolution size of the intermediate frame with a lower resolution is smaller than the resolution size of the first frame with a higher resolution.

14. A method for interpolating video frames according to claim 13, wherein the size of the resolution of the intermediate frame with a lower resolution is less than or equal to one-quarter of the size of the resolution of the first higher resolution frame along both the X and Y axes.

15. A method for interpolating video frames according to claim 13, wherein the size of the resolution of the intermediate frame with a lower resolution is 1 / 4 to 1 / 2 of the size of the resolution of the first higher resolution frame along both the X and Y axes.

16. A method for interpolating video frames according to claim 12, wherein the resolution of the first higher resolution frame, the resolution of the second higher resolution frame, and the resolution of the lower resolution intermediate frame are the same.

17. The method for interpolating video frames according to claim 16, wherein the upscaling of intermediate frames of lower resolution is performed via pixel duplication.

18. A method for interpolating video frames according to claim 12, wherein the concatenation of the first higher resolution frame, the second higher resolution frame, and the upscaled intermediate frame is on a channel-by-channel basis.

19. A method for interpolating video frames according to claim 18, further comprising down-shuffling the first higher resolution frame, the second higher resolution frame, and the upscaled intermediate frame after upscaling the intermediate frame of a lower resolution.

20. A system for interpolating video frames, A trained video frame interpolation (VFI) model, One or more processors, A non-temporary memory having executable instructions, wherein, when an executable instruction is executed, it is sent to one or more processors. The system receives a first full resolution, a second full resolution, and an intermediate frame of lower resolution at a temporal position between the first full resolution and the second full resolution. An intermediate frame with a lower resolution than the above is upscaled to create an upscaled intermediate frame, wherein the resolution of the upscaled intermediate frame is the same as the resolution of the first full resolution and the second full resolution. The first full resolution, the second full resolution, and the upscaled intermediate frame are concatenated into a concatenated frame. The connecting frame is provided to the trained VFI model. A system for interpolating video frames, wherein an interpolated frame is output at the aforementioned temporal position, and the resolution of the interpolated frame is the same as the resolution of the first full resolution and the second full resolution.

21. A system for interpolating video frames according to claim 20, wherein the first full resolution, the second full resolution, and intermediate frames of lower resolution are compressed in accordance with a video compression standard selected from the H.264 Advanced Video Coding (AVC) standard and the H.265 High Efficiency Video Coding (HEVC) standard.

22. The system for interpolating video frames according to claim 20, wherein the upscaling of the intermediate frames is performed via pixel duplication.

23. A system for interpolating video frames according to claim 20, wherein the concatenation of the first full resolution, the second full resolution, and the upscaled intermediate frames is on a channel-by-channel basis.

24. When the aforementioned executable instruction is executed, it further causes one or more processors to: A system for interpolating video frames according to claim 20, wherein the first full resolution, the second full resolution, and the upscaled intermediate frames are down-shuffled before concatenating the first full resolution, the second full resolution, and the upscaled intermediate frames.

Images