CNN-based system and method for video frame interpolation

Abstract

A system for a convolutional neural network (CNN) that synthesizes an intermediate non-existing frame based on an input frame pair includes: a coarse CNN that receives a pair of images acquired at consecutive time points, a registration module, a refined CNN, an adder, and a motion-compensated frame interpolation (MC-FI) module. The coarse CNN outputs a previous feature map, a next feature map, a coarse interpolated motion vector field (IMVF), and an occlusion map based on the pair of images, the registration module uses the coarse IMVF to warp the previous feature map and the next feature map to align with the pixel position of the coarse IMVF, and outputs the registered previous feature map and the next feature map, the refined CNN uses the registered previous feature map and the next feature map to correct the coarse IMVF, and the adder adds the coarse IMVF to the correction and outputs the final IMVF.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to systems and methods for video processing, and more particularly to CNN-based systems and methods for video frame interpolation. Background Art

[0002] In video frame interpolation (VFI), intermediate non-existing frames are synthesized from the original input frames. The classic application requiring VFI is frame rate up-conversion, which is used to deal with problems such as display motion blur and jitter in LED / LC displays. Other applications include frame recovery in video encoding and streaming, slow motion effects, and novel view synthesis.

[0003] Conventional methods for VFI, which include carefully hand-designed modules, usually include the following steps: bidirectional motion estimation, motion interpolation and occlusion inference, and motion-compensated frame interpolation (MC-FI). Such methods are prone to various artifacts such as halos, ghosting, and tearing due to insufficient quality of any of the above components.

[0004] Over the past few years, deep learning, especially convolutional neural networks (CNNs), has become a leading approach for many image processing and computer vision tasks. VFI processing can benefit from the use of these cutting-edge techniques, such as replacing some steps in the VFI algorithm pipeline with one or more pre-trained CNNs. Summary of the invention

[0005] Exemplary embodiments of the present disclosure relate to systems and methods for synthesizing intermediate non-existing frames from input frames in a given video. In contrast to conventional methods for VFI, embodiments of the present disclosure focus on designing a CNN-based framework that retains simple hand-crafted modules and replaces more complex components with CNNs that have been pre-trained on a large number of examples. The methods according to embodiments of the present disclosure can reduce the level of halo, ghosting, and breakup artifacts, which are typical features of conventional solutions. In addition, by selecting a well-balanced mixture of conventional components and CNN-based components, the methods according to embodiments of the present disclosure can be effectively implemented on various platforms from GPUs to neural processors to dedicated hardware.

[0006] According to an embodiment of the present disclosure, a system for synthesizing intermediate non-existing frames based on input frame pairs in a given video using a convolutional neural network (CNN) is provided, the system comprising: a coarse convolutional neural network (CNN), receiving a pair of images acquired at consecutive time points, wherein the pair of images comprises a previous image and a next image; a registration module, connected to the coarse CNN; a refinement CNN, connected to the registration module and the coarse CNN; an adder, connected to the refinement CNN and the coarse CNN; and a motion-compensated frame interpolation (MC-FI) module, connected to the adder and the coarse CNN. The coarse CNN outputs a previous feature map and a next feature map according to a previous image and a next image, outputs a coarse interpolated motion vector field (IMVF) and an occlusion map according to a pair of images, the registration module uses the coarse IMVF to warp the previous feature map and the next feature map to align with the pixel positions of the coarse IMVF, and outputs the registered previous feature map and the registered next feature map, the refined CNN uses the registered previous feature map and the registered next feature map to correct the coarse IMVF, and the adder adds the coarse IMVF to the correction of the coarse IMVF and outputs the final IMVF.

[0007] According to another embodiment of the present disclosure, a motion-compensated frame interpolation (MC-FI) module generates an interpolated frame corresponding to a time between time points of a previous image and a next image by warping the previous image and the next image using a final IMVF and weighted blending the warped previous image and the warped next image using an occlusion weight from an occlusion map.

[0008] According to another embodiment of the present disclosure, a coarse CNN receives the pair of images at multiple resolution levels. The coarse CNN includes: a feature extraction subnetwork that generates a pair of feature maps corresponding to a pair of images at each resolution level; an encoder-decoder subnetwork that cascades a pair of feature maps at each resolution level into a single feature map, and processes the single feature map to generate a new single feature map with reduced spatial resolution; a fusion subnetwork that merges the new single feature maps at each resolution level into a single merged feature map by performing weighted averaging on the new single feature maps at each resolution level, wherein the weights used for the weighted averaging are learned in the training phase and are different for each pixel; and an estimation subnetwork that outputs the horizontal and vertical components of the coarse IMVF and an occlusion map. The feature extraction subnetwork includes a conjoined layer.

[0009] According to another embodiment of the present disclosure, the estimation subnetwork includes a horizontal submodule, a vertical submodule, and an occlusion map submodule, wherein each submodule receives a single merged feature map output from the fusion subnetwork. The horizontal submodule and the vertical submodule output a horizontal probability map and a vertical probability map, respectively, each probability map having S probability values ​​for each pixel. Each probability value represents the probability that the motion vector is one of the S displacement values ​​for the pixel. The horizontal submodule and the vertical submodule calculate the first-order moment of the probability value for each pixel, respectively, to determine the desired horizontal component and vertical component for each pixel, wherein the pair consisting of the desired horizontal component and vertical component for each pixel includes a correction for the rough IMVF.

[0010] According to yet another embodiment of the present disclosure, the occlusion map submodule outputs an occlusion map including per-pixel weights for performing weighted averaging between a previous image and a next image.

[0011] According to another embodiment of the present disclosure, the refinement CNN includes: an encoder-decoder subnetwork, which concatenates the aligned previous feature map with the aligned next feature map and outputs a new set of feature maps, the new set of feature maps having a spatial resolution that is resized relative to the full resolution of the previous image and the next image; and an estimation subnetwork, which estimates the correction of the horizontal component and the vertical component of the rough IMVF for each block in the aligned next feature map and the aligned previous feature map to output a corrected IMVF.

[0012] According to another embodiment of the present disclosure, the estimation subnetwork includes a horizontal submodule and a vertical submodule. The horizontal submodule and the vertical submodule output a horizontal probability map and a vertical probability map, respectively, wherein each probability map has S probability values ​​for each pixel, wherein each probability value represents the probability that the motion vector is one of the S displacement values ​​for the pixel. The horizontal submodule and the vertical submodule respectively calculate the first-order moment of the probability value for each pixel to determine the expected horizontal component and vertical component for each pixel, wherein the pair consisting of the expected horizontal component and vertical component for each pixel includes a correction for the rough IMVF.

[0013] According to another embodiment of the present disclosure, a method for synthesizing an intermediate non-existing frame based on a pair of input frames in a given video using a convolutional neural network (CNN) is provided, including: receiving a pyramid representation of a pair of consecutive input frames, wherein the pair of consecutive input frames includes a previous image and a next image, wherein the pyramid representation includes multiple pairs of input frames, each pair of input frames is at a different spatial resolution level; generating a pair of feature maps from each resolution level of the pyramid representation, and estimating a coarse interpolated motion vector field (IMVF) and an occlusion map based on each pair of feature maps; aligning a pair of feature maps of the same resolution level based on the coarse IMVF and the occlusion map by warping each feature map in the pair of feature maps to align with the pixel position of the coarse IMVF, and outputting the aligned previous feature map and the aligned next feature map; correcting the coarse IMVF using the aligned previous feature map and the aligned next feature map to generate a correction to the coarse IMVF; adding the correction to the coarse IMVF to the coarse IMVF to generate a refined IMVF; and generating a synthesized intermediate frame based on the pair of consecutive input frames, the refined IMVF, and the occlusion map.

[0014] According to another embodiment of the present disclosure, generating a pair of feature maps includes: generating a pair of feature maps at each spatial resolution level for each pair of input frames in multiple pairs of input frames, wherein each pair of feature maps has a spatial resolution reduced relative to the resolution of a pair of input frames; cascading the feature maps at each resolution level, and processing the cascaded feature maps to generate a new set of feature maps, the new set of feature maps having a spatial resolution reduced relative to the resolution of a pair of consecutive input frames; merging the new sets of feature maps for all spatial resolution levels into a single merged feature map by performing weighted averaging on the new set of feature maps for each resolution level, wherein the weights used for the weighted averaging are learned in a training phase and are different for each pixel; and estimating horizontal and vertical components of a coarse IMVF and an occlusion map for each block in the single merged feature map, wherein the occlusion map includes per-pixel weights for performing weighted averaging between a previous image and a next image.

[0015] According to another embodiment of the present disclosure, estimating the horizontal component and the vertical component of the rough IMVF includes: generating a horizontal probability map and a vertical probability map, wherein each probability map has S probability values ​​for each pixel, wherein each probability value represents the probability that the motion vector is one of the S displacement values ​​for the pixel, and calculating the first-order moment of the probability value for each pixel to determine the expected horizontal component and vertical component for each pixel, wherein the pair consisting of the expected horizontal component and vertical component for each pixel includes a correction for the rough IMVF.

[0016] According to another embodiment of the present disclosure, the rough IMVF includes two 2D maps of horizontal and vertical shifts pointing from the rough IMVF to the next image, and the registration feature map pair includes: using opposite values ​​of these shifts to warp the features from the previous image to the position of the coarse IMVF, and warping the input features from the next image to the said position of the coarse IMVF using the horizontal and vertical shifts as is.

[0017] According to another embodiment of the present disclosure, the registered feature map pair also includes: when the previous feature map of the registration has an occluded area, replacing the occluded area in the previous feature map of the registration with the corresponding area in the next feature map of the registration, and when the next feature map of the registration has an occluded area, replacing the occluded area in the next feature map of the registration with the corresponding area in the previous feature map of the registration.

[0018] According to another embodiment of the present disclosure, correcting a rough IMVF includes: concatenating features of a previous feature map and a next feature map that are aligned, and processing the concatenated feature maps to generate a new set of feature maps, the new set of feature maps having a spatial resolution that is resized relative to the full resolutions of the previous image and the next image; and estimating corrections to horizontal and vertical components of the rough IMVF for each block in the rough IMVF to generate corrections to the rough IMVF.

[0019] According to yet another embodiment of the present disclosure, generating a synthesized intermediate frame includes: warping a pair of consecutive input frames according to a refined IMVF; and performing weighted blending on the warped pair of images using occlusion weights from an occlusion map to generate a synthesized intermediate frame.

[0020] According to yet another embodiment of the present disclosure, the method includes removing blocking artifacts from motion boundaries by performing weighted averaging of interpolated pixels in a synthesized intermediate frame using estimates from neighboring blocks.

[0021] According to another embodiment of the present disclosure, there is provided a computer-readable, non-transitory program storage device tangibly embodying a program of instructions executed by a computer to perform method steps for synthesizing intermediate non-existing frames from input frame pairs in a given video using a convolutional neural network (CNN). BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A pair of consecutive input frames is shown according to an embodiment of the present disclosure.

[0023] Figure 2 A pair of consecutive input frames with complex motion is shown according to an embodiment of the present disclosure.

[0024] Figure 3An example of a coarse CNN with 3 pyramid levels according to an embodiment of the present disclosure is shown.

[0025] Figure 4 An occlusion-aware registration module according to an embodiment of the present disclosure is shown.

[0026] Figure 5 An example of a refined CNN according to an embodiment of the present disclosure is shown.

[0027] Figure 6 An overview of a CNN-based framework for VFI according to an embodiment of the present disclosure is shown.

[0028] Figure 7 The embodiment of the present disclosure shows Figure 1 The result of applying the Hierarchical Coarse CNN and MC-FI modules to a pair of frames.

[0029] Figure 8 The embodiment of the present disclosure shows Figure 2 The results of applying the coarse CNN and the refined CNN to a pair of frames.

[0030] Fig. 9 The embodiment of the present disclosure shows Figure 2 Results of performing a CNN-based VFI solution on a pair of frames.

[0031] Fig.10 is a block diagram of a system implementing a method for synthesizing intermediate non-existing frames from input frame pairs in a given video using a CNN according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Exemplary embodiments of the present disclosure as described herein generally provide systems and methods for synthesizing intermediate non-existing frames from input frames in a given video using a CNN. Although the embodiments are susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that this is not intended to limit the present disclosure to the specific form disclosed, but rather, the present disclosure shall cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the present disclosure.

[0033] Figure 1 A pair of consecutive input frames is depicted, with the previous frame at the top and the next frame at the bottom. In this example, the background remains stationary while the left hand and the rest of the upper body each move with a different motion vector.

[0034] Basic Assumptions

[0035] According to an embodiment of the present disclosure, a linear motion between a pair of consecutive input frames is assumed. Specifically, this means that the motion from each pixel position in the estimated intermediate frame to the previous frame is equal to the motion from that position to the next frame.

[0036] Task Definition

[0037] The method according to an embodiment of the present disclosure decomposes the task of synthesizing an intermediate frame according to a pair of two consecutive input frames into three steps:

[0038] 1) Estimate the motion from each pixel position in the synthesized frame to its corresponding position in each of the input frames. The set of motion vectors from all such pixel positions is considered to be the interpolated motion vector field (IMVF).

[0039] 2) Classify each pixel position in the synthesized frame into one of the following categories: "closed", "open", or "non-occluded". The first category means that the pixel position can only be found in the previous input frame, the second category means that it can only be found in the next frame, and the last category means that it can be found in both the previous and next frames. The map of these classification categories is considered the occlusion map.

[0040] 3) Perform motion-compensated warping of the input frame according to the occlusion map and perform weighted blending of the warped input, which acts as a local weight for the blending process.

[0041] CNN-based framework for VFI

[0042] The system according to an embodiment of the present disclosure includes two CNNs and two conventional modules to handle registration and motion-compensated frame interpolation (MC-FI). First, a coarse CNN according to an embodiment estimates a coarse IMVF and an occlusion map for each K×K block in the output interpolated frame. The registration module uses these estimates to create a registered feature map, which serves as the input of a second CNN according to an embodiment that refines the IMVF. The refined IMVF and occlusion map are input to the MC-FI module that produces an interpolated intermediate frame.

[0043] Each CNN in the two CNNs according to the embodiment plays a different role. The coarse CNN according to the embodiment can capture the most important motion in the scene, which is usually a combination of global motion and a set of simple rigid local motions. The coarse CNN can also find occlusion areas mainly on the motion boundaries. Using the estimates of the coarse CNN for IMVF and occlusion maps, the input features extracted from these frames are distorted. After the distortion, errors and residual motion may still remain. The second refined CNN according to the embodiment operates on the distorted input features, and can correct errors and capture secondary motion.

[0044] According to an embodiment, splitting the IMVF estimation into two steps is useful for scenes with complex motion. An example of a pair of input frames with complex motion is Figure 2 , where the previous frame is at the top and the next frame is at the bottom. The car closest to the camera is moving to the right and its wheels are also rotating, while another car farther from the camera is moving in a different direction. A coarse CNN according to an embodiment can capture the camera motion as well as the local motion of the car closest to the camera. However, it is challenging to accurately capture the motion of the rotating wheels at the same time. In this case, a refined CNN according to an embodiment comes into play.

[0045] According to an embodiment, the basic building blocks used to construct the two CNNs include convolution, average pooling, bilinear upsampling, cascading, and soft-max layers; all of which are common practices in the field of deep learning. Using these building blocks, the following types of sub-networks can be designed:

[0046] 1. Feature extraction;

[0047] 2. Encoder-Decoder;

[0048] 3. Feature fusion; and

[0049] 4. Estimate.

[0050] Each subnetwork is composed of specific layers, which include the following types:

[0051] (i) Convolution;

[0052] (ii) Average pooling;

[0053] (iii) Cascade;

[0054] (iv) Bilinear upsampling;

[0055] (v) SoftMax; and

[0056] (vi) Centroid.

[0057] The first five types are very common practices in the field of deep learning. Note that each convolutional layer that is not followed by a SoftMax layer includes a nonlinear (usually a rectified linear) unit activation. The last type is a linear operation in which the first-order moment of a given distribution is calculated. The two CNNs according to the embodiment are composed of some or all of these types of subnetworks. Each convolutional layer in the neural network has its own set of parameters. The parameters of the convolutional layer in the neural network include weights and biases. In the convolutional layer, the weights correspond to the coefficients of the 3D convolution kernel, and the bias is a set of offsets added to the result of the 3D convolution operation. For example, if the convolutional layer supports a size of 3×3 filters, consider a pixel and a 3×3 pixel grid around the pixel. Multiply each of these pixels by the weight value, and then add all these weights. Then add the bias and generate an output feature value. Repeat this operation for all pixel positions and all output channels, where different 3×3 filters and biases can be applied for each output channel. The weights and biases of the convolutional layer are learned when training the CNN. In order to learn the parameters of each CNN, the CNN is trained with a given MC-FI module to generate an output frame. This allows training a CNN in an end-to-end manner using a dataset of frame triplets, where in each triplet the middle frame serves as the ground truth for frame synthesis from the other two frames. Specifically, this means that the ground truth for VFI can be easily obtained by subsampling the video.

[0058] Rough CNN

[0059] The hierarchical coarse CNN according to an embodiment receives a pair of input frames at several resolutions, for example: level 0 - full resolution, level 1 - reduced to 1 / 2, level 2 - reduced to 1 / 4, and so on. For a description of the image pyramid, see Figure 3 . To generate an image pyramid for each input frame, standard image resizing operations are used with 1 / 2 (for level 1) and 1 / 4 (for level 2) per axis. These resizing operations are based on bilinear or bicubic interpolation. Each input passes through the same pair of layers with the same set of layer parameters. Such layer pairs are often referred to as Siamese layers. Siamese layers produce a set of feature maps for each input, outputting N values ​​per pixel, with the number of pixels per axis reduced to 1 / M relative to the input resolution. An exemplary, non-limiting value for N is N=25. As with the rest of the network, weights and biases can be learned in this case. The low-level features extracted from the Siamese layers of a CNN typically include strong responses to image edges and color representations.

[0060] According to an embodiment, when moving from input to output, the encoder-decoder subnetwork is trained to extract features with increased high-level context. The output of the encoder-decoder is a 3D image, in which each spatial entry in the image is a high-dimensional representation. Each pair of feature maps obtained from the same pyramid level in two input frames is cascaded and processed by the encoder-decoder subnetwork layer to obtain a new set of feature maps whose spatial resolution is reduced to 1 / L relative to the full resolution. Cascading means appending the N feature channels from the second input frame to the N feature channels of the first input frame, producing a total of 2N feature channels. The 2N feature channels are passed as input to the encoder-decoder layer, thereby producing an output of J values ​​(channels) per pixel, and the number of pixels on each axis is reduced to 1 / L relative to the input image. For example, in some embodiments, J=2N. The processing performed by the encoder-decoder subnetwork is a deep cascade of convolution, average pooling, and upsampling layers. Each convolution layer is also followed by a nonlinear unit, such as a rectified linear unit. Taken together, the subnetwork according to an embodiment learns how to extract features with increasing high-level context as processing moves from input to output.The output of the encoder-decoder is a 3D image, where each spatial entry in the image is a high-dimensional feature vector.

[0061] Then, according to an embodiment, the outputs of all decoders are merged into a coarse IMVF in a locally adaptive manner using a fusion subnetwork. According to an embodiment, there are three encoder-decoder submodules, one for each level of resolution, and each encoder-decoder submodule produces an output of the same size: W / L×H / L×2N, where W and H are the width and height of the input frame in pixels. The fusion submodule takes these three outputs and combines them into a single output by performing a weighted average on the three inputs. The weights of this average are adaptive in the sense that they are learned, and local in the sense that they are different for each pixel.

[0062] According to an embodiment, non-limiting values ​​of L and M are L = 8 and M = 4. These values ​​are fundamental properties of the architecture of the network and are set by the number and structure of layers within the CNN that change the spatial resolution.

[0063] According to an embodiment, the merged decoder output is input into an estimation subnetwork layer, which estimates the horizontal and vertical components of the rough IMVF and the occlusion map for each K×K block in the output image. For example, in some embodiments, K=L. The rough estimation layer includes three submodules: a horizontal rough IMVF submodule, a vertical rough IMVF submodule, and an occlusion map submodule, and each submodule receives the output of the fusion subnetwork. The horizontal rough IMVF submodule and the vertical rough IMVF submodule each output a similar structure: a rough horizontal probability map and a rough vertical probability map of size W / L×H / L×S, that is, there are S values ​​for each pixel in the probability map. Each of the S probabilities of each pixel represents the probability of the motion component becoming one of the S displacement values, for example, for S=25, the displacement can be -96, -88, ... 0, ... 88, 96 (in pixel units). For example, in some embodiments, S=N. Then, the first-order moment (centroid) of the distribution is calculated for each pixel to determine the horizontal and vertical components of each pixel. The expected value pair of the horizontal component and the vertical component of each pixel is the IMVF.

[0064] According to an embodiment, the estimation path for each of the horizontal / vertical motion components includes a shallow cascade of convolutional layers, soft-max layers, and centroid layers. The convolutional layer processes the output of the merged decoder output so that it will match the specific estimation task in progress. The soft-max layer converts the output of the last convolutional layer into a normalized discrete probability vector, where each spatial position has a probability vector. Each motion estimation path in the motion estimation path uses S classes, each class corresponding to a specific motion value. The centroid layer calculates the first-order moment of each probability distribution and then converts the probability vector into an expected motion value.

[0065] According to an embodiment, when a new intermediate frame is created, the previous frame and the next frame are separately registered according to the IMVF, after which they are combined into a single frame. This is done using an occlusion map, which includes per-pixel weights for performing a weighted average between the previous frame and the next frame. The occlusion weight for each pixel is calculated through a separate estimation path. Similar to the motion estimation path, a shallow cascade of convolutional layers and soft-max layers is applied to the merged decoder output. In this estimation path, there are two classes - "covered" and "uncovered". The occlusion weight is calculated by taking the probability of the first class. When the value of the occlusion map is 0.5, the pixels from the previous frame and the next frame will be averaged with equal weights. When the value of the occlusion map is 1, only the previous frame pixels are used, and when the value of the occlusion map is 0, only the next frame pixels are used. These values ​​of 0 and 1 are used in the case of occlusion. For example, if the ball moves between two frames, the pixels behind the ball in the previous frame are now visible, and when performing interpolation, only some of the pixels from the next image (the new pixels that are now exposed) and some of the pixels from the previous frame (the pixels hidden by the advancing ball in the next frame) are used. The occlusion map is determined by a submodule similar to the IMVF submodule, but instead of getting S channels per pixel, only two channels are output, of which the occlusion map is one of the channels.

[0066] An example of a coarse CNN with 3 pyramid levels is shown in Figure 3 Now refer to Figure 3 , two consecutive input frames given at three resolutions are shown on the left. The three resolutions are indicated by level 0, level 1 and level 2, respectively. Note that this number of resolution levels is exemplary and non-restrictive, and there may be more or fewer resolution levels in other embodiments. Each pair of input frames is input into the Siamese layer of the feature extraction subnetwork 31, which outputs a pair of feature maps for each pair of input frames. The pair of feature maps output by the Siamese layer is input into the encoder-decoder subnetwork 32, which outputs the processed feature maps, which are provided to the feature fusion subnetwork 33. The feature fusion subnetwork 33 merges the outputs of the encoder-decoder subnetwork 32 to produce a rough IMVF and an occlusion map 34. The estimated IMVF (overlaid on the output image) and the occlusion map are shown on the right.

[0067] Occlusion-aware registration

[0068] According to an embodiment, the registration module receives a pair of input feature maps extracted by the convolutional layer of the Siamese subnetwork of a given pyramid level. Using the coarse IMVF output from the feature fusion subnetwork, the registration module warps each of these feature maps to align with the pixel position of the output frame, and outputs a registered previous feature map and a registered next feature map.

[0069] According to an embodiment, the IMVF includes two 2D maps of horizontal and vertical shifts from the output frame to the next frame. In the registration module, these shifts are used to generate two versions of the warped input features:

[0070] (1) The opposite values ​​of these shifts are used to warp the input features from the previous frame to the positions of the output frame.

[0071] (2) These shifts are used as is to warp the input features from the next frame to the positions of the output frame.

[0072] Figure 4 An exemplary flow of an occlusion-aware registration module according to an embodiment is shown. Due to occlusion, some areas in the output frame can only be synthesized from one of the input frames. This means that in the distorted feature map, there are areas with very low confidence. To handle these areas, according to an embodiment, the following occlusion-aware compensation can be applied. Figure 4 The top row of , replaces the occluded areas (white holes) in the warped feature map of the previous frame 42a with the warped features of the next frame at these positions 42b to produce an output frame 43a, and reference Figure 4 In the bottom row of FIG. 4 , the occluded areas in the warped feature map of the next frame 42b are replaced with the warped features of the previous frame at these locations 42a to produce an output frame 43b. After this compensation, the registered previous and next feature maps will coincide at the occluded pixel locations, so that there is no further room for motion refinement at these locations.

[0073] Refine CNN

[0074] Figure 5 An example of a refined CNN according to an embodiment is shown in , where the refined CNN receives the registered feature map from one level of the image pyramid. The refined CNN uses the two coarsely registered input features as input, concatenates them, and then processes them similarly to the coarse CNN. Similar to the coarse CNN, the refined CNN according to an embodiment includes a horizontal submodule and a vertical submodule. Reference Figure 5, the refined CNN receives a pair of registered feature maps produced by the registration module. Similar to the coarse CNN, these features are cascaded and processed by the encoder-decoder subnetwork 51, resulting in a new set of feature maps in which the spatial resolution is reduced to 1 / Q relative to the full resolution. An exemplary non-limiting value of Q is Q=4. The decoder output and the coarse IMVF are then provided to the estimation subnetwork layer 52, which estimates the correction of the horizontal and vertical components of the coarse IMVF for each P×P block in the output image to output a refined IMVF. For example, in some embodiments, P=Q. The refined CNN according to the embodiment uses the coarsely registered input frame instead of directly outputting the IMVF correction from the coarse IMVF. The IMVF correction is then added to the resized version of the coarse IMVF to create the final refined IMVF. The resizing operation aligns the outputs estimated by the coarse CNN and the refined CNN to the same resolution. In the example of K=8, L=8, P=4, and Q=4, an upward scaling operation with a factor of 2 on each axis is used.

[0075] MC-FI module

[0076] The MC-FI module according to an embodiment uses two full-resolution input frames, a refined IMVF and an occlusion map to create a new interpolated frame at a given timestamp. According to an embodiment, similar to how the registration module warps each of these feature maps, the MC-FI module generates an output frame based on the input frame using the IMVF, where the occlusion weights are used to blend the two input frames. Similar to the first step of the registration module according to an embodiment, the MC-FI module first warps the input frame according to the IMVF. The MC-FI module then performs weighted blending on the warped image using the occlusion weights. Since the IMVF and occlusion weights are estimated for each K×K block in the output image, the MC-FI module includes a deblocking mechanism that removes blockiness artifacts from motion boundaries. More specifically, the MC-FI module performs weighted averaging on interpolated pixels obtained using estimates from adjacent blocks.

[0077] Algorithm Flow

[0078] Figure 6 The process of the CNN-based framework according to an embodiment for VFI is shown in FIG. Referring to the accompanying drawings, according to an embodiment, the entire algorithm process includes:

[0079] 1. Receive a pair of consecutive input frames;

[0080] 2. Generate a pyramid representation based on these frames;

[0081] 3. Provide the image pyramid pairs to the first hierarchical coarse CNN 61, which outputs a pair of feature maps for each pair of image pyramids and estimates a coarse IMVF and occlusion map;

[0082] 4. The registration module 62 aligns the feature map pairs generated by the coarse CNN 61 according to the IMVF and occlusion map estimated by the coarse CNN 61;

[0083] 5. Provide the coarse IMVF and the registered feature map pair to the second refined CNN 63, which estimates the correction to the coarse IMVF;

[0084] 6. The correction to the coarse IMVF is added to the upscaled version of the coarse IMVF by adder 64 to create a final IMVF; and

[0085] 7. The original input frame pair along with the final IMVF and estimated occlusion map are provided to the MC-FI module 65 which produces a synthesized intermediate frame.

[0086] Note that: (1) the coarse CNN can receive one or more levels from the image pyramid; and (2) if fast inference is prioritized over high accuracy, the registration module and refinement CNN can be bypassed.

[0087] Effect

[0088] The method according to the embodiment can incorporate deep learning modules into the VFI framework to replace some or all of the conventional hand-designed components. The pre-trained neural network can:

[0089] 1. More accurate estimation of occluded areas and complex motion; and

[0090] 2. More robust to sudden brightness changes.

[0091] These features improve frame interpolation quality, effectively reducing the levels of halo, ghosting and tearing artifacts typically observed with standard VFI methods.

[0092] In addition, the method according to the embodiment can learn from examples to perform the following tasks:

[0093] 1. Estimate the IMVF directly from the deep feature space, instead of computing the forward and backward MVFs and then interpolating them to the desired output grid.

[0094] 2. Estimating the occlusion map directly from the deep feature space, rather than through a hand-crafted analysis of forward and backward MVFs.

[0095] The method according to the embodiment can balance between using pre-trained neural networks and conventional hand-designed modules in VFI solutions. A good balance between the two types of components and a careful design of the neural network architecture can facilitate efficient implementation of high-quality VFI on platforms such as GPUs, neural processors, and dedicated hardware.

[0096] Embodiments of the present disclosure may be used in various applications that rely on VFI:

[0097] 1. Frame rate up-conversion for reducing display motion blur and jitter in LED / LCD displays;

[0098] 2. Frame recovery in video encoding and streaming;

[0099] 3. Improve the frame rate in video conference calls;

[0100] 4. Slow motion effects; and

[0101] 5. Synthesize novel views.

[0102] In scenes such as (3) and (4), motion and occlusion may be too complex, and image quality requirements may be too stringent for conventional VFI methods to handle. A system using CNN-based techniques according to an embodiment can better cope with the challenges faced in these scenes.

[0103] Example

[0104] Figures 7 to 9 Results obtained by a CNN-based solution for VFI according to an embodiment of the present disclosure are shown.

[0105] Figure 7 Describes when applied to Figure 1 , the outputs produced by the hierarchical coarse CNN and MC-FI modules according to an embodiment, including an estimated IMVF and an occlusion map, when a pair of input frames is shown in . It also shows the intermediate frames synthesized by the MC-FI module using these estimates. The top row shows, from left to right: the estimated horizontal and vertical components of the IMVF and the estimated occlusion map. The bottom row shows, from left to right: the estimated IMVF overlaid on the synthesized frame and the estimated occlusion map overlaid on the synthesized frame. Note that area 72 depicts an "off" occlusion type and the remaining pixel locations are not occluded.

[0106] Next, it is shown Figure 2 Results for a bunch of input frames with complex motion as shown in . Figure 8Depicted are the coarse IMVF obtained by the coarse CNN, which uses only the finest level of the image pyramid (full resolution), and the estimated correction of the IMVF produced by the refined CNN. The top row shows from left to right: the horizontal and vertical components of the IMVF estimated by the coarse CNN using only the finest level of the image pyramid (full resolution). The bottom row shows from left to right: the horizontal and vertical components of the ΔIMVF estimated by the refined CNN. It can be seen that the second CNN detects errors in the IMVF of the leading vehicle and also recovers the secondary motion of the wheels of that vehicle.

[0107] Fig. 9 Shows the Figure 2 A pair of frames of Figure 1 show the results of executing a CNN-based solution for VFI, and show the intermediate frames synthesized by two possible choices of the proposed framework: (i) Rough CNN + MC-FI; and (ii) Rough CNN + Registration + Refinement CNN + MC-FI (full pipeline). The top row shows the synthesized frames generated by the VFI solution (including Rough CNN + MC-FI). The bottom row shows the synthesized frames generated by the entire pipeline of the VFI solution (Rough CNN + Registration + Refinement CNN + MC-FI) according to an embodiment. When comparing the results of these two frameworks, it is easy to observe the quality improvement achieved by using the full pipeline: the text looks clearer, the boundaries of the car parts are sharper, and the level of detail of the wheels is also improved.

[0108] System Implementation

[0109] It should be understood that the embodiments of the present disclosure can be implemented in various forms of hardware, software, firmware, dedicated processing, or a combination thereof. In some embodiments, the present disclosure can be implemented in the form of hardware as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). In other embodiments, the present disclosure can be implemented in the form of software as an application program tangibly embodied on a computer-readable program storage device. The application program can be uploaded to and executed by a machine including any suitable architecture.

[0110] Fig.10 is a block diagram of a system according to an embodiment of the present disclosure, the system implementing a method for synthesizing intermediate non-existing frames from input frame pairs in a given video using a CNN. Now refer to Fig.10, the computer system 101 for implementing the present invention may include, among other things, a processor 102, a memory 103, and an input / output (I / O) interface 104. The computer system 101 is typically coupled to a display 105 and various input devices 106 (such as a mouse and keyboard) via the I / O interface 104. The support circuits may include circuits such as cache memory, power supply, clock circuits, and communication buses. The processor 102 may be a graphics processing unit (GPU), a neural processor, or dedicated hardware (HW). The GPU and the neural processor are suitable for running deep neural networks, and according to an embodiment, the GPU and the dedicated HW are a good match for the registration and MC-FI modules. The memory 103 may include a random access memory (RAM), a read-only memory (ROM), a disk drive, a tape drive, etc., or a combination thereof. The present disclosure may be implemented as a routine 107, which is stored in the memory 103 and executed by the processor 102. In this way, the computer system 101 is a general-purpose computer system, which becomes a special-purpose computer system when the routine 107 of the present invention is executed. Alternatively, as described above, embodiments of the present disclosure may be implemented as an ASIC or FPGA 107 in signal communication with the processor 102 .

[0111] The computer system 101 also includes an operating system and microinstruction codes. The various processes and functions described herein may be part of the microinstruction codes, or may be part of an application program (or a combination thereof) executed by the operating system. In addition, various other peripheral devices may be connected to the computer platform, such as additional data storage devices and printing devices.

[0112] It should also be understood that, because some of the constituent system components and method steps depicted in the accompanying drawings can be implemented in software form, the actual connections between the system components (or, processing steps) can be different according to the programming of the present invention. Given the teachings of the present invention provided herein, those of ordinary skill in the relevant art will be able to consider these and similar embodiments or configurations of the present invention.

[0113] Although the present invention has been described in detail with reference to exemplary embodiments, those skilled in the art will appreciate that various modifications and substitutions may be made thereto without departing from the spirit and scope of the present invention as set forth in the appended claims.

Claims

1. A system for synthesizing intermediate non-existing frames from pairs of input frames in a given video using a convolutional neural network (CNN), comprising: A coarse convolutional neural network (CNN) receives a pair of images acquired at consecutive time points, wherein the pair of images includes a previous image and a next image; A registration module, connected to the rough CNN; A refined CNN connected to the registration module and the coarse CNN; An adder, connected to the refined CNN and the coarse CNN; as well as A motion-compensated frame interpolation MC-FI module, connected to the adder and the coarse CNN, The rough CNN outputs a previous feature map and a next feature map according to the previous image and the next image, outputs a rough interpolated motion vector field IMVF and an occlusion map according to the pair of images, the registration module uses the rough IMVF to warp the previous feature map and the next feature map to align with the pixel positions of the rough IMVF, and outputs the registered previous feature map and the registered next feature map, the refined CNN uses the registered previous feature map and the registered next feature map to correct the rough IMVF, and the adder adds the rough IMVF to the correction of the rough IMVF and outputs the final IMVF.

2. The system according to claim 1, wherein: The motion-compensated frame interpolation MC-FI module generates an interpolated frame corresponding to the time between the time points of the previous image and the next image by warping the previous image and the next image using the final IMVF and weighted blending the warped previous image and the warped next image using the occlusion weights from the occlusion map.

3. The system according to claim 1, wherein: The coarse CNN receives the pair of images at multiple resolution levels, wherein the coarse CNN comprises: A feature extraction subnetwork generates a pair of feature maps corresponding to the pair of images at each resolution level; an encoder-decoder subnetwork that concatenates the pair of feature maps at each resolution level into a single feature map and processes the single feature map to produce a new single feature map with reduced spatial resolution; a fusion subnetwork that merges the new single feature maps at each resolution level into a single merged feature map by performing weighted averaging on the new single feature maps at each resolution level, wherein the weights for the weighted averaging are learned in a training phase and are different for each pixel; and An estimation subnetwork outputs the horizontal component and the vertical component of the rough IMVF and an occlusion map; and Wherein, the feature extraction sub-network includes a conjoined layer.

4. The system according to claim 3, wherein: The estimation subnetwork includes a horizontal submodule, a vertical submodule, and an occlusion map submodule, wherein each submodule receives the single merged feature map output from the fusion subnetwork, The horizontal submodule and the vertical submodule output a horizontal probability map and a vertical probability map, respectively, wherein each probability map has S probability values ​​for each pixel, wherein each probability value represents the probability that the motion vector is one of the S displacement values ​​for the pixel, The horizontal submodule and the vertical submodule respectively calculate the first-order moment of the probability value for each pixel to determine the desired horizontal component and vertical component for each pixel, wherein the pair consisting of the desired horizontal component and vertical component for each pixel includes the correction for the rough IMVF.

5. The system according to claim 4, wherein: The occlusion map submodule outputs the occlusion map, which includes per-pixel weights for performing a weighted average between the previous image and the next image.

6. The system according to claim 3, wherein: The refined CNN includes: an encoder-decoder subnetwork that concatenates the registered previous feature map with the registered next feature map and outputs a new set of feature maps having a spatial resolution resized relative to the full resolution of the previous image and the next image; and An estimation subnetwork estimates corrections to the horizontal component and the vertical component of the rough IMVF for each block in the registered next feature map and the registered previous feature map to output a corrected IMVF.

7. The system according to claim 6, wherein: The estimation sub-network includes a horizontal sub-module and a vertical sub-module, The horizontal submodule and the vertical submodule output a horizontal probability map and a vertical probability map, respectively, wherein each probability map has S probability values ​​for each pixel, wherein each probability value represents the probability that the motion vector is one of the S displacement values ​​for the pixel, The horizontal submodule and the vertical submodule respectively calculate the first-order moment of the probability value for each pixel to determine the desired horizontal component and vertical component for each pixel, wherein the pair consisting of the desired horizontal component and vertical component for each pixel includes the correction for the rough IMVF.

8. A method for synthesizing intermediate non-existing frames from input frames in a given video using a convolutional neural network (CNN), comprising the following steps: receiving a pyramid representation of a pair of consecutive input frames, wherein the pair of consecutive input frames comprises a previous image and a next image, wherein the pyramid representation comprises a plurality of pairs of input frames, each pair of input frames being at a different level of spatial resolution; Generating a pair of feature maps from each resolution level of the pyramid representation, and estimating a coarse interpolated motion vector field IMVF and an occlusion map based on each pair of feature maps; Registering a pair of feature maps of the same resolution level according to the coarse IMVF and the occlusion map by warping each feature map in the pair of feature maps to align with the pixel position of the coarse IMVF, and outputting a registered previous feature map and a registered next feature map; Correcting the coarse IMVF using the registered previous feature map and the registered next feature map to generate a correction to the coarse IMVF; adding the correction to the coarse IMVF to the coarse IMVF to generate a refined IMVF; as well as A synthesized intermediate frame is generated based on the pair of consecutive input frames, the refined IMVF, and the occlusion map.

9. The method according to claim 8, wherein: Generating a pair of feature maps includes: generating a pair of feature maps at each spatial resolution level for each pair of input frames in the plurality of pairs of input frames, wherein each pair of feature maps has a spatial resolution that is reduced relative to a resolution of the pair of input frames; cascading the feature maps at each resolution level and processing the cascaded feature maps to generate a new set of feature maps, the new set of feature maps having a reduced spatial resolution relative to a resolution of the pair of consecutive input frames; merging the new set of feature maps for all spatial resolution levels into a single merged feature map by performing a weighted average on the new set of feature maps for each resolution level, wherein the weights for the weighted average are learned in a training phase and are different for each pixel; and Horizontal and vertical components of the coarse IMVF and an occlusion map are estimated for each block in the single merged feature map, wherein the occlusion map includes per-pixel weights for performing a weighted average between the previous image and the next image.

10. The method according to claim 9, wherein: Estimating the horizontal and vertical components of the rough IMVF includes: Generate a horizontal probability map and a vertical probability map, wherein each probability map has S probability values ​​for each pixel, wherein each probability value represents the probability that the motion vector is one of the S displacement values ​​for the pixel, A first order moment of the probability value for each pixel is calculated to determine a desired horizontal component and a vertical component for each pixel, wherein a pair consisting of the desired horizontal component and the vertical component for each pixel comprises the correction for the coarse IMVF.

11. The method according to claim 8, wherein: The rough IMVF includes two 2D maps of horizontal and vertical shifts from the rough IMVF to the next image, Wherein, registering the feature map pair comprises: using the opposite values ​​of these shifts to warp the features from the previous image to the position of the coarse IMVF; and using the horizontal shift and the vertical shift as is to warp the input features from the next image to the position of the coarse IMVF.

12. The method according to claim 11, wherein: The registration feature map pair also includes: When the previous feature map of the registration has an occluded area, replacing the occluded area in the previous feature map of the registration with a corresponding area in the next feature map of the registration, and When the next feature map of the registration has an occluded area, the occluded area in the next feature map of the registration is replaced by a corresponding area in the previous feature map of the registration.

13. The method according to claim 8, wherein: Correcting the rough IMVF includes: cascading features of the registered previous feature map and the registered next feature map, and processing the cascaded feature maps to generate a new set of feature maps, the new set of feature maps having a spatial resolution resized relative to the full resolution of the previous image and the next image; and Corrections to the horizontal and vertical components of the coarse IMVF are estimated for each block in the coarse IMVF to generate corrections to the coarse IMVF.

14. The method according to claim 8, wherein: The intermediate frames generated by the synthesis include: warping the pair of consecutive input frames according to the refined IMVF; and A weighted blending is performed on the warped pair of images using occlusion weights from the occlusion map to generate the synthesized intermediate frame.

15. The method according to claim 14, further comprising: Blockiness artifacts are removed from motion boundaries by performing a weighted average of interpolated pixels in the synthesized intermediate frame using estimates from neighboring blocks.

16. A computer-readable non-transitory program storage device tangibly embodying a program of instructions executable by the computer to perform method steps for synthesizing intermediate non-existing frames from input frame pairs in a given video using a convolutional neural network (CNN), the method steps comprising the steps of: receiving a pyramid representation of a pair of consecutive input frames, wherein the pair of consecutive input frames comprises a previous image and a next image, wherein the pyramid representation comprises a plurality of pairs of input frames, each pair of input frames being at a different level of spatial resolution; Generating a pair of feature maps from each resolution level of the pyramid representation, and estimating a coarse interpolated motion vector field IMVF and an occlusion map based on each pair of feature maps; Registering a pair of feature maps of the same resolution level according to the coarse IMVF and the occlusion map by warping each feature map in the pair of feature maps to align with the pixel position of the coarse IMVF, and outputting a registered previous feature map and a registered next feature map; Correcting the coarse IMVF using the registered previous feature map and the registered next feature map to generate a correction to the coarse IMVF; adding the correction to the coarse IMVF to the coarse IMVF to generate a refined IMVF; as well as A synthesized intermediate frame is generated based on the pair of consecutive input frames, the refined IMVF, and the occlusion map.

17. The computer readable non-transitory program storage device of claim 16, wherein generating a pair of feature maps comprises: generating a pair of feature maps at each spatial resolution level for each pair of input frames in the plurality of pairs of input frames, wherein each pair of feature maps has a spatial resolution that is reduced relative to a resolution of the pair of input frames; cascading the feature maps at each resolution level and processing the cascaded feature maps to generate a new set of feature maps, the new set of feature maps having a reduced spatial resolution relative to a resolution of the pair of consecutive input frames; merging the new set of feature maps for all spatial resolution levels into a single merged feature map by performing a weighted average on the new set of feature maps for each resolution level, wherein the weights used for the weighted average are learned in a training phase and are different for each pixel; as well as estimating, for each block in the single merged feature map, horizontal and vertical components of the coarse IMVF and an occlusion map, wherein the occlusion map comprises per-pixel weights for performing a weighted average between the previous image and the next image, Wherein, estimating the horizontal component and the vertical component of the rough IMVF comprises: Generate a horizontal probability map and a vertical probability map, wherein each probability map has S probability values ​​for each pixel, wherein each probability value represents the probability that the motion vector is one of the S displacement values ​​for the pixel, A first order moment of the probability value for each pixel is calculated to determine a desired horizontal component and a vertical component for each pixel, wherein a pair consisting of the desired horizontal component and the vertical component for each pixel comprises the correction for the coarse IMVF.

18. The computer-readable non-transitory program storage device of claim 16, wherein the coarse IMVF comprises two 2D maps of horizontal and vertical shifts pointing from the coarse IMVF to the next image, in, Registering the feature map pair comprises: warping the features from the previous image to the position of the coarse IMVF using the opposite values ​​of these shifts, and warping the input features from the next image to the position of the coarse IMVF using the horizontal shift and the vertical shift as is, When the previous feature map of the registration has an occluded area, replacing the occluded area in the previous feature map of the registration with a corresponding area in the next feature map of the registration, and When the next feature map of the registration has an occluded area, the occluded area in the next feature map of the registration is replaced by a corresponding area in the previous feature map of the registration.

19. The computer-readable non-transitory program storage device of claim 16, wherein: Correcting the rough IMVF includes: cascading features of the registered previous feature map and the registered next feature map, and processing the cascaded feature maps to generate a new set of feature maps, the new set of feature maps having a spatial resolution resized relative to the full resolution of the previous image and the next image; and Corrections to the horizontal and vertical components of the coarse IMVF are estimated for each block in the coarse IMVF to generate corrections to the coarse IMVF.

20. The computer-readable non-transitory program storage device of claim 16, wherein: The intermediate frames generated by the synthesis include: warping the pair of consecutive input frames according to the refined IMVF; performing weighted fusion on the warped pair of images using occlusion weights from the occlusion map to generate the synthesized intermediate frame; and Blockiness artifacts are removed from motion boundaries by performing a weighted average of interpolated pixels in the synthesized intermediate frame using estimates from neighboring blocks.

Images