Method for frame interpolation and related apparatus

Abstract

A method for frame interpolation and related apparatus are provided in embodiments of the present disclosure. The method includes: obtaining a previous frame and a current frame of a video stream; obtaining one or more intermediate frames based on the previous frame and the current frame, and obtaining one or more confidence values respectively corresponding to the one or more intermediate frames; and in a case that the one or more confidence values satisfy a pre-defined condition, interpolating the one or more intermediate frames between the previous frame and the current frame. When the confidence value meets the pre-defined condition, e.g., the confidence value is greater than or equal to a threshold, the intermediate frame can be interpolated between the previous frame and the current frame. The decision is made to get a tradeoff between smoothness and level of ghosting, in other words, ghost-prevention may be implemented.

Description

METHOD FOR FRAME INTERPOLATION AND RELATED APPARATUSTECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to a method for frame interpolation and related apparatus.BACKGROUND

[0002] A series of images being played continuously can implement a smooth and continuous visual effect, such images can be referred to as a video, and each image may be referred to as a frame.

[0003] Frame interpolation technology reconstructs one or more frames between two consequent frames. It can be used in Cloud Media Services to reduce resource cost of real-time video rendering in case of original rendering operation is heavy compared to frame interpolation.

[0004] This background information is provided to reveal information believed by the applicant to be of possible relevance to the present disclosure. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art against the present disclosure.SUMMARY

[0005] In a first aspect, a method for frame interpolation is provided by the present disclosure, and the method includes: obtaining a previous frame and a current frame of a video stream; obtaining one or more intermediate frames based on the previous frame and the current frame, and obtaining one or more confidence values respectively corresponding to the one or more intermediate frames; and in a case that the one or more confidence values satisfy a pre-defined condition, interpolating the one or more intermediate frames between the previous frame and the current frame.

[0006] The confidence value corresponding to the intermediate frame is introduced in determining whether the intermediate frame can be used for frame interpolation, and when the confidence value meets the pre-definedcondition, e.g., the confidence value is greater than or equal to a threshold, the intermediate frame can be interpolated between the previous frame and the current frame. The decision on whether to interpolate the intermediate frame(s) can be made to prevent or reduce ghosting caused by frame interpolation and get a tradeoff between smoothness and level of ghosting.

[0007] The quantity of intermediate frames may depend on an original frame rate of the video stream and a target frame rate. For example, in a case that the target frame rate is twice the original frame rate, one intermediate frame may be obtained for a pair of adjacent frames, and in a case that the target frame rate is three times the original frame rate, two intermediate frames may be obtained for a pair of adjacent frames.

[0008] The threshold for the confidence value may be pre-configured, e.g., by a user, and / or adjusted during frame interpolation. The threshold may have a default value, which may be used in a case that the threshold is not configured with a specific value by a user at the beginning of frame interpolation.

[0009] In a possible implementation of the first aspect, the obtaining the one or more intermediate frames based on the previous frame and the current frame includes: obtaining a first feature pyramid including a set of first feature maps with different resolutions for the previous frame and a second feature pyramid including a set of second feature maps with different resolutions for the current frame, where a first feature map and a second feature map for a same level have same resolution, a first feature map for a top level has a resolution of the top level and a first feature map for a bottom level has a resolution of the bottom level; and obtaining, according to the first feature pyramid and the second feature pyramid, the one or more intermediate frames, where intermediate feature maps with different resolutions of the one or more intermediate frames are built based on the first feature pyramid and the second feature pyramid from the bottom level to the top level.

[0010] A feature pyramid including feature maps with different resolutions may be obtained for each of the previous frame and the current frame and then used for obtaining the intermediate frame(s). The intermediate frame(s) may be obtained by using a frame interpolation model, where the intermediate frame(s) can be an output of the model with the previous and the current frames as inputs. The model may include a pyramidal encoder and a chain of decoders, where the chain of decoders are arranged in a sequence of the bottom level to the top level. The pyramidal encoder may produce the first and the second feature maps with different resolutions for the previous and the current frames respectively, and the first and the second feature maps pass through the chain of decoders to generate theintermediate frame(s). With such data flows, features of different resolutions are considered, and the obtained intermediate frame(s) can be reliable and accurate, in other words, in good image quality. Combing such intermediate frame with the confidence value to perform frame interpolation, a better ghost-prevention may be achieved.

[0011] In a possible implementation of the first aspect, the obtaining, according to the first feature pyramid and the second feature pyramid, the one or more intermediate frames includes: obtaining, according to a first feature map and a second feature map for the bottom level, one or more first optical flows with a first resolution, one or more second optical flows with the first resolution, one or more intermediate feature maps with the first resolution, one or more first flow selection mask with the first resolution, one or more second flow selection mask with the first resolution, and one or more feature selection masks with the first resolution, where the first resolution is a resolution of a level which is higher than and next to the bottom level, where each of the one or more first optical flows with the first resolution is a forward optical flow corresponding to a respective one in the one or more intermediate frames, and each of the one or more second optical flows with the first resolution is a backward optical flow corresponding to a respective one in the one or more intermediate frames; where the one or more first optical flows with the first resolution, the one or more second optical flows with the first resolution, the one or more intermediate feature maps with the first resolution, the one or more first flow selection mask with the first resolution, the one or more second flow selection mask with the first resolution, and the one or more feature selection masks with the first resolution are used for producing the one or more intermediate frames.

[0012] The chain of decoders may include a bottom-level decoder with the first and the second feature maps as inputs. The bottom-level decoder may produce forward and backward optical flows (i.e., first and second optical flows), one or more intermediate feature maps, and selection masks which may include one or more feature selection masks, one or more first selection masks and one or more second flow selection masks. The inputs of the bottomlevel decoder have the resolution of the bottom level (i.e., the lowest resolution), while outputs of the bottom-level decoder have a resolution of a level which is higher than and next to the bottom level (i.e., a next higher resolution of the lowest resolution), therefore the outputs of the bottom-level decoder can be compatible with the first and the second feature maps with the next higher resolution of the lowest resolution in terms of spatial dimensions, and the resolution is the same as that of inputs of a next higher-level decoder. The selection masks regulate how much the forward and backward optical flows and the intermediate feature map(s) decoded by the bottom-level decoder are trusted. The data flows of the bottom-level decoder form a part to produce the intermediate frame(s) more reliablyand more accurately.

[0013] In a possible implementation of the first aspect, the obtaining, according to the first feature pyramid and the second feature pyramid, the one or more intermediate frames includes: obtaining one or more first warped feature maps with a middle resolution by warping a first feature map with the middle resolution using one or more first optical flows with the middle resolution, where the middle resolution is higher than the resolution of the bottom level and lower than the resolution of the top level; obtaining one or more second warped feature maps with the middle resolution by warping the second feature map with the middle resolution using one or more second optical flows with the middle resolution; and obtaining one or more first optical flow residuals with a second resolution, one or more second optical flow residuals with the second resolution, one or more intermediate feature maps with the second resolution, one or more first flow selection masks with the second resolution, one or more second flow selection masks with the second resolution, and one or more feature selection masks with the second resolution according to one or more intermediate feature maps with the middle resolution, the one or more first warped feature maps with the middle resolution, the one or more second warped feature maps with the middle resolution, one or more first flow selection masks with the middle resolution, one or more second flow selection masks with the middle resolution, and one or more feature selection masks with the middle resolution, where the second resolution is a resolution of a level which is higher than and next to a level of the middle resolution; where the one or more first optical flow residuals with the second resolution, the one or more second optical flow residuals with the second resolution, the one or more intermediate feature maps with the second resolution, the one or more first flow selection masks with the second resolution, the one or more second flow selection masks with the second resolution, and the one or more feature selection masks with the second resolution are used for producing the one or more intermediate frames.

[0014] The chain of decoders may include one or more middle-level decoders. A middle-level decoder takes warped feature maps of its level, the intermediate feature map(s) produced by the former decoder, and the selection masks produced by the former decoder. The middle-level decoder produces optical flow residuals, an intermediate feature map and selection masks. Outputs of the middle-level decoder have a next higher resolution than inputs of the middle-level decoder. The selection masks regulate how much other outputs decoded by the middle-level decoder are trusted. The data flows of the middle-level decoder form a part to produce the intermediate frame more reliably and more accurately.

[0015] In a possible implementation of the first aspect, the obtaining, according to the first feature pyramid and the second feature pyramid, the one or more intermediate frames includes: obtaining one or more first warped feature maps with the resolution of the top level by warping the first feature map with the resolution of the top level using one or more first optical flows with the resolution of the top level; obtaining one or more second warped feature maps with the resolution of the top level by warping the second feature map with the resolution of the top level using one or more second optical flows with the resolution of the top level; and obtaining the one or more intermediate frames according to one or more intermediate feature maps with the resolution of the top level, the one or more first warped feature maps with the resolution of the top level, the one or more second warped feature maps with the resolution of the top level, one or more first flow selection masks with the resolution of the top level, one or more second flow selection masks with the resolution of the top level, and one or more feature selection masks with the resolution of the top level.

[0016] The chain of decoders may include a top-level decoder. The top-level decoder takes warped feature maps with the resolution of the top level (i.e., the highest resolution), the intermediate feature map(s) produced by the former decoder, and the selection masks produced by the former decoder. The top-level decoder produces one or more intermediate frames as the output of the model. With such dataflows, the intermediate frame(s) for the pervious frame and the current frame can be obtained.

[0017] In a possible implementation of the first aspect, the method further includes: obtaining one or more first optical flows with a third resolution by up-scaling one or more first optical flows with a fourth resolution to obtain one or more up-scaled first optical flows with the third resolution; and correcting the one or more up-scaled first optical flows with the third resolution by one or more first optical flow residuals with the third resolution; and obtaining one or more second optical flows with the third resolution by up-scaling one or more second optical flows with the fourth resolution to obtain one or more up- scaled second optical flows with the third resolution; and correcting the one or more up-scaled second optical flows with the third resolution by the one or more second optical flow residuals with the third resolution;where the third resolution is a resolution of a level higher than and next to a level of the fourth resolution, and the fourth resolution is higher than the resolution of the bottom level.

[0018] Regarding a decoder producing an optical flow residual, optical flow correction is performed to obtain a corresponding optical flow as an input of a next decoder. This can improve accuracy of optical flows inputting into the next decoder, thus, and the intermediate frame(s) obtained through the chain of decoders may be more accurate.

[0019] In a possible implementation of the first aspect, the obtaining the one or more confidence values respectively corresponding to the one or more intermediate frames includes: obtaining the one or more confidence values respectively corresponding to the one or more intermediate frames based on one or more first warped feature maps with a fifth resolution and one or more second warped feature maps with a fifth resolution, where the fifth resolution is greater than or equal to a first resolution threshold.

[0020] Whether the obtained intermediate frame(s) can be inserted between the pervious frame and the current frame may depend on the confidence value for each intermediate frame. A confidence value corresponding to an intermediate frame can be produced by a confidence estimator with two warped feature maps of a specific level as inputs. The confidence estimator may take a difference between two warped feature maps associated with optical flows, normalize that difference to feature magnitudes, and aggregate the feature magnitudes to obtain a confidence value. The confidence value is used for determining whether a corresponding intermediate frame can be interpolated. With the aforementioned obtaining manner, the confidence value is reliable, thus, the determination on whether to insert the intermediate frame(s) will also be reliable, thereby preventing ghosting caused by frame interpolation.{0021] In a possible implementation of the first aspect, the first resolution threshold is associated with a size of details to keep in the video stream.

[0022] This threshold is associated with a size of details to keep / track in the video stream and can be configurable to ensure that desired details can be tracked. In general, when a higher first resolution threshold is employed, smaller details may be kept or tracked, but more computational overhead may be caused, while when a lower first resolution threshold is employed, details to keep or track may be larger but the introduced computational overhead may be less.

[0023] In a possible implementation of the first aspect, the obtaining the first feature pyramid including the set of first feature maps with different resolutions for the previous frame includes: obtaining the first feature map for the top level based on an image of the previous frame with an original resolution of the video stream;obtaining a first feature map for a level next to the top level, where the first feature map for the next level of the top level includes a first part obtained based on a down-sampled image of the previous frame for the next level of the top level, and a second part obtained based on the first feature map for the top level, where the down-sampled image of the previous frame for the next level of the top level is obtained by down sampling the image for the previous frame with the original resolution by a pre-defined factor; and obtaining a first feature map for each level of one or more levels lower than the level next to the top level, where the first feature map for the each level includes a first part obtained based on a down-sampled image of the previous frame for the each level, and a second part obtained based on a first part of a first feature map for a level upper than and next to the each level, where the down-sampled image of the previous frame for the each level is obtained by down sampling the down-sampled image of the previous frame for the level upper than and next to the each level by a predefined factor.

[0024] The first feature maps of respective levels can be obtained for the previous frame by means of the steps described above, and similarly the second feature maps of respective levels can be obtained for the current frame. The feature map for each of the bottom level and the one or more middle levels includes two parts, where a first part is obtained based on down sampling, and a second part is obtained based on a first part of a feature map for a level upper than and next to the specific level without an additional step of down sampling. Since the feature maps of the bottom and middle levels each include features obtained from some upper-level features in addition to features obtained from the images with the resolutions of respective levels, the features of the bottom and middle levels may be more abundant and accurate, and the feature maps of these levels and thus the feature pyramid can be generated more accurately and efficiently.

[0025] In a possible implementation of the first aspect, a quantity of levels of the first feature pyramid is based on an original resolution of the video stream and a second resolution threshold, where a resolution of the first feature map for the bottom level is lower than the second resolution threshold and a resolution of a first feature map for a level upper than and next to the bottom level is higher than the second resolution threshold.

[0026] The quantity of levels of the feature pyramid may be determined as described above, both the first feature pyramid and the second feature pyramid have the same quantity of levels. The feature map for the top level may be obtained by down sampling an image of the previous / current frame with the original resolution. The pyramidal encoder adds a new level of the feature pyramid by down sampling a down-sampled image while a size of the down- sampled image is greater than the second resolution threshold. By setting the aforementioned determination principle,the quantity of levels or the resolution of the bottom level of the feature pyramid can be determined, and it may be beneficial to achieve a tradeoff between computational overhead and accuracy in terms of the frame interpolation solution.10027] In a possible implementation of the first aspect, the second resolution threshold is based a motion level between the previous frame and the current frame.

[0028] The second resolution threshold intends to handle proper level of motion for the whole image while using limited motion compensation at each level of pyramid.

[0029] In a possible implementation of the first aspect, a first feature map and a second feature map for a same level have a same quantity of features, and a quantity of features for the first feature map of the top level is smaller than a quantity of features for a first feature map of a lower level.

[0030] The quantity of features for the first / second feature map of each level is associated with the aforementioned data flows of the pyramidal encoder, the first / second feature map for the top level has only one part, while the first / second feature map for other levels has two parts.

[0031] In a possible implementation of the first aspect, a quantity of the one or more intermediate frames is based on an original frame rate of the video stream, and a target frame rate of the video stream.

[0032] The intermediate frame quantity can be determined based on the original frame rate and the target frame rate, for example, may be a ratio of the target frame rate and the original rate, which can be changed according to actual requirements.

[0033] In a possible implementation of the first aspect, a quantity of the one or more intermediate frames is based on a first operation of a user, where the first operation indicates a target frame rate of the video stream or the quantity of the one or more intermediate frames.

[0034] The intermediate frame quantity can be determined based on the user operation, which improves flexibility of the design. Auser may directly define an interpolation factor (the intermediate frame quantity) by which the original / input frame rate should be increased. The user may also input the target frame rate, and the intermediate frame quantity can be obtained based on the input frame rate and the target frame rate, for example, by dividing the target frame rate by the input frame rate and then subtracting one from the quotient. Specific implementation manners can be set according to actual requirements.

[0035] In a possible implementation of the first aspect, a color space in which to perform the frame interpolation is determined based on a characteristic of a scene of the video stream or a second operation of a user, where thesecond operation indicates the color space in which to perform the frame interpolation.

[0036] The color space in which to perform the frame interpolation may be an RGB color space or a YUV color space. A user can directly choose the color space, or the color space can be determined based on the scene of the video stream, which improves flexibility of the design. For example, for natural looking scenes with complex textures and geometry, YUV color space may be used; for synthetic looking scenes with simplified graphics, RGB color space may be used.

[0037] In a possible implementation of the first aspect, the method further includes: in a case that the one or more confidence values do not satisfy the pre-defined condition, repeating the previous frame or the current frame one or more times.

[0038] When a confidence value corresponding to an intermediate frame does not satisfy the pre-defined condition, e.g., the confidence value is smaller than a threshold, the intermediate frame is ignored, and the previous frame or the current frame is repeated. In this manner, ghosting may be avoided by ignoring the intermediate frame with a low confidence value, thereby achieving ghost-prevention.

[0039] In a possible implementation of the first aspect, the frame interpolation is performed in an RGB color space, each of the set of first feature maps and the set of second feature maps includes red channel information, blue channel information and green channel information, and one or more first optical flows with a sixth resolution and one or more second optical flows with the sixth resolution are obtained based on the red channel information, the blue channel information and the green channel information in one or more first feature maps and one or more second feature maps with a resolution lower than the sixth resolution, where the sixth resolution is higher than the resolution of the bottom level.

[0040] When the frame interpolation is performed in an RGB color space, an RGB model may be used for generating the intermediate frame(s) and the confidence value(s) corresponding to the intermediate frame(s). The encoding and decoding processing for the RGB model are performed based on three-channel information. The RGB model can be used for synthetic looking scenes with simplified graphics.

[0041] In a possible implementation of the first aspect, the frame interpolation is performed in a YUV color space, each of the set of first feature maps and the set of second feature maps includes gray channel information, chroma-blue channel information and chroma-red channel information, and one or more first optical flows with a seventh resolution and one or more second optical flows with the seventh resolution are obtained based on the gray channel information of one or more first feature maps and one ormore second feature maps with a resolution of the gray channel information lower than the seventh resolution.

[0042] When the frame interpolation is performed in a YUV color space, a YUV model may be used for generating the intermediate frame(s) and the confidence value(s) corresponding to the intermediate frame(s). The encoding and decoding processing for the YUV model are performed based on single-channel information, i.e., the gray channel information, and motion estimation results are applied to chromatic channels. The YUV model can be used for natural looking scenes with complex textures and geometry. Compared with the RGB model with the same interpolation mode, the YUV model may be more efficient in terms of data processing.

[0043] In a possible implementation of the first aspect, the one or more intermediate frames are obtained through a frame interpolation model, and the frame interpolation model is determined from multiple frame interpolation models.10044] The multiple frame interpolation models can be selected according to actual requirements.

[0045] In a possible implementation of the first aspect, the multiple frame interpolation models include a first frame interpolation model that doubles a frame rate of the video stream and performs frame interpolation in an RGB color space, or a second frame interpolation model that triples a frame rate of the video stream and performs frame interpolation in an RGB color space, or a third frame interpolation model that doubles a frame rate of the video stream and perform frame interpolation in a YUV color space, or a fourth frame interpolation model that triples a frame rate of the video stream and perform frame interpolation in a YUV color space.|0046] The multiple frame interpolation models may be used for implementing different frame rate increase in different color spaces, which provides a diversity for the frame interpolation solution.

[0047] In a possible implementation of the first aspect, the frame interpolation model is determined according to at least one of an original frame rate of the video stream, a target frame rate of the video stream, or a characteristic of scenes of the video stream.

[0048] A frame interpolation model may be determined by taking into account: an interpolation mode and a color space. The interpolation mode (e.g., x2 interpolation or x3 interpolation) of the frame interpolation model may be determined based on the original frame rate and the target frame rate. A color space of the frame interpolation model may be determined based on the characteristic of scenes of the video stream.

[0049] In a possible implementation of the first aspect, the frame interpolation model is determined based on a third operation of a user, where the third operation indicates the frame interpolation model selected by the user.

[0050] A user may directly specify the frame interpolation model to be used, and user control may improve userexperience.

[0051] In a second aspect, an apparatus for frame interpolation is provided by the present disclosure, and the apparatus includes a first obtaining module, configured to obtain a previous frame and a current frame of a video stream; a second obtaining module, configured to obtain one or more intermediate frames based on the previous frame and the current frame, and obtain one or more confidence values respectively corresponding to the one or more intermediate frames; and an interpolating module, configured to in a case that the one or more confidence values satisfy a predefined condition, interpolate the one or more intermediate frames between the previous frame and the current frame.

[0052] In a possible implementation of the second aspect, the second obtaining module includes: a pyramidal encoder, configured to obtain a first feature pyramid including a set of first feature maps with different resolutions for the previous frame and a second feature pyramid including a set of second feature maps with different resolutions for the current frame, where a first feature map and a second feature map for a same level have same resolution, a first feature map for a top level has a resolution of the top level and a first feature map for a bottom level has a resolution of the bottom level; and a chain of decoders, configured to obtain, according to the first feature pyramid and the second feature pyramid, the one or more intermediate frames, where intermediate feature maps with different resolutions of the one or more intermediate frames are built based on the first feature pyramid and the second feature pyramid from the bottom level to the top level.

[0053] In a possible implementation of the second aspect, the chain of decoders includes a bottom-level decoder, and the bottom-level decoder is configured to: obtain, according to a first feature map and a second feature map with the resolution of the bottom level, one or more first optical flows with a first resolution, one or more second optical flows with the first resolution, one or more intermediate feature maps with the first resolution, one or more first flow selection mask with the first resolution, one or more second flow selection mask with the first resolution, and one or more feature selection masks with the first resolution, where the first resolution is a resolution of a level which is higher than and next to the bottom level, where each of the one or more first optical flows with the first resolution is a forward optical flow corresponding to a respective one in the one or more intermediate frames, and each of the one or more second optical flows with the first resolution is a backward optical flow corresponding to a respective one in the one or moreintermediate frames; where the one or more first optical flows with the first resolution, the one or more second optical flows with the first resolution, the one or more intermediate feature maps with the first resolution, the one or more first flow selection mask with the first resolution, the one or more second flow selection mask with the first resolution, and the one or more feature selection masks with the first resolution are used for producing the one or more intermediate frames.

[0054] In a possible implementation of the second aspect, the bottom-level decoder includes: a first convolution layer, configured to: obtain, according to a first input tensor, a first extended tensor, where the first input tensor is obtained according to the first feature map and the second feature map with the resolution of the bottom level; where a quantity of channels of the first extended tensor is greater than a quantity of channels of the first input tensor.

[0055] In a possible implementation of the second aspect, the bottom-level decoder further includes: a first shuffle layer module, configured to up scale tensors split from a first aggregated output tensor to produce up-scaled tensors with the first resolution; where the first aggregated output tensor is obtained according to the first extended tensor; where the up-scaled tensors with the first resolution are used for producing the one or more first optical flows with the first resolution, the one or more second optical flows with the first resolution, the one or more intermediate feature maps with the first resolution, the one or more first flow selection masks with the first resolution, the one or more second flow selection masks with the first resolution, and the one or more feature selection masks with the first resolution.

[0056] In a possible implementation of the second aspect, the bottom-level decoder further includes: a first tangent layer module, configured to apply a first hyperbolic tangent function to produce, according to one or more first up-scaled tensors of the up-scaled tensors with the first resolution, the one or more first optical flows with the first resolution, and produce, according to one or more second up-scaled tensors of the up-scaled tensors with the first resolution, the one or more second optical flows with the first resolution; and / or a first activation layer, configured to apply a first softmax function to produce, according to one or more third up-scaled tensors of the up-scaled tensors with the first resolution, the one or more first flow selection masks with the first resolution, the one or more second flow selection masks with the first resolution, and the one or morefeature selection masks with the first resolution.

[0057] In a possible implementation of the second aspect, the chain of decoders includes one or more middlelevel decoders, and the one or more middle-level decoders are configured to: obtain one or more first warped feature maps with a middle resolution by warping a first feature map with the middle resolution using one or more first optical flows with the middle resolution, where the middle resolution is higher than the resolution of the bottom level and lower than the resolution of the top level; obtain one or more second warped feature maps with the middle resolution by warping the second feature map with the middle resolution using one or more second optical flows with the middle resolution; and obtain one or more first optical flow residuals with a second resolution, one or more second optical flow residuals with the second resolution, one or more intermediate feature maps with the second resolution, one or more first flow selection masks with the second resolution, one or more second flow selection masks with the second resolution, and one or more feature selection masks with the second resolution according to one or more intermediate feature maps with the middle resolution, the one or more first warped feature maps with the middle resolution, the one or more second warped feature maps with the middle resolution, one or more first flow selection masks with the middle resolution, one or more second flow selection masks with the middle resolution, and one or more feature selection masks with the middle resolution, where the second resolution is a resolution of a level higher than and next to a level of the middle resolution; where the one or more first optical flow residuals with the second resolution, the one or more second optical flow residuals with the second resolution, the one or more intermediate feature maps with the second resolution, the one or more first flow selection masks with the second resolution, the one or more second flow selection masks with the second resolution, and the one or more feature selection masks with the second resolution are used for producing the one or more intermediate frames.[0058| In a possible implementation of the second aspect, each of the one or more middle-level decoders includes: a second convolution layer, configured to: obtain, according to a second input tensor, a second extended tensor, where the second input tensor is obtained according to the one or more intermediate feature maps with the middle resolution, the one or more first warped feature maps with the middle resolution, the one or more second warped feature maps with the middle resolution, the one or more first flow selection masks with the middle resolution, the one or more second flow selection masks with the middle resolution, and the one or more feature selection maskswith the middle resolution; where a quantity of channels of the second extended tensor is greater than a quantity of channels of the second input tensor.

[0059] In a possible implementation of the second aspect, each of the one or more middle-level decoders further includes: a second shuffle layer module, configured to up scale tensors split from a second aggregated output tensor to produce up-scaled tensors with the second resolution; where the second aggregated output tensor is obtained according to the second extended tensor; where the up-scaled tensors with the second resolution are used for producing the one or more intermediate feature maps with the second resolution, the one or more first optical flow residuals with the second resolution, the one or more second optical flow residuals with the second resolution, the one or more first flow selection masks with the second resolution, the one or more second flow selection masks with the second resolution, and the one or more feature selection masks with the second resolution.

[0060] In a possible implementation of the second aspect, each of the one or more middle-level decoders further includes: a second tangent layer module, configured to apply a second hyperbolic tangent function to produce, according to one or more first up-scaled tensors of the up-scaled tensors with the second resolution, the one or more first optical flow residuals with the second resolution, and produce, according to one or more second up-scaled tensors of the up-scaled tensors with the second resolution, the one or more second optical flow residuals with the second resolution; and / or a second activation layer, configured to apply a second softmax function to produce, according to one or more third up-scaled tensors of the up-scaled tensors with the second resolution, the one or more first flow selection masks with the second resolution, the one or more second flow selection masks with the second resolution, and the one or more feature selection masks with the second resolution.

[0061] In a possible implementation of the second aspect, the chain of decoders includes a top-level decoder, and the top-level decoder is configured to: obtain one or more first warped feature maps with the resolution of the top level by warping the first feature map with the resolution of the top level using one or more first optical flows with the resolution of the top level;obtain one or more second warped feature maps with the resolution of the top level by warping the second feature map with the resolution of the top level using one or more second optical flows with the resolution of the top level; and obtain the one or more intermediate frames according to one or more intermediate feature maps with the resolution of the top level, the one or more first warped feature maps with the resolution of the top level, the one or more second warped feature maps with the resolution of the top level, one or more first flow selection masks with the resolution of the top level, one or more second flow selection masks with the resolution of the top level, and one or more feature selection masks with the resolution of the top level.

[0062] In a possible implementation of the second aspect, the apparatus further includes an optical flow correcting module, and the optical flow correcting module is configured to: obtain one or more first optical flows with a third resolution by up-scaling one or more first optical flows with a fourth resolution to obtain one or more up-scaled first optical flows with the third resolution; and correcting the one or more up-scaled first optical flows with the third resolution by one or more first optical flow residuals with the third resolution; and obtain one or more second optical flows with the third resolution by up-scaling one or more second optical flows with the fourth resolution to obtain one or more up- scaled second optical flows with the third resolution; and correcting the one or more up-scaled second optical flows with the third resolution by the one or more second optical flow residuals with the third resolution; where the third resolution is a resolution of a level higher than and next to a level of the fourth resolution, and the fourth resolution is higher than the resolution of the bottom level.

[0063] In a possible implementation of the second aspect, the second obtaining module includes a confidence estimator, and the confidence estimator is configured to: obtain the one or more confidence values respectively corresponding to the one or more intermediate frames based on one or more first warped feature maps with a fifth resolution and one or more second warped feature maps with a fifth resolution, where the fifth resolution is greater than or equal to a first resolution threshold.

[0064] In a possible implementation of the second aspect, the first resolution threshold is associated with a size of details to keep in the video stream

[0065] In a possible implementation of the second aspect, the pyramidal encoder is configured to: obtain the first feature map for the top level based on an image of the previous frame with an original resolution of the video stream; obtain a first feature map for a level next to the top level, where the first feature map for the level next to the top level includes a first part obtained based on a down-sampled image of the previous frame for the level next to the top level, and a second part obtained based on the first feature map for the top level, where the down-sampled image of the previous frame for the level next to the top level is obtained by down sampling the image for the previous frame with the original resolution by a pre-defined factor; and obtain a first feature map for each level of one or more levels lower than the level next to the top level, where the first feature map for the each level includes a first part obtained based on a down-sampled image of the previous frame for the each level, and a second part obtained based on a first part of a first feature map for a level upper than and next to the each level, where the down-sampled image of the previous frame for the each level is obtained by down sampling the down-sampled image of the previous frame for the level upper than and next to the each level by a pre-defined factor.

[0066] In a possible implementation of the second aspect, a quantity of levels of the first feature pyramid is based on an original resolution of the video stream and a second resolution threshold, where a resolution of the first feature map for the bottom level is lower than the second resolution threshold and a resolution of a first feature map for a level upper than and next to the bottom level is higher than the second resolution threshold.

[0067] In a possible implementation of the second aspect, the second resolution threshold is based a motion level between the previous frame and the current frame.

[0068] In a possible implementation of the second aspect, a first feature map and a second feature map for a same level have a same quantity of features, and a quantity of features for the first feature map of the top level is smaller than a quantity of features for a first feature map of a lower level.

[0069] In a possible implementation of the second aspect, a quantity of the one or more intermediate frames is based on an original frame rate of the video stream, and a target frame rate of the video stream.

[0070] In a possible implementation of the second aspect, a quantity of the one or more intermediate frames is based on a first operation of a user, where the first operation indicates a target frame rate of the video stream or the quantity of the one or more intermediate frames.

[0071] In a possible implementation of the second aspect, a color space in which to perform the frameinterpolation is determined based on a characteristic of a scene of the video stream or a second operation of a user, where the second operation indicates the color space in which to perform the frame interpolation.

[0072] In a possible implementation of the second aspect, the interpolating module is further configured to in a case that the one or more confidence values do not satisfy the pre-defined condition, repeat the previous frame or the current frame one or more times.

[0073] In a possible implementation of the second aspect, the frame interpolation is performed in an RGB color space, each of the set of first feature maps and the set of second feature maps includes red channel information, blue channel information and green channel information, and one or more first optical flows with a sixth resolution and one or more second optical flows with the sixth resolution are obtained based on the red channel information, the blue channel information and the green channel information in one or more first feature maps and one or more second feature maps with a resolution lower than the sixth resolution, where the sixth resolution is higher than the resolution of the bottom level.

[0074] In a possible implementation of the second aspect, the frame interpolation is performed in a YUV color space, each of the set of first feature maps and the set of second feature maps includes gray channel information, chroma-blue channel information and chroma-red channel information, and one or more first optical flows with a seventh resolution and one or more second optical flows with the seventh resolution are obtained based on the gray channel information of one or more first feature maps and one or more second feature maps with a resolution of the gray channel information lower than the seventh resolution.

[0075] In a possible implementation of the second aspect, the one or more intermediate frames are obtained through a frame interpolation model, where the frame interpolation model is determined from multiple frame interpolation models.

[0076] In a possible implementation of the second aspect, the multiple frame interpolation models comprise a first frame interpolation model that doubles a frame rate of the video stream and performs frame interpolation in an RGB color space, or a second frame interpolation model that triples a frame rate of the video stream and performs frame interpolation in an RGB color space, or a third frame interpolation model that doubles a frame rate of the video stream and perform frame interpolation in a YUV color space, or a fourth frame interpolation model that triples a frame rate of the video stream and perform frame interpolation in a YUV color space.

[0077] In a possible implementation of the second aspect, the frame interpolation model is determined according to at least one of an original frame rate of the video stream, a target frame rate of the video stream, or a characteristicof scenes of the video stream.

[0078] In a possible implementation of the second aspect, the frame interpolation model is determined based on a third operation of a user, where the third operation indicates the frame interpolation model selected by the user.

[0079] In a third aspect, an apparatus for frame interpolation is provided by the present disclosure, and the apparatus is configured to execute the method according to the first aspect or any possible implementation of the first aspect.

[0080] In a fourth aspect, a computing device cluster including at least one computing device is provided by the present disclosure, and the computing device includes at least one processor and a memory coupled with the processor, where the memoiy stores instructions which, when executed by the at least one processor, cause the computing device cluster to execute the method according to the first aspect or any possible implementation of the first aspect.

[0081] In a fifth aspect, a computer-readable medium is provided by the present disclosure, and the computer- readable medium stores computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute the method according to the first aspect or any possible implementation of the first aspect.

[0082] In a sixth aspect, a computer program product is provided by the present disclosure, and the computer program product includes computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute the method according to the first aspect or any possible implementation of the first aspect.

[0083] In a seventh aspect, a computer program is provided by the present disclosure, and the computer program includes computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute the method according to the first aspect or any possible implementation of the first aspect.

[0084] A method for frame interpolation and related apparatus are provided by the present disclosure. The confidence value corresponding to the intermediate frame is introduced in determining whether the intermediate frame can be used for frame interpolation, and when the confidence value meets the pre-defined condition, e.g., the confidence value is greater than or equal to a threshold, the intermediate frame can be interpolated between the previous frame and the cunent frame. The decision on whether to interpolate the intermediate frame(s) can be made to prevent or reduce ghosting caused by frame interpolation and thus get a tradeoff between smoothness and level of ghosting.BRIEF DESCRIPTION OF DRAWINGS

[0085] The accompanying drawings are used to provide a further understanding of the present disclosure, constitute a part of the specification, and are used to explain the present disclosure together with the following specific embodiments, but should not be construed as limiting the present disclosure.

[0086] FIG. 1 is a schematic illustration of optical flow according to one or more embodiments of the present disclosure.

[0087] FIG. 2A is a schematic illustration of an original pipeline in an example of virtual teacher according to one or more embodiments of the present disclosure.

[0088] FIG. 2B is a schematic illustration of a modified pipeline in an example of virtual teacher according to one or more embodiments of the present disclosure.

[0089] FIG. 3A is a schematic illustration of an original pipeline in an example of virtual assistant according to one or more embodiments of the present disclosure.

[0090] FIG. 3B is a schematic illustration of a modified pipeline in an example of virtual assistant according to one or more embodiments of the present disclosure.

[0091] FIGS. 4A and 4B are schematic illustrations of ghosting in prior art.

[0092] FIG. 4C is a schematic illustration of a relationship between motion level and frame rate in prior art.

[0093] FIG. 5 is a flowchart of a method for frame interpolation according to one or more embodiments of the present disclosure.

[0094] FIG. 6 is a schematic illustration of components for a cloud media service according to one or more embodiments of the present disclosure.

[0095] FIG. 7 is a block diagram of RGB X2 model according to one or more embodiments of the present disclosure.

[0096] FIG. 8 is a block diagram of RGB X3 model according to one or more embodiments of the present disclosure.

[0097] FIG. 9 is a block diagram of a pyramidal encoder in RGB X2 / X3 model according to one or more embodiments of the present disclosure.

[0098] FIG. 10 is a block diagram of the bottom decoder in RGB X3 model according to one or more embodiments of the present disclosure.

[0099] FIG. 11 is a block diagram of the bottom decoder in RGB X3 model according to one or moreembodiments of the present disclosure.

[0100] FIG. 12 is a block diagram of the middle and the top decoders in RGB X2 model according to one or more embodiments of the present disclosure.

[0101] FIG. 13 is a block diagram of the middle and the top decoders in RGB X3 model according to one or more embodiments of the present disclosure.

[0102] FIG. 14 is a block diagram of the image decoder in RGB X2 model according to one or more embodiments of the present disclosure.

[0103] FIG. 15 is a block diagram of the image decoder in RGB X3 model according to one or more embodiments of the present disclosure.

[0104] FIG. 16 is a block diagram of the confidence estimator in RGB X2 model according to one or more embodiments of the present disclosure.

[0105] FIG. 17 is a block diagram of an apparatus for frame interpolation according to one or more embodiments of the present disclosure.

[0106] FIG. 18 is a schematic structural diagram of a computing device according to one or more embodiments of the present disclosure.

[0107] FIG. 19 is a schematic diagram of an architecture of a computing device cluster according to one or more embodiments of the present disclosure.

[0108] FIG. 20 is a schematic diagram of a connection between computing devices over a network according to one or more embodiments of the present disclosure.DESCRIPTION OF EMBODIMENTS

[0109] In the following description, reference is made to the accompanying figures, which form part of the present disclosure, and which show, by way of illustration, specific aspects of embodiments of the present disclosure or specific aspects in which embodiments of the present disclosure may be used. It is understood that embodiments of the present disclosure may be used in other aspects and include structural or logical changes not depicted in the figures. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0110] The following introduces possible terminologies of the present disclosure.

[0111] Image / Frame - 2D array (matrix) of pixels. Each pixel is encoded by few numbers (color channels)that represent color information and intensity information.

[0112] Color Space - a way to represent (encode) color information of a single pixel or a group of pixels (macropixel).

[0113] RGB Color Space - a family of color spaces in which color information is represented by three color components: red, green and blue. This color space naturally fits the model of human color perception. Correspondence between tristimulus values and RGB values may vary from one color model to another.

[0114] YUV Color Space - a family of color spaces in which color information is represented by three components: one gray component (luma) and two chromatic components (chroma-blue and chroma-red). This color space is used for compact representation of color information. While the gray component is represented in full resolution, the chroma-components are usually represented in two- or four-times smaller resolution. So, two chromatic values encode color information for a macro-pixel of 2x2 or 4x4 original pixels. On one hand, such reduction of color information has minimal impact on human’s perception of the image because the most significant information - edges and textures - is carried by a gray channel. On the other hand, this reduction of color information leads to minimization of binary data size.

[0115] Video Content / Video Stream - a sequence of images / frames with timestamp information. Video can be encoded (compressed) or represented in raw format.

[0116] In media systems, video and audio streams are processed independently. But as for the content delivery, both streams are joint and delivered together. There are two special components serving this need: multiplexer (mux) and de-multiplexer (demux). Multiplexer joins video and audio streams (raw or compressed) into a sequence of packets synchronizing them by timestamps. De-multiplexer splits a sequence of packets back into video and audio streams (raw or compressed).

[0117] The present disclosure focuses on video part only leaving audio part aside. But the audio part exists and works as it was working before integration of the solution. This is because audio-processing components, placement of mux and demux components, synchronization algorithm, a way the media gets packetized and transferred over the network - all of those things are not affected by the present disclosure.|0118] Media Content / Media Stream - a combination of one or many video and audio streams available to users as a single unit of delivery. Audio and video streams can be accompanied by text information. All streams are time-synchronized.

[0119] Video Encoding - a process of converting video stream to a sequence of compressed video packets.Widely used video compression standards are H.264 / H.265.10120] Video Decoding - a process of converting a sequence of compressed video packets into a video stream.

[0121] Video Resolution - a size of pixel array (matrix) usually denoted by a pair of numbers: width and height.In media industry video resolutions are grouped into families.

[0122] High Definition Video (HD Video) - in the present disclosure, it is a family of the following video resolutions: HD Ready - 1280x720, Full HD - 1920x1080, and Quad HD - 2560x1440.

[0123] Frame Rate - the frequency at which consecutive frames are captured or displayed. Frame rate is measured in frames per second (FPS). Frame rate can be fixed or variable. Typical frame rates are: 25, 30, 50, 60 FPS. Frame rate lower than 25 FPS degrades human perception of the video.

[0124] Video Transcoding - a process of changing video stream properties such as frame rate or video resolution.Usual goal of video transcoding is to adopt original high-quality stream to capabilities of customer’s display device and network bandwidth. For that the original video stream is decoded, then converted (both resolution and frame rate), and then encoded.

[0125] Video Rendering / Video Content Generation - a process of generating video content using non-video scene description such as object geometry data, textures, light source positions, and others.

[0126] 3D Video Rendering - a process of rendering video content using 3D scene description, where the scene is defined as a set of volumetric / solid objects, surfaces, cloud-points (particles) accompanied with material information, texture information and other properties that affects rendering (roughness, reflection, translucency, others). Scene setup may also define light sources, special effects, camera position and view angle. Separate layers may contain text (sub-titles) and elements of user interface. Most of the data is specified in 3D-space with X, Y, Z coordinates and three orientation angles. Layer information (text, user-interface elements) may be defined in 2D- space with X, Y coordinates and one orientation angle. Scene information evolves in time. So, object positions, surface properties, camera view and other scene components can change from frame to frame.

[0127] 2D Video Rendering - a process of rendering video content using 2D scene description, where the scene is defined as several layers. Each layer contains planar objects with color textures, transparency masks and some other information. In addition, text (sub-titles) and / or elements of user interface can be provided as a separate layer (overlay). Some layers can be fully dedicated for playing other video content with specified crop, scale, transparency. Geometry information is specified in 2D-space with Y coordinates and one orientation angle. Scene information can change in time.

[0128] Live Video Rendering / Real-time Video Rendering - a process of rendering video content that produces video stream synchronized with physical clock. Frames can be presented in natural speed as if they were captured from a real camera. To be real-time, one second of video content should be rendered in less than one second of physical time.

[0129] Cloud Service - software application and resource infrastructure available on the internet.

[0130] Cloud Service Provider - a company that hosts a cloud service.

[0131] Cloud Media Service - a cloud service related to media content generation or processing.

[0132] Content Delivery (Distribution) Network (CDN) - is a geographically distributed network of proxy servers and their data centers. The goal is to provide high availability and performance by distributing the service spatially relative to end users.

[0133] Media Content Delivery (Distribution) Network - a content delivery network that distributes media content - video and audio. Usually such network combines features of content delivery and effective transcoding. So, media content is adopted to customer needs considering capabilities of customer’s display device and network bandwidth.

[0134] Service Density - a number of application instances simultaneously served by a single resource unit, for example, a number of video rendering instances running concurrently on a single Graphical Processing Unit.

[0135] Frame Interpolation (FI) - a technology that reconstructs one or more intermediate frames between two available consequent frames. This technology is used to increase frame rate.

[0136] Video Frame Interpolation (VFI) - a frame interpolation technology that uses frame information only (array of pixels) and does not require any additional render targets such as depth-map, albedo, normals, motion vectors and others.

[0137] x2 / x3 Frame Interpolation - frame interpolation that doubles (x2) or triples (x3) frame rate. So x2 interpolation reconstructs one frame and x3 interpolation reconstructs two frames. These numbers (x2 and x3) can be called interpolation factor or rate up-sampling factor.

[0138] Meta Human / Digital Human - a technology or a service that renders human-like person. Content can be generated out of pre-recorded human character or from real human whose mimic, pose and gestures are captured live and repeated by virtual character real-time.

[0139] Artificial Neural Networks (ANN) - a computational model defined as an ordered acyclic graph where each node represents a layer - an operation on a set of input tensors that produces one or more output tensors, andthe edge represents data links - which outputs are connected to what inputs. Main data unit is tensor. The graph can include fully connected linear layers, convolutional layers, activation layers, special operations such as maximum pooling or average pooling and many others.

[0140] Convolutional Neural Networks (CNN) - a sub-class of Artificial Neural Networks specifically designed for processing spatial data such as images or audio signals. CNN includes convolutional layers that compute convolutions with trainable kernel of limited size, e.g., 3x3, 5x5, 7x7.

[0141] Tensor - in the present disclosure, this is a multi-dimensional array of real numbers. Tensor is a main data unit in artificial neural networks. This definition is different from the mathematical one. To note a single element of a tensor, brackets with zero-based indices can be used. So elements of 3 dimensional tensor x with number of channels C, height H and width W can be addressed as x[k, i,J] where k e {0, 1, ... C — 1], i e {0, 1, ... H — 1), and j 6 {0, 1, ... W — 1}. Please note that this is one of the possible formats.

[0142] Tensor formats NCHW, CHW, NHWC, HWC - defines which tensor dimension corresponds to what dimension of a spatial object. Here C stands for channel, H stands for height, W stands for width, and N stands for batch-size -a number of spatial objects combined into a single tensor. For example, ten HD RGB images (3-channels, 1280x720 each) combined into a single tensor will have the following dimensions: 10x3x720x1280 in NCHW and 10x720x1280x3 in NHWC.

[0143] Tensor Size - is tensor dimensions.

[0144] Tensor Depth / Feature Map Depth - number of tensor channels.

[0145] Feature Space -n-dimensional space of real numbers ( / ?") that represents a local area (e.g., patch or segment) of a spatial object (e.g., image or audio sequence) using specific features such as derivatives, frequencies / phases or any other. Usually feature space carries information about geometric structure of an image patch, texture specifics or other higher-level information.

[0146] Feature Map - a tensor of features obtained by converting all local areas of a spatial object into a feature space. For example, an RGB image represented by a tensor of size 3 x Hinx Wtncan be converted into a feature map of size n x Houtx Wout, where Hinand Winare height and width of the 3-channel image, Houtand Woutare height and width of n-dimensional feature space.

[0147] Tensor up sampling I down sampling - a procedure of increasing or decreasing spatial dimensions of a tensor. Usually when dimension increases, tensors are interpolated along that dimension. And when dimension decreases, tensor values are averaged along that dimension. All channels are processed independently. Commonlyused re-sampling algorithms are: nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, area interpolation (down-sampling only), Lanczos filter and SINC filter.

[0148] Image Pyramid - a sequence of images where a next image is obtained by down sampling a current image by constant factor (usually by factor of two).|0149] Feature Pyramid - a sequence of feature maps generated from image pyramid.

[0150] Feature map magnitude (features magnitude) - a single channel tensor of the same spatial dimensions as the feature map with values equal to square root of sum of squares of channels for specific column and row: magn( / )[0, i,y] = -J'Zkf2[k,i,j].

[0151] Optical Flow (OF) - usually defined as point-to-point correspondence between two frames. As shown in FIG. 1, a common way to define optical flow is to provide a tensor of X, Y offsets such that for every destination pixel, a source pixel can be computed to take. Offsets can be provided with sub-pixel precision. In that case color information of source pixel is mixed with its neighbors according to fractions.

[0152] Flow magnitude, Motion magnitude, Motion Level - Length of optical flow vectors computed as V2+ Y2and aggregated into a metric, for example: average motion magnitude or maximum motion magnitude. This metric is computed for a pair of consequent frames.

[0153] Flow up sampling / down sampling - Up sampling and down sampling of the flow tensor. This is done by tensor up sampling / down sampling followed by scaling tensor values by scale factors: for X-channel scale by ratio of new to old width, for Y-channel by ratio of new to old height.|0154] Convolution with stride one (i.e., non-strided convolution): core operation in convolutional neuralOutput tensor dimension depends on padding. For zero padding, output tensor size is reduced by R from each side, so it is Coutx H — 2R x W — 2R. For padding = R, output tensor size is Coutx H x IV. In case of padding x[kin, i + Si,j + <5j] = 0 for indices outside of tensor dimensions.

[0155] Convolution with stride greater than one (i.e., strided convolution): for simplicity it is considered stridewhere K is a convolution kernel size, R = (K — l) / 2 , tensor x is an input tensor of size CQUt X H x IV, tensor w is a tensor of weights of size Coutx Cinx K X K.Output tensor dimension depends on padding. Output tensor size is twice as lower as the original one: COutxH / 2 x IV / 2. Please note that x[kin, i + 6i,j + <5j' ] = 0 for indices outside of tensor dimensions.

[0156] Activation Layers are non-linear functional layers that keep spatial dimensions of input tensors but can

[0158] Splitting and Concatenation of tensor splits or concatenates tensor(s) along channel dimension.

[0159] Peak Signal to Noise Ratio (PSNR) - a metric for measuring image reconstruction quality defined as decimal logarithm of ratio of maximum possible (peak) signal and square root of mean squared error (MSE). Usually measured in decibels (dB). Typical well-matched images have PSNR value 30 dB and higher.

[0160] Structured Similarity Index Measure (SSIM) - another classical metric for measuring difference between reconstructed and reference images. A unitless quantity that has a range between zero and one. Where one means complete match and zero means complete mismatch. Typical well-matched images have SSIM value 0.95 and higher.

[0161] Learned Perceptual Image Patch Similarity (LPIPS) - a metric based on special neural network that tries to predict human perception of images similarity. A unitless non-negative quantity for which zero means complete match and large numbers mean some level of mismatch. Typical well-matched images have LPIPS value 0.05 and lower.

[0162] Ghosting - a corruption of a reconstructed image that looks like a blended replication of a moving object or its parts. Ghosting is a common problem of such technologies as frame interpolation and super resolution.

[0163] Frame interpolation technology reconstructs one or more frames between two consequent frames. It can be used in Cloud Media Services to reduce resource cost of real-time video rendering in case if original renderingoperation is heavy compared to frame interpolation. Below are two examples of such applications.

[0164] In an example of virtual teacher, FIG. 2A illustrates an original pipeline and FIG. 2B illustrates a modified pipeline. FIG. 2A shows that a camera transmits a picture of user to cloud side. At the cloud side, Al (Artificial Intelligence) estimates user pose, 3D engine renders the user’s avatar, and then content (stream) is delivered to MCDN (Media Content Delivery Network). In the original pipeline, a user is acting as a teacher (or lecturer). He / she connects a camera to a Cloud Media Service. Then, the camera streams a picture of the user to the cloud server. An Al-based algorithm estimates the user’s pose and mimics. Aspecial rendering component generates 3D-avatar of the user from a pre-defined character. And, finally, a streaming component encodes generated video content and sends it to a Media Content Delivery Network, which, in turn, broadcasts live stream to recipients (students). All components operate at 30 FPS which is resource consumptive.

[0165] In the modified pipeline, a new element is inserted - a frame interpolation component. This element can double or even triple the input frame rate. This is called x2 and x3 interpolation respectively. Resource consumptive components such as Pose Estimation and Avatar Renderer can operate at a lower frame rate: 15 or 10 FPS. The frame interpolation component reconstructs the missing frames. The x2 interpolation inserts one additional frame and gets 30 FPS out of 15 FPS. The x3 interpolation inserts two additional frames and gets 30 FPS out of 10 FPS. Then streaming component operates and encodes video at 30 FPS, and sends stream to the Media Content Delivery Network.

[0166] In another example of virtual assistant, FIG. 3A illustrates an original pipeline and FIG. 3B illustrates a modified pipeline. FIG. 3 A shows that interactive-AI-assistant can answer user’s questions about some product with 2D-rendered and animated virtual person. In the original pipeline, a user communicates with a virtual assistant who consults or guides user on some topic of interest. Text-based interface is used for that. Assistant character is rendered using Digital Human technology. Then rendered content is encoded and streamed to user.

[0167] In the modified pipeline, a frame interpolation component is inserted. So, rendering can happen at a lower frame rate. To maintain video smoothness, the missing frames are reconstructed by frame interpolation component increasing frame rate by a factor of two or three. Video is encoded and streamed to user.,0168] In the above examples, the pipelines are modified with integrated frame interpolation, this may increase a service density. Same resource unit can support more rendering application instances with a modified pipeline compared to the original pipeline. This may also make solution real-time ready in case if performance of the original pipeline does not allow to reach a target frame rate.|0169] However, with the aforementioned modified pipeline, there are some challenges in aspects of quality, motion level, performance, user control, etc. Regarding quality, it may be how to avoid corruption of interpolated frame. Typical problems are so-called “ghosting”, deformed objects edges, deformed background nearby object edges, deformed space around light areas. FIGS. 4Aand 4B are examples of “ghosting”.

[0170] Regarding motion level, it may be how to keep operating with appropriate quality in case if motion magnitude reaches high values. Usually motion level is related to frame rate, and FIG. 4C illustrates a relationship between motion level and frame rate. High frame rate usually corresponds to small motion, because time difference between two frames is small. In contrast, low input frame rate usually introduces large motion. But sometime contracases may happen. Frame interpolation solutions can be categorized by motion level they can handle.

[0171] Regarding performance, frame interpolation should be fast enough to bring economical effect. There are two major categories of frame interpolation solutions: real-time solutions and offline solutions. For offline solutions, which are used in film production, requirements are simple - operate faster than rendering. For real-time solutions, requirements are strict. Performance should be enough to increase a number of application instances handled by the same computing unit (e.g., GPU).

[0172] Table 1 illustrates upper-limits on frame interpolation time in milliseconds required to achieve desired number of instances with 30 FPS each. Sign “+” means interpolation is not required because target is already achieved. Sign “-” means desired number of instances is not achievable.Table 1

[0173] As show in Table 1, if rendering time takes 20ms, then system can support one instance only running without interpolation. With x2 interpolation the system can support 2 instances if interpolation takes not more than 13.3ms and 3 instances if it takes not more than 2.2ms. With x3 interpolation the system supports 2, 3, and 4 instances if interpolation time does not exceed 30.0 ms, 13.3 ms, and 5.0 ms respectively. Each instance produces 30 FPS stream.

[0174] Regarding user control, since flame interpolation may introduce quality degradation, any particular implementation of this technology should introduce some level of user control. For example, user control solution may let user disable or enable frame interpolation in gaming using a hot-key. Service application programming interface (API) may let user configure frame interpolation for specific application instance.

[0175] Besides the aforementioned Video Content Rendering in Cloud, frame interpolation technology has other application scenarios, for example, gaming, video editing, video streaming, animated film production, etc.

[0176] Regarding gaming, frame interpolation can improve gaming experience by increasing frame-rate and motion smoothness in high-resolution. Some indicators about gaming scenario are as follows: up-sampling (50-60 ->100-120) FPS, motion level (small / medium), ultra-low latency (1-2 ms), desktop GPU (Graphical Processing Unit). Solutions use help of render targets produced by 3D rendering pipeline such as motion vectors, depth-map, color, albedo, and others.

[0177] Regarding video editing, frame interpolation can make video smoother by increasing original frame rate by a factor of two or more: 25 / 30 — > 50 / 60 — > 100 / 120. Offline solution has no strict requirements on performance. Some indicators about video editing scenario are as follows: moderate motion to handle, quality requirements: from moderate to high, desktop CPU (Central Processing Unit) / GPU.

[0178] Regarding video streaming, frame rate conversion may be required to adapt video to regional requirements e.g., 60 FPS in US, and 50 FPS in Europe. Some indicators about video streaming scenario are as follows: real-time scenarios, cloud media services, moderate motion to handle, high quality, performance enough to operate real-time, server CPU / GPU / NPU (Neural Processing Unit).

[0179] Regarding animated film production, it requires to minimize total film rendering time by rendering at low frame rate and interpolate to a target rate. Frame rendering may take hours to complete. So frame interpolation can run seconds and minutes. Some indicators about animated film production scenario are as follows: up-sampling: (6 -7 — 60 FPS), large motion, high quality, server CPU / GPU / NPU.

[0180] The above describes possible application scenarios of frame interpolation, next related frame interpolation solutions will be introduced.

[0181] Some frame interpolation solutions are designed to improve gaming experience by increasing motion smoothness on fast changing scenes. They work with render targets as a part of 3D rendering pipeline. One such solution uses motion vectors as initial estimation (hints) of the optical flow. More accurate optical flow is obtained by further estimation with help of hardware accelerated correlation blocks. In addition to that some refinement is performed using neural network. That solution supports only one instance of frame generation, and requires special hardware blocks in order to achieve ultra-low latency. These frame interpolation solutions for gaming require specific 3D rendered data such as motion vectors and depths, so they are 3D only solutions and do not support 2D rendering cases. They are designed for gaming scenarios with high input frame rate (e.g., 45+ fps), and provide low quality for the case of low input frame.

[0182] In another frame interpolation solution, optical flow is estimated by computing correlations between two frames acting at different levels of pyramid, then interpolate flow and do in-painting of holes caused by occlusions and dis-occlusions. To compute correlation, the solution uses help of hardware-accelerated blocks for flow estimation. In-painting is performed with help of neural network using kernels for inference. However, the time for frame interpolation is long.

[0183] Some other frame interpolation solutions are designed based on residual-blocks architecture. These solutions are targeted at applications such as video editing and video content generation. One solution uses a model based on coarse-to-fine flow estimation which can be divided into two major parts: one part performs intermediate flow estimation, and the other part synthesizes the image. The first part implements direct prediction of intermediate optical flow along with fusion mask. Coarse-to-fine flow estimation is implemented by chaining flow estimation blocks. Each block in the chain takes and returns data in original resolution while doing actual estimation in some internal resolution which may be lower or be the same. The second part estimates image residual based on the computed outputs of the first part and the two original frames (the previous and the current frames) to obtain the final frame (the intermediate frame). Another solution employs a model including pyramid of encoders and pyramid of decoders where decoders estimate both intermediate feature maps and the intermediate flow. Top-level decoder estimates image residual, intermediate optical flow, and the mask for fusing left and right warped images (images obtained based on a warp operation of a previous image and a corresponding optical flow, a warp operation of a current image and a corresponding optical flow). The fused image and the image residual are combined to obtain thefinal image (the intermediate image). However, the time for frame interpolation for these solutions is relatively long. [0184| In view of the above, some related frame interpolation solutions can only support 3D video with high frame rate but cannot support 2D cases and provide low quality for the case of low input frame rate. Some related frame interpolation solutions can support 2D cases, but the time for frame interpolation is long, and thus they may not be suitable for real-time applications, and may have low service density even if they can be applied. So, a frame interpolation solution that take less time for frame interpolation and can work with low-input frame rate for rendering scenarios is desired|0185| In addition, the aforementioned solutions also lack options for automatically control quality in unattended scenarios under fixed resource constrains.

[0186] For cloud rendering scenario, a user need an option to leave rendering and streaming service unattended. For example, in the case of Virtual Teacher, the user configures the service once, and then he or she is fully busy with his / her primary work - lecturing the audience using 3D avatar. In another example with Virtual Assistant, the user may configure assistant service, do few rounds of test, and then leave the service unattended with hope that desired level of quality is constantly maintained.

[0187] Usage of frame interpolation reduces resource costs but also introduces quality degradation. Quality degradation usually depends on scene complexity and dynamics such as the number of overlapping objects and motion magnitude. Specific artifacts such as “ghosting”, edge deforming or space deforming can appear in case of fast motion or sudden light condition change. At the same time resource usage should not exceed the quota for which user pays. Otherwise user may lose main advantage of frame interpolation technology - request two / three times less resource quota per instance and pay two / three times less money.

[0188] Regarding quality control, some frame interpolation solutions provide on / off options for frame interpolation (generation) on the fly by some user action, like key press, some solutions allow the user to select quality preset on start, while some solutions allow the user to select model type on start. By selecting model type, the user may find trade-off between quality and performance. All solutions above do not support automatic quality control, but require user to monitor the generated stream. There is no automatics that would temporally disable or switch frame interpolation into a different mode if quality degradation is expected.(0189] In view of the above, the present disclosure proposes a frame interpolation solution to solve at least one of the problems.(0190] FIG. 5 shows a flowchart of a method for frame interpolation according to one or more embodiments ofthe present disclosure. The method may include the following steps.

[0191] Step 502, obtaining a previous frame and a current frame of a video stream.

[0192] As mentioned before, a video stream is a sequence of frames with timestamp information. The previous frame is a frame with timestamp just before that of the current frame. For example, in the detailed embodiments of the present disclosure, the previous frame may be referred to as / 0, and the current frame may be referred to as / t.

[0193] Step 504, obtaining one or more intermediate frames based on the previous frame and the current frame, and obtaining one or more confidence values respectively corresponding to the one or more intermediate frames.

[0194] An intermediate frame is a frame with timestamp between that of the previous frame and the current frame. The previous frame may be denoted as Zo, the current frame may be denoted as / t.

[0195] The quantity of intermediate frames to be inserted between the previous frame and the current frame may depend on an original frame rate of the video stream and a taiget frame rate. For example, in a case that the target frame rate is twice the original frame rate, x2 frame interpolation may be performed to reconstruct one intermediate frame between the previous and current frames, which may be denoted as which may be a frame at the middle time point between the previous and current frames. Assuming the timestamp for the previous frame is t0, and the timestamp for the current frame is t1;then the timestamp for this intermediate frame / 1 / 2may be t0+ | At, where At = — t0represents the time interval between the previous and current frames. In a case that the target frame rate is three times the original frame rate, x3 frame interpolation may be performed to reconstruct two intermediate frames between the previous and current frames, which may be denoted as / 1 / 3and Z2 / 3, respectively, and the timestamps for the two intermediate frames Z1 / 3and I2 / 3may be t0+ -At and t0+ -At.

[0196] A confidence value corresponding to an intermediate frame indicates image quality of the intermediate frame, and the confidence value may be in a range from zero to one, where 0 represents that the intermediate frame is in low image quality and 1 represents that the intermediate frame is in high image quality. How to obtain the one or more intermediate frames and one or more confidence values respectively corresponding to the one or more intermediate frames will be described later.

[0197] Step 506, in a case that the one or more confidence values satisfy a pre-defined condition, interpolating the one or more intermediate frames between the previous frame and the current frame.

[0198] The confidence value can be used in ghost-prevention mechanism. The pre-defined condition may be set based on a specified threshold. If the estimated confidence value is greater than or equal to the specified threshold, the one or more intermediate frames may be interpolated between the previous frame and the current frame. In acase that the one or more confidence values do not satisfy the pre-defined condition, for example, if the estimated confidence value is smaller than the specified threshold, the one or more interpolated frames are ignored and the previous frame or the current frame is repeated one or more times. It should be noted that, in some implementations, the intermediate frame(s) may be interpolated in a case that the estimated confidence value is greater than the specified threshold; and in a case that the estimated confidence value is smaller than or equal to the specified threshold, the previous frame or the current frame is repeated, the specific condition is not limited in the embodiments of the present disclosure. It should also be noted that, for x2 interpolation, the previous frame or the current frame may be repeated for one time; and for x3 interpolation, the previous frame or the current frame may be repeated for two times, or, both the previous frame and the current frame may be repeated for one time, which is not limited in the embodiments of the present disclosure.

[0199] The confidence value is introduced in determining whether the intermediate frame(s) can be used for frame interpolation, and when the confidence value meets the pre-defined condition, the intermediate frame can be interpolated between the previous frame and the current frame. The decision on whether to interpolate the intermediate frame(s) can be made to prevent or reduce ghosting caused by frame interpolation and get a tradeoff between smoothness and level of ghosting.

[0200] In a possible implementation, when one intermediate frame needs to be interpolated between the previous and current frames, a confidence value is obtained for the intermediate frame, and the intermediate frame will be interpolated between the previous and current frames if the confidence value meets the pre-defined condition.

[0201] In a possible implementation, when multiple intermediate frames need to be interpolated between the previous and current frames, multiple confidence values may be obtained which respectively correspond to the multiple intermediate frames, and whether the multiple intermediate frames can be interpolated depends on the multiple confidence values. In a possible implementation, the multiple intermediate frames will be interpolated between the previous and current frames if all the confidence values are greater than or equal to the threshold. In another possible implementation, an intermediate frame of the multiple intermediate frames will be interpolated if its corresponding confidence value is greater than or equal to the threshold.

[0202] In another possible implementation, when multiple intermediate frames need to be interpolated between the previous and current frames, one confidence value may be obtained for the multiple intermediate frames, and when the confidence value meets the pre-defined condition, the multiple intermediate frames can be interpolated between the previous frame and the current frame.

[0203] In a possible implementation, the intermediate frame(s) may be obtained through an encoding process and a decoding process. The encoding process may include generation of feature maps with different resolutions for the previous frame and the current frame. The decoding process may include generation of the intermediate frame and the confidence value corresponding to the intermediate frame. The intermediate frame(s) may be obtained by using a frame interpolation model including a pyramidal encoder and a chain of decoders, where the intermediate frame(s) can be an output of the model with the previous and the current frames as inputs.

[0204] Regarding the encoding process, a feature pyramid including feature maps with different resolutions may be obtained for each of the previous frame and the current frame and then used for obtaining the intermediate frame(s). In a possible implementation, a first feature pyramid for the previous frame and a second feature pyramid for the current frame are obtained. The first feature pyramid includes a set of first feature maps with different resolutions for the previous frame and the second feature pyramid includes a set of second feature maps with different resolutions for the current frame. A first feature map and a second feature map for a same level have same resolution, a first feature map for a top level has a resolution of the top level (e.g., a highest resolution) and a first feature map for a bottom level has a resolution of the bottom level (e.g, a lowest resolution).

[0205] Each pyramid can be arranged into levels according to resolutions of feature maps going from the highest resolution for the top level to the lowest resolution for the bottom level with each level downwards the pyramid has resolution by a certain factor (e.g., a factor of two) compared to the previous one, and with the same number of levels in the first and the second feature pyramids. A pyramidal encoder may be used for producing the first and the second32x24, in a case that the original resolution of the video stream is 1024x768.

[0206] In a possible implementation, the pyramidal encoder may first obtain the first feature map for the top level based on an image of the previous frame with an original resolution of the video stream; then obtain a first feature map for a level next to the top level, where the first feature map for the level next to the top level includes a first part obtained based on a down-sampled image of the previous frame for the level next to the top level, and a second part obtained based on the first feature map for the top level. The down-sampled image of the previous frame for the level next to the top level is obtained by down sampling the image for the previous frame with the original resolution by a pre-defined factor, e.g., a factor of 2. Next, the pyramidal encoder may obtain a first feature map for each level of one or more levels lower than the level next to the top level, where the first feature map for each levelincludes a first part obtained based on a down-sampled image of the previous frame for the each level, and a second part obtained based on a first part of a first feature map for a level upper than and next to the each level. The down- sampled image of the previous frame for the each level can be obtained by down sampling the down sampled image of the previous frame for the level upper than and next to the each level by a pre-defined factor, e.g., a factor of 2.

[0207] The above describes how to get the first feature maps of respective levels for the previous frame, it should be noted that, it can also apply for the second feature maps of respective levels for the current frame, and will not repeated here for the sake of brevity. At the top level of the pyramid i = 0, the encoder takes the input frame / image lk(A=0 or 1) and produces the feature map of level 0 f ° For every next level of the pyramid i > 0,feature map for level 1).

[0208] For a feature map of a specific level including two parts, a first part is obtained based on down sampling, and a second part is obtained based on a first part of a feature map for a next upper level of the specific level (i.e., a level upper than and next to the specific level) without an additional step of down sampling. Since the feature maps of the bottom and middle levels each include features obtained from some upper-level features in addition to features obtained from the images with the resolutions of respective levels, the features of the bottom and middle levels may be more abundant and accurate, and the feature maps of these levels and thus the feature pyramid can be generated more accurately and efficiently.

[0209] In a possible implementation, a quantity of levels of the first feature pyramid is based on an original resolution of the video stream and a second resolution threshold, where a resolution of the first feature map for the bottom level is lower than the second resolution threshold and a resolution of a first feature map for a level upper than and next to the bottom level is higher than the second resolution threshold. As mentioned before, the quantity of levels of the first feature pyramid is the same as the quantity of levels of the second feature pyramid, therefore, the determination principle of the quantity of levels of the first feature pyramid can also apply for that of the second feature pyramid.

[0210] For example, in the case that the original resolution of the video stream is 1024x768 and the secondresolution threshold is 32x32, level 0 of the first and the second feature pyramids correspond to resolution of 512x384 by down sampling the image with resolution of 1024x768, level 1 of the first and the second feature pyramids correspond to resolution of 256x192 by down sampling the image with resolution of 512x384, and so on. The pyramidal encoder adds a new level of feature pyramid by down sampling image while the image size is greater than256x192, 128x96, 64x48, 32x24. By setting the aforementioned determination principle, the quantity of levels or the lowest resolution of the feature pyramid can be determined, and it may be beneficial to achieve a tradeoff between computational overhead and accuracy in terms of the frame interpolation solution.

[0211] In a possible implementation, the second resolution threshold is based on a motion level between the previous frame and the current frame. The second resolution threshold intends to handle proper level of motion for the whole image while using limited motion compensation at each level of pyramid. With effective encoder receptive field of 7x7, motion of magnitude 4 can be estimated (compensated) which is 12.5% of 32x32 for the bottom level. With the help of pyramid this result can be propagated to the upper level and for the whole image. The second resolution threshold can be changed if needed, for example, by increasing pyramid depth and decreasing the resolution of the bottom level at very low cost or almost no-cost, since bottom levels are fast to compute in the case of low resolution. Architecture is scalable and does not require re-training. If the bottom-level stops at 64x64 (i.e., the second resolution threshold is 64x64), then the maximum motion magnitude that can be handled would be 4 / 64 which is 1 / 16. This is less than what is typically adopted but still a valid value as an option. It should be noted that, 32x32 is just a reasonable value for the second resolution threshold, and other lower or higher resolution such as 16x16 may be used.

[0212] In a possible implementation, a first feature map and a second feature map for a same level have a same quantity of features, and a quantity of features for the first feature map of the top level is smaller than a quantity of features for a first feature map of a lower level. The quantity of features for the first / second feature map of each level may be associated with the aforementioned data flows of the pyramidal encoder, where the first / second feature map for the top level has only one part, while the first / second feature map for other levels has two parts. For example, the first and the second feature maps of level 0 have one number of channels while the first and the second feature maps for levels 1, 2, 3, 4 have another number of channels. For example, for RGB model, the number of channels may be 6 for level 0 and 18 for levels 1, 2, 3, 4.

[0213] Then, the one or more intermediate frames are obtained according to the first and the second featurepyramids, where intermediate feature maps with different resolutions of the one or more intermediate frames are built based on the first feature pyramid and the second feature pyramid from the bottom level to the top level.

[0214] Regarding the decoding process, it may be implemented by a chain of decoders, which may include a bottom-level decoder, one or more middle-level decoder, a top-level decoder, etc.

[0215] As mentioned before, the intermediate frame(s) can be an output of the frame interpolation model with the previous and the current frames as inputs, where the model includes a pyramidal encoder and a chain of decoders. The chain of decoders may be arranged in a sequence of the bottom level to the top level. The pyramidal encoder may produce the first and the second feature maps with different resolutions for the previous and the current frames respectively, and the first and the second feature maps pass through the chain of decoders to generate the intermediate frame(s). With such data flows, features of different resolutions are considered, and the obtained intermediate frame(s) can be reliable and accurate, in other words, in good image quality. Combing such intermediate frame with the confidence value to perform frame interpolation, a better ghost-prevention may be achieved.

[0216] As for data processing of the bottom-level decoder, in a possible implementation, one or more first optical flows with a first resolution, one or more second optical flows with the first resolution, one or more intermediate feature maps with the first resolution, one or more first flow selection mask with the first resolution, one or more second flow selection mask with the first resolution, and one or more feature selection masks with the first resolution are obtained according to a first feature map and a second feature map for a bottom level (with the lowest resolution). The first resolution is a next higher resolution of the lowest resolution (i.e., a resolution of a level higher than and next to the bottom level), where each of the one or more first optical flows with the first resolution is a forward optical flow corresponding to a respective one in the one or more intermediate frames, and each of the one or more second optical flows with the first resolution is a backward optical flow corresponding to a respective one in the one or more intermediate frames. Taking RGB X2 model as an example, if the lowest resolution is 32x24,

[0217] The chain of decoders may include a bottom-level decoder with the first and the second feature maps as inputs. The bottom-level decoder may produce forward and backward optical flows (i.e., first and second optical flows), one or more intermediate feature map(s) and selection masks (which may include one or more featureselection masks, one or more first flow selection masks and one or more second flow selection masks). The inputs of the bottom-level decoder have the lowest resolution, while outputs of the bottom-level decoder have a next higher resolution of the lowest resolution (i.e., a resolution of a level higher than and next to the bottom level), therefore the outputs of the bottom-level decoder can be compatible with the first and the second feature maps with the next higher resolution of the lowest resolution in terms of spatial dimensions, and the resolution is the same as that of inputs of a next higher-level decoder. The selection masks regulate how much the forward and backward optical flows, and the intermediate feature map(s) decoded by the bottom-level decoder are trusted. The data flows of the bottom-level decoder form a part to produce the intermediate frame(s) more reliably and more accurately.

[0218] As for data processing of the one or more middle-level decoders, inputs of each middle-level decoder may be with a resolution, referred to as a middle resolution, which is higher than the lowest resolution (i.e., a resolution of a bottom level) and lower than the highest resolution (i.e., a resolution of a top level); and outputs of each middle-level decoder may be with a resolution, referred to as a second resolution, which is a next higher resolution of the middle resolution (i.e., a resolution of a level higher than and next to a level of the middle resolution). For example, in a case that the lowest resolution is 32x24, the highest resolution is 1024x768, and there are three middle-level decoders 64x48, 128x96 and 256x192. A middle resolution for the first middle-level decoder (i.e., the resolution of inputs of the first middle-level decoder) may be the aforementioned first resolution (the resolution of outputs of the bottom-level decoder, e.g., 64x48), and the second resolution for the first middle-level decoder (i.e., the resolution of outputs of the first middle-level decoder) may be a next higher resolution of the first resolution, e.g., 128x96. A middle resolution for the second middle-level decoder (i.e., the resolution of inputs of the second middle-level decoder) may be 128x96, and the second resolution for the second middle-level decoder (i.e., the resolution of outputs of the second middle-level decoder) may be anext higher resolution of 128x96, e.g., 256x192. Amiddle resolution for the third middle-level decoder (i.e., the resolution of inputs of the third middle-level decoder) may be 256x192, and the second resolution for the third middle-level decoder (i.e., the resolution of outputs of the third middle-level decoder) may be a next higher resolution of 256x192, e.g., 512x384.

[0219] In a possible implementation, one or more first warped feature maps with a middle resolution are obtained by warping a first feature map with the middle resolution using one or more first optical flows with the middle resolution, where the middle resolution is higher than the lowest resolution and lower than the highest resolution. Similarly, one or more second warped feature maps with the middle resolution are obtained by warping the second feature map with the middle resolution using one or more second optical flows with the middle resolution.Then, one or more first optical flow residuals with a second resolution, one or more second optical flow residuals with the second resolution, one or more intermediate feature maps with the second resolution, one or more first flow selection masks with the second resolution, one or more second flow selection masks with the second resolution, and one or more feature selection masks with the second resolution are obtained according to one or more intermediate feature maps with the middle resolution, the one or more first warped feature maps with the middle resolution, the one or more second warped feature maps with the middle resolution, one or more first flow selection masks with the middle resolution, one or more second flow selection masks with the middle resolution, and one or more feature selection masks with the middle resolution, where the second resolution is a next higher resolution of the middle resolution (i.e., a resolution of a level which is higher than and next to a level of the middle resolution).

[0220] There may be one or more middle-level decoders, in a case that the pyramidal encoder produces five- level feature maps with resolutions 512x384, 256x192, 128x96, 64x48, 32x24, there may be three middle-level decoders. Still taking RGB X2 model as an example, if the middle resolution is 64x48, the second resolution would be 128x96, a middle-level decoder D1-3) takes the following inputs: warped feature maps (the first and the secondand feature selection mask M^2with resolution 128x96. If the middle resolution is 128x96, the second resolution would be 256x192, a middle-level decoder D(2)takes warped feature maps using corrected optical flows f0(^1 / 2=resolution 256x192.

[0221] The chain of decoders may include one or more middle-level decoders. A middle-level decoder takes warped feature maps of its level, one or more intermediate feature maps produced by the former decoder, and the selection masks produced by the former decoder. The middle-level decoder produces optical flow residuals, one ormore intermediate feature maps and selection masks. Outputs of the middle-level decoder have a next higher resolution than inputs of the middle-level decoder. The selection masks regulate how much other outputs decoded by the middle-level decoder are trusted. The data flows of the middle-level decoder form a part to produce the intermediate frame more reliably and more accurately.

[0222] As for data processing of the top-level decoder, in a possible implementation, one or more first warped feature maps for the top level (with the highest resolution) are obtained by warping the first feature map for the top level (with the highest resolution) using one or more first optical flows with the highest resolution. Similarly, one or more second warped feature maps for the top level (with the highest resolution) are obtained by warping the second feature map for the top level (with the highest resolution) using one or more second optical flows with the highest resolution. Then, the one or more intermediate frames are obtained according to one or more intermediate feature maps with the highest resolution, the one or more first warped feature maps with the highest resolution, the one or more second warped feature maps with the highest resolution, one or more first flow selection masks with the highest resolution, one or more second flow selection masks with the highest resolution, and one or more feature selection masks with the highest resolution.

[0223] Still taking RGB X2 model as an example, if the highest resolution is 512x384, a top-level decoderAflJ2with resolution 512x384. The decoder £>(0)produces an intermediate image / 1 / 2with resolution 1024x768.

[0224] The chain of decoders may include a top-level decoder. The top-level decoder takes warped feature maps with the highest resolution, an intermediate feature map produced by the former decoder, and the selection masks produced by the former decoder. The top-level decoder produces an intermediate frame as the output of the model. With such data flows, the intermediate frame(s) can be more reliably and more accurately.

[0225] A decoder in the chain of decoders may output optical flows directly, or output optical flow residuals which are further processed to get respective optical flows. As mentioned before, the bottom-level decoder outputs optical flows directly, while the middle-level decoder outputs optical flow residuals.

[0226] Regarding how to get optical flows using optical flow residuals, in a possible implementation, one or more first optical flows with a third resolution may be obtained by up-scaling one or more first optical flows with afourth resolution to obtain one or more up-scaled first optical flows with the third resolution, and the one or more up-scaled first optical flows with the third resolution are corrected by one or more first optical flow residuals with the third resolution. Similarly, one or more second optical flows with the third resolution are obtained by up-scaling one or more second optical flows with the fourth resolution to obtain one or more up-scaled second optical flows with the third resolution and the one or more up-scaled second optical flows with the third resolution are corrected by the one or more second optical flow residuals with the third resolution. The third resolution is a next higher resolution of the fourth resolution (i.e., a resolution of a level higher than and next to a level of the fourth resolution), and the fourth resolution is higher than the resolution of the bottom level (i.e., the lowest resolution).

[0227] As mentioned before, taking a middle-level decoder D(3)next to the bottom-level decoder in RGB X2 model as an example, this decoder outputs a forward optical flow residual AF®1 / 2and a backward optical flow

[0228] When a decoder produces an optical flow residual, optical flow correction is performed to obtain a corresponding optical flow as an input of a next decoder. This can improve accuracy of optical flows inputting into the next decoder, thus, the intermediate frame(s) obtained through the chain of decoders may be more accurate.

[0229] Regarding the process for generating the confidence value corresponding to the intermediate frame, in a possible implementation, one or more confidence values respectively corresponding to one or more intermediate frames may be obtained based on one or more first warped feature maps and one or more second warped feature maps, where the one or more first warped feature maps and the one or more second warped feature maps may have a fifth resolution which is greater than or equal to a first resolution threshold. A confidence value corresponding to an intermediate frame may be output by a confidence estimator with the first and the second (forward and backward) warped feature maps as inputs. For example, the confidence estimator may be attached to a decoder in the chain of decoders which has output resolution greater than or equal to a threshold. The fifth resolution may refer to the output resolution of the decoder. The confidence estimator may be attached to the output side of the decoder to take the first and second warped feature maps with the fifth resolution. The decoder may be a first decoder, from the bottom level, whose output resolution is greater than or equal to the threshold. For another example, the confidence estimator may be attached to the decoder whose input resolution is greater than or equal to a threshold (i.e., the first resolution threshold), and the fifth resolution may refer to the input resolution of the decoder. The confidence estimator may attached to the input side of the decoder to take the warped feature maps with the fifth resolution as the inputs of theconfidence estimator, and in this case, the decoder may be a first decoder, from the bottom level, whose input resolution is greater than or equal to the threshold. Taking the chain of decoders including five decoders (D'o>. D(1), O(2), £>(3), pW)with output resolutions 64x48, 128x96, 256x192, 512x384, 1024x768 from the bottom to the top as an example, if the first resolution threshold is 128x128, the confidence estimator may be attached to the output side of decoder D(2>(the first decoder in the chain of decoders which has output resolution greater than 128x128)than 128x128). That is, the first and the second warped feature maps with resolution 256x192 serve as inputs of the confidence estimator. The confidence estimator may be fast to compute in the case of low resolution while guaranteeing computation accuracy.

[0230] A confidence value corresponding to an intermediate frame can be produced by a confidence estimator with two warped feature maps of a specific level as inputs. The confidence estimator may take a difference between two warped feature maps associated with optical flows, normalize that difference to feature magnitudes, and aggregate the feature magnitudes to obtain a confidence value. The confidence value is used for determining whether the intermediate frame can be interpolated. With the aforementioned obtaining manner, the confidence value is reliable, thus, the determination on whether to insert the intermediate frame(s) will also be reliable, thereby preventing ghosting caused by frame interpolation.

[0231] In a possible implementation, the first resolution threshold is associated with a size of details to keep in the video stream. This threshold is associated with level of details which are important to track on the image, and may be configurable. For example, in some scenarios, the scale 128x128 can be selected as the first resolution threshold. On one hand important details to track can be considered, on the other hand this low resolution does not introduce much of computational overhead, i.e., the computational overhead is low. If an image contains smaller details that are important to keep, then the confidence estimator can be attached at a higher level of pyramid, e.g., if it is an image from some surveillance camera which observes large enough area like square full of people, it is needed to track smaller details of the image. In view of the above, this value of the first resolution threshold is related to detail size to track.

[0232] The above describes data flows for the frame interpolation solution of the present disclosure, the one or more intermediate frames can be obtained through a frame interpolation model, and the frame interpolation model is determined from multiple frame interpolation models. The multiple frame interpolation models can be selected according to actual requirements, and some selection principles will be described later. The multiple frameinterpolation models may be used for implementing different frame rate increase in different color spaces, which provides a diversity for the frame interpolation solution. In a possible implementation, the multiple frame interpolation models include a first frame interpolation model that doubles a frame rate of the video stream and performs frame interpolation in an RGB color space (RGB X2 model), or a second frame interpolation model that triples a frame rate of the video stream and performs frame interpolation in an RGB color space (RGB X3 model), or a third frame interpolation model that doubles a frame rate of the video stream and perform frame interpolation in a YUV color space (YUV X2 model), or a fourth frame interpolation model that triples a frame rate of the video stream and perform frame interpolation in a YUV color space (YUV X2 model). There may be other frame interpolation models, which is not limited in the present disclosure.

[0233] As described above, a frame interpolation model may be determined by taking into account: an interpolation mode (e.g. x2 Frame Interpolation or x3 Frame Interpolation) and a color space. Since each interpolation mode corresponds to a unique quantity of intermediate frames, the interpolation mode may be determined or the intermediate frame quantity of intermediate frames to be interpolated may be determined. In a possible implementation, the interpolation mode or the (intermediate frame) quantity of the one or more intermediate frames may be based on an original frame rate of the video stream, and a target frame rate of the video stream. The intermediate frame quantity may be determined based on the original frame rate and the target frame rate, for example, may be a ratio of the target frame rate and the original rate, which may be changed according to actual requirements. For example, in a case that the original frame rate of the video stream is 15 FPS and the target frame rate is 30 FPS, it means that frame rate is increased by a factor of two (30 / 15), that is, x2 interpolation can be used or in other words, one intermediate frame is needed to be interpolated. For another example, in a case that the original frame rate of the video stream is 10 FPS and the target frame rate is 30 FPS, it means that frame rate is increased by a factor of three (30 / 10), that is, x3 interpolation can be used or in other words, two intermediate frames are needed to be interpolated. Compared with x2 interpolation, x3 interpolation can gain bigger increase of the number of application instances. It should be noted that, there may be other determination manners and other interpolation modes, which is not limited in the present disclosure.

[0234] In a possible implementation, the interpolation mode or the (intermediate frame) quantity of the one or more intermediate frames is based on a first operation of a user, where the first operation can indicate a target frame rate of the video stream or the quantity of the one or more intermediate frames. The intermediate frame quantity can be determined based on the user operation, which improves flexibility of the design. Specific implementationmanners can be set according to actual requirements. A user may directly define, through a user interface, a factor (an interpolation factor) by which the original / input frame rate should be increased. The user may also input the target frame, and the factor can be obtained by dividing the target frame rate by the input frame rate (i.e., the aforementioned original frame rate). This factor should be an integer, and can be 2, 3, 4, 5, and others.

[0235] Regarding how to determine the color space for the frame interpolation model, in a possible implementation, a color space in which to perform the frame interpolation may be determined based on a characteristic of a scene of the video stream or a second operation of a user, where the second operation can indicate the color space in which to perform the frame interpolation. The color space in which to perform the frame interpolation may be an RGB color space or a YUV color space. A user can directly choose the color space, or the color space can be determined based on the scene of the video stream, which improves flexibility of the design. For example, for natural looking scenes with complex textures and geometry, YUV color space may be used; for synthetic looking scenes with simplified graphics, RGB color space may be used.

[0236] In a possible implementation, the frame interpolation model is determined according to at least one of an original frame rate of the video stream, a taiget frame rate of the video stream, or a characteristic of scenes of the video stream. As mentioned before, the interpolation mode (e.g., x2 interpolation or x3 interpolation) of the frame interpolation model may be determined based on the original frame rate and the taiget frame rate. A color space of the frame interpolation model may be determined based on the characteristic of scenes of the video stream. In a possible implementation, the frame interpolation model is determined based on a third operation of a user, where the third operation indicates a frame interpolation model selected by the user. A user may directly specify the frame interpolation model, user control may improve user experience. The frame interpolation model can be set according to properties associated with the video stream, while a user can adjust the settings. Frame interpolation can be performed in RGB color space using an RGB model, or in YUV color space using a YUV model. For example, for natural looking scenes with complex textures and geometry, the RGB model may be used; whil e for synthetic looking scenes with simplified graphics, the YUV model may be used. In addition, x2 interpolation and x3 interpolation may be applied to the RGB model and the YUV model, which can be selected according to actual requirements.

[0237] In a possible implementation, the frame interpolation is performed in an RGB color space, each of the set of first feature maps and the set of second feature maps includes red channel information, blue channel information and green channel information. One or more first optical flows with a resolution (which may be referred to as a sixth resolution) higher than the lowest resolution and one or more second optical flows with the sixthresolution, may be obtained based on the red channel information, the blue channel information and the green channel information in one or more first feature maps and one or more second feature maps with a resolution lower than the sixth resolution. As mentioned before, the lowest resolution is determined by the pyramidal encoder based on the input resolution and a resolution threshold. In a case that the input resolution is 1024x768 and the resolution threshold is 32x32, 32x24 may be a specific example of the lowest resolution. When the frame interpolation is performed in an RGB color space, an RGB model is used for generating the intermediate frame(s) and the confidence value(s) corresponding to the intermediate frame(s). The encoding and decoding processing for the RGB model are performed based on three-channel information. As mentioned before, the RGB model can be used for synthetic looking scenes with simplified graphics.

[0238] In a possible implementation, the frame interpolation is performed in a YUV color space, each of the set of first feature maps and the set of second feature maps includes gray channel information, chroma-blue channel information and chroma-red channel information, one or more first optical flows with a resolution (which may be referred to as a seventh resolution) higher than the lowest resolution and one or more second optical flows with the seventh resolution may be obtained based on the gray channel information of one or more first feature maps and one or more second feature maps with a resolution of the gray channel information lower than the seventh resolution. As mentioned before, the lowest resolution is determined by the pyramidal encoder based on the input resolution and a resolution threshold. In a case that the input resolution is 1024x768 and the resolution threshold is 32x32, 32x24 may be a specific example of the lowest resolution. When the frame interpolation is performed in a YUV color space, a YUV model is used for generating the intermediate frame and the confidence value corresponding to the intermediate frame. The encoding and decoding processing for the YUV model are performed based on singlechannel information. As mentioned before, the YUV model can be used for natural looking scenes with complex textures and geometry. Compared with the RGB model with the same interpolation mode, the YUV model may be more efficient in terms of data processing.

[0239] In order to elaborate the method for frame interpolation of the present disclosure more clearly, in the following, specific examples will be given for elaborating the frame interpolation solution of the present disclosure more clearly. It should be noted that the model architecture is illustrative but not restrictive.

[0240] The frame interpolation solution may be integrated into a cloud media service which initially includes a rendering component and a streaming component. FIG. 6 illustrates components for the cloud media service. A frame interpolation component is inserted into this pipeline between the rendering component and the streaming component.Frame interpolation takes frames from the rendering component, generates a stream of frames with doubled or tripled frame rate, and pass frames to the streaming component that encodes stream and transfer it over the network.

[0241] The frame interpolation component may have an API available to a user where the user can select one of four models: RGB X2 model, RGB X3 model, YUV X2 model, YUV X3 model; and a confidence threshold: a value from zero to one. The user may adjust options live and visually control the quality of the output stream.

[0242] The frame interpolation component may include a pre-processing phase, a model inference phase, a postprocessing phase, and a controller that switches models and keeps settings for a confidence threshold Cminaccording to user settings.

[0243] At the pre-processing phase, input frames can be converted to required format, if conversion is needed. Then converted tensors are padded with zero values to align width and height to 2n, where n is a depth of decoder’s pyramid.

[0244] For RGB X2ZX3 model, input frame ftis converted to 3-channel tensor Itwith channels corresponding to red, green, and blue color planes. Depending on inference backend used in particular implementation, the input tensor is converted to CHW or HWC formats and scaled to required dynamic range. Then tensor is padded with zero values to align up tensor sizes to 2n, where n is a depth of decoder’s pyramid.

[0245] For YUV X2 / X3 model, input frame ftis converted to 3 single-channel tensors Yt, Ut, Vt, where Ytkeeps gray plane, Ut- chroma-blue, and Vt- chroma-red planes. Then tensors are converted to CHW or HWC format scaling dynamic range to required one. Then tensors are padded with zero values to align up tensor sizes. For tensor Ytthe required alignment is 2n. For tensors Utand Vtthe required alignment is 2”-1, where n is a depth of decoder’s pyramid.

[0246] At the model inference phase, the neural network model is inferenced and output tensors are generated.

[0247] RGB X2 model takes image tensors Io= It.x, Ix= Itand produces an image tensor I1 / 2and a confidence tensor Cj / 2, where is a previous image tensor and Itis a current image tensor.

[0248] RGB X3 model takes image tensors Io= It-i, h = 4 and produces image tensors / 1 / 3, / 2 / 3 and confidence tensors C1 / 3, C2 / 3.

[0249] YUV X2 model takes tensors Ya= Yt-1, U0= Ut-i, V0= Vt-1, Yt = Yt, Ux= Ut, V1= Vtand produces image tensors Yj / 2, U1 / 2 , V-L / Z and confidence tensor C1 / 2, where Yt-t, U^ , are previous image tensors, and Yt, Ut, Vtare current image tensors.

[0250] YUV X3 model takes tensors Yo= Y^, Uo= Ut-t, Vo= Vt-i , >1 = Yt, Ut= Ut, Vi = Vtandproduces image tensors Y3 / 3, U1 / 3, V1^3, Y2 / 3, U2 / 3, ^2 / 3 and confidence tensor C3 / 3, C2 / 3.

[0251] At the post-processing phase, output tensors are converted to frames, cropped to original size, and returned from component along with a previous frame.]0252] For RGB X2 model, if confidence C1 / 2is greater than or equal to user-specified confidence threshold Cmin, a 3-channel output tensor l1 / 2is cropped to match input frame size and then converted to RGB frame where tensor channels correspond to red, green, and blue color planes. Component produces two frames: andA-1 / 2 where is a previous frame. If confidence C3 / 2is less than confidence threshold Cmin, the previous frame is repeated with updated timestamp, so component produces and ft-1 / 2=ft-i-

[0253] For RGB X3 model, if both values C1 / 3and C2 / 3 are greater than or equal to Cmin, two 3-channel output tensors and l2 / 3are cropped to match input frame size and then converted into two RGB frames ft-2 / 3and ft-1 / 3 with channels corresponding to red, green, and blue color planes. Component produces three frames: ft-t, ft—2 / 3, ft-i / 3- But if at least one confidence value C1 / 3or C2 / 3is less than Cmin, the previous frame is repeated twice with updated timestamps, so component produces ft-2 / 3— ft-i, ft-1 / 3=ft-i-

[0254] For YUV X2 model, if confidence C1 / 2> Cmin, three single channel tensors F1 / 2, U1 / 2, V1 / 2are cropped to match input frame size and then converted to YUV frame ft-1 / 2with tensors corresponding to gray, chroma-blue, and chroma-red planes. Component produces two frames: ft-3and ft-1 / 2. If confidence C1 / 2< Cmin, component repeats the previous frame with updated timestamp, so it produces ft-rand ft_ i / 2= ft-3.

[0255] For YUV X3 model, if both C1 / 3> Cminand C2 / 3> Cmin, six single-channel tensors X1 / 3, U1 / 3, V1 / 3and f2 / 3, U2 / 3, V2 / 3are cropped to match input frame size and then converted to two YUV frames ft-2 / 3and ft-1 / 3 with tensors corresponding to gray, chroma-blue, and chroma-red planes. Component produces three frames: ft-x, ft-2 / 3, ft-i / 3 - But if at least one confidence is less than the threshold, that is C1 / 3< Cminor C2 / 3 < Cmin, component repeats the previous frame twice updating timestamps and producing ft-t, ft-2 / 3= ft-i, ft-1 / 3=ft-i-

[0256] Frame and tensor dimensions, alignment, padding and crop rectangle for the case of Full HD video are as follows.Table 2

[0257] Next, specific structures for RGB X2 / X3 model will be described.

[0258] RGB X2 / X3 Model

[0259] FIG. 7 illustrates a block diagram of RGB X2 model, while FIG. 8 illustrates a block diagram of RGB X3 model. Model takes two frames Ioand fx, each frame is a 3-channel tensor, the first 3 channels are red, green and blue components. RGB X2 model produces one intermediate frame I1 / 2, while RGB X3 model produces two intermediate frames / 3, / 2 / 3- In addition to intermediate frames, model produces one confidence value C1 / 2or two confidence values Clj3lC2 / 3. Model includes a pyramidal (feature) encoder, a chain of (feature) decoders, an image decoder, and a confidence estimator (or referred to as a confidence estimation block).

[0260] Both input frames Ioand Ixare passed through a pyramidal encoder that produces feature maps. Pyramid depth depends on input resolution. Encoder is adding new level of pyramids by down sampling image while current image size is greater than 32x32, it should be noted that, 32x32 is only illustrative but not restrictive, it may be other values in some cases, e.g., 16x16. For example, if the input resolution is 1024x768, the encoder produces aresolutions 512x384, 256x192, 128x96, 64x48, 32x24. Specifics of encoder work is that feature maps of level 0 have one number of channels while feature maps for levels 1, 2, 3, 4 have the other number of channels. For example, for RGB model, number of channels is 6 for level 0 and 18 for levels 1, 2, 3, 4. More details about the encoder implementation will be introduced later.pairs of flows (the aforementioned optical flows), one or two intermediate feature maps, and one or two sets of selection masks depending on if it is X2 or X3 model.

[0262] For RGB X2 model, the bottom decoder Df4)(a specific example of the aforementioned bottom-levelF^1 / 2and selection masks M^^2, M^2come in a resolution of one level-up in the pyramid. So, they are compatible with feature maps fg3>and in terms of spatial dimensions - width and height In addition, intermediate feature map f3*2has same number of channels as fg3>and

[0263] For RGB X3 model, the bottom decoder £>(4)takes encoded feature maps / 0(4)and y^4', and produces two pairs of forward and backward flows: Fo(42 / 3, ^1^2 / 3’tw0intermediate feature maps , / 243, two pairs of flow selection masks M34)1 / 3, Af^2 / 3, M^2 / 3, and two feature selection masks M^3,M243. Here element-wise sum of Mg^^3, M^1 / 3, and M^3should be “all-ones” tensor. And element-wise sum of ^0^2 / 3’Wi^2 / 3’M2 / 3sh°uld“all-ones” tensor. Masks regulate level of trust to respective sources of information.

[0264] A middle decoder D(3)(a specific example of the aforementioned middle-level decoder) takes warped feature maps of its level using flows produced by D(4), intermediate feature maps decoded by f>(4), and selection masks. Decoder D(3)produces flow residuals-corrections (the aforementioned optical flow residual), intermediate feature maps and selection masks.

[0265] For RGB X2 model, the decoder D(3)takes the following inputs: warped feature maps / 0(3)1 / 2=warP( / of3).FoC4>)i / 2) , / j^i / 2 = warp(f33),F^1 / 2), intermediate feature / j(42, flow selection masks Mg^1 / 2, M^1 / 2, and feature selection mask Decoder D(3)produces forward and backward flow residuals AF®1 / 2, AF® ,,, intermediate feature f.^, flow selection masks for forward and backward flows and feature selection mask

[0266] For RGB X3 model, the decoder D(3)takes warped feature maps f0^1 / 3= warp( / 0(3),F0(4)1 / 3), fi^i / 3= warP(fi3)>F^i / 3^ fot)2 / 3= warPOo'3)-Fot)2 / 3^ fi%3 =mrP(fi3)’^2 / 3)’decodedintermediate features f^3, f2*3, flow selection masks M34)1 / 3, Af^2 / 3, ^1^2 / 3’ featureselection masks M^3,. The decoder D<3>produces two pairs of flow residuals AF®1 / 3, AF®^ , AF®2 / 3, AF^^ , two intermediate feature maps f^33, f233, two pair of flow selection masks Mg^1 / 3, ^^1 / 3’ ^0^2 / 3’ ^1^2 / 3andtwo feature selection masks M233, M233.

[0267] Another middle decoder D(2>(a specific example of the aforementioned middle-level decoder) takes warped feature maps of its level using up-scaled flows from D(4)and corrected by residuals from D<3>, intermediate feature maps decoded by D(3>, selection masks. The decoder D(2)produces flow residuals, intermediate feature maps and selection masks.

[0268] For RGB X2 model, the decoder O(2)takes feature maps warped using corrected flows / 0(^1 / 2—two decoded intermediate feature maps and f&3, flow selection masks Af^1 / 3, M^1 / 3, M®2 / 3, M®2 / ,3, and feature selection masks M^3, M® . The decoder Dmproduces flow residuals AF0{^1 / 3, AF^1 / 3, AF0c^2 / 3, AF®2 / 3, intermediate features f^3, f2*3, flow selection masks M^1 / 3, M^J1 / 3, Mg22 / 3- Af3^2 / 3, feature selection masks M^3, M2 / 3-

[0273] The image decoder D(o)(a specific example of the aforementioned top-level decoder) does intermediate image decoding with refinement. This is the last decoder in the chain. It takes warped feature maps of zero-level using up-scaled and corrected flows, intermediate feature maps and selection masks from the decoder D'11. The decoder D<0>produces intermediate images which are the actual output of the network.

[0274] For RGB X2 model, the decoder D'-' takes feature maps warped using corrected flows f^1 / 2— «arp( / 0(0),F0® / 2) , f1%2= vvarp( / 1(0),F1(‘)1 / 2) , where F™1 / 2= upflow (FO® / 2) + AFO(^1 / 2, and F^jyj = upflow (F®1 / 2) + AF3^1 / 2. Also, the decoder D(o)takes intermediate feature map / j^2, flow selection masks M^1 / 2, and a feature selection mask M^2. The decoder D<0' produces an intermediate image I1 / 2that is one of model outputs.

[0275] For RGB X3 model, the decoder D(o)takes feature maps warped using corrected flows f^1 / 3= warp( / 0(0\F0(^1 / 3), f^\ / 3= warp( / 1(0),F™1 / 3), f™2 / 3= warp( / 0(0),F™2 / 3) , f^2 / 3= warp( / j(0),F™2 / 3), where F™1 / 3= upflow (F®1 / 3) -F AF^ , F™1 / 3= upflow (F®1 / 3) + AF™1 / 3Rnl? / 3 = upflow (Fn®2„) + AFn(V, / , and F.™ = upflow + AE^,,,. Also, the decoder D'0)takes two decoded intermediate feature maps f3,3and f2j3, flow selection masks Mg2i / 3>Mit\ / 3> ^0^2 / 3* ^1^2 / 3’ and feature selection masks M3J3, M2*3. The decoder D(o)produces two intermediate images 4 / 3and I2 / 3that are ones of model outputs.

[0276] The confidence estimator is attached to the first decoder in the chain which has output resolution greater than or equal to 128x128 by both spatial dimensions - width and height except the zero-level. The confidence estimator takes warped features of specific level and analyze its difference and produce confidence values. It should be noted that, the value 128x128 is illustrative but not restrictive.

[0277] For RGB X2 model, the confidence estimator is attached to the output of decoder where i > 0. It takes warped feature maps using corrected flows f^i / z= warP( / o°>Fo^i / 2)’ A^i / 2 = warpC / ^F^1^), where Fg^2, and F^ffy is computed as it is stated in the description of the decoder D(l+ 1). The Confidence Estimator produces a confidence value C1 / 2- one of model outputs.

[0278] For RGB X3 model, the confidence estimator is attached to the output of decoder D(‘) where i > 0. Itas it is stated in the description of the decoder D<1+1>. The confidence estimator produces two confidence values Cx / 3, C2 / 3- ones of model outputs.

[0279] Tensor dimensions for RGB X2 model with input resolution 1024x768 are as follows.Table 3

[0280] Tensor dimensions for RGB X3 model with input resolution 1024x768 are as follows.Table 4|0281] Encoder ((or referred to as a pyramidal encoder, an encoder block) takes one of input frame Ik(k = 0, k = 1) and produces feature map pyramid. The depth depends on input resolution. As it was previously mentioned, the encoder adds new level of pyramids by down sampling image while image size is greater than 32x32 in both directions. For example, if input resolution is 1024x768, the encoder produces a five-level pyramid: fk°\ ff, fff, ff. A?’ with resolutions 512x384, 256x192, 128x96, 64x48, 32x24.

[0282] FIG. 9 illustrates a block diagram of a pyramidal encoder. Pyramidal encoder uses two types of encoders and E® arranged in a cascade, where i denotes pyramid level: 0, 1, 2, ... At the top level of pyramid i = 0, the encoder block uses just one encoder Ef that takes input image Ikand produces features of the zero level fk°\ For every next level of pyramid i > 0, the image is down sampled by factor of two = down and is passed to the encoder Ef that produces first part of the features for this level The second part of features fk:2)are produced by the second encoder E25that takes first part of features of the previous level as input. When referring zero level, it is considered if = Ik, fko r>=ff ■ Output features for the layer ff are concatenated from two parts and fkl:2\ Encoders Ef and Ef have the same structure: two convolutions3x3, the first has stride 2, padding 1, and the second has stride 1, padding 1. Encoder E® takes 3-channel tensor - the image, and produces 6-channel tensor - features fk^. Encoder E® takes 6-channel tensor fk~i r>and produces 12-channel tensor ff2\ Each encoder down samples resolution by factor of two because first convolution has stride two. All instances of encoders E® have shared weights. Same for encoders E^ - they have shared weights.

[0283] Tensor dimensions for the encoder with input resolution 1024x768 are as follows.

[0284] FIG. 10 illustrates a block diagram of the bottom decoder in RGB X2 model, while FIG. 11 illustrates a block diagram of the bottom decoder in RGB X3 model. The bottom decoder takes encoded feature maps - for input resolution 1024x768, these features are fj4)and f*4\ The decoder concatenates them and passes to convolution 1x1 extending feature space to the number of features required to produce output. Those features are passed to a sequence of two processing blocks. Each block consists of three 3x3 convolution blocks with stride 1, padding 1 followed by LRELU. Each processing block has residual connection that bypass block input and add it to output of the last (third) convolution before passing it to LRELU. After processing blocks, an additional block of 1x1 convolution adapts the dynamic range and forms an aggregated output tensor 0(4).

[0285] For RGB X2 model and input resolution 1024x768, the aggregated output tensor 0(4)is split into 4 tensors: O^4\ 024\ 024\ 0^4>. First two tensors o[4), 024^ have features for flows. The third tensor O24)haswhich increase resolution by factor of two and lowering depth by factor of four. Then, upscaled tensors for flows are passed to hyperbolic tangent layer followed by layer that scales value by factor of 16. These are output flow tensorsand is split into output masks Mg421 / 2, ^±21 / 2’ ^1 / 2 ■ Below are dimensions of internal tensors before shuffle and splitting one of tensors to masks. In addition, the table contains respective output tensors.Table 6

[0286] For RGB X3 model and input resolution 1024x768, the aggregated output tensor 0(4)is split into 8 tensors: o54), 0,4\ O^4\ o^4\ 0?4\ 0<4), 0,4), and 0„4\ Tensors O.(4), 0„(4)have features for the first pair of flows. Those tensors are upscaled using shuffle layer, then processed by hyperbolic tangent layer and multiplied byTable 7

[0287] FIG. 12 illustrates a block diagram of the middle and the top decoders in RGB X2 model, while FIG. 13 illustrates a block diagram of the middle and the top decoders in RGB X3 model. Middle and top decoders take

[0288] Then, the decoder concatenates warped input features and fused features into one tensor and passes them to convolution 1x1 extending feature space to the number of features required to produce output. Those features are passed to a sequence of two processing block similar to the bottom decoder. So, each processing block consists of three 3x3 convolution blocks with stride 1, padding 1 followed by LRELU. Each block has residual connection that bypass block input to and add it to output of the last (third) convolution before passing it to LRELU. After two processing blocks, an additional block of 1x1 convolution adapts the dynamic range and forms an aggregated output tensor 0w.

[0289] For RGB X2 model and input resolution 1024x768, the aggregated output tensor 0wis split into 4decoded feature. The fourth tensor 0® have features for 3 masks. Tensors 0®, 0® are upscaled by shuffle layers and passed to hyperbolic tangent layer followed by layer that scales value by factor of 16. These are output flowM® I / 3> M®3ar)d ^O'2Z / 3’ M®Z / 3> M®3- Below are dimensions of internal tensors before shuffle and splitting one of tensors to masks. In addition, the table contains respective output tensors.Table 9

[0291] FIG. 14 illustrates a block diagram of the image decoder in RGB X2 model, while FIG. 15 illustrates a block diagram of the image decoder in RGB X3 model. Image decoder takes encoded feature maps of zero level

[0292] Then, the decoder concatenates warped input features and fused features into one tensor and passes them to convolution 1x1 extending feature space to the number of features required to produce output. Those features are passed to a sequence of two processing blocks similar to the bottom-level decoder. So, each processing block consists of three 3x3 convolution blocks with stride 1, padding 1 followed by LRELU. Each block has residual connection that bypass block input to and add it to output of the last (third) convolution before passing it to LRELU. After two processing blocks, an additional block of 1x1 convolution adapts the dynamic range and forms an aggregated output tensor O(o).

[0293] For RGB X2 model and input resolution 1024x768. The aggregated output tensor O(0>is upscaled by shuffle layer and returned as the output image Ijp = shuffle(O(0)). Table 10 illustrates corresponding dimensions of internal tensors and output tensors.Table 10

[0294] For RGB X3 model and input resolution 1024x768, the aggregated output tensor O(o)is split into 2 tensors: O®\ O®\ Each tensor is upscaled by a shuffle layer and outputs predicted images. So two images are provided l1 / 3= shuffle(Oj0:)), and / 2 / 3 — shuffle(O20)). Table 11 illustrates corresponding dimensions of internal tensors and output tensors.Table 11

[0295] FIG. 16 illustrates a block diagram of the confidence estimator in RGB X2 model. For each output image, the confidence estimator takes pair of warped features at specific pyramid level and produces a confidence tensor. For RGB X2 model, the confidence estimator takes f^i / z’anc^ produce Ci / z- For RGB X3 model, the confidence estimator takes / 0®1 / 3, Zi-^i / 3’ folz / y fv^z / z and produce C1 / 3, C2 / 3. Pyramid level I at which the confidence estimator is attached to the pyramid is defined by resolution of warped features: the resolution should be greater than or equal to 128x128 by both dimensions. If the input resolution is 1024x768, then encoded feature maps have the following resolutions: level 0 - 6x384x512, level 1 - 18x192x256, level 2 - 18x96x128, level 3 - 18x48x64, level 4 - 18x24x32. So, level of attachment I = 1.

[0296] For RGB X2 model, a pair of warped features fi2i / zaredownscaled, then magnitudes are computed for each downscaled tensor magn^y2= magn(down( / 0®1 / 2)) , magn^^2— magn(down( / j®1 / 2)). The error map is computed as a magnitude of element-wise difference of downscaled feature maps err^1;= magn(down( / 0®1 / 2) — down( / t®1 / 2)). Then error map is passed through two Error Reduction blocks. First error reduction block takes error map err^*11as input, computes maxpoo\(err^2and avgpoolCerr^1)) using 2x2 window with zero padding and builds an output error tensor err^^l 2>with elements from maxpool result tensor if the element value is greater than 0.4 of respective element from avgpool result tensor, and with zero values overwise. Second reduction block does the same processing taking err^' / l22>as input andproducing err^? as an output. Then relative error is computed as a ratio of maximum error to maximum feature magnitude: re / err1 / 2= maxerr1 / 2 / maxmagnr / 2where maxerr1 / 2= maxCerT^2)[0, i, / ]) , and maxma5n1 / 2= max(max(ma5n^iy2[0, i,;]) , niax(ma5ng^2[0, i, / ])) . In addition, a kind of contrast is. , ~ . avamaon, / , , estimated according to formula: contrast^ = 1 —maxmagn> where avgmang^2=. Finally, the confidence is computed according to formula C1 / 2= clamp[0 11(l - 0.3 x relerr1 / 2x k1 / 2), where fc1 / 2= 0.3 + clamp[o,o.7](l - contrast1 / 2). For low-contrast scene contrast1 / 2« 0.0, the impact of relative error is high fc1 / 2« 1.0. For high-contrast scene (contrast1 / 2~ 1.0), the impact is low k1 / 2~ 0.3.

[0297] Tensor dimensions for RGB X2 model with input resolution 1024x768 attached at I = 1 are as follows.Then each pair is processed independently using exactly the same processing pipeline as for RGB X2 model and produce confidence tensors C1 / 3, C2 / 3. Input, intermediate and output tensors have same dimensions in the RGB X2 model case.

[0299] In the aforementioned illustrative block diagrams for RGB X2 / X3 model, some symbols or marks in thevii) LRELU: Leaky RELU layer. viii) Tanh: hyperbolic tangent layer activation layer. ix) Softmax: softmax activation layer. x) Convlxl: Convolution lxl with stride 1 padding 0. xi) Convlxl + LRELU'. Convolution lxl with stride 1 padding 0, followed by LRELU activation. xii) Conv3x3: Convolution 3x3 with stride 1 padding 1. xiii) Conv3x3s2: Convolution 3x3 with stride 2 padding 1. xiv) Conv3x3s2 + LRELU: Convolution 3x3 with stride 2 padding 1, followed by LRELU activation. xv) Shuffle: Pixel shuffle operation that expands channels into spatial dimensions increasing width and height by factor of two while decreasing number of channels by factor of four. xvi) Block "xl6”: Multiply input tensor by value 16. xvii) Block “x”: elementwise multiplication of input tensors. xviii) Block “E elementwise summation of input tensors. xix) Max: maximum of all elements. xx) Avg: average of all elements. xxi) MaxPool: a maximum pooling layer with 2x2 window, stride 1 and zero padding. xxii) AvgPool: an average pooling layer with 2x2 window, stride 1 and zero padding. xxiii) Clamp[aClamp every tensor element by values a and b byformula max(min(x, b) , a).

[0300] YUV X2 / X3 Model

[0301] Model takes six single-channel tensors representing two frames in YUV420 format Yo, Uo, Voand Yr, If, Vt, where Yoand Yrare gray components, Uoand If are chroma-blue components, and Vaand \f are chroma-red components. Chromatic components come in twice as lower resolution as gray components. YUV X2 model produces one intermediate frame encoded by three tensors T1 / 2, lf / 2, V1 / 2. YUV X3 model produces two intermediate frames encoded by six tensors K1 / 3, lf / 3, V1 / 3and Y2 / 3, U2 / 3, V2 / 3. In addition, the model produces confidence values: C1 / 2or C1 / 3, C2 / 3. Model architecture is similar to RGB X2 / X3 model with the following several modifications.

[0302] Instead of encoding three-channel image tensors, YUV model encodes a single-channel tensor (gray) using a smaller number of features: 4 and 6 channels for the first and the second types of encoders. Decoders produce features, flows and masks in same way as RGB model but with different number of features. Image decoder produce output gray tensors: Y1 / 2for X2 model or Yr / 3, Y2 / 3for X3 model. Confidence estimator works same as in RGB model but operating with tensors of different dimensions. Top-level flows and masks are used to warp and blend chromatic channels.

[0303] Tensor dimensions for YUV X2 model with input resolution 1024x768 are as follows.Table 13

[0304] Tensor dimensions for YUV X3 model with input resolution 1024x768 are as follows.Table 14

[0305] Encoder works same as in RGB models but dimensions are different, see table 15.

[0306] Decoders works same as in RGB models but dimensions are different.(0307] For YUV X2 model with input resolution 1024x768, corresponding dimensions of internal and output tensors are as follows.Table 16

[0308] For YUV X3 model and input resolution 1024x768, corresponding dimensions of internal and output tensors are as follows.Table 17

[0309] Chromatic channels are warped and blended according to formulas similar to gray channel with one exception: half of mask for previously decoded features are added to flow selection masks because there is no previously decoded features for chromatic channels.

[0310] For YUV X2 model, output chromatic channels are computed according to formulas:V1 / 2= warp(v0,^1 / 2) x (M£1 / 2+ |MVZ)+^(^F^) X (M™ / 2+ |«$)

[0311] For YUV X3 model, output chromatic channels are computed according to formulas:

[0312] YUV model uses YUV frame format and performs interpolation on gray channel only using smaller number of features in encoder / decoder pyramids. Then, it may apply motion estimation results to chromatic channels processing them in twice as lower resolution. Processing in YUV color space has xl.3 times speed-up for x2 interpolation and xl.5 speed-up for x3 interpolation compared to processing in RGB color space.

[0313] At the same time, the frame interpolation solution also leaves user an option to operate in RGB color space using separate models for that. Recommendations for model selection are: use YUV model for natural looking scenes with complex textures and geometry, and use RGB color space for synthetic looking scenes with simplified graphics.

[0314] In addition to x2 interpolation mode, x3 interpolation mode effectively reuses computed data to produce two intermediate frames instead of one, and do feature decoding in more optimal way. Using x3 interpolation let us gain bigger increase of number of rendering service compared to x2 interpolation at cost of affordable quality decrease.

[0315] Confidence estimation algorithm that automatically disables frame interpolation for frames with low prediction confidence value and simply repeats the previous frame. The confidence estimation algorithm takes difference between two feature maps warped by decoded optical flows, normalizes that difference to feature magnitudes, aggregates them into a single value and adjust that value to the scene contrast. Finally, that aggregated and adjusted error is converted into a confidence value in the range from zero to one. This value is used in the ghostprevention mechanism. If the estimated confidence is less than user specified threshold, then interpolated frames are ignored and previous frame is repeated. This automatic quality control mechanism temporally drops frame rate to input rate in case if ghosting is expected. This mechanism lets user control a tradeoff between smoothness and level of ghosting in case of large scene changes. In general, this option lets the user leave the rendering service running unattended. That is very important for cloud rendering scenario.|0316] Table 18 illustrates potential increase An of the number of instances when using frame interpolation models of the present disclosure, where tris render time and tflis frame interpolation inference time in seconds. Measurements are performed on a same computing platform. Quality is estimated on a complex proprietary dataset published for the purpose of comparison between models (not for the purpose of comparison with the aforementioned frame interpolation solutions). Table 19 illustrates potential increase An of the number of instances when using some aforementioned frame interpolation solutions based on the same computing platform. It can be seen from the comparison of table 18 and table 19, compared to the aforementioned frame interpolation solutions, the number of instances can be increased with frame interpolation models of the present disclosure.Table 18Table 19

[0317] It should be noted that, besides cloud rendering scenario, frame interpolation solutions of the present disclosure can also be applied to video playing scenarios, e.g., run frame interpolation when playing movie on smart TV sets to increase smoothness of the video, run frame interpolation in video players on both desktop and mobile devices, etc. The frame interpolation solutions can also be applied to video editing scenarios, e.g., increase video smoothness by increasing frame rate of some old video (25 / 30 fps — > 50 / 60 fps or even 25 / 30 fps — > 75 / 90 fps),generate slow-motion effect, etc. The frame interpolation solutions can also be applied to minimize traffic for video conferencing, e.g., network traffic can be reduced by factor of two without significant lose in quality by inserting frame interpolation component on a receiving side and making sender side capture and code video with twice as lower frame rate as original.

[0318] Next, embodiments of products related to the method for frame interpolation will be described.

[0319] FIG. 17 shows a schematic structural diagram of an apparatus for frame interpolation according to one or more embodiments of the present disclosure. As shown in FIG. 17, the apparatus for frame interpolation 1700 may include: a first obtaining module 1702, configured to obtain a previous frame and a current frame of a video stream; a second obtaining module 1704, configured to obtain one or more intermediate frames based on the previous frame and the current frame, and obtain one or more confidence values respectively corresponding to the one or more intermediate frames; and an interpolating module 1706, configured to in a case that the one or more confidence values satisfy a pre-defined condition, interpolate the one or more intermediate frames between the previous frame and the current frame.

[0320] It should be noted that, the first obtaining module and the second obtaining module are only illustrative for their functions, and in practice, the functions of the first obtaining module and the second obtaining module may be implemented by one obtaining module, which is not limited here. It should also be noted that, the functions of the obtaining modules and the interpolating module may also be implemented by a single module. The apparatus may be in a form of a computing device, an electronic device, etc.

[0321] In a possible implementation, the second obtaining module includes: a pyramidal encoder, configured to obtain a first feature pyramid including a set of first feature maps with different resolutions for the previous frame and a second feature pyramid including a set of second feature maps with different resolutions for the current frame, where a first feature map and a second feature map for a same level have same resolution, a first feature map for a top level has a resolution of the top level and a first feature map for a bottom level has a resolution of the bottom level; and a chain of decoders, configured to obtain, according to the first feature pyramid and the second feature pyramid, the one or more intermediate frames, where intermediate feature maps with different resolutions of the oneor more intermediate frames are built based on the first feature pyramid and the second feature pyramid from the bottom level to the top level.

[0322] In a possible implementation, the chain of decoders includes a bottom-level decoder, and the bottomlevel decoder is configured to: obtain, according to a first feature map and a second feature map with the resolution of the bottom level, one or more first optical flows with a first resolution, one or more second optical flows with the first resolution, one or more intermediate feature maps with the first resolution, one or more first flow selection mask with the first resolution, one or more second flow selection mask with the first resolution, and one or more feature selection masks with the first resolution, where the first resolution is a resolution of a level which is higher than and next to the bottom level, where each of the one or more first optical flows with the first resolution is a forward optical flow corresponding to a respective one in the one or more intermediate frames, and each of the one or more second optical flows with the first resolution is a backward optical flow corresponding to a respective one in the one or more intermediate frames; where the one or more first optical flows with the first resolution, the one or more second optical flows with the first resolution, the one or more intermediate feature maps with the first resolution, the one or more first flow selection mask with the first resolution, the one or more second flow selection mask with the first resolution, and the one or more feature selection masks with the first resolution are used for producing the one or more intermediate frames.

[0323] In a possible implementation, the bottom-level decoder includes: a first convolution layer, configured to: obtain, according to a first input tensor, a first extended tensor, where the first input tensor is obtained according to the first feature map and the second feature map with the resolution of the bottom level; where a quantity of channels of the first extended tensor is greater than a quantity of channels of the first input tensor.

[0324] In a possible implementation, the bottom-level decoder further includes: a first shuffle layer module, configured to up scale tensors split from a first aggregated output tensor to produce up-scaled tensors with the first resolution; where the first aggregated output tensor is obtained according to the first extended tensor; where the up-scaled tensors with the first resolution are used for producing the one or more first opticalflows with the first resolution, the one or more second optical flows with the first resolution, the one or more intermediate feature maps with the first resolution, the one or more first flow selection masks with the first resolution, the one or more second flow selection masks with the first resolution, and the one or more feature selection masks with the first resolution.

[0325] In a possible implementation, the bottom-level decoder further includes: a first tangent layer module, configured to apply a first hyperbolic tangent function to produce, according to one or more first up-scaled tensors of the up-scaled tensors with the first resolution, the one or more first optical flows with the first resolution, and produce, according to one or more second up-scaled tensors of the up-scaled tensors with the first resolution, the one or more second optical flows with the first resolution; and / or a first activation layer, configured to apply a first softmax function to produce, according to one or more third up-scaled tensors of the up-scaled tensors with the first resolution, the one or more first flow selection masks with the first resolution, the one or more second flow selection masks with the first resolution, and the one or more feature selection masks with the first resolution.

[0326] In a possible implementation, the chain of decoders includes one or more middle-level decoders, and the one or more middle-level decoders are configured to: obtain one or more first warped feature maps with a middle resolution by warping a first feature map with the middle resolution using one or more first optical flows with the middle resolution, where the middle resolution is higher than the resolution of the bottom level and lower than the resolution of the top level; obtain one or more second warped feature maps with the middle resolution by warping the second feature map with the middle resolution using one or more second optical flows with the middle resolution; and obtain one or more first optical flow residuals with a second resolution, one or more second optical flow residuals with the second resolution, one or more intermediate feature maps with the second resolution, one or more first flow selection masks with the second resolution, one or more second flow selection masks with the second resolution, and one or more feature selection masks with the second resolution according to one or more intermediate feature maps with the middle resolution, the one or more first warped feature maps with the middle resolution, the one or more second warped feature maps with the middle resolution, one or more first flow selection masks with the middle resolution, one or more second flow selection masks with the middle resolution, and one or more feature selection masks with the middle resolution, where the second resolution is a resolution of a level higher than and next to a level of the middle resolution;where the one or more first optical flow residuals with the second resolution, the one or more second optical flow residuals with the second resolution, the one or more intermediate feature maps with the second resolution, the one or more first flow selection masks with the second resolution, the one or more second flow selection masks with the second resolution, and the one or more feature selection masks with the second resolution are used for producing the one or more intermediate frames.

[0327] In a possible implementation, each of the one or more middle-level decoders includes: a second convolution layer, configured to: obtain, according to a second input tensor, a second extended tensor, where the second input tensor is obtained according to the one or more intermediate feature maps with the middle resolution, the one or more first warped feature maps with the middle resolution, the one or more second warped feature maps with the middle resolution, the one or more first flow selection masks with the middle resolution, the one or more second flow selection masks with the middle resolution, and the one or more feature selection masks with the middle resolution; where a quantity of channels of the second extended tensor is greater than a quantity of channels of the second input tensor.

[0328] In a possible implementation, each of the one or more middle-level decoders further includes: a second shuffle layer module, configured to up scale tensors split from a second aggregated output tensor to produce up-scaled tensors with the second resolution; where the second aggregated output tensor is obtained according to the second extended tensor; where the up-scaled tensors with the second resolution are used for producing the one or more intermediate feature maps with the second resolution, the one or more first optical flow residuals with the second resolution, the one or more second optical flow residuals with the second resolution, the one or more first flow selection masks with the second resolution, the one or more second flow selection masks with the second resolution, and the one or more feature selection masks with the second resolution.

[0329] In a possible implementation, each of the one or more middle-level decoders further includes: a second tangent layer module, configured to apply a second hyperbolic tangent function to produce, according to one or more first up-scaled tensors of the up-scaled tensors with the second resolution, the one or more first optical flow residuals with the second resolution, and produce, according to one or more second up-scaled tensors of the up-scaled tensors with the second resolution, the one or more second optical flow residuals with the second resolution; and / ora second activation layer, configured to apply a second softmax function to produce, according to one or more third up-scaled tensors of the up-scaled tensors with the second resolution, the one or more first flow selection masks with the second resolution, the one or more second flow selection masks with the second resolution, and the one or more feature selection masks with the second resolution.

[0330] In a possible implementation, the chain of decoders includes a top-level decoder, and the top-level decoder is configured to: obtain one or more first warped feature maps with the resolution of the top level by warping the first feature map with the resolution of the top level using one or more first optical flows with the resolution of the top level; obtain one or more second warped feature maps with the resolution of the top level by warping the second feature map with the resolution of the top level using one or more second optical flows with the resolution of the top level; and obtain the one or more intermediate frames according to one or more intermediate feature maps with the resolution of the top level, the one or more first warped feature maps with the resolution of the top level, the one or more second warped feature maps with the resolution of the top level, one or more first flow selection masks with the resolution of the top level, one or more second flow selection masks with the resolution of the top level, and one or more feature selection masks with the resolution of the top level.

[0331] In a possible implementation, the apparatus further includes an optical flow correcting module, and the optical flow correcting module is configured to: obtain one or more first optical flows with a third resolution by up-scaling one or more first optical flows with a fourth resolution to obtain one or more up-scaled first optical flows with the third resolution; and correcting the one or more up-scaled first optical flows with the third resolution by one or more first optical flow residuals with the third resolution; and obtain one or more second optical flows with the third resolution by up-scaling one or more second optical flows with the fourth resolution to obtain one or more up- scaled second optical flows with the third resolution; and correcting the one or more up-scaled second optical flows with the third resolution by the one or more second optical flow residuals with the third resolution;where the third resolution is a resolution of a level higher than and next to a level of the fourth resolution, and the fourth resolution is higher than the resolution of the bottom level.

[0332] In a possible implementation, the second obtaining module includes a confidence estimator, and the confidence estimator is configured to: obtain the one or more confidence values respectively corresponding to the one or more intermediate frames based on one or more first warped feature maps with a fifth resolution and one or more second warped feature maps with a fifth resolution, where the fifth resolution is greater than or equal to a first resolution threshold.

[0333] In a possible implementation, the first resolution threshold is associated with a size of details to keep in the video stream

[0334] In a possible implementation, the pyramidal encoder is configured to: obtain the first feature map for the top level based on an image of the previous frame with an original resolution of the video stream; obtain a first feature map for a level next to the top level, where the first feature map for the level next to the top level includes a first part obtained based on a down-sampled image of the previous frame for the level next to the top level, and a second part obtained based on the first feature map for the top level, where the down-sampled image of the previous frame for the level next to the top level is obtained by down sampling the image for the previous frame with the original resolution by a pre-defined factor; and obtain a first feature map for each level of one or more levels lower than the level next to the top level, where the first feature map for the each level includes a first part obtained based on a down-sampled image of the previous frame for the each level, and a second part obtained based on a first part of a first feature map for a level upper than and next to the each level, where the down-sampled image of the previous frame for the each level is obtained by down sampling the down-sampled image of the previous frame for the level upper than and next to the each level by a pre-defined factor.

[0335] In a possible implementation, a quantity of levels of the first feature pyramid is based on an original resolution of the video stream and a second resolution threshold, where a resolution of the first feature map for the bottom level is lower than the second resolution threshold and a resolution of a first feature map for a level upper than and next to the bottom level is higher than the second resolution threshold.

[0336] In a possible implementation, the second resolution threshold is based a motion level between the previous frame and the current frame.

[0337] In a possible implementation, a first feature map and a second feature map for a same level have a same quantity of features, and a quantity of features for the first feature map of the top level is smaller than a quantity of features for a first feature map of a lower level.

[0338] In a possible implementation, a quantity of the one or more intermediate frames is based on an original frame rate of the video stream, and a taiget frame rate of the video stream.

[0339] In a possible implementation, a quantity of the one or more intermediate frames is based on a first operation of a user, where the first operation indicates a target frame rate of the video stream or the quantity of the one or more intermediate frames.

[0340] In a possible implementation, a color space in which to perform the frame interpolation is determined based on a characteristic of a scene of the video stream or a second operation of a user, where the second operation indicates the color space in which to perform the frame interpolation.

[0341] In a possible implementation, the interpolating module is further configured to in a case that the one or more confidence values do not satisfy the pre-defined condition, repeat the previous frame or the current frame one or more times.

[0342] In a possible implementation, the frame interpolation is performed in an RGB color space, each of the set of first feature maps and the set of second feature maps includes red channel information, blue channel information and green channel information, and one or more first optical flows with a sixth resolution and one or more second optical flows with the sixth resolution are obtained based on the red channel information, the blue channel information and the green channel information in one or more first feature maps and one or more second feature maps with a resolution lower than the sixth resolution, where the sixth resolution is higher than the resolution of the bottom level.

[0343] In a possible implementation, the frame interpolation is performed in a YUV color space, each of the set of first feature maps and the set of second feature maps includes gray channel information, chroma-blue channel information and chroma-red channel information, and one or more first optical flows with a seventh resolution and one or more second optical flows with the seventh resolution are obtained based on the gray channel information of one or more first feature maps and one or more second feature maps with a resolution of the gray channel information lower than the seventh resolution.

[0344] In a possible implementation, the one or more intermediate frames are obtained through a frame interpolation model, where the frame interpolation model is determined from multiple frame interpolation models.

[0345] In a possible implementation, the multiple frame interpolation models include a first frame interpolation model that doubles a frame rate of the video stream and performs frame interpolation in an RGB color space, or a second frame interpolation model that triples a frame rate of the video stream and performs frame interpolation in an RGB color space, or a third frame interpolation model that doubles a frame rate of the video stream and perform frame interpolation in a YUV color space, or a fourth frame interpolation model that triples a frame rate of the video stream and perform frame interpolation in a YUV color space.

[0346] In a possible implementation, the frame interpolation model is determined according to at least one of an original frame rate of the video stream, a target frame rate of the video stream, or a characteristic of scenes of the video stream.

[0347] In a possible implementation, the frame interpolation model is determined based on a third operation of a user, where the third operation indicates the frame interpolation model selected by the user.

[0348] The apparatus for frame interpolation may be the implemented as described in the above method embodiments. It should be understood by a person skilled in the art that, the relevant description of the above modules in the embodiments of the present disclosure may be understood with reference to the relevant description of the method for frame interpolation in the embodiments of the present disclosure.

[0349] FIG. 18 is a schematic structural diagram of a computing device according to one or more embodiments of the present disclosure. As shown in FIG. 18, the computing device 1800 includes a processor 1802, an interface 1804 for communicating with other devices, a memory 1806, and a bus 1808. The processor 1802, the interface 1804, and the memory 1806 are in communication with each other via the bus 1808. The memory 1806 may be stored with computer execution instructions, and the processor 1802 executes computer execution instructions stored in the memory 1806 to enable the computing device to execute any of the above methods. The computing device 1800 can be a server or a terminal device. The number of processors and memories in the computing device 1800 is not limited in the embodiments of the present disclosure.

[0350] The bus 1808 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in FIG. 18, but it does not mean that there is only one bus or one type of bus. The bus 1808 may include pathways for transferring information between various components of the computing device 1800 (e.g., the memory 1806, the processor 1802, or the interface 1804).

[0351] The processor 1802 can include any one or more of a central processing unit (CPU), a graphics processingunit (GPU), a microprocessor (MP), or a digital signal processor (DSP) processor.

[0352] The memory 1806 can include volatile memory, such as random access memory (RAM). The processor 1802 can also include non-volatile memory, such as read-only memory (ROM), flash memory, a mechanical hard disk (HDD), or a solid state drive (SSD).

[0353] An executable program code is stored in the memory 1806, and the processor 1802 executes the executable program code to implement the functions of the aforementioned first obtaining module 1702, the second obtaining module 1704, and the interpolating module 1706, respectively, so as to implement the method described in the above embodiments. That is, the memory 1806 has instructions on it for executing the method described in the above embodiments.

[0354] The structure of the computing device 1800 enumerated above is only an exemplary illustration, and the present disclosure is not limited thereto. The computing device 1800 of embodiments of the present application includes a variety of hardware found in computer systems in the prior art, e.g., the computing device 1800 also includes memory other than the memory 1806, such as, for example, a disk memory, and the like. It should be understood by those skilled in the art that the computing device 1800 can also include other devices necessary to achieve proper operation. In addition, depending on the specific needs, one of skill in the art should understand that the above-described computing device 1800 may also include other hardware devices for realizing other additional functions. In addition, one of skill in the art should understand that the above-described computing device 1800 may also include only the devices necessary to realize the embodiments of the present disclosure, and need not include all of the devices shown in FIG. 18.

[0355] FIG. 19 is a schematic diagram of an architecture of a computing device cluster according to one or more embodiments of the present disclosure. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0356] As shown in FIG. 19, the cluster of computing devices described includes at least one computing device 1900. The memory 1906 in one or more of the computing devices 1900 in the cluster of computing devices can hold the same instructions for performing the method described in the above embodiments.

[0357] In some possible implementations, some of the instructions for performing the method described in the above embodiments can also be separately held in the memory 1906 of the one or more computing devices 1900 inthe cluster of computing devices. In other words, a combination of the one or more computing devices 1900 can jointly execute instructions for performing the method described in the above embodiments.

[0358] It is noted that the memories 1906 in the different computing devices 1900 in the cluster of computing devices can store different instructions for performing some of the functions of the computing devices 1900, respectively. That is, the instructions stored in the memories 1906 in the different computing devices 1900 can realize the functions of one or more of the first obtaining module 1702, the second obtaining module 1704, and the interpolating module 1706.

[0359] In some possible implementations, one or more computing devices in a cluster of computing devices can be connected via a network. Among other things, the network can be a wide area network or a local area network, etc. FIG. 20 illustrates one possible implementation. FIG. 20 is a schematic diagram of a connection between computing devices 2000Aand 2000B over a network according to one or more embodiments of the present disclosure. The two computing devices 2000Aand 2000B are connected to each other via a network Specifically, the connection to said network is made through a communication interface in each computing device. In this class of possible implementations, the memory 2006A in the computing device 2000A holds instructions for performing the functions of the first obtaining module 1702 and the second obtaining module 1704. At the same time, the memory 2006B in the computing device 2000B holds instructions for performing the functions of the interpolating module 1706.

[0360] The connection between the clusters of computing devices shown in FIG. 20 can be such that, considering that the method provided in the present disclosure requires processing a large number of data, it is contemplated that the functions realized by the second obtaining module 1704 be performed by the computing device 2000B.

[0361] The functions of computing device 2000A illustrated in FIG. 20 can also be accomplished by multiple computing devices. Similarly, the functions of computing device 2000B can be accomplished by multiple computing devices.

[0362] In some aspects of the present disclosure, there is provided an apparatus for frame interpolation including processing circuitry for executing any of the above methods. It should be understood that the apparatus can execute the steps in the above method embodiments, which will not be repeated here.

[0363] In some aspects of the present disclosure, there is provided a computing device cluster including at least one computing device, and the computing device includes a processor and a memory coupled with the processor. The memory is storing instructions which, when executed by the processor, cause the computing device cluster to perform any of the above methods.

[0364] In some aspects of the present disclosure, there is provided a chip, including an input / output (I / O) interface and a processor, where the processor is configured to call and run computer execution instructions stored in a memory, to enable a device installing with the chip to execute any of the above methods.

[0365] In some aspects of the present disclosure, there is provided a computer-readable medium storing computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute any of the above methods.

[0366] In some aspects of the present disclosure, there is provided a computer program product including computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute any of the above methods.

[0367] In some aspects of the present disclosure, there is provided a computer program including computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute any of the above methods.

[0368] Although the present disclosure describes methods and processes with steps in a certain order, one or more steps of the methods and processes may be omitted or altered as appropriate. One or more steps may take place in an order other than that in which they are described, as appropriate.

[0369] Note that the expression “at least one of A or B”, as used herein, is interchangeable with the expression “A and / or B”. It refers to a list in which you may select A or B or both A and B. Similarly, “at least one of A, B, or C”, as used herein, is interchangeable with “A and / or B and / or C” or “A, B, and / or C”. It refers to a list in which you may select: A or B or C, or both A and B, or both A and C, or both B and C, or all of A, B and C. The same principle applies for longer lists having a same format.

[0370] Although the present disclosure is described, at least in part, in terms of methods, a person of ordinary skill in the art will understand that the present disclosure is also directed to the various components for performing at least some of the aspects and features of the described methods, be it by way of hardware components, software or any combination of the two. Accordingly, the technical solution of the present disclosure may be embodied in the form of a software product. A suitable software product may be stored in a pre-recorded storage device or other similar non-volatile or non-transitory computer readable medium, including DVDs, CD-ROMs, USB flash disk, a removable hard disk, or other storage media, for example. The software product includes instructions tangibly stored thereon that enable a processing device (e.g., a personal computer, a server, or a network device) to execute examples of the methods disclosed herein. The machine-executable instructions may be in the form of code sequences,configuration information, or other data, which, when executed, cause a machine (e.g., a processor or other processing device) to perform steps in a method according to examples of the present disclosure.

[0371] The present disclosure may be embodied in other specific forms without departing from the subject matter of the claims. The described example embodiments are to be considered in all respects as being only illustrative and not restrictive. Selected features from one or more of the above-described embodiments may be combined to create alternative embodiments not explicitly described, features suitable for such combinations being understood within the scope of this disclosure.

[0372] All values and sub-ranges within disclosed ranges are also disclosed. Also, although the systems, devices and processes disclosed and shown herein may include a specific number of elements / components, the systems, devices and assemblies could be modified to include additional or fewer of such elements / components. For example, although any of the elements / components disclosed may be referenced as being singular, the embodiments disclosed herein could be modified to include a plurality of such elements / components. The subject matter described herein intends to cover and embrace all suitable changes in technology.

[0373] Although embodiments have been described above with reference to the accompanying drawings, those of skill in the art will appreciate that variations and modifications may be made without departing from the scope thereof as defined by the appended claims.

Claims

CLAIMS1. A method for frame interpolation, comprising: obtaining a previous frame and a current frame of a video stream; obtaining one or more intermediate frames based on the previous frame and the current frame, and obtaining one or more confidence values respectively corresponding to the one or more intermediate frames; and in a case that the one or more confidence values satisfy a pre-defined condition, interpolating the one or more intermediate frames between the previous frame and the current frame.

2. The method according to claim 1, wherein the obtaining the one or more intermediate frames based on the previous frame and the current frame comprises: obtaining a first feature pyramid comprising a set of first feature maps with different resolutions for the previous frame and a second feature pyramid comprising a set of second feature maps with different resolutions for the current frame, wherein a first feature map and a second feature map for a same level have same resolution, a first feature map for a top level has a resolution of the top level and a first feature map for a bottom level has a resolution of the bottom level; and obtaining, according to the first feature pyramid and the second feature pyramid, the one or more intermediate frames, wherein intermediate feature maps with different resolutions of the one or more intermediate frames are built based on the first feature pyramid and the second feature pyramid from the bottom level to the top level.

3. The method according to claim 2, wherein the obtaining, according to the first feature pyramid and the second feature pyramid, the one or more intermediate frames comprises: obtaining, according to a first feature map and a second feature map for a bottom level, one or more first optical flows with a first resolution, one or more second optical flows with the first resolution, one or more intermediate feature maps with the first resolution, one or more first flow selection mask with the first resolution, one or more second flow selection mask with the first resolution, and one or more feature selection masks with the first resolution, wherein the first resolution is a resolution of a level which is higher than and next to the bottom level, wherein each of the one or more first optical flows with the first resolution is a forward optical flow corresponding to a respective one in the one or more intermediate frames, and each of the one or more second optical flows with the first resolution is a backward optical flow corresponding to a respective one in the one or more intermediate frames; wherein the one or more first optical flows with the first resolution, the one or more second optical flows withthe first resolution, the one or more intermediate feature maps with the first resolution, the one or more first flow selection mask with the first resolution, the one or more second flow selection mask with the first resolution, and the one or more feature selection masks with the first resolution are used for producing the one or more intermediate frames.

4. The method according to claim 2 or 3, wherein the obtaining, according to the first feature pyramid and the second feature pyramid, the one or more intermediate frames comprises: obtaining one or more first warped feature maps with a middle resolution by warping a first feature map with the middle resolution using one or more first optical flows with the middle resolution, wherein the middle resolution is higher than the resolution of the bottom level and lower than the resolution of the top level; obtaining one or more second warped feature maps with the middle resolution by warping the second feature map with the middle resolution using one or more second optical flows with the middle resolution; and obtaining one or more first optical flow residuals with a second resolution, one or more second optical flow residuals with the second resolution, one or more intermediate feature maps with the second resolution, one or more first flow selection masks with the second resolution, one or more second flow selection masks with the second resolution, and one or more feature selection masks with the second resolution according to one or more intermediate feature maps with the middle resolution, the one or more first warped feature maps with the middle resolution, the one or more second warped feature maps with the middle resolution, one or more first flow selection masks with the middle resolution, one or more second flow selection masks with the middle resolution, and one or more feature selection masks with the middle resolution, wherein the second resolution is a higher resolution of a level higher than and next to a level of the middle resolution; wherein the one or more first optical flow residuals with the second resolution, the one or more second optical flow residuals with the second resolution, the one or more intermediate feature maps with the second resolution, the one or more first flow selection masks with the second resolution, the one or more second flow selection masks with the second resolution, and the one or more feature selection masks with the second resolution are used for producing the one or more intermediate frames.

5. The method according to any one of claims 2 to 4, wherein the obtaining, according to the first feature pyramid and the second feature pyramid, the one or more intermediate frames comprises: obtaining one or more first warped feature maps with the resolution of the top level by warping the first feature map with the resolution of the top level using one or more first optical flows with the resolution of the top level;obtaining one or more second warped feature maps for the top level by warping the second feature map for the top level using one or more second optical flows with the resolution of the top level; and obtaining the one or more intermediate frames according to one or more intermediate feature maps with the resolution of the top level, the one or more first warped feature maps with the resolution of the top level, the one or more second warped feature maps with the resolution of the top level, one or more first flow selection masks with the resolution of the top level, one or more second flow selection masks with the resolution of the top level, and one or more feature selection masks with the resolution of the top level.

6. The method according to any one of claims 3 to 5, further comprising obtaining one or more first optical flows with a third resolution by up-scaling one or more first optical flows with a fourth resolution to obtain one or more up-scaled first optical flows with the third resolution; and correcting the one or more up-scaled first optical flows with the third resolution by one or more first optical flow residuals with the third resolution; and obtaining one or more second optical flows with the third resolution by up-scaling one or more second optical flows with the fourth resolution to obtain one or more up-scaled second optical flows with the third resolution; and correcting the one or more up-scaled second optical flows with the third resolution by the one or more second optical flow residuals with the third resolution; wherein the third resolution is a resolution of a level higher than and next to a level of the fourth resolution, and the fourth resolution is higher than the resolution of the bottom level.

7. The method according to any one of claims 1 to 6, wherein obtaining the one or more confidence values respectively corresponding to the one or more intermediate frames comprises: obtaining the one or more confidence values respectively corresponding to the one or more intermediate frames based on one or more first warped feature maps with a fifth resolution and one or more second warped feature maps with a fifth resolution, wherein the fifth resolution is greater than or equal to a first resolution threshold.

8. The method according to claim 7, wherein the first resolution threshold is associated with a size of details to keep in the video stream.

9. The method according to any one of claims 2 to 8, wherein the obtaining the first feature pyramid comprising the set of first feature maps with different resolutions for the previous frame comprises:obtaining the first feature map for the top level based on an image of the previous frame with an original resolution of the video stream; obtaining a first feature map for a level next to the top level, wherein the first feature map for the level next to the top level comprises a first part obtained based on a down-sampled image of the previous frame for the level next to the top level, and a second part obtained based on the first feature map for the top level, wherein the down-sampled image of the previous frame for the level next to the top level is obtained by down sampling the image for the previous frame with the original resolution by a pre-defined factor; and obtaining a first feature map for each level of one or more levels lower than the level next to the top level, wherein the first feature map for the each level comprises a first part obtained based on a down-sampled image of the previous frame for the each level, and a second part obtained based on a first part of a first feature map for a level upper than and next to the each level, wherein the down-sampled image of the previous frame for the each level is obtained by down sampling the down-sampled image of the previous frame for the level upper than and next to the each level by a pre-defined factor.

10. The method according to any one of claims 2 to 8, wherein a quantity of levels of the first feature pyramid is based on an original resolution of the video stream and a second resolution threshold, wherein a resolution of the first feature map for the bottom level is lower than the second resolution threshold and a resolution of a first feature map for a level upper than and next to the bottom level is higher than the second resolution threshold.

11. The method according to claim 10, wherein the second resolution threshold is based a motion level between the previous frame and the current frame.

12. The method according to any one of claims 2 to 11, wherein a first feature map and a second feature map for a same level have a same quantity of features, and a quantity of features for the first feature map of the top level is smaller than a quantity of features for a first feature map of a lower level.

13. The method according to any one or claims 1 to 12, wherein quantity of the one or more intermediate frames is based on an original frame rate of the video stream and a target frame rate of the video stream.

14. The method according to any one of claims 1 to 13, wherein quantity of the one or more intermediate frames is based on a first operation of a user, wherein the first operation indicates a target frame rate of the video stream or the quantity of the one or more intermediate frames.

15. The method according to claim 14, wherein a color space in which to perform the frame interpolation is determined based on a characteristic of a scene of the video stream or a second operation of a user, wherein thesecond operation indicates the color space in which to perform the frame interpolation.

16. The method according to any one or claims 1 to 15, further comprising: in a case that the one or more confidence values do not satisfy the pre-defined condition, repeating the previous frame or the current frame one or more times.

17. The method according to any of claims 3 to 5, wherein the frame interpolation is performed in an RGB color space, each of the set of first feature maps and the set of second feature maps comprises red channel information, blue channel information and green channel information, and one or more first optical flows with a sixth resolution and one or more second optical flows with the sixth resolution are obtained based on the red channel information, the blue channel information and the green channel information in one or more first feature maps and one or more second feature maps with a resolution lower than the sixth resolution, wherein the sixth resolution is higher than the resolution of the bottom level.

18. The method according to any of claims 3 to 5, wherein the frame interpolation is performed in a YUV color space, each of the set of first feature maps and the set of second feature maps comprises gray channel information, chroma-blue channel information and chroma-red channel information, and one or more first optical flows with a seventh resolution and one or more second optical flows with the seventh resolution are obtained based on the gray channel information of one or more first feature maps and one or more second feature maps with a resolution of the gray channel information lower than the seventh resolution.

19. The method according to any one of claims 1 to 18, wherein the one or more intermediate frames are obtained through a frame interpolation model, and the frame interpolation model is determined from multiple frame interpolation models.

20. The method according to claim 19, wherein the multiple frame interpolation models comprise a first frame interpolation model that doubles a frame rate of the video stream and performs frame interpolation in an RGB color space, or a second frame interpolation model that triples a frame rate of the video stream and performs frame interpolation in an RGB color space, or a third frame interpolation model that doubles a frame rate of the video stream and perform frame interpolation in a YUV color space, or a fourth frame interpolation model that triples a frame rate of the video stream and perform frame interpolation in a YUV color space.

21. The method according to claims 19 or 20, wherein the frame interpolation model is determined according to at least one of an original frame rate of the video stream, a target frame rate of the video stream, or a characteristic of scenes of the video stream.

22. The method according to claim 19 or 20, wherein the frame interpolation model is determined based on a third operation of a user, wherein the third operation indicates the frame interpolation model selected by the user.

23. The method according to any one of claims 1 to 22, wherein the pre-defined condition comprises the one or more confidence values being greater than or equal to a threshold.

24. An apparatus for frame interpolation, comprising: a first obtaining module, configured to obtain a previous frame and a current frame of a video stream; a second obtaining module, configured to obtain one or more intermediate frames based on the previous frame and the current frame, and obtain one or more confidence values respectively corresponding to the one or more intermediate frames; and an interpolating module, configured to in a case that the one or more confidence values satisfy a pre-defined condition, interpolate the one or more intermediate frames between the previous frame and the current frame.

25. The apparatus according to claim 24, wherein the second obtaining module comprises: a pyramidal encoder, configured to obtain a first feature pyramid comprising a set of first feature maps with different resolutions for the previous frame and a second feature pyramid comprising a set of second feature maps with different resolutions for the current frame, wherein a first feature map and a second feature map for a same level have same resolution, a first feature map for a top level has a resolution of the top level and a first feature map for a bottom level has a resolution of the bottom level; and a chain of decoders, configured to obtain, according to the first feature pyramid and the second feature pyramid, the one or more intermediate frames, wherein intermediate feature maps with different resolutions of the one or more intermediate frames are built based on the first feature pyramid and the second feature pyramid from the bottom level to the top level.

26. The apparatus according to claim 25, wherein the chain of decoders comprises a bottom-level decoder, and the bottom-level decoder is configured to: obtain, according to a first feature map and a second feature map with the resolution of the bottom level, one or more first optical flows with a first resolution, one or more second optical flows with the first resolution, one or more intermediate feature maps with the first resolution, one or more first flow selection mask with the first resolution, one or more second flow selection mask with the first resolution, and one or more feature selection masks with the first resolution, wherein the first resolution is a resolution of a level which is higher than and next to the resolution of the bottom level, wherein each of the one or more first optical flows with the first resolution is a forward opticalflow corresponding to a respective one in the one or more intermediate frames, and each of the one or more second optical flows with the first resolution is a backward optical flow corresponding to a respective one in the one or more intermediate frames; wherein the one or more first optical flows with the first resolution, the one or more second optical flows with the first resolution, the one or more intermediate feature maps with the first resolution, the one or more first flow selection mask with the first resolution, the one or more second flow selection mask with the first resolution, and the one or more feature selection masks with the first resolution are used for producing the one or more intermediate frames.

27. The apparatus according to claim 26, wherein the bottom-level decoder comprises: a first convolution layer, configured to: obtain, according to a first input tensor, a first extended tensor, wherein the first input tensor is obtained according to the first feature map and the second feature map with the resolution of the bottom level; wherein a quantity of channels of the first extended tensor is greater than a quantity of channels of the first input tensor.

28. The apparatus according to claim 27, wherein the bottom-level decoder further comprises: a first shuffle layer module, configured to up scale tensors split from a first aggregated output tensor to produce up-scaled tensors with the first resolution; wherein the first aggregated output tensor is obtained according to the first extended tensor; wherein the up-scaled tensors with the first resolution are used for producing the one or more first optical flows with the first resolution, the one or more second optical flows with the first resolution, the one or more intermediate feature maps with the first resolution, the one or more first flow selection masks with the first resolution, the one or more second flow selection masks with the first resolution, and the one or more feature selection masks with the first resolution.

29. The apparatus according to claim 28, wherein the bottom-level decoder further comprises: a first tangent layer module, configured to apply a first hyperbolic tangent function to produce, according to one or more first up-scaled tensors of the up-scaled tensors with the first resolution, the one or more first optical flows with the first resolution, and produce, according to one or more second up-scaled tensors of the up-scaled tensors with the first resolution, the one or more second optical flows with the first resolution; and / or a first activation layer, configured to apply a first softmax function to produce, according to one or more thirdup-scaled tensors of the up-scaled tensors with the first resolution, the one or more first flow selection masks with the first resolution, the one or more second flow selection masks with the first resolution, and the one or more feature selection masks with the first resolution.

30. The apparatus according to any one of claims 25 to 29, wherein the chain of decoders comprises one or more middle-level decoders, and the one or more middle-level decoders are configured to: obtain one or more first warped feature maps with a middle resolution by warping a first feature map with the middle resolution using one or more first optical flows with the middle resolution, wherein the middle resolution is higher than the resolution of the bottom level and lower than the resolution of the top level; obtain one or more second warped feature maps with the middle resolution by warping the second feature map with the middle resolution using one or more second optical flows with the middle resolution; and obtain one or more first optical flow residuals with a second resolution, one or more second optical flow residuals with the second resolution, one or more intermediate feature maps with the second resolution, one or more first flow selection masks with the second resolution, one or more second flow selection masks with the second resolution, and one or more feature selection masks with the second resolution according to one or more intermediate feature maps with the middle resolution, the one or more first warped feature maps with the middle resolution, the one or more second warped feature maps with the middle resolution, one or more first flow selection masks with the middle resolution, one or more second flow selection masks with the middle resolution, and one or more feature selection masks with the middle resolution, wherein the second resolution is a resolution of a level higher than and next to a level of the middle resolution; wherein the one or more first optical flow residuals with the second resolution, the one or more second optical flow residuals with the second resolution, the one or more intermediate feature maps with the second resolution, the one or more first flow selection masks with the second resolution, the one or more second flow selection masks with the second resolution, and the one or more feature selection masks with the second resolution are used for producing the one or more intermediate frames.

31. The apparatus according to claim 30, wherein each of the one or more middle-level decoders comprises: a second convolution layer, configured to: obtain, according to a second input tensor, a second extended tensor, wherein the second input tensor is obtained according to the one or more intermediate feature maps with the middle resolution, the one or more first warped feature maps with the middle resolution, the one or more second warped feature maps with the middle resolution, the one or more first flow selection masks with the middle resolution, theone or more second flow selection masks with the middle resolution, and the one or more feature selection masks with the middle resolution; wherein a quantity of channels of the second extended tensor is greater than a quantity of channels of the second input tensor.

32. The apparatus according to claim 31, wherein each of the one or more middle-level decoders further comprises: a second shuffle layer module, configured to up scale tensors split from a second aggregated output tensor to produce up-scaled tensors with the second resolution; wherein the second aggregated output tensor is obtained according to the second extended tensor; wherein the up-scaled tensors with the second resolution are used for producing the one or more intermediate feature maps with the second resolution, the one or more first optical flow residuals with the second resolution, the one or more second optical flow residuals with the second resolution, the one or more first flow selection masks with the second resolution, the one or more second flow selection masks with the second resolution, and the one or more feature selection masks with the second resolution.

33. The apparatus according to claim 32, wherein each of the one or more middle-level decoders further comprises: a second tangent layer module, configured to apply a second hyperbolic tangent function to produce, according to one or more first up-scaled tensors of the up-scaled tensors with the second resolution, the one or more first optical flow residuals with the second resolution, and produce, according to one or more second up-scaled tensors of the up- scaled tensors with the second resolution, the one or more second optical flow residuals with the second resolution; and / or a second activation layer, configured to apply a second softmax function to produce, according to one or more third up-scaled tensors of the up-scaled tensors with the second resolution, the one or more first flow selection masks with the second resolution, the one or more second flow selection masks with the second resolution, and the one or more feature selection masks with the second resolution.

34. The apparatus according to any one of claims 25 to 33, wherein the chain of decoders comprises a top-level decoder, and the top-level decoder is configured to: obtain one or more first warped feature maps with the resolution of the top level by warping the first feature map with the resolution of the top level using one or more first optical flows with the resolution of the top level;obtain one or more second warped feature maps with the resolution of the top level by warping the second feature map with the resolution of the top level using one or more second optical flows with the resolution of the top level; and obtain the one or more intermediate frames according to one or more intermediate feature maps with the resolution of the top level, the one or more first warped feature maps with the resolution of the top level, the one or more second warped feature maps with the resolution of the top level, one or more first flow selection masks with the resolution of the top level, one or more second flow selection masks with the resolution of the top level, and one or more feature selection masks with the resolution of the top level.

35. The apparatus according to any one of claims 26 to 34, wherein the apparatus further comprises an optical flow correcting module, and the optical flow correcting module is configured to: obtain one or more first optical flows with a third resolution by up-scaling one or more first optical flows with a fourth resolution to obtain one or more up-scaled first optical flows with the third resolution; and correcting the one or more up-scaled first optical flows with the third resolution by one or more first optical flow residuals with the third resolution; and obtain one or more second optical flows with the third resolution by up-scaling one or more second optical flows with the fourth resolution to obtain one or more up-scaled second optical flows with the third resolution; and correcting the one or more up-scaled second optical flows with the third resolution by the one or more second optical flow residuals with the third resolution; wherein the third resolution is a resolution of a level higher than and next to a level of the fourth resolution, and the fourth resolution is higher than the resolution of the bottom level.

36. The apparatus according to any one of claims 24 to 35, wherein the second obtaining module comprises a confidence estimator, and the confidence estimator is configured to: obtain the one or more confidence values respectively corresponding to the one or more intermediate frames based on one or more first warped feature maps with a fifth resolution and one or more second warped feature maps with a fifth resolution, wherein the fifth resolution is greater than or equal to a first resolution threshold.

37. The apparatus according to claim 36, wherein the first resolution threshold is associated with a size of details to keep in the video stream.

38. The apparatus according to any one of claims 25 to 37, wherein the pyramidal encoder is configured to: obtain the first feature map for the top level based on an image of the previous frame with an original resolution of the video stream; obtain a first feature map for a level next to the top level, wherein the first feature map for the level next to the top level comprises a first part obtained based on a down-sampled image of the previous frame for the level next to the top level, and a second part obtained based on the first feature map for the top level, wherein the down-sampled image of the previous frame for the level next to the top level is obtained by down sampling the image for the previous frame with the original resolution by a pre-defined factor; and obtain a first feature map for each level of one or more levels lower than the level next to the top level, wherein the first feature map for the each level comprises a first part obtained based on a down-sampled image of the previous frame for the each level, and a second part obtained based on a first part of a first feature map for a level upper than and next to the each level, wherein the down-sampled image of the previous frame for the each level is obtained by down sampling the down-sampled image of the previous frame for the level upper than and next to the each level by a pre-defined factor.

39. The apparatus according to any one of claims 25 to 37, wherein a quantity of levels of the first feature pyramid is based on an original resolution of the video stream and a second resolution threshold, wherein a resolution of the first feature map for the bottom level is lower than the second resolution threshold and a resolution of a first feature map for a level upper than and next to the bottom level is higher than the second resolution threshold.

40. The apparatus according to claim 39, wherein the second resolution threshold is based a motion level between the previous frame and the current frame.

41. The apparatus according to any one of claims 25 to 40, wherein a first feature map and a second feature map for a same level have a same quantity of features, and a quantity of features for the first feature map of the top level is smaller than a quantity of features for a first feature map of a lower level.

42. The apparatus according to any one or claims 24 to 41, wherein a quantity of the one or more intermediate frames is based on an original frame rate of the video stream, and a target frame rate of the video stream.

43. The apparatus according to any one of claims 24 to 42, wherein a quantity of the one or more intermediate frames is based on a first operation of a user, wherein the first operation indicates a target frame rate of the video stream or the intermediate frame quantity of the one or more intermediate frames.

44. The apparatus according to claim 43, wherein a color space in which to perform the frame interpolation isdetermined based on a characteristic of a scene of the video stream or a second operation of a user, wherein the second operation indicates the color space in which to perform the frame interpolation.

45. The apparatus according to any one or claims 24 to 44, wherein the interpolating module is further configured to in a case that the one or more confidence values do not satisfy the pre-defined condition, repeat the previous frame or the current frame one or more times.

46. The apparatus according to any one of claims 26 to 34, wherein the frame interpolation is performed in an RGB color space, each of the set of first feature maps and the set of second feature maps comprises red channel information, blue channel information and green channel information, and one or more first optical flows with a sixth resolution and one or more second optical flows with the sixth resolution are obtained based on the red channel information, the blue channel information and the green channel information in one or more first feature maps and one or more second feature maps with a resolution lower than the sixth resolution, wherein the sixth resolution is higher than the resolution of the bottom level.

47. The apparatus according to any of claims 26 to 34, wherein the frame interpolation is performed in a YUV color space, each of the set of first feature maps and the set of second feature maps comprises gray channel information, chroma-blue channel information and chroma-red channel information, and one or more first optical flows with a seventh resolution and one or more second optical flows with the seventh resolution are obtained based on the gray channel information of one or more first feature maps and one or more second feature maps with a resolution of the gray channel information lower than the seventh resolution.

48. The apparatus according to any one of claims 24 to 47, wherein the one or more intermediate frames are obtained through a frame interpolation model, wherein the frame interpolation model is determined from multiple frame interpolation models.

49. The apparatus according to claim 48, wherein the multiple frame interpolation models comprise a first frame interpolation model that doubles a frame rate of the video stream and performs frame interpolation in an RGB color space, or a second frame interpolation model that triples a frame rate of the video stream and performs frame interpolation in an RGB color space, or a third frame interpolation model that doubles a frame rate of the video stream and perform frame interpolation in a YUV color space, or a fourth frame interpolation model that triples a frame rate of the video stream and perform frame interpolation in a YUV color space.

50. The apparatus according to claims 48 or 49, wherein the frame interpolation model is determined according to at least one of an original frame rate of the video stream, a target frame rate of the video stream, or a characteristicof scenes of the video stream.

51. The apparatus according to claim 48 or 49, wherein the frame interpolation model is determined based on a third operation of a user, wherein the third operation indicates the frame interpolation model selected by the user.

52. The apparatus according to any one of claims 24 to 51, wherein the pre-defined condition comprises the one or more confidence values being greater than or equal to a threshold.

53. A computing device cluster, comprising at least one computing device, wherein the computing device comprises at least one processor and a memory coupled with the processor, wherein the memory stores instructions which, when executed by the at least one processor, cause the computing device cluster to execute the method according to any one of claims 1 to 23.

54. A computer-readable medium storing computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute the method according to any one of claims 1 to 23.

55. A computer program product comprising computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute the method according to any one of claims 1 to 23.

Images