Video frame insertion method and device, and computer readable medium

Abstract

The present disclosure provides a method of video frame insertion, comprising: down-sampling a first input feature at a plurality of levels, producing an image feature at a lower level; each level of first input feature comprises a video frame or the image feature of the level; performing pixel recombination on the to-be-sampled features at multiple levels to generate additional image features of a higher level; each level of to-be-sampled features comprises the image features of the level and the additional image features generated by recombining the pixels of the lower level; up-sampling the second input feature at a plurality of levels, resulting in an intermediate image feature at a higher level; the second input feature of each level comprises the additional image feature and the intermediate image feature of the level. The invention further provides video frame insertion equipment and a computer readable medium.

Description

Technical Field

[0001] The present disclosure relates to the technical field of video frame insertion, and in particular to a video frame insertion method and device, and a computer-readable medium. Background Art

[0002] In order to increase the frame rate, improve the video smoothness and reduce the picture jitter, video frame interpolation (VFI) can be performed, that is, inserting new frames (insertion frames) between the original frames (video frames) of the video.

[0003] In some related technologies, deep learning models (such as neural networks) can be used to analyze videos to generate inserted frames. However, video frame insertion technology using deep learning models has high computational overhead, requires a large number of channels, and has a limited scope of application. Summary of the invention

[0004] The present disclosure provides a method and device for video frame insertion, and a computer-readable medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for video frame insertion, comprising:

[0006] Downsampling the first input feature at multiple levels to generate image features at a lower level; the first input feature at each level includes a video frame or the image feature at the current level;

[0007] Recombining pixels of features to be sampled at multiple levels to generate additional image features of a higher level; the features to be sampled at each level include the image features of the current level and the additional image features generated by pixel recombining at a lower level;

[0008] The second input features are upsampled at multiple levels to generate intermediate image features at a higher level; the second input features at each level include the additional image features and the intermediate image features at the current level.

[0009] In a second aspect, an embodiment of the present disclosure provides a device for video interpolation, which includes a memory and a processor; the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, the method for video interpolation of the embodiment of the present disclosure is implemented.

[0010] In a third aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, and the computer program is executed by a processor to implement the video interpolation method of the embodiment of the present disclosure.

[0011] In the disclosed embodiment, in addition to decoding, upsampling is performed through pixel shuffle, and deep (low-level, small-scale) features are returned to the upper layer (high-level, large-scale), realizing a feature fusion process similar to optical reflection in multiple medium layers, increasing the connection between deep features and shallow features, and increasing the final weight of decoding. Better results can be achieved with the same number of channels, so that fewer channels are required; and since pixel shuffle does not require training, it can save computing overhead; at the same time, the deep learning model of the disclosed embodiment is applicable to both image feature-based and optical flow-based video interpolation, and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the accompanying drawings of the embodiments of the present disclosure:

[0013] Figure 1 A flowchart of a video frame insertion method provided by an embodiment of the present disclosure;

[0014] Figure 2 A flowchart of another video frame insertion method provided by an embodiment of the present disclosure;

[0015] Figure 3 A schematic diagram of the structure of a deep learning model used in a video frame insertion method provided in an embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram of the structure of a deep learning model used in another video frame insertion method provided by an embodiment of the present disclosure;

[0017] Figure 5 for Figure 4 A roadmap of information transmission for deep learning models;

[0018] Figure 6 A circuit diagram of information transmission of a deep learning model used in another video frame insertion method provided by an embodiment of the present disclosure;

[0019] Figure 7 A block diagram of a video frame insertion device provided by an embodiment of the present disclosure;

[0020] Figure 8 A block diagram of a computer-readable medium provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the video frame insertion method and device, and computer-readable medium provided by the embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0022] The present disclosure will be described more fully below with reference to the accompanying drawings, but the embodiments shown may be embodied in different forms, and the present disclosure should not be construed as being limited to the embodiments set forth below. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete, and will enable those skilled in the art to fully understand the scope of the present disclosure.

[0023] The accompanying drawings of the embodiments of the present disclosure are used to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification, and are used together with the detailed embodiments to explain the present disclosure, and do not constitute a limitation of the present disclosure. By describing the detailed embodiments with reference to the accompanying drawings, the above and other features and advantages will become more apparent to those skilled in the art.

[0024] The present disclosure may be described with reference to plan views and / or cross-sectional views by means of ideal schematic views of the present disclosure. Therefore, the exemplary illustrations may be modified according to manufacturing techniques and / or tolerances.

[0025] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0026] The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure. The term "and / or" as used in the present disclosure includes any and all combinations of one or more related enumerated items. The singular forms "one" and "the" as used in the present disclosure are also intended to include plural forms, unless the context clearly indicates otherwise. The terms "including", "made of..." as used in the present disclosure specify the presence of features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined in this disclosure.

[0028] The present disclosure is not limited to the embodiments shown in the drawings, but includes modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the drawings have schematic properties, and the shapes of the regions shown in the drawings illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.

[0029] In order to increase the frame rate, improve the video smoothness, and reduce the picture jitter, video interpolation can be performed, that is, inserting newly generated interpolation frames between the original video frames of the video, and video interpolation can be based on different principles.

[0030] For example, one video interpolation principle is based on image features, that is, extracting image features of video frames in receptive fields of different levels (scales) to calculate each pixel in the interpolated frame.

[0031] For another example, another principle of video frame insertion is based on optical flow, that is, the motion characteristics (optical flow) of pixels in the video frame are analyzed, so as to "move" the pixels of the video frame to generate an inserted frame.

[0032] Video interpolation can be achieved through a deep learning model (such as a neural network). For example, the deep learning model based on image features can be a model with an "encoding-decoding" structure such as a UNet model.

[0033] However, the deep learning models in related technologies (such as UNet models) usually have a large number of parameters, so the computational overhead is high and a large number of channels are required; moreover, the above deep learning models can only be used for video interpolation based on image features, but cannot be used for video interpolation based on optical flow, so the scope of application is limited.

[0034] First, refer to Figures 1 to 6 , an embodiment of the present disclosure provides a method for video frame insertion.

[0035] The method of the disclosed embodiment is used to perform video frame interpolation (VFI), that is, to analyze the original frames (video frames) in the video to determine the new frames that should be inserted between adjacent video frames, namely the inserted frames (or intermediate frames), and insert the inserted frames between the corresponding video frames to obtain a video with a higher frame rate.

[0036] The frequency of interpolation can be set as needed, for example, interpolation frames can be added between any two adjacent video frames to double the frame rate of the video; or, interpolation frames can be added between some adjacent video frames.

[0037] Reference Figure 1 The video frame insertion method of the embodiment of the present disclosure includes:

[0038] S101. Downsample the first input feature at multiple levels to generate image features at a lower level.

[0039] The first input feature of each level includes a video frame or an image feature of the current level.

[0040] S102, performing pixel reorganization on the features to be sampled at multiple levels to generate additional image features at a higher level.

[0041] The features to be sampled at each level include the image features at this level and the additional image features generated by pixel reorganization at the next lower level.

[0042] S103 , up-sample the second input feature at multiple levels to generate an intermediate image feature at a higher level.

[0043] The second input features of each level include additional image features and intermediate image features of the current level.

[0044] In the disclosed embodiments, "level" refers to the scale of the corresponding image / feature or the layer in which it is located. The higher the "level", the larger the scale of the image / feature is or the shallower the layer in which it is located.

[0045] In the disclosed embodiment, the relevant features of the video frames in the video are down-sampled (decoded) at multiple levels to generate image features of lower and lower levels; thereafter, the image features are up-sampled (encoded) at multiple levels to generate intermediate image features of higher and higher levels.

[0046] At the same time, in the embodiment of the present disclosure, the image features generated by each level of downsampling are pixel reorganized (Pixel Shuffle) together with other partial features to generate additional image features of a higher level (Extra Feature Levelx), and the additional image features are also input for upsampling at the corresponding level.

[0047] The above downsampling, pixel reorganization, and upsampling are equivalent to "feature fusion" or "feature extraction", and the features generated by them can be further processed to obtain the corresponding insertion frame.

[0048] In the disclosed embodiment, in addition to decoding, upsampling is performed through pixel shuffle, and deep (low-level, small-scale) features are returned to the upper layer (high-level, large-scale), realizing a feature fusion process similar to optical reflection in multiple medium layers, increasing the connection between deep features and shallow features, and increasing the final weight of decoding. Better results can be achieved with the same number of channels, so that fewer channels are required; and since pixel shuffle does not require training, it can save computing overhead; at the same time, the deep learning model of the disclosed embodiment is applicable to both image feature-based and optical flow-based video interpolation, and has a wide range of applications.

[0049] In some embodiments, downsampling, pixel binning, and upsampling are performed in a deep learning model.

[0050] As a method of an embodiment of the present disclosure, it can be specifically possible to implement the method of the embodiment of the present disclosure with a preset "deep learning model (such as a neural network)", that is, the structure of the deep learning model is preset so that the deep learning model can perform the operations of each step in the method of the embodiment of the present disclosure; thus, as long as the video frame is input into the deep learning model, the output inserted frame can be obtained through the processing of the deep learning model.

[0051] For example, as one embodiment of the present disclosure, when using a deep learning model, refer to Figure 2 , the method of the embodiment of the present disclosure may include:

[0052] A101. Input the video frame into the deep learning model and obtain the output interpolated frame.

[0053] Accordingly, the structure of the deep learning model should ensure that it can implement the above methods.

[0054] For example, refer to Figure 3 The deep learning model may include a feature fusion module for performing the above feature fusion process.

[0055] The feature fusion module includes:

[0056] A decoding submodule, comprising a plurality of sequentially connected downsampling units for outputting image features;

[0057] An encoding submodule, comprising a plurality of upsampling units connected in sequence and used to output intermediate image features;

[0058] The additional upsampling submodule connected between the decoding submodule and the encoding submodule includes a multi-level pixel recombination unit; the pixel recombination unit is used to perform pixel recombination on the features to be sampled to generate a higher-level additional image feature, and send it to the corresponding level of the encoding submodule, and the features to be sampled include the image features generated by the higher-level downsampling unit and the additional image features generated by the lower-level pixel recombination unit.

[0059] Furthermore, the feature fusion module as a whole is a pyramid structure, which includes an encoding submodule for feature encoding (Feature Encoder) and a decoding submodule for feature decoding (Feature Decoder).

[0060] Reference Figure 3 The encoding submodule includes multiple levels (scales or layers) of downsampling units connected in sequence for the above "downsampling", each downsampling unit is used to downsample the content input therein, so as to obtain a lower-level (i.e., smaller scale or deeper layer) image feature (Feature Levelx).

[0061] Among them, except for the lowest-level downsampling unit, the image features generated by each downsampling unit are output to the lower-level downsampling unit, and the lowest-level downsampling unit directly receives the features of the input video frame (Img) (such as Feature Level0); accordingly, except for the input (first input feature) of the highest-level downsampling unit including the video frame, the input (first input feature) of other downsampling units includes the image features generated by the previous-level downsampling unit.

[0062] The decoding submodule includes multiple levels of sequentially connected upsampling units for performing the above "upsampling". Each upsampling unit is used to upsample the content input therein, thereby obtaining a higher-level (i.e., larger scale or shallower) intermediate image feature (Mid Feature Levelx).

[0063] Among them, except for the highest-level upsampling unit, the intermediate image features generated by each other upsampling unit are output to the higher-level upsampling unit, and the intermediate image features output by the high-level and low-level upsampling units can be used to generate inserted frames; accordingly, the input (second input features) of the lowest-level upsampling unit includes the image features generated by the lowest-level downsampling unit, and the inputs (second input features) of other upsampling units include not only the image features of the corresponding level, but also the intermediate image features generated by the next-level upsampling unit.

[0064] The encoding submodule and the decoding submodule are connected, that is, the image features (Feature Level 4) generated by the lowest level downsampling unit of the encoding submodule are directly output to the lowest level upsampling unit of the decoding submodule as the input of the upsampling unit.

[0065] Among them, the image features generated by some down-sampling units (such as Feature Level 3) can be directly output to the up-sampling unit in the decoding submodule in addition to being output to the down-sampling unit of the next level.

[0066] The number of downsampling units in the encoding submodule may be the same as the number of upsampling units in the decoding submodule. Figure 3 It is different, depending on the scale of the features required to generate the interpolated frame.

[0067] Reference Figure 3According to the embodiment of the present disclosure, the feature fusion module also includes an additional upsampling (Extra Upsample) submodule connected between the encoding submodule and the decoding submodule, that is, at least part of the image features generated by the downsampling unit of the encoding submodule are not directly output to the corresponding upsampling unit of the encoding submodule, but are output to the decoding submodule at a higher level (larger scale) after upsampling by the additional upsampling submodule.

[0068] Reference Figure 3 The additional upsampling submodule includes multiple pixel reorganization units for performing the above "pixel reorganization". Each reorganization unit is used to reorganize the pixels (Pixel Shuffle) of the input content to generate additional image features (Extra Feature Levelx) of a higher level (i.e., larger scale or shallower layer), and output them to the corresponding position of the decoding submodule, such as merging with the intermediate image features (such as Extra Feature Level2 and Mid Feature Level2) and then inputting them into the upsampling unit together, or merging with the intermediate image features generated by the highest-level upsampling unit (such as Extra Feature Level1 and Mid Feature Level1) as the output of the feature fusion module as a whole.

[0069] Among them, pixel reassembly is a known upsampling method, which divides a pixel into "multiple parts" to generate multiple new pixels.

[0070] Reference Figure 3 The input (features to be sampled) received by the pixel recombination unit in the additional upsampling submodule includes image features of the same level (i.e., image features generated by the downsampling unit at a higher level) and additional image features of the same level (i.e., additional image features generated by the pixel recombination unit at a lower level).

[0071] It should be understood that the number of pixel reorganization units in the additional upsampling submodule can be set as needed, for example, Figure 3 , the image features (Feature Level4) generated by the lowest level downsampling unit can be directly output to the lowest level upsampling unit without going through the additional upsampling submodule; and the image features (Feature Level1) generated by the highest level downsampling unit can also be directly merged with the intermediate image features (Mid Feature Level1) generated by the corresponding upsampling unit without going through the additional upsampling submodule.

[0072] It should be understood that the input of the lowest-level pixel binning unit only includes image features, but not additional image features (because there is no lower-level pixel binning unit).

[0073] Correspondingly, the additional image features generated by all pixel recombination units are output to the corresponding positions of the decoding submodule (input into the upsampling unit, or merged with the intermediate image features); and except for the highest-level pixel recombination unit, the additional image features generated by other pixel recombination units are also output to the higher-level pixel recombination unit (because there are no higher-level upsampling units and pixel recombination units above the highest-level pixel recombination unit).

[0074] In some embodiments, the method of the present disclosure further includes:

[0075] S100: Generate multi-level original image features according to video frames.

[0076] In addition to the first input features at the highest level, the first input features at other levels also include original image features at the same level.

[0077] As a method of an embodiment of the present disclosure, the video frame may be processed to collect its original image features at multiple different levels, and accordingly, the original image features at each level may be downsampled together as part of the first input features.

[0078] For example, when using the above deep learning model, refer to Figure 4 , Figure 5 , the deep learning model also includes: an image pyramid module, which is used to generate multi-level original image features according to video frames; the highest level downsampling unit is used to receive video frames, and each of the other downsampling units is also used to receive the original image features of the same level.

[0079] Reference Figure 4 , Figure 5 As one method of the embodiment of the present disclosure, the deep learning model may also include an image pyramid (Pyramid Frames) module to enrich the features processed by the encoding submodule. The image pyramid module can analyze the video frame, obtain multiple original image features of different levels (scales), and output them to the downsampling units of the corresponding levels respectively. Therefore, except for the highest level downsampling unit directly receiving the video frame (Img or Feature Level 0), the other downsampling units simultaneously receive the original image features and image features of the corresponding level.

[0080] In some embodiments, downsampling includes convolution downsampling, or includes sampling downsampling and pooling downsampling; upsampling includes deconvolution upsampling.

[0081] As one mode of an embodiment of the present disclosure, the above downsampling may be convolution downsampling (Convolution), or may further include pooling downsampling (Pooling); correspondingly, the above downsampling may be deconvolution (Transposed Convolution) upsampling.

[0082] For example, when a deep learning model is adopted, the downsampling unit includes a convolution downsampling unit, or includes a sampling downsampling unit and a pooling downsampling unit; the upsampling unit includes a deconvolution upsampling unit.

[0083] As one embodiment of the present disclosure, the downsampling unit in the above encoding submodule may be a convolution downsampling unit for performing convolution downsampling, and the upsampling unit in the above decoding submodule may be a deconvolution upsampling unit for performing deconvolution (Transposed upsampling).

[0084] Among them, convolution is the process of generating an output image by traversing the input image with a convolution kernel according to a certain convolution step length and performing point multiplication. Its expression can be:

[0085]

[0086] in, Represents the pixels of the output image, I represents the pixels of the input image, i and j represent the coordinates, n represents the size of the convolution kernel, k and l represent the convolution step, and W represents the convolution kernel.

[0087] Furthermore, the downsampling unit may also include a pooling downsampling unit for performing pooling downsampling to enrich the features of the downsampling.

[0088] In some embodiments, the method of the present disclosure further includes:

[0089] S1021. Pre-convolute the image features and additional image features at the same level to generate features to be sampled.

[0090] As one method of an embodiment of the present disclosure, since the input of pixel reconstruction (features to be sampled) includes multiple different features (image features and additional image features), these input features can also be pre-convolved and "merged" into one feature before pixel reconstruction as the feature to be sampled.

[0091] For example, when using the above deep learning model, refer to Figure 4 , Figure 5 The additional upsampling submodule also includes: a pre-convolution unit, which is used to perform pre-convolution on the features to be sampled before they enter the pixel reorganization unit.

[0092] As before, except for the lowest level pixel rebinning unit, the input of other pixel rebinning units includes at least two different features, namely the image feature from the downsampling unit and the additional image feature from the low-level pixel rebinning unit.

[0093] For this purpose, refer to Figure 4 , Figure 5 , the features of the two parts can be convolved without changing the scale (level) through the pre-convolution unit, that is, the two parts are "merged" into one feature through convolution and then input into the pixel recombination unit so that the pixel recombination unit can perform pixel recombination on it.

[0094] It should be understood that it is also feasible to achieve the above feature combination through other methods.

[0095] It should be understood that reference Figure 4 , Figure 5 For other units with multiple different feature inputs (such as upsampling units), features can also be merged in some way.

[0096] In some embodiments, the method of the present disclosure further includes:

[0097] S104 , generating a subframe according to the relevant features of each of two adjacent video frames.

[0098] The relevant features of the video frame include the video frame, the highest-level intermediate image features generated based on the video frame, and the highest-level additional image features generated based on the video frame.

[0099] S105 , merging two subframes generated according to two adjacent video frames to obtain an inserted frame.

[0100] As a method A of an embodiment of the present disclosure, various features generated in the feature fusion process can also be used to generate actual insertion frames (ImgT) through feature refinement (Feature Refine).

[0101] Specifically, the inserted frame between two adjacent video frames can be generated based on the two video frames, that is, for each of the video frames, a "sub-frame" can be generated based on the video frame itself, as well as the highest-level intermediate image features and additional image features (related features) generated by the video frame; then, the two sub-frames are merged to obtain the inserted frame.

[0102] For example, when using the above deep learning model, refer to Figure 4 , Figure 5 ,The deep learning model also includes a feature refinement module, which includes:

[0103] Two subframe generation units, each for processing one of two adjacent video frames; each subframe generation unit is used to generate a subframe according to the video frame it processes, the intermediate image features generated by the highest level upsampling unit according to the video frame, and the additional image features generated by the highest level pixel reassembly unit according to the video frame;

[0104] The merging unit is used to merge two sub-frames generated according to two adjacent video frames to obtain an inserted frame.

[0105] As one method of an embodiment of the present disclosure, a deep learning model may be used to perform video interpolation based on image features. In this case, the interpolated frame should be obtained based on the video frames before and after it, that is, the interpolated frame between two adjacent video frames can be determined.

[0106] Reference Figure 4 , Figure 5 At this time, after the feature fusion module, a feature refining module (such as a feature refining network) for feature refinement can also be connected to further calculate the final inserted frame (ImgT) according to the intermediate image features generated by the feature fusion module for different video frames.

[0107] Reference Figure 4 , Figure 5 The feature refining module includes two sub-frame generation units. For two adjacent video frames that need to be inserted with an insertion frame, each sub-frame generation unit processes one of the video frames to obtain a sub-frame. After that, the merging unit merges the two sub-frames into one insertion frame.

[0108] Among them, when the subframe generation unit generates a subframe based on a video frame, the data (related features) based on it include: the video frame itself (that is, the video frame also needs to be directly input into the subframe generation unit), the highest-level intermediate image features generated based on the video frame, and the highest-level additional image features generated based on the video frame.

[0109] In some embodiments, generating a subframe according to the relevant feature (S104) includes:

[0110] S1041. Perform a dilated deformable convolution on relevant features to generate subframes.

[0111] As one method of an embodiment of the present disclosure, the above subframes may be specifically generated by means of dilation DCN.

[0112] For example, when the above deep learning model is adopted, the subframe generation unit includes a hole deformable convolution unit.

[0113] As one method of an embodiment of the present disclosure, the subframe generation unit may specifically be a dilated deformable convolution unit that performs dilated deformable convolution, and the dilated deformable convolution may provide a larger receptive field.

[0114] For example, the dilated deformable convolution unit may be an AdaCoF Dilation DCN (Adaptive Collaboration of Flows for Video Frame Interpolation Dilation Deformable Convolution).

[0115] Among them, the dilated deformable convolution is a convolution formed by combining the dilated convolution with the deformable convolution, and its expression is:

[0116]

[0117] in, Represents the pixel of the output image, I represents the pixel of the input image, i and j represent the coordinates, n represents the size of the convolution kernel, k and l represent the convolution step, W represents the convolution kernel, α represents the offset in the i direction (which can be understood as the superposition in the optical flow), and β represents the offset in the j direction (which can be understood as the superposition in the optical flow).

[0118] In some embodiments, the method of the present disclosure further includes:

[0119] S106, determining the optical flow information of the current level according to the intermediate image features of the previous level, the additional image features of the current level, and the optical flow information of the previous level, and flipping the intermediate image features of the previous level and the additional image features of the current level according to the optical flow information of the current level.

[0120] As another method of the embodiment of the present disclosure, video interpolation can also be performed based on "optical flow", that is, there is no need to generate interpolated frames through feature refinement, but the optical flow information (pixel motion features, Optical Flow Levelx) is determined based on the comparison of various features generated in feature fusion, and then the image is flipped (Warp) based on the optical flow information, that is, the pixels in the image are moved to obtain a new image, such as an interpolated frame.

[0121] For example, when the above deep learning model is adopted, the decoding submodule also includes: a multi-level optical flow unit, which is used to determine the optical flow information of the current level based on the intermediate image features of the previous level, the additional image features of the current level, and the optical flow information of the previous level, and flip the intermediate image features of the previous level and the additional image features of the current level according to the optical flow information.

[0122] Reference Figure 6As another embodiment of the present disclosure, the deep learning model can also perform video interpolation based on optical flow; in this case, it is not necessary to set up a feature refinement module, but to add an optical flow unit to the decoding submodule.

[0123] Each optical flow unit is used to generate optical flow information (pixel motion features), and each level of optical flow information (Optical Flow Levelx) is derived based on the optical flow information of the next level, the intermediate image features of the next level (the lowest level is the image features), and the additional image features of this level. The features (intermediate image features, additional image features) used in the decoding process (such as those entering the upsampling unit) need to be flipped (Warp) based on the above optical flow information, that is, the pixels in it are moved.

[0124] Further, see Figure 6 , after flipping the highest-level image features (such as Img or Feature Level0) using the highest-level optical flow information (Optical Flow Level0), the inserted frame (ImgT) can be obtained.

[0125] Among them, the expression of optical flow flip can be:

[0126]

[0127] in, Represents the pixel of the output image, I represents the pixel of the input image, i and j represent the coordinates, α represents the optical flow in the i direction, and β represents the optical flow in the j direction.

[0128] Furthermore, optical flow flipping is actually a kind of convolution, and its expression is:

[0129]

[0130] in, Represents the pixel of the output image, I represents the pixel of the input image, i and j represent the coordinates, α represents the optical flow in the i direction, and β represents the optical flow in the j direction.

[0131] As one embodiment of the present disclosure, when the above deep learning model is adopted, refer to Figure 2 , the method of the embodiment of the present disclosure further includes:

[0132] A1001. Input the training video frames into the deep learning model and obtain the output training insertion frames.

[0133] A1002. Determine a loss function based on the training insertion frame and the preset standard insertion frame.

[0134] A1003. In response to the training termination condition not being met, the deep learning model is adjusted according to the loss function, and the step of inputting the training video frame into the deep learning model and obtaining the output training insertion frame (A1001) is returned.

[0135] When a deep learning model is used in the embodiments of the present disclosure, the deep learning model may be obtained by copying or the like, or may be obtained by training an initial deep learning model.

[0136] That is, the deep learning model is used to process the training video frames (training samples) in the preset training set (data set) to obtain the training insertion frames, and the training insertion frames are compared with the preset standard insertion frames to determine the loss function (i.e., the difference between the training insertion frames and the standard insertion frames); thereafter, if the training termination condition is met, the training can be terminated, and the deep learning model at this time can be used for subsequent video insertion; and if the training termination condition is met, the parameters in the deep learning model can be adjusted according to the loss function (but not including the adjustment of the pixel recombination unit), and then return to step A1001, reselect the training video frames, and start a new round of training.

[0137] Among them, the training termination conditions can be set specifically as needed, such as including the loss function being small enough, the loss function converging, reaching a predetermined number of training times, reaching a predetermined training time, etc.

[0138] Among them, the training video frames and standard insertion frames can be selected from preset datasets, such as the Viemo90k dataset, the Middlebury dataset, the adobe240fps dataset, the X4K1000FPS dataset, etc.

[0139] In some embodiments, the loss function includes L1 loss, perceptual loss, feature unified loss.

[0140] As one method of an embodiment of the present disclosure, the loss function may specifically include three parts: L1 loss, perceptual loss, and feature unified loss, and the three parts of the loss constitute the total loss function according to their respective weights.

[0141] For example, the L1 loss can be expressed as:

[0142] L1=||I out -I gt ||1;

[0143] Among them, L1 represents L1 loss, I out represents the output image (training insertion frame), I gt represents the reference image (standard insertion frame), and ‖‖1 represents a norm.

[0144] The perceptual loss can make the generated inserted frames more "realistic". It can be calculated using a VGG network (such as a VGG16 network), and the perceptual loss can be expressed as:

[0145] L vgg =||F(I out )-F(I gt )||2;

[0146] Among them, L vgg represents perceptual loss, F represents VGG16 network, I out represents the output image (training insertion frame), I gt represents the reference image (standard insertion frame), and ‖‖2 represents the binary norm.

[0147] The feature unification loss can increase the constraints on the pixel neighborhood in the final output inserted frame, thereby improving the effect of the inserted frame. Specifically, the difference between the Census sequence generated by stereo matching in the output and the true value can be used as the feature unification loss. The feature unification loss can be expressed as:

[0148] L census =||C(I out )-C(I gt )||2;

[0149] Among them, L census represents the unified feature loss, C represents the Census sequence, and I out represents the output image (training insertion frame), I gt represents the reference image (standard insertion frame), and ‖‖2 represents the binary norm.

[0150] Therefore, the overall loss function Loss can be expressed as:

[0151] Loss=L1+λ*L vgg +μ*L census ;

[0152] Among them, λ and μ are weight parameters.

[0153] In some embodiments, the loss function also includes a reverse supervision loss.

[0154] Furthermore, when the deep learning model is used for video interpolation based on optical flow, since it is based on optical flow, which displays motion information, and the optical flow between the video frame and the inserted frame is reverse, reverse supervision can also be set, and the optical flow process before and after the flipping is used as the corresponding supervision.

[0155] It should be understood that the above reverse supervision is only used in the training process, and does not exist in the actual deep learning model, and will not affect the speed of the deep learning model when used (for video interpolation).

[0156] Specifically, the reverse supervision loss can be expressed as:

[0157] L backward =‖W(Flow foreward )+Flow backward ‖2;

[0158] Among them, L backward represents the reverse supervision loss, W represents the optical flow flip (Warp) process, Flow foreward Represents the reference optical flow used for supervision, Flow backward represents the output optical flow, and ‖‖2 represents the two-norm.

[0159] Therefore, when the deep learning model is used for video interpolation based on optical flow, its overall loss function Loss can be expressed as:

[0160] Loss=L1+λ*L vgg +μ*L census +ε*L backward ;

[0161] Among them, ε is the weight parameter.

[0162] Second, refer to Figure 7 The embodiment of the present disclosure provides a device for video interpolation, which includes a memory and a processor; the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, the method for video interpolation of the embodiment of the present disclosure is implemented.

[0163] Thirdly, refer to Figure 8 The embodiment of the present disclosure provides a computer-readable medium on which a computer program is stored. When the computer program is executed by a processor, the video frame insertion method of the embodiment of the present disclosure is implemented.

[0164] Among them, the processor is a device with data processing capabilities, including but not limited to the central processing unit (CPU); the memory is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, and can realize information exchange between the memory and the processor, including but not limited to the data bus (Bus), etc.

[0165] Those skilled in the art will appreciate that all or some of the steps, systems, and functional modules / units in the apparatus disclosed above may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0166] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation.

[0167] Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-temporary medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical disk storage; magnetic cassettes, magnetic tapes, disk storage or other magnetic storage; any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0168] The present disclosure has disclosed example embodiments, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A method for inserting video frames, wherein: include: Downsampling the first input feature at multiple levels to generate image features at a lower level; the first input feature at each level includes a video frame or the image feature at the current level; Recombining pixels of the features to be sampled at multiple levels to generate additional image features of a higher level; the features to be sampled at each level include the image features of the current level and the additional image features generated by pixel recombining at a lower level; Upsampling the second input feature at multiple levels to generate a higher-level intermediate image feature; The second input features of each level include the additional image features and the intermediate image features of the current level.

2. The method according to claim 1, wherein: Also includes: A plurality of levels of original image features are generated according to the video frame; except for the first input features at the highest level, the first input features at other levels also include the original image features at the same level.

3. The method according to claim 1, wherein: The downsampling includes convolution downsampling, or includes sampling downsampling and pooling downsampling; The upsampling includes deconvolution upsampling.

4. The method according to claim 1, wherein: Also includes: Pre-convolution is performed on the image features and the additional image features at the same level to generate the features to be sampled.

5. The method according to claim 1, wherein: Also includes: generating a subframe according to the correlation characteristics of each of two adjacent video frames; The relevant features of the video frame include the video frame, the highest level of the intermediate image features generated based on the video frame, and the highest level of the additional image features generated based on the video frame; The two sub-frames generated according to two adjacent video frames are merged to obtain the inserted frame.

6. The method according to claim 5, wherein: Generating a subframe according to relevant characteristics comprises: Performing a dilated deformable convolution on the relevant features to generate a subframe.

7. The method according to claim 1, wherein: Also includes: The optical flow information of the current level is determined according to the intermediate image features of the previous level, the additional image features of the current level, and the optical flow information of the previous level, and the intermediate image features of the previous level and the additional image features of the current level are flipped according to the optical flow information of the current level.

8. The method according to claim 1, wherein: The downsampling, the pixel reconstruction, and the upsampling are performed in a deep learning model.

9. The method according to claim 8, wherein: Also includes: Input the training video frame into the deep learning model and obtain the output training insertion frame; Determine a loss function according to the training insertion frame and a preset standard insertion frame; In response to the training termination condition not being met, the deep learning model is adjusted according to the loss function, and the step of inputting the training video frame into the deep learning model and obtaining the output training insertion frame is returned.

10. The method according to claim 9, wherein: The loss functions include L1 loss, perceptual loss, and feature unified loss.

11. The method according to claim 10, wherein: The loss function also includes reverse supervision loss.

12. A video interpolation device, comprising a memory and a processor; the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, the video interpolation method described in any one of claims 1 to 11 is implemented.

13. A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the video frame insertion method according to any one of claims 1 to 11.

Images