Video frame interpolation method and system, computer device, and storage medium

Abstract

Embodiments of the present application disclose a video frame interpolation method. The method comprises: acquiring a plurality of consecutive frames from a video file to be processed, the plurality of consecutive frames comprising a target frame and the frame preceding the target frame and the frame following the target frame; on the basis of an optical flow estimation model, obtaining a reverse optical flow map between the target frame and the preceding frame and a forward optical flow map between the target frame and the following frame; on the basis of the similarity between the forward optical flow map and the reverse optical flow map, determining whether a transition exists among the plurality of consecutive frames; and, when no transition exists among the plurality of consecutive frames, interpolating a frame between the target frame and the following frame by means of an optical flow frame interpolation model. The embodiments of the present application further disclose a video frame interpolation system, a computer device, and a computer storage medium. The technical solution provided by the embodiments of the present application can prevent the appearance of artifacts in a synthesized intermediate frame due to the use of an unsuitable optical flow frame interpolation model for frame interpolation when a transition exists, while also avoiding misjudgment of a transition scene.

Description

Video frame insertion method, system, computer device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on December 27, 2023, with application number 202311828859.7 and invention name “Video interpolation method, system, computer device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to a video frame insertion method, system, computer device, and computer-readable storage medium. Background Art

[0003] Video interpolation, as the name suggests, involves adding several frames between consecutive frames in a video's time sequence, shortening the display time between frames and improving the video's frame rate and smoothness. Currently, there are three common methods for interpolation: the first is to directly copy the previous or next frame as an interpolated frame (duplication); the second is to blur the two frames, similar to double exposure, to create an intermediate frame (blending); and the third is an interpolation method based on deep learning models. This method analyzes and models the two frames to generate optical flow, thereby obtaining a linear mapping relationship between the frames and ultimately combining them to create an intermediate frame.

[0004] However, the inventors realized that the first interpolation method does not bring visual improvement in actual application, and sometimes even causes the video to freeze. The second interpolation method uses a simple double exposure blur, which will cause more serious artifacts. At the same time, one frame is clear and the other is blurred, which will bring additional burden to video encoding and decoding. Compared with the first two interpolation methods, the third interpolation method synthesizes more reasonable intermediate frames. However, when the two frames to be interpolated are the previous and next frames in the scene switch, the frames synthesized by the optical flow model will also have very serious artifacts.

[0005] Summary of the Invention

[0006] The purpose of the embodiments of the present application is to provide a video interpolation method, system, computer device and computer-readable storage medium to solve the technical problem of how to determine whether there is a transition in a video frame to guide the optical flow model interpolation method.

[0007] One aspect of an embodiment of the present application provides a video frame insertion method, the method comprising:

[0008] Acquire multiple consecutive frames from a video file to be processed, including a target frame and a previous frame and a next frame of the target frame;

[0009] Obtaining a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the subsequent frame based on an optical flow estimation model;

[0010] Determining whether there is a transition between the plurality of consecutive frames according to the similarity between the forward optical flow map and the backward optical flow map;

[0011] In a case where there is no transition between the plurality of consecutive frames, an optical flow interpolation model is used to interpolate frames between the target frame and the subsequent frame.

[0012] Optionally, obtaining a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the subsequent frame based on an optical flow estimation model includes:

[0013] Inputting the target frame and the previous frame into the optical flow estimation model to obtain an optical flow map from the target frame to the previous frame, which is the backward optical flow map;

[0014] The target frame and the subsequent frame are input into the optical flow estimation model to obtain an optical flow map from the target frame to the subsequent frame, which is the forward optical flow map.

[0015] Optionally, the determining whether there is a transition between the plurality of consecutive frames according to the similarity between the forward optical flow map and the backward optical flow map includes:

[0016] Performing an absolute value operation on the optical flow values ​​of the backward optical flow map and the forward optical flow map respectively;

[0017] Calculate the similarity of the two optical flow maps after taking the absolute value;

[0018] It is determined whether there is a transition between the plurality of consecutive frames according to the similarity and a preset threshold.

[0019] Optionally, the determining whether there is a transition between the plurality of consecutive frames according to the similarity between the forward optical flow map and the backward optical flow map includes:

[0020] Performing a smoothing operation on the backward optical flow map and the forward optical flow map respectively;

[0021] Take the absolute value of the optical flow values ​​of the two optical flow maps after the smoothing operation;

[0022] Calculate the similarity of the two optical flow maps after taking the absolute value;

[0023] It is determined whether there is a transition between the plurality of consecutive frames according to the similarity and a preset threshold.

[0024] Optionally, determining whether there is a transition between the plurality of consecutive frames according to the similarity and a preset threshold comprises:

[0025] When the similarity is greater than the preset threshold, determining that there is no transition between the target frame and the subsequent frame;

[0026] When the similarity is less than or equal to the preset threshold, it is determined that there is a transition between the target frame and the subsequent frame.

[0027] Optionally, the performing smoothing operations on the backward optical flow map and the forward optical flow map respectively includes:

[0028] At least one smoothing operation based on image erosion and image dilation is performed on the backward optical flow map and the forward optical flow map respectively.

[0029] Optionally, the method further includes:

[0030] In the case where there is a transition between the plurality of consecutive frames, a frame is inserted between the target frame and the subsequent frame by copying the target frame or the subsequent frame.

[0031] One aspect of an embodiment of the present application further provides a video frame insertion system, the system comprising:

[0032] An acquisition module is used to acquire a plurality of consecutive frames from a video file to be processed, including a target frame and a previous frame and a next frame of the target frame;

[0033] an estimation module, configured to obtain a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the subsequent frame based on an optical flow estimation model;

[0034] A determination module, configured to determine whether there is a transition between the plurality of consecutive frames based on the similarity between the forward optical flow map and the backward optical flow map;

[0035] The processing module is used to insert a frame between the target frame and the subsequent frame by using an optical flow interpolation model when there is no transition between the multiple consecutive frames.

[0036] Another aspect of an embodiment of the present application provides a computer device, the computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein when the processor executes the computer-readable instructions, the following steps are implemented:

[0037] Acquire multiple consecutive frames from a video file to be processed, including a target frame and a previous frame and a next frame of the target frame;

[0038] Obtaining a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the subsequent frame based on an optical flow estimation model;

[0039] Determining whether there is a transition between the plurality of consecutive frames according to the similarity between the forward optical flow map and the backward optical flow map;

[0040] In a case where there is no transition between the plurality of consecutive frames, an optical flow interpolation model is used to interpolate frames between the target frame and the subsequent frame.

[0041] One aspect of an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to cause the at least one processor to implement the following steps:

[0042] Acquire multiple consecutive frames from a video file to be processed, including a target frame and a previous frame and a next frame of the target frame;

[0043] Obtaining a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the subsequent frame based on an optical flow estimation model;

[0044] Determining whether there is a transition between the plurality of consecutive frames according to the similarity between the forward optical flow map and the backward optical flow map;

[0045] In a case where there is no transition between the plurality of consecutive frames, an optical flow interpolation model is used to interpolate frames between the target frame and the subsequent frame.

[0046] The video interpolation method, system, computer device, and computer-readable storage medium provided in the embodiments of the present application can utilize an optical flow estimation model to obtain a backward optical flow map between the first two frames and a forward optical flow map between the last two frames based on multiple consecutive frames of a video. Based on these two optical flow maps, the method determines whether a video frame has a transition based on similarity, thereby guiding the subsequent interpolation method. This avoids using an inappropriate optical flow interpolation model for interpolation when there is a transition, which results in artifacts in the synthesized intermediate frames. It also avoids mistakenly judging consecutive frames from the same lens as frames before and after the transition when scenes with large dynamic changes are involved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] FIG1 schematically shows a schematic diagram of an operating environment of an embodiment of the present application;

[0048] FIG2 schematically shows a flow chart of a video frame insertion method according to the first embodiment of the present application;

[0049] FIG3 schematically shows a sub-step flow chart of step S204 in FIG2 ;

[0050] FIG4 schematically shows another sub-step flow chart of step S204 in FIG2 ;

[0051] FIG5 schematically shows a program module diagram of a video frame insertion system according to a second embodiment of the present application;

[0052] FIG6 schematically shows a hardware architecture diagram of a computer device suitable for a video frame insertion method according to the third embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] It should be noted that the descriptions of "first", "second", etc. in the embodiments of the present application are for descriptive purposes only and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various application embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0055] The following is an explanation of the terms used in this application:

[0056] Video interpolation: Add several frames between every two frames of the original picture, shorten the display time between frames, and improve the frame rate and smoothness of the video.

[0057] Optical flow: The instantaneous speed of pixel movement of a moving object in space on the observation imaging plane.

[0058] Peak Signal-to-Noise Ratio (PSNR) is an engineering term that represents the ratio of the maximum possible signal power to the destructive noise power that affects its representation accuracy. Because many signals have very wide dynamic ranges, PSNR is often expressed in logarithmic decibel units. PSNR is often used as a measure of signal reconstruction quality in fields such as image compression and is often simply defined as the mean square error (MSE).

[0059] Structural Similarity (SSIM): A metric that measures the similarity between two images. This metric was first proposed by the Image and Video Engineering Laboratory at the University of Texas at Austin. Humans are insensitive to the absolute brightness or color of pixels but are highly sensitive to the location of edges and textures. SSIM mimics human perception by focusing primarily on edge and texture similarity.

[0060] Of the three commonly used frame interpolation methods, the first, duplicating frames, relies on completely copying the previous or next frame to increase the frame rate. However, in practice, this doesn't provide a visual improvement and can sometimes even cause the video to appear jerky. While the second, blending frames, references information from both the preceding and following frames, the simple double-exposure blurring can lead to severe artifacts. Furthermore, having one frame clear and one blurred can place an additional burden on video encoding and decoding. The third, based on deep learning optical flow models, effectively models the mapping relationship between the target intermediate frame and the preceding and following frames using methods like fitting, thereby restoring the intermediate frame. Compared to the first two interpolation methods, the synthesized intermediate frame is more reasonable. Numerous practical application results have also demonstrated that interpolation using deep learning optical flow models yields far superior results to those achieved by duplicating or blending frames.

[0061] However, a common problem with the third interpolation method is that before interpolating two consecutive frames in a video, it is necessary to determine whether the two frames to be interpolated are consecutive frames from the same shot or the two frames before and after a scene change. If they are consecutive frames from the same shot, interpolation can be performed directly; otherwise, interpolation cannot be performed. Because the two frames before and after a scene change are often discontinuous in content and image quality, and there are significant differences, the intermediate frames synthesized based on the optical flow model will produce severe artifacts.

[0062] To circumvent this situation, a common solution in the industry is to calculate metrics such as PSNR or SSIM between two frames before interpolating to determine whether the two frames are at the scene change moment. Both PSNR and SSIM are pixel-level image quality metrics. They only reflect the signal-to-noise ratio or structural similarity of the image, but not its semantic information or visual perception. PSNR and SSIM are also global metrics, assigning equal weight to every pixel in the image, regardless of its location or importance. Most importantly, these metrics fail to incorporate the temporal information of the video, which is often used for video interpolation. For example, optical flow-based interpolation models use the temporal information between two frames to perform motion estimation and synthesize the intermediate frames. These metrics are often unreliable when dealing with scenes with large dynamic changes.

[0063] In view of this, the embodiments of the present application provide a new video interpolation solution. In this technical solution, the continuity of the content between frames is determined by judging the optical flow changes between every two frames of N consecutive video frames, avoiding the problem of traditional solutions that only compare global pixel differences without paying attention to the semantic changes between frames.

[0064] Figure 1 shows a schematic diagram of an environmental application according to an embodiment of the present application. In an exemplary embodiment, the terminal 10 is used to receive or obtain a video file, and send the video file to the server 20 for frame insertion processing. After receiving the video file, the server 20 obtains a plurality of continuous frames from the video file, including a target frame and the previous frame and the next frame of the target frame. Then, based on the optical flow estimation model, a forward optical flow map between the target frame and the previous frame, and a backward optical flow map between the target frame and the next frame are obtained, and then, based on the similarity between the forward optical flow map and the backward optical flow map, it is determined whether there is a transition between the plurality of continuous frames. In the case that there is no transition between the plurality of continuous frames, a frame is inserted between the target frame and the next frame by using the optical flow interpolation model. In the case that there is a transition between the plurality of continuous frames, a frame is inserted between the target frame and the next frame by copying frames.

[0065] In the exemplary embodiment, the terminal 10 includes, but is not limited to, a personal computer (PC), a mobile phone, a tablet personal computer, a laptop computer, or a photographic device such as a camera or a video camera. The server 20 may be a computing device such as a rack server, a blade server, a tower server, or a cabinet server, and may be a standalone server or a server cluster consisting of multiple servers.

[0066] Of course, in other optional embodiments, the server 20 may directly obtain the video file without the terminal 10 sending the video file, or the server 20 may receive the video file from another device or download the video file from the network. The terminal 10 may also directly perform frame insertion processing on the video file without sending it to the server 20 for processing.

[0067] The following describes the technical solutions of the present application through multiple embodiments, taking the server 20 as an example. It should be noted that these embodiments can be implemented in various forms and should not be construed as being limited to the embodiments described herein.

[0068] Example 1

[0069] FIG2 schematically illustrates a flowchart of the steps of the video frame insertion method according to the first embodiment of the present application. It is understood that the flowchart in this method embodiment is not intended to limit the order in which the steps are executed. The execution subject of this method embodiment can be either a client or a server, without limitation.

[0070] As shown in FIG2 , the video frame insertion method may include steps S200 to S206 , wherein:

[0071] Step S200 , obtaining a plurality of continuous frames from a video file to be processed, including a target frame and a previous frame and a next frame of the target frame.

[0072] Different from the traditional solution that determines whether there is a transition and performs interpolation processing through optical flow estimation between the previous and next frames, this embodiment performs the above judgment and processing based on at least three consecutive frames. When it is necessary to perform interpolation processing on a video file, it is first necessary to obtain these three consecutive frames from the video file. In this embodiment, the second frame of the three consecutive frames is used as the target frame, that is, interpolation is performed after the second frame. Therefore, the three consecutive frames obtained are the target frame, and the previous frame and the next frame of the target frame. For example, assuming there are three consecutive frames I1, I2, and I3, then I2 is the target frame, I1 is the previous frame, and I3 is the next frame. It is necessary to insert frame I2 after the target frame I1. 2.5 .

[0073] Step S202 : obtaining a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the next frame based on an optical flow estimation model.

[0074] Specifically, the target frame and the previous frame are input into the optical flow estimation model to obtain an optical flow map from the target frame to the previous frame, which is the backward optical flow map. The target frame and the next frame are input into the optical flow estimation model to obtain an optical flow map from the target frame to the next frame, which is the forward optical flow map.

[0075] In this embodiment, the optical flow estimation model can adopt any existing feasible deep learning model, such as an optical flow estimation model based on FlowNet, etc., and is not limited here.

[0076] Step S204 : determining whether there is a transition between the plurality of consecutive frames based on the similarity between the forward optical flow map and the backward optical flow map.

[0077] In a conventional solution, for three consecutive frames I1, I2, and I3, when determining whether there is a transition between I1 and I2, the similarity psnr_1_2 between I1 and I2 can be calculated based on the PSNR. When the similarity psnr_1_2 is less than a pre-set threshold, it is determined that there is a transition between I1 and I2. When determining whether there is a transition between I2 and I3, the similarity psnr_2_3 between I2 and I3 can be calculated. When the similarity psnr_2_3 is less than a pre-set threshold, it is determined that there is a transition between I2 and I3.

[0078] In practice, this traditional approach can indeed detect most transitions. However, when encountering rapid camera movements, the similarity between the two frames will be very low. In this case, even if I1, I2, and I3 are three consecutive frames captured by the same camera, each pair of frames will still be judged as a transition due to the low similarity value.

[0079] In this embodiment, in order to avoid the above problem, an optical flow judgment method is used to determine whether a transition occurs.

[0080] The following provides an exemplary solution for determining whether there is a transition between the plurality of consecutive frames.

[0081] In an optional embodiment, as shown in FIG3 , step S204 may include:

[0082] Step S300 , performing an absolute value operation on the optical flow values ​​of the backward optical flow map and the forward optical flow map respectively.

[0083] For example, for three consecutive frames I1, I2, and I3, I1 and I2, I2 and I3 are input into the optical flow estimation model F respectively, and the corresponding optical flow maps are output to obtain the backward optical flow map f from I2 to I1. 2-＞1 And the forward optical flow graph f from I2 to I3 2-＞3 Then, f 2-＞1 and f 2-＞3 Take the absolute value of the optical flow value in and get |f 2-＞1 | and |f 2-＞3 |.

[0084] Step S302 : calculating the similarity of the two optical flow maps after taking the absolute values.

[0085] In this embodiment, this step can calculate the similarity based on the PSNR index. 2-＞1 | and |f 2-＞3 | Calculate the similarity based on the PSNR index and obtain the similarity psnr = PSNR(|f 2-＞1 |,|f 2-＞3 |).

[0086] Step S304: determining whether there is a transition between the plurality of consecutive frames according to the similarity and a preset threshold.

[0087] In this embodiment, whether a transition exists between the plurality of consecutive frames specifically refers to whether a transition exists between the target frame and the subsequent frame. If the calculated similarity is greater than a preset threshold, it is determined that no transition exists between the target frame and the subsequent frame; otherwise, if the similarity is less than or equal to the preset threshold, it is determined that a transition exists between the target frame and the subsequent frame.

[0088] For example, when the similarity psnr=PSNR(|f 2-＞1 |,|f 2-＞3 |) is greater than the preset threshold, it indicates that there is a transition between I2 and I3; otherwise, it indicates that there is no transition between I2 and I3.

[0089] The following provides an exemplary solution for determining whether there is a transition between the plurality of consecutive frames.

[0090] In an optional embodiment, as shown in FIG4 , step S204 may further include:

[0091] Step S400 : performing a smoothing operation on the backward optical flow map and the forward optical flow map respectively.

[0092] The deep learning-based optical flow estimation model has strong noise immunity. In this embodiment, to further enhance this immunity, the optical flow map output by the model undergoes several (at least one) smoothing operations based on image erosion and dilation. Erosion removes image artifacts and details, and is generally used to eliminate noise and segment independent image elements. It is essentially a form of spatial filtering: a mask is set, and the center of the mask slides over each pixel one by one. The value of the current pixel (i.e., the location corresponding to the mask center) is set to the minimum value of the pixels in the area covered by the mask. The erosion operation eliminates object boundary points, shrinking the target, and can eliminate noise points smaller than the structural element. Dilation is an operation that finds local maxima. It convolves the structural element with the image, calculating the maximum value of the pixels in the area covered by the structural element and assigning this maximum value to the pixel designated by the reference point. This gradually increases the highlight area in the image. The dilation operation merges all background points that touch the object into the object, enlarging the target and filling holes in the object. The core of dilation and erosion operations is the structuring element, which is generally composed of a matrix with elements set to 1 or 0. The area where the structuring element is set to 1 defines the area of ​​the image, and the pixels within the area are considered when performing morphological operations such as dilation and erosion.

[0093] For example, for three consecutive frames I1, I2, and I3, I1 and I2, I2 and I3 are input into the optical flow estimation model F respectively, and the corresponding optical flow maps are output to obtain the backward optical flow map f from I2 to I1. 2-＞1 And the forward optical flow graph f from I2 to I3 2-＞3 Then, f 2-＞1 and f 2-＞3 Perform smoothing operation to get smooth_f 2-＞1 and smooth_f 2-＞3 .

[0094] Step S402 , performing an absolute value operation on the optical flow values ​​of the two optical flow maps after the smoothing operation.

[0095] For example, the above smooth_f 2-＞1 and smooth_f 2-＞3 Take the absolute value operation and get |smooth_f 2-＞1 |and|smooth_f 2-＞3 |.

[0096] Step S404 : calculating the similarity of the two optical flow maps after taking the absolute values.

[0097] In this embodiment, this step can calculate the similarity based on the PSNR index. 2-＞1 |and|smooth_f 2-＞3 | Calculate similarity based on PSNR index, and obtain similarity psnr = PSNR(|smooth_f 2-＞1 |,|smooth_f 2-＞1 |).

[0098] Step S406 : determining whether there is a transition between the plurality of consecutive frames according to the similarity and a preset threshold.

[0099] In this embodiment, whether a transition exists between the plurality of consecutive frames specifically refers to whether a transition exists between the target frame and the subsequent frame. If the calculated similarity is greater than the preset threshold, it is determined that there is no transition between the target frame and the subsequent frame; otherwise, it is determined that there is a transition between the target frame and the subsequent frame.

[0100] For example, when the similarity psnr=PSNR(|smooth_f 2-＞1 |,|smooth_f 2-＞3 |) is greater than the preset threshold, it indicates that there is a transition between I2 and I3; otherwise, it indicates that there is no transition between I2 and I3.

[0101] The following uses a specific example to illustrate the difference between this embodiment and the traditional solution in determining whether a transition exists.

[0102] Suppose a black square with a side length of 50 is moved within a square frame with a pure white background of 150*150 pixels, resulting in I1, I2, and I3. I1 contains a square with a center at (x=25, y=75). I2 is the square in I1 shifted 50 pixels to the right, resulting in the center of the square at (x=75, y=75). Similarly, I3 is shifted another 50 pixels to the right, resulting in the center of the square at (x=125, y=75).

[0103] Using the traditional solution, it can be calculated that the PSNR similarity of I1 and I2, and I3 and I2 is 6.53, while the general transition threshold is much larger than this value. The optical flow corresponding to each pixel point contained in the square during the movement from I2 to I1 is f_2->1=-50, and the optical flow corresponding to the square during the movement from I2 to I3 is f_2->3=50. Although the directions of the optical flows of the two are different, the values ​​are the same, that is, the optical flow scalar maps of the two are completely consistent. Therefore, this embodiment can determine whether a transition exists by calculating the similarity of the optical flow scalar maps.

[0104] This embodiment converts the pixel similarity judgment between two frames in the traditional solution into motion similarity judgment, and completes the judgment process in another semantics that is more consistent with human eye perception, which is also more suitable for actual conditions.

[0105] Returning to FIG. 2 , in step S206 , when there is no transition between the plurality of consecutive frames, interpolating a frame between the target frame and the subsequent frame is performed using an optical flow interpolation model.

[0106] Specifically, if there is no transition between the target frame and the subsequent frame, an optical flow interpolation model can be directly used to interpolate between the target frame and the subsequent frame. That is, at least one intermediate frame is inserted between the target frame and the subsequent frame. The optical flow interpolation model can be any existing feasible deep learning-based model and is not limited here.

[0107] Step S208 : When there is a transition between the plurality of consecutive frames, insert a frame between the target frame and the subsequent frame by copying the frame.

[0108] If there is a transition between the target frame and the subsequent frame, the optical flow interpolation model cannot be used to directly interpolate between the target frame and the subsequent frame. In this embodiment, a copy frame method can be used to interpolate between the target frame and the subsequent frame. In other words, the target frame or the subsequent frame is copied as an interpolated intermediate frame.

[0109] Of course, in other optional embodiments, when there is a transition between the target frame and the subsequent frame, other methods other than the optical flow interpolation model can also be used to perform interpolation processing between the target frame and the subsequent frame, which will not be repeated here.

[0110] After completing the interpolation process between the target frame and the subsequent frame according to the above steps, the subsequent frame is used as the next target frame and the above steps are repeated until the interpolation of the entire video is completed.

[0111] The video interpolation method described in the first embodiment above can use an optical flow estimation model to obtain a backward optical flow map between the first two frames and a forward optical flow map between the last two frames based on three consecutive frames of the video. Based on these two optical flow maps, it is judged whether there is a transition in the video frame according to similarity, and the subsequent interpolation method is guided. Thus, it is avoided that an inappropriate optical flow interpolation model is used for interpolation when there is a transition, resulting in artifacts in the synthesized intermediate frame. At the same time, the time domain information of the video is combined when judging whether there is a transition, and it is also avoided that the consecutive frames under the same lens are mistakenly judged as frames before and after the transition when scenes with large dynamic changes are involved. In addition, this solution is scalable and can be used as a plug-and-play extension module, applicable to any interpolation model.

[0112] Example 2

[0113] Figure 5 schematically illustrates a program module diagram of a video interpolation system according to Example 2 of the present application. The video interpolation system can be divided into one or more program modules, one or more of which are stored in a storage medium and executed by one or more processors to implement the embodiment of the present application. The program modules referred to in the embodiment of the present application are a series of computer-readable instruction segments that can perform specific functions. The following description will specifically introduce the functions of each program module in this embodiment.

[0114] As shown in FIG5 , the video frame insertion system 800 can be used in a terminal or a server, and can include an acquisition module 810 , an estimation module 820 , a determination module 830 , and a processing module 840 , wherein:

[0115] The acquisition module 810 is configured to acquire a plurality of continuous frames from the video file to be processed, including a target frame and a previous frame and a next frame of the target frame.

[0116] Unlike traditional approaches that use optical flow estimation between two frames to determine whether a transition exists and perform frame interpolation, this embodiment performs the above determination and processing based on at least three consecutive frames. When frame interpolation is required for a video file, the three consecutive frames must first be obtained from the video file. In this embodiment, the second of the three consecutive frames is used as the target frame, that is, the frame interpolation is performed after the second frame. Therefore, the three consecutive frames obtained are the target frame, the frame before it, and the frame after it.

[0117] The estimation module 820 is configured to obtain a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the subsequent frame based on an optical flow estimation model.

[0118] Specifically, the target frame and the previous frame are input into the optical flow estimation model to obtain an optical flow map from the target frame to the previous frame, which is the backward optical flow map. The target frame and the next frame are input into the optical flow estimation model to obtain an optical flow map from the target frame to the next frame, which is the forward optical flow map.

[0119] In this embodiment, the optical flow estimation model can adopt any existing feasible deep learning model, such as an optical flow estimation model based on FlowNet, etc., and is not limited here.

[0120] The determination module 830 is configured to determine whether there is a transition between the plurality of consecutive frames according to the similarity between the forward optical flow map and the backward optical flow map.

[0121] Whether there is a transition between the plurality of consecutive frames specifically refers to whether there is a transition between the target frame and the subsequent frame. In an optional embodiment, the determination module 830 can determine whether there is a transition between the plurality of consecutive frames by the following process:

[0122] (1) Performing an absolute value operation on the optical flow values ​​of the backward optical flow map and the forward optical flow map respectively.

[0123] (2) Calculate the similarity of the two optical flow maps after taking the absolute value.

[0124] (3) Determine whether there is a transition between the plurality of consecutive frames based on the similarity and a preset threshold.

[0125] If the calculated similarity is greater than the preset threshold, it is determined that there is no transition between the target frame and the subsequent frame; otherwise, if the similarity is less than or equal to the preset threshold, it is determined that there is a transition between the target frame and the subsequent frame.

[0126] In another optional embodiment, the determination module 830 may also determine whether there is a transition between the plurality of consecutive frames through the following process:

[0127] (1) Smoothing operations are performed on the backward optical flow map and the forward optical flow map respectively.

[0128] The optical flow estimation model based on deep learning has a strong anti-interference ability against noise. In this embodiment, in order to further enhance the anti-interference ability, the optical flow map output by the model will be subjected to several (at least one) smoothing operations based on image erosion and image expansion.

[0129] (2) Take the absolute value of the optical flow values ​​of the two optical flow maps after the smoothing operation.

[0130] (3) Calculate the similarity of the two optical flow maps after taking the absolute value.

[0131] (4) Determine whether there is a transition between the plurality of consecutive frames based on the similarity and a preset threshold.

[0132] This embodiment converts the pixel similarity judgment between two frames in the traditional solution into motion similarity judgment, and completes the judgment process in another semantics that is more consistent with human eye perception, which is also more suitable for actual conditions.

[0133] The processing module 840 is configured to interpolate a frame between the target frame and the subsequent frame by using an optical flow interpolation model when there is no transition between the plurality of consecutive frames.

[0134] Specifically, if there is no transition between the target frame and the subsequent frame, an optical flow interpolation model can be directly used to interpolate between the target frame and the subsequent frame. That is, at least one intermediate frame is inserted between the target frame and the subsequent frame. The optical flow interpolation model can be any existing feasible deep learning-based model and is not limited here.

[0135] In an optional embodiment, the processing module 840 is further configured to insert a frame between the target frame and the subsequent frame by copying frames when there is a transition between the plurality of consecutive frames.

[0136] If there is a transition between the target frame and the subsequent frame, the optical flow interpolation model cannot be used to directly interpolate between the target frame and the subsequent frame. In this embodiment, a copy frame method can be used to interpolate between the target frame and the subsequent frame. In other words, the target frame or the subsequent frame is copied as an interpolated intermediate frame.

[0137] Of course, in other optional embodiments, when there is a transition between the target frame and the subsequent frame, other methods other than the optical flow interpolation model can also be used to perform interpolation processing between the target frame and the subsequent frame, which will not be repeated here.

[0138] After completing the interpolation process between the target frame and the subsequent frame according to the above process, the subsequent frame is used as the next target frame and the above process is repeated until the interpolation of the entire video is completed.

[0139] The specific functions of the above modules are described in the first embodiment and will not be repeated here.

[0140] Example 3

[0141] Figure 6 schematically illustrates a hardware architecture diagram of a computer device suitable for the video frame interpolation method according to Example 3 of the present application. In this embodiment, the computer device 10000 is a device, such as a server or terminal, that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions.

[0142] As shown in FIG6 , the computer device 10000 includes at least but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other via a system bus.

[0143] The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Of course, the memory 10010 may also include both the internal storage module of the computer device 10000 and its external storage devices. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed on the computer device 10000, such as the computer-readable instructions of the above-mentioned video frame insertion method. In addition, the memory 10010 can also be used to temporarily store various data that has been output or is about to be output.

[0144] In some embodiments, the processor 10020 may be a CPU, a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to execute computer-readable instructions stored in the memory 10010 or process data.

[0145] The network interface 10030 may include a wireless network interface or a wired network interface. The network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal via a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, or Wi-Fi.

[0146] It should be noted that FIG6 only shows a computer device having components 10010 - 10030 , but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0147] In this embodiment, the video interpolation method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (processor 10020 in this embodiment) to complete this application.

[0148] Example 4

[0149] This embodiment further provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor, the following steps are implemented:

[0150] Acquire multiple consecutive frames from a video file to be processed, including a target frame and a previous frame and a next frame of the target frame;

[0151] Obtaining a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the subsequent frame based on an optical flow estimation model;

[0152] Determining whether there is a transition between the plurality of consecutive frames according to the similarity between the forward optical flow map and the backward optical flow map;

[0153] In a case where there is no transition between the plurality of consecutive frames, an optical flow interpolation model is used to interpolate frames between the target frame and the subsequent frame.

[0154] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk equipped with the computer device, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Of course, the computer-readable storage medium may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store the operating system and various application software installed on the computer device, such as the computer-readable instructions of the video interpolation method in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are about to be output.

[0155] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented using computer-readable instructions executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0156] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video frame interpolation method, the method comprising: Obtaining a plurality of consecutive frames from a video file to be processed, including a target frame and the frame before and after the target frame; Obtaining a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the next frame based on an optical flow estimation model; Determining whether there is a transition between the plurality of consecutive frames according to the similarity between the forward optical flow map and the backward optical flow map; In the case where there is no transition between the plurality of consecutive frames, performing frame interpolation between the target frame and the next frame through an optical flow interpolation model.

2. The method according to claim 1, wherein obtaining the backward optical flow map between the target frame and the previous frame, and the forward optical flow map between the target frame and the next frame based on the optical flow estimation model comprises: Inputting the target frame and the previous frame into the optical flow estimation model to obtain an optical flow map from the target frame to the previous frame, which is the backward optical flow map; Inputting the target frame and the next frame into the optical flow estimation model to obtain an optical flow map from the target frame to the next frame, which is the forward optical flow map.

3. The method according to claim 1, wherein determining whether there is a transition between the plurality of consecutive frames according to the similarity between the forward optical flow map and the backward optical flow map comprises: Performing an absolute value operation on the optical flow values of the backward optical flow map and the forward optical flow map respectively; Calculating the similarity for the two optical flow maps after taking the absolute value; Determining whether there is a transition between the plurality of consecutive frames according to the similarity and a preset threshold.

4. The method according to claim 1, wherein determining whether there is a transition between the plurality of consecutive frames according to the similarity between the forward optical flow map and the backward optical flow map comprises: Performing a smoothing operation on the backward optical flow map and the forward optical flow map respectively; Performing an absolute value operation on the optical flow values of the two optical flow maps after the smoothing operation respectively; Calculating the similarity for the two optical flow maps after taking the absolute value; Determining whether there is a transition between the plurality of consecutive frames according to the similarity and a preset threshold.

5. The method according to claim 3, wherein determining whether there is a transition between the plurality of consecutive frames according to the similarity and a preset threshold comprises: In the case where the similarity is greater than the preset threshold, determining that there is no transition between the target frame and the next frame; In the case where the similarity is less than or equal to the preset threshold, determining that there is a transition between the target frame and the next frame.

6. The method according to claim 4, wherein determining whether there is a transition between the plurality of consecutive frames according to the similarity and a preset threshold comprises: In the case where the similarity is greater than the preset threshold, determining that there is no transition between the target frame and the next frame; In the case where the similarity is less than or equal to the preset threshold, determining that there is a transition between the target frame and the next frame.

7. The method according to claim 4, wherein performing a smoothing operation on the backward optical flow map and the forward optical flow map respectively comprises: Perform at least one smoothing operation based on image erosion and image dilation on the backward optical flow map and the forward optical flow map respectively.

8. The method according to claim 1, wherein the method further comprises: In the case where there is a transition between the consecutive multiple frames, insert frames between the target frame and the subsequent frame by copying the target frame or copying the subsequent frame.

9. A video frame interpolation system, the system comprising: An acquisition module, configured to acquire consecutive multiple frames from a video file to be processed, including a target frame and the previous frame and the subsequent frame of the target frame; An estimation module, configured to obtain a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the subsequent frame based on an optical flow estimation model; A determination module, configured to determine whether there is a transition between the consecutive multiple frames according to the similarity between the forward optical flow map and the backward optical flow map; A processing module, configured to insert frames between the target frame and the subsequent frame through an optical flow interpolation model in the case where there is no transition between the consecutive multiple frames.

10. A computer device, the computer device comprising a memory, a processor, and computer-readable instructions stored on the memory and executable on the processor, wherein when the processor executes the computer-readable instructions, the following steps are implemented: Acquire consecutive multiple frames from a video file to be processed, including a target frame and the previous frame and the subsequent frame of the target frame; Obtain a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the subsequent frame based on an optical flow estimation model; According to the similarity between the forward optical flow map and the backward optical flow map, determine whether There is a transition; In the case where there is no transition between the consecutive multiple frames, insert frames between the target frame and the subsequent frame through an optical flow interpolation model.

11. The computer device according to claim 10, wherein obtaining the backward optical flow map between the target frame and the previous frame, and the forward optical flow map between the target frame and the subsequent frame based on the optical flow estimation model comprises: Input the target frame and the previous frame into the optical flow estimation model to obtain an optical flow map from the target frame to the previous frame, which is the backward optical flow map; Input the target frame and the subsequent frame into the optical flow estimation model to obtain an optical flow map from the target frame to the subsequent frame, which is the forward optical flow map.

12. The computer device according to claim 10, wherein determining whether there is a transition between the consecutive multiple frames according to the similarity between the forward optical flow map and the backward optical flow map comprises: Perform an absolute value operation on the optical flow values of the backward optical flow map and the forward optical flow map respectively; Calculate the similarity for the two optical flow maps after taking the absolute value; Determine whether there is a transition between the consecutive multiple frames according to the similarity and a preset threshold.

13. The computer device according to claim 10, wherein determining whether there is a transition between the consecutive multiple frames according to the similarity between the forward optical flow map and the backward optical flow map comprises: Perform smoothing operations on the backward optical flow map and the forward optical flow map respectively; Perform absolute value operations on the optical flow values of the two smoothed optical flow maps respectively; Calculate the similarity for the two optical flow maps after taking the absolute values; Determine whether there is a transition between the consecutive multiple frames according to the similarity and a preset threshold.

14. The computer device according to claim 12, wherein the determining whether there is a transition between the consecutive multiple frames according to the similarity and the preset threshold includes: In the case where the similarity is greater than the preset threshold, determine that there is no transition between the target frame and the subsequent frame; In the case where the similarity is less than or equal to the preset threshold, determine that there is a transition between the target frame and the subsequent frame.

15. The computer device according to claim 13, wherein the determining whether there is a transition between the consecutive multiple frames according to the similarity and the preset threshold includes: In the case where the similarity is greater than the preset threshold, determine that there is no transition between the target frame and the subsequent frame; In the case where the similarity is less than or equal to the preset threshold, determine that there is a transition between the target frame and the subsequent frame.

16. The computer device according to claim 13, wherein the performing smoothing operations on the backward optical flow map and the forward optical flow map respectively includes: Perform at least one smoothing operation based on image erosion and image dilation on the backward optical flow map and the forward optical flow map respectively.

17. The computer device according to claim 10, when the processor executes the computer-readable instructions, further implements the following steps: In the case where there is a transition between the consecutive multiple frames, insert a frame between the target frame and the subsequent frame by copying the target frame or copying the subsequent frame.

18. A computer-readable storage medium, on which computer-readable instructions are stored, and the computer-readable instructions can be executed by at least one processor, so that the at least one processor executes the following steps: Obtain consecutive multiple frames from a video file to be processed, including a target frame and the previous frame and the subsequent frame of the target frame; Obtain a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the subsequent frame based on an optical flow estimation model; Determine whether there is a transition between the consecutive multiple frames according to the similarity between the forward optical flow map and the backward optical flow map; In the case where there is no transition between the consecutive multiple frames, insert a frame between the target frame and the subsequent frame through an optical flow interpolation model.

19. The computer-readable storage medium according to claim 18, wherein the obtaining a backward optical flow map between the target frame and the previous frame, and a forward optical flow map between the target frame and the subsequent frame based on the optical flow estimation model includes: Input the target frame and the previous frame into the optical flow estimation model to obtain an optical flow map from the target frame to the previous frame, which is the backward optical flow map; Input the target frame and the subsequent frame into the optical flow estimation model to obtain an optical flow map from the target frame to the subsequent frame, which is the forward optical flow map.

20. The computer-readable storage medium according to claim 18, wherein determining whether there is a transition between the consecutive multiple frames according to the similarity between the forward optical flow map and the backward optical flow map includes: Perform an absolute value operation on the optical flow values of the backward optical flow map and the forward optical flow map respectively; Calculate the similarity for the two optical flow maps after taking the absolute value; Determine whether there is a transition between the consecutive multiple frames according to the similarity and a preset threshold.

Images