Video frame insertion method and device

Abstract

The present application provides a video interpolation method and device, wherein the video interpolation method includes: obtaining a continuous first video frame and a second video frame from a video to be interpolated; determining a first subtitle area in the first video frame and a second subtitle area in the second video frame based on subtitle position parameters in the video to be interpolated; determining a first similarity between the first video frame and the second video frame, and determining a second similarity between the first subtitle area and the second subtitle area; determining a target interpolation frame based on the first similarity and the second similarity, and inserting the target interpolation frame between the first video frame and the second video frame. In this way, the similarity of the two video frames and the similarity of the subtitle areas in the two video frames are considered at the same time, and the target interpolation frame is comprehensively determined, so that subtitle artifacts are avoided and the interpolation quality and interpolation effect are improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video frame insertion method. The present application also relates to a video frame insertion device, a computing device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of computer technology and network technology, various videos emerge in an endless stream, and watching videos has become an important way for people to work, relax, and entertain. In order to improve the frame rate and smoothness of the video, a synthetic frame can be inserted between two consecutive video frames in the video to shorten the inter-frame display time. Usually, before inserting two consecutive video frames, it is necessary to determine whether the two video frames to be inserted belong to the continuous frames of the same shot, or to the two frames before and after the scene switching. If they are continuous frames in the same shot, they can be inserted directly; if not, they cannot be inserted.

[0003] In the prior art, two consecutive video frames are often obtained, and then the similarity of the two video frames is determined to determine whether the two video frames switch shots, thereby determining whether to perform interpolation. However, in the above method, the similarity of the two video frames is considered. If the two video frames include subtitles, even if the overall pictures and contents of the two are continuous, if the subtitles are different, interpolation at this time may cause artifacts in the subtitles, thereby resulting in poor interpolation quality and interpolation effect. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a video frame insertion method. The present application also relates to a video frame insertion device, a computing device, and a computer-readable storage medium to solve the technical problems of subtitle artifacts, poor frame insertion quality, and poor frame insertion effect in the prior art.

[0005] According to a first aspect of an embodiment of the present application, a video frame insertion method is provided, comprising:

[0006] Acquire a first video frame and a second video frame continuously from the video to be inserted;

[0007] Based on the subtitle position parameters in the video frame to be inserted, determining a first subtitle area in the first video frame, and determining a second subtitle area in the second video frame;

[0008] determining a first similarity between the first video frame and the second video frame, and determining a second similarity between the first subtitle region and the second subtitle region;

[0009] Based on the first similarity and the second similarity, a target frame to be inserted is determined, and the target frame to be inserted is inserted between the first video frame and the second video frame.

[0010] According to a second aspect of an embodiment of the present application, a video frame insertion device is provided, including:

[0011] An acquisition module is configured to acquire a continuous first video frame and a second video frame from the video to be inserted;

[0012] A first determination module is configured to determine a first subtitle area in a first video frame and a second subtitle area in a second video frame based on subtitle position parameters in the video frame to be inserted;

[0013] a second determination module configured to determine a first similarity between the first video frame and the second video frame, and to determine a second similarity between the first subtitle region and the second subtitle region;

[0014] The third determination module is configured to determine a target frame to be inserted based on the first similarity and the second similarity, and insert the target frame to be inserted between the first video frame and the second video frame.

[0015] According to a third aspect of an embodiment of the present application, a computing device is provided, including:

[0016] Memory and processor;

[0017] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to implement the following method:

[0018] Acquire a first video frame and a second video frame continuously from the video to be inserted;

[0019] Based on the subtitle position parameters in the video frame to be inserted, determining a first subtitle area in the first video frame, and determining a second subtitle area in the second video frame;

[0020] determining a first similarity between the first video frame and the second video frame, and determining a second similarity between the first subtitle region and the second subtitle region;

[0021] Based on the first similarity and the second similarity, a target frame to be inserted is determined, and the target frame to be inserted is inserted between the first video frame and the second video frame.

[0022] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of an arbitrary video interpolation method are implemented.

[0023] The video interpolation method provided by the present application obtains a continuous first video frame and a second video frame from a video to be interpolated; determines a first subtitle area in the first video frame based on subtitle position parameters in the video to be interpolated, and determines a second subtitle area in the second video frame; determines a first similarity between the first video frame and the second video frame, and determines a second similarity between the first subtitle area and the second subtitle area; determines a target frame to be interpolated based on the first similarity and the second similarity, and inserts the target frame to be interpolated between the first video frame and the second video frame.

[0024] In this case, the subtitle area of ​​the first video frame and the second video frame can be determined based on the obtained subtitle position parameters. When inserting frames between the first video frame and the second video frame, the first similarity between the first video frame and the second video frame can be calculated, and the second similarity between the first subtitle area and the second subtitle area can be calculated. Based on the above two similarities, the corresponding target frame to be inserted is determined, and then the target frame to be inserted is inserted between the first video frame and the second video frame. In this way, the similarity of the two video frames and the similarity of the subtitle area in the two video frames are considered at the same time. Even if the overall picture and content of the two video frames are continuous, the similarity of the subtitle area is further considered, and the target frame to be inserted is comprehensively determined, which avoids the appearance of subtitle artifacts and improves the quality and effect of interpolation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1a is a schematic diagram of a video frame provided by an embodiment of the present application;

[0026] Figure 1b is a schematic diagram of another video frame provided by an embodiment of the present application;

[0027] Figure 1c is a schematic diagram of a synthetic frame provided by an embodiment of the present application;

[0028] Figure 2 is a flow chart of a video frame insertion method provided by an embodiment of the present application;

[0029] Figure 3 This is a schematic diagram of an interactive interface provided by an embodiment of the present application;

[0030] Figure 4 is a processing flow chart of a video frame insertion method applied to an offline video processing tool provided by an embodiment of the present application;

[0031] Figure 5 It is a structural schematic diagram of a video frame insertion device provided by an embodiment of the present application;

[0032] Figure 6 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0033] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0034] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0035] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0036] First, the terms involved in one or more embodiments of the present application are explained.

[0037] Video interpolation: adding several frames between every two consecutive frames in the video sequence to shorten the display time between video frames and improve the frame rate and smoothness of the video. The embodiment of the present application mainly focuses on adding one frame between two consecutive video frames, that is, double interpolation.

[0038] Optical flow: The instantaneous speed of pixel movement of a moving object in space on the observation imaging plane.

[0039] Artifacts: Unnatural, abnormal traces, areas, flaws, etc. in the synthesized frame that can be seen as artificial processing.

[0040] Image similarity: Structural similarity, referred to as SSIM, is an evaluation index that measures the similarity between two images from three aspects: brightness, contrast, and structure.

[0041] Scene switching: refers to the switching of the screen from one shot to another, with the same shot appearing as a series of continuous frames in the video frame.

[0042] It should be noted that there are currently three common methods of interpolation: the first method is to copy the previous or next frame as a synthetic frame and insert it between two consecutive video frames, that is, a copy frame; the second method is to blur the previous and next video frames similar to double exposure to obtain a synthetic frame, that is, a mixed frame; the third method is to interpolate frames based on a deep learning model, by analyzing and modeling the previous and next frames to generate optical flow, thereby obtaining a linear mapping relationship between frames, and finally combining them to form a synthetic frame.

[0043] In addition to being frequently deployed in the cloud for task calls to perform one-time fully automatic frame insertion, the video interpolation function is often embedded as a separate functional module in offline tool platforms (such as video processing tools). Compared with online calls, users can adjust function parameters according to actual conditions when using offline tools for frame insertion, so they have greater freedom.

[0044] Whether online or offline, before inserting two consecutive video frames, there is usually an essential step, which is to determine whether the two video frames to be inserted belong to the same shot or the two frames before and after the scene switch. If they are consecutive frames in the same shot, you can insert the frames directly; if not, you cannot insert the frames, because the two video frames before and after the scene switch are often discontinuous in content and picture, and there are huge differences. At this time, the synthesized frame synthesized by the interpolation model will have very serious artifacts. In order to avoid this situation, the image similarity of the two video frames can be calculated before inserting the frames (or other indicators) to determine whether the two video frames are at the scene switch moment. If so, copy one of the frames as the target frame to be inserted instead of determining the synthesized target frame to be inserted through the interpolation model.

[0045] In the above method, if there is a scene switch in a local area of ​​the video frame, for example, when the subtitles of a movie or animation are played continuously, the upper and lower subtitles are switched, but the background content is not switched, directly judging the image similarity of the two video frames will fail, because the subtitle area generally only occupies a very small part of the entire video frame, which will not have much impact on calculating the similarity of the two video frames. Therefore, even if the subtitles displayed in two consecutive video frames are different, the two video frames behind the subtitles belong to the same shot, which usually makes the two video frames meet the similarity requirements, resulting in the picture after the subtitles being able to be obtained normally through interpolation synthesis, while the subtitle area causes serious artifacts due to the large gap before and after the subtitle switching.

[0046] For example, Figure 1a is a schematic diagram of a video frame provided by an embodiment of the present application, Figure 1b is a schematic diagram of another video frame provided by an embodiment of the present application, Figure 1c It is a schematic diagram of a synthetic frame provided by an embodiment of the present application. Figure 1a and Figure 1b The following are two consecutive video frames that need to be interpolated: Figure 1a and Figure 1b As shown, the overall picture and content of the two video frames are continuous, but the lyrics are two different sentences. Figure 1a The subtitle in the film is "A pipe that is not a pipe". Figure 1b The subtitle in the video is "The pigeon on the face did not fly away". When calculating the image similarity of the entire video frame, the similarity between the two is high, which is greater than the threshold required for interpolation. The synthetic frame obtained by directly interpolating is as follows: Figure 1c As shown, Figure 1c It can be seen that obvious artifacts appear in the subtitle area, while the content behind the subtitles is the normal interpolation result.

[0047] Therefore, in an embodiment of the present application, before performing video interpolation, an interactive operation can be embedded to allow the user to select the subtitle area in the video to be interpolated. Based on the user's selection, subtitle position parameters are provided in the subsequent actual interpolation process. The subtitle area can be subsequently identified based on the subtitle position parameters, and the similarity is calculated in combination with the subtitle area. The user only needs to operate once instead of operating before each interpolation, so as to improve the interpolation effect of the subtitle area. In this way, the subtitle position parameters can be used as a priori information and input into the video processing platform. Before interpolating, the video processing platform must not only calculate the picture similarity of the entire video frame, but also calculate the similarity of the subtitle area. If one of the two conditions is not met, the operation of copying the frame will be triggered instead of performing the model interpolation operation.

[0048] In the present application, a video interpolation method is provided. The present application also relates to a video interpolation device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0049] It should be noted that when users edit videos through video processing platforms, they sometimes need to improve video smoothness through frame insertion, or slow down the video without causing video freezes, etc. These functions need to be implemented through video frame insertion methods, and improving the quality of frame insertion plays a vital role in user experience. Among them, the video processing platform can be an online video processing webpage or an offline video processing tool.

[0050] Figure 2 A flowchart of a video frame insertion method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0051] Step 202: Acquire a first video frame and a second video frame in succession from the video to be inserted.

[0052] It should be noted that the video to be inserted is a video in which an intermediate synthetic frame needs to be inserted between two consecutive video frames in the temporal sequence of the video, and the video to be inserted is a video including subtitles. The first video frame and the second video frame are two consecutive frames in the video to be inserted, and the first video frame is located before the second video frame in the temporal sequence.

[0053] In actual applications, the video processing platform can intercept the video to be inserted at a set frequency, obtain the video frames included in the video to be inserted, select two consecutive video frames, and use the earlier one in time sequence as the first video frame and the later one in time sequence as the second video frame.

[0054] For example, it is assumed that a video to be inserted is intercepted to obtain video frame 1, video frame 2, video frame 3, ..., video frame N-1, and video frame N, and video frame 1, video frame 2, video frame 3, ..., video frame N-1, and video frame N are arranged in chronological order. At this time, video frame 1 and video frame 2 are continuous, video frame 1 is the first video frame, and video frame 2 is the second video frame; video frame 2 and video frame 3 are continuous, video frame 2 is the first video frame, and video frame 3 is the second video frame; ...; video frame N-1 and video frame N are continuous, video frame N-1 is the first video frame, and video frame N is the second video frame.

[0055] In an optional implementation of this embodiment, before obtaining the continuous first video frame and the second video frame from the video to be inserted and performing the actual frame insertion process, the subtitle position parameter corresponding to the video to be inserted may be obtained first, so as to indicate the area corresponding to the subtitle in the video frame in the subsequent actual frame insertion process, that is, before obtaining the continuous first video frame and the second video frame from the video to be inserted, the method further includes:

[0056] Obtaining a video to be inserted into a frame, wherein the video to be inserted into a frame is a video with subtitles;

[0057] A frame selection operation of a user on the video to be inserted is received, and subtitle position parameters are determined based on the frame selection operation, wherein the frame selection operation is used to frame a subtitle area in the video to be inserted.

[0058] In actual applications, the user imports the video to be inserted with subtitles into the video processing platform. The video processing platform can obtain the video to be inserted imported by the user and display the video to be inserted to the user. The user can select a certain frame of the video based on the progress bar of the video to be inserted, and then frame the area where the subtitles are located in the video frame. At this time, the video processing platform can receive the user's frame selection operation on the video to be inserted, and determine the subtitle position parameters corresponding to the subtitle area framed by the user based on the frame selection operation.

[0059] Of course, in actual applications, when the video processing platform displays the video to be inserted to the user, it can display a default frame (such as the first frame), and the user can directly select the area where the subtitles are located in the default frame. This application does not impose any restrictions on this.

[0060] Among them, the subtitle position parameter can refer to the position parameter of the subtitle area selected by the user, such as the subtitle position parameter can be the vertex coordinates of the selection box selected by the selection operation, or the subtitle position parameter can also be the center point coordinates and the selection box size of the selection box selected by the selection operation.

[0061] It should be noted that when the user interactively divides a selection box containing subtitles in the video to be inserted, the video processing platform can determine the specific coordinate values ​​of the upper left corner and the lower right corner of the selection box in the video screen. The specific coordinate values ​​are the subtitle position parameters. When the video processing platform subsequently calls the insertion module to implement the specific insertion process, the subtitle position parameters will be passed to the insertion module. Similarly, when subtitles exist in multiple places in the video to be inserted, the position parameters of multiple selection boxes can be passed by selecting them multiple times.

[0062] Furthermore, when the video processing platform displays the video to be inserted to the user, it can also display prompt information to prompt the user to select the subtitle area in the video to be inserted, and the selected area can be slightly larger to avoid the situation where the subtitles in some video frames are too long to be fully included.

[0063] For example, Figure 3 is a schematic diagram of an interactive interface provided by an embodiment of the present application, such as Figure 3 As shown, the video processing platform can display the video to be inserted and the progress bar corresponding to the video to be inserted. The user selects a frame of video based on the progress bar, and then selects the subtitle area in the video through the selection box. After that, the video processing platform can determine the position parameters of the selection box as the subtitle position parameters.

[0064] In the embodiment of the present application, the interactive feature with the user is utilized, by embedding an interactive operation in the video processing platform, so that the user can frame the subtitle area in the video to be inserted, and based on the user's frame selection, subtitle position parameters are provided in the subsequent actual insertion process. The subtitle area can be identified based on the subtitle position parameters, and the similarity is calculated in combination with the subtitle area. The user only needs to operate once instead of every time before insertion, so the insertion effect of the subtitle area can be improved, and the user can freely frame, which improves processing efficiency and freedom and saves processing resources.

[0065] In an optional implementation of this embodiment, in addition to manually selecting the subtitle area by the user, the video processing platform may also automatically identify the subtitle area in the video to be inserted, that is, before obtaining the continuous first video frame and the second video frame from the video to be inserted, the process further includes:

[0066] Obtaining a video to be inserted into a frame, wherein the video to be inserted into a frame is a video with subtitles;

[0067] Determine each video frame included in the video to be inserted, and identify the subtitle area in each video frame;

[0068] A target subtitle area with the largest area range among the subtitle areas is determined, and the position parameters of the target subtitle area are used as subtitle position parameters.

[0069] In actual applications, after the video processing platform obtains the video to be inserted, it can determine the various video frames included in the video to be inserted, and identify the subtitle areas in each video frame through an image recognition algorithm, determine the target subtitle area with the largest area range among the various subtitle areas, and use the position parameters of the target subtitle area as subtitle position parameters to provide to the subsequent actual insertion process. In this way, there is no need to manually identify the subtitle area, avoiding interaction with the user. The video processing platform can automatically identify the subtitle area, thereby determining the subtitle position parameters corresponding to the video to be inserted, and passing them to the subsequent actual insertion process, with high recognition efficiency and reduced user operations.

[0070] Step 204: Based on the subtitle position parameters in the video frame to be inserted, determine a first subtitle area in the first video frame, and determine a second subtitle area in the second video frame.

[0071] It should be noted that the subtitle position parameter can be a parameter determined in advance by the video processing platform based on the user's selection operation, or it can be a parameter automatically determined by the video processing platform through an image recognition algorithm. The subtitle position parameter can indicate the position of the subtitle in the video to be inserted. Therefore, based on the subtitle position parameter in the video to be inserted, the first subtitle area in the first video frame can be determined, and the second subtitle area in the second video frame can be determined.

[0072] In practical applications, if the subtitle position parameter is the vertex coordinates of the selection box, that is, the vertex coordinates of the subtitle area, then the corresponding area image in the first video frame can be intercepted based on the vertex coordinates as the first subtitle area, and the corresponding area image in the second video frame can be intercepted based on the vertex coordinates as the second subtitle area. That is, the first subtitle area and the second subtitle area refer to the subtitle sub-images in the first video frame and the second video frame, respectively.

[0073] Step 206: Determine a first similarity between the first video frame and the second video frame, and determine a second similarity between the first subtitle region and the second subtitle region.

[0074] It should be noted that after determining the first subtitle area in the first video frame and the second subtitle area in the second video frame, the first similarity between the first video frame and the second video frame can be calculated, and the second similarity between the first subtitle area and the second subtitle area can be calculated at the same time. In other words, before each frame insertion, the similarity between the entire screen and the subtitle area of ​​the two video frames is calculated respectively, and based on the two similarities, it is determined whether the first video frame and the second video frame are similar, so as to determine whether the two video frames switch scenes, thereby determining the corresponding frame insertion strategy.

[0075] In an optional implementation of this embodiment, the first similarity between the first video frame and the second video frame is determined, and the specific implementation process may be as follows:

[0076] Determine a first brightness similarity, a first contrast similarity, and a first structure similarity between the first video frame and the second video frame respectively, and determine a first similarity between the first video frame and the second video frame based on the first brightness similarity, the first contrast similarity, and the first structure similarity;

[0077] Accordingly, determining a second similarity between the first subtitle region and the second subtitle region includes:

[0078] A second brightness similarity, a second contrast similarity, and a second structural similarity between the first subtitle region and the second subtitle region are determined respectively, and a second similarity between the first subtitle region and the second subtitle region is determined based on the second brightness similarity, the second contrast similarity, and the second structural similarity.

[0079] It should be noted that image similarity has three measurement dimensions, namely brightness, contrast, and structure, and thus the similarity between two images can be determined by combining these three dimensions.

[0080] In practical applications, the first brightness similarity, the first contrast similarity, and the first structural similarity between the first video frame and the second video frame may be determined respectively, and then the first brightness similarity, the first contrast similarity, and the first structural similarity may be combined to determine the first similarity between the first video frame and the second video frame. Similarly, the second brightness similarity, the second contrast similarity, and the second structural similarity between the first subtitle region and the second subtitle region may be determined respectively, and then the second brightness similarity, the second contrast similarity, and the second structural similarity may be combined to determine the second similarity between the first subtitle region and the second subtitle region.

[0081] In an embodiment of the present application, the three measurement dimensions of brightness, contrast, and structure can be comprehensively considered to calculate the first similarity between the first video frame and the second video frame, and the second similarity between the first subtitle area and the second subtitle area. The calculated first similarity and second similarity have high accuracy, thereby ensuring the quality of subsequent inserted frames.

[0082] In an optional implementation of this embodiment, respectively determining a first brightness similarity, a first contrast similarity, and a first structure similarity between the first video frame and the second video frame includes:

[0083] Determine a first average brightness of the first video frame based on the pixel value of each pixel point in the first video frame, and determine a second average brightness of the second video frame based on the pixel value of each pixel point in the second video frame; determine a first brightness similarity based on the first average brightness and the second average brightness;

[0084] Determine a first standard deviation of the pixel values ​​in the first video frame based on the pixel values ​​of each pixel point in the first video frame and the first average brightness, and determine a second standard deviation of the pixel values ​​in the second video frame based on the pixel values ​​of each pixel point in the second video frame and the second average brightness; determine a first contrast similarity based on the first standard deviation and the second standard deviation;

[0085] A first structural similarity is determined based on the pixel values ​​of each pixel in the first video frame and the second video frame, as well as the first average brightness, the first standard deviation, the second average brightness, and the second standard deviation.

[0086] It should be noted that the similarity in the embodiment of the present application refers to the similarity between two images, namely SSIM. SSIM is an evaluation index. Specifically, the two images x and y can be compared in three dimensions, namely brightness I(x, y), contrast c(x, y) and structure s(x, y). Finally, the image similarity of x and y is a function of these three.

[0087] In practical applications, based on the pixel values ​​of each pixel point in the first video frame, the first average brightness of the first video frame is determined, and based on the pixel values ​​of each pixel point in the second video frame, the second average brightness of the second video frame is determined. Specifically, this can be achieved by the following formulas (1) and (2):

[0088]

[0089]

[0090] Among them, μ x is the first average brightness of the first video frame; N is the number of pixels in the first video frame or the second video frame, that is, the number of pixels in the first video frame and the second video frame is the same, both N; x iIt refers to the pixel value of the i-th pixel in the first video frame; μ y is the second average brightness, y i Refers to the pixel value of the i-th pixel in the second video frame.

[0091] In addition, the first brightness similarity is determined according to the first average brightness and the second average brightness, which can be specifically implemented by the following formula (3):

[0092]

[0093] Where, l(x,y) represents the first brightness similarity; μ x Represents the first average brightness, μ y Represents the second average brightness; C1 is a constant to prevent the denominator from being zero.

[0094] Furthermore, based on the pixel values ​​of each pixel point in the first video frame and the first average brightness, a first standard deviation of the pixel values ​​in the first video frame is determined, and based on the pixel values ​​of each pixel point in the second video frame and the second average brightness, a second standard deviation of the pixel values ​​in the second video frame is determined. The specific implementation process can be implemented by the following formula (4) and formula (5):

[0095]

[0096]

[0097] Among them, σ x is the first standard deviation, N is the number of pixels in the first video frame or the second video frame, x i refers to the pixel value of the i-th pixel in the first video frame, μ x is the first average brightness of the first video frame; σ y is the second standard deviation, y i refers to the pixel value of the i-th pixel in the second video frame, μ y is the second average brightness.

[0098] The first contrast similarity is determined according to the first standard deviation and the second standard deviation. The specific implementation process can be implemented by the following formula (6):

[0099]

[0100] Among them, c(x,y) refers to the first contrast similarity, σ x is the first standard deviation, σ y is the second standard deviation, and C2 is also a constant to prevent the denominator from being zero.

[0101] In a specific implementation, the first structural similarity is determined based on the pixel values ​​of each pixel in the first video frame and the second video frame, as well as the first average brightness, the first standard deviation, the second average brightness and the second standard deviation. The specific implementation process can be implemented by the following formulas (7) and (8):

[0102]

[0103]

[0104] Where s(x,y) is the first structural similarity, N is the number of pixels in the first video frame or the second video frame, x represents the first video frame, μ x is the first average brightness of the first video frame, σ x is the first standard deviation; y represents the second video frame, μ y represents the second average brightness, σ y is the second standard deviation; C3 is also a constant to prevent the denominator from being zero.

[0105] It should be noted that after determining the first brightness similarity, the first contrast similarity and the first structure similarity, the first similarity between the first video frame and the second video frame can be determined by the following formula (9):

[0106] SSIM(x,y)=[l(x,y)] α [c(x,y)] β ·[s(x,y)] γ (9)

[0107] Among them, SSIM(x,y) represents the first similarity, l(x,y) represents the first brightness similarity, c(x,y) represents the first contrast similarity, s(x,y) represents the first structural similarity, and α, β, and γ are set constants.

[0108] In specific implementation, α=β=γ=1 can be generally set. At this time, the above formula (9) can be simplified to the following formula (10):

[0109]

[0110] It should be noted that the value range of SSIM is 0 to 1. The more similar the two images are, the closer the images are. When the two images are exactly the same, SSIM(x, y) = 1.

[0111] In addition, the second brightness similarity, the second contrast similarity, and the second structural similarity between the first subtitle area and the second subtitle area are respectively determined, and based on the second brightness similarity, the second contrast similarity, and the second structural similarity, the second similarity between the first subtitle area and the second subtitle area is determined. The specific implementation process is the same as the specific implementation process of determining the first similarity above, and this application will not repeat it here.

[0112] In the embodiment of the present application, three metrics, namely, the first brightness similarity, the first contrast similarity, and the first structure similarity, are calculated between the first video frame and the second video frame respectively. Then, the first similarity between the first video frame and the second video frame can be calculated in combination with the three metric dimensions. The second similarity between the first subtitle area and the second subtitle area can be calculated by the same method. The calculated first similarity and second similarity have high accuracy, thereby ensuring the quality of subsequent inserted frames.

[0113] Step 208: Based on the first similarity and the second similarity, determine a target frame to be inserted, and insert the target frame to be inserted between the first video frame and the second video frame.

[0114] It should be noted that after determining the first similarity between the first video frame and the second video frame, and determining the second similarity between the first subtitle area and the second subtitle area, the two similarities can be combined to determine whether the first video frame and the second video frame meet the similarity requirements, thereby determining whether the first video frame and the second video frame are similar. Based on the determination result, a corresponding strategy is adopted to determine the target frame to be inserted, thereby realizing video insertion.

[0115] In an optional implementation of this embodiment, based on the first similarity and the second similarity, the target frame to be inserted is determined, and the specific implementation process may be as follows:

[0116] When both the first similarity and the second similarity satisfy the similarity condition, a corresponding synthesized frame is generated according to the first video frame and the second video frame, and the synthesized frame is used as a target frame to be inserted;

[0117] When the first similarity or the second similarity does not satisfy the similarity condition, the first video frame or the second video frame is copied to obtain a copied video frame, and the copied video frame is used as a target frame to be inserted.

[0118] Specifically, the similarity condition may refer to a pre-set condition for indicating that two images are similar, such as the similarity condition that the first similarity is greater than a first threshold and the second similarity is greater than a second threshold. The first threshold and the second threshold may be the same or different.

[0119] In practical applications, when both the first similarity and the second similarity meet the similarity conditions, that is, the first similarity is greater than the first threshold, and the second similarity is greater than the second threshold, it means that the overall picture and structure of the first video frame and the second video frame are similar, there is no shot switching, and the first subtitle area and the second subtitle area are also similar. At this time, it can be determined that the first video frame and the second video frame are two consecutive video frames, and there is no shot switching and subtitle changes. At this time, the first video frame and the second video frame can be mixed to generate a synthetic frame, which is used as the target frame to be inserted between the first video frame and the second video frame.

[0120] When the first similarity or the second similarity does not satisfy the similarity condition, that is, the first similarity is not greater than the first threshold, or the second similarity is not greater than the second threshold, it means that the overall picture and structure of the first video frame and the second video frame are not similar, or the first subtitle area and the second subtitle area are not similar. At this time, it can be determined that there is a shot switch between the first video frame and the second video frame, or there is a subtitle change. At this time, the first video frame and the second video frame cannot be mixed, which will cause artifacts in the video picture or subtitles. Therefore, the first video frame or the second video frame can be directly copied at this time to obtain a copied video frame as a target frame to be inserted between the first video frame and the second video frame.

[0121] For example, the image similarity (SSIM) of the two previous and next video frames can be determined. t ) and the similarity of the subtitle regions selected in the two video frames (SSIM s ), assuming that both similarity thresholds are 0.6, then the image similarity SSIM of the current and next two video frames is t Greater than 0.6, and the similarity SSIM of the subtitle area s If it is greater than 0.6, it means that the overall picture and structure of the two video frames are similar, and the first subtitle area and the second subtitle area are also similar, and the model mixed interpolation is performed; otherwise, the previous frame or the next frame is copied as the interpolation result.

[0122] It should be noted that the subtitle position parameters can be used as a priori information and input into the video processing platform. Before each frame insertion, the video processing platform must not only calculate the picture similarity of the entire video frame, but also calculate the similarity of the subtitle area. If one of the two conditions is not met, the operation of copying the frame will be triggered instead of performing the model mixed interpolation operation, which avoids the appearance of subtitle artifacts and improves the interpolation quality and effect.

[0123] In an optional implementation of this embodiment, the first video frame and the second video frame may be mixed based on a pre-trained interpolation model to determine a corresponding synthetic frame, that is, the corresponding synthetic frame may be generated according to the first video frame and the second video frame. The specific implementation process may be as follows:

[0124] Determining a target frame insertion time according to the first video frame and the second video frame;

[0125] The first video frame, the second video frame and the target interpolation moment are input into the trained interpolation model to obtain a synthetic frame output by the interpolation model, wherein the synthetic frame is a synthetic frame corresponding to the target interpolation moment.

[0126] In practical applications, the first video frame and the second video frame can be first input into the trained optical flow estimation model to obtain the inter-frame optical flow map output by the optical flow estimation model, and then the optical flow intensity index between the first video frame and the second video frame is determined based on the inter-frame optical flow map, and the target interpolation time corresponding to the optical flow intensity index is determined according to the correspondence between the optical flow intensity range and the interpolation time. The optical flow estimation model can be any optical flow estimation model based on deep learning (for example: RAFT (Recurrent All Pairs Field Transforms for Optical Flow), a new optical flow deep neural architecture; FlowNet (Learning Optical Flow with Convolutional Networks, neural optical flow network), using convolutional networks to achieve optical flow prediction; ... etc.) or a traditional optical flow estimation algorithm (for example: Lucas-Kanade, a two-frame differential optical flow estimation algorithm).

[0127] In an optional implementation of this embodiment, determining an optical flow intensity index between the first video frame and the second video frame based on the inter-frame optical flow map includes:

[0128] Determine the horizontal axis component of the target pixel point in the horizontal axis direction and the vertical axis component in the vertical axis direction, wherein the target pixel point is any pixel point in the inter-frame optical flow map;

[0129] Based on the horizontal axis components of each pixel point in the inter-frame optical flow map, determine the average horizontal axis component, and based on the vertical axis components of each pixel point in the inter-frame optical flow map, determine the average vertical axis component;

[0130] An optical flow intensity index between the first video frame and the second video frame is determined based on the horizontal axis component, the vertical axis component, the average horizontal axis component, and the average vertical axis component.

[0131] In specific implementation, the optical flow intensity index between the first video frame and the second video frame can be determined by the following formula (11):

[0132] Ind=max(max 99 (F x ) / max(1.0,mean(abs(F x ))), max99 (F y ) / max(1.0,mean(abs(F y )))) (11)

[0133] Among them, Ind represents the optical flow intensity index; max is the maximum value function, min is the minimum value function, mean is the average value function, and abs is the absolute value function. x is the horizontal component of the optical flow F of the target pixel in the horizontal direction, F y is the longitudinal component of the optical flow F of the target pixel in the longitudinal direction; max 99 It represents the maximum value. The 99th percentile value of the optical flow of each pixel is taken as the maximum optical flow value. Its purpose is to eliminate the influence of abnormally large values ​​on the final result.

[0134] It can be seen from the above formula (1) that the ratio of the maximum optical flow value and the average optical flow value on the two components is calculated respectively, instead of directly using the maximum value of the optical flow as the final optical flow intensity index, which can reduce the effect of the motion amplitude in the inter-frame optical flow map. In addition, the value of mean(abs(Fx)) may be less than 1, so the lower limit of the average optical flow value is set to 1 in the above formula (1), which can prevent the optical flow intensity index result from being amplified when the average optical flow value is less than 1, and also prevent the influence of abnormal small values ​​on the result, and then take the largest ratio of the two components as the final optical flow intensity index.

[0135] It should be noted that optical flow is a vector. Generally, the inter-frame optical flow map is a two-channel (h, w, 2) image data with the same size as the original video frame. If the same object in two video frames is only the size of a pixel during movement, then the distance and direction of movement of the object between the two video frames is the optical flow of the object or the pixel. The horizontal movement distance and direction are the horizontal component of the optical flow, and the vertical movement distance and direction are the vertical component of the optical flow. The two vectors constitute the final optical flow vector.

[0136] In practical applications, the smaller the optical flow intensity index, the smaller the movement, and the middle moment can be selected as the target interpolation moment; the larger the optical flow intensity index, the greater the movement, and the frame should be moved as close to the first video frame as possible or as close to the second video frame as possible. Therefore, in the correspondence between the optical flow intensity range and the interpolation moment, one optical flow intensity range may correspond to two interpolation moments, one interpolation moment is shifted forward to be close to the first video frame, and the other interpolation moment is shifted backward to be close to the second video frame, and the values ​​of the forward shift and the backward shift are symmetrical. When determining the target interpolation moment corresponding to the optical flow intensity index based on the correspondence between the optical flow intensity range and the interpolation moment, the interpolation moment that is shifted forward or backward can be arbitrarily selected, but for a video to be interpolated, the selection rules should be consistent, that is, for a video to be interpolated, when the optical flow intensity index is large, each interpolation is shifted forward or backward.

[0137] As an example, the correspondence table between the preset optical flow intensity range and the interpolation time is shown in Table 1 below. Assuming that the optical flow intensity index between the first video frame and the second video frame is determined to be 30 based on the inter-frame optical flow graph, and assuming that when the optical flow intensity index is large, the interpolation time is shifted to the left, based on Table 1 below, the corresponding target interpolation time t=0.25 can be determined.

[0138] Table 1 Correspondence between optical flow intensity range and interpolation time

[0139] Optical flow intensity range Insertion moment Less than or equal to 15 0.5 Greater than 15 and less than or equal to 20 0.4 (or 0.6) Greater than 20 and less than or equal to 25 0.3 (or 0.7) Greater than 25 0.25 (or 0.75)

[0140] In addition, after determining the target interpolation time, the first video frame, the second video frame and the target interpolation time can be input into the trained interpolation model to obtain a synthetic frame output by the interpolation model, wherein the synthetic frame is a synthetic frame corresponding to the target interpolation time.

[0141] Among them, the interpolation model can be any model based on deep learning and supports interpolation at any time (for example: RIFE (Real-Time Intermediate Flow Estimation for Video Frame Interpolation, a real-time intermediate flow estimation algorithm), IFRNet (Intermediate Feature Refine Network for Efficient Frame Interpolation, a video interpolation network that only contains a codec structure), etc.). This interpolation model is used because of its fast inference speed and the ability to interpolate at any time position between two video frames.

[0142] It should be noted that the first video frame, the second video frame and the interpolation time information can be input into the trained interpolation model to obtain the target synthetic frame output by the interpolation model for the target interpolation time. The interpolation model based on optical flow can interpolate at any time position between two frames, and the closer the time is to any input frame, the fewer the interpolation artifacts. With the help of an additional optical flow estimation model, the size of the optical flow between frames is determined to determine the interpolation time position, thereby avoiding the problem of serious artifacts that may occur when the model directly interpolates at the intermediate moment, and greatly improving the sensory effect of the final interpolation result.

[0143] In an optional implementation of this embodiment, the first video frame, the second video frame and the interpolation time information are input into the trained interpolation model to obtain the target synthetic frame output by the interpolation model. The specific implementation process may be as follows:

[0144] Inputting the first video frame, the second video frame and the interpolation time information into the optical flow analysis layer of the interpolation model, and determining the first optical flow from the first timestamp to the target interpolation time, and the second optical flow from the target interpolation time to the second timestamp through the optical flow analysis layer, wherein the first timestamp is the timestamp of the first video frame, and the second timestamp is the timestamp of the second video frame;

[0145] Sampling from the first video frame based on the first optical flow through the sampling layer of the interpolation model to obtain a first sampling result, and sampling from the second video frame based on the second optical flow to obtain a second sampling result;

[0146] The first sampling result and the second sampling result are fused based on the set fusion weight through the fusion layer of the interpolation model to obtain and output the target synthetic frame.

[0147] In practical applications, two consecutive video frames and the target interpolation time that needs to be interpolated are input into the interpolation model. The optical flow analysis layer of the interpolation model will first generate a first optical flow from the first timestamp to the target interpolation time and a second optical flow from the target interpolation time to the second timestamp, and then sample from the two input video frames respectively through the layer mapping operation (warp operation) to obtain the first sampling result and the second sampling result. The fusion layer of the interpolation model fuses the two sampling results according to the fusion weights generated by the interpolation model to generate the target synthetic frame corresponding to the final target interpolation time.

[0148] It should be noted that the movement direction and size of a certain pixel point in the first video frame between the two video frames can be known from the obtained optical flow, and then the corresponding pixel point can be mapped back to the corresponding coordinate position according to the two obtained optical flows through a mapping operation (warp operation), so as to realize sampling of the two input video frames. In addition, setting the fusion weight is an intermediate output result of the interpolation model, and the interpolation model can calculate the weight of each pixel point of the two video frames before and after by itself. For example, pixel point a of the first video frame and pixel point b of the second video frame are both mapped to a certain position C of the middle synthetic frame, but only one pixel point can be placed in one position. At this time, it is necessary to weigh the proportion of the two pixel points a and b in C. At this time, the above-mentioned setting of fusion weights can be used to balance.

[0149] In the embodiment of the present application, the optical flow between the two frames can be obtained with the help of an additional optical flow estimation model, and as a priori information, the interpolation time of the interpolation model can be adaptively guided to move forward or backward. Moving forward, the target synthetic frame will be closer to the first video frame. At this time, the optical flow from the first timestamp to the target interpolation time will be relatively small. Although the optical flow from the target interpolation time to the second timestamp will be large, even if the optical flow estimation from the target interpolation time to the second timestamp is inaccurate, the fusion weight of the optical flow from the target interpolation time to the second timestamp will be very small during the fusion process due to the interpolation model mechanism, so it will not have a substantial impact on the final result. The final smaller fusion weight will reduce the impact of the optical flow from the target interpolation time to the second timestamp on the result, ensuring that it will not cause a situation that is not similar to the previous and next two frames, but closer to the first video frame; similarly, if the target synthetic frame time is moved backward, the target synthetic frame can be made more similar to the second video frame. In this way, the characteristics of interpolation at any time and the closer the interpolation is to any input frame, the fewer interpolation artifacts there are, which greatly improves the problem of artifacts in the interpolation of the intermediate moment caused by large motion.

[0150] In an optional implementation of this embodiment, after inserting the target frame to be inserted between the first video frame and the second video frame, the method further includes:

[0151] Determine whether the interpolation end condition is currently met;

[0152] If the interpolation end condition is met, the video after the target interpolation frame is inserted is used as the obtained interpolation video;

[0153] If the frame insertion end condition is not met, the operation step of obtaining a continuous first video frame and a second video frame from the video to be inserted is continued.

[0154] Specifically, the interpolation end condition is a pre-set condition that must be met for the video to be interpolated to complete interpolation. For example, the interpolation end condition can be that the target synthetic frame has been inserted between any two consecutive video frames in the video to be interpolated; or, the interpolation end condition can also be that the target synthetic frame has been inserted between any two consecutive video frames specified in the video to be interpolated.

[0155] In practical applications, after inserting the target frame to be inserted between the first video frame and the second video frame, it can be determined whether the frame insertion end condition is currently met; if the frame insertion end condition is met, it means that the frame insertion is completed, and the video after the target frame to be inserted can be used as the obtained frame insertion video; if the frame insertion end condition is not met, the operation step of determining the continuous first video frame and the second video frame from the frame to be inserted is continued, and the target frame to be inserted is continued to be inserted at the target frame insertion time between the two video frames until the frame insertion end condition is met.

[0156] It should be noted that the video interpolation method provided in the embodiment of the present application can be used as a plug-and-play extension module and can be used in any interpolation model that supports interpolation at any time.

[0157] The video interpolation method provided by the present application can determine the subtitle area of ​​the first video frame and the second video frame based on the acquired subtitle position parameters. When interpolating between the first video frame and the second video frame, the first similarity between the first video frame and the second video frame can be calculated, and the second similarity between the first subtitle area and the second subtitle area can be calculated. Based on the above two similarities, the corresponding target frame to be interpolated is determined, and then the target frame to be interpolated is inserted between the first video frame and the second video frame. In this way, the similarity of the two video frames and the similarity of the subtitle area in the two video frames are considered at the same time. Even if the overall picture and content of the two video frames are continuous, the similarity of the subtitle area is further considered, and the target frame to be interpolated is comprehensively determined, which avoids the appearance of subtitle artifacts and improves the interpolation quality and interpolation effect.

[0158] The following combination Figure 4 , taking the application of the video interpolation method provided by the present application in an offline video processing tool as an example, the video interpolation method is further described. Figure 4 A processing flow chart of a video frame insertion method applied to an offline video processing tool provided by an embodiment of the present application is shown, which specifically includes the following steps:

[0159] Step 402: The user imports a video with subtitles into a video processing tool, and the user roughly selects a subtitle area in the video, and the video processing tool determines subtitle position parameters of the subtitle area selected by the user.

[0160] Step 404: the video processing tool obtains a continuous first video frame and a second video frame from the video to be inserted, and determines a first subtitle area in the first video frame and a second subtitle area in the second video frame based on the subtitle position parameter.

[0161] Step 406: Determine a first similarity between the first video frame and the second video frame, and determine a second similarity between the first subtitle region and the second subtitle region.

[0162] Step 408: When both the first similarity and the second similarity satisfy the similarity condition, a corresponding synthetic frame is generated according to the first video frame and the second video frame through the frame insertion model, and the synthetic frame is used as a target frame to be inserted.

[0163] Step 410: when the first similarity or the second similarity does not satisfy the similarity condition, copy the first video frame or the second video frame to obtain a copied video frame, and use the copied video frame as a target frame to be inserted.

[0164] Step 412: insert the target frame to be inserted between the first video frame and the second video frame, and the video processing platform executes step 404 in a loop until the frame insertion is completed.

[0165] The video interpolation method provided by the present application can determine the subtitle area of ​​the first video frame and the second video frame based on the acquired subtitle position parameters. When interpolating between the first video frame and the second video frame, the first similarity between the first video frame and the second video frame can be calculated, and the second similarity between the first subtitle area and the second subtitle area can be calculated. Based on the above two similarities, the corresponding target frame to be interpolated is determined, and then the target frame to be interpolated is inserted between the first video frame and the second video frame. In this way, the similarity of the two video frames and the similarity of the subtitle area in the two video frames are considered at the same time. Even if the overall picture and content of the two video frames are continuous, the similarity of the subtitle area is further considered, and the target frame to be interpolated is comprehensively determined, which avoids the appearance of subtitle artifacts and improves the interpolation quality and interpolation effect.

[0166] Corresponding to the above method embodiment, the present application also provides a video frame insertion device embodiment, Figure 5 FIG. 1 is a schematic diagram showing the structure of a video frame insertion device provided by an embodiment of the present application. Figure 5 As shown, the device comprises:

[0167] An acquisition module 502 is configured to acquire a continuous first video frame and a second video frame from the video to be inserted;

[0168] A first determination module 504 is configured to determine a first subtitle region in a first video frame and a second subtitle region in a second video frame based on subtitle position parameters in the video frame to be inserted;

[0169] A second determination module 506 is configured to determine a first similarity between the first video frame and the second video frame, and to determine a second similarity between the first subtitle region and the second subtitle region;

[0170] The third determination module 508 is configured to determine a target frame to be inserted based on the first similarity and the second similarity, and insert the target frame to be inserted between the first video frame and the second video frame.

[0171] Optionally, the device further includes a fourth determining module, configured to:

[0172] Obtaining a video to be inserted into a frame, wherein the video to be inserted into a frame is a video with subtitles;

[0173] A frame selection operation of a user on the video to be inserted is received, and subtitle position parameters are determined based on the frame selection operation, wherein the frame selection operation is used to frame a subtitle area in the video to be inserted.

[0174] Optionally, the device further includes a fifth determining module, configured to:

[0175] Obtaining a video to be inserted into a frame, wherein the video to be inserted into a frame is a video with subtitles;

[0176] Determine each video frame included in the video to be inserted, and identify the subtitle area in each video frame;

[0177] A target subtitle area with the largest area range among the subtitle areas is determined, and the position parameters of the target subtitle area are used as subtitle position parameters.

[0178] Optionally, the third determining module 508 is further configured to:

[0179] When both the first similarity and the second similarity satisfy the similarity condition, a corresponding synthesized frame is generated according to the first video frame and the second video frame, and the synthesized frame is used as a target frame to be inserted;

[0180] When the first similarity or the second similarity does not satisfy the similarity condition, the first video frame or the second video frame is copied to obtain a copied video frame, and the copied video frame is used as a target frame to be inserted.

[0181] Optionally, the third determining module 508 is further configured to:

[0182] Determining a target frame insertion time according to the first video frame and the second video frame;

[0183] The first video frame, the second video frame and the target interpolation moment are input into the trained interpolation model to obtain a synthetic frame output by the interpolation model, wherein the synthetic frame is a synthetic frame corresponding to the target interpolation moment.

[0184] Optionally, the second determining module 506 is further configured to:

[0185] Determine a first brightness similarity, a first contrast similarity, and a first structure similarity between the first video frame and the second video frame respectively, and determine a first similarity between the first video frame and the second video frame based on the first brightness similarity, the first contrast similarity, and the first structure similarity;

[0186] A second brightness similarity, a second contrast similarity, and a second structural similarity between the first subtitle region and the second subtitle region are determined respectively, and a second similarity between the first subtitle region and the second subtitle region is determined based on the second brightness similarity, the second contrast similarity, and the second structural similarity.

[0187] Optionally, the second determining module 506 is further configured to:

[0188] Determine a first average brightness of the first video frame based on the pixel value of each pixel point in the first video frame, and determine a second average brightness of the second video frame based on the pixel value of each pixel point in the second video frame; determine a first brightness similarity based on the first average brightness and the second average brightness;

[0189] Determine a first standard deviation of the pixel values ​​in the first video frame based on the pixel values ​​of each pixel point in the first video frame and the first average brightness, and determine a second standard deviation of the pixel values ​​in the second video frame based on the pixel values ​​of each pixel point in the second video frame and the second average brightness; determine a first contrast similarity based on the first standard deviation and the second standard deviation;

[0190] A first structural similarity is determined based on the pixel values ​​of each pixel in the first video frame and the second video frame, as well as the first average brightness, the first standard deviation, the second average brightness, and the second standard deviation.

[0191] Optionally, the device further includes a sixth determining module, configured to:

[0192] Determine whether the interpolation end condition is currently met;

[0193] If the interpolation end condition is met, the video after the target interpolation frame is inserted is used as the obtained interpolation video;

[0194] If the frame insertion end condition is not met, the process continues to return to the execution of the acquisition module 502 .

[0195] The video interpolation device provided by the present application can determine the subtitle area of ​​the first video frame and the second video frame based on the acquired subtitle position parameters. When interpolating between the first video frame and the second video frame, the first similarity between the first video frame and the second video frame can be calculated, and the second similarity between the first subtitle area and the second subtitle area can be calculated. Based on the above two similarities, the corresponding target frame to be interpolated is determined, and then the target frame to be interpolated is inserted between the first video frame and the second video frame. In this way, the similarity of the two video frames and the similarity of the subtitle area in the two video frames are considered at the same time. Even if the overall picture and content of the two video frames are continuous, the similarity of the subtitle area is further considered, and the target frame to be interpolated is comprehensively determined, which avoids the appearance of subtitle artifacts and improves the interpolation quality and interpolation effect.

[0196] The above is a schematic scheme of a video interpolation device of this embodiment. It should be noted that the technical scheme of the video interpolation device and the technical scheme of the above-mentioned video interpolation method belong to the same concept, and the details not described in detail in the technical scheme of the video interpolation device can be referred to the description of the technical scheme of the above-mentioned video interpolation method.

[0197] Figure 6 The block diagram of a computing device according to an embodiment of the present application is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0198] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0199] In one embodiment of the present application, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0200] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 600 may also be a mobile or stationary server.

[0201] The processor 620 is used to execute the following computer executable instructions to implement the following method:

[0202] Acquire a first video frame and a second video frame continuously from the video to be inserted;

[0203] Based on the subtitle position parameters in the video frame to be inserted, determining a first subtitle area in the first video frame, and determining a second subtitle area in the second video frame;

[0204] determining a first similarity between the first video frame and the second video frame, and determining a second similarity between the first subtitle region and the second subtitle region;

[0205] Based on the first similarity and the second similarity, a target frame to be inserted is determined, and the target frame to be inserted is inserted between the first video frame and the second video frame.

[0206] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned video frame insertion method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned video frame insertion method.

[0207] An embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the steps of any video frame insertion method when executed by a processor.

[0208] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned video frame insertion method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned video frame insertion method.

[0209] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0210] Computer instructions include computer program codes, which may be in source code form, object code form, executable files, or some intermediate form, etc. Computer readable media may include: any entity or device capable of carrying computer program codes, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memories, read-only memories (ROM), random access memories (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0211] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0212] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0213] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. A video frame insertion method, characterized in that: include: Acquire a first video frame and a second video frame continuously from the video to be inserted; Based on the subtitle position parameters in the video frame to be inserted, determining a first subtitle area in the first video frame, and determining a second subtitle area in the second video frame; determining a first similarity between the first video frame and the second video frame, and determining a second similarity between the first subtitle region and the second subtitle region; Determine a target frame to be inserted based on the first similarity and the second similarity, and insert the target frame to be inserted between the first video frame and the second video frame; The determining of the first similarity between the first video frame and the second video frame includes: respectively determining a first brightness similarity, a first contrast similarity, and a first structure similarity between the first video frame and the second video frame, and determining a first similarity between the first video frame and the second video frame based on the first brightness similarity, the first contrast similarity, and the first structure similarity; Accordingly, the determining the second similarity between the first subtitle region and the second subtitle region includes: A second brightness similarity, a second contrast similarity, and a second structural similarity between the first subtitle region and the second subtitle region are respectively determined, and a second similarity between the first subtitle region and the second subtitle region is determined based on the second brightness similarity, the second contrast similarity, and the second structural similarity.

2. The video frame insertion method according to claim 1, characterized in that: Before acquiring the continuous first video frame and the second video frame from the video to be inserted, the method further includes: Obtaining a video to be inserted into a frame, wherein the video to be inserted into a frame is a video with subtitles; A frame selection operation of a user on the video to be inserted is received, and the subtitle position parameter is determined based on the frame selection operation, wherein the frame selection operation is used to frame a subtitle area in the video to be inserted.

3. The video frame insertion method according to claim 1, characterized in that: Before acquiring the continuous first video frame and the second video frame from the video to be inserted, the method further includes: Determine each video frame included in the video to be inserted, and identify the subtitle area in each video frame; A target subtitle area with the largest area range among the subtitle areas is determined, and a position parameter of the target subtitle area is used as the subtitle position parameter.

4. The video frame insertion method according to claim 1, characterized in that: The determining a target frame to be inserted based on the first similarity and the second similarity includes: In the case where both the first similarity and the second similarity satisfy the similarity condition, generating a corresponding synthesized frame according to the first video frame and the second video frame, and using the synthesized frame as the target frame to be inserted; When the first similarity or the second similarity does not satisfy the similarity condition, the first video frame or the second video frame is copied to obtain a copied video frame, and the copied video frame is used as the target frame to be inserted.

5. The video frame insertion method according to claim 4, characterized in that: The step of generating a corresponding synthesized frame according to the first video frame and the second video frame includes: Determining a target frame insertion time according to the first video frame and the second video frame; The first video frame, the second video frame and the target interpolation moment are input into a trained interpolation model to obtain a synthetic frame output by the interpolation model, wherein the synthetic frame is a synthetic frame corresponding to the target interpolation moment.

6. The video frame insertion method according to claim 1, characterized in that: The determining respectively a first brightness similarity, a first contrast similarity, and a first structure similarity between the first video frame and the second video frame comprises: Determine a first average brightness of the first video frame based on the pixel value of each pixel point in the first video frame, and determine a second average brightness of the second video frame based on the pixel value of each pixel point in the second video frame; determine the first brightness similarity according to the first average brightness and the second average brightness; Determine a first standard deviation of pixel values ​​in the first video frame based on pixel values ​​of each pixel point in the first video frame and the first average brightness, and determine a second standard deviation of pixel values ​​in the second video frame based on pixel values ​​of each pixel point in the second video frame and the second average brightness; determine the first contrast similarity based on the first standard deviation and the second standard deviation; The first structural similarity is determined based on the pixel values ​​of each pixel in the first video frame and the second video frame, as well as the first average brightness, the first standard deviation, the second average brightness, and the second standard deviation.

7. The video frame insertion method according to any one of claims 1 to 5, characterized in that: After inserting the target frame to be inserted between the first video frame and the second video frame, the method further includes: Determine whether the interpolation end condition is currently met; If the frame insertion end condition is met, the video after the target frame to be inserted is used as the obtained frame insertion video; If the frame insertion end condition is not met, the operation step of acquiring a continuous first video frame and a second video frame from the video to be inserted is continued.

8. A video frame insertion device, characterized in that: include: An acquisition module is configured to acquire a continuous first video frame and a second video frame from the video to be inserted; A first determination module is configured to determine a first subtitle area in the first video frame and a second subtitle area in the second video frame based on subtitle position parameters in the video frame to be inserted; a second determination module configured to determine a first similarity between the first video frame and the second video frame, and to determine a second similarity between the first subtitle region and the second subtitle region; A third determining module is configured to determine a target frame to be inserted based on the first similarity and the second similarity, and insert the target frame to be inserted between the first video frame and the second video frame; The determining of the first similarity between the first video frame and the second video frame includes: respectively determining a first brightness similarity, a first contrast similarity, and a first structure similarity between the first video frame and the second video frame, and determining a first similarity between the first video frame and the second video frame based on the first brightness similarity, the first contrast similarity, and the first structure similarity; Accordingly, the determining the second similarity between the first subtitle region and the second subtitle region includes: A second brightness similarity, a second contrast similarity, and a second structural similarity between the first subtitle region and the second subtitle region are respectively determined, and a second similarity between the first subtitle region and the second subtitle region is determined based on the second brightness similarity, the second contrast similarity, and the second structural similarity.

9. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the following method: Acquire a first video frame and a second video frame continuously from the video to be inserted; Based on the subtitle position parameters in the video frame to be inserted, determining a first subtitle area in the first video frame, and determining a second subtitle area in the second video frame; determining a first similarity between the first video frame and the second video frame, and determining a second similarity between the first subtitle region and the second subtitle region; Determine a target frame to be inserted based on the first similarity and the second similarity, and insert the target frame to be inserted between the first video frame and the second video frame; The determining of the first similarity between the first video frame and the second video frame includes: respectively determining a first brightness similarity, a first contrast similarity, and a first structure similarity between the first video frame and the second video frame, and determining a first similarity between the first video frame and the second video frame based on the first brightness similarity, the first contrast similarity, and the first structure similarity; Accordingly, the determining the second similarity between the first subtitle region and the second subtitle region includes: A second brightness similarity, a second contrast similarity, and a second structural similarity between the first subtitle region and the second subtitle region are respectively determined, and a second similarity between the first subtitle region and the second subtitle region is determined based on the second brightness similarity, the second contrast similarity, and the second structural similarity.

10. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the video frame insertion method described in any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images