Video frame insertion method and device, computer equipment and readable medium

Abstract

The invention provides a video frame insertion method, which is applied to a video decoding end and comprises the following steps: decoding a code stream to obtain a current frame image, and generating an intermediate frame image according to the current frame image when frame insertion is determined to be performed on the current frame image, the intermediate frame image being used for frame insertion before or after the current frame image; frame insertion and code stream analysis are carried out at the video decoding end at the same time, so that the efficiency can be saved, and the purpose of time division multiplexing is achieved; the embodiment of the invention can be applied to an AV1 video transmission scene, and the decoding efficiency and the display efficiency are not influenced while the video fluency is increased. The invention further provides a video frame insertion device, computer equipment and a readable medium.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a video frame insertion method, device, computer equipment and readable medium. Background Art

[0002] As people's demand for high-quality videos increases, video interpolation technology has been derived to increase the frame rate. This method inserts an intermediate frame between the previous and next frames to improve the smoothness of the video and the user's viewing experience.

[0003] In traditional video interpolation schemes, the middle frame is usually determined by using the two frames before and after. Currently, for HEVC (High Efficiency Video Coding) video interpolation, the arrangement of coded frames is IBPB, and frames are inserted between I frames and B frames. This scheme is for decoded frames with fixed arrangement, and this method is not applicable to AV1 videos, so it has limitations. Summary of the invention

[0004] The present disclosure provides a video frame insertion method, apparatus, computer equipment and readable medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a video frame insertion method, which is applied to a video decoding end and includes:

[0006] Decode the code stream to obtain the current frame image;

[0007] In the case where it is determined to perform frame insertion on the current frame image, an intermediate frame image is generated according to the current frame image, and the intermediate frame image is used for frame insertion before or after the current frame image.

[0008] On the other hand, the embodiments of the present disclosure further provide a computer device, comprising: one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by the one or more processors, the one or more processors implement the video interpolation method as described above.

[0009] In another aspect, the present disclosure further provides a video frame insertion device, including:

[0010] A decoding module is used to decode the code stream to obtain the current frame image;

[0011] A feature judgment module, used for obtaining a first feature value of the current frame image, and determining whether to perform interpolation on the current frame image according to the first feature value;

[0012] The phase interpolation module is used to generate an intermediate frame image according to the current frame image when it is determined to interpolate the current frame image, and the intermediate frame image is used to interpolate before or after the current frame image.

[0013] On the other hand, the embodiments of the present disclosure further provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed, implements the video frame insertion method as described above.

[0014] The interpolation processing method provided by the embodiment of the present disclosure is applied to the video decoding end, including: decoding the code stream to obtain the current frame image, and when it is determined to interpolate the current frame image, generating an intermediate frame image according to the current frame image, and the intermediate frame image is used to interpolate before or after the current frame image; the video decoding end performs interpolation and code stream parsing at the same time, which can save efficiency and achieve the purpose of time division multiplexing; the embodiment of the present disclosure can be applied to AV1 video transmission scenarios, while increasing video smoothness, without affecting decoding efficiency and display efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of a video frame insertion method flow chart provided by an embodiment of the present disclosure;

[0016] Figure 2 A schematic diagram of a process for generating an intermediate frame image provided by an embodiment of the present disclosure Figure 1 ;

[0017] Figure 3 A schematic diagram of a process for generating an intermediate frame image provided by an embodiment of the present disclosure Figure 2 ;

[0018] Figure 4 A schematic diagram of phase interpolation provided in an embodiment of the present disclosure;

[0019] Figure 5 A schematic diagram of a process for determining the phase of each level of the third image pyramid provided by an embodiment of the present disclosure;

[0020] Figure 6 A schematic diagram of a process for correcting and adjusting a phase difference provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.

[0022] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0023] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded.

[0024] The embodiments described herein may be described with reference to plan views and / or cross-sectional views by means of idealized schematic diagrams of the present disclosure. Therefore, the example illustrations may be modified according to manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to the embodiments shown in the accompanying drawings, but include modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the accompanying drawings have schematic properties, and the shapes of the regions shown in the figures illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.

[0025] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0026] The embodiment of the present disclosure provides a video frame insertion method, which is applied to a video decoding end. The video frame insertion method of the embodiment of the present disclosure can be applied to AV1 video transmission scenarios. The following description is made by taking AV1 video stream video frame insertion as an example. Figure 1 As shown, the video frame insertion method includes the following steps:

[0027] Step S1, decoding the code stream to obtain the current frame image.

[0028] The AV1 decoding module at the video decoding end can be used to decode the bitstream and parse and output the current frame image.

[0029] Step S2: when it is determined to insert a frame into the current frame image, an intermediate frame image is generated according to the current frame image, and the intermediate frame image is used to insert a frame before or after the current frame image.

[0030] The frame feature module at the video decoding end extracts a first feature (such as a frame type) from the current frame image according to the AV1 decoding syntax element, and determines whether to perform frame interpolation on the current frame image by judging the first feature. In some embodiments, the frame type (FRAME_TYPE) of the current frame image is determined, and if the frame type is not a preset frame type, it is determined to perform frame interpolation on the current frame image. In some embodiments, the preset frame type includes one of the following: a key frame (KEY_FRAME), an intra-frame (INTRA_FRAME), and a switch frame (SWITCH_FRAME). That is, if the frame type of the current frame image is not a key frame, an intra-frame, or a switch frame, an interpolation operation is required; if the frame type of the current frame image is one of a key frame, an intra-frame, or a switch frame, an interpolation operation is not required.

[0031] If interpolation is required, an intermediate frame image is generated based on the current frame image. The intermediate frame image is the frame image to be inserted and can be located before or after the current frame image. After the entire code stream is decoded to obtain a decoded image, an interpolation operation is performed based on the intermediate frame image, the current frame image and the decoded image to generate a video stream.

[0032] The video interpolation method provided by the embodiment of the present disclosure is applied to the video decoding end, including: decoding the code stream to obtain the current frame image, and when it is determined to interpolate the current frame image, generating an intermediate frame image according to the current frame image, and the intermediate frame image is used to interpolate before or after the current frame image; the video decoding end performs interpolation and code stream parsing at the same time, which can save efficiency and achieve the purpose of time division multiplexing; the embodiment of the present disclosure can be applied to AV1 video transmission scenarios, while increasing video smoothness, it does not affect decoding efficiency and display efficiency.

[0033] In the embodiments of the present disclosure, Figure 2 As shown, the step of generating an intermediate frame image according to the current frame image (i.e., step S2) includes the following steps:

[0034] Step S21, determining each reference frame image of the current frame image, and determining a target reference frame image according to each reference frame image.

[0035] In some embodiments, determining the target reference frame image according to each reference frame image includes: calculating the similarity between each reference frame image and the current frame image, and selecting the reference frame image with the highest similarity as the target reference frame image. The peak signal-to-noise ratio (PSNR) of each reference frame image can be calculated, and the reference frame image corresponding to the maximum PSNR is determined as the target reference frame image.

[0036] It should be noted that after determining the target reference frame, the second feature of the target reference frame can also be extracted, and the interpolation mode can be determined according to the second feature of the target reference frame. Exemplarily, the second feature may include the relative position of the frame, for example, it may include LAST_FRAME (the nearest reference frame), ALTREF_FRAME (the farthest reference frame), etc. The interpolation mode may include forward interpolation or backward interpolation. Forward interpolation means that the intermediate frame image is interpolated before the current frame image, and backward interpolation means that the intermediate frame image is interpolated after the current frame image.

[0037] Step S22, generating an intermediate frame image according to the target reference frame image and the current frame image.

[0038] Figure 3 A schematic diagram of a process for generating an intermediate frame image provided by an embodiment of the present disclosure, Figure 4 A schematic diagram of a phase interpolation frame provided in an embodiment of the present disclosure. Figure 3 and Figure 4 As shown, the step of generating an intermediate frame image (i.e., step S22) according to the target reference frame image and the current frame image includes the following steps:

[0039] Step S31 : establishing a first image pyramid of a current frame image, and establishing a second image pyramid of a target reference frame image.

[0040] The target reference frame image (I k ) and the current frame image (I m ) uses a steerable filter (Ψ(ω,θ)) to obtain an image pyramid of different frequencies and directions. The image pyramid can be represented by the amplitude and phase of each level, where different frequencies ω correspond to different image pyramid levels. The current frame image (I m ) corresponds to the first image pyramid, the target reference frame image (I k ) corresponds to the second image pyramid. The mathematical expression of the first image pyramid is as shown in formula (1), and the mathematical expression of the second image pyramid is as shown in formula (2):

[0041]

[0042]

[0043] in, is the cosine part, indicating an even symmetric filter response; is the sinusoidal part, representing the odd-symmetric filter response; ω represents the frequency, and θ represents the direction; Indicates the current frame image (I m ), Represents the target reference frame image (I k) is a second image pyramid.

[0044] Step S32, establishing a third image pyramid of the intermediate frame image according to the first image pyramid and the second image pyramid; wherein the phase of each level of the third image pyramid is determined according to the phase difference between the levels of the first image pyramid and the second image pyramid, and the amplitude of each level of the third image pyramid is determined according to the first image pyramid and the second image pyramid.

[0045] Since the phase method can well resolve the boundary of object motion by extracting motion information from coarse to fine, and restores the edge details of the object well, in the embodiment of the present disclosure, the phase method is combined with the image pyramid to determine the intermediate frame image.

[0046] The amplitude of each level of the pyramid of the third image can be obtained by superimposing the amplitudes of the first image pyramid and the same level of the second image pyramid.

[0047] Step S33: generating an intermediate frame image according to the amplitude and phase of each level of the third image pyramid.

[0048] The intermediate frame image is obtained by synthesizing the amplitude and phase of each level of the third image pyramid.

[0049] In some embodiments, Figure 5 As shown, the step of determining the phase of each level of the third image pyramid according to the phase difference of the levels between the first image pyramid and the second image pyramid (i.e., step S32) includes the following steps:

[0050] Step S321 : determining a first amplitude and a first phase of each level of the first image pyramid, and a second amplitude and a second phase of each level of the second pyramid.

[0051] According to the above formulas (1) and (2), the current frame image (I m ) of the first image pyramid at different levels and Phase As shown in the following formulas (3) and (4):

[0052]

[0053]

[0054] According to the above formulas (1) and (2), the target reference frame (I k ) is the amplitude of the second image pyramid and Phase As shown in the following formulas (5) and (6):

[0055]

[0056]

[0057] Step S322 , calculating the phase difference of each level between the first image pyramid and the second image pyramid.

[0058] Based on the motion assumption of minimum phase, the phase difference required for interpolation between two frames is calculated. In some embodiments, the phase difference of each level between the first image pyramid and the second image pyramid can be calculated according to the following formula (7):

[0059] in, represents the phase difference at level l between the first image pyramid and the second image pyramid, is the phase of the first image pyramid at level l, is the phase of the lth level of the second image pyramid.

[0060] According to the above formula (7), the phase difference of each level between the first image pyramid and the second image pyramid can be calculated respectively.

[0061] Step S323 , calculating the third phase of each level of the third image pyramid according to the phase difference of each level between the first image pyramid and the second image pyramid, the interpolation factor, and the second phase of each level of the second image pyramid.

[0062] In some embodiments, the target reference frame image (I k ) and the expected output sequence of the current frame image (I m ) is determined by the expected output order of the target reference frame image (I k ) and the expected output sequence of the current frame image (I m ) is determined by the expected output sequence of the target reference frame image (I k ) and the current frame image (I m ), and determine the interpolation factor (α) according to the distance.

[0063] In some embodiments, the third phase of each level of the third image pyramid can be obtained by correcting the second phase of each level of the second image pyramid using the product of the interpolation factor (α) and the phase difference of each level between the first image pyramid and the second image pyramid.

[0064] In some embodiments, determining the amplitude of each level of the third image pyramid according to the first image pyramid and the second image pyramid includes: for each level of the first image pyramid and the second image pyramid, superimposing the first amplitude of the level of the first image pyramid and the second amplitude of the corresponding level of the second pyramid to obtain the third amplitude of the level of the third image pyramid. Each level is amplitude superimposed in the above manner, thereby obtaining the third amplitude of each level of the third image pyramid.

[0065] In some embodiments, the step of calculating the third phase of each level of the third image pyramid according to the phase difference of each level between the first image pyramid and the second image pyramid, the interpolation factor, and the second phase of each level of the second image pyramid (i.e., step S323) includes the following steps: when the phase difference of each level between the first image pyramid and the second image pyramid meets a preset condition, the third phase of each level of the third image pyramid is calculated according to the phase difference of each level between the first image pyramid and the second image pyramid, the interpolation factor, and the second phase of each level of the second image pyramid. That is, if the phase difference of each level between the first image pyramid and the second image pyramid meets the preset condition, the phase difference of each level between the first image pyramid and the second image pyramid calculated in step S322 may not be corrected or adjusted, and the third phase of each level of the third image pyramid may be directly calculated using the phase difference of each level. The calculation formula of the third phase of the first level of the third image pyramid is: in, is the third phase of the third image pyramid at level l, is the second phase of the lth level of the second image pyramid, is the phase difference at the lth level between the first image pyramid and the second image pyramid.

[0066] In some embodiments, the step of calculating the third phase of each level of the third image pyramid according to the phase difference of each level between the first image pyramid and the second image pyramid, the interpolation factor, and the second phase of each level of the second image pyramid (i.e., step S323) includes the following steps: when the phase difference of at least one level between the first image pyramid and the second image pyramid does not meet the preset condition, determining the phase difference of the first target level that does not meet the preset condition, correcting and adjusting the phase difference of the first target level to obtain the precise phase difference of the first target level; calculating the third phase of the first target level of the third image pyramid according to the precise phase difference of the first target level, the interpolation factor, and the second phase of each level of the second image pyramid; calculating the third phase of the second target level of the third image pyramid according to the phase difference of the second target level between the first image pyramid and the second image pyramid, the interpolation factor, and the second phase of each level of the second image pyramid; wherein the second target level is the level corresponding to the phase difference that meets the preset condition. That is to say, for the level corresponding to the phase difference that meets the preset conditions (i.e., the second target level), the phase difference of the level does not need to be corrected or adjusted, and the phase difference of the level is directly used to calculate the third phase of the level of the third image pyramid; for the level corresponding to the phase difference that does not meet the preset conditions (i.e., the first target level), the phase difference of the level needs to be corrected and adjusted to obtain the precise phase difference of the level, and the precise phase difference of the level is used to calculate the third phase of the level of the third image pyramid. In this case, the calculation formula for the third phase of the first target level l of the third image pyramid is: in, is the third phase of the third image pyramid at level l, is the second phase of the lth level of the second image pyramid, is the precise phase difference at level l between the first image pyramid and the second image pyramid.

[0067] By performing Gabor filtering on the target reference frame image and the current frame image, the frequency spectrum diagrams of different frequencies and directions (i.e., image pyramid) are obtained, the phases and amplitudes corresponding to different levels of the image pyramid are calculated, and the phase differences corresponding to different levels of the image pyramid are calculated and corrected; the corrected phase difference is then adjusted, and the phase and amplitude of the intermediate frame are obtained according to the interpolation factor, and the interpolated intermediate frame is obtained by image recombination.

[0068] In some embodiments, the step of determining that the phase difference between the first image pyramid and the second image pyramid does not meet the preset condition comprises: determining that the phase difference between the first image pyramid and the next level of the first image pyramid does not meet the preset condition according to the phase difference between the first level and the next level of the first level. and the phase difference between the next level and the next two levels below the level Calculate the absolute value of the phase difference (|φl |). The absolute value of the phase difference (|φ l |) is greater than π / 2, it is determined that the phase difference of the level between the first image pyramid and the second image pyramid does not meet the preset condition, and the phase difference needs to be corrected and adjusted.

[0069] In some embodiments, the phase difference is calculated according to the following formula (8):

[0070]

[0071] Among them, φ l is the phase difference of the first target level (i.e. level l), is the phase difference between the first target level and the level below the first target level, is the phase difference between one level below the first target level and two levels below the first target level, and λ is a preset scaling factor.

[0072] Using formula (8), the coarse level phase difference is scaled according to an arbitrary scaling factor λ, and a scale-independent estimate can be obtained.

[0073] The purpose of setting the preset condition is: since atan2 (inverse tangent function) is used to calculate the phase difference of the same level of the pyramid, and the value range of atan2 function is [-π,π], the motion difference between the corresponding levels may exceed multiple 2π; therefore, the preset condition is needed to correct and adjust the phase difference of the current level. When the displacement of the object in the image is large, atan2 cannot accurately calculate the phase corresponding to the displacement of the object, so it is necessary to limit the phase difference calculated for different levels of the pyramid to obtain a more accurate phase difference.

[0074] Based on the assumption that the phase difference between two resolution levels is not arbitrarily different, the phase difference between different levels can be used as a confidence measure to quantify whether the calculated phase shift is reliable. First, the phase offset Adding integer multiples of 2π, the absolute difference in phase between consecutive different resolution levels will not be greater than the given tolerance. Therefore, using π as the tolerance distance, the phase is modified in such a way that the pixel phase difference between two levels will never be greater than π.

[0075] In some embodiments, Figure 6 As shown, the correction and adjustment of the phase difference of the first target level to obtain the precise phase difference of the first target level includes the following steps:

[0076] Step S61 , scaling the phase difference between the level below the first target level and the two levels below the first target level according to a preset scaling factor to obtain a corrected phase difference between the first target level and the level below the first target level.

[0077] The corrected phase difference between the first target level and the level below the first target level can be calculated according to the following formula (9):

[0078]

[0079] in, is the phase difference between the level below the first target level and the two levels below the first target level, is the corrected phase difference between the first target level and the level below the first target level, and λ is a preset scaling factor.

[0080] Step S62, calculating an adjustment factor of the first target level according to the corrected phase difference between the first target level and a level below the first target level and the phase difference between the first target level and a level below the first target level.

[0081] In this step, you can and Calculate the adjustment factor γ for the first target level l* .

[0082] Step S63, adjusting the phase difference according to the phase difference between the first target level and the level below the first target level and the adjustment factor of the first target level to obtain a precise phase difference.

[0083] The precise phase difference can be calculated according to the following formula (10):

[0084]

[0085] in, is the phase difference between the first target level and the level below the first target level, γ l* is the adjustment factor of the first target level, γ l* is a positive integer, It is the precise phase difference between the first target level and the level below the first target level.

[0086] Due to the phase correction, is no longer equal to φ l+1 , or not equal to φ l+1 ±γ2π, in order to make the intermediate frame image smoother, it is necessary to ensure that the phase value φ calculated at different levels of the pyramid l and φ l+1 , until an integer multiple of 2π, and at the same time, the phase shift correction difference needs to be maintained Therefore, by searching for a γ l* Make Infinite Approach

[0087] In some embodiments, the step of calculating the adjustment factor of the first target level (i.e., step S63) according to the corrected phase difference between the first target level and the level below the first target level and the phase difference between the first target level and the level below the first target level comprises the following steps:

[0088] Step S631 : calculating an intermediate adjustment factor of the first target level according to a phase difference between the first target level and a level below the first target level, a phase of the first target level of the first image pyramid, and a phase of the first target level of the second image pyramid.

[0089] The intermediate adjustment factor of the first target level can be calculated according to the following formula (11):

[0090]

[0091] in, is the phase difference between the first target level and the level below the first target level, φ0 is the phase of the first target level of the first image pyramid, φ1 is the phase of the first target level of the second image pyramid, γ l is the intermediate adjustment factor for the first target level.

[0092] Step S632, calculating the adjustment factor of the first target level according to the corrected phase difference between the first target level and the level below the first target level, the phase difference between the first target level and the level below the first target level, and the intermediate adjustment factor of the first target level.

[0093] The adjustment factor of the first target level can be calculated according to the following formula (12):

[0094]

[0095] in, is the corrected phase difference between the first target level and the level below the first target level, is the phase difference between the first target level and the level below the first target level, γ l is the intermediate adjustment factor of the first target level, γ l* is the adjustment factor for the first target level.

[0096] The disclosed embodiment uses the phase method as an example for explanation. Considering the influence of block division in video encoding and decoding, the details of the object are blurred due to filtering. Therefore, the use of the phase method for video interpolation can effectively reduce the blur caused by the block division. It should be noted that the disclosed embodiment does not exclude the use of other methods for video interpolation, such as the optical flow method, etc., which can still determine whether the movement of the object in the video is a large movement or a small movement.

[0097] The disclosed embodiment determines whether to insert a current frame according to the features of a decoded frame (such as a key frame, a switching frame, and an intra-frame), calculates the peak signal-to-noise ratio between each corresponding reference frame and the current frame to determine the source frame as input, and then calculates the distance between the current frame and the reference frame as the input of the intermediate frame to be inserted. Compared with the solution of using previous and next frames for interpolation in the prior art, the peak signal-to-noise ratio of the inserted frame can be improved.

[0098] At present, there are different ways to convert frame rates for different codec protocols, including optical flow-based method, motion compensation interpolation method and implicit phase method; the optical flow-based method and motion compensation interpolation method mainly use local matching, and the video interpolation is performed on the display end, which makes it difficult to solve the interference problems caused by extreme lighting and occlusion. In order to achieve more accurate matching, the embodiment of the present application introduces a phase-based method in AV1 interpolation, taking into account that the global motion of objects in the image is conducive to optimizing the edges of objects, and the image is smoother between inserted frames, avoiding the expensive cost of global optimization required by typical optical flow. The disclosed embodiment can improve the quality of interpolation on the basis of improving the smoothness of the video, and at the same time use the phase method for interpolation, which can retain the processed results of the input of the inserted frame and apply it to the next frame to improve the efficiency of video interpolation.

[0099] The present disclosure also provides a video frame insertion device, including:

[0100] A decoding module is used to decode the code stream to obtain the current frame image;

[0101] A feature judgment module, used for obtaining a first feature value of the current frame image, and determining whether to perform interpolation on the current frame image according to the first feature value;

[0102] The phase interpolation module is used to generate an intermediate frame image according to the current frame image when it is determined to interpolate the current frame image, and the intermediate frame image is used to interpolate before or after the current frame image.

[0103] The embodiments of the present disclosure also provide a computer device, which includes: one or more processors and a storage device; wherein one or more programs are stored on the storage device, and when the one or more programs are executed by the one or more processors, the one or more processors implement the video interpolation method provided in the aforementioned embodiments.

[0104] The embodiments of the present disclosure further provide a computer-readable medium on which a computer program is stored, wherein when the computer program is executed, the video frame insertion method provided in the aforementioned embodiments is implemented.

[0105] It will be appreciated by those skilled in the art that all or some of the steps in the method disclosed above, the functional modules / units in the device may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0106] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A video frame insertion method, characterized in that: The method is applied to a video decoding end, comprising: Decode the code stream to obtain the current frame image; In the case where it is determined to perform frame insertion on the current frame image, an intermediate frame image is generated according to the current frame image, and the intermediate frame image is used for frame insertion before or after the current frame image.

2. The method according to claim 1, characterized in that The step of generating an intermediate frame image according to the current frame image comprises: Determine each reference frame image of the current frame image, and determine a target reference frame image from each reference frame image; The intermediate frame image is generated according to the target reference frame image and the current frame image.

3. The method according to claim 2, characterized in that The step of generating an intermediate frame image according to the target reference frame image and the current frame image comprises: Establishing a first image pyramid of the current frame image, and establishing a second image pyramid of the target reference frame image; A third image pyramid of the intermediate frame image is established according to the first image pyramid and the second image pyramid; wherein the phase of each level of the third image pyramid is determined according to the phase difference of each level between the first image pyramid and the second image pyramid, and the amplitude of each level of the third image pyramid is determined according to the first image pyramid and the second image pyramid; The intermediate frame image is generated according to the amplitude and phase of each level of the third image pyramid.

4. The method according to claim 3, characterized in that The determining the phase of each level of the third image pyramid according to the phase difference of each level between the first image pyramid and the second image pyramid includes: determining a first amplitude and a first phase of each level of the first image pyramid, and a second amplitude and a second phase of each level of the second pyramid; Calculating the phase difference at each level between the first image pyramid and the second image pyramid; The third phase of each level of the third image pyramid is calculated according to the phase difference of each level between the first image pyramid and the second image pyramid, the interpolation factor, and the second phase of each level of the second image pyramid.

5. The method according to claim 4, characterized in that The calculating, according to the phase difference at each level between the first image pyramid and the second image pyramid, the interpolation factor, and the second phase at each level of the second image pyramid, the third phase at each level of the third image pyramid comprises: When the phase difference at each level between the first image pyramid and the second image pyramid meets a preset condition, the third phase at each level of the third image pyramid is calculated according to the phase difference at each level between the first image pyramid and the second image pyramid, an interpolation factor, and the second phase at each level of the second image pyramid.

6. The method according to claim 4, characterized in that The calculating, according to the phase difference at each level between the first image pyramid and the second image pyramid, the interpolation factor, and the second phase at each level of the second image pyramid, the third phase at each level of the third image pyramid comprises: When the phase difference of at least one level between the first image pyramid and the second image pyramid does not meet a preset condition, determine that the level corresponding to the phase difference that does not meet the preset condition is a first target level, and correct and adjust the phase difference of the first target level to obtain an accurate phase difference of the first target level; Calculating a third phase of the first target level of the third image pyramid according to the precise phase difference of the first target level, the interpolation factor, and the second phases of each level of the second image pyramid; A third phase of the second target level of the third image pyramid is calculated according to the phase difference of the second target level between the first image pyramid and the second image pyramid, the interpolation factor, and the second phase of each level of the second image pyramid; wherein the second target level is a level corresponding to the phase difference that satisfies the preset condition.

7. The method according to claim 6, characterized in that The following steps are used to determine that the phase difference between the first image pyramid and the second image pyramid does not meet a preset condition: Calculate an absolute value of a phase difference according to a phase difference between the level and a level below the level, a phase difference between a level below the level and two levels below the level, and a preset scaling factor; When the absolute value of the phase difference is greater than π / 2, it is determined that the phase difference at the level between the first image pyramid and the second image pyramid does not satisfy a preset condition.

8. The method according to claim 6, characterized in that The correcting and adjusting the phase difference of the first target level to obtain the accurate phase difference of the first target level includes: Scaling the phase difference between one level below the first target level and two levels below the first target level according to a preset scaling factor to obtain a corrected phase difference between the first target level and one level below the first target level; calculating an adjustment factor of the first target level according to the corrected phase difference between the first target level and a level below the first target level and the phase difference between the first target level and a level below the first target level; The phase difference is adjusted according to the phase difference between the first target level and a level below the first target level and an adjustment factor of the first target level to obtain a precise phase difference.

9. The method according to claim 8, characterized in that The calculating the adjustment factor of the first target level according to the corrected phase difference between the first target level and a level below the first target level and the phase difference between the first target level and a level below the first target level includes: calculating an intermediate adjustment factor of the first target level according to a phase difference between the first target level and a level below the first target level, a phase of the first target level of the first image pyramid, and a phase of the first target level of the second image pyramid; An adjustment factor of the first target level is calculated according to the corrected phase difference between the first target level and a level below the first target level, the phase difference between the first target level and a level below the first target level, and an intermediate adjustment factor of the first target level.

10. The method according to claim 4, characterized in that The interpolation factor is determined according to the expected output order of the target reference frame image and the expected output order of the current frame image.

11. The method according to claim 4, characterized in that The step of determining the amplitude of each level of the third image pyramid according to the first image pyramid and the second image pyramid comprises: For each level of the first image pyramid and the second image pyramid, a first amplitude of the level of the first image pyramid and a second amplitude of the level of the second pyramid are superimposed to obtain a third amplitude of each level of the third image pyramid.

12. A video frame insertion device, comprising: A decoding module is used to decode the code stream to obtain the current frame image; A feature judgment module, used for obtaining a first feature value of the current frame image, and determining whether to perform interpolation on the current frame image according to the first feature value; The phase interpolation module is used to generate an intermediate frame image according to the current frame image when it is determined to interpolate the current frame image, and the intermediate frame image is used to interpolate before or after the current frame image.

13. A computer device, wherein: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the video frame insertion method as described in any one of claims 1-11.

14. A computer readable medium having a computer program stored thereon, wherein: When the program is executed, the video frame insertion method as described in any one of claims 1-11 is implemented.

Images