Video coding method and apparatus based on frame interpolation

Abstract

The application discloses a video coding method and device based on frame insertion. The method obtains frame insertion identification information in a decoding process, generates an inserted image by using already decoded images according to the frame insertion identification information, and outputs the inserted image at a target time. The method can reduce the number of images to be processed during coding, reduce the data consumption of coded videos while obtaining high-quality inserted images, and improve coding efficiency.

Description

Technical Field

[0001] This invention relates to the field of video encoding and decoding technology, and more specifically, to a video encoding and decoding method and apparatus based on frame interpolation. Background Technology

[0002] Existing video codec technologies, such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, AVS series, and AV1 series, typically cannot generate interpolated images to increase the frame rate during video decoding. Instead, they can only skip some images during decoding to reduce the frame rate. Although existing video codec technologies delegate the image insertion operation to post-processing, the post-processing does not consider the effects of frame insertion during encoding, and post-processing is unaware of which images were skipped during encoding. This means that the frame insertion effect in post-processing cannot effectively restore the original video's frame rate and quality. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a video encoding and decoding method and apparatus based on frame interpolation. During the decoding process, this method acquires frame interpolation identification information, generates an interpolated image using the already decoded image based on the frame interpolation identification information, and outputs the interpolated image at the target time. This method reduces the number of images that need to be processed during encoding, obtains high-quality interpolated images while reducing the amount of data consumed in encoding the video, and improves encoding efficiency.

[0004] Therefore, the first objective of this invention is to propose a video decoding method based on frame interpolation, which performs the following operations on the input video bitstream:

[0005] Obtain the encoded data of at least one first image from the bitstream, and decode it to obtain at least one first decoded image and the output time of the first decoded image;

[0006] The frame interpolation identifier information corresponding to the second image is obtained from the bitstream. The frame interpolation identifier information includes index information of at least one first decoded image used to generate the second image, and also includes the output time of the second image.

[0007] Based on the interpolation identifier information, a second image is generated using the at least one first decoded image;

[0008] Based on the output time of the second image and the output time of the first decoded image, the video composed of the first decoded image and the second image is output in the order of the output times.

[0009] Furthermore, the frame interpolation identification information is included in the image header of the corresponding second image in the bitstream, and the bitstream does not contain the strip data of the second image.

[0010] Furthermore, the frame interpolation identification information is included in the encoded data of a first image in the bitstream, and the first decoded image obtained by decoding the first image is one of the at least one first decoded images used to generate the second image.

[0011] Furthermore, the method also includes:

[0012] The frame interpolation method information corresponding to the second image is obtained from the bitstream, and the frame interpolation method information indicates the generation method used to generate the second image;

[0013] Based on the interpolation identifier information corresponding to the second image, the second image is generated using the at least one first decoded image and the generation method indicated by the interpolation method information.

[0014] The second objective of this invention is to propose a video coding method based on frame interpolation, which performs the following operations on the input video:

[0015] Extract at least one image from the input video and skip the others;

[0016] For the extracted image, the extracted image is encoded using an existing encoding method to obtain a first bitstream;

[0017] For the skipped image, the frame insertion identification information corresponding to the skipped image is put into the second bitstream. The frame insertion identification information includes the index information of at least one decoded image of the extracted image. The decoded image is used to generate the skipped image corresponding to the frame insertion identification information. The frame insertion identification information also includes the output time of the skipped image.

[0018] The first and second bitstreams are merged to obtain the output bitstream.

[0019] Furthermore, the method also includes:

[0020] The image header corresponding to the skipped image is inserted into the second bitstream;

[0021] The interpolation identification information is placed in the image header;

[0022] The encoding of the strip data of the skipped image is skipped.

[0023] Furthermore, the method also includes:

[0024] The frame interpolation identifier information is placed into the encoded data of an extracted image in the first bitstream. The decoded image of the extracted image is one of at least one decoded image used to generate the skipped image.

[0025] A third objective of this invention is to provide a video decoding apparatus based on frame interpolation, comprising:

[0026] processor;

[0027] Memory for storing bitstream and image data to be decoded; and

[0028] One or more programs are used to perform the frame-interpolation-based video decoding method as described in the first objective above.

[0029] The fourth objective of this invention is to provide a frame interpolation-based video encoding apparatus, comprising:

[0030] processor;

[0031] Memory for storing bitstreams and images to be encoded; and

[0032] One or more programs are used to perform the frame interpolation-based video coding method as described in the second objective above.

[0033] The beneficial effects of this invention are as follows:

[0034] On the one hand, when processing video, the encoding end can extract a portion of the images from the video for encoding, which can reduce the amount of data consumed in encoding the video. At the same time, a portion of the saved data can be used to improve the quality of the encoded images, which can effectively improve encoding efficiency and reduce the computational complexity of the encoding end.

[0035] On the other hand, when processing video, the decoding end can generate high-quality inserted images based on the high-quality images that have already been decoded. This can effectively improve the frame rate and reconstruction quality of the reconstructed video, and also ensure that the number of images in the video output by the decoding end is consistent with the number of images in the original video of the encoding end, avoiding mismatch between the encoding and decoding ends. In addition, since frame extraction at the encoding end and frame insertion at the decoding end do not affect the inherent encoding and decoding operations, the method proposed in this invention also has compatibility with different video encoding and decoding methods. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of a video decoding method based on frame interpolation according to an embodiment of the present invention;

[0038] Figure 2 This is a flowchart of a video encoding method based on frame interpolation according to an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0040] Definition of noun:

[0041] VVC: Versatile Video Coding, also known as H.266, is a video coding standard that can compress input video into a bitstream. This bitstream can contain multiple sub-bitstreams, and there can be reference relationships between the sub-bitstreams.

[0042] DPB: Short for Decoded Picture Buffer, this DPB exists in the VVC codec and stores and manages decoded images for output or to provide reference frames for subsequent decoded video frames. When managing decoded images, the DPB attaches tags to them to identify their attributes, such as the image's index number and layer number, whether it is used for short-term or long-term reference, and whether it is no longer in use.

[0043] A slice is a data unit defined in traditional video encoding and decoding methods. An image typically contains at least one slice. There are several types of slices, including intra slices, single-hypothesis inter-frame slices (P slices), and double-hypothesis inter-frame slices (B slices), which are encoded and decoded using intra-prediction, single-hypothesis inter-frame prediction, and double-hypothesis inter-frame prediction methods, respectively.

[0044] NAL packets, short for Network Abstract Layer units, are a video stream encapsulation format. NAL packets consist of a header and payload. The header contains information such as layer identifiers, type identifiers, and temporal layer identifiers, allowing the system layer to quickly identify NAL packets and distinguish their functions and data types.

[0045] Image header: A set of syntax elements that contains encoding and decoding information shared by all stripes of an image.

[0046] Image insertion: Adding an image to a video at a specific display order or moment to increase the video's frame rate.

[0047] Image generation: Use a given set of images to generate another image.

[0048] This embodiment discloses a video decoding method based on frame interpolation, the flowchart of which is as follows: Figure 1As shown, the implementer of this method can be a decoder. The decoding method is described in detail below.

[0049] Specifically, for the input bitstream, interpolation identification information is obtained from the bitstream. This identification information directly or indirectly contains the index information of the decoded image used to generate the interpolated image. The decoder can use the decoded image to obtain the interpolated image using traditional or intelligent generation methods. This identification information also directly or indirectly contains the output time or presentation time of the obtained interpolated image. The decoder can output the interpolated image based on the time information. Typically, the bitstream does not contain the encoded data of the interpolated image, or only contains a portion of the encoded data of the interpolated image. Based on this portion of encoded data, the complete pixel information of the interpolated image cannot be decoded. The advantage of this is that some or all of the pixel information of the interpolated image can be generated from other decoded images, which can effectively reduce the data volume of the bitstream. When the bitstream contains the complete encoded data of an image, the decoder uses an existing decoding method to decode the image to obtain its decoded image. The existing decoding method can be a decoding method of standard specifications such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, AVS series, or AV1 series.

[0050] The preceding embodiments have provided an overview of the entire decoding method process. The following will describe the details of each operation in conjunction with specific examples.

[0051] In one embodiment, the frame interpolation-based video decoding method obtains explicit frame interpolation identification information from the syntax elements of the bitstream. Specifically, the decoder obtains an image header from the bitstream, which contains a generation insertion image identification information to indicate whether the pixel information of the image corresponding to the image header needs to be generated and inserted into the output video. When the generation insertion image identification information indicates that the current image needs to be generated and inserted into the output video, the image header in the bitstream does not contain the strip data of that image, and the image needs to be generated from other decoded images; when the generation insertion image identification information indicates that the current image does not need to be generated, the image header in the bitstream should contain the strip data of that image, and the decoded image is obtained using an existing decoding method. The image header also contains display information corresponding to the inserted image, such as whether to output display or display number, etc., and the decoder can determine the output of the inserted image based on this display information.

[0052] In one possible implementation, the above embodiments can be further modified. Specifically, the generated inserted image identification information can further indicate that a portion of the corresponding image needs to be generated from other decoded images, while the encoded data of other portions is carried in the strip data following the image header of the corresponding image. This implementation method can obtain pixel information of simpler regions using the generation method and obtain pixel information of more complex regions using the encoding and decoding method when the video image content contains regions of varying complexity. This avoids the problem that existing image generation methods are insufficient to handle complex content in images, and improves the fidelity of complex content in video images as much as possible while reducing bitstream data.

[0053] In another implementation, the generated interpolated image identifier information may also indicate the generation method used to generate the interpolated image, thereby allowing the decoder to select one of its supported generation methods for generating the interpolated image. These generation methods may include, but are not limited to: methods that average pixel information from multiple decoded images, methods that weight pixel information from multiple decoded images, or intelligent generation methods based on neural networks, such as Real-Time Intermediate Flow Estimation for Video Frame Interpolation (RIFE).

[0054] In another implementation, generating the insert image identification information may further include identification information of the decoded image used to generate the insert image. The decoder can retrieve at least one decoded image for generating the insert image from the decoded image buffer based on the identification information of the decoded image, and generate the insert image using the generation method. The identification information of the decoded image can be explicit information, such as the display number or decoding number of the decoded image; the identification information of the decoded image can also be implicit information, such as the difference between the display number of the decoded image and the display number of the insert image, or a reference image queue recorded in the image header of the insert image, i.e., the decoder can retrieve available decoded images from the reference image queue for the generation of the insert image.

[0055] In another implementation, the identifier information of the decoded image used to generate the inserted image, included in the generated insert image identifier information, can also be a reference image queue recorded in the image parameter set or sequence parameter set referenced by the inserted image. In yet another implementation, when different image regions or different stripes in the inserted image can use different decoded images to generate pixels for the inserted image, the identifier information of the decoded image can also be a reference image queue recorded in the strip header of the inserted image. The advantage of using a reference image queue is that it allows reuse of the decoded image buffer management based on reference image queue information in existing video coding methods without introducing additional management operations; simultaneously, it ensures that the decoded images required to generate the inserted image exist in the decoded image buffer. The reference image queue typically contains multiple reference images and records their index numbers. Considering that the generation of an insert image does not necessarily require the use of all reference images, in one implementation, the first few reference images in the reference image queue arranged in chronological order, such as the first two reference images, can be selected as the decoded images required to generate the insert image; in another implementation, the index of the decoded image required to generate the insert image in the reference image queue can be placed in the image header or strip header of the insert image to limit the decoded images available for the insert image.

[0056] The identification information used in the above implementation should appear in the syntax elements of the bitstream. Without loss of generality, in conjunction with the syntax structure of H.266 / VVC, an example of a syntax element for the identification information used above can be shown in the following table.

[0057]

[0058] The italicized syntax elements are those defined in this invention; the other syntax elements are existing syntax elements in the H.266 / VVC syntax structure and will not be elaborated further here. The meanings of the syntax elements defined in this invention are as follows:

[0059] 1) ph_interpolate_flag is the enable flag in the generated interpolated image identifier information, indicating whether the corresponding image needs to be generated and inserted into the output video;

[0060] 2) ph_interpolate_method_id is the generation method index in the generated interpolated image identifier information, indicating the generation method used to generate the interpolated image;

[0061] 3) ph_num_dec_pic and ph_delta_poc_dec_pic[i] are the identifiers of the decoded images used to generate the inserted image in the generated inserted image identifier information, indicating the number of decoded images and the difference between the display number of the i-th decoded image and the display number of the inserted image, respectively.

[0062] 4) The existing syntax elements ph_pic_order_cnt_lsb and ph_poc_msb_cycle_val (if they exist) in H.266 / VVC indicate the display number of the inserted image.

[0063] In another implementation, the identification information in the image header may be present in the supplemental information (SEI), which is placed after the bitstream of the image closest to the inserted image in the display order.

[0064] In another implementation, the insertion image identifier information is not present in the image header of the insertion image, but rather in the syntax element of a decoded image used to generate the insertion image. After parsing the insertion image identifier information in the syntax element of the decoded image, the decoder can obtain the decoded image used to generate the insertion image based on the insertion image identifier information, generate the insertion image using a given generation method, and insert the insertion image into the output video according to the display number of the insertion image indicated in the insertion image identifier information. In this case, the insertion image identifier information must explicitly record the index information of the decoded image used to generate the insertion image, the generation method information, and the display information of the insertion image. Typically, to ensure timely generation of the insertion image, the insertion image identifier information can be present in the last decoded image among all the decoded images used to generate the insertion image. However, this method does not exclude the possibility that the insertion image identifier information may exist in other decoded images. In one possible implementation, since there may be multiple insert images before the decoded image, the decoded image needs to be used to generate multiple insert images. The insert image generation identification information in the decoded image needs to include the number of insert images or the display number information of all insert images.

[0065] In one implementation, considering the inconsistency between the encoding order and the display order in video encoding and decoding, it is necessary to constrain the range of decoded images that the decoder can use when generating the inserted image, in order to avoid excessive decoding and display delays.

[0066] In one embodiment, the frame-interpolation-based video decoding method obtains implicit frame-interpolation identification information from the syntax elements of the bitstream. For example, the decoder determines whether to generate an insert image based on the continuity of the display numbers of the decoded images in the video. Specifically, the decoder decodes the images in the bitstream in sequence and outputs the decoded images to the output video according to their display numbers. Before each output of a decoded image, the decoder compares the display number of the current decoded image with the display number of the nearest preceding decoded image in the display order. If the two display numbers are not consecutive, several insert images are inserted before the current decoded image to make the display numbers consecutive. The decoded images used to generate the insert images are the current decoded image and the nearest preceding decoded image in the display order.

[0067] In one embodiment, the interpolation frame identification information used to generate an inserted image can be recorded in the image-level parameter set corresponding to the inserted image, or it can be referenced from the sequence-level parameter set corresponding to the sequence to which the inserted image belongs. The sequence-level parameter set can record multiple sets of different interpolation frame identification information to provide different inserted images in the sequence with reference to the interpolation frame identification information. The advantage of this is that when most inserted images in the sequence use the same or similar interpolation frame identification information, these inserted images can share and reference it from the sequence-level parameter set, without needing to record it in the image parameter set corresponding to each inserted image, thus saving the amount of data occupied by each inserted image. When some inserted images need to use different interpolation frame identification information, on the one hand, these inserted images can reference other sets of interpolation frame identification information recorded in the sequence-level parameter set; on the other hand, these inserted images can also record the interpolation frame identification information they need in their own corresponding image-level parameter set. This design also maintains the flexibility of generating inserted images. A specific syntax structure and operation example are as follows: 1) Multiple sets of interpolation frame identification information recorded in the sequence-level parameter set.

[0068]

[0069] Where sps_num_temporal_restoration_data represents the number of groups of interpolation identification information, and temporal_restoration_data(i) represents the i-th group of interpolation identification information.

[0070] 2) Frame interpolation identification information recorded in the image-level parameter set

[0071]

[0072] Among them, prd_picture_order_cnt_lsb is used to directly or indirectly represent the image number corresponding to the image level parameter set; prd_decoded_pic_flag indicates whether the image is an inserted image or a decoded image that can be used to generate the inserted image. If it is a decoded image, the interpolation identification information is not needed because the image is decoded from the bitstream. For inserted images, if the value of `sps_num_temporal_restoration_data` is equal to 0, then there is no interpolation frame identifier information recorded in the sequence-level parameter set that can be used for reference. In this case, the inserted image can only record the interpolation frame identifier information in the image-level parameter set. If the value of `sps_num_temporal_restoration_data` is greater than 0, then the image-level parameter set can use `prd_use_srd_temporal_restoration_data_flag` to indicate whether its interpolation frame identifier information is referenced from the sequence-level parameter set or recorded in itself. `prd_srd_temporal_restoration_data_idx` represents the number of the interpolation frame identifier information referenced from the sequence-level parameter set, and `temporal_restoration_data(sps_num_temporal_restoration_data)` is the interpolation frame identifier information recorded in the image-level parameter set. The number can be set to `sps_num_temporal_restoration_data`. When sps_num_temporal_restoration_data equals 1 and prd_use_srd_temporal_restoration_data_flag equals 1, only one set of interpolation identifier information is recorded in the sequence-level parameter set. At this time, the image-level parameter set does not need to use prd_srd_temporal_restoration_data_idx, but can use this one set of interpolation identifier information by default.

[0073] 3) Frame interpolation identification information

[0074]

[0075] Wherein, trdIdx represents the number of the interpolation frame identifier information; delta_poc_of_backword_ref_pic_minus1[trdIdx] represents the difference between the number of the decoded image used to generate the interpolated image recorded in the trdIdx group of interpolation frame identifier information and the number of the interpolated image, and the display order of the decoded image is before the interpolated image; delta_poc_of_forward_ref_pic_minus1[trdIdx] represents the difference between the number of the decoded image used to generate the interpolated image recorded in the trdIdx group of interpolation frame identifier information and the number of the interpolated image, and the display order of the decoded image is after the interpolated image.

[0076] In another implementation, the frame interpolation identifier information can also be recorded more flexibly in the following form:

[0077]

[0078] Wherein, num_ref_pic_minus1[trdIdx] represents the number of decoded images used to generate the inserted image recorded in the trdIdx group of interpolation identification information, abs_delta_poc_of_ref_pic_minus1[trdIdx][i] represents the absolute value of the difference between the number of the i-th decoded image in the trdIdx group of interpolation identification information and the number of the inserted image, and sign_of_delta_poc[trdIdx][i] represents the sign of the difference. A positive difference indicates that the decoded image is after the display order of the inserted image and is recorded as a backward reference image, and a negative difference indicates that the decoded image is before the display order of the inserted image and is recorded as a forward reference image.

[0079] Based on the above information, the decoding operation can construct a list of decoded images for each inserted image, which can be used to generate the inserted image. When the list of decoded images contains both a forward reference image and a backward reference image, the operation to generate the inserted image can use interpolation, that is, using multiple images in time sequence to generate an image at a specific time in the middle of the corresponding time period. When the list of decoded images contains only a forward reference image or only a backward reference image, the operation to generate the inserted image can use extrapolation, that is, using multiple images in time sequence to generate an image at a time outside the corresponding time period. When the list of decoded images contains only one decoded image, the inserted image can be generated by copying or prediction.

[0080] This embodiment also discloses a video coding method based on frame interpolation, the flowchart of which is as follows: Figure 2As shown, the implementing entity of this method can be an encoder. For the input video, the encoder extracts a portion of images from the video according to a certain rule, encodes them, and generates a bitstream. The extraction rule can be extraction at fixed intervals, or extraction based on the richness of image content after analyzing the video content. In one embodiment, the encoder does not completely skip the encoding of unextracted images. Instead, it inserts the image header information of the unextracted images into the bitstream, and includes insertion image identification information. This information includes the index information of the decoded images used to generate the insertion images, generation method information, and display information of the insertion images. Then, the encoder skips the encoding of the strip data of that image and proceeds to the encoding of the next image, ultimately obtaining the bitstream. In another embodiment, the encoder completely skips the encoding of the unextracted images and includes insertion image identification information in the syntax element of the extracted image used to generate the insertion images. This information includes the number or display number of all insertion images generated from the extracted image.

[0081] This embodiment discloses a decoding device, which includes a processor and a memory, for executing the decoding method disclosed in this invention.

[0082] This embodiment discloses an encoding device, which includes a processor and a memory, for executing the encoding method disclosed in this invention.

[0083] The above description of the embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A video decoding method based on frame interpolation, characterized in that, For the input video stream, perform the following operations: Obtain the encoded data of at least one first image from the bitstream, and decode it to obtain at least one first decoded image and the output time of the first decoded image; Obtain a first identifier from the bitstream, the first identifier being used to identify the method of obtaining the frame interpolation identifier information; According to the first identifier, the interpolation identifier information of the corresponding second image is obtained from the sequence-level parameter set or image-level parameter set of the bitstream. The interpolation identifier information includes the index information of at least one first decoded image used to generate the second image, and also includes the output time of the second image. Based on the interpolation identifier information, a second image is generated using the at least one first decoded image; Based on the output time of the second image and the output time of the first decoded image, the video composed of the first decoded image and the second image is output in the order of the output times.

2. The method according to claim 1, characterized in that, The frame interpolation identifier information is included in the image-level parameter set corresponding to the second image in the bitstream, and the bitstream does not contain the strip data of the second image.

3. The method according to claim 1, characterized in that, The frame interpolation identification information is contained in the encoded data of a first image in the bitstream, and the first decoded image obtained by decoding the first image is one of the at least one first decoded images used to generate the second image.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: The frame interpolation method information corresponding to the second image is obtained from the bitstream, and the frame interpolation method information indicates the generation method used to generate the second image; Based on the interpolation identifier information corresponding to the second image, the second image is generated using the at least one first decoded image and the generation method indicated by the interpolation method information.

5. A video coding method based on frame interpolation, characterized in that, For the input video, perform the following operations: Extract at least one image from the input video and skip the others; For the extracted image, the extracted image is encoded using an existing encoding method to obtain a first bitstream, and a first identifier is added to the first bitstream. The first identifier is used to identify the acquisition method of the interpolation identifier information. For a skipped image, interpolation identification information corresponding to the skipped image is placed in the sequence-level parameter set or image-level parameter set of the second bitstream. The interpolation identification information includes the index information of at least one decoded image of the extracted image. The decoded image is used to generate a skipped image corresponding to the interpolation identification information. The interpolation identification information also includes the output time of the skipped image. The first and second bitstreams are merged to obtain the output bitstream.

6. The method according to claim 5, characterized in that, The method further includes: The image-level parameter set corresponding to the skipped image is inserted into the second bitstream; The interpolation identifier information is placed into the image-level parameter set; The encoding of the strip data of the skipped image is skipped.

7. The method according to claim 5, characterized in that, The method further includes: The frame interpolation identifier information is placed into the encoded data of an extracted image in the first bitstream. The decoded image of the extracted image is one of at least one decoded image used to generate the skipped image.

8. A video decoding apparatus based on frame interpolation, comprising: processor; A memory used to store bitstream and image data to be decoded; as well as One or more programs are used to perform the frame-interpolation-based video decoding method as described in any one of claims 1-4.

9. A video encoding apparatus based on frame interpolation, comprising: processor; Memory used to store bitstreams and images to be encoded; as well as One or more programs are used to perform the frame interpolation-based video coding method as described in any one of claims 5-7.

Images