Video coding and decoding method and device based on frame insertion

Abstract

The invention discloses a video coding and decoding method and device based on frame insertion, and the method comprises the steps: obtaining frame insertion identification information in a decoding process, generating an insertion image through employing a decoded image according to the frame insertion identification information, and outputting the insertion image at a target moment. According to the method, the number of images needing to be processed during coding can be reduced, the data volume consumed by video coding is reduced while high-quality insertion images are obtained, and the coding efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video coding and decoding, and more specifically, to a video coding and decoding method and device based on frame insertion. Background Art

[0002] Existing video codec technologies such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, AVS series or AV1 series, etc., are usually unable to generate inserted images to increase the frame rate of the video when decoding the video, but can only skip the decoding of some images during decoding to reduce the frame rate. Although the existing video codec technology leaves the operation of inserting images to post-processing, the effect of post-processing frame insertion is not considered during encoding and the post-processing does not know which images were skipped during encoding, so the post-processing frame insertion effect cannot restore the frame rate and quality of the original video well. Summary of the invention

[0003] In order to solve the above-mentioned shortcomings of the prior art, the present invention proposes a video encoding and decoding method and device based on interpolation. The method obtains interpolation identification information during the decoding process, generates an inserted image using the decoded image according to the interpolation identification information, and outputs the inserted image at the target time. The method can reduce the number of images that need to be processed during encoding, reduce the amount of data consumed by the encoded video while obtaining high-quality inserted images, and improve encoding efficiency.

[0004] To this end, the first object of the present invention is to propose a video decoding method based on interpolation, and for an input video code stream, the following operations are performed:

[0005] Acquire coded data of at least one first image from the code stream, and decode to obtain at least one first decoded image and an output time of the first decoded image;

[0006] Acquire interpolation identification information corresponding to the second image from the bitstream, where the interpolation identification information includes index information of at least one first decoded image used to generate the second image, and also includes an output time of the second image;

[0007] Generate a second image using the at least one first decoded image according to the interpolation frame identification information;

[0008] According to the output time of the second image and the output time of the first decoded image, a video consisting of the first decoded image and the second image is output in the order of the output time.

[0009] Furthermore, the interpolation frame identification information is included in the image header corresponding to the second image in the code stream, and the code stream does not include the strip data of the second image.

[0010] Furthermore, the interpolation identification information is included in the encoded data of a first image in the code stream, and a first decoded image obtained by decoding the first image is one of the at least one first decoded images used to generate the second image.

[0011] Furthermore, the method further comprises:

[0012] Acquire interpolation method information corresponding to the second image from the code stream, where the interpolation method information indicates a generation method for generating the second image;

[0013] According to the interpolation identification information corresponding to the second image, the second image is generated using the at least one first decoded image and the generation method indicated by the interpolation method information.

[0014] The second object of the present invention is to provide a video encoding method based on interpolation, for an input video, performing the following operations:

[0015] Extract at least one image from the input video and skip other images;

[0016] For the extracted image, use an existing encoding method to encode the extracted image to obtain a first code stream;

[0017] For a skipped image, inserting interpolation identification information corresponding to the skipped image into the second bitstream, the interpolation identification information including index information of a decoded image of at least one extracted image, the decoded image being used to generate the skipped image corresponding to the interpolation identification information, the interpolation identification information also including an output time of the skipped image;

[0018] The first bit stream and the second bit stream are combined to obtain an output bit stream.

[0019] Furthermore, the method further comprises:

[0020] Putting a picture header corresponding to the skipped picture into the second bitstream;

[0021] Putting the interpolation frame identification information into the image header;

[0022] Encoding of the slice data of the skipped picture is skipped.

[0023] Furthermore, the method further comprises:

[0024] The interpolation identification information is placed in the coded data of an extracted image in the first code stream, and the decoded image of the extracted image is one of the at least one decoded image used to generate the skipped image.

[0025] A third object of the present invention is to provide a video decoding device based on interpolation, comprising:

[0026] processor;

[0027] A memory for storing code streams and image data to be decoded; and

[0028] One or more programs are used to implement the interpolation-based video decoding method as described in the first purpose above.

[0029] A fourth object of the present invention is to provide a video encoding device based on interpolation, comprising:

[0030] processor;

[0031] A memory for storing code streams and images to be encoded; and

[0032] One or more programs are used to implement the interpolation-based video encoding method as described in the second purpose above.

[0033] The beneficial effects of the present invention are:

[0034] On the one hand, when processing the video, the encoder can extract a part of the image from the video for encoding, which can reduce the amount of data consumed by encoding the video. At the same time, part of the saved data can also be used to improve the quality of the encoded image, which can effectively improve the encoding efficiency and reduce the computational complexity of the encoder.

[0035] On the other hand, when processing the video, the decoding end can generate a high-quality inserted image based on the decoded high-quality image, which can effectively improve the frame rate and reconstruction quality of the reconstructed video, and can also ensure that the number of images in the video output by the decoding end is consistent with the number of images in the original video of the encoding end, thereby avoiding mismatch between the encoding and decoding ends; in addition, since the frame extraction at the encoding end and the frame insertion at the decoding end do not affect the inherent encoding and decoding operations, the method proposed in the present invention is also compatible with different video encoding and decoding methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 A flowchart of a video decoding method based on interpolation according to an embodiment of the present invention;

[0038] Figure 2 The present invention is a flowchart of a video encoding method based on interpolation. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0040] Definition of terms:

[0041] VVC: Versatile Video Coding, also known as H.266, is a video coding standard that can compress the input video to obtain a bitstream. The bitstream can contain multiple layers of sub-bitstreams, and the sub-bitstreams can have reference relationships.

[0042] DPB: Decoded Picture Buffer, which exists in the VVC codec and can store and manage decoded images for outputting images or providing reference frames for subsequently decoded video frames. When managing decoded images, DPB will add some tags to the decoded images to identify the attributes of the decoded images, such as the index number of the decoded image and the number of the layer to which it belongs, whether it is used for short-term reference or long-term reference, whether it is no longer used, etc.

[0043] Slice: A data unit defined in traditional video coding and decoding methods. A picture usually contains at least one slice, and there are many types of slices, including intra slices, single-hypothesis inter slices (P slices) and double-hypothesis inter slices (B slices), which are coded and decoded using intra prediction, single-hypothesis inter prediction and double-hypothesis inter prediction methods respectively.

[0044] NAL packet: full name Network Abstract Layer units, is a video code stream encapsulation format. NAL packet consists of header data and payload data, where the header data contains information such as layer identification, type identification and time domain layer identification, which is used by the system layer to quickly identify NAL packets to distinguish the functions of different NAL packets and the data types they contain.

[0045] Picture header: A set of syntax elements that contains codec information common to all slices of a picture.

[0046] Image insertion: Add an image at a certain display sequence or display time in the video to increase the frame rate of the video.

[0047] Image generation: Using several given images to obtain another image.

[0048] This embodiment discloses a video decoding method based on interpolation, and its flow chart is as follows: Figure 1As shown, the implementation subject of the method can be a decoder. The decoding method is described in detail below.

[0049] Specifically, for the input bitstream, the interpolation frame identification information is obtained from the bitstream, and the identification information directly or indirectly includes the index information of the decoded image used to generate the inserted image, and the decoder can use the decoded image for the traditional or intelligent generation method to obtain the inserted image; the identification information also directly or indirectly includes the output time or presentation time of the obtained inserted image, and the decoder can output the inserted image according to the time information. Usually, the bitstream does not contain the encoded data of the inserted image, or only contains a part of the encoded data of the inserted image, and the complete pixel information of the inserted image cannot be decoded according to this part of the encoded data. The advantage of this is that part or all of the pixel information of the inserted image is generated by other decoded images, which can effectively reduce the data volume of the bitstream. When the bitstream contains the complete encoded data of an image, the decoder uses an existing decoding method to decode the image to obtain its decoded image, and the existing decoding method can be a decoding method of a standard specification such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, AVS series or AV1 series.

[0050] The above embodiments have summarized the process of the entire decoding method. The following will describe the details of each operation in detail with reference to specific examples.

[0051] In one embodiment, a video decoding method based on interpolation obtains explicit interpolation identification information from the syntax elements of the bitstream. Specifically, the decoder obtains an image header from the bitstream, and the image header includes a generation insertion image identification information, which is used to identify whether the pixel information of the image corresponding to the image header needs to be generated by a generation method and inserted in the output video. When the generation insertion image identification information indicates that the current image needs to be generated and inserted into the output video, there is no strip data of the image after the image header in the bitstream, and the image needs to be generated from other decoded images; when the generation insertion image identification information indicates that the current image does not need to be generated, there should be strip data of the image after the image header in the bitstream, and the decoded image is decoded using the existing decoding method. The image header also includes display information corresponding to the inserted image, such as whether to output the display or the display number, etc. The decoder can decide the output of the inserted image based on this display information.

[0052] The above-mentioned embodiment may be further varied in a possible implementation scheme. Specifically, the generated inserted image identification information may also indicate that part of the corresponding image area needs to be generated by other decoded images, while the encoded data of other part of the area is carried in the strip data after the image header of the corresponding image. This implementation scheme can use the generation method to obtain pixel information of areas with simple content when there are areas of different complexity in the video image content, and use the encoding and decoding method to obtain pixel information of areas with complex content. This can avoid the problem that the existing image generation method is not sufficient to cope with the complex content in the image, and can improve the fidelity of the complex content in the video image as much as possible while reducing the code stream data.

[0053] In another embodiment, the generated inserted image identification information may also indicate the generation method used to generate the inserted image, and accordingly, the decoder selects one of the multiple generation methods it can support to generate the inserted image. These generation methods may include, but are not limited to: a method of averaging pixel information of multiple decoded images, a method of weighting pixel information of multiple decoded images, or an intelligent generation method based on a neural network, such as a real-time intermediate optical flow estimation method for video frame interpolation (RIFE).

[0054] In another embodiment, generating the identification information of the inserted image may further include identification information of a decoded image used to generate the inserted image, and the decoder may obtain at least one decoded image used to generate the inserted image from the decoded image cache according to the identification information of the decoded image, and generate the inserted image using the generation method. The identification information of the decoded image may be explicit information, such as the display number or decoding number of the decoded image; the identification information of the decoded image may also be implicit information, such as the difference between the display number of the decoded image and the display number of the inserted image, or a reference image queue recorded in the image header of the inserted image, that is, the decoder may obtain an available decoded image from the reference image queue for generating the inserted image.

[0055] In another embodiment, the identification information of the decoded image used to generate the inserted image included in the generated inserted image identification information may also be a reference image queue recorded in an image parameter set or a sequence parameter set referenced by the inserted image. In another embodiment, when different image areas or different strips in the inserted image can use different decoded images to generate pixels of the inserted image, the identification information of the decoded image may also be a reference image queue recorded in a strip header of the inserted image. The advantage of using a reference image queue is that the decoded image cache management based on the reference image queue information in the existing video encoding method can be reused without introducing additional management operations; at the same time, it can also ensure that the decoded image required to generate the inserted image exists in the decoded image cache. The reference image queue usually contains multiple reference images and records the index numbers of the reference images. Taking into account that, in general, the generation of an inserted image does not necessarily require the use of all reference images, in one embodiment, the first several reference images arranged in chronological order in the reference image queue, such as the first two reference images, may be selected as decoded images required for generating the inserted image; in another embodiment, the index of the decoded image required for generating the inserted image in the reference image queue may be placed in the image header or slice header of the inserted image to limit the decoded images available for the inserted image.

[0056] The identification information used in the above implementation should appear in the syntax elements in the bitstream. Without loss of generality, in combination with the syntax structure of H.266 / VVC, an example of a syntax element of the identification information used above can be shown in the following table.

[0057]

[0058] The syntax elements in italics are the syntax elements defined in the present invention, and the other syntax elements are the existing syntax elements in the syntax structure of H.266 / VVC, which will not be repeated here. The meanings of the syntax elements defined in the present invention are:

[0059] 1) ph_interpolate_flag is an enable flag in the generated inserted image identification information, indicating whether the corresponding image needs to be generated and inserted into the output video;

[0060] 2) ph_interpolate_method_id is the generation method index in the generated insert image identification information, indicating the generation method used to generate the insert image;

[0061] 3) ph_num_dec_pic and ph_delta_poc_dec_pic[i] are identification information of a decoded image used to generate an inserted image in the generated inserted image identification information, indicating the number of decoded images and the difference between the display number of the i-th decoded image and the display number of the inserted image, respectively.

[0062] 4) The existing syntax elements ph_pic_order_cnt_lsb and ph_poc_msb_cycle_val (if present) of H.266 / VVC indicate the display number of the inserted picture.

[0063] In another implementation, the identification information in the above-mentioned image header may exist in Supplemental Information (SEI), and the SEI information is placed after the code stream of the image that is most adjacent to the inserted image in display order.

[0064] In another embodiment, the generated inserted image identification information does not exist in the image header of the inserted image, but exists in the syntax element of a decoded image used to generate the inserted image. After parsing the generated inserted image identification information in the syntax element of the decoded image, the decoder can obtain the decoded image used to generate the inserted image according to the generated inserted image identification information, and use a given generation method to generate the inserted image, and insert the inserted image into the output video according to the display number of the inserted image indicated in the generated inserted image identification information. At this time, the generated inserted image identification information must explicitly record the index information of the decoded image used to generate the inserted image, the generation method information, and the display information of the inserted image. Generally, in order to generate the inserted image in time, the generated inserted image identification information can exist in a decoded image with the latest decoding order among all decoded images used to generate the inserted image. Of course, this method does not exclude the situation where the generated inserted image identification information exists in other decoded images. In a possible implementation, since there may be multiple inserted images before the decoded image, the decoded image needs to be used to generate multiple inserted images, and the generated inserted image identification information in the decoded image needs to include the number information of the inserted images, or include the display number information of all the inserted images.

[0065] In one implementation, considering the inconsistency between the encoding order and the display order in video encoding and decoding, it is necessary to constrain the range of decoded images that the decoder can use when generating an inserted image to avoid excessive decoding and display delays.

[0066] In one embodiment, the interpolation-based video decoding method obtains implicit interpolation identification information from the syntax elements of the code stream, for example, the decoder determines whether it is necessary to generate an inserted image based on the continuity of the display number of the decoded image in the video. Specifically, the decoder decodes the images in the code stream according to the code stream order, and outputs the decoded images to the output video according to the display number of the images. Before outputting the decoded images each time, the decoder compares the display number of the current decoded image with the display number of the most adjacent previous decoded image in the display order. If the two display numbers are not continuous, several inserted images are inserted before the current decoded image to make the display numbers continuous. The decoded images used to generate the inserted image are the current decoded image and the most adjacent previous decoded image in the display order.

[0067] In one embodiment, the interpolation identification information used to generate an inserted image can be recorded in the image-level parameter set corresponding to the inserted image, or can be referenced from the sequence-level parameter set corresponding to the sequence to which the inserted image belongs. Multiple sets of different interpolation identification information can be recorded in the sequence-level parameter set to provide different inserted images in the sequence with reference as interpolation identification information. The advantage of this is that when most of the inserted images in the sequence use the same or similar interpolation identification information, these inserted images can be referenced and shared from the sequence-level parameter set without having to be recorded in the image parameter set corresponding to each inserted image, which can save the amount of data occupied by each inserted image. When a part of the inserted images need to use different interpolation identification information, on the one hand, these inserted images can reference other sets of interpolation identification information recorded in the sequence-level parameter set, and on the other hand, these inserted images can also record the interpolation identification information they need in their corresponding image-level parameter set. Such a design also maintains the flexibility of generating inserted images. A specific grammatical structure and operation example are described as follows: 1) Multiple sets of interpolation identification information recorded in the sequence-level parameter set

[0068]

[0069] Among them, sps_num_temporal_restoration_data represents the number of groups of interpolation frame identification information, and temporal_restoration_data(i) represents the i-th group of interpolation frame identification information.

[0070] 2) Interpolation identification information recorded in the image-level parameter set

[0071]

[0072] Among them, prd_picture_order_cnt_lsb is used to directly or indirectly indicate the number of the picture corresponding to the picture-level parameter set; prd_decoded_pic_flag indicates whether the picture is an inserted picture or a decoded picture that can be used to generate an inserted picture. If it is a decoded picture, there is no need to use the interpolation identification information because the picture is decoded from the code stream. For an inserted image, if the value of sps_num_temporal_restoration_data is equal to 0, there is no interpolation identification information recorded in the sequence-level parameter set that can be used for reference. At this time, the inserted image can only record the interpolation identification information in the picture-level parameter set; if the value of sps_num_temporal_restoration_data is greater than 0, the picture-level parameter set can use prd_use_srd_temporal_restoration_data_flag to indicate whether its interpolation identification information is referenced from the sequence-level parameter set or recorded in itself, prd_srd_temporal_restoration_data_idx indicates the number of the interpolation identification information referenced from the sequence-level parameter set, and temporal_restoration_data (sps_num_temporal_restoration_data) is the interpolation identification information recorded in the picture-level parameter set, and the number can be set to sps_num_temporal_restoration_data. When sps_num_temporal_restoration_data is equal to 1 and prd_use_srd_temporal_restoration_data_flag is equal to 1, only one set of interpolation identification information is recorded in the sequence level parameter set. At this time, the picture level parameter set does not need to use prd_srd_temporal_restoration_data_idx, and this set of interpolation identification information can be used by default.

[0073] 3) Insertion frame identification information

[0074]

[0075] Among them, trdIdx represents the number of the interpolation identification information; delta_poc_of_backword_ref_pic_minus1[trdIdx] represents the difference between the number of the decoded image used to generate the inserted image recorded in the trdIdx-th group of interpolation identification information and the number of the inserted image, and the display order of the decoded image is before the inserted image; delta_poc_of_forward_ref_pic_minus1[trdIdx] represents the difference between the number of the decoded image used to generate the inserted image recorded in the trdIdx-th group of interpolation identification information and the number of the inserted image, and the display order of the decoded image is after the inserted image.

[0076] In another implementation, the interpolation frame identification information may also be recorded more flexibly in the following form:

[0077]

[0078] Among them, num_ref_pic_minus1[trdIdx] represents the number of decoded images used to generate the inserted image recorded in the trdIdx-th group of interpolation identification information, abs_delta_poc_of_ref_pic_minus1[trdIdx][i] represents the absolute value of the difference between the number of the i-th decoded image in the trdIdx-th group of interpolation identification information and the number of the inserted image, and sign_of_delta_poc[trdIdx][i] represents the positive or negative sign of the difference. A positive difference indicates that the decoded image is after the display order of the inserted image and is recorded as a backward reference image. A negative difference indicates that the decoded image is before the display order of the inserted image and is recorded as a forward reference image.

[0079] According to the above information, the decoding operation can construct a decoded image list for each inserted image, in which the decoded images can be used to generate the inserted image. When the decoded image list contains both forward reference images and backward reference images, the operation of generating the inserted image can use the interpolation method, that is, using multiple images with a time sequence to generate a certain image at the middle moment of the time period corresponding to these images; when the decoded image list contains only forward reference images or only backward reference images, the operation of generating the inserted image can use the extrapolation method, that is, using multiple images with a time sequence to generate a certain image at a moment outside the time period corresponding to these images; when the decoded image list contains only one decoded image, the inserted image can be generated by copying or predicting.

[0080] This embodiment also discloses a video encoding method based on interpolation, and its flow chart is as follows: Figure 2As shown, the implementation subject of the method can be an encoder. For the input video, the encoder extracts a part of the image from the video according to a certain rule to encode and generate a code stream. The extraction rule can be extraction at a fixed interval, or extraction according to the richness of the image content after analyzing the video content. In one embodiment, the encoder does not completely skip the encoding of the unextracted image, but puts the image header information of the unextracted image into the code stream, and puts the generated inserted image identification information therein, which contains the index information of the decoded image used to generate the inserted image, the generation method information and the display information of the inserted image, and then the encoder skips the encoding of the strip data of the image, and performs the encoding of the next image, and finally obtains the code stream. In another embodiment, the encoder completely skips the encoding of the unextracted image, and puts the generated inserted image identification information into the syntax element of the extracted image used to generate the inserted image, and the information contains the number or display number of all inserted images used to generate the extracted image.

[0081] This embodiment discloses a decoding device, which includes a processor and a memory, and is used to execute the decoding method disclosed in the present invention.

[0082] This embodiment discloses a coding device, which includes a processor and a memory, and is used to execute the coding method disclosed in the present invention.

[0083] The above embodiments are only used to help understand the method and core idea of ​​the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A video decoding method based on interpolation, characterized in that: For the input video stream, perform the following operations: Acquire coded data of at least one first image from the code stream, and decode to obtain at least one first decoded image and an output time of the first decoded image; Acquire interpolation identification information corresponding to the second image from the bitstream, where the interpolation identification information includes index information of at least one first decoded image used to generate the second image, and also includes an output time of the second image; Generate a second image using the at least one first decoded image according to the interpolation frame identification information; According to the output time of the second image and the output time of the first decoded image, a video consisting of the first decoded image and the second image is output in the order of the output time.

2. The method according to claim 1, characterized in that The interpolation frame identification information is included in the image header corresponding to the second image in the code stream, and the code stream does not include the strip data of the second image.

3. The method according to claim 1, characterized in that The interpolation identification information is included in the encoded data of a first image in the code stream, and the first decoded image obtained by decoding the first image is one of the at least one first decoded images used to generate the second image.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire interpolation method information corresponding to the second image from the code stream, where the interpolation method information indicates a generation method for generating the second image; According to the interpolation identification information corresponding to the second image, the second image is generated using the at least one first decoded image and the generation method indicated by the interpolation method information.

5. A video encoding method based on interpolation, characterized in that: For the input video, do the following: Extract at least one image from the input video and skip other images; For the extracted image, use an existing encoding method to encode the extracted image to obtain a first code stream; For a skipped image, inserting interpolation identification information corresponding to the skipped image into the second bitstream, the interpolation identification information including index information of a decoded image of at least one extracted image, the decoded image being used to generate the skipped image corresponding to the interpolation identification information, the interpolation identification information also including an output time of the skipped image; The first bit stream and the second bit stream are combined to obtain an output bit stream.

6. The method according to claim 5, characterized in that The method further comprises: Putting a picture header corresponding to the skipped picture into the second bitstream; Putting the interpolation frame identification information into the image header; Encoding of the slice data of the skipped picture is skipped.

7. The method according to claim 5, characterized in that The method further comprises: The interpolation identification information is placed in the coded data of an extracted image in the first code stream, and the decoded image of the extracted image is one of the at least one decoded image used to generate the skipped image.

8. A video decoding device based on interpolation, comprising: processor; A memory for storing code streams and image data to be decoded; as well as One or more programs are used to implement the interpolation-based video decoding method as described in any one of claims 1-4.

9. A video encoding device based on interpolation, comprising: processor; A memory for storing code streams and images to be encoded; as well as One or more programs are used to implement the interpolation-based video encoding method as described in any one of claims 5-7.

Images