Video coding and decoding method and device based on frame insertion

Abstract

The invention discloses a video coding and decoding method and device based on frame insertion, and the method comprises the steps: obtaining frame insertion identification information in a decoding process, generating a frame insertion image through employing a decoded image according to the frame insertion identification information, and outputting the frame insertion image at a target moment. According to the method, the number of images needing to be processed during coding can be reduced, the data volume consumed by video coding is reduced while high-quality frame insertion images are obtained, and the coding efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of video encoding and decoding technology, and more specifically, to a video encoding and decoding method and apparatus based on frame interpolation. Background Technology

[0002] Existing video codec technologies, such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, AVS series, and AV1 series, typically cannot generate interpolated frames to increase the frame rate during video decoding. Instead, they can only skip some images during decoding to reduce the frame rate. Although existing video codec technologies delegate frame interpolation to post-processing, the effects of post-processing frame interpolation are not considered during encoding, and post-processing is unaware of which images were skipped during encoding. This means that post-processing frame interpolation cannot effectively restore the original video's frame rate and quality. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a video encoding and decoding method and apparatus based on frame interpolation. During the decoding process, this method acquires frame interpolation identification information, generates interpolated frames using the already decoded images based on the identification information, and outputs the interpolated frames at the target time. This method reduces the number of images that need to be processed during encoding, obtains high-quality interpolated frames, reduces the amount of data consumed in encoding the video, and improves encoding efficiency.

[0004] Therefore, the first objective of this invention is to propose a video decoding method based on frame interpolation, which performs the following operations on the bitstream:

[0005] Reconstructed data, which includes frame interpolation identification information, is obtained from the bitstream.

[0006] Obtain at least one decoded image from the bitstream;

[0007] Based on the interpolation information in the reconstructed data, generate interpolated images using at least one decoded image;

[0008] The decoded image and the interpolated image are combined to obtain the decoded video.

[0009] Furthermore, the method also includes:

[0010] Obtain the end identifier from the reconstructed data. This end identifier describes the number of interpolated frames that need to be generated at the end of the video.

[0011] Based on the end identifier, at least one decoded image is used to generate the required number of interpolated images.

[0012] Furthermore, the method also includes:

[0013] Obtain the number of sequence-level interpolated images from the reconstructed data;

[0014] Based on the number of sequence-level interpolated images, an interpolated image matching the stated number is generated before each decoded image.

[0015] Furthermore, the method also includes:

[0016] Obtain the target frame rate of the decoded video from the reconstructed data;

[0017] The decoded video is obtained by combining the decoded image and the interpolated image according to the target frame rate.

[0018] Furthermore, the method also includes:

[0019] Image cache management information is obtained from the bitstream. This management information is used to indicate the images that should be stored in the image cache, which are used to generate interpolated images or are waiting to be displayed.

[0020] Based on the image cache management information, images other than those that should be stored in the image cache are marked as no longer being referenced. These no longer-referenced images can be deleted, replaced, or updated.

[0021] The second objective of this invention is to propose a video coding method based on frame interpolation, which performs the following operations on the input video:

[0022] Extract at least one image from the input video and skip the others;

[0023] For the extracted image, the extracted image is encoded using an existing encoding method to obtain a first bitstream;

[0024] For the skipped image, the frame insertion identification information corresponding to the skipped image is put into the second bitstream. The frame insertion identification information includes the index information of at least one decoded image of the extracted image. The decoded image is used to generate the skipped image corresponding to the frame insertion identification information. The frame insertion identification information also includes the output time of the skipped image.

[0025] The first and second bitstreams are merged to obtain the output bitstream.

[0026] A third objective of this invention is to provide a video decoding apparatus based on frame interpolation, comprising:

[0027] processor;

[0028] Memory for storing bitstream and image data to be decoded; and

[0029] One or more programs are used to perform the frame-interpolation-based video decoding method as described in the first objective above.

[0030] The fourth objective of this invention is to provide a frame interpolation-based video encoding apparatus, comprising:

[0031] processor;

[0032] Memory for storing bitstreams and images to be encoded; and

[0033] One or more programs are used to perform the frame interpolation-based video coding method as described in the second objective above.

[0034] A fifth objective of the present invention is to provide a storage device comprising a bitstream obtained using the frame-interpolation-based video encoding method described in the second objective above, and which can be decoded using the frame-interpolation-based video decoding method described in the first objective above.

[0035] The beneficial effects of this invention are as follows:

[0036] On the one hand, when processing video, the encoding end can extract a portion of the images from the video for encoding, which can reduce the amount of data consumed in encoding the video. At the same time, a portion of the saved data can be used to improve the quality of the encoded images, which can effectively improve encoding efficiency and reduce the computational complexity of the encoding end.

[0037] On the other hand, when processing video, the decoding end can generate high-quality interpolated images based on the high-quality images that have already been decoded. This can effectively improve the frame rate and reconstruction quality of the reconstructed video, and also ensure that the number of images in the video output by the decoding end is consistent with the number of images in the original video of the encoding end, avoiding mismatch between the encoding and decoding ends. In addition, since frame extraction at the encoding end and frame interpolation at the decoding end do not affect the inherent encoding and decoding operations, the method proposed in this invention also has compatibility with different video encoding and decoding methods. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of a video decoding method based on frame interpolation according to an embodiment of the present invention;

[0040] Figure 2 This is a flowchart of a video encoding method based on frame interpolation according to an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0042] Definition of noun:

[0043] VVC: Versatile Video Coding, also known as H.266, is a video coding standard that can compress input video into a bitstream. This bitstream can contain multiple sub-bitstreams, and there can be reference relationships between the sub-bitstreams.

[0044] DPB: Short for Decoded Picture Buffer, this DPB exists in the VVC codec and stores and manages decoded images for output or to provide reference frames for subsequent decoded video frames. When managing decoded images, the DPB attaches tags to them to identify their attributes, such as the image's index number and layer number, whether it is used for short-term or long-term reference, and whether it is no longer in use.

[0045] A slice is a data unit defined in traditional video encoding and decoding methods. An image typically contains at least one slice, and there are several types of slices, including intra slices, single-hypothesis inter-frame slices (P slices), and double-hypothesis inter-frame slices (B slices), which are encoded and decoded using intra-prediction, single-hypothesis inter-frame prediction, and double-hypothesis inter-frame prediction methods, respectively.

[0046] NAL packets, short for Network Abstract Layer units, are a video stream encapsulation format. NAL packets consist of a header and payload. The header contains information such as layer identifiers, type identifiers, and temporal layer identifiers, allowing the system layer to quickly identify NAL packets and distinguish their functions and data types.

[0047] Image header: A set of syntax elements that contains encoding and decoding information shared by all stripes of an image.

[0048] Image insertion: Adding an image to a video at a specific display order or moment to increase the video's frame rate.

[0049] Image generation: Use a given set of images to generate another image.

[0050] This embodiment discloses a video decoding method based on frame interpolation, the flowchart of which is as follows: Figure 1As shown, the implementer of this method can be a decoder. The decoding method is described in detail below.

[0051] Specifically, for the input bitstream, interpolation identification information is obtained from the bitstream. This identification information directly or indirectly contains the index information of the decoded image used to generate the interpolated image. The decoder can use the decoded image to obtain the interpolated image using traditional or intelligent generation methods. This identification information also directly or indirectly contains the output time or presentation time of the obtained interpolated image. The decoder can output the interpolated image based on the time information. Typically, the bitstream does not contain the encoded data of the interpolated image, or only contains a portion of the encoded data of the interpolated image. Based on this portion of encoded data, the complete pixel information of the interpolated image cannot be decoded. The advantage of this is that some or all of the pixel information of the interpolated image can be generated from other decoded images, which can effectively reduce the data volume of the bitstream. When the bitstream contains the complete encoded data of an image, the decoder uses an existing decoding method to decode the image to obtain its decoded image. The existing decoding method can be a decoding method of standard specifications such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, AVS series, or AV1 series.

[0052] The preceding embodiments have provided an overview of the entire decoding method process. The following will describe the details of each operation in conjunction with specific examples.

[0053] In one embodiment, the frame interpolation-based video decoding method obtains explicit frame interpolation identification information from the syntax elements of the bitstream. Specifically, the decoder obtains an image header from the bitstream, which contains a frame interpolation image generation identification information to indicate whether the pixel information of the image corresponding to the image header needs to be generated and inserted into the output video. When the frame interpolation image generation identification information indicates that the current image needs to be generated and inserted into the output video, the image header in the bitstream does not contain the strip data of that image, and the image needs to be generated from other decoded images; when the frame interpolation image generation identification information indicates that the current image does not need to be generated, the image header in the bitstream should contain the strip data of that image, and the decoded image is obtained using an existing decoding method. The image header also contains display information for the corresponding frame interpolation image, such as whether to output display or display number, etc., and the decoder can determine the output of the frame interpolation image based on this display information.

[0054] In one possible implementation, the above embodiments can be further modified. Specifically, the generated interpolated image identifier information can further indicate that a portion of the corresponding image needs to be generated from other decoded images, while the encoded data of other portions is carried in the strip data following the image header of the corresponding image. This implementation method can obtain pixel information of simpler regions using the generation method and obtain pixel information of more complex regions using the encoding and decoding method when the video image content contains regions of varying complexity. This avoids the problem that existing image generation methods are insufficient to handle complex content in images, and improves the fidelity of complex content in video images as much as possible while reducing bitstream data.

[0055] In another implementation, the generated interpolated image identification information may also indicate the generation method used to generate the interpolated image. Accordingly, the decoder selects one of the multiple generation methods it supports for generating the interpolated image. These generation methods may include, but are not limited to: methods that average pixel information from multiple decoded images, methods that weight pixel information from multiple decoded images, or intelligent generation methods based on neural networks, such as Real-Time Intermediate Flow Estimation for Video Frame Interpolation (RIFE).

[0056] In another implementation, the interpolation image identification information may further include identification information of decoded images used to generate the interpolation image. The decoder can retrieve at least one decoded image for generating the interpolation image from the decoded image buffer based on the identification information of the decoded image, and generate the interpolation image using the generation method. The identification information of the decoded image can be explicit information, such as the display number or decoding number of the decoded image; it can also be implicit information, such as the difference between the display number of the decoded image and the display number of the interpolation image, or a reference image queue recorded in the image header of the interpolation image, i.e., the decoder can retrieve available decoded images from the reference image queue for the generation of the interpolation image.

[0057] In another implementation, the identifier information of the decoded image used to generate the interpolated frame image, included in the generated interpolated frame image identifier information, can also be a reference image queue recorded in the image parameter set or sequence parameter set referenced by the interpolated frame image. In another implementation, when different image regions or different stripes in the interpolated frame image can use different decoded images for the pixels of the interpolated frame image, the identifier information of the decoded image can also be a reference image queue recorded in the strip header of the interpolated frame image. The advantage of using a reference image queue is that it allows reuse of the decoded image buffer management based on reference image queue information in existing video coding methods without introducing additional management operations; simultaneously, it ensures that the decoded images required to generate the interpolated frame image exist in the decoded image buffer. The reference image queue typically contains multiple reference images and records the index numbers of the reference images. Considering that the generation of interpolated images does not necessarily require the use of all reference images, in one implementation, the first few reference images in the reference image queue arranged in chronological order, such as the first two reference images, can be selected as the decoded images required to generate the interpolated image; in another implementation, the index of the decoded image required to generate the interpolated image in the reference image queue can be placed in the image header or strip header of the interpolated image to limit the decoded images available for the interpolated image.

[0058] The identification information used in the above implementation should appear in the syntax elements of the bitstream. Without loss of generality, in conjunction with the syntax structure of H.266 / VVC, an example of a syntax element for the identification information used above can be shown in the following table.

[0059]

[0060] The italicized syntax elements are those defined in this invention; the other syntax elements are existing syntax elements in the H.266 / VVC syntax structure and will not be elaborated further here. The meanings of the syntax elements defined in this invention are as follows:

[0061] 1) ph_interpolate_flag is the enable flag in the frame interpolation image identifier information, indicating whether the corresponding image needs to be generated and inserted into the output video;

[0062] 2) ph_interpolate_method_id is the generation method index in the generated interpolated image identifier information, indicating the generation method used to generate the interpolated image;

[0063] 3) ph_num_dec_pic and ph_delta_poc_dec_pic[i] are the identifiers of the decoded images used to generate the interpolated images in the interpolated image identifier information, respectively indicating the number of decoded images and the difference between the display number of the i-th decoded image and the display number of the interpolated image.

[0064] 4) The existing syntax elements ph_pic_order_cnt_lsb and ph_poc_msb_cycle_val (if they exist) in H.266 / VVC indicate the display number of the interpolated frame image.

[0065] In another implementation, the identification information in the image header may be present in the supplemental information (SEI), which is placed after the bitstream of the image closest to the interpolated image in the display order.

[0066] In another implementation, the interpolation image identifier information is not present in the image header of the interpolated image, but rather in the syntax element of a decoded image used to generate the interpolated image. After parsing the interpolation image identifier information in the syntax element of the decoded image, the decoder can obtain the decoded image used to generate the interpolated image based on the interpolation image identifier information, generate the interpolated image using a given generation method, and insert the interpolated image into the output video according to the display number of the interpolated image indicated in the interpolation image identifier information. At this time, the interpolation image identifier information must explicitly record the index information of the decoded image used to generate the interpolated image, the generation method information, and the display information of the interpolated image. Typically, to ensure timely generation of the interpolated image, the interpolation image identifier information can exist in the last decoded image among all the decoded images used to generate the interpolated image. However, this method does not exclude the possibility that the interpolation image identifier information exists in other decoded images. In one possible implementation, since there may be multiple interpolated frames before the decoded image, the decoded image needs to be used to generate multiple interpolated frames. The interpolated frame generation identification information in the decoded image needs to include the number of interpolated frames or the display number information of all interpolated frames.

[0067] In one implementation, considering the inconsistency between the encoding order and the display order in video encoding and decoding, it is necessary to constrain the range of decoded images that the decoder can use when generating interpolated frames in order to avoid excessive decoding and display delays.

[0068] In one embodiment, the frame-interpolating video decoding method obtains implicit frame-interpolating identification information from the syntax elements of the bitstream. For example, the decoder determines whether to generate interpolated frames based on the continuity of the display numbers of the decoded images in the video. Specifically, the decoder decodes the images in the bitstream in sequence and outputs the decoded images to the output video according to their display numbers. Before each output of a decoded image, the decoder compares the display number of the current decoded image with the display number of the nearest preceding decoded image in the display order. If the two display numbers are not consecutive, several interpolated frames are inserted before the current decoded image to make the display numbers consecutive. The decoded images used to generate the interpolated frames are the current decoded image and the nearest preceding decoded image in the display order.

[0069] In one embodiment, the interpolation identification information used to generate an interpolated image can be recorded in the image-level parameter set corresponding to the interpolated image, or it can be referenced from the sequence-level parameter set corresponding to the sequence to which the interpolated image belongs. The sequence-level parameter set can record multiple sets of different interpolation identification information for different interpolated images in the sequence to reference as interpolation identification information. The advantage of this is that when most interpolated images in the sequence use the same or similar interpolation identification information, these interpolated images can be referenced and shared from the sequence-level parameter set, without needing to record it in the image parameter set corresponding to each interpolated image, thus saving the amount of data occupied by each interpolated image. When some interpolated images need to use different interpolation identification information, on the one hand, these interpolated images can reference other sets of interpolation identification information recorded in the sequence-level parameter set; on the other hand, these interpolated images can also record their own required interpolation identification information in their own corresponding image-level parameter set. This design also maintains the flexibility of generating interpolated images. A specific syntax structure and operation example are described below:

[0070] 1) Multiple sets of interpolation identifier information recorded in the sequence-level parameter set

[0071]

[0072] Where sps_num_temporal_restoration_data represents the number of groups of interpolation identification information, and temporal_restoration_data(i) represents the i-th group of interpolation identification information.

[0073] 2) Frame interpolation identification information recorded in the image-level parameter set

[0074]

[0075] Among them, prd_picture_order_cnt_lsb is used to directly or indirectly represent the image number corresponding to the image level parameter set; prd_decoded_pic_flag indicates whether the image is an interpolated image or a decoded image that can be used to generate an interpolated image. If it is a decoded image, the interpolation identification information is not needed because the image is decoded from the bitstream. For interpolated images, if the value of `sps_num_temporal_restoration_data` is equal to 0, then there is no interpolation identifier information recorded in the sequence-level parameter set that can be used for reference. In this case, the interpolated image can only record the interpolation identifier information in the image-level parameter set. If the value of `sps_num_temporal_restoration_data` is greater than 0, then the image-level parameter set can use `prd_use_srd_temporal_restoration_data_flag` to indicate whether its interpolation identifier information is referenced from the sequence-level parameter set or recorded in itself. `prd_srd_temporal_restoration_data_idx` represents the number of the interpolation identifier information referenced from the sequence-level parameter set, and `temporal_restoration_data( sps_num_temporal_restoration_data )` is the interpolation identifier information recorded in the image-level parameter set, and the number can be set to `sps_num_temporal_restoration_data`. When sps_num_temporal_restoration_data equals 1 and prd_use_srd_temporal_restoration_data_flag equals 1, only one set of interpolation identifier information is recorded in the sequence-level parameter set. At this time, the image-level parameter set does not need to use prd_srd_temporal_restoration_data_idx, but can use this one set of interpolation identifier information by default.

[0076] 3) Frame interpolation identification information

[0077]

[0078] Wherein, trdIdx represents the interpolation identification information number; delta_poc_of_backword_ref_pic_minus1[trdIdx] represents the difference between the number of the decoded image used to generate the interpolated image recorded in the trdIdx group of interpolation identification information and the number of the interpolated image, and the display order of the decoded image is before the interpolated image; delta_poc_of_forward_ref_pic_minus1[trdIdx] represents the difference between the number of the decoded image used to generate the interpolated image recorded in the trdIdx group of interpolation identification information and the number of the interpolated image, and the display order of the decoded image is after the interpolated image.

[0079] In another implementation, the frame interpolation identifier information can also be recorded more flexibly in the following form:

[0080]

[0081] Wherein, num_ref_pic_minus1[trdIdx] represents the number of decoded images used to generate interpolated images recorded in the trdIdx group of interpolation identification information, abs_delta_poc_of_ref_pic_minus1[trdIdx][i] represents the absolute value of the difference between the number of the i-th decoded image in the trdIdx group of interpolation identification information and the number of the interpolated image, and sign_of_delta_poc[trdIdx][i] represents the sign of the difference. A positive difference indicates that the decoded image is after the display order of the interpolated image and is recorded as a backward reference image, and a negative difference indicates that the decoded image is before the display order of the interpolated image and is recorded as a forward reference image.

[0082] In another embodiment, when the generation of the interpolated image uses decoded images before and after the display order as reference images more often, two queues, forward and backward, can also be used to describe the reference image information for generating the interpolated image, as shown in the table below.

[0083]

[0084] Where num_pos_ref_pic_minus1[trdIdx] and num_neg_ref_pic_minus1[trdIdx] represent the number of backward and forward decoded images recorded in the trdIdx group of interpolation identification information used to generate the interpolated image, respectively. abs_delta_poc_of_pos_ref_pic_minus1[trdIdx][i] and abs_delta_poc_of_neg_ref_pic_minus1[trdIdx][i] represent the absolute values ​​of the differences between the numbers of the i-th backward and i-th forward decoded images in the trdIdx group of interpolation identification information and the number of the interpolated image. The advantage of this description is that it can save the bit of adding a positive or negative sign for each difference in the number of decoded images, and at the same time, it is compatible with application scenarios where the decoder uses extrapolation to generate interpolated images.

[0085] Based on the above information, the decoding operation can construct a list of decoded images for each interpolated image, and these decoded images can be used to generate the interpolated image. When the list of decoded images contains both a forward reference image and a backward reference image, the operation to generate the interpolated image can use interpolation, that is, using multiple images in time sequence to generate an image at a certain time in the middle of the corresponding time period. When the list of decoded images contains only a forward reference image or only a backward reference image, the operation to generate the interpolated image can use extrapolation, that is, using multiple images in time sequence to generate an image at a time outside the corresponding time period. When the list of decoded images contains only one decoded image, the interpolated image can be generated by copying or prediction.

[0086] In one embodiment, generating interpolated images based on decoded images requires effective management of the image cache. This avoids storing too many useless images in the cache and also prevents useful images from being accidentally deleted and unusable for subsequent interpolated image generation. Existing video encoding / decoding methods also have a set of management operations for decoded image caches, but these are overly complex. Directly applying them to the image cache management in this method would introduce additional operations. This is because existing video encoding / decoding methods use a dual-queue reference image queue, with each queue containing both forward and backward reference images. This method is highly flexible but complex. Since generating interpolated images requires a certain degree of similarity in content and motion regularity of the reference images, using reference images that are too far apart in time or too numerous may not necessarily produce good generation results. Therefore, this method uses a simpler operation and syntax to manage the image cache. It deletes or frees up storage space by describing images in the image cache that do not need to be stored. An example syntax is shown in the table below. Obviously, in many other implementations, images that must be stored in the image cache can also be directly described, allowing the decoder to delete images that are not described.

[0087]

[0088] In another embodiment, since both the image buffer management information and the frame interpolation identification information are needed for each image, and the image buffer management information also has a certain regularity, the implementation method can merge the image buffer management information and the frame interpolation identification information into a single syntax structure. This allows the image buffer management information to be stored in sequence-level information and referenced by each decoded image or interpolated image, or it can be specifically set in image-level information. A syntax example is shown in the table below. In this case, the decoded image also needs to use this information to manage the image buffer.

[0089]

[0090] Based on the above information, the decoding method can manage the image cache before or after decoding or generating each image. First, it parses the image cache management information in the bitstream to obtain the number of the image to be removed. This number can be calculated based on the difference between the number of the current image to be decoded / interpolated or the currently decoded / interpolated image and the number of the removable image described in the management information. Then, the decoding method iterates through each decoded / interpolated image in the image cache, deleting images whose numbers match the numbers of the removable images. In one application, to avoid additional data operations, images considered removable in the image cache can be marked as "no longer for reference" or "can be overwritten," thus eliminating the need for data deletion. Newly decoded / generated images can then directly use the storage space of these removable images. When the image cache management information records the numbers of some removable images, but the image cache does not contain decoded / interpolated images with these numbers, the decoding method can assume that these removable images have been removed by methods other than decoding, and skip the deletion or marking of these images. In another embodiment, after the bitstream is clipped or reassembled, the image buffer may contain some decoded / interpolated images before the clipping point, whose numbers match the numbers of removable images described in the image buffer management information after the clipping point. This could lead to accidental deletion. One possible solution is for the decoding method to identify the clipping point and distinguish between the images before and after the clipping point, ensuring that the image buffer management information after the clipping point does not affect the decoded / interpolated images before the clipping point.

[0091] The above implementation method provides sufficient flexibility for frame interpolation-based video decoding methods, enabling the decoding method to support the use of any combination of reference images that can improve coding efficiency for the generation of interpolated images. It also allows the decoding method to effectively manage image buffers and avoid excessive buffering overhead. In some application scenarios, the reference images that the frame interpolation method can use are relatively fixed. This is because the filters or neural networks used for frame interpolation usually have certain requirements regarding the temporal, motion, or difference relationships between the reference images and the interpolated images in order to generate high-quality interpolated images. In such scenarios, overly flexible frame interpolation identification information can lead to a waste of bitstream data. Therefore, this invention also discloses a frame interpolation-based video decoding method with limited flexibility but low data consumption and simple operation.

[0092] Specifically, the decoding method decodes a currently decoded image and its timing information from the bitstream. This timing information can be an chronologically increasing number, a timestamp, or a time interval. Based on the timing information of the currently decoded image, the method infers the number of interpolated frames that can be generated using the decoded image, as well as their timing information. If necessary, at least one decoded image whose timing precedes the current decoded image can be obtained for generating the interpolated frames. Based on the inference results, interpolated frames are generated in a certain order and using a specific generation method, and corresponding timing information is assigned to them. The decoded image and the generated interpolated frames are then combined to obtain the decoded video.

[0093] Based on the above decoding method, more specific embodiments are provided below to further clarify the detailed operation and advantages of the decoding method. Other embodiments described in this invention can also be combined with this decoding method in more detailed ways, which will not be elaborated here.

[0094] In one embodiment, the decoding method defaults to using an interpolation generation method, which generates interpolated frames using at least two decoded images, one forward and one backward, in temporal order. In this case, due to the constraint of the temporal information of the forward and backward decoded images, the decoding method can infer the temporal information of the interpolated frames that can be generated between these decoded images. For example, if the interpolation generation method can generate an interpolated frame at an intermediate time point between the forward and backward images, then the decoding method, based on the temporal information t1 and t2 of the forward decoded image d1 and the backward decoded image d2, first generates an interpolated frame p1 at half the time point, with a temporal information of (t1+t2) / 2. Then, based on the forward decoded image and the interpolated frame p1, it generates an interpolated frame p2 at half the time point, with a temporal information of (t1+(t1+t2) / 2) / 2, and so on. However, in this case, the decoding method cannot determine the number of interpolated frames needed between the decoded images, because the temporal information can be infinitely subdivided, and the frame rate of the decoded video can be infinitely increased. Therefore, the decoding method needs to determine the number of interpolated frames based on additional information. In one implementation, when the timing information used by the decoding method is a number, such as Picture Order Count (POC) often used in video encoding and decoding, it can be agreed that the number of the interpolated frames must be an integer. In this case, the maximum number of interpolated frames between d1 and d2 is t2-t1-1. This method requires constraints on the numbering of the decoded images, requiring the numbering method to calculate and increment the number of the decoded images and the number of the interpolated frames together. This prevents the number of interpolated frames in the decoded video from being too many or too few, thus failing to achieve the target frame rate expected by the application. In another implementation, when the timing information used by the decoding method is a timestamp, the decoding method should obtain the target frame rate of the decoded video from the bitstream, or the application scenario should set the target frame rate of the decoded video required by the decoding method. In this case, the decoding method can calculate the number of interpolated frames to be generated between any two decoded images according to the target frame rate, generate the corresponding number of interpolated frames, and allocate timing information according to the target frame rate. This method requires constraining the timestamp information of the decoded image, which should indicate the actual display time of the decoded image in the decoded video. In another implementation of this method, the timing information of the decoded image can be calculated by describing its actual frame rate. The decoding method also obtains the actual frame rate of the decoded image from the bitstream and calculates the timestamp of the decoded image in conjunction with the Proof-of-Concept (POC) of the decoded image.

[0095] An example of a syntax for identifying the target frame rate in a bitstream is shown in the table below.

[0096]

[0097] Here, `vcm_num_units_in_tick` represents the unit time length occupied by each image in the decoded video, and this time length is in units of clock frequency `vcm_time_scale`. Furthermore, the method for identifying the actual frame rate of the decoded image in the bitstream is similar and will not be elaborated further.

[0098] In another implementation, the interpolation generation method is not used to generate interpolated frame images at half the time point, but can generate interpolated frame images at asymmetrical time points such as one-third or one-fifth. The advantage of this method is that it can use asymmetrical decoded images to generate interpolated frame images, enabling efficient encoding and decoding of videos with more diverse content.

[0099] In one implementation, the decoding method defaults to using an extrapolation generation method, which generates interpolated frames using one or more temporally preceding images. In this case, the decoding method can still infer the temporal information of the interpolated frames in the same way as when using an interpolation generation method. There are several different scenarios for how the decoding method infers the number of interpolated frames. In one implementation, the bitstream is already fully stored at the decoding end. The decoding method can obtain the encoded data of the temporally preceding images to be decoded in advance, decode it to obtain its temporal information, and thus infer the number of interpolated frames that need to be generated before that image. In another implementation, the bitstream is transmitted from the encoding end to the decoding end in real time. In this case, the decoding method cannot obtain the encoded data of the preceding images to be decoded in advance. It can first generate interpolated frames based on at least one temporally preceding decoded image or an interpolated frame, and assign temporal information to the interpolated frame. When the decoding method receives the encoded data of a subsequent image to be decoded, if the temporal information of the decoded image matches the temporal information of the interpolated frame, then the decoded image can be used to replace the interpolated frame. In another implementation, before generating the interpolated image, it is also possible to detect whether the encoded data of the decoded image has been received at the current moment within the time limit. This can skip the generation of the interpolated image and avoid wasting computing resources.

[0100] In another implementation, when the timing information of the decoded image cannot indicate its actual display time, timing information needs to be set for each decoded image. One approach is to allocate additional timing information to the decoded image that matches its actual display time. This timing information can be carried in the reconstructed data outside the encoded data of the decoded image. This reconstructed data is associated with the decoded image. The association can be achieved by binding the encoded data and reconstructed data of the image together in the bitstream, using the reconstructed data as extended data of the encoded data, or recording the actual number in the encoded data of the decoded image in the reconstructed data. In another implementation, the number of interpolated images to be generated before or after the decoded image can also be set for each decoded image. The decoding method obtains the number of interpolated frames that are timed before or after the decoded image from the encoded data or reconstructed data of the decoded image, then generates the interpolated frames, and infers the timing information of the interpolated frames based on the timing information of the decoded image. In one implementation, the decoding method obtains the number of interpolated frames that precede the decoded image from the data of the decoded image, and generates interpolated frames based on the decoded image and one or more other decoded images that may precede the decoded image. In another implementation, the decoding method obtains the number of interpolated frames that follow the decoded image from the data of the decoded image, and generates interpolated frames based on the decoded image and one or more other decoded images that may precede or follow the decoded image. This method can achieve the same function as an end marker at the end of the video. In one implementation, if the number of interpolated frames required before or after most decoded images is the same, but the number differs for a portion of decoded images, the decoding method obtains a common number of interpolated frames from the sequence-level data in the bitstream for each decoded image. Then, it retrieves an identifier from the encoded or reconstructed data corresponding to each decoded image to determine whether the image uses a different number of interpolated frames. If the decoded image uses this identifier, the decoding method retrieves the number of interpolated frames required before or after that image from the bitstream to replace the common number of interpolated frames. In another implementation, the decoding method also needs to obtain an identifier for the interpolation direction from the bitstream to determine whether the interpolated frame is before or after the current decoded image.

[0101] The syntax design for identifying the number of interpolated images is shown in the table below.

[0102]

[0103]

[0104] Where srd_temporal_resampling_ratio_idx represents the number of shared interpolated frames, srd_prd_temporal_resampling_ratio_change_allowed_flag indicates whether there are any decoded frames in the video stream that use a number different from the number of shared interpolated frames; these two syntaxes can be obtained from the sequence-level data; prd_temporal_resampling_ratio_changed_flag indicates whether its corresponding decoded frame uses a number different from the number of shared interpolated frames, and prd_temporal_resampling_ratio_index indicates the value of the number of decoded frames that use a number different from the number of shared interpolated frames.

[0105] In the above implementation, if the video length is limited, some interpolated frames may appear at the end of the video. The number of these interpolated frames cannot be inferred from the timing information of other decoded images because there is no longer any encoded data for subsequent decoded images in the bitstream. In this case, the bitstream should also carry an end marker with timing information. The decoding method can obtain this end marker from the bitstream to infer the number of interpolated frames to be generated at the end of the video. One method for implementing the end marker is to include the number of interpolated frames to be generated after the last decoded image in the video's timing sequence into the encoded data of that image. This way, the decoding method knows how to generate subsequent images after decoding the last image. However, this method is not user-friendly to the encoding end, as the encoding end may not be able to determine whether an image is the last image of the video when encoding it, especially in low-latency encoding scenarios. Therefore, this method requires the encoding method to modify the encoded data of the last encoded image to describe the end marker after detecting the end of the video. The advantage of this method is that it simplifies the operation requirements of the decoding method and does not introduce additional computational waste. Another method for implementing the end marker is to indicate the number of interpolated frames to be generated at the end in the end data packet at the end of the video stream. In existing video encoding methods, an end data packet, such as the EOS (End of Sequence) packet used by VVC and HEVC, is usually inserted at the end of the video stream to mark the end of the video stream and distinguish it from subsequent video streams. In this case, the decoding method can determine the number of interpolated frames to be generated at the end of the video when scanning or receiving the end data. For the interpolation method, when the decoding method receives the end data, it can generate the required number of interpolated frames using at least one previously decoded image based on the end marker carried therein. This introduces a certain delay, but it will not exceed the delay inherent in the interpolation method itself. For the extrapolation method, the decoding method can generate interpolated frames first using the previously decoded image as described in the previous embodiments. When the end marker in the end data packet is received, the number of interpolated frames to be retained can be determined based on the number of interpolated frames described therein. Another way to implement end-of-video marking is to explicitly assign timing information to the end data packet at the end of the video stream, indicating the exact time point at which the video should end. In this case, the decoding method can generate a certain number of interpolated frames according to the target frame rate and the timing information of the last decoded image, following the chronological order and the end time point to meet the end requirements of the video. In real-time transmission scenarios, the end data packet may not explicitly carry timing information. The decoding method can terminate the generation of interpolated frames and complete decoding upon receiving the end data packet.However, this method is affected by latency in network transmission and system management. Therefore, while it can save some data consumption, it places higher demands on the operation from the encoding end to the decoding end.

[0106] In a more specific decoding method implementation, the bitstream contains two parts of data: encoded data of the decoded image and reconstructed data corresponding to the decoded image. The reconstructed data describes the reconstruction information needed to reconstruct the decoded video after the decoded image is decoded again, such as the frame interpolation identifier information, timing information, and end markers described in this invention. This reconstructed data can also be recorded in the encoded data of the decoded image. The advantage of this is that the bitstream structure of existing encoding / decoding methods can be directly utilized without designing a new bitstream structure. The disadvantage is that it cannot be compatible with various existing or potentially more efficient video encoding / decoding methods in the future. Therefore, the decoding method disclosed in this invention can separate the reconstructed data and encoded data, and without affecting the use of encoded data generated by existing or potentially future video encoding / decoding methods, use the reconstructed data to reconstruct the decoded image to achieve higher visual quality or improve its effectiveness for machine intelligence.

[0107] Specifically, the decoding method obtains information for frame interpolation from the sequence-level and image-level reconstructed data, and one syntax structure is designed as shown in the table below.

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] Among them, `sequence_restoration_data_rbsp` is the sequence-level reconstructed data packet, recording the packet number `srd_sequence_restoration_data_id`, the target frame rate information `vcm_num_units_in_tick` and `vcm_time_scale`, and the sequence-level interpolation information `srd_temporal_restoration_data()`. The sequence-level interpolation information records the actual frame rate information of the decoded image `cvd_num_units_in_tick` and `cvd_time_scale`, the number of shared interpolated frames `srd_temporal_resampling_ratio_idx`, and the switch `srd_prd_temporal_resampling_ratio_change_allowed_flag` that allows the image to use a different number of interpolated frames than the number of shared interpolated frames. The image-level reconstruction data packet `picture_restoration_data_rbsp` records the number of the sequence-level reconstruction data packet it references, the low-order bits of the temporal number of the image-level reconstruction data `prd_picture_order_cnt_lsb`, and `prd_temporal_restoration_data`. The image-level frame interpolation information records the corresponding image's switch `prd_temporal_resampling_ratio_changed_flag` and the actual number of interpolated frames used `prd_temporal_resampling_ratio_index`. The end-of-stream reconstruction data packet `end_of_nal_sample_stream_rbsp` records the number of interpolated frames to be generated at the end of the video, `erd_num_temporal_remain`.

[0114] Based on the above information, the decoding method performs the following processing:

[0115] 1) Calculate the actual frame rate of the decoded image: CVDClockTick = cvd_num_units_in_tick ÷ cvd_time_scale, and use this frame rate to calculate the display time of the decoded image or the output time of the decoded image buffer.

[0116] 2) Based on `prd_picture_order_cnt_lsb`, calculate the temporal sequence number of the image-level reconstructed data using a high-order cumulative method similar to existing video coding methods. Then, calculate the temporal information of the image-level reconstructed data, also known as the activation time, based on the target frame rate `VCMClockTick = vcm_num_units_in_tick ÷ vcm_time_scale`. Confirm the correspondence between the image-level reconstructed data and the decoded image based on the consistency between the activation time and the display time of the decoded image.

[0117] 3) Based on the number of interpolated images recorded in the image-level reconstruction data corresponding to the decoded image, either prd_temporal_resampling_ratio_index or the number of shared interpolated images, srd_temporal_resampling_ratio_idx, use the decoded image and any other possible decoded images to generate interpolated images that meet the required number.

[0118] 4) Combine the decoded image and the interpolated image to obtain the decoded video.

[0119] This embodiment also discloses a video coding method based on frame interpolation, the flowchart of which is as follows: Figure 2 As shown, the implementing entity of this method can be an encoder. For the input video, the encoder extracts a portion of images from the video according to a certain rule for encoding and generating a bitstream. The extraction rule can be extraction at fixed intervals, or extraction based on the richness of image content after video content analysis. In one embodiment, the encoder does not completely skip the encoding of unextracted images. Instead, it inserts the image header information of the unextracted images into the bitstream, and includes interpolation image identifier information therein. This information includes the index information of the decoded images used to generate the interpolated images, generation method information, and display information of the interpolated images. Then, the encoder skips the encoding of the strip data of that image and proceeds to the encoding of the next image, ultimately obtaining the bitstream. In another embodiment, the encoder completely skips the encoding of the unextracted images and includes interpolation image identifier information in the syntax elements of the extracted images used to generate interpolated images. This information includes the number or display number of all interpolated images generated from the extracted image.

[0120] This embodiment discloses a decoding device, which includes a processor and a memory, for executing the decoding method disclosed in this invention.

[0121] This embodiment discloses an encoding device, which includes a processor and a memory, for executing the encoding method disclosed in this invention.

[0122] The above description of the embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A video decoding method based on frame interpolation, wherein for the bitstream, the following operations are performed: Reconstructed data, which includes frame interpolation identification information, is obtained from the bitstream. Obtain at least one decoded image from the bitstream; Based on the interpolation information in the reconstructed data, generate interpolated images using at least one decoded image; The decoded image and the interpolated image are combined to obtain the decoded video.

2. The method according to claim 1, further comprising: Obtain the end identifier from the reconstructed data. This end identifier describes the number of interpolated frames that need to be generated at the end of the video. Based on the end identifier, at least one decoded image is used to generate the required number of interpolated images.

3. The method according to claim 1, further comprising: Obtain the number of sequence-level interpolated images from the reconstructed data; Based on the number of sequence-level interpolated images, an interpolated image matching the stated number is generated before each decoded image.

4. The method according to claim 1, further comprising: Obtain the target frame rate of the decoded video from the reconstructed data; The decoded video is obtained by combining the decoded image and the interpolated image according to the target frame rate.

5. The method according to claim 1, further comprising: Image cache management information is obtained from the bitstream. This management information is used to indicate the images that should be stored in the image cache, which are used to generate interpolated images or are waiting to be displayed. Based on the image cache management information, images other than those that should be stored in the image cache are marked as no longer being referenced. These no longer-referenced images can be deleted, replaced, or updated.

6. A video coding method based on frame interpolation, wherein for the input video, the following operations are performed: Extract at least one image from the input video and skip the others; For the extracted image, the extracted image is encoded using an existing encoding method to obtain a first bitstream; For the skipped image, the frame insertion identification information corresponding to the skipped image is put into the second bitstream. The frame insertion identification information includes the index information of at least one decoded image of the extracted image. The decoded image is used to generate the skipped image corresponding to the frame insertion identification information. The frame insertion identification information also includes the output time of the skipped image. The first and second bitstreams are merged to obtain the output bitstream.

7. A video decoding apparatus based on frame interpolation, comprising: processor; A memory used to store bitstream and image data to be decoded; as well as One or more programs are used to perform the frame-interpolation-based video decoding method as described in any one of claims 1-4.

8. A video encoding apparatus based on frame interpolation, comprising: processor; Memory used to store bitstreams and images to be encoded; as well as One or more programs are used to perform the frame interpolation-based video coding method as described in any one of claims 5-7.

9. A storage device, characterized in that, The storage device contains a bitstream obtained using the video encoding method as described in claim 6, and can be decoded using the video decoding methods as described in claims 1-5.

Images