Method of managing a decoded picture buffer and video encoder or video decoder

Abstract

A video encoder or a video decoder is disclosed which uses Intra Block Copy (IntraBC) mode. If IntraBC mode is enabled and a loop filter is not used for a current picture, a current decoded reference picture for the current picture is marked as a long-term reference picture before and during the decoding of the current picture. The current decoded reference picture is assigned to a reference picture list construction for IntraBC reference of the current picture, and by considering a buffer which stores only one reference picture, a buffer storage degree of a decoded picture buffer is calculated before starting to decode the current picture. According to another method, if a maximum decoded picture buffer size is 1, a variable or a syntax element is set to a target value to limit to use only one picture storage buffer in the decoded picture buffer to store a reference picture for encoding or decoding the current picture.

Description

[0001] Priority Statement

[0002] This invention claims priority to U.S. Provisional Patent Application No. 62 / 215,287, filed September 8, 2015, and U.S. Provisional Patent Application No. 62 / 245,372, filed October 23, 2015. The entirety of the aforementioned U.S. Provisional Patent Applications is incorporated herein by reference. Technical Field

[0003] This invention relates to palette coding for video data. Specifically, it relates to the management / operation of decoded image buffers when the coding system uses coding modes including inter-frame prediction mode and intra-block copy mode. Background Technology

[0004] High Efficiency Video Coding (HEVC) is a relatively new coding standard that has emerged in recent years. In HEVC, the fixed-size macroblocks of H.264 / AVC are replaced by flexible blocks called Coding Units (CUs). Pixels within a Coding Unit share the same encoding / decoding parameters to improve encoding / decoding efficiency. Coding Units can start from the Largest Coding Unit (LCU), also known as a Coded Tree Unit (CTU) in HEVC. In addition to the concept of Coding Units, HEVC also introduces Prediction Units (PUs). Once the hierarchical tree of Coding Units is partitioned, each leaf Coding Unit is further subdivided into one or more Prediction Units based on the prediction type and prediction unit classification. HEVC also supports slice structures, where an image is divided into multiple slices, each of which can use its own encoding / decoding parameters or configurations.

[0005] With the development of the HEVC standard, the development of HEVC extensions has also begun. HEVC extensions include Range Extension (RExt), which targets non-4:2:0 color formats such as 4:2:2 and 4:4:4, as well as video with higher bit depths, such as 12 bits, 14 bits, and 16 bits per sample. One potential application of RExt is screen sharing based on wired or wireless connections. Due to the specific characteristics of screen content, encoding and decoding tools have been developed, and significant progress has been made in encoding and decoding efficiency.

[0006] Intra-block copy prediction

[0007] In the current development of screen content coding for the HEVC standard, several tools have been adopted due to their improved efficiency in encoding and decoding screen content. For intra-blocks, intra-prediction is performed using reconstructed pixels from neighboring blocks, following traditional methods. Intra-prediction can select an intra-mode from a set of intra-modes, including horizontal, vertical, and various angle prediction modes. For HEVC screen content encoding and decoding, a new intra-codec mode called Intra-block copy (IntraBC) has been adopted. IntraBC technology was initially proposed by Bulagavi in ​​AHG8, namely: Video coding using Intra motion compensation, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 13th Meeting: Incheon, KR, 18–26 Apr. 2013, Document: JCTVC-M0350 (hereinafter referred to as JCTVC-M0350). An example based on JCTVC-M0350 is as follows... Figure 1 As shown, intra-frame motion compensation (MC) is used, and the current CU (i.e., 110) is encoded and decoded. The prediction block (i.e., 120) is located based on the current CU and the displacement vector (i.e., 112). In this example, the search area is limited to the current coding tree unit, the left CTU, and the second left CTU. The prediction block is obtained from the reconstructed area. Subsequently, the displacement vector used for the current CU, also known as the block vector (BV), and the residual are encoded and decoded. As is well known, HEVC uses coding tree units and coding unit block structures as basic units to encode and decode video data. Each image is segmented into multiple coding tree units, and each coding tree unit is recursively segmented into multiple coding units. During the prediction phase, each coding unit can be segmented into multiple blocks, called prediction units, for performing prediction processing. After the prediction residuals are formed for each coding unit, the residuals associated with each coding unit are segmented into multiple blocks, called transform units (TUs), for applying the transform.

[0008] In JCTVC-M0350, intra-frame motion compensation differs from motion compensation used for inter-frame prediction in at least the following aspects:

[0009] • The motion vector (MV) is restricted to one dimension for intra-frame motion compensation (i.e., vertical or horizontal), while inter-frame prediction uses two-dimensional motion estimation.

[0010] • Binarization is of fixed length for intra-frame motion compensation, while inter-frame prediction uses exponential-Golomb.

[0011] • A new syntax element has been introduced in intra-frame motion compensation to indicate whether the motion vector is horizontal or vertical.

[0012] Based on JCTVC-M0350, some variations were disclosed by Pang et al. in Non-RCE3, namely: Intra Motion Compensation with 2-D MVs, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16WP 3 and ISO / IEC JTC1 / SC 29 / WG 11, 14th Meeting: Vienna, AT, 25 July–2 August 2013, Document: JCTVC-N0256 (hereinafter referred to as JCTVC-N0256). First, intra-frame motion compensation was extended to support two-dimensional motion vectors, allowing both motion vector components to be non-zero simultaneously. This provides more flexibility for intra-frame motion compensation than the original method, where motion vectors were strictly limited to horizontal or vertical.

[0013] JCTVC-N0256 discloses two block vector encoding and decoding methods:

[0014] • Method 1 - Block Vector Prediction. The left or upper block vector is selected as the block vector predictor, and the resulting block vector difference (BVD) is encoded and decoded. One flag is used to indicate whether the block vector difference is 0. When the block vector difference is not 0, a third-order exponential Golomb code is used to encode and decode the residual absolute degree of the block vector difference. Another flag is used to encode and decode this symbol.

[0015] Method 2: Blockless Vector Prediction. Using exponential Golomb codes for BVD in HEVC, BV is encoded and decoded.

[0016] Another difference disclosed in JCTVC-N0256 is the further integration of 2D IntraBC with pipeline-friendly methods:

[0017] 1. No interpolation filter was used.

[0018] 2. The block vector search area is limited. Two cases are described:

[0019] a. The search area is the current CTU and the CTU to the left, or

[0020] b. The search area is the rightmost four columns of samples of the current CTU and the leftmost CTU.

[0021] In the method proposed in JCTVC-N0256, the two-dimensional IntraBC, the removal of the interpolation filter, and the constraint on the search region of the current CTU and the left CTU have been used in the new version of the draft HEVC RExt standard.

[0022] IntraBC is already included in the existing SCC Draft Standard (SCM).

[0023] In JCTV-T0227 (Pang, et al., Non-CE2 Test1: Intra block copy and intersignalling unification, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 20th Meeting: Geneva, CH, 10–18 Feb. 2015, Document: JCTVC-T0227), the IntraBC mode is unified with the inter-frame coding and decoding mode. In other words, the current image is processed as a reference image and inserted into one or two reference image lists (i.e., list L0 and list L1). Block vector prediction and coding / decoding are handled in the same way as inter-frame motion vector prediction and coding / decoding. This unification simplifies the design of the codec.

[0024] Adaptive motion resolution

[0025] In JCTV-S0085 (Li, et al., Adaptive motion vector resolution for screencontent, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 19th Meeting: Strasbourg, FR, 17–24 Oct. 2014, Document: JCTVC-S0085), the flag “use_integer_mv_flag” is used for each slice to indicate the resolution of the motion vectors in the slice. As specified in the HEVC standard, if the flag is 1, all decoded motion vectors in the slice are represented using an integer pixel resolution; otherwise, these motion vectors are represented using a quarter pixel resolution.

[0026] The current SCC draft standard considers memory bandwidth.

[0027] When reconstructing a block, a reference block is needed for prediction. If the motion vector points to a fractional location, additional pixels surrounding the reference block are also needed for interpolation purposes. When the reference block and its surrounding pixels need to be retrieved from external memory, data fetching is performed in batches depending on the memory model used. For example, if the memory model used is a 4x4 sample block, the smallest unit for accessing data is a 4x4 pixel even if only one pixel is needed. Therefore, more data than directly related pixels will be retrieved and sent to on-chip memory / buffers for further processing, all of which consume memory bandwidth. In HEVC, the worst-case memory bandwidth consumption occurs in 8x8 dual prediction mode where both motion vectors have fractional pixels. When IntraBC mode is used, in addition to the filtered version of the current image used for inter-frame prediction mode, the unfiltered version of the current image also needs to be stored in external memory. This is considered an additional cost of HEVC.

[0028] In JCTVC-U0078 (Rapaka, et al., CE2: Test 5 on intra block copy constraints on prediction, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 21st Meeting: Warsaw, PL, 19 June–26 June 2015, Document: JCTVC-U0078), some restrictions are imposed on the use of 8x8 dual prediction mode when IntraBC is used. For example, the use of 8x8 dual prediction mode is disabled when the Sequence Parameter Set (SPS) flag of IntraBC is enabled and the slice header flag use_integer_mv_flag (indicating the MV resolution in the slice) is disabled.

[0029] Decoded image buffer in the current SCC draft standard

[0030] In HEVC, all reference images are stored in a buffer called the decoded picture buffer (DPB). This is performed whenever an image is decoded, and the currently decoded image after the loop filtering operation is placed into the DPB (called the filtered version of the currently decoded image). For IntraBC, its reference image is the currently decoded image before the loop filtering (called the unfiltered version of the currently decoded image), which is an additional image to be stored in the DPB compared to HEVC version 1. In JCTVC-U0181 (Xu, et al., On storage of filtered and unfiltered currently decoded pictures, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 21st Meeting: Warsaw, PL, 19 June–26 June 2015, Document: JCTVC-U0181), both the filtered and unfiltered versions of the currently decoded image are placed in the DPB for image buffer management. For convenience, by omitting the word "decoded," the filtered or unfiltered version of the currently decoded image can be referred to as the filtered or unfiltered version of the current image. After decoding the current image, the unfiltered version of the current image is discarded, and the storage buffer for this image is released.

[0031] The case where two versions of the currently decoded image are identical is taken into account. This situation occurs when no loop filter (deblocking or sample adaptive offset) is used for the current image. In the current HEVC SCC working draft described in JCTVC-U1005 (Joshi, et al., High Efficiency Video Coding (HEVC) Screen Content Coding: Draft 3, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16WP 3 and ISO / IEC JTC1 / SC 29 / WG 11, 21st Meeting: Warsaw, PL, 19 June–26 June 2015, Document: JCTVC-U1005), a variable called TwoVersionsOfCurrDecPicFlag is used to identify whether the current image is used as a reference image. If so, there may be two different versions of the current image due to the use of loop filters in the image.

[0032] 1) The currently decoded image after the loop filter is invoked is stored in the DPB. Once the decoding of the current image is complete, the image is marked as "for short-term reference". When decoding is complete, the image will be stored in the DPB for subsequent use.

[0033] 2) When the flag TwoVersionsOfCurrDecPicFlag equals 1, the currently decoded image before the loop filter is invoked is stored in the DPB in a separate buffer. This image is marked "for long-term reference" and used for IntraBC compensation. When decoding is complete, this image is removed from the DPB.

[0034] According to the HEVC specification, short-term reference image sets are designed to provide references to images that are temporally close to the current image, such as images with the same image structure. On the other hand, long-term reference image sets can be used to resolve selected images for longer-term reference purposes. These long-term reference image sets are constructed independently of short-term image sets. As the management of the decoded image buffer changes when the current image is the reference image, certain constraints need to be imposed to ensure the proper functioning of the decoded image buffer management. Summary of the Invention

[0035] This invention discloses a method and system for managing a decoded image buffer, used by a video codec or video decoder employing encoding / decoding modes including inter-frame prediction mode and intra-frame block copy mode. If intra-frame block copy mode is enabled and the loop filter is not used for the current image, a set of steps is executed to ensure correct decoded image buffer operation. The set of steps includes: marking the currently decoded reference image used for the current image as a long-term reference image before and during decoding of the current image; allocating the currently decoded reference image to a reference image list construction for use as an intra-frame block copy reference for the current image; and calculating the buffer storage degree of the decoded image buffer before starting decoding of the current image by considering a buffer that stores only one reference image. The currently decoded reference image corresponds to an empty image storage buffer in the decoded image buffer. Additionally, the method may include: marking the currently decoded reference image in the decoded image buffer as a short-term reference image after decoding of the current image; and retaining the currently decoded reference image in the decoded image buffer for subsequent use.

[0036] This invention discloses another method and system for managing a decoded image buffer, wherein if the maximum decoded image buffer size corresponds to 1, the method includes: setting a variable or syntax element to a target value to restrict the use of only one image storage buffer in the decoded image buffer to store a reference image for encoding or decoding the current image. For example, if the intra-block copy mode is not used for the current image, setting the variable or syntax element to the target value corresponds to setting the slice type of multiple slices in the current image to I slice. As another example, setting the variable or syntax element to the target value corresponds to setting the variable TwoVersionsOfCurrDecPicFlag to a first value corresponding to the use of only one version of the currently decoded image. The variable TwoVersionsOfCurrDecPicFlag indicates whether two versions of the currently decoded image are used. In yet another example, setting the variable or syntax element to the target value corresponds to setting the intra-block copy flag in the image parameter set to a second value to indicate that the intra-block copy mode is not used for the current image.

[0037] This invention discloses yet another method and system for managing a decoded image buffer, wherein the total number of reference images used for decoding the current image or slice is calculated by considering information about whether two versions of the currently decoded image are used. The set of reference images used for decoding the current image or slice includes short-term reference images and long-term reference images. For example, the total number of reference images is calculated as the sum of a first total number of short-term reference image sets having image sequence count values ​​less than the current image's image sequence count value, a second total number of short-term reference image sets having image sequence count values ​​greater than the current image's image sequence count value, a third total number of long-term reference image sets, and version values ​​related to information about whether two versions of the currently decoded image are used. If both versions of the currently decoded image are used, the version value is 1; if one version of the currently decoded image is used, the version value is 0. If the total number of reference images is greater than the maximum required size of the decoded image buffer specified in the sequence parameter set minus 1, the total number of reference images is set to the maximum required size of the decoded image buffer minus 1 or less. Attached Figure Description

[0038] Figure 1 An example of an intra-block copy mode is shown, where the current block is predicted by a reference block in the current image.

[0039] Figure 2 A flowchart illustrating an exemplary codec system including the management / operation of a decoded image buffer according to an embodiment of the present invention is shown.

[0040] Figure 3 A flowchart illustrating an exemplary codec system according to an embodiment of the present invention includes management / operation of a decoded image buffer with a maximum DPB size of 1.

[0041] Figure 4 A flowchart of an exemplary encoding / decoding system according to an embodiment of the present invention is shown, wherein the total number of reference images for DPB is calculated by taking into account information about whether two versions of a currently decoded image are used. Detailed Implementation

[0042] The following description describes a preferred mode for carrying out the invention. This description is intended to illustrate the general principles of the invention and is not intended to be limiting. The scope of protection of this invention is defined by the appended claims.

[0043] As mentioned above, existing decoded image buffer management may not function properly when the currently reconstructed image can be used as a reference image based on the intra-block copy mode. To overcome the problems with DPB operations when using IntraBC, various methods for operating DPB are disclosed.

[0044] Method 1: Calculation of the maximum size of the decoded image buffer

[0045] The syntax element `sps_max_dec_pic_buffering_minus1` is specified in the SPS layer for each time layer bitstream to limit the total number of allowed decoded and stored images. This syntax element should be less than or equal to `(MaxDPBSize-1)`, where `MaxDPBSize` is used to determine the maximum allowed DPB size in the standard for a given image resolution. In HEVC, one image buffer in the DPB is reserved for storing a filtered version of the currently decoded image. The remainder of the DPB is used to store other time reference images or decoded images awaiting output. The sum of all these stored images in the DPB, including time reference images and images awaiting output, should be less than or equal to the value of the SPS syntax element `sps_max_dec_pic_buffering_minus1` for this layer, which specifies the maximum required size of the DPB for the coded video sequence (CVS) in the image storage buffer unit according to JCTVC-U1005. The value of the SPS syntax element sps_max_dec_pic_buffering_minus1 should be in the range of 0 to MaxDpbSize–1.

[0046] When the current image is used as a reference image, an unfiltered version of the current image may also need to be placed in the DPB. According to one embodiment of Method 1, the number of reference images allowed in terms of DPB capacity is calculated by considering whether the unfiltered version of the currently decoded image is the same as the filtered version of the currently decoded image. For example, when the two versions are the same, it is not necessary to create or reserve an image buffer for the unfiltered version of the current image, which may occur when no loop filter is available for the currently decoded image. Therefore, this unused image buffer can be allocated so that one more temporal reference image can be stored in the DPB and used to predict the current image. Otherwise, when the current image is used as a reference image and the two versions of the currently decoded image are different, both versions of the current image need to be stored in the DPB. The remainder of the DPB will be used to store the temporal reference image and other possible images to be output. In all cases, the sum of the number of reference images and the number of the current image (one or two versions) should be less than or equal to the maximum allowed number of decoded images in the DPB.

[0047] In one embodiment, the variable TwoVersionsOfCurrDecPicFlag is used to identify whether the current image is used as a reference image and whether the two versions of the current image are different. When this variable equals 1, it means that there are two different versions of the current image, and the current image is used as a reference image. When this variable equals 0, it means that only one version of the current image exists, or the current image is never used as a reference image for decoding the current image. An exemplary DPB size calculation according to an embodiment of the present invention is shown below, wherein the portion “General Segment Header Semantics” specified in subclause 7.4.7.1 of JCTVC-U1005 is modified as follows, and italicized text indicates the added text:

[0048] `num_long_term_pics` specifies the number of entries in the long-term reference picture set (RPS) of the current image, which is directly indicated in the slice header. When it does not exist, the value of `num_long_term_pics` is inferred to be equal to 0. When `nuh_layer_id` is equal to 0, the sum of `NumNegativePics[CurrRpsIdx]`, `NumPositivePics[CurrRpsIdx]`, `num_long_term_sps`, `TwoVersionsOfCurrDecPicFlag`, and `num_long_term_pics` will be less than or equal to `sps_max_dec_pic_buffering_minus1[sps_max_sub_layers_minus1]`.

[0049] `NumNegativePics[CurrRpsIdx]` specifies the number of entries in the `CurrRpsIdx`th candidate short-term RPS with a `Picture Order Count` (POC) value less than that of the current image. `NumPositivePics[CurrRpsIdx]` specifies the number of entries in the `CurrRpsIdx`th candidate short-term RPS with a `POC` value greater than that of the current image. `num_long_term_sps` specifies the number of entries in the long-term RPS of the current image, which is derived based on the candidate long-term reference images specified in the active SPS. `sps_max_dec_pic_buffering_minus1` specifies the maximum required size of the decoded image buffer for the encoded / decoded video sequence. As shown above, according to an embodiment of the present invention, the DPB size is calculated as the sum of short-term RPS (i.e., NumNegativePics and NumPositivePics), long-term RPS (i.e., num_long_term_sps and num_long_term_pics), and the variable TwoVersionsOfCurrDecPicFlag.

[0050] Method 2: Constraints on the use of IntraBC when the maximum allowed DPB size is limited.

[0051] When the maximum decoded image buffer size is 1 (e.g., sps_max_dec_pic_buffering_minus1 containing a sublayer of the current image equals 0), this means that only one available decoded image buffer is used to store the currently decoded image (i.e., the filtered version) for this layer. In this case, if the IntraBC is not used for the current image, the slice type used for slices in this image should be an I-slice. When the maximum decoded image buffer size for the current encoded / decoded video stream is 1 (e.g., sps_max_dec_pic_buffering_minus1 containing a sublayer of the current image equals 0), Method 2 of the present invention does not allow the current image to be used as a reference image for decoding the current image.

[0052] In one embodiment, the syntax element pps_curr_pic_ref_enabled_flag is used as a Picture Parameter Set (PPS) flag to indicate whether IntraBC is used for the current image referencing the PPS. An exemplary constraint is shown below, where the portion of "General Segment Header Semantics" specified in subclause 7.4.7.1 of JCTVC-U1005 is modified:

[0053] Slice type semantics...

[0054] When `nal_unit_type` has a value in the range from `BLA_W_LP` to `RSV_IRAP_VCL23` including both endpoints (i.e., the image is an IRAP image), `nuh_layer_id` equals 0, and `pps_curr_pic_ref_enabled_flag` equals 0, the slice...

[0055] _type should be equal to 2.

[0056] When sps_max_dec_pic_buffering_minus1[TemporalId] equals 0 and nuh_layer_id equals 0, slice_type should equal 2.

[0057] An example constraint is shown below, in which the portion “Image Parameter Set Range Extension Semantics” specified in subclause 7.4.3.3.2 of JCTVC-U1005 is modified, wherein the italicized text indicates the added text:

[0058] A value of 1 for `pps_curr_pic_ref_enabled_flag` indicates that an image from the referenced PPS can be included in the list of reference images for its own slice. A value of 0 for `pps_curr_pic_ref_enabled_flag` indicates that an image from the referenced PPS is never included in the list of reference images for its own slice. When it does not exist, the value of `pps_curr_pic_ref_enabled_flag` is inferred to be 0.

[0059] The requirement for bitstream consistency is that when sps_curr_pic_ref_enabled_flag equals 0, the value of pps_curr_pic_ref_enabled_flag should also equal 0.

[0060] The requirement for bitstream consistency is that when sps_max_dec_pic_buffering_minus1[TemporalId] equals 0 and nuh_layer_id equals 0, pps_curr_pic_ref_enabled_flag should be 0.

[0061] In another alternative to Method 2, the current image should not be used as a reference image for decoding the current image when the maximum decoded image buffer size is 1 (e.g., sps_max_dec_pic_buffering_minus1 containing the sublayer of the current image is equal to 0).

[0062] In this method, the variable TwoVersionsOfCurrDecPicFlag is used to identify whether the current image is used as a reference image and whether the two versions of the current image are different. When this variable equals 1, it means that two different versions of the current image exist, and the current image is used as a reference image. When this variable equals 0, it means that only one version of the current image exists, or the current image is never used as a reference image for decoding the current image. In one embodiment, the requirement for bitstream consistency is that if TwoVersionsOfCurrDecPicFlag equals 1, then the syntax element sps_max_dec_pic_buffering_minus1 of the sublayer containing the current image should be greater than 0. In another embodiment, the requirement for bitstream consistency is that if sps_max_dec_pic_buffering_minus1 of the sublayer containing the current image equals 0, then the value of TwoVersionsOfCurrDecPicFlag should be 0.

[0063] In another alternative to Method 2, if no loop filter is applied to the current image, then the current image can be used as a reference image for decoding the current image when the maximum decoded image buffer size is 1 (sps_max_dec_pic_buffering_minus1 containing the sublayer of the current image equals 0). Furthermore, this current image should be the only reference image that can be used to decode the current image.

[0064] Method 3: Store the reference image for IntraBC in the image buffer when no loop filter is available for the current image.

[0065] When IntraBC is applied to the current image, the reference image used for IntraBC is the unfiltered version of the current image. If no loop filter is used for the current image, then the two versions of the currently decoded image are identical. There is no image buffer created in DPB to store the additional version (unfiltered version) of the currently decoded image, which is the reference image used for the current image. In this case, method 3 of the present invention modifies the allocation of the reference image buffer so that IntraBC can reference the correct image buffer, which stores the currently decoded image as the reference image used for the current image.

[0066] When IntraBC is used and no loop filter is used, only one version of the current image exists. Therefore, according to method 3, in DPB, only one image buffer will be created to store the currently decoded image. This image will be used as a reference image for IntraBC. In one embodiment, this image, referred to as the filtered version of the current image, will be used as the reference image for IntraBC. In another embodiment, when using IntraBC and without a loop filter, when only one version of the current image exists, the current image is marked as "for long-term reference" at the beginning and during the current image decoding process. After decoding the current image is complete, this image is marked as "for short-term reference". In yet another embodiment, at the beginning and during the current image decoding process, the filtered version of the current image is marked as "for long-term reference". After decoding the current image is complete, the filtered version of the current image is marked as "for short-term reference".

[0067] In one embodiment, the variable TwoVersionsOfCurrDecPicFlag is used to identify whether the current image is used as a reference image and whether the two versions of the current image are different. When this variable equals 1, it means that there are two different versions of the current image, and the current image is used as a reference image. When this variable equals 0, it means that only one version of the current image exists, or the current image has never been used as a reference image for decoding the current image. By modifying the decoding process of existing video codec standards, embodiments of the present invention can be applied to existing video codec standards (e.g., HEVC). For example, part of the subclause 8.1.3 of JCTVC-U1005 can be modified as follows, where the text enclosed in brackets and asterisks (i.e., [*deleted text*]) represents the deleted text, and the italicized text represents the added text:

[0068] 8.1.3 Decoding process for a decoded image with a nuh_layer_id equal to 0;

[0069] 3. The procedures in clauses 8.4, 8.5, 8.6, and 8.7 specify the decoding procedures using syntactic elements across all syntactic structure layers. The requirement for bitstream consistency is that the encoded / decoded slices of an image should include segment data for each coding tree unit of the image, such that the segmentation of the image into slices, the segmentation of slices into segments, and the segmentation of segments into coding tree units constitute the segmentation of the image. As specified in clause 8.7, the decoded sample values ​​of the current image after the call to the loop filter procedure are stored in the image storage buffer of the current image.

[0070] [*and not marked as "for long-term reference"*]. When TwoVersionsOfCurrDecPicFlag equals 0, the current image after the call to the loop filter procedure, as specified in clause 8.7, is marked as "for long-term reference". When TwoVersionsOfCurrDecPicFlag equals 1, the decoded sample values ​​of the current image before the call to the loop filter procedure, as specified in clause 8.7, are stored in the current image's image storage buffer and marked as "for long-term reference".

[0071] The subclause 8.3.4 in JCTVC-U1005 can be modified as follows, where the text enclosed in parentheses and asterisks (i.e., [*deleted text*]) represents the deleted text, and the italicized text represents the added text:

[0072] 8.3.4 Decoding process for constructing a reference image list;

[0073] [*Assume the variable currPic is the currently decoded image marked "for long-term reference"*]. When TwoVersionsOfCurrDecPicFlag equals 1, assume the variable currPic is the currently decoded image before the loop filter procedure is invoked, as specified in clause 8.7; otherwise (i.e., TwoVersionsOfCurrDecPicFlag equals 0), assume the variable currPic is the currently decoded image after the loop filter procedure is invoked, as specified in clause 8.7.

[0074] Appendix C 3.4 in JCTVC-U1005 can be modified as follows, where italicized text indicates added text:

[0075] Appendix C3.4 Tagging and storage of currently decoded images;

[0076] When TwoVersionsOfCurrDecPicFlag equals 1

[0077] – As specified in clause 8.7, the currently decoded image after the loop filter procedure is invoked is stored in the DPB in the empty image storage buffer, the fullness of the DPB is increased by 1, and this image is marked as “for short-term reference”.

[0078] otherwise,

[0079] – Following the loop filter procedure as specified in clause 8.7, the currently decoded image is stored in the DPB in the empty image storage buffer, the DPB's storage level is increased by 1, and this image is marked "for long-term reference". After all slices of the current image have been decoded, following the loop filter procedure as specified in clause 8.7, the currently decoded image is marked "for short-term reference".

[0080] Use and tagging of image caches for IntraBC

[0081] The currently decoded image after loop filtering needs to be stored in the DPB for subsequent use (i.e., output or as a reference image for future images). When the IntraBC is used for the current image, the reference image used for the IntraBC is the unfiltered version of the current image (i.e., the decoded image before loop filtering). If no loop filter is used for the current image, then the two versions of the currently decoded image are identical. In this case, it is not necessary to use an additional image buffer in the DPB to store a copy of the currently decoded image. Therefore, only one version of the currently decoded image needs to be stored in the DPB. According to one embodiment of the method, the allocation of the reference image buffer is modified so that the IntraBC can reference the correct image buffer, which stores the currently decoded image as a reference image for the current image.

[0082] Specifically, when IntraBC is enabled and no loop filter is used, according to one embodiment of this method, only one version of the current image is used as the reference image. Before and during decoding the current image, this current image is marked as "for long-term reference" and used as the reference image for IntraBC operation. After decoding the current image is complete, the current image is marked as "for short-term reference".

[0083] The syntax and semantics of the existing HEVC SCC standard draft (i.e., JCTVC-U1005) can be modified to implement the above-mentioned cache management. Exemplary modifications to the syntax and semantics are shown below.

[0084] Some parts of subclause 8.1.3, namely “decoding process of encoded and decoded images with nuh_layer_id equal to 0”, have been modified. Specifically, item 3 of subclause 8.1.3 has been modified as follows, where italicized text indicates added text.

[0085] 3. The procedures in clauses 8.4, 8.5, 8.6, and 8.7 specify the decoding procedures using syntactic elements across all syntactic structure layers. The requirement for bitstream consistency is that the encoded / decoded slices of an image should include fragment data for each coding tree unit of the image, such that the segmentation of the image into slices, the segmentation of slices into fragments, and the segmentation of fragments into coding tree units constitute the segmentation of the image. When TwoVersionsOfCurrDecPicFlag equals 1, the decoded sample values ​​of the current image after the loop filter procedure is invoked, as specified in clause 8.7, are stored in the image storage buffer used for the current image. The decoded sample values ​​of the current image before the loop filter procedure is invoked, as specified in clause 8.7, are stored in another image storage buffer used for the current image and are marked "for long-term reference". When TwoVersionsOfCurrDecPicFlag equals 0 and pps_curr_pic_as_ref_enabled_flag equals 0, the decoded sample values ​​of the current image after the loop filter procedure is invoked, as specified in clause 8.7, are stored in the storage buffer for the current image. When TwoVersionsOfCurrDecPicFlag equals 0 and pps_curr_pic_as_ref_enabled_flag equals 1, the decoded sample values ​​of the current image before the loop filter procedure is invoked, as specified in clause 8.7, are stored in the image storage buffer for the current image and are marked "for long-term reference".

[0086] In another embodiment, the last sentence in item 3 of the sub-clause can be changed to “The decoded sample values ​​of the current image after the invocation of the loop filter procedure, as specified in clause 8.7, are stored in the image storage buffer for the current image and marked as ‘for long-term reference’.”

[0087] Some parts of the subclause 8.3.4, namely “Decoding process for constructing a list of reference images”, have been modified as follows, where the text enclosed in parentheses and asterisks (i.e., [*deleted text*]) indicates deleted text, and the text in italics indicates added text.

[0088] [*Assume the variable currPic is the currently decoded image marked "for long-term reference".*] When TwoVersionsOfCurrDecPicFlag equals 1, assume the variable currPic is the currently decoded image before the loop filter procedure is invoked, as specified in clause 8.7; otherwise (i.e., TwoVersionsOfCurrDecPicFlag equals 0), assume the variable currPic is the currently decoded image after the loop filter procedure is invoked, as specified in clause 8.7.

[0089] In another embodiment, the last sentence of the above embodiment can be modified to "Otherwise (i.e. TwoVersionsOfCurrDecPicFlag equals 0), assume that the variable currPic is the currently decoded image as specified in clause 8.7 before the invocation of the loop filter procedure (immediately before)".

[0090] Similarly, some parts of the subclause Appendix C 3.4, namely “marking and storage of currently decoded images”, also need to be modified as follows, where the text contained in parentheses and asterisks (i.e., [*deleted text*]) indicates deleted text, and the italicized text indicates added text.

[0091] If TwoVersionsOfCurrDecPicFlag equals 1, then

[0092] – As specified in clause 8.7, the currently decoded image after the loop filter procedure is invoked is stored in the DPB in the empty image storage buffer, the storage level of the DPB is increased by 1, and this image is marked as “for short-term reference”.

[0093] –[*When TwoVersionsOfCurrDecPicFlag equals 1,*] As specified in clause 8.7, the currently decoded image before the loop filter procedure call is stored in the DPB in the empty image storage buffer, and the storage level of the DPB is increased by 1.

[0094] And this image is labeled "for long-term reference".

[0095] ...

[0096] Otherwise, if TwoVersionsOfCurrDecPicFlag equals 0 and pps_curr_pic_ref_enabled_flag equals 0, then

[0097] – As specified in clause 8.7, the currently decoded image after the loop filter procedure is invoked is stored in the DPB in the empty image storage buffer, the storage level of the DPB is increased by 1, and this image is marked as “for short-term reference”.

[0098] Otherwise (i.e., TwoVersionsOfCurrDecPicFlag equals 0 and pps_curr_pic_ref_enabled_flag equals 1),

[0099] – As specified in clause 8.7, the currently decoded image after the loop filter procedure is invoked is stored in the DPB in the empty image storage buffer, the DPB's storage level is increased by 1, and this image is marked "for long-term reference". After all slices of the current image have been decoded, as specified in clause 8.7, the currently decoded image after the loop filter procedure is invoked is marked "for short-term reference".

[0100] In another embodiment, the last sentence of the above paragraph can be modified to "The currently decoded image before the invocation of the loop filter procedure (immediately before) is marked as 'for short-term reference', as specified in clause 8.7".

[0101] Figure 2 A flowchart of an exemplary codec system including decoded image buffer management / operation according to an embodiment of the present invention is shown. In step 210, before decoding the current image, the process determines whether the IntraBC mode is enabled and whether any loop filters are used for the current image. In step 220, it is checked whether the IntraBC mode is enabled and whether loop filters are used for the current image. If the result is "yes", steps 230 to 250 are executed. If the result is "no", the process terminates. In step 230, before and during decoding the current image, the currently decoded reference image used for the current image is marked as a long-term reference image, wherein the currently decoded reference image corresponds to an empty image storage buffer in the DPB. In step 240, the currently decoded reference image is allocated to the reference image list construction for use as an IntraBC reference for the current image. In step 250, the buffer storage degree of the DPB is calculated before starting to decode the current image by considering a buffer that stores only one reference image.

[0102] Figure 3A flowchart of an exemplary encoding / decoding system according to an embodiment of the present invention, including management / operation of a decoded image buffer with a maximum DPB size equal to 1, is shown. In step 310, the process determines the maximum DPB size. In step 320, if the maximum DPB size corresponds to 1, a variable or syntax element is set to a target value to restrict the use of only one image storage buffer in the DPB to store a reference image for encoding or decoding the current image.

[0103] Figure 4 A flowchart of an exemplary encoding / decoding system according to an embodiment of the present invention is shown, wherein the total number of reference images for a Decoder Base (DPB) is calculated by taking into account information about whether two versions of a currently decoded image are used. In step 410, the total number of reference images for the DPB is calculated by taking into account information about whether two versions of a currently decoded image are used. The DPB includes short-term reference images and long-term reference images. As shown in step 420, a maximum total number of reference images are allocated in the DPB. As shown in step 430, the current image is then encoded or decoded using the reference images stored in the DPB.

[0104] The flowchart shown is intended to illustrate an example of video encoding according to the present invention. Those skilled in the art can modify each step, reorganize these steps, separate a step, or combine these steps to implement the invention without departing from the spirit of the invention. In this invention, specific syntax and semantics have been used to illustrate different examples of implementing embodiments of the invention. Those skilled in the art can implement the invention by replacing the syntax and semantics with equivalent syntax and semantics without departing from the spirit of the invention.

[0105] The foregoing description enables those skilled in the art to implement the invention within the content and requirements of specific applications. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, the invention is not limited to the specific embodiments shown and described, but is to be given the maximum scope consistent with the principles and novel features disclosed herein. Various specific details have been set forth in the foregoing detailed description in order to provide a thorough understanding of the invention. Nevertheless, it will be understood by those skilled in the art that the invention is capable of being practiced.

[0106] The embodiments of the present invention described above can be implemented in various hardware, software code, or a combination of both. For example, an embodiment of the present invention may be one or more circuits integrated within a video compression chip slice, or program code integrated into video compression software to perform the processes described herein. An embodiment of the present invention may also be program code executed on a digital signal processor (DSP) to perform the processes described herein. The present invention may also include functions executed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array (FPGA). According to the present invention, processors can be configured to perform specific tasks by executing machine-readable software code or firmware code that defines the specific methods implemented by the present invention. The software code or firmware code may be developed in different programming languages ​​and different formats or styles. The software code may also be compiled for use on different target platforms. However, different code formats, software code styles and languages, and other forms of configuration code for performing the tasks of the present invention do not depart from the spirit and scope of the present invention.

[0107] This invention is practiced in other specific forms without departing from its spirit or essential characteristics. The examples described are illustrative in all respects only and not limiting. Therefore, the scope of the invention is indicated by the appended claims, not by the foregoing description. The meaning of the claims, and all variations within the same scope, should be included within their scope.

Claims

1. A method for managing a decoded image buffer for a video encoder or video decoder using encoding / decoding modes including inter-frame prediction mode and intra-frame block copy mode, the method comprising: Determine the total number of reference images used for the decoded image buffer, wherein the decoded image buffer includes short-term reference images and long-term reference images; Allocate a maximum of the total number of reference images in the decoded image buffer; as well as Using the total number of reference images stored in the decoded image buffer, the current image is encoded or decoded. If the intra-block copy mode is enabled, and the current image does not use a loop filter but only uses one version of the current decoded image, the current decoded image is marked as a long-term reference at the beginning or during the decoding of the current image, and the current decoded image in the decoded image buffer is marked as a short-term reference image after decoding the current image, wherein the one version of the current decoded image is the unfiltered version of the current decoded image.

2. The method for managing a decoded image buffer as described in claim 1, characterized in that, The total number of reference images is calculated as the sum of a first total number of short-term reference image sets having an image sequence count value less than that of the current image, a second total number of short-term reference image sets having an image sequence count value greater than that of the current image, a third total number of long-term reference image sets, and a version value related to information about whether two versions of the currently decoded image are used, wherein the version value is 1 if both versions of the currently decoded image are used, and the version value is 0 if only one version of the currently decoded image is used.

3. The method for managing a decoded image buffer as described in claim 2, characterized in that, If the total number of reference images is greater than the maximum required size of the decoded image buffer specified in the sequence parameter set minus 1, then the total number of reference images is set to the maximum required size of the decoded image buffer minus 1, or smaller.

4. The method for managing a decoded image buffer as described in claim 1, characterized in that, Further includes: The currently decoded image is marked as a short-term reference image, and After decoding the current image, the currently decoded image is retained in the decoded image buffer.

5. The method for managing a decoded image buffer as described in claim 1, characterized in that, Information about whether two versions of the currently decoded image are used is determined based on a flag indicating that the loop filter is not used for the current image and the intra-block copy mode is enabled.

6. A video encoder or video decoder using an encoding / decoding mode including an inter-frame prediction mode and an intra-frame block copy mode, comprising a decoded image buffer and one or more electronic circuits, wherein the one or more electronic circuits are used for: Determine the total number of reference images used for the decoded image buffer, wherein the decoded image buffer includes short-term reference images and long-term reference images; Allocate a maximum of the total number of reference images in the decoded image buffer; and Using the total number of reference images stored in the decoded image buffer, the current image is encoded or decoded. If the intra-block copy mode is enabled, and the current image does not use a loop filter but only uses one version of the current decoded image, the current decoded image is marked as a long-term reference at the beginning or during the decoding of the current image, and the current decoded image in the decoded image buffer is marked as a short-term reference image after decoding the current image, wherein the one version of the current decoded image is the unfiltered version of the current decoded image.

Images