Symbol prediction in video coding

Abstract

Methods, systems, and bitstream syntax are described for symbol prediction in video coding. The method includes selection of top and left neighbors based on an image continuity check, an intra mode of a current coding unit (CU), merge motion vector or adaptive motion vector prediction, symbol prediction based on a residual domain of the current CU or a neighboring CU, symbol prediction based on an approximately reconstructed sample, and symbol prediction based on an approximately reconstructed sample. And by combining symbol prediction with symbol data hiding, the number of selected coefficients used for sorting is reduced, and the sequence search cost is reduced.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to Indian Provisional Patent Application No. 202241021948 filed on April 12, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] This document relates generally to image and video coding. More specifically, embodiments of the invention relate to symbol prediction in video coding. Background Art

[0004] In 2020, the MPEG expert group in the International Organization for Standardization (ISO) and the International Telecommunication Union (ITU) jointly released the first version of the generic video coding standard (VVC), also known as H.266 (reference [1]). Recently, the MPEG expert group has been working on the development of next-generation coding standards that provide improved coding performance compared to existing video coding technologies. As part of this investigation, new coding technologies are also studied.

[0005] As appreciated by the inventors herein, improved techniques for symbol prediction in image and video coding are desired and such techniques are described herein.

[0006] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art simply by virtue of their inclusion in this section. Similarly, unless otherwise indicated, it should not be assumed that issues identified with respect to one or more approaches are recognized in any prior art based on this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the invention are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0008] Figure 1 depicts an example pixel configuration for sign prediction in video coding;

[0009] Figure 2A depicts an example processing pipeline for symbol prediction according to the prior art;

[0010] Figure 2B depicts an example processing pipeline for symbol prediction according to an embodiment of the present invention;

[0011] Figure 3 depicts an example of symbol prediction according to an embodiment of the present invention;

[0012] Figure 4 depicts an example showing continuity to the top or to the left based on a motion vector according to an embodiment of the present invention;

[0013] Figure 5 An example diagram depicting a coding unit (CU) and its neighbors;

[0014] Figure 6 depicts an example subdivision of a picture for processing a transform unit according to an embodiment of the present invention;

[0015] Fig. 7A , Figure 7B and Figure 7C An example of a process flow for symbol prediction according to an embodiment of the present invention is depicted; and

[0016] Figure 8 An example of reducing the area of ​​sorting prediction coefficients according to an embodiment of the present invention is depicted. DETAILED DESCRIPTION

[0017] Example embodiments related to symbol prediction in video coding are described herein. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. However, it will be apparent that various embodiments of the present invention may be practiced without these specific details. In other cases, well-known structures and devices are not described in detail to avoid unnecessarily obscuring, obscuring or confusing embodiments of the present invention.

[0018] Overview

[0019] Example embodiments described herein relate to sign prediction of transform coefficients in image and video coding. Embodiments for improving sign prediction include: selection of top and left neighbors based on image continuity check, intra mode of the current coding unit (CU), merged motion vector or adaptive motion vector prediction, sign prediction based on the residual domain of the current CU or neighboring CUs, sign prediction based on approximate reconstructed samples, and reducing the number of selected coefficients for sorting and simplifying the sequential search cost by combining sign prediction with sign data hiding.

[0020] Data symbol prediction and coding in video coding

[0021] In traditional video coding, the signs of the residual coefficients can be transmitted uncompressed and can account for 10% of the bit rate in the compressed stream. Symbol prediction aims to improve compression efficiency by reducing the bit overhead of the residual symbols. The symbols that have been predicted are no longer signaled uncompressed in the bitstream, but are replaced by a coded "residual" signaled using the relevant arithmetic coding (e.g., CABAC) context, which indicates whether the prediction was correct.

[0022] Symbol Prediction Algorithm

[0023] As described in references [2-3], the basic idea of ​​the coefficient sign prediction method is to compute the reconstructed residuals of the negative and positive sign combinations of the applicable transform coefficients and then select the hypothesis with the minimum cost function. In order to derive the best sign, the cost function is defined as a discontinuity measure across block boundaries, such as Figure 1 The cost functions are measured for all hypotheses, and the one with the smallest cost is chosen as the predictor of the coefficient sign.

[0024] The cost function is defined as the sum of the absolute second-order derivatives in the residual domain with respect to the rows on the top and the columns on the left, as follows:

[0025]

[0026] where w represents the predicted width, h represents the predicted height, R represents the reconstructed neighbors (105), P represents the prediction for the current block (110), and r is the residual hypothesis. The term (-R -1 +2R0-P1), and only the residual assumption is subtracted.

[0027] When predicting n symbols in a transform unit (TU), the encoder and decoder perform n+1 partial inverse transforms and 2n boundary reconstructions corresponding to 2n symbol combination hypotheses, where each boundary reconstruction has a boundary cost metric. These costs are checked to determine the symbol prediction value, and the encoder uses two additional CABAC contexts to transmit a symbol residual for each predicted symbol, which indicates whether the prediction of the symbol is correct. The decoder reads these symbol residuals, calculates the hypothesis reconstruction to calculate the predictor being used, and then uses the received residuals to determine the correct symbol.

[0028] Reference [4] proposes to improve the symbol prediction process by the following changes:

[0029] (1) Applying qIdx-based coefficient selection, where qIdx represents the dequantized transform coefficient level after compensating for the effects of multiple quantizers in dependent quantization (DQ).

[0030] (2) Apply symbol prediction to the largest region selected from the four allowed block size values: 4, 8, 16, and 32 (signaled by the encoder in the sequence parameter set (SPS))

[0031] (3) Apply symbol prediction to low frequency non-separable transform (LFNST) (up to 4 symbols and 4x4 regions)

[0032] As reported in Ref. [5], the horizontal qIdx depends on the DQ state and can be calculated as follows:

[0033] qIdx=(abs(level)<<1)-(state&1).

[0034] The qIdx values ​​represent the absolute values ​​of the dequantized coefficients. Sorting by "level" may not give the best results, since the levels do not accurately reflect the quantization due to the use of two quantizers.

[0035] The proposed symbol prediction is quite complex and adds non-trivial complexity to the hardware decoding pipeline. As described in [4]:

[0036] 1) The inverse transform and sign prediction of the current block requires the brightness reconstruction pixels of the neighboring blocks.

[0037] 2) Two pixel rows of the top block and two pixel columns of the left block are needed to measure the boundary discontinuity loss.

[0038] Since the inverse transform depends on the reconstructed pixels in the neighborhood, there is a stall in the pipeline, which is shown as an example of a hypothetical decoder pipeline in Figure 2A As shown in Figure 2A As shown, the pipeline includes: entropy decoding, motion vector decoding and boundary strength (for deblocking filtering) derivation, inverse quantization, symbol prediction and inverse transform, inter-frame prediction, intra-frame prediction and reconstruction, and loop filtering. Conventional pipeline delays are represented by "x". Stalls are represented by "S". The pipeline shows the dependencies across multiple modules assuming that there is one coding unit (CU) in a virtual pipeline data unit (VPDU). In the case where the number of CUs in the VPDU exceeds 1, the dependencies will be at the micro-pipeline level, where the pause duration varies with the size of the CU.

[0039] The embodiments presented here aim to improve the symbol prediction process from different aspects:

[0040] 1) Quality Improvement (QI) proposals: targeting compression efficiency compared to current coding tools, preferably without any additional hardware (HW) implementation issues.

[0041] 2) Proposals for removing HW dependency / pipeline issues (HWPI) and for HW / SW complexity reduction (CR): Aims to address HW dependency / pipeline issues and similar tool coordination by considering certain guidelines and HW / SW-friendly solutions (e.g., complexity reduction with minimal impact on compression efficiency).

[0042] 3) Coordination with other VVC tools (e.g., Symbol Data Hiding (SDH)).

[0043] Smart selection of top / left neighbors for image continuity check

[0044] Motivation: Improve prediction accuracy by intelligently selecting neighboring pixels

[0045] Proposal: The current algorithm tries to minimize the cost with respect to both top and left neighbors; however, depending on the scene characteristics, image continuity may be true in only one direction. There are three cases for cost calculation: 1) using only top neighbors; 2) using only left neighbors; 3) using both top and left neighbors.

[0046] Top neighbor cost

[0047]

[0048] Left neighbor cost:

[0049]

[0050] Cost per pixel of top neighbor = top-cost / w

[0051] The per-pixel cost of the left neighbor = left_cost / h

[0052] Among them, w and h represent the width and height respectively.

[0053] Different indicators can be selected to make a decision among the three options, such as:

[0054] 1. If the absolute difference between the per-pixel cost of the left neighbor and the top neighbor is above a certain threshold

[0055] If |top_cost / w-left_cost / h|>T diff , then replace the cost in equation (1) with min(top_cost, left_cost) for symbol prediction. Threshold T diffThis can be determined in a variety of ways: for example: 1) using a fixed threshold derived from experimental results; 2) selecting a threshold from a set of predetermined thresholds (e.g., T based on content characteristics (e.g., the variance of the boundary area); diff_set ={T diff1 , T diff2 , T diff3}) and / or quantization parameters (QP) to select a T diff ; 3) T is adaptively calculated using a mathematical formula based on QP and content characteristics, etc. dif f = f(QP, content).

[0056] 2. If the ratio between max(left_cost, top_cost) and min(left_cost, top_cost) is greater than the positive integer threshold T ratio

[0057] If max(left_cost, top_cost) / min(left_cost, top_cost)>T ratio , then replace the cost in equation (1) with For symbol prediction. Threshold T ratio Can be done with T diff A similar method is used to determine

[0058] 3. If the percentage difference between the per-pixel cost of the left neighbor and the upper neighbor is above the threshold T percent , the percentage is measured relative to the maximum cost difference between the upper neighbor and the neighbor to the left.

[0059] If |top-cost / w-left-cost / h|>T percent *max(left_cost / h, top_cost / w), then replace the cost in equation (1) with min(top_cost / w, left_cost / h) for symbol prediction. Threshold T percent Can be done with T diff A similar method is used to determine

[0060] Intelligent selection of top / left neighbors based on the intra mode of the current CU

[0061] Motivation: Improve prediction accuracy by intelligently selecting neighbors using intra prediction mode

[0062] Proposal: For intra blocks, the intra mode gives an indication of the prediction direction of the pixel value. For example, if the intra mode is in the vertical direction, the top pixels may be more reliable for computing the cost. As a further improvement to this, the cost can also be computed in the direction of the intra mode. For example, if the intra mode points 45 degrees to vertical, the cost in equation (1) can be computed by considering neighboring pixels at an angle, such as Figure 3 As shown by the arrow in . For other intra-frame modes, when the angle does not point to a full pixel position, it is necessary to interpolate the pixel values ​​of the neighbors to generate a sub-pixel position value for cost calculation. Any known pixel interpolation technique known in the art can be applied.

[0063] Intelligent selection of top / left neighbor based on merged motion vector (MV) or adaptive motion vector prediction (AMVP) MV direction

[0064] Motivation: Improve prediction accuracy by using the motion vector list information of the current CU to make intelligent choices of neighbors. If the motion information of the current CU and the neighboring CU are similar, they are most likely to pass the continuity check.

[0065] Proposal: The motion vector list of the current CU consists of various spatial and temporal candidates. The current CU can give priority to neighbors whose motion information is similar to that of the current CU.

[0066] For example, in Figure 4 In , the motion vectors of the current CU (405) and the left neighbor (415) point to the left, and the motion vector of the top neighbor (410) points to the top left. The left and current CUs are more likely to belong to similar regions and will show better image continuity. If there are multiple partitions on the left neighbor and the right neighbor, some guidelines can be followed to consider the MVs from the left and top. For example, if Figure 5 As shown, for the current CU (510), motion vectors corresponding to pixel regions A, B, C, and D may be considered for neighbor selection.

[0067] Proposals to address pipelining issues and reduce complexity

[0068] Sign prediction based only on the residual domain of the current CU

[0069] Motivation: Removes dependency on neighboring reconstructed pixels. Since neighboring samples are removed, the location of the residual hypothesis template can be expanded in a variety of ways if necessary, including (but not limited to):

[0070] 1) The first row and the first column are the same as the current symbol prediction usage

[0071] 2) The first and last rows; the first and last columns cover all internal boundaries of the TU

[0072] 3) The first and last row; the first and last column and the center 2x2 or 4x4 area covers all borders as well as the center area.

[0073] Proposal: Select a symbolic prediction hypothesis that meets one or more of the following criteria:

[0074] a. Compare the maximum absolute value of the spatial domain residual value of each hypothesis (L abs ). Select L abs Minimal assumptions. Specifically:

[0075] For each hypothesis k = 0, 1, .. (2 M -1), calculate L abs (k) as follows

[0076] L abs (k) = max(abs(r(i, j))) for all valid positions i, j of the LUT-based residual hypothesis (r) of the current TU

[0077] Select the best hypothesis based on minimum cost, for example, min(L abs (k))

[0078] b. Calculate the sum of the absolute magnitudes of all residuals for each hypothesis in the spatial domain (S abs ). Select S abs Minimal assumptions. Specifically:

[0079] For each hypothesis k = 0, 1, .. (2M-1), calculate S abs (k) as follows

[0080] S abs (k) = ∑(abs(r(i, j))) for all valid positions i, j of the LUT-based residual hypothesis (r) of the current TU

[0081] Select the best hypothesis based on minimum cost, for example, min(S abs (k))

[0082] c. By L abs and S abs Assign some weights (w, 1-w) to combine (a) and (b). Select the hypothesis with the smallest weighted sum [L abs * w+(S abs / N)*(1-w)] or [(L abs *w+((S abs -Labs ) / (N-1))*(1-w)]

[0083] d. Select the hypothesis with the smallest variance

[0084] Removes dependency on neighboring reconstructed pixels by using residual data from neighboring pixels

[0085] Motivation: Remove dependency on neighboring reconstructed pixels

[0086] Proposal: Select a sign prediction hypothesis that satisfies one or more of the following criteria for the spatial domain residual values ​​of the current CU and neighboring CUs. This solution is proposed for inter CUs that have at least one neighbor as an inter.

[0087] a. The average amplitude of the spatial domain residual value of each hypothesis (L avg ) and the average amplitude of the left neighboring block and the top neighboring block (L Ltavg and L Topavg ) for comparison. If both the left neighbor and the top neighbor are inter-frame, the hypothesis closest to the two neighbors is selected; otherwise, it must be closest to the left neighbor or the top neighbor of the inter-frame. Specifically:

[0088] For each hypothesis k = 0, 1, .. (2 M -1), calculate the average residual error L avg (k) as follows

[0089] L avg (k) = ∑(r(i,j))) / N for all valid positions i, j of the LUT-based residual hypothesis (r) of the current TU, where N is the number of samples in the residual hypothesis

[0090] ·L leftavg and L topavg are the average residuals of the left neighbor samples and the top neighbor samples based on their true symbols.

[0091] Select the best hypothesis based on the closest left and top neighbors, i.e.,

[0092]

[0093] b. The sum of the absolute magnitudes of all residual values ​​for each hypothesis in the spatial domain (S abs ) and the sum of the absolute amplitudes of the left neighboring block and the top neighboring block (S Ltavg and S Topavg ) for comparison. If both the left neighbor and the top neighbor are inter-frame, the hypothesis closest to the two neighbors is selected; otherwise, it must be closest to the left neighbor or the top neighbor of the inter-frame. Specifically:

[0094] For each hypothesis k = 0, 1, .. (2M-1), calculate S abs (k) as follows

[0095] S abs (k) = ∑abs(r(i, j))) for all valid positions i, j of the LUT-based residual hypothesis (r) of the current TU, where N is the number of samples in the residual hypothesis

[0096] ·L leftsabs and L topsabs are the sum of the absolute residual errors of the left neighboring samples and the top neighboring samples based on their true signs.

[0097] Select the best hypothesis based on the closest left neighbor and top neighbor, i.e.,

[0098]

[0099] c. By giving L abs and S abs Assign some weight (w, 1-w) to combine (a) and (b). [L abs *w+(S abs / N)*(1-w)] or [(L abs *w+((S abs -L abs ) / (N-1))*(1-w)]

[0100] Use constrained top neighbors or left neighbors for cost calculation to reduce dependencies on immediate neighbors in decoding order

[0101] Motivation: Current symbol prediction solutions require the immediate top neighbor reconstructed pixels and the immediate left neighbor reconstructed pixels. This introduces strong pipeline dependencies in the decoding pipeline, since the immediate neighbor reconstructed pixels are required to calculate the symbol value of the current TU.

[0102] Proposal: Decode TUs in Z-scan order. The proposal will use neighbors based on the following criteria:

[0103] If the left TU is the immediately previous TU in decoding order, use the top TU for neighbor cost. Figure 6 , TU 3 will use the top TU1 to calculate the neighbor cost instead of the left TU2, which is its immediately previous TU in the decoder order. Similarly, TU9 can use all top neighbor samples, but can use only some left neighbor samples from TU 7, because the left neighbor samples from TU 8 belong to the immediately previous TU in the decoding order.

[0104] If the top TU is the immediately preceding TU in decoding order, the left TU is used as the neighbor cost. Figure 6 , TU 6 will use the left TU 4 to calculate the neighbor cost instead of the top TU 5, which is its immediately previous TU in the decoder order.

[0105] Use the approximate reconstructed samples of the top inter-prediction CU and the left inter-prediction CU for sign prediction

[0106] Motivation: Remove dependency on neighboring reconstructed pixels

[0107] Proposal: The need for reconstructed pixels of the immediate neighbors introduces a strong pipeline dependency in decoding. Therefore, in an embodiment, the approximate reconstructed pixels of the neighbors may be used for sign prediction. The approximate reconstructed samples of the top inter-CU and the left inter-CU may be computed using (i) the prediction samples and (ii) the approximate residual samples of 2 rows and 2 columns of the neighboring CU using an inverse transform lookup table (the required residual samples must be stored during sign prediction of the corresponding CU). This approach will have pipeline dependencies only on the prediction samples. In another embodiment, a filtered version of the prediction samples (by linear or nonlinear filtering) may be used to approximate the neighboring reconstructed pixels.

[0108] This skips the complex serialization dependency on intraPred+Recon (the last stage of construction).This method cannot be applied to CUs whose neighbors are intra, and thus this method will lose the benefit of intra slices or CUs whose neighbors are intra.

[0109] The pipeline dependencies with this change are as follows Figure 2B As shown. Figure 2A In contrast, the pipeline dependency on intra prediction and reconstruction (IntraPredRec) has been removed. Since the dependency is restricted to inter prediction of neighboring pixels, the latency slots are reduced from 2 to 1. For intra CUs or CUs with intra neighbors, the previously proposed methods can be applied.

[0110] Reduce the ordering complexity of coefficients to be selected for sign prediction by reducing the maximum area

[0111] Motivation: In the current solution, the region used for symbol prediction is limited to a square region of size 4x4, 8x8, 16x16, or 32x32. Note that a 32x32 region would require us to sort an array of size 1024, which significantly increases the complexity of TU-level processing. This also significantly increases the LUT size.

[0112] Proposal: Starting from 32x32, reduce the area of ​​symbol prediction by using one or more of the following methods:

[0113] a. Reduce the maximum area to the upper triangular region, since high amplitude coefficients are more likely to reside in this region. This will reduce the area to WxH / 2

[0114] / / The upper left triangle does not exceed 50% TU area

[0115] signPredEnable=(Xpos+Ypos)<((W+H) / 2)

[0116] b. The maximum area can be further reduced by limiting the maximum intercept of the triangle area (e.g., to 32).

[0117] / / Additional constraint that the maximum intercept in a+TU is 32

[0118] signPredEnable = (Xpos + Ypos) <min(32,((W+H) / 2))

[0119] c. The region of interest can be further reduced. For example, to reduce the region to the region where the product of the x and y coordinates is < 64, the following constraint can be added

[0120] / / Case 2 + additional constraint of maximum area of ​​64

[0121] signPredEnable = (Xpos + Ypos) <min(32,((W+H) / 2))

[0122] signPredEnable&=((Xpos+1)*(Ypos+1)<=64)

[0123] These techniques will help define a region (801-a) in the upper left corner of a transform unit (TU) (801) where high amplitude coefficients are more likely to be located. Figure 8 An example of such processing is depicted for a 32×64 TU (801). Figure 8 In , A) depicts the results of the current Enhanced Compression Model (ECM) software in JVET, and B) to D) depict the results from proposals a) to c), respectively.

[0124] Reduce the ordering complexity of the coefficients to be selected for sign prediction by selecting the coefficients based on a threshold

[0125] Motivation: In current solutions, the region used for symbol prediction is limited to a square region of size 4x4, 8x8, 16x16, or 32x32; however, a 32x32 region requires sorting an array of size 1024, which significantly increases the complexity of TU-level processing. This also significantly increases the LUT size.

[0126] Proposal: Instead of sorting all coefficients, select coefficients based on absolute level (e.g., qIdx) in a specific order (useful when the number of coded coefficients is much larger than the maximum number of symbol prediction coefficients threshold (e.g., 8))

[0127] a. In the first pass, select qIdx with a scan order greater than 4 (e.g., >= 5)

[0128] b. If the maximum number of symbol prediction coefficients is not reached, select all qIdx in a scanning order greater than 2 (e.g., >= 3)

[0129] c. If the maximum number threshold of symbol prediction coefficients is still not reached, the remaining coding coefficients are selected in scanning order until the threshold is reached.

[0130] Cost simplification of the sequence search process in symbol prediction

[0131] Motivation: For simplicity, dequantized transform coefficients are represented as "coeff" or just coefficient. The sign prediction of the first dequantized transform coefficient (highest magnitude) is first predicted, and then the true sign of the first coefficient is used to predict the signs of subsequent coefficients. In the current process, all hypothesis costs need to be stored, and then the minimum cost is searched in the range where the true sign matches the hypothesis. This proposal aims to reduce the storage cost and the sequential operation of searching for the minimum cost.

[0132] proposal:

[0133] 1. Method 1: Select the predicted sign of all coeffs based only on the minimum cost

[0134] a. Reduced internal memory as no assumption cost needs to be stored.

[0135] b. Avoid the sequential process of searching

[0136] 2. Method 2: Select the predicted symbols for the remaining coefficients based on the correct symbol with the first largest coeff (with the highest qIdx magnitude) in the scan order. If the maximum coeff symbol is correctly predicted based on the minimum cost of all hypotheses, the same hypothesis is used to predict the remaining coefficient symbols, otherwise the 2 correct symbols with the largest coeff are selected. n-1 The minimum cost of the hypotheses is used to predict the signs of the remaining coefficients.

[0137] a. Store only the data from all hypotheses (2 n ) and from the assumption that the maximum coefficient has the correct sign (2 (n-1) ) minimum cost. Note: For the positive and negative signs of the maximum coefficient, use (2 (n-1)) hypothesized minimum costs, the minimum costs can be pre-realized without depending on the true sign of the maximum coefficient

[0138] b. Avoid the sequential process of searching

[0139] 3. Method 3: Select the predicted symbols for the remaining coefficients based on the correct symbol of the first largest coefficient (coeff) (highest qIdx magnitude) in the scan order. If the largest coeff symbol is correctly predicted based on the minimum cost of all hypotheses, the remaining coeff symbols are predicted using the same hypothesis, otherwise the 2 correct symbols with the largest coeff are selected. n-1 The sign of the remaining coeff is predicted by minimizing the cost of the top or left neighbors of the hypotheses.

[0140] a. Store the minimum cost for all hypotheses along with the left-only minimum and top-only minimum of the correct and incorrect maximum coefficient predictions.

[0141] Note: For the positive and negative signs of the maximum coeff, use (2 (n-1) ) hypothesized minimum costs, all of which can be realized in advance without depending on the true sign of the maximum coefficient.

[0142] b. Avoid the sequential process of searching

[0143] Fig. 7A , Figure 7B and Figure 7C Examples of proposed data flows for the three methods described herein are depicted.

[0144] Combining symbol prediction with symbol data hiding (SDH)

[0145] In reference [2], it is proposed to combine sign prediction with SDH. The basic idea of ​​SDH is to omit the encoding of the sign of a non-zero quantized coefficient and instead derive the sign from the parity of the sum of the absolute values ​​of all quantized coefficients. SDH is applied based on coefficient groups (CGs). In most cases, the CG size is 4x4. If the difference between the scan index of the last non-zero level and the first non-zero level (in the coding order) within the CG is greater than 3, the sign of the last non-zero level of the CG is not encoded but derived based on the sum of the absolute values, where an odd sum indicates a negative value. In reference [2], when SDH is combined with sign prediction, the order is used to first perform sign data hiding and then perform sign prediction on the remaining coefficients.

[0146] In an example embodiment, it is proposed to improve coding efficiency by changing the rules for which quantized coefficients should apply SDH. For example, in the current embodiment, since the coefficients are sorted, SDH can be applied to the highest qIdx coefficients, and then symbol prediction can be applied to the remaining coefficients. In an alternative solution, based on statistics, rules can be generated to predict coefficients that have low precision when symbol prediction is used, and then SDH is applied to those coefficients.

[0147] SDH is applied to CG (mainly 4x4), while symbol prediction can be applied to variable block sizes from 4x4 to 32x32. In another embodiment, the symbol prediction block size rule can be changed. For example, if SDH is used, only 4x4 is allowed.

[0148] References

[0149] Each of the references listed herein is incorporated herein by reference in its entirety.The term JVET refers to the Joint Video Experts Group of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29.

[0150] [1] “Versatile Video Coding”, Rec. ITU-T H.266, August 2020.

[0151] [2] JVET-D0031, “Residual Coefficient Sign Prediction”, F. Henry, G. Clare, Chengdu, October 2016.

[0152] [3] JVET-J0021, “Description of SDR, HDR and 360° video coding technology proposal by Qualcomm and Technicolor–low and high complexity versions”, Y.-W Chen et al., San Diego, April 2018.

[0153] [4] JVET-Y0141, “EE2-4.3related: More combined test results for signprediction,” J. Chen et al., online conference, January 2022.

[0154] [5] JVET-X0120, “AHG12: On sign prediction”, MG Sarwer et al., online conference, October 2021.

[0155] Example Computer System Implementation

[0156] Embodiments of the present invention may be implemented using a computer system, a system configured with electronic circuits and components, an integrated circuit (IC) device (such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or implement instructions related to symbol prediction in image and video encoding, such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to symbol prediction in image and video encoding described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0157] Certain embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the method of the present invention. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement the method related to symbol prediction in the image and video encoding as described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention can also be provided in the form of a program product. The program product may include any non-transient and tangible medium that carries a set of computer-readable signals including instructions, which, when executed by a data processor, causes the data processor to perform the method of the present invention. The program product according to the present invention may take any form in various non-transient and tangible forms. The program product may include, for example, a physical medium, such as a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signal on the program product may be optionally compressed or encrypted.

[0158] Where a component (e.g., software module, processor, component, device, circuit, etc.) is mentioned above, unless otherwise indicated, reference to the component (including reference to "means") should be interpreted to include any component that performs the function of the described component as an equivalent (e.g., functionally equivalent) to the component, including components that are not structurally equivalent to the disclosed structure that performs the function in the illustrated example embodiments of the invention.

[0159] Equivalents, extensions, alternatives and miscellaneous

[0160] Thus described are example embodiments related to symbol prediction in image and video coding. In the foregoing description, embodiments of the present invention have been described with reference to many specific details, which may vary from implementation to implementation. Therefore, the only and exclusive indication of the present invention and the applicant's inventive intent is the set of claims issued in a specific form according to the present application, wherein such claim issuance includes any subsequent corrections. Any definitions expressly set forth herein for terms contained in such claims should govern the meaning of such terms as used in the claims. Therefore, limitations, elements, properties, features, advantages or attributes not expressly recited in a claim should not limit the scope of such claim in any way. Therefore, this specification and drawings should be viewed in an illustrative rather than a restrictive sense.

Claims

1. A method for performing symbol prediction in video coding, the method comprising: Accessing a current transform unit (TU), a left neighboring TU of the current TU, and a top neighboring TU of the current TU, wherein the neighboring TUs include neighboring pixels; Access thresholds associated with symbol predictions; calculating a top cost value for each pixel and a left cost value for each pixel; and A hypothesized cost for symbol prediction is generated based on the threshold, the top cost value for each pixel, and the left cost value for each pixel.

2. The method of claim 1, wherein: The top cost value of each pixel = top_cost / w, The left cost value of each pixel = left_cost / j, Wherein, w and h represent width and height respectively, R represents the reconstructed neighbor (105), P represents the prediction of the current CU, and r is the residual hypothesis, Among them, given the threshold T diff , T ratio and T percent If |top_cost / w-left_cost / h|>T diff , then the hypothesized cost including all neighboring pixels is replaced by min(top_cost, left_cost) for symbol prediction; If max(left_cost, top)cost) / min(left_cost, top)>T ratio , then the hypothesized cost including all neighboring pixels is replaced by min(top_cost, left_cost) for symbol prediction; If |top_cost / w-left_cost / h|>T percent *max(left_cost / h, top_cost / w), then the hypothesized cost including all neighboring pixels is replaced by min(top-cost / w, left_cost / h) for symbol prediction.

3. A method for performing symbol prediction in video coding, the method comprising: Accessing a current coding unit (CU), a left neighboring CU of the current CU, and a top neighboring CU of the current CU, wherein the neighboring CUs include neighboring pixels; If the current CU is encoded in intra mode, only neighboring pixels in the direction of the intra mode are considered when calculating the hypothesis cost to determine the sign prediction.

4. A method for performing symbol prediction in video coding, the method comprising: Accessing a current transform unit (TU), a left neighboring TU of the current TU, and a top neighboring TU of the current TU, wherein the neighboring TUs include neighboring pixels; Accessing a vector list of the current TU; and The hypothesis cost is calculated by considering only the neighboring TUs having similar motion information to determine the symbol prediction.

5. A method for performing symbol prediction in video coding, the method comprising: Select a symbolic prediction hypothesis that meets one or more of the following criteria: a) Compare the maximum absolute value of the spatial domain residual value of each hypothesis in the hypothesis (L abs ), and select the one with the smallest L abs Assumptions; b) Calculate the sum of the absolute magnitudes of all residuals for each hypothesis in the spatial domain (S abs ), and select the one with the smallest S abs Assumptions; c) By sending L abs and S abs Assign some weights (w, 1-w) to combine (a) and (b), and select the hypothesis with the smallest sum of weights [L abs *w+(S abs / N)*(1-w)] or [(L abs *w+((S abs -L abs ) / (N-1))*(1-w)].

6. A method for performing symbol prediction in video coding, the method comprising: Selecting a symbol prediction hypothesis that satisfies one or more of the following criteria with respect to spatial domain residual values ​​of a current TU and neighboring TUs of the current TU: a) The average amplitude (L) of the spatial domain residual value of each hypothesis in the hypothesis avg ) and the average amplitude of the left neighboring block and the top neighboring block (L Ltavg and L Topavg ) for comparison; as well as If both the left neighbor and the top neighbor are inter-frame, the hypothesis closest to the two neighbors is selected; otherwise, selecting the hypothesis of the left neighbor or the top neighbor that is closest to the frame interval; b) The sum of the absolute magnitudes of all residual values ​​for each hypothesis in the spatial domain (S abs ) and the sum of the absolute amplitudes of the left neighboring block and the top neighboring block (S Ltavg and S Topavg ) for comparison; as well as If both the left neighbor and the top neighbor are inter-frames, the hypothesis that is closest to the two neighbors is selected; otherwise, the hypothesis that is closest to the left neighbor or the top neighbor is selected. c) By sending L abs and S abs Assign some weight (w, 1-w) to combine (a) and (b), [L abs *w+(S abs / N)*(1-w)] or [(L abs *w+((S abs -L abs ) / (N-1))*(1-w)].

7. A method for performing symbol prediction in video coding, the method comprising: Decode transform units (TUs) in Z-scan order; and Signal prediction is applied using the neighboring pixel area of ​​the current TU, where the neighboring TU is constrained as follows in signal prediction: If the left TU is the immediately previous TU in decoding order, the top TU is used for the neighbor cost; If the top TU is the immediately preceding TU in decoding order, the left TU is used for the neighbor cost.

8. A method for performing symbol prediction in video coding, the method comprising: When computing the sign prediction hypothesis, instead of using fully reconstructed pixel values ​​from neighboring coding units (CUs), approximate residual samples of the two immediately adjacent rows of the top neighboring CU or the two immediately adjacent columns of the left neighboring CU are used.

9. A method for performing symbol prediction in video coding, the method comprising: Starting with a 32x32 pixel region, reduce the maximum area for symbol prediction by using one or more of the following methods: reducing the maximum area to an upper triangular area; Further reducing the maximum area by limiting the maximum intercept of the triangular area; and The maximum area is further reduced by limiting the product of the x and y coordinates to below 64.

10. A method for performing symbol prediction in video coding, the method comprising selecting coefficients based on a dequantized transform coefficient level (qIdx) and a threshold, wherein: The options include: The first pass, in which qIdx values ​​are selected in a scanning order greater than 4; If the maximum number of symbol prediction coefficients threshold is not reached, selecting all qIdx in a scan order greater than 2; and If the maximum number threshold of symbol prediction coefficients is still not reached, the remaining coding coefficients are selected in a scanning order until the threshold is reached.

11. A method for performing symbol prediction in video coding, the method comprising: computing a minimum host hypothesis and predicting the sign of the dequantized transform coefficient having the highest magnitude based on the minimum cost; Sign prediction of the remaining coefficients is then performed using one or more of the following: selecting predicted signs for all coefficients based solely on said minimum cost; selecting said predicted symbols for the remaining coefficients based on the correct sign of the first largest coefficient (having the highest qIdx magnitude) in the scan order; If the largest coefficient sign is predicted correctly based on the minimum cost of all hypotheses, then the same hypothesis is used to predict the remaining coefficient signs, otherwise the 2 with the correct sign of the largest coefficient is selected n-1 The minimum cost of the hypothesis is used to predict the signs of the remaining coefficients; selecting the predicted symbols for the remaining coefficients based on the correct symbol for the first largest coefficient in the scan order; if the largest coefficient symbol is correctly predicted based on the minimum cost of all hypotheses, predicting the remaining coefficient symbols using the same hypothesis, otherwise selecting the 2 largest coefficients with the correct symbol n-1 The sign of the remaining coefficients is predicted by taking the minimum of the top neighbor costs or the left neighbor costs of the two hypotheses. 12 . A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions being used to execute the method according to claim 1 using one or more processors.

13. An apparatus comprising a processor and configured to perform the method according to any one of claims 1 to 11.

Images