Generalized dual prediction for video coding with reduced coding complexity
By adopting generalized double prediction weight adaptation and weighted double prediction based on time layer and quantization parameters in video decoding technology, combined with the reuse of motion vector accuracy and early termination conditions, the efficiency problem of the double prediction mode under rapid illumination changes is solved. , reduces encoding complexity and signaling overhead, and improves video encoding efficiency.
Patent Information
- Application Number
- CN202410891836.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-01-24
- Filing Date
- 2019-01-23
- Publication Date
- 2024-11-08
AI Technical Summary
In the existing video decoding technology, the dual prediction mode is not efficient when dealing with rapid illumination changes, and generalized dual prediction has challenges in encoding complexity and decoding efficiency, especially in the compression process of high-resolution video signals.
By adopting generalized double prediction weight adaptation technology based on time layer and quantization parameters during video encoding and decoding, the number of available weights is reduced, and weighted double prediction is performed on the encoder and decoder side, using the sent in the bitstream Weight indexes are used to optimize the combination of predicted signals, combining reuse of motion vector accuracy and early termination conditions to reduce computational complexity.
Effectively reduce the encoding complexity of generalized double prediction and signaling overhead in the decoding process, improve video encoding efficiency, especially in the compression and decoding process of high-resolution video signals, improve decoding performance and bandwidth utilization. .
Smart Images

Figure CN118921472A_ABST
Abstract
Description
This application is a divisional application of Chinese invention patent application No. 201980016309.0, filed on January 23, 2019, entitled “Generalized dual prediction for video decoding with reduced decoding complexity”, the entire contents of which are incorporated herein by reference in their entirety. CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a non-provisional application of and claims the benefit of 35 U.S.C. §119(e) of U.S. Provisional Patent Application Serial No. 62 / 621,524, filed on January 24, 2018, entitled “GENERALIZED BI-PREDICTION FOR VIDEO CODING WITH REDUCED CODING COMPLEXITY,” the entire contents of which are incorporated herein by reference in their entirety. Background Art
[0002] Video coding systems are widely used to compress digital video signals to reduce the storage requirements and / or transmission bandwidth of such signals. Among various types of video coding systems, such as block-based systems, wavelet-based systems, and object-based systems, block-based hybrid video coding systems are the most widely used and deployed today. Examples of block-based video coding systems include multiple international video coding standards, such as MPEG-1 / 2 / 4 Part 2, H.264 / MPEG-4 Part 10 AVC, VC-1, and the latest video coding standard known as High Efficiency Video Coding (HEVC), which is developed by ITU-T / SG16 / Q.6 / VCEG and JCT-VC (Joint Collaboration Team on Video Coding) of ISO / IEC / MPEG. Summary of the invention
[0003] Exemplary embodiments include methods for use in video encoding and decoding (collectively referred to as "decoding"). A method of decoding a video comprising a plurality of pictures, the plurality of pictures comprising a current picture, a first reference picture, and a second reference picture, wherein each picture comprises a plurality of blocks. In one method, for at least one current block in the current picture, a number of available bi-prediction weights is determined based at least in part on a quality metric of the current picture, such as a temporal layer and / or a quantization parameter. A pair of weights (a first weight and a second weight) is identified from the determined number of available bi-prediction weights. The current block is then predicted as a weighted sum of a first reference block in the first reference picture and a second reference block in the second reference picture, wherein the first reference block is weighted by the first weight and the second reference block is weighted by the second weight. The same prediction may be performed on both the encoder and decoder sides, and the encoder may signal a prediction residual in a bitstream.
[0004] In some embodiments, said number of available bi-prediction weights is signaled in said bitstream.
[0005] In some embodiments, information identifying a pair of bi-prediction weights is signaled in the bitstream, and if one of the weights in the pair is greater than the other, the larger weight is used with a higher quality reference picture. In such embodiments, it is not necessary to signal in the bitstream which weight in the pair is associated with which reference picture, as this information can be inferred from the comparative qualities of the reference pictures. In some embodiments, the higher quality reference picture may be a reference picture that is closer in time to the current picture. In some embodiments, the higher quality reference picture may be a reference picture with a lower quantization parameter.
[0006] In some embodiments, the number of available bi-prediction weights depends at least in part on the similarity between the reference blocks, wherein the higher the level of similarity between the reference blocks, the lower the number of available bi-prediction weights.
[0007] In additional exemplary embodiments, techniques are described for selecting appropriate bi-prediction weights for encoding a video block without having to search all available bi-prediction weights.
[0008] Further embodiments provide encoders and decoders comprising a processor and a non-transitory computer-readable medium storing instructions operable to perform the encoding and decoding operations described herein. Additional embodiments include a non-transitory computer-readable storage medium storing a bitstream generated using the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 is a functional block diagram of a block-based video encoder.
[0010] Figure 2 It is a functional block diagram of a video decoder.
[0011] Figure 3 It is a template T used for local illumination compensation c Schematic diagram of an example of its prediction blocks T0 and T1.
[0012] Figure 4 is a functional block diagram of a video encoder according to some embodiments.
[0013] Figure 5 It is used for example Figure 4 Functional block diagram of an exemplary generalized bi-prediction module in an exemplary video encoder of an encoder.
[0014] Figure 6 is a functional block diagram of a video decoder according to some embodiments.
[0015] Figure 7 It is used for example Figure 6 Functional block diagram of a generalized bi-prediction module in an exemplary video decoder of a decoder of .
[0016] Figure 8 It is a schematic diagram of multiple time layers and a hierarchical prediction structure.
[0017] Fig. 9 is a flow chart of a generalized bi-prediction (GBi) adaptation and acceleration method employed in some embodiments.
[0018] Fig.10 is a schematic diagram of an example architecture of a two-layer scalable video encoder.
[0019] Fig.11 is a schematic diagram of an example architecture of a two-layer scalable video decoder.
[0020] Fig.12 is a diagram showing an example of a decoded bit stream structure.
[0021] Fig.13 is a schematic diagram illustrating an example communication system.
[0022] Fig.14 is a schematic diagram illustrating an example wireless transmit / receive unit (WTRU). DETAILED DESCRIPTION Overview of block-based video coding.
[0023] Figure 11 is a functional block diagram of a block-based hybrid video coding system. An input video signal 102 is processed block by block. In HEVC, extended block sizes (called "coding units" or CUs) are used to efficiently compress high-resolution (1080p and above) video signals. In HEVC, a CU can be up to 64x64 pixels. A CU can be further partitioned into prediction units or PUs, for which separate prediction methods can be applied. For each input video block (MB or CU), spatial prediction (160) and / or temporal prediction (162) can be performed. Spatial prediction (or "intra-frame prediction") uses pixels from coded neighboring blocks in the same video picture / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also known as "inter-frame prediction" or "motion compensated prediction") uses pixels from coded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given video block is typically signaled by one or more motion vectors that indicate the amount and direction of motion between the current block and its reference blocks. Furthermore, if multiple reference pictures are supported (as is the case for recent video coding standards such as H.264 / AVC or HEVC), for each video block its reference picture index is additionally sent and is used to identify which reference picture in a reference picture store (164) the temporal prediction signal comes from. After spatial and / or temporal prediction, a mode decision block (180) in the encoder selects the best prediction mode, for example based on a rate-distortion optimization method. The prediction block is then subtracted (116) from the current video block to generate a prediction residual, and the prediction residual is decorrelated using a transform (104) and quantization (106) to achieve a target bit rate. The quantized residual coefficients are inverse quantized (110) and inverse transformed (112) to form a reconstructed residual, which is then added back (126) to the prediction block to form a reconstructed video block. Further in-loop filtering, such as a deblocking filter and an adaptive loop filter, may be applied (166) to the reconstructed video block before it is placed in a reference picture store (164) and used to decode future video blocks. To form the output video bitstream 120, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit (108) to be further compressed and packed to form the bitstream.
[0024] Figure 22 is a functional block diagram of a block-based video decoder. The video bitstream 202 is first unpacked and entropy decoded in the entropy decoding unit 208. The decoding mode and prediction information are sent to the spatial prediction unit 260 (if intra-frame coding) or the temporal prediction unit 262 (if inter-frame coding) to form the prediction block. The residual transform coefficients are sent to the inverse quantization unit 210 and the inverse transform unit 212 to reconstruct the residual block. The prediction block and the residual block are then added together at 226. The reconstructed block can be further in-loop filtered before being stored in the reference picture repository 264. The reconstructed video in the reference picture repository is then sent out to drive the display device and used to predict future video blocks.
[0025] In modern video codecs, bidirectional motion compensated prediction (MCP) is known for its high efficiency in removing temporal redundancy by exploiting temporal correlations between pictures, and it has been widely used in most state-of-the-art video codecs. However, the bi-prediction signal is simply formed by combining two single prediction signals using a weight value equal to 0.5. This is not necessarily the best way to combine two single prediction signals, especially under conditions where the illumination changes rapidly from one reference picture to another. Therefore, several prediction techniques aim to compensate for illumination variations over time by applying some global or local weights and offset values to each sample value in the reference picture. Weighted bi-prediction.
[0026] Weighted bi-prediction is a coding tool primarily used to compensate for illumination variations over time, such as fading transitions, when performing motion compensation. For each slice, two sets of multiplicative weight values and additive offset values are explicitly indicated and applied separately to the motion compensated prediction, one at a time for each reference list. This technique works best when illumination varies linearly from picture to picture and the illumination variation is global at the picture / slice level. Local illumination compensation.
[0027] Local illumination compensation solves this problem by adapting parameters (two pairs of multiplication weight values and additive offset values) on a block-by-block basis. Unlike weighted bi-prediction, which indicates these parameters at the slice level, this technique takes the approach of adapting the template (T C ) and the illumination change between the reconstructed signal of the template (T0 and T1) (see Figure 3 ). By minimizing T C and T0 (for the first pair of weights and bias values) and T C The derived parameters are optimized based on the illumination difference between T1 and T2 (for the second pair of weights and offset values). The derived parameters are then used to perform the same motion compensation process as for weighted bi-prediction. Generalized bi-prediction.
[0028] Generalized bi-prediction (GBi) has been proposed to improve motion compensated prediction of bi-prediction mode. Generalized bi-prediction is described, for example, in C.-C. Chen, X. Xiu, Y. He, Y. Ye, "Generalized Bi-prediction for Inter Coding", JVET-C0047, May 2016; and International Application Publication No. WO2017197146, published on November 16, 2017, "Systems and Methods for Generalized Multi-Hypothesis Prediction for Video Coding".
[0029] In bi-prediction mode, the prediction signal at sample x is given by and can be calculated by: P[x]=w0*P0[x+v0]+w1*P1[x+v1], Formula 1 Where P[x] represents the result prediction signal of sample x at position x in the image, P i [x+v i ] is the motion vector (MV) v using the i-th list (e.g. list 0, list 1) i The motion compensation prediction signal of x, and w0 and w1 are two weight values shared among all samples in the block. Based on this formula, multiple prediction signals can be obtained by adjusting the weight values w0 and w1. Some configurations of w0 and w1 may imply the same predictions as conventional uni-prediction and bi-prediction, for example (w0, w1) = (1, 0) for uni-prediction using reference list L0, (0, 1) for uni-prediction using reference list L1, and (0.5, 0.5) for conventional bi-prediction using two reference lists. The weights are signaled for each CU. In order to reduce signaling overhead, one constraint that can be applied is w0+w1 = 1. Under this constraint, only one weight needs to be signaled. (In this way, signaling of a single weight can be referred to as signaling about a pair of weights). Therefore, Formula 1 can be further simplified to: P[x]=(1-w1)*P0[x+v0]+w1*P1[x+v1]. Formula 2
[0030] To further reduce the weight signaling overhead, w1 can be discretized into a limited number of available dual prediction weights, such as seven weights {-1 / 4, 1 / 4, 3 / 8, 1 / 2, 5 / 8, 3 / 4, 5 / 4}, so that each weight value can be indicated by an index value in a very small limited range. In this case, the available dual prediction weight pairs can be represented as (-1 / 4, 5 / 4), (1 / 4, 3 / 4), (3 / 8, 5 / 8), (1 / 2, 1 / 2), (5 / 8, 3 / 8), (3 / 4, 1 / 4), and (5 / 4, -1 / 4).
[0031] Figure 4 is a block diagram of a video encoder according to some embodiments. Figure 1 Similar to the video encoder shown in FIG, spatial prediction and temporal prediction are two basic pixel domain prediction modules. The spatial prediction module can be used with Figure 1 Same as described in, however, with Figure 1 Compared with the motion prediction module 162, Figure 4 The motion prediction module 462 provides GBi support by allowing bi-prediction to combine two separate prediction signals in a weighted average manner. In addition, the selected weight index is signaled in the bitstream. Figure 5 As shown, the exemplary GBi estimation module 500 performs a two-step process to generate the final inter-frame prediction signal. It includes a motion estimation module 502 for performing motion estimation in (one or more) reference pictures, which includes searching for two optimal motion vectors (MVs) pointing to two reference blocks. It further includes a weight value estimation module 504 for searching for an optimal weight index to minimize the weighted bi-prediction error between the current video block and the bi-prediction. Then, the prediction signal of the generalized bi-prediction is calculated as a weighted average of the two prediction blocks.
[0032] Figure 6 is decoded by e.g. Figure 4 Functional block diagram of an exemplary video decoder for a bitstream generated by a video encoder of the present invention. The decoding mode and prediction information are used to derive the prediction signal using spatial prediction or motion compensated prediction (with generalized bi-prediction support). For generalized bi-prediction, the block motion information and weight values can be received and decoded to generate the prediction block.
[0033] By using the block motion information and the weight value, the generalized bi-prediction module (eg, Figure 7 The module 750 described in ) calculates the prediction signal of the generalized bi-prediction as the weighted average of the two motion compensated prediction blocks. The generalized bi-prediction module 750 may include a motion compensation module 752 and a weighted average module 754. Hierarchical prediction and temporal layers.
[0034] Hierarchical prediction structures are widely used in random access configurations to provide efficient temporal prediction. Figure 8 An example of hierarchical prediction using four temporal layers (TL) is schematically shown, where arrows depict the prediction relationship between a current picture and its reference pictures (arrows start from the reference picture and go to the current picture being predicted). As shown, in hierarchical prediction, higher TL pictures are usually predicted based on temporally closer reference pictures (e.g., TL-3 pictures are predicted based on temporally neighboring pictures), while lower TL layer pictures have a larger temporal distance from their reference pictures (e.g., TL0 pictures with picture order count (POC) or POC8 are 8 pictures away from their reference pictures with picture order count POC0). Also note that at the highest TL ( Figure 8 TL-3 in ), the pictures are not used as reference pictures by any other pictures, i.e., they are non-reference pictures. In this embodiment, four TLs are used. A deeper level can be used, for example, the JVET Common Test Condition (CTC) uses five TLs in its random access configuration.
[0035] In hierarchical prediction, the picture / slice level quantization parameter (QP) value can be adjusted according to the TL that the current picture is in. Specifically, if QP0 is used for TL0 picture, QP0+delta(TLx) can be used for the picture at TLx, where delta() is a function based on the temporal level and delta() is zero or a positive integer. In one example, delta(TLx) can be set to TLx. Overview of exemplary embodiments.
[0036] On the encoder side, deriving the best motion vector and the best GBi weight may significantly increase the computational complexity. In order to find the best weight / motion vector combination, the encoder can operate to iteratively refine the two MVs and the weights for dual prediction, i.e., v0, v1, and w1 in Formula 2. This process is performed at the CU level, and if an exhaustive search is used, the computational complexity will become extremely high. For some CUs, testing all seven weights {-1 / 4, 1 / 4, 3 / 8, 1 / 2, 5 / 8, 3 / 4, 5 / 4} may not be effective. For example, referring to Formula 2, when the two predictions P0 and P1 of the current CU to be decoded are similar, no matter what weight value is applied, the final dual prediction will be similar. This shows that the encoder does not have to test so many weight options and signal the best weight selected from so many weights. On the contrary, in the case where P0 and P1 are very similar, allowing more weights will result in more signaling overhead for the weight, which may reduce the decoding efficiency of GBi.
[0037] Exemplary embodiments disclosed herein are directed to improving decoding efficiency and reducing coding complexity of GBi.
[0038] In some embodiments, the number of available GBi weights is reduced based on the temporal layer, the temporal distance of two reference pictures, or the relative quality of two prediction signals.
[0039] In some embodiments, the encoding complexity of GBi is reduced by reusing GBi weights for different motion vector precisions.
[0040] Some embodiments include techniques for determining when to terminate the process of determining GBi weights to reduce encoding complexity.
[0041] Some embodiments include techniques for quickly determining GBi weights based on motion estimation costs.
[0042] This disclosure describes techniques for reducing the complexity of generalized bi-prediction. These techniques can be used Figure 4 The encoder and Figure 6 Some embodiments are used to reduce GBi weight signaling overhead. Some embodiments are operable to reduce the complexity of GBi weight selection. Picture / slice level GBi adaptation based on TL and QP.
[0043] As discussed above, QP adaptation is widely used in hierarchical prediction. When QP adaptation is used in conjunction with bi-prediction, if two reference pictures are from two temporal layers, they may have different qualities. Part of the GBi gain comes from this, because a larger weight can be applied to the reference block from the reference picture with higher quality (e.g., lower QP) to get a better final prediction. With this consideration, the GBi weights can be adapted when decoding pictures at different temporal layers.
[0044] In a first embodiment, the number of allowed GBi weights can be adapted according to the TL of the current picture. The allowed number of GBi weight indices of temporal layer i is denoted as c i For a coding structure with k temporal layers, the number of weights that can be used for each temporal layer can be expressed as {c0,c1,…,c k-1}. Typically, lower temporal layer pictures have higher quality and will be used more frequently as reference pictures to decode other pictures (from the same or higher temporal level). Therefore, the decoding quality of lower TL pictures is more important in achieving better overall decoding efficiency. Therefore, in some embodiments, lower TL pictures can use a larger number of GBi weights in order to improve the prediction quality and decoding quality of lower TL pictures. For example, the condition c0≤c1≤…≤c k-1, so that there is a larger number of weight choices for lower TL pictures. At higher temporal layers, the number of allowed GBi weights is reduced compared to lower temporal layers. This can not only reduce the signaling overhead of higher TL pictures, but also speed up the encoding of higher TL pictures. For example, the lowest one or two temporal layers (e.g., TL0 and / or TL1) can allow seven different sets of weights, the higher intermediate temporal layers (e.g., TL2) can allow only three different sets of weights, and at the highest temporal level (e.g., TL4) only three different sets of weights can be allowed. Figure 8 At TL3 in , for non-reference pictures, only conventional bi-prediction may be allowed (ie, only one set of (0.5, 0.5) weights is allowed).
[0045] The GBi gain of blocks in high temporal layer pictures may be relatively small compared to blocks in lower temporal layer pictures. This is because high temporal layer pictures usually have reference pictures that are very close in temporal distance, so even without GBi, blocks in these high TLs tend to be more accurately or fully predicted. In addition, higher QPs are usually used for high temporal layer pictures, which leads to more skipped blocks and / or zero residual blocks. Therefore, in addition to adapting GBi based on the TL of the current picture, the QP value of the current picture can also be considered in GBi adaptation. For example, if the slice / picture level QP of the current picture is high enough, only conventional dual prediction with equal weights is allowed (or a smaller set of GBi weights is allowed). This may bring some quality degradation, but the loss can be compensated by reducing signaling overhead. Since GBi is turned off or a smaller set of GBi weights is used, GBi index signaling can be skipped, or GBi indexes can be decoded using fewer bits. An embodiment using slice-level QP-based GBi adaptation can be used in combination with the above-mentioned TL-based GBi adaptation. If the TL and / or slice level QP of the current picture is high enough, the number of allowed GBi weights can be reduced, or GBi can be disabled completely (e.g., only equal weights are allowed for bi-prediction). At the decoder side, the temporal layer and / or slice QP values are known to the decoder. Therefore, if the decoder determines that GBi has been turned off based on the TL and / or slice QP values, it can skip the GBi weight information parsing at the CU level. By skipping the GBi process for certain temporal layer and / or slice QP values, GBi can be accelerated with negligible performance loss.
[0046] In another embodiment, instead of determining the number of available weights for the current picture by using the TL of the current picture and / or utilizing the QP of the current picture, the encoder may adaptively select the number of GBi weights it uses when encoding the current picture. In this case, the encoder may signal in the bitstream the number of GBi weights allowed at the slice / picture level and the actual allowed GBi weights. This advanced signaling of the allowed GBi weights may be signaled as part of the slice header, or may be signaled in the picture parameter set PPS. At the CU level, only the weights allowed for the current slice / picture may be used, and only the GBi indices associated with these allowed weights may be signaled. Block-level GBi adaptation based on reference signal quality.
[0047] As described above, some embodiments adapt the number of GBi weights at the picture level based on TL and / or slice level QP.In addition to or as an alternative to adjusting GBi weights at the picture level, block level GBi adjustment may also be employed.
[0048] In bi-prediction, the current block is predicted from two reference pictures, one for each reference picture list. Then, in GBi, the two reference signals P0 and P1 are combined using Formula 2, and in some embodiments, the GBi weights may be adapted at the block level based on the relative quality of the two prediction signals P0 and P1. Instead of allowing a full set of, for example, seven weights in GBi for the current block, the allowable weights may be restricted in such a way that only larger weights are used for predictions with higher quality, and only smaller weights are used for predictions with lower quality. For example, in unconstrained GBi, w1 is the GBi weight for P1, which may have, for example, seven values, which may be {-1 / 4, 1 / 4, 3 / 8, 1 / 2, 5 / 8, 3 / 4, 5 / 4}. For the current block coded in bi-prediction mode, if P1 has a higher quality than P0, then for w1, only a subset of w1 values greater than or equal to (1 / 2) may be allowed. For example, for the current block, only the subset {1 / 2, 5 / 8, 3 / 4, 5 / 4} may be allowed. On the other hand, if P0 has a higher quality than P1, only a subset of w1 values less than or equal to (1 / 2) may be allowed. For example, for the current block, only the subset {-1 / 4, 1 / 4, 3 / 8, 1 / 2} may be allowed. In this way, the GBi weight signaling overhead may be reduced and encoding may also be accelerated.
[0049] Different embodiments may use different techniques to estimate the relative quality of P0 and P1. In a first embodiment, the temporal distance between the current slice and each of the reference pictures in its list 0 and list 1 can be used to estimate the relative quality of P0 and P1. Specifically, the prediction signal from the reference picture that is closer to the current picture in time is more likely to have a higher correlation and therefore provide a higher quality prediction. R0 and R1 represent the reference pictures from which P0andP1 are obtained, respectively. And d0andd1 represent the temporal distance between R0andR1 and the current picture, respectively. The temporal distance between two pictures can be measured by the difference in their picture sequence counts. If d0≤d1, then P0 is likely to have a higher quality than P1, and therefore the w1 value should be limited to a value less than or equal to (1 / 2), that is, only a larger weight is assigned to P0. In some embodiments, in the case of equal temporal distance d0=d1, all weight pairs can be considered.
[0050] In a second embodiment, the QP values associated with P0 and P1 may be considered when determining the relative quality. This may be a slice-level QP value for decoding reference pictures R0 and R1, or it may be a block-level QP value for decoding prediction blocks P0 and P1, which may occur, for example, when a block-level QP change is applied when decoding reference pictures R0 and R1. Because fractional-pel precision motion vectors may be used for motion compensated prediction to obtain P0 and P1, and P0 and P1 may correspond to more than one block partition in R0 and R1. Take P0 as an example. If it covers N block partitions in R0, and if each of these N block partitions is decoded using a different block-level QP, the block-level QP associated with P0 may be calculated using the average of these block-level QPs. QP0 and QP1 represent the QP values associated with P0 and P1, respectively. Without loss of generality, assume that QP0≤QP1. Then P0 is likely to have a higher quality than P1, so the w1 value should be restricted to values less than or equal to 1 / 2, ie, only assign a larger weight to P0. In some embodiments, all weight pairs may be considered in case of equal QP values.
[0051] In a third embodiment, the temporal distances of P0 and P1 from the current picture and the QPs associated with P0 and P1 may be used in combination to determine the number of allowed GBi weights for the current block. For example, if d0≤d1 and QP0≤QP1, the allowed GBi weights for the current block may be further constrained to support larger weights for P0. For example, in this case, only the largest two weights may be used for P0. Conversely, of the full set of, for example, seven weight values for w1, only the two smallest values {-1 / 4, 1 / 4} are allowed for the current block.
[0052] Similar to picture-level GBi adaptation, block-level GBi adaptation can also be used to accelerate encoding. During the motion estimation stage, dual-prediction motion search can be performed as follows: select one reference picture from each reference picture list, search for the best motion vector for the reference picture selected from one list, fix the motion vector for the reference picture selected from this list, and refine the motion vector for the reference picture selected from the other list. During this stage, in an exemplary embodiment, two temporal distances of P0 and P1 and / or the QP values associated with P0 and P1 can be calculated to determine a subset of the allowable GBi weights for the current block. During the encoder search, if a larger weight is being applied to a reference block from a more distant reference picture (and thus is not allowed for the current block), then the dual-prediction motion search can be stopped for this reference picture pair, and the process can continue to the next reference picture pair. GBi adaptation based on the similarity between two prediction signals.
[0053] GBi prediction is calculated using Equation 2, where two prediction signals P0 and P1 are weighted averaged. When P0 and P1 are sufficiently similar, applying a larger set of weights will not be very helpful because the final predictions will be very similar to each other regardless of how different the weights are. Therefore, in some embodiments, the set of GBi weights can be constrained by evaluating the similarity between P0 and P1 and determining the similarity level between P0 and P1. In some embodiments, the similarity level is determined by calculating the sum of absolute differences (SAD) value between P0 and P1. If the SAD value is less than a predetermined threshold, unequal GBi weights can be skipped for the current block, and only traditional dual-prediction can be allowed. Other similarity metrics, such as sum of squared errors (SSE), can also be used. In some embodiments, a hierarchical constraint method can be applied. For example, a set of two thresholds can be used, where T0 < T1. If SAD(P0,P1) < T0, only conventional dual-prediction (with equal weights) is allowed. If T0 ≤ SAD(P0,P1) < T1, only a smaller set of GBi weights is allowed, e.g., a constrained set containing three weights {1 / 4, 1 / 2, 3 / 4} (in this case, the two most biased weights and the equal weight). Finally, if T1 ≤ SAD(P0,P1), a full set of, e.g., seven GBi weights is allowed.
[0054] In some embodiments, the similarity-based GBi adaptation can be used in combination with the above-described picture-level or block-level GBi adaptation. For example, in the above hierarchical constraint method, if T0 ≤ SAD(P0, P1) < T1, the set of three weights of the constraint can be further constrained according to the quality of P0 and P1. Without loss of generality, assume that P0 has higher quality than P1 (e.g., it comes from a temporally closer reference picture, and / or the QP associated with P0 is lower), then the set of three weights can be further constrained to a set of two weights including {1 / 4, 1 / 2} that are allowed for w1. Encoder acceleration: Reuse of GBi weights for different motion vector precisions.
[0055] Currently in JEM, for explicit (i.e., non-merged) inter prediction modes, the motion vectors can have different precisions: 1 / 4 pixel precision, 1 pixel precision, and 4 pixel precision. Allowing more motion vector precisions at the block level provides more trade-off points between bitrate and quality. Without encoder acceleration techniques, taking the seven-weight configuration as an example, with three different motion vector precisions, for dual prediction, the motion search and rate-distortion processing will be performed 21 times, which will result in very high encoder complexity.
[0056] Generally, the final motion vectors determined by explicit inter prediction modes with different motion vector precisions are very similar to each other. As a result, the best reference block determined by one motion vector precision may be almost the same as those from other motion vector precisions. From this observation, it is reasonable to reuse the best GBi weights from one motion precision for other motion vector precisions.
[0057] In one embodiment, according to the current implementation of the JEM software, the 1 / 4 pixel precision is always tested first in the motion search of the explicit inter prediction mode. In an exemplary embodiment, after cycling through all GBi weights to find the best GBi weight for the 1 / 4 pixel precision case, for the motion search of the remaining motion vector precisions, other GBi weights can be skipped. For example, subsequent searches can test only the traditional (1 / 2, 1 / 2) weight pair plus the best GBi weight selected by the 1 / 4 pixel precision search. By doing so, the number of GBi weight iterations can be reduced, thus reducing the encoder complexity. In an alternative embodiment, the coding commands regarding the motion vector precision can be changed, and thus the reuse can be flexible for other combinations, e.g., the best GBi weight selected by 1 pixel can be used for 1 / 4 pixel and 4 pixel coding.
[0058] In another embodiment, rather than reusing only the best GBi weight for another motion vector precision, the best two or N weights are selected as candidates for another motion vector precision to mitigate the potential loss that may have been inflicted on the encoder by this simplified approach. A threshold may be used to determine whether another GBi weight other than the best GBi weight is still tested by the other motion vector precision. If the rate-distortion cost is less than the threshold calculated based on the best GBi weight cost, then that particular GBi weight will also be reused for the other motion vector precision.
[0059] The embodiment of reusing the optimal GBi weights can be combined with the above-mentioned GBi adaptation method to further improve the flexibility of the encoder, thereby finding a better compromise between complexity and performance. By doing so, the complexity of the GBi encoding process can be reduced while still providing high performance. Encoder speedup: Early termination of GBi process.
[0060] In some embodiments, the GBi weight search process is simplified by early termination. Two types of early termination conditions are described below.
[0061] One early termination condition used in some embodiments is based on the partition depth of the current block. In the current design of JEM, a quadtree binary tree CU hierarchy is adopted, in which the coding tree unit (CTU) is first divided using quadtree partitions. The quadtree leaf node is used as the root of the binary tree (BT). Generally, when the CU size is very small, the GBi effect is poor. Therefore, in some embodiments, the current BT depth is checked to decide whether to terminate the GBi search process in advance. Regardless of the BT depth, equal weights are always applied to motion search. For unequal weight pairs, if it is found that the current BT depth is already greater than a threshold, the remaining GBi weights can be skipped in the motion search. Instead of using the BT depth, the BT size can also be used to determine whether non-equal GBi weights will be tested in the motion search. If the BT size (e.g., BT width multiplied by BT height) is less than a predetermined threshold, non-equal GBi weights can be skipped.
[0062] In some embodiments, the method can also be used to standardize the disabling of GBi for the current BT if the current BT has a depth greater than a preset threshold and / or if its size is less than a preset threshold. When the method is used standardly, the decoder can be operated to skip parsing GBi weights for such blocks.
[0063] Another early termination condition used in some embodiments is based on the decoded block flag (CBF). The CBF flag is used in the JEM codec to indicate whether the residual error calculated by subtracting the prediction block from the current block will be encoded and signaled to the decoder. Generally, when the encoder decides to skip the residual error, it indicates that the prediction is good enough because all quantized transform coefficients can be set to 0. Taking advantage of this, in some embodiments, if the encoder has found a GBi weight (equal or unequal) for which the CBF flag can be set to 0 (indicating that the residual error does not need to be decoded), the check of the remaining GBi weights in the motion search process is terminated because this indicates that a valid weight pair for the current block has been found.
[0064] There may be two cases where the encoder will decide to set the CBF to 0. The first case is when the encoder tries a mode where the CBF is set to 0 in an attempt to reduce overhead. The second case is when all transform coefficients are effectively 0 after quantization of the transform coefficients. From a performance perspective, early termination of the GBi weight test due to the first case may result in performance degradation. Therefore, in some embodiments, early termination based on the CBF flag may be performed only for the second case. Encoder acceleration: GBi simplification based on merge mode.
[0065] In the current JEM implementation, the explicit inter prediction mode where the GBi weight index search occurs is located after all other inter prediction modes. With this design, in some embodiments, information from other inter prediction modes can be used to speed up the GBi search process. More specifically, if the best inter prediction mode is a skip merge candidate when the explicit inter prediction mode is about to be tested, and the GBi weights from the skip merge mode are unequal weights (the GBi weights for a skip merge block are inherited from the selected skip merge candidate), then there is no need to check the unequal weights of the explicit inter prediction mode, because using the GBi weights merged from spatial or temporal neighboring blocks can already provide very good predictions without the need to transmit residuals.
[0066] In another embodiment, to further ensure the performance of GBi, during the explicit inter prediction mode test, in addition to equal weights, we may also want to test the GBi weights that are the same as the merge weights from the skip merge mode.
[0067] In another embodiment, during the explicit inter prediction mode test, all GBi weights (including equal weights) may be tested except for the GBi weight that is the same as the merge weight. Encoder acceleration: GBi simplification based on motion estimation cost.
[0068] The current GBi implementation is a rate-distortion based implementation, where after motion estimation, the rate-distortion (RD) cost of encoding the current block using each of the possible GBi weights is calculated. Then, the GBi weight with the minimum RD cost is selected. The RD process is relatively complex because it involves transformation and quantization. Therefore, in some embodiments, the complexity of GBi can be further reduced by selecting the best GBi weight at the motion estimation stage.
[0069] In the motion estimation stage, the bit rate has been estimated for uni-prediction and bi-prediction in order to decide whether to use bi-prediction. In addition, the SAD or the sum of absolute transform differences (SATD) is also calculated (depending on the motion vector accuracy, SAD or SATD can be used) as distortion. Using the estimated bit rate and distortion, the cost can be calculated. In an exemplary embodiment, the cost from the motion estimation stage can be used to select the best GBi weight (i.e., the GBi weight is selected as the weight with the lowest SAD or SATD). By doing this, only one RD cost calculation needs to be performed, thereby reducing the complexity of GBi.
[0070] Fig. 9 is a flowchart summarizing the proposed GBi adaptation and acceleration method. In order to provide a better trade-off between the signaling overhead and performance of GBi and to accelerate encoding, various embodiments are disclosed in the present disclosure. These embodiments can be used in various combinations. Fig. 9 In the example of , several embodiments are combined. Given a complete set of GBi weights, picture-level GBi weight adaptation can be applied (902), for example, based on the TL, QP, or encoder decision of the current picture, and a reduced set of GBi weights can be selected for the current picture. Then, when decoding each block (904) in the current picture, block-level GBi weight adaptation (906) can be applied, for example, based on the temporal distance of two prediction blocks, their relative quality, and / or their similarity to each other, and a further reduced set of GBi weights can be selected for the current block. Then, within the set of allowed GBi weights for the current block, the encoder can use the acceleration method described herein (e.g., based on the depth of the current block, the partition size, the CBF flag, etc.) to select the best decoding mode (908) for the current block, which includes the best motion vector, reference index, and the best corresponding GBi weight.
[0071] Note that although Fig. 9A flow chart is depicted in which several disclosed embodiments are used, but a subset of the disclosed embodiments may alternatively be used. For example, the picture-level GBi adaptation (902) may be bypassed, and a complete set of GBi weights may be fed directly into the block-level GBi adaptation block. Other combinations are possible, for example, encoder acceleration (908) may not be applied, and instead, an exhaustive search for optimal GBi weights may be performed.
[0072] Although many embodiments are described herein as starting with a set of seven GBi weights {-1 / 4, 1 / 4, 3 / 8, 1 / 2, 5 / 8, 3 / 4, 5 / 4} as a complete set of GBi weights, the proposed method is generally applicable to any set of starting GBi weights (which may include a different number of weights) and / or a different set of weights. Overview of Scalable Video Coding
[0073] As described above, some embodiments employ multi-layer video coding techniques. For example, multi-layer video coding techniques such as scalable video coding and / or multi-view video coding may be designed to process more than one video layer, wherein each layer may be decoded to reconstruct a video signal of a specific spatial resolution, temporal resolution, fidelity, and / or view.
[0074] Scalable video coding can improve the quality of experience of video applications running on devices with different capabilities on heterogeneous networks. Scalable video coding can encode the signal once at the highest representation (e.g., temporal resolution, spatial resolution, quality, etc.), but enable decoding from a subset of the video stream according to the specific rate and representation required by certain applications running on the client device. Compared with non-scalable solutions, scalable video decoding can save bandwidth and / or storage. International video standards (e.g., MPEG-2 video, H.263, MPEG4 vision, H.264, etc.) can have a variety of tools and / or profiles that support scalability modes.
[0075] Table 1 provides examples of different types of scalability and corresponding standards that can support them. Bit depth scalability and / or chroma format scalability can be associated with video formats (e.g., higher than 8-bit video, and chroma sampling formats higher than YUV4:2:0), which, for example, may be primarily used by professional video applications. Aspect ratio scalability can be provided. Table 1
[0076] Scalable video coding may use a base layer bitstream to provide a first video quality level associated with a first set of video parameters. Scalable video coding may use one or more enhancement layer bitstreams to provide one or more higher quality levels associated with one or more enhancement parameters. The set of video parameters may include one or more of the following: spatial resolution, frame rate, reconstructed video quality (e.g., in the form of SNR, PSNR, VQM, visual quality, etc.), 3D capabilities (e.g., with two or more views), luminance and chrominance bit depths, chrominance formats, and underlying single-layer coding standards. Different use cases may use different types of scalability, for example, as shown in Table 1. A scalable coding architecture may provide a common structure that may be configured to support one or more scalabilities (e.g., the scalability listed in Table 1). The scalable coding architecture may be flexible so as to support different scalabilities with minimal configuration effort. A scalable coding architecture may include at least one preferred operation mode, which may not require changes to block-level operations, so that coding logic (e.g., encoding logic and / or decoding logic) can be reused to the maximum extent in a scalable coding system. For example, a scalable coding architecture based on picture-level inter-layer processing and management units may be provided, wherein the inter-layer prediction may be performed at the picture level.
[0077] Fig.10 is a schematic diagram of an example architecture of a two-layer scalable video encoder. A video encoder 950 may receive a video (e.g., an enhancement layer video input). A downsampler 952 may be used to downsample the enhancement layer video to create a lower-level video input (e.g., a base layer video input). The enhancement layer video input and the base layer video input may correspond to each other via the downsampling process, and spatial scalability may be achieved. A base layer encoder 954 (e.g., an HEVC encoder in this example) may encode the base layer video input block by block and generate a base layer bitstream. Figure 1 Can be used as Fig.10 Schematic diagram of an exemplary block-based single-layer video encoder for the base layer encoder in.
[0078] At the enhancement layer, an enhancement layer (EL) encoder 956 may receive the EL input video input, which may have a higher spatial resolution (e.g., and / or higher values of other video parameters) than the base layer video input. The EL encoder 956 may generate an EL bitstream in a substantially similar manner to the base layer video encoder 954, such as by using spatial prediction and / or temporal prediction to achieve compression. Inter-layer prediction (ILP) may be available at the EL encoder 956 to improve its decoding performance. Unlike spatial prediction and temporal prediction, which may derive a prediction signal based on a decoded video signal in the current enhancement layer, inter-layer prediction may derive a prediction signal based on a coded video signal from a base layer (e.g., and / or other lower layers when there are more than two layers in a scalable system). At least two forms of inter-layer prediction (picture-level ILP and block-level ILP) may be used in the scalable system. Picture-level ILP and block-level ILP are discussed herein. A bitstream multiplexer 958 may combine the base layer bitstream and the enhancement layer bitstream together to generate a scalable bitstream.
[0079] Fig.11 is a schematic diagram of an example architecture of a two-layer scalable video decoder. Fig.11 The two-layer scalable video decoder architecture may correspond to Fig.10 The video decoder 1000 may receive a scalable bitstream from, for example, a scalable encoder (e.g., scalable encoder 900). The demultiplexer 1002 may separate the scalable bitstream into a base layer bitstream and an enhancement layer bitstream. The base layer decoder 1004 may decode the base layer bitstream and may reconstruct the base layer video. Figure 2 Can be used as Fig.11 Schematic diagram of an exemplary block-based single-layer video decoder for the base layer decoder in .
[0080] The enhancement layer decoder 1006 may decode the enhancement layer bitstream. The EL decoder 1006 may decode the EL bitstream in a substantially similar manner as the base layer video decoder 1004. The enhancement layer decoder may use information from the current layer and / or information from one or more dependent layers (e.g., the base layer) to do so. For example, such information from one or more dependent layers may be processed inter-layer, which may be accomplished when using picture-level ILP and / or block-level ILP. Although not shown, additional ILP information may be multiplexed with the base and enhancement layer bitstreams at the MUX 958. The ILP information may be demultiplexed by the DEMUX 1002. An embodiment using a temporal layer to determine the number of available bi-prediction weights.
[0081] In some embodiments, a method of decoding a video including a plurality of pictures is provided, the plurality of pictures including a current picture, a first reference picture, and a second reference picture, each picture including a plurality of blocks. The method includes, for at least a current block in the current picture: determining a number of available bi-prediction weights based at least in part on a temporal layer of the current picture; identifying a first weight and a second weight from the determined number of available bi-prediction weights; and predicting the current block as a weighted sum of a first reference block in the first reference picture and a second reference block in the second reference picture, wherein the first reference block is weighted by the first weight and the second reference block is weighted by the second weight.
[0082] In some embodiments, the number of bits used to identify the bi-prediction weights in the bitstream is based on the determined number of available bi-prediction weights. In some embodiments, an index identifying at least the first weight is signaled in the bitstream only when the number of available bi-prediction weights is greater than one. In some embodiments, the sum of the first and second weights is one. In some such embodiments, the number of available bi-prediction weights is reduced for increased temporal layers. In some embodiments, seven bi-prediction weights may be used for at least one lowest temporal layer. In some embodiments, three bi-prediction weights may be used for at least one temporal layer above the lowest temporal layer. In some embodiments, one bi-prediction weight of 0.5 may be used for at least one highest temporal layer.
[0083] In some embodiments, a method for encoding a video comprising a plurality of pictures is provided, the plurality of pictures comprising a current picture, a first reference picture, and a second reference picture, each picture comprising a plurality of blocks. The method comprises, for at least a current block in the current picture: determining, based at least in part on a temporal layer of the current picture, a number of available bi-prediction weights; selecting, from among the determined number of available bi-prediction weights, a first weight and a second weight for prediction of the current block; and signaling in a bitstream an index identifying at least the first weight only if the determined number of the available bi-prediction weights is greater than one. In some such methods, the number of bits used to signal the index is based on the determined number of available bi-prediction weights.
[0084] In some embodiments, a method for decoding a video from a bitstream is provided, wherein the video comprises a plurality of pictures, the plurality of pictures comprising a current picture, a first reference picture, and a second reference picture, each picture having a plurality of blocks. The method comprises, for at least a current block in the current picture: determining, based at least in part on a temporal layer of the current picture, a number of available bi-prediction weights; and parsing from the bitstream an index identifying at least a first bi-prediction weight only if the determined number of available bi-prediction weights is greater than one. In some such embodiments, the number of bits used to signal the index is based on the determined number of available bi-prediction weights. Embodiments in which a quantization parameter is used to determine the number of available weights.
[0085] In some embodiments, a method of decoding a video including multiple pictures is provided, the multiple pictures including a current picture, a first reference picture, and a second reference picture, each picture including multiple blocks. The method includes, for at least a current block in the current picture: determining a number of available bi-prediction weights based at least in part on a quantization parameter of the current picture; identifying a first weight and a second weight from the determined number of available bi-prediction weights; and predicting the current block as a weighted sum of a first reference block in the first reference picture and a second reference block in the second reference picture, wherein the first reference block is weighted by the first weight and the second reference block is weighted by the second weight.
[0086] In some such embodiments, the number of bits in the bitstream used to identify the bi-prediction weights is based on the determined number of available bi-prediction weights. In some embodiments, an index identifying at least the first weight is signaled in the bitstream only if the number of available bi-prediction weights is greater than one. In some embodiments, the sum of the first and second weights is one. In some embodiments, the number of available bi-prediction weights decreases for an increasing quantization parameter.
[0087] In some embodiments, seven bi-prediction weights may be used for at least one lowest value of the quantization parameter. In some embodiments, three bi-prediction weights may be used for at least one value of the quantization parameter above the lowest value of the quantization parameter. In some embodiments, one bi-prediction weight of 0.5 may be used for at least one highest value of the quantization parameter.
[0088] In some embodiments, a method for encoding a video comprising a plurality of pictures is provided, the plurality of pictures comprising a current picture, a first reference picture, and a second reference picture, each picture comprising a plurality of blocks. The method comprises, for at least a current block in the current picture: determining, based at least in part on a quantization parameter of the current picture, a number of available bi-prediction weights; selecting, from among the determined number of available bi-prediction weights, a first weight and a second weight for prediction of the current block; and signaling in a bitstream an index identifying at least the first weight only if the determined number of available bi-prediction weights is greater than one. In some such embodiments, the number of bits used to signal the index is based on the determined number of available bi-prediction weights.
[0089] A method of decoding a video from a bitstream, the video comprising a plurality of pictures, the plurality of pictures comprising a current picture, a first reference picture, and a second reference picture, each picture comprising a plurality of blocks. The method comprises, for at least a current block in the current picture: determining, based at least in part on a quantization parameter of the current picture, a number of available bi-prediction weights; and parsing from the bitstream an index identifying at least a first bi-prediction weight only if the determined number of available bi-prediction weights is greater than one. In some such embodiments, the number of bits used to signal the index is based on the determined number of available bi-prediction weights. An embodiment using a quality metric to determine the number of available weights.
[0090] In some embodiments, a method for decoding a video including a plurality of pictures is provided, the plurality of pictures including a current picture, a first reference picture, and a second reference picture, each picture including a plurality of blocks. The method comprises, for at least a current block in the current picture: determining a number of available bi-prediction weights based at least in part on a quality metric of the current picture; identifying a first weight and a second weight from the determined number of available bi-prediction weights; and predicting the current block as a weighted sum of a first reference block in the first reference picture and a second reference block in the second reference picture, wherein the first reference block is weighted by the first weight and the second reference block is weighted by the second weight.
[0091] In some such embodiments, the quality metric is determined by a temporal layer of the current picture. In some embodiments, the quality metric is determined by a quantization parameter of the current picture. In some embodiments, the quality metric is determined by a quantization parameter and a temporal layer of the current picture.
[0092] In some embodiments, a method for decoding a video comprising a plurality of pictures is provided, the plurality of pictures comprising a current picture, a first reference picture, and a second reference picture, each picture comprising a plurality of blocks. The method comprises, for at least a current block in the current picture: determining a number of available bi-prediction weight pairs based at least in part on a quality metric of the current picture; identifying a weight pair from the determined number of available bi-prediction weight pairs, the identified weight pair comprising a higher weight and a lower weight; determining which of the first and second reference pictures is a higher quality reference picture and which is a lower quality reference picture; and predicting the current block as a weighted sum of a higher quality reference block in the higher quality reference picture and a lower quality reference block in the lower quality reference picture, wherein the higher quality reference block is weighted by the higher weight and the lower quality reference block is weighted by the lower weight. An embodiment of signaling the number of available weights.
[0093] In some embodiments, a method of decoding a video comprising a plurality of pictures is provided, the plurality of pictures comprising a current picture, a first reference picture, and a second reference picture, each picture comprising a plurality of blocks. The method comprises, for at least a current block in the current picture: determining a number of available bi-prediction weights, wherein the number of available bi-prediction weights is signaled in a bitstream; from the determined number of available bi-prediction weights, identifying a first weight and a second weight; and predicting the current block as a weighted sum of a first reference block in the first reference picture and a second reference block in the second reference picture, wherein the first reference block is weighted by the first weight and the second reference block is weighted by the second weight. In some embodiments, the number of bits used to identify the bi-prediction weights in the bitstream is based on the determined number of available bi-prediction weights. In some embodiments, an index identifying at least the first weight is signaled in the bitstream only if the number of available bi-prediction weights is greater than one. An embodiment of assigning weights based on reference picture quality.
[0094] In some embodiments, a method for encoding a video comprising a plurality of pictures is provided, the plurality of pictures comprising a current picture, a first reference picture, and a second reference picture, each picture comprising a plurality of blocks. The method comprises, for at least a current block in the current picture: determining which of the first and second reference pictures is a higher quality reference picture and which is a lower quality reference picture; identifying a first weight and a second weight from a plurality of available dual prediction weights, wherein the second weight is not greater than the first weight; and predicting the current block as a weighted sum of a higher quality reference block in the higher quality reference picture and a lower quality reference block in the lower quality reference picture, wherein the higher quality reference block is weighted by the first weight and the lower quality reference block is weighted by the second weight.
[0095] In some such embodiments, determining which of the first and second reference pictures is the higher quality reference picture and which is the lower quality reference picture comprises selecting one of the first and second reference pictures that is temporally closer to the current picture as the higher quality reference picture. In some embodiments, each of the first and second reference pictures has an associated quantization parameter, and wherein determining which of the first and second reference pictures is the higher quality reference picture and which is the lower quality reference picture comprises selecting one of the first and second reference pictures that has a lower associated quantization parameter as the higher quality reference picture. The associated quantization parameter may be a slice-level quantization parameter of the first and second reference pictures. The associated quantization parameter may be a block-level quantization parameter of a corresponding reference block in the first reference picture and the second reference picture. The block-level quantization parameter of the corresponding reference block may be determined by averaging the block-level quantization parameters of a block partition used to decode the corresponding reference block.
[0096] Some embodiments include determining a number of available bi-prediction weights, wherein the number of available bi-prediction weights is determined based at least in part on a quality difference between the higher quality and lower quality reference pictures. A larger quality difference may correspond to a lower number of available bi-prediction weights. The quality difference may be determined based on both a respective associated quantization parameter of a respective reference picture and a temporal distance of the respective reference picture from the current picture.
[0097] In some embodiments, a level of similarity between the first reference block and the second reference block is determined; and based at least in part on the level of similarity, a number of available bi-prediction weights is determined, wherein the identifying of the first weight and the second weight is performed from among the determined number of available bi-prediction weights. In some embodiments, a number of available bi-prediction weights is determined based at least in part on a temporal layer of the current picture, wherein the identifying of the first weight and the second weight is performed from among the determined number of available bi-prediction weights. In some embodiments, a number of available bi-prediction weights is determined based at least in part on a quantization parameter of the current picture, wherein the identifying of the first weight and the second weight is performed from among the determined number of available bi-prediction weights.
[0098] In some embodiments, the first and second weights are identified using an index signaled in a bitstream. In some such embodiments, the index identifies an unordered weighted pair selected from the group consisting of {5 / 4, -1 / 4}, {3 / 4, 1 / 4}, {5 / 8, 3 / 8}, and {1 / 2, 1 / 2}. An embodiment of using picture similarity to determine the number of available weights.
[0099] In some embodiments, a method of decoding a video including a plurality of pictures is provided, the plurality of pictures including a current picture, a first reference picture, and a second reference picture, each picture including a plurality of blocks. The method includes, for at least a current block in the current picture: identifying a first reference block in the first reference picture and a second reference block in the second reference picture; determining a similarity level between the first reference block and the second reference block; determining a number of available bi-prediction weights based at least in part on the similarity level; identifying a first weight and a second weight from the determined number of available bi-prediction weights; and predicting the current block as a weighted sum of a first reference block in the first reference picture and a second reference block in the second reference picture, wherein the first reference block is weighted by the first weight and the second block is weighted by the second weight. In some such embodiments, determining the similarity level includes determining a sum of absolute differences between the first reference block and the second reference block.
[0100] In some embodiments, in response to determining that the similarity level is above a threshold, the first weight and the second weight are both 0.5. In some embodiments, for at least one minimum similarity level, the number of available bi-predictive weights is seven. In some embodiments, for at least one similarity level above the minimum similarity level, the number of available bi-predictive weights is three. An embodiment using a multi-stage motion search.
[0101] In some embodiments, a method of encoding video in a bitstream is provided. The method includes, for at least one current block in a current picture: using a first level of precision, performing a first bi-prediction motion search to select a first set of motion vectors and a first pair of bi-prediction weights, the first search determining a set of N pairs of best bi-prediction weights; using at least a second level of precision different from the first level of precision, performing at least a second search to select at least a second set of motion vectors and at least a second pair of bi-prediction weights, the second search being performed only on the N pairs of best bi-prediction weights; selecting a selected set of motion vectors and bi-prediction weights for decoding of the current block from (i) the first set of motion vectors and the first pair of bi-prediction weights and (ii) at least the second set of motion vectors and the second pair of bi-prediction weights; and encoding the current block using the selected set of motion vectors and bi-prediction weights. In some embodiments, the first level of precision is 1 / 4 pixel. In some embodiments, the second level of precision is 1 pixel.
[0102] In some embodiments, the set of N best pairs is a single best pair of bi-predictive weights. In some embodiments, the set of N best pairs includes the pair (1 / 2, 1 / 2). In some embodiments, the set of N best pairs includes the bi-predictive weights of a best pair and the pair (1 / 2, 1 / 2). In some embodiments, the set of N best pairs includes all pairs that give a rate-distortion cost less than a predetermined threshold.
[0103] In some embodiments, the first search is performed on a predetermined set of seven bi-prediction weights. In some embodiments, the first search is performed on a predetermined set of available bi-prediction weights, wherein the number of available bi-prediction weights is determined based at least in part on a temporal level of a current picture. In some embodiments, the first search is performed on a predetermined set of available bi-prediction weights, wherein the number of available bi-prediction weights is determined based at least in part on a quantization parameter of the current picture. In some embodiments, encoding the current block comprises: decoding information in the bitstream that identifies a selected set of motion vectors and bi-prediction weights. An embodiment of using a binary tree depth to limit motion searches.
[0104] In some embodiments, a method of encoding a video in a bitstream is provided. The method includes, for at least one current block in a current picture: determining whether a binary tree depth of the current block is not greater than a predetermined threshold; and only if the binary tree depth is not greater than the threshold: performing a search for a set of best motion vectors and bi-prediction weights, the search being performed on multiple pairs of bi-prediction weights; and encoding the current block using the best pair of bi-prediction weights. In some such embodiments, in response to determining that the binary tree depth is greater than the threshold, performing a search for a set of best motion vectors, the search being performed using only equal bi-prediction weights. An embodiment of using block size to limit motion search.
[0105] In some embodiments, a method of encoding a video in a bitstream is provided. The method includes, for at least one current block in a current picture: determining whether a block size of the current block is less than a predetermined threshold; and only if the block size is less than the threshold: performing a search for a set of best motion vectors and bi-prediction weights, the search being performed on multiple pairs of bi-prediction weights; and encoding the current block using the best pair of bi-prediction weights. In some embodiments, the method further includes, in response to a determination that the binary tree depth is less than the threshold, performing a search for a set of best motion vectors, the search being performed using only equal bi-prediction weights. In some such embodiments, the block size is the product of the width and height of the current block. An embodiment of using CBF to constrain motion search.
[0106] In some embodiments, a method of encoding video in a bitstream is provided. The method includes, for at least one current block in a current picture: performing a bi-prediction motion search among a plurality of bi-prediction weights; during the bi-prediction motion search, determining whether a pair of bi-prediction weights has been found so that a decoding block flag can be set to zero; in response to determining that a pair of bi-prediction weights has been found so that a decoding block flag can be set to zero, terminating the search; and encoding the current block using the found pair of bi-prediction weights.
[0107] In some embodiments, a method of encoding a video in a bitstream is provided. The method includes, for at least one current block in a current picture: performing a bi-prediction motion search among a plurality of bi-prediction weights; during the bi-prediction motion search, determining whether a pair of bi-prediction weights has been found such that a prediction of the current block made using the pair of bi-prediction weights has all-zero transform coefficients; in response to determining that a pair of bi-prediction weights has been found such that a prediction of the current block made using the pair of bi-prediction weights has all-zero transform coefficients, terminating the search; and encoding the current block using the found pair of bi-prediction weights. An embodiment of using skip merge mode to limit motion search.
[0108] In some embodiments, a method for encoding video in a bitstream is provided. The method includes, for at least one current block in a current picture: performing a first inter-frame prediction search among a plurality of merge candidate inter-frame prediction modes to identify an optimal inter-frame prediction merge mode for prediction of the current block; determining whether the optimal inter-frame prediction merge mode is a skip merge candidate associated with a pair of unequal generalized bi-prediction weights; performing a second inter-frame prediction search among a plurality of explicit inter-frame prediction modes, wherein the second search is performed on a plurality of pairs of unequal generalized bi-prediction weights only in response to determining that the optimal inter-frame prediction merge mode is not a skip merge candidate associated with a pair of unequal generalized bi-prediction weights; and encoding the current block using an inter-frame prediction mode selected from the first or second inter-frame prediction search.
[0109] In some such methods, an inter-prediction mode candidate is a skip merge mode candidate only if a prediction about the current block made using the inter-prediction mode candidate has all-zero transform coefficients. In some methods, in response to determining that the best inter-prediction merge mode is a skip merge candidate associated with a pair of unequal generalized bi-prediction weights, the second search comprises searching over explicit inter-prediction modes using at least the pair of unequal generalized bi-prediction weights. In some methods, in response to determining that the best inter-prediction merge mode is a skip merge candidate associated with a pair of unequal generalized bi-prediction weights, the second search comprises searching over explicit inter-prediction modes excluding the pair of unequal generalized bi-prediction weights. An embodiment of using a distortion metric in a motion search.
[0110] In some embodiments, a method of encoding a video in a bitstream is provided. The method includes, for at least one current block in a current picture: during a motion estimation stage, calculating a distortion metric for each of a plurality of combinations of a bi-predictive motion vector and a bi-predictive weight pair; based on the distortion metric, selecting a bi-predictive motion vector and an associated bi-predictive weight pair; using the selected bi-predictive motion vector and the associated bi-predictive weight pair, calculating a rate-distortion cost for predicting the current block; determining whether to use the selected bi-predictive motion vector and the associated bi-predictive weight pair to encode the current block based at least in part on the rate-distortion cost; and encoding the current block based on the determination. The distortion metric may be, for example, a sum of absolute differences or a sum of absolute transform differences.
[0111] In some such embodiments, the selected bi-predictive motion vector and associated bi-predictive weight pair is the only bi-predictive motion vector and associated bi-predictive weight pair for which a rate-distortion cost for the current block is calculated.
[0112] In some embodiments, a method for decoding a video comprising a plurality of pictures is provided, the plurality of pictures comprising a current picture, a first reference picture, and a second reference picture, each picture comprising a plurality of blocks. The method comprises, for at least a current block in the current picture: determining a number of available bi-prediction weight pairs based at least in part on a quality metric of the current picture; identifying a weight pair from the determined number of available bi-prediction weight pairs, the identified weight pair comprising a higher weight and a lower weight; determining which of the first and second reference pictures is a higher quality reference picture and which is a lower quality reference picture; and predicting the current block as a weighted sum of a higher quality reference block in the higher quality reference picture and a lower quality reference block in the lower quality reference picture, wherein the higher quality reference block is weighted by the higher weight and the lower quality reference block is weighted by the lower weight. Decoded bitstream structure.
[0113] Fig.1213 is a schematic diagram showing an example of a decoded bitstream structure. The decoded bitstream 1300 consists of several NAL (Network Abstraction Layer) units 1301. The NAL unit may contain decoded sample data (e.g., decoded slices 1306), or high-level syntax metadata, such as parameter set data, slice header data 1305, or supplemental enhancement information data 1307 (which may be referred to as SEI messages). The parameter set is a high-level syntax structure containing basic syntax elements, which may be applied to multiple bitstream layers (e.g., video parameter set 1302 (VPS)), or may be applied to a decoded video sequence within a layer (e.g., sequence parameter set 1303 (SPS)), or may be applied to several decoded pictures within a decoded video sequence (e.g., picture parameter set 1304 (PPS)). The parameter set may be sent together with the decoded pictures of the video bitstream, or sent by other means (including out-of-band transmission using a reliable channel, hard decoding, etc.). The slice header 1305 is also a high-level syntax structure that may contain some picture-related information that is relatively small or only relevant to certain slices or picture types. The SEI message 1307 carries information that may not be required by the decoding process but can be used for various other purposes, such as picture output timing or display and loss detection and concealment. Communications equipment and systems.
[0114] Fig.13 1400 is a schematic diagram showing an example of a communication system. The communication system 1400 may include an encoder 1402, a communication network 1404, and a decoder 1406. The encoder 1402 may communicate with the network 1404 via a connection 1408, which may be a wired connection or a wireless connection. The encoder 1402 may be similar to Figure 1 The encoder 1402 may include a single-layer codec (e.g., Figure 1 ) or multi-layer codecs (e.g. Fig.10 ). For example, the encoder 1402 may be similar to Fig. 9 The decoder 1406 may communicate with the network 1404 via a connection 1410, which may be a wired connection or a wireless connection. The decoder 1406 may be similar to Figure 2 The decoder 1406 may include a single-layer codec (e.g., Figure 2 ) or multi-layer codecs (e.g. Fig.11 ). For example, the decoder 1406 may be similar to Fig.11 A multi-layer (e.g., two-layer) scalable decoding system with picture-level ILP support.
[0115] The encoder 1402 and / or the decoder 1406 may be incorporated into a variety of wired communication devices and / or wireless transmit / receive units (WTRUs), such as, but not limited to, digital televisions, wireless broadcast systems, network elements / terminals, servers (e.g., content or network servers (e.g., such as hypertext transfer protocol (HTTP) servers)), personal digital assistants (PDAs), laptop or desktop computers, tablet computers, digital cameras, digital recording devices, video gaming devices, video game consoles, cellular or satellite radio telephones, digital media players, and / or the like.
[0116] The communication network 1404 may be a communication network of a suitable type. For example, the communication network 1404 may be a multiple access system that provides content such as voice, data, video, messaging, broadcasting, etc. to multiple wireless users. The communication network 1404 may enable multiple wireless users to access such content by sharing system resources including wireless bandwidth. For example, the communication network 1404 may use one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA) and / or the like. The communication network 1404 may include multiple connected communication networks. The communication network 1404 may include the Internet and / or one or more private commercial networks, such as cellular networks, WiFi hotspots and / or Internet service provider (ISP) networks, etc.
[0117] Fig.14 1500 , a processor 1518 , a transceiver 1520 , a transmit / receive element 1522 , a speaker / microphone 1524 , a keypad or keyboard 1526 , a display / touchpad 1528 , a non-removable memory 1530 , a removable memory 1532 , a power supply 1534 , a global positioning system (GPS) chipset 1536 , and / or other peripherals 1538 . It will be appreciated that the WTRU 1500 may include any sub-combination of the foregoing elements while remaining consistent with the embodiments. In addition, a terminal incorporating an encoder (e.g., encoder 100 ) and / or a decoder (e.g., decoder 200 ) may be included herein. Fig.14 Some or all of the elements described and depicted for the WTRU 1500.
[0118] The processor 1518 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a graphics processing unit (GPU), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 1518 may perform signal coding, data processing, power control, input / output processing, and / or any other functions that enable the WTRU 1500 to operate in a wired and / or wireless environment. The processor 1518 may be coupled to the transceiver 1520, which may be coupled to the transmit / receive element 1522. Although Fig.14 The processor 1518 and the transceiver 1520 are depicted as separate components, but it will be appreciated that the processor 1518 and transceiver 1520 may be integrated together in an electronic package and / or chip.
[0119] The transmit / receive element 1522 may be configured to transmit a signal to another terminal and / or receive a signal from another terminal via the air interface 1515. For example, in one or more embodiments, the transmit / receive element 1522 may be an antenna configured to transmit and / or receive RF signals. In one or more embodiments, the transmit / receive element 1522 may be a transmitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In one or more embodiments, the transmit / receive element 1522 may be configured to transmit and / or receive both RF and optical signals. It will be appreciated that the transmit / receive element 1522 may be configured to transmit and / or receive any combination of wireless signals.
[0120] Furthermore, although the transmit / receive element 1522 is Fig.14 1515. Although depicted as a single element, the WTRU 1500 may include any number of transmit / receive elements 1522. More specifically, the WTRU 1500 may employ MIMO technology. Thus, in one embodiment, the WTRU 1500 may include two or more transmit / receive elements 1522 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 1515.
[0121] The transceiver 1520 may be configured to modulate signals to be transmitted by the transmit / receive element 1522 and / or to demodulate signals received by the transmit / receive element 1522. As described above, the WTRU 1500 may have multi-mode capabilities. Thus, the transceiver 1520 may include multiple transceivers to enable the WTRU 1500 to communicate via multiple RATs, such as UTRA and IEEE 802.11.
[0122] The processor 1518 of the WTRU 1500 may be coupled to a speaker / microphone 1524, a keyboard 1526, and / or a display / touchpad 1528 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), and may receive user input data therefrom. The processor 1518 may also output user data to the speaker / microphone 1524, the keyboard 1526, and / or the display / touchpad 1528. In addition, the processor 1518 may access information from and store data in any type of suitable memory, such as a non-removable memory 1530 and / or a removable memory 1532. The non-removable memory 1530 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 1532 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, or the like. In one or more embodiments, the processor 1518 may access information from, and store data in, memory that is not physically located on the WTRU 1500, such as on a server or a home computer (not shown).
[0123] The processor 1518 may receive power from a power source 1534 and may be configured to distribute and / or control the power to other components in the WTRU 1500. The power source 1534 may be any suitable device for powering the WTRU 1500. For example, the power source 1534 may include one or more dry cell batteries (e.g., nickel-cadmium, nickel-zinc, nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.
[0124] The processor 1518 may be coupled to a GPS chipset 1536, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 1500. In addition to or in lieu of the information from the GPS chipset 1536, the WTRU 1500 may receive location information from a terminal (e.g., a base station) over the air interface 1515, and / or determine its location based on the timing of signals received from two or more neighboring base stations. It will be appreciated that the WTRU 1500 may acquire location information by any suitable location-determination method while remaining consistent with an embodiment.
[0125] The processor 1518 may also be coupled to other peripherals 1538, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 1538 may include an accelerometer, an orientation sensor, a motion sensor, a proximity sensor, an electronic compass, a satellite transceiver, a digital camera and / or video recorder (e.g., for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, modules, frequency modulation (FM) radio units, and software modules such as digital music players, media players, video game player modules, Internet browsers, etc.
[0126] By way of example, the WTRU 1500 may be configured to transmit and / or receive wireless signals and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smart phone, a laptop computer, a netbook, a tablet computer, a personal computer, a wireless sensor, a consumer electronic product, or any other terminal capable of receiving and processing compressed video communications.
[0127] The WTRU 1500 and / or the communication network (e.g., the communication network 804) may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may use Wideband CDMA (WCDMA) to establish the air interface 1515. WCDMA may include communication protocols such as High Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed Downlink Packet Access (HSDPA) and / or High Speed Uplink Packet Access (HSUPA). The WTRU 1500 and / or the communication network (e.g., the communication network 804) may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may use Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) to establish the air interface 1515.
[0128] The WTRU 1500 and / or the communication network (e.g., the communication network 804) may implement a radio technology such as IEEE 802.16 (e.g., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 IX, CDMA2000EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), GSM EDGE (GERAN), etc. The WTRU 1500 and / or the communication network (e.g., the communication network 804) may implement a radio technology such as IEEE 802.11, or IEEE 802.15, etc.
[0129] Note that one or more of the various hardware elements in the described embodiments are referred to as "modules" that perform (i.e., run, execute, etc.) the various functions described herein in conjunction with the corresponding modules. As used herein, a module includes hardware that a person skilled in the relevant art deems suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices). Each described module may also include executable instructions for performing one or more functions described as being performed by the corresponding module, and note that these instructions may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, and / or software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium or media, such as those commonly referred to as RAM, ROM, etc.
[0130] Although the features and elements are described above in a specific combination or order, one of ordinary skill in the art will recognize that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated into a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, buffer memory, semiconductor storage devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor associated with software may be used to implement a radio frequency transceiver used in a WTRU, UE, terminal, base station, RNC, or any host computer.
Claims
1. A video decoding method, comprising: identifying a first weight and a second weight from a set of available bi-prediction weights; as well as A current block in the current picture is predicted as a weighted sum of a first reference block in a first reference picture and a second reference block in a second reference picture, wherein the first reference block is weighted by the first weight and the second reference block is weighted by the second weight. 2 . The method of claim 1 , wherein a number of available bi-prediction weights is based on a quantization parameter of the current picture, and wherein the number of available bi-prediction weights decreases for increasing quantization parameters.
3. The method according to claim 2, wherein: For at least one quantization parameter of the current picture, the number of available bi-prediction weights is one.
4. The method of claim 1 , wherein identifying the first weight and the second weight comprises: Information identifying at least one of the first weight and the second weight is parsed from a bitstream.
5. An apparatus comprising a processor, the processor being configured to at least perform: identifying a first weight and a second weight from a set of available bi-predictive weights; and A current block in the current picture is predicted as a weighted sum of a first reference block in a first reference picture and a second reference block in a second reference picture, wherein the first reference block is weighted by the first weight and the second reference block is weighted by the second weight. 6 . The device of claim 5 , wherein a number of available bi-prediction weights is based on a quantization parameter of the current picture, and wherein the number of available bi-prediction weights decreases for increasing quantization parameters.
7. The device according to claim 6, wherein: For at least one quantization parameter of the current picture, the number of available bi-prediction weights is one.
8. The device of claim 5, wherein the device is a video encoder.
9. The device of claim 5, wherein the device is a video decoder.
10. A video encoding method, comprising: identifying a first weight and a second weight from a set of available bi-prediction weights; as well as A current block in the current picture is predicted as a weighted sum of a first reference block in a first reference picture and a second reference block in a second reference picture, wherein the first reference block is weighted by the first weight and the second reference block is weighted by the second weight.
11. The method of claim 10, wherein a number of available bi-prediction weights is based on a quantization parameter of the current picture, and wherein the number of available bi-prediction weights decreases for increasing quantization parameters.
12. The method according to claim 11, wherein: For at least one value of the quantization parameter, the determined number of available bi-prediction weights is one.
13. The method according to claim 10, further comprising: Information identifying at least one of the first weight and the second weight is signaled in a bitstream.
Citation Information
Patent Citations
Systems and methods for generalized multi-hypothesis prediction for video coding
WO2017197146A1
