Systems and methods for generalized multi-hypothesis prediction for video coding

Abstract

To provide systems and methods for video coding using generalized bi-prediction.SOLUTION: A video encoding method comprises: for at least a current block in a current image, encoding block-level information identifying a first weight and a second weight, where at least one of the first and second weights has a value not equal to 0, 0.5 or 1; and predicting the current block as a weighted sum of a first reference block in a first reference image and a second reference block in a second reference image, where the first reference block is weighted by the first weight and the second reference block is weighted by the second weight. Encoding the block-level information includes mapping a weight index to a codeword using a truncated unary code, and entropy-encoding the codeword in a bitstream.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a system and method for generalized multi-hypothesis prediction for video coding. [Background technology]

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application of and claims the benefit under 35 U.S.C. § 119(c) of the following U.S. provisional patent applications: Serial No. 62 / 336227, entitled "Systems and Methods for Generalized Multi-Hypothesis Prediction for Video Coding," filed May 13, 2016; Serial No. 62 / 342772, entitled "Systems and Methods for Generalized Multi-Hypothesis Prediction for Video Coding," filed May 27, 2016; Serial No. 62 / 399234, entitled "Systems and Methods for Generalized Multi-Hypothesis Prediction for Video Coding," filed September 23, 2016; and Serial No. 62 / 415187, entitled "Systems and Methods for Generalized Multi-Hypothesis Prediction for Video Coding," filed October 31, 2016. All of these applications are incorporated herein by reference in their entirety.

[0003] Video coding systems are widely used to compress digital video signals to reduce the storage requirements and / or transmission bandwidth of such signals. Among various types of video coding systems, such as block-based, wavelet-based, and object-based systems, block-based hybrid video coding systems are most widely used and deployed today. Examples of block-based video coding systems include international video coding standards such as MPEG-1 / 2 / 4 part 2, H.264 / MPEG-4 part 10 AVC, VC-1, and the latest video coding standard called High Efficiency Video Coding (HEVC), developed by ITU-T / SG16 / Q.6 / VCEG and ISO / IEC / MPEG's JCT-VC (Joint Collaborative Team on Video Coding).

[0004] Video encoded using block-based coding accounts for a significant percentage of data transmitted electronically, for example, over the Internet. It is desirable to increase the efficiency of video compression so that high-quality video content can be stored and transmitted using fewer bits. Summary of the Invention

[0005] In an exemplary embodiment, a system and method for performing generalized bi-prediction (GBi) is described. The exemplary method includes encoding and decoding (collectively "encoding") video including multiple images, including a current image, a first reference image, and a second reference image, each image including multiple blocks. In the exemplary method, a block-level index is encoded that identifies a first weight and a second weight from a set of weights for at least a current block in the current image, and at least one of the weights in the set of weights has a value not equal to 0, 0.5, or 1. The current block is predicted as a weighted sum of a first reference block in the first reference image and a second reference block in the second reference image, where the first reference block is weighted by the first weight and the second block is weighted by the second weight.

[0006] In some embodiments (or for some blocks), block-level information identifying the first and second weights for a current block can be coded by means other than coding an index for that block. For example, the block can be coded in merge mode. In such a case, the block-level information can be information identifying a candidate block from multiple merge candidate blocks. In that case, the first and second weights can be identified based on the weights used to code the identified candidate block.

[0007] In some embodiments, the first reference block and the second reference block are further scaled by at least one scaling factor signaled in the bitstream for the current picture.

[0008] In some embodiments, the sets of weights are coded into the bitstream, allowing different weight sets to be adapted for use in different slices, images, or sequences. In other embodiments, the sets of weights are predetermined. In some embodiments, only one of the two weights is signaled in the bitstream, and the other weight is derived by subtracting the signaled weight from one.

[0009] In some embodiments, a codeword is assigned to each weight, and the weights are identified using the corresponding codeword. The assignment of codewords to weights can be predetermined, or the assignment can be adapted based on weights used in previously coded blocks.

[0010] Exemplary encoders and decoders for performing generalized bi-prediction are also described herein.

[0011] The systems and methods described herein provide novel techniques for predicting blocks of sample values. Such techniques can be used by both encoders and decoders. In an encoding method, predicting a block results in a block of sample values ​​that can be subtracted from the original input block to determine a residual that is encoded into a bitstream. In a decoding method, the residual can be decoded from the bitstream and added to the predicted block to obtain a reconstructed block that is the same as, or approximately the same as, the original input block. Thus, prediction methods as described herein improve the operation of video encoders and decoders by, at least in some implementations, reducing the number of bits required to encode and decode video. Further benefits of exemplary prediction methods to the operation of video encoders and decoders are provided in the detailed description. [Brief explanation of the drawings]

[0012] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings, which are first briefly described below.

[0013] [Figure 1] FIG. 1 is a functional block diagram illustrating an example of a block-based video encoder. [Figure 2] FIG. 1 is a functional block diagram illustrating an example of a block-based video decoder. [Figure 3] 1 is a schematic diagram of prediction using a template TC and associated prediction blocks T0 and T1. [Figure 4] 1 is a graph providing a schematic representation of illumination change over time. [Figure 5] FIG. 1 is a functional block diagram illustrating a video encoder configured to use generalized bi-prediction, according to some embodiments. [Figure 6] FIG. 1 is a functional block diagram of an example generalized bi-prediction module for use in a video encoder. [Figure 7] FIG. 10 is a schematic diagram of an exemplary decoder-side derivation of implicit weight values ​​for use in generalized bi-prediction. [Figure 8] FIG. 10 is a schematic diagram of a tree structure for binarizing weight_idx, where each circle represents a signaled bit. [Figure 9] FIG. 1 is a functional block diagram illustrating a video decoder configured to use generalized bi-prediction, according to some embodiments. [Figure 10] FIG. 2 is a functional block diagram of an example generalized bi-prediction module for use in a video decoder. [Figure 11A] FIG. 1 is a schematic diagram of a fixed allocation method of codeword allocation. [Figure 11B] 1 is a schematic diagram of an alternative codeword allocation method; [Figure 12A]FIG. 10 is a schematic diagram providing weight value fields of an example of block adaptive codeword allocation. [Figure 12B] FIG. 10 is a schematic diagram providing the resulting codeword allocation updated from a constant allocation in an example of block adaptive codeword allocation. [Figure 13] FIG. 1 is a schematic diagram of merge candidate locations. [Figure 14] 1 is a schematic diagram of an example of overlapped block motion compensation (OBMC), where m is a basic processing unit for performing OBMC, N1 to N8 are sub-blocks in a causal neighborhood, and B1 to B7 are sub-blocks in the current block. [Figure 15] FIG. 10 is a diagram showing an example of frame rate up-conversion (FRUC), where v0 is a given motion vector corresponding to reference list L0, and v1 is a scaled MV based on v0 and temporal distance. [Figure 16] FIG. 1 illustrates an example of an encoded bitstream structure. [Figure 17] FIG. 1 illustrates an exemplary communication system. [Figure 18] FIG. 1 illustrates an exemplary wireless transmit / receive unit (WTRU) that can be used as an encoder or decoder in some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0014] Block-based encoding FIG. 1 is a block diagram of a generic block-based hybrid video encoding system 100. An input video signal 102 is processed block by block. In HEVC, an extended block size (called a "coding unit" or CU) is used to efficiently compress high-resolution (1080p and above) video signals. In HEVC, a CU can be up to 64x64 pixels. A CU can be further partitioned into prediction units or PUs, to which different prediction methods are applied. For each input video block (MB or CU), spatial prediction (160) and / or temporal prediction (162) can be performed. Spatial prediction (or "intra prediction") predicts the current video block using pixels from already-encoded neighboring blocks within the same video image / slice. Spatial prediction reduces spatial redundancy inherent in the video signal. Temporal prediction (also called "inter prediction" or "motion-compensated prediction") predicts the current video block using pixels from already-encoded video images. Temporal prediction reduces temporal redundancy inherent in video signals. The temporal prediction signal for a given video block can be signaled by one or more motion vectors, which indicate the amount and direction of motion between the current block and its reference block. Additionally, if multiple reference images are supported (as is the case in recent video coding standards such as H.264 / AVC or HEVC), a reference index of the reference image can also be transmitted for each video block. The reference index is used to identify which reference image in the reference image store (164) the temporal prediction signal originates from. After spatial and / or temporal prediction, a mode decision block (180) in the encoder selects the best prediction mode, for example, based on a rate-distortion optimization method. The prediction block is then subtracted (116) from the current video block, and the prediction residual is decorrelated using a transform (104) and quantized (106) to achieve a target bit rate.The quantized residual coefficients are inverse quantized (110) and inverse transformed (112) to form a reconstructed residual, which is then added back to the prediction block (126) to form a reconstructed video block. Further, in-loop filtering, such as a deblocking filter and an adaptive loop filter, may be applied to the reconstructed video block (166), which is then placed into a reference image store (164) and used to encode additional video blocks. To form the output video bitstream 120, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy coding unit (108) for further compression and packing to form the bitstream.

[0015] Block-Based Decoding 2 provides an overall block diagram of a block-based video decoder 200. A video bitstream 202 is unpacked and entropy decoded in an entropy decoding unit 208. Coding mode and prediction information are sent to a spatial prediction unit 260 (for intra-coding) or a temporal prediction unit 262 (for inter-coding) to form a prediction block. Residual transform coefficients are sent to an inverse quantization unit 210 and an inverse transform unit 212 to reconstruct a residual block. The prediction block and the residual block are then summed at 226. The reconstructed block may further pass in-loop filtering, after which it is stored in a reference picture store 264. The reconstructed video in the reference picture store is then used to predict further video blocks as well as to drive a display device.

[0016] In modern video codecs, bidirectional motion compensation prediction (MCP) is known for its high efficiency in removing temporal redundancy by exploiting the temporal correlation between images, and is widely adopted in most modern video codecs. However, a bi-predictive signal is formed by simply combining two uni-prediction signals using a weight value equal to 0.5. This is not always optimal for combining uni-prediction signals, especially in certain conditions where illumination changes rapidly when switching from one reference image to another. Therefore, several prediction techniques have been developed, aiming to compensate for illumination variations over time by applying several global or local weights and offset values ​​to each sample value in the reference image.

[0017] Weighted Bi-Prediction Weighted bi-prediction is a coding tool primarily used to compensate for illumination changes over time, such as fading transitions, when performing motion compensation. For each slice, two sets of multiplicative weights and additive offsets are explicitly specified and applied separately to the motion-compensated prediction, one at a time, per reference list. This technique is most effective when illumination varies linearly and equally across images.

[0018] Local illumination compensation In local illumination compensation, parameters (two pairs of multiplicative weights and additive offsets) are adapted for each block. Unlike weighted bi-prediction, which expresses these parameters at the slice level, this technique uses a template (T C We resort to fitting optimal parameters to the illumination variation between the reconstructed signal of T and the predicted signals of the template (T and T) (see Figure 3). The resulting parameters are T (for the first pair of weights and offset values). C and T0, and (for the second pair of weights and offset values) T Cand T1 separately. The same motion compensation process as for weighted bi-prediction is then performed with the derived parameters.

[0019] Effect of changing illumination The variation of illumination over space and time can seriously affect the performance of motion compensated prediction. As can be seen in Figure 4, when illumination weakens along the time direction, motion compensated prediction does not provide good performance. For example, if a sample of an object moves over a time period from t-3 to t, and the intensity value of this sample changes along its motion trajectory, v t-3 From v t Assuming this sample is to be predicted in the t-th image, its predicted value is v t-3 From v t-1 , and therefore may result in poor motion-compensated prediction. The above-mentioned techniques of weighted bi-prediction and local illumination compensation may not solve this problem sufficiently. Because illumination can vary wildly within an image, weighted bi-prediction may fail. Local illumination compensation sometimes produces poor estimates of weights and offset values ​​due to low illumination correlation between a block and its associated template block. These examples show that global and template-based local descriptions are not sufficient to represent illumination variations across space and time.

[0020] Illustrative Embodiments Exemplary embodiments described herein can improve prediction efficiency for weighted motion-compensated prediction. In some embodiments, systems and methods for generalized multi-hypothesis prediction are proposed, using motion-compensated prediction and block-level weight values ​​for linearly combining multi-hypothesis prediction signals. In some embodiments, a generalized bi-prediction framework using weight values ​​is described. In some embodiments, a finite set of weights is used at the sequence, image, or slice level, and a construction process for the set of weights is described. In some embodiments, weight values ​​are determined based on a given weight set and optimized taking into account signals of the current block and its reference block. An exemplary encoding method for signaling weight values ​​is described. Exemplary encoder search criteria for the motion estimation process for the proposed prediction are described, and the proposed prediction process combined with the disclosed temporal prediction technique is described.

[0021] In this disclosure, systems and methods for temporal prediction using generalized multi-hypothesis prediction are described. Exemplary encoders and decoders using generalized bi-prediction are described with reference to Figures 5 and 9. The systems and methods disclosed herein are organized in sections as follows: Section "Generalized Multi-hypothesis Prediction" describes an exemplary embodiment using generalized multi-hypothesis prediction. Section "Generalized Bi-prediction" discloses an exemplary framework and prediction process for generalized bi-prediction. Sections "Weight Set Construction" and "Weight Index Coding" respectively describe an exemplary construction process for a weight set and an exemplary technique for signaling the selection of weights within this set. Section "Extension to Advanced Temporal Prediction Techniques" describes systems and methods for combining the exemplary proposed prediction method with advanced inter-prediction techniques, including local illumination compensation and weighted bi-prediction, merge mode, overlapped block motion compensation, affine prediction, bi-predictive optical flow, and a decoder-side motion vector derivation technique called frame rate up-conversion bi-prediction. In the section "GBi Prediction Search Strategy," an exemplary encoder-specific method for improving the efficiency of the exemplary prediction method is described.

[0022] Generalized multi-hypothesis prediction The exemplary systems and methods described herein utilize generalized multi-hypothesis prediction. Generalized multi-hypothesis prediction can be described as a generalized form of multi-hypothesis prediction that provides an estimate of a pixel's intensity value based on a linear combination of multiple motion-compensated prediction signals. Generalized multi-hypothesis prediction can utilize the benefits of multiple predictions with different qualities by combining them together. To arrive at an accurate estimate, the motion-compensated prediction signals can be processed through a predetermined function f(●) (e.g., gamma correction, local illumination correction, dynamic range correction) and then linearly combined. Generalized multi-hypothesis prediction can be described with reference to equation (1):

[0023]

number

[0024] where P[x] represents the resulting predicted signal for sample x placed at image location x, and w i represents the weight value applied to the i-th motion hypothesis from the i-th reference image, and P i [x+v i ] is the motion vector (MV) v i is the motion compensated prediction signal of x using , and n is the total number of motion hypotheses.

[0025] One factor to consider with motion-compensated prediction is how to balance the accuracy of the motion field with the required motion overhead to achieve maximum rate-distortion performance. An accurate motion field implies better prediction; however, the required motion overhead can sometimes outweigh the benefit of prediction accuracy. Therefore, in an exemplary embodiment, the proposed video encoder can adaptively switch between different numbers of motion hypotheses n, and the n value that provides optimal rate-distortion performance is found for each PU. Since two motion hypotheses are commonly used in most modern video coding standards, a value of n=2 is selected as an example in the following sections to facilitate the explanation of how generalized multi-hypothesis prediction works, but other values ​​of n can alternatively be used. To simplify understanding of the exemplary embodiment, the formula f(●) is treated as an identity function and therefore will not be explicitly described. The application of the systems and methods disclosed herein to cases where f(●) is not an identity function will be apparent to those skilled in the art in light of this disclosure.

[0026] Generalized Bi-Prediction The term generalized bi-prediction (GBi) is used herein to refer to a special case of generalized multi-hypothesis prediction where the number of motion hypotheses is limited to two, i.e., n = 2. In this case, the predicted signal at sample x given by equation (1) is P[x]=w0*P0[x+v0]+w1*P1[x+v1] (2) where w0 and w1 are two weight values ​​shared across all samples in the block. Based on this formula, a wide variety of prediction signals can be generated by adjusting the weight values ​​w0 and w1. Some configurations for w0 and w1 can result in the same prediction as traditional uni-prediction and bi-prediction, such as (w0, w1) = (1, 0) for uni-prediction using reference list L0, (w0, w1) = (0, 1) for uni-prediction using reference list L1, and (w0, w1) = (0.5, 0.5) for bi-prediction using two reference lists. In the cases of (1, 0) and (0, 1), only one set of motion information is signaled because the other set associated with a weight value equal to zero has no effect on the prediction signal P[x].

[0027] In particular, flexibility in the values ​​of w0 and w1 at a high level of precision can incur the cost of high signaling overhead. To reduce the signaling overhead, in some embodiments, a unit gain constraint of w0+w1=1 is applied, and thus only one weight value per block is explicitly indicated for GBi-coded PUs. To further reduce the weight signaling overhead, weight values ​​can be signaled at the CU level instead of the PU level. For ease of explanation, in the description of this disclosure, w1 is signaled, and thus equation (2) becomes P[x]=(1-w1)*P0[x+v0]+w1*P1[x+v1] (3) In an exemplary embodiment, to further limit the signaling overhead, the frequently used weight values ​​are (hereafter referred to as W L1 ) so that each weight value can be arranged in a set called W L1 The weight_idx can be indicated by an index value within a restricted range that indicates which entry in the weight_idx to occupy.

[0028] In an exemplary embodiment, generalized bi-prediction does not introduce additional decoding load to support the generation of a weighted average of two reference blocks. Since most modern video standards (e.g., AVC, HEVC) support weighted bi-prediction, the same prediction module can be adapted for use in GBi prediction. In an exemplary embodiment, generalized bi-prediction can be applied not only to traditional uni-prediction and bi-prediction, but also to other advanced temporal prediction techniques, such as affine prediction, advanced temporal motion vector derivation, and bi-predictive optical flow. These techniques aim to derive motion field representations in finer units (e.g., 4x4) with very low motion overhead. Affine prediction is a model-based motion field coding method, and the motion of each unit within one PU can be derived based on model parameters. Advanced temporal motion vector derivation involves deriving the motion of each unit from the motion field of a temporal reference image. Bi-predictive optical flow involves deriving a motion refinement for each pixel using an optical flow model. Whatever the size of the unit, once the weight values ​​are specified at the block level, the proposed video codec can perform generalized bi-prediction for each unit using these derived motions and the given weight values.

[0029] Exemplary encoders and decoders that utilize generalized bi-prediction are described in more detail below.

[0030] Exemplary Encoder for Generalized Bi-Prediction FIG. 5 is a block diagram of an exemplary video encoder adapted to perform generalized bi-prediction. Similar to the video encoder shown in FIG. 1, spatial prediction and temporal prediction are two basic pixel-domain prediction modules in the exemplary video encoder. The spatial prediction module may be the same as that shown in FIG. 1. The temporal prediction module named "motion prediction" in FIG. 1 may be replaced by a generalized bi-prediction (GBi) module 502. The generalized bi-prediction (GBi) module may be operable to combine two separate motion-compensated prediction (MCP) signals in a weighted average manner. As shown in FIG. 6, the GBi module may implement a process for generating a final inter-prediction signal as follows: The GBi module may perform motion estimation on a reference image to search for two optimal motion vectors (MVs) that point to two reference blocks that minimize the weighted bi-prediction error between the current video block and the bi-prediction signal. The GBi module may fetch these two prediction blocks through motion compensation using the two optimal MVs. The GBi module can then calculate a generalized bi-predictive prediction signal as a weighted average of the two prediction blocks.

[0031] In some embodiments, all available weight values ​​are specified in a single set. If weight values ​​are signaled for both reference lists at the PU level, it means signaling two separate weight values ​​per bi-predictive PU, which can cost a large number of bits, so a unit gain constraint (the sum of the weight values ​​is equal to 1) can be applied. Under this constraint, only one single weight value per PU is signaled, while the other one can be derived from subtracting the signaled weight value from 1. For ease of explanation, in this disclosure, the weight values ​​associated with reference list L1 are signaled, and the set of weight values ​​is W L1To further reduce the signaling overhead, the weight value is expressed as W L1 The weight_idx is an index value that indicates the position of the entry in the W L1 With appropriate assignments to W, both conventional uni-prediction (with weights equal to 0 for one reference list and 1 for the other) and conventional bi-prediction (with weight values ​​equal to 0.5 for both reference lists) can be represented under the GBi framework. L1 In the special case of ={0,0.5,1}, the GBi module can achieve the same functionality as the motion estimation module shown in FIG.

[0032] In addition to {0,0.5,1}, W L1 Additional weight values ​​for can be specified at the slice, picture, or sequence level, with the non-negative integer extra_number_of_weights indicating their number, so that within the GBi framework there are extra_number_of_weights+3 distinct weights. In particular, in an exemplary embodiment, when extra_number_of_weights is greater than zero, one of these additional weight values ​​can be derived on a block-by-block basis, depending on the control of a flag implicit_weight_flag, which exists at the slice, picture, or sequence level. When this flag is set equal to 1, this particular weight value is not signaled, but can be derived by finding one that can minimize the difference between the generalized bi-predictive signal of the direct inverse-L neighborhood (called the template) and the reconstructed signal of the template, as shown in FIG. 7. L1 The above-described process related to constructing {right arrow over (x)} can be performed by the weight set construction module 504.

[0033] W L1In order to adapt the additional weight values ​​in to images with high dynamic illumination changes, two scaling factors (gbi_scaling_factors) can be applied and signaled at the image level. Using them, the weight set construction module can scale the values ​​of the additional weights for GBi prediction. After inter prediction (which is GBi prediction in the proposed video encoder) and intra prediction, the original signal can be subtracted from this final prediction signal, and thus the resulting prediction residual signal for encoding is generated.

[0034] In the exemplary proposed video encoder, block motion (motion vector and reference picture index) and weight value index are the only block-level information indicated for each inter-coded PU.

[0035] In an exemplary embodiment, block motion information for GBi prediction is coded in the same way as that of the underlying video codec. Except when weight_idx is associated with a weight equal to 0 or 1, i.e., the equivalent case of uni-prediction, two sets of motion information per PU are signaled.

[0036] In an exemplary video encoder, a weight index encoding module 506 is used to binarize weight_idx for each PU. The output of the weight index encoding module may be a unique binary representation of weight_idx, binary_weight_idx. The tree structure of an exemplary binarization scheme is shown in FIG. 8. As in conventional inter prediction, the first bit of binary_weight_idx indicates, for each inter PU, whether uni-prediction (weight index associated with a weight value equal to 0 or 1) or bi-prediction (W L1In a uni-predictive branch, another bit is signaled to indicate whether the L0 reference list (weight indices associated with weight values ​​other than 0 and 1 in W) or the L1 reference list (weight indices associated with weight values ​​equal to 1) is referenced. In a bi-predictive branch, each leaf node is signaled with a bit L1 8, each weight is assigned a unique weight index value associated with one of the remaining weight values ​​in the bi-predictive branch, i.e., weight values ​​that are neither 0 nor 1. At the slice or picture level, an exemplary video encoder can adaptively switch between several predetermined assignment schemes or dynamically assign each weight to a unique leaf node on a per-PU basis based on the use of weight values ​​from previous coding blocks. Generally, frequently used weight indices are assigned to leaf nodes closer to the root in the bi-predictive branch, while others, in contrast, are assigned to deeper leaf nodes farther from the root. Traversing this tree in FIG. 8, every weight_idx can be converted to a unique binary_weight_idx for entropy coding.

[0037] A generalized bipredictive decoding framework Figure 9 is a block diagram of a video decoder in some embodiments. The decoder of Figure 9 may be operable to decode the bitstream generated by the video encoder shown in Figure 5. The coding mode and prediction information may be used to derive a prediction signal using spatial prediction or generalized bi-prediction. For generalized bi-prediction, block motion information and weight values ​​are received and decoded.

[0038] The weight index decoding module 902 decodes the weight indexes coded by the weight index coding module 506 in the proposed video encoder. The weight index decoding module 902 reconstructs the same tree structure as specified in FIG. 8, and each leaf node on the tree is assigned a unique weight_idx in the same way as in the proposed video encoder. In this way, the tree is synchronized between the proposed video encoder and decoder. Traversing the tree, every received binary_weight_idx can find its associated weight_idx at a leaf node on the tree. The exemplary video decoder uses the weight set W, similar to the video encoder of FIG. 5. L1 When implicit_weight_flag is equal to 1, W L1 One of the additional weight values ​​in W can be derived instead of being explicitly signaled. L1 All additional weight values ​​in can be further scaled using the scaling factors indicated by gbi_scaling_factors. The weight value reconstruction is then performed using W L1 This can be done by fetching the one pointed to by weight_idx from

[0039] The decoder can receive one or two sets of motion information depending on the selection of the weight value for each block. When the reconstructed weight value is neither 0 nor 1, two sets of motion information can be received; otherwise (when it is 0 or 1), only one set of motion information associated with the non-zero weight is received. For example, if the weight value is equal to 0, only the motion information for reference list L0 is signaled; otherwise, if the weight value is equal to 1, only the motion information for reference list L1 is signaled.

[0040] Using the block motion information and weight values, the generalized bi-prediction module 1050 shown in FIG. 10 can operate to calculate a generalized bi-prediction prediction signal as a weighted average of two motion compensated prediction blocks.

[0041] Depending on the coding mode, a spatial prediction signal or a generalized bi-predictive signal may be summed with the reconstructed residual signal to obtain a reconstructed video block signal.

[0042] Building the weight set A weight set W, which uses explicitly signaled weights, decoder-side derived weights, and scaled weights. L1 An exemplary construction process for the weight set W L1 This is described below along with an exemplary reduction process to compact the size of .

[0043] Explicit Weight Values Explicit weight values ​​can be signaled and managed hierarchically at each of the sequence, image, and slice levels. Weights specified at lower levels can replace those at higher levels. Assuming the number of explicit weights at a higher level is p and that at a relatively lower level is q, the following rules for substitution can be applied when constructing weight value lists at lower levels: If p>q, then the last q weights at the higher level are replaced by the q weights at the lower level. When p≦q, all weights at higher levels are replaced by those specified at lower levels.

[0044] The number of explicit weight values ​​can be indicated by extra_number_of_weights at each of the sequence, picture, and slice levels. In some embodiments, at the slice level, the base weight set always includes three default values ​​forming {0, 0.5, 1} for GBi to support conventional uni-prediction and bi-prediction, so that a total of (extra_number_of_weights+3) weights can be used per block. For example, if the value of extra_number_of_weights exists at the sequence, picture, and slice levels and is 2 (e.g., w A , w B ), 1 (e.g., w C ), and 3 (e.g., w D , w E , w F ), the available weight values ​​at the sequence, image, and slice levels are {w A ,w B}, {w A ,w C}, and {0,0.5,1}∪{w D ,w E ,w F In this example, the W mentioned in the section "Generalized Bi-Prediction" is L1 is the slice-level weight set.

[0045] Implicit weight derivation process In some embodiments, the slice-level weight set W L1 The weight values ​​in are derived through template matching at both the encoder and decoder without signaling. As shown in Figure 7, the implicit weight values ​​are derived from the predicted signal of the template with the motion information of the current block (T0 and T1) and the reconstructed signal of the template (i.e., T C ) can be derived by minimizing the difference between w * = argmin w Σ X(T C [x]-(1-w)*T0[x+v0]-w*T1[x+v1]) 2 (4) where v0 and v1 are the motion vectors of the current block. Since equation (4) is a quadratic function, if T0 and T1 are not exactly the same, a closed-form expression for the derived weights can be obtained, i.e.,

[0046]

number

[0047] This becomes:

[0048] The effectiveness of this method can be seen when the weight values ​​of the signal of the current block are correlated with those of the associated template prediction signal; however, this is not always guaranteed, especially when the pixels in the current block and its associated template are located in different motion objects. To maximize the prediction performance of GBi, when extra_number_of_weights≧1, a flag implicit_weight_flag can be signaled at the slice, picture, or sequence level to determine whether implicit weights are used. Once this is set equal to 1, W L1 The weight values ​​for the last slice level in are derived and therefore do not need to be signaled. For example, the w F does not need to be signaled and when implicit_weight_flag is equal to 1, the weight for the block can be derived implicitly.

[0049] Weight scaling process In some embodiments, the explicit weight values ​​can be further scaled by using two scaling factors gbi_scaling_factors indicated at the picture level. Due to possible illumination changes with high dynamics over time within a picture, the dynamic range of these weight values ​​may not be sufficient to cover all these cases. Weighted bi-prediction can compensate for illumination differences between pictures, but it is not always guaranteed that this is possible in the underlying video codec. Therefore, these scaling factors can be used to adjust for illumination differences across multiple reference pictures when weighted bi-prediction is not used.

[0050] The first scaling factor is W L1 Each explicit weight value in can be increased. With this, the prediction function of GBi in equation (3) becomes

[0051]

number

[0052] where α is the first scaling coefficient of the current image, and w1' represents the scaled weight value (i.e., α*(w1-0.5)+0.5). The first equation of Equation (6) can be expressed in the same format as Equation (3). The only difference is the weight value applied to Equation (6) and Equation (3).

[0053] A second scaling factor can be used to reduce the difference in illumination between the associated reference images of P0 and P1. With this scaling factor, equation (6) becomes:

[0054]

number

[0055] where s, s0, and s1 represent the signaled second scaling factors in the current image and its two reference images, respectively. According to equation (7), one optimal assignment for the variable s can be the average value of the samples in the current image. Therefore, the average values ​​of the reference images can be expected to be similar after the second scaling factor is applied. Due to commutativity, applying the scaling factors P0 and P1 is the same as applying them to the weight values, and thus equation (7) becomes

[0056]

number

[0057] can be reinterpreted as follows:

[0058] Therefore, the weight set construction process can be expressed as a function of explicit weights, implicit weights, scaling factors, and reference images. For example, the slice-level weight set W L1 is {0,0.5,1}∪{(s / s1)*w D ',(s / s1)*w E ',(s / s1)*w F '}, and the weight set for L0 is {1,0.5,1}∪{(s / s0)*(1-w D '),(s / s0)*(1-w E '),(s / s0)*(1-w F ')} where s1 is the average sample value of the reference images in list L1 for the current block and s0 is the average sample value of the reference images in list L0 for the current block.

[0059] Weight reduction An exemplary embodiment is L1The method operates to further reduce the number of weight values ​​in the range between 0 and 1. Two exemplary techniques for reducing the weight values ​​are described in detail below. The first technique operates in response to motion compensation prediction results, and the second technique operates based on weight values ​​outside the range between 0 and 1.

[0060] Prediction-based approach. Given the motion information of a PU, not all weights produce substantially different bi-predictive signals. An exemplary embodiment eliminates redundant weight values ​​(that produce similar bi-predictive signals) and keeps only one weight among the redundant values ​​to produce a bi-predictive signal. L1 This property can be exploited by making the weights more compact. To do so, a function can be used that calculates the similarity between the bi-predictive signals of two different weight values. This function can be, but is not limited to, a cosine similarity function, which operates as follows:

[0061]

number

[0062] where w (i) and w (j) is W L1 where v0 and v1 are given bi-predictive motion information, and P[x;w,v0,v1] represents the same prediction function as shown in equations (3), (6), and (8) with given w, v0, and v1. When the value of equation (9) is below a given threshold (indicated by weight_pruning_threshold at the slice level), one of the weights can be reduced according to the slice-level syntax pruning_smaller_weight_flag. If this flag is set equal to 1, the reduction process is performed by pruning smaller weights. L1 From w (i) and w (j) The smaller weight of W is removed. Otherwise (when this flag is set equal to 0), the larger one is removed. In an exemplary embodiment, this reduction process isL1 is applied to each pair of weight values ​​in the resulting W L1 No two weight values ​​in will produce similar bi-predictive signals. The similarity between two weight values ​​can also be evaluated by using the sum of absolute transformed differences (SATD). To reduce computational complexity, this similarity can be evaluated using two sub-sampled prediction blocks. For example, it can be calculated using sub-sampled rows or sub-sampled columns of samples in both the horizontal and vertical directions.

[0063] Weight-Based Approach. Weight values ​​outside the range between 0 and 1 (or out-of-range weights, for short) can behave differently with respect to coding performance under different coding structures (e.g., hierarchical or low-latency structures). Exploiting this fact, exemplary embodiments utilize a set of sequence-level indices, weight_control_idx, to restrict the use of out-of-range weights separately for each temporal layer. In such an embodiment, each weight_control_idx is associated with all images in a particular temporal layer. Depending on how this index is constructed, out-of-range weights can be conditionally available for use or reduced, as follows: ● If weight_control_idx=0, W L1 remains unchanged for the associated image. ● When weight_control_idx=1, W L1 Weights outside the range within are not available for the associated image. ● When weight_control_idx=2, W L1 Out-of-range weights within are available for some of the associated images whose reference frames come purely from the past (e.g., low-delay configurations in HEVC and JEM). ● When weight_control_idx=3, W L1 Out-of-range weights in are available for the associated picture only when the slice-level flag mvd_ll_zero_flag in HEVC and JEM is enabled.

[0064] Weight Index Encoding Exemplary systems and methods for binarization and codeword assignment for weight index coding are described in more detail below.

[0065] Binarization process for weight index coding In an exemplary embodiment, each weight index (weight_idx) is converted to a unique binary representation (binary_weight_idx) through a systematic code before entropy encoding. For illustrative purposes, the tree structure of the proposed binarization method is shown in Figure 8. The first bit of binary_weight_idx is used to distinguish between uni-prediction (associated with weights equal to 0 or 1) and bi-prediction. In uni-prediction branches, another bit is signaled to indicate which of two reference lists is referenced: reference list L0 (associated with a weight index pointing to a weight value equal to 0) or reference list L1 (associated with a weight index pointing to a weight value equal to 1). In bi-prediction branches, each leaf node is signaled with W L1Each leaf node in a bi-predictive branch is assigned a unique weight index value associated with one of the remaining weight values ​​in the bi-predictive branch, i.e., the weight values ​​that are neither 0 nor 1. Exemplary video codecs support a variety of systematic codes for binarizing bi-predictive branches, such as truncated unary codes (e.g., FIG. 8) and Exponential-Golomb codes. An exemplary technique in which each leaf node in a bi-predictive branch is assigned a unique weight_idx is described in more detail below. Through examining this tree structure, each weight index can be mapped to or recovered from a unique codeword (e.g., binary_weight_idx).

[0066] Adaptive codeword assignment for weight index coding. In an exemplary binary tree structure, each leaf node corresponds to one codeword. To reduce the signaling overhead of weight indices, various adaptive codeword allocation methods can be used to map each leaf node in a bi-predictive branch to a unique weight index. Exemplary methods include predetermined codeword allocation, block adaptive codeword allocation, temporal layer-based codeword allocation, and time-delay CTU adaptive codeword allocation. These exemplary methods update codeword allocation in a bi-predictive branch based on the occurrence of weight values ​​used in previously coded blocks. Frequently used weights can be assigned to codewords of shorter length (i.e., shallower leaf nodes in a bi-predictive branch), while the others can be assigned to codewords of relatively longer length.

[0067] 1) Predetermined Codeword Allocation. Using predetermined codeword allocation, a fixed codeword allocation can be provided for leaf nodes in bi-predictive branches. In this method, the weight index associated with a weight of 0.5 is assigned the shortest codeword, i.e., node i in FIG. 8 . Weight values ​​other than 0.5 can be divided into two sets: set 1 contains all values ​​greater than 0.5, ordered in ascending order, and set 2 contains all values ​​less than 0.5, ordered in descending order. These two sets are then interleaved to form set 3, starting from set 1 or set 2. All remaining codewords, ordered from shortest to longest, are assigned weight values ​​in set 3 in order. For example, if the set of all possible weight values ​​in a bi-predictive branch is {0.1, 0.3, 0.5, 0.7, 0.9}, then set 1 is {0.7, 0.9}, set 2 is {0.3, 0.1}, and set 3 is {0.7, 0.3, 0.9, 0.1} if interleaving starts with set 1. Codewords ordered from shortest to longest length are assigned, in order, to 0.5, 0.7, 0.3, 0.9, and 0.1.

[0068] This allocation can change in circumstances where a codec can drop one motion vector difference when two sets of motion information are transmitted. For example, this behavior can be seen in HEVC from the slice-level flag mvd_ll_zero_flag. In this case, the alternative codeword allocation is based on the weight value (e.g., w) that is greater than 0.5 and closest to it. + ) and assign it to the weight index associated with w +The weight index associated with the nth smallest (or largest) weight value among those larger (or smaller) than (n+1) is assigned the (2n+1)th (or 2nth) shortest codeword. Based on the previous example, the codewords ordered from shortest to longest in length are assigned, in order, to 0.7, 0.5, 0.9, 0.3, and 0.1. The resulting assignments for both examples are shown in Figures 11A-11B.

[0069] 2) Block-adaptive codeword allocation using causal-neighboring weights. The weight values ​​used in causal neighboring blocks can be correlated with those used for the current block. Based on this knowledge and a given codeword allocation method (e.g., constant allocation or alternate allocation), weight indices that can be found from causal neighboring blocks are promoted to leaf nodes in the bi-predictive branch with shorter codeword lengths. Similar to the motion vector prediction list construction process, causal neighboring blocks can be accessed in an ordered order as shown in FIG. 12A, and at most two weight indices can be promoted. As can be seen, from the bottom-left block to the left block, the first available weight index (if any) can be promoted with the shortest codeword length, and from the top-right block to the top-left block, the first available weight index (if any) can be promoted with the second shortest codeword length. For the other weight indices, they can be assigned to the remaining leaf nodes according to their codeword lengths in the initial given assignment, from shallowest to deepest. Figure 12B provides an example showing how a given codeword assignment can adapt itself to causal neighborhood weights. In this example, a constant assignment is used, and weight values ​​equal to 0.3 and 0.9 are encouraged.

[0070] 3) Temporal Layer-Based Codeword Allocation. In an exemplary method using temporal layer-based codeword allocation, the proposed video encoder adaptively switches between constant and alternative codeword allocation. Based on the use of weight indices from previously coded pictures in the same temporal layer or with the same QP value, the optimal codeword allocation method using weight indices with the smallest expected codeword length can be found as follows:

[0071]

number

[0072] where L m (w) represents the codeword length of w using a codeword allocation method m, and W L1 Bi is the set of weights used for bi-prediction only, and Prob k (w) represents the cumulative probability of w over k images in the temporal layer. Once the best codeword assignment method is determined, it can be applied to encoding the weight index for the current image or analyzing the binary weight index.

[0073] Several different methods are contemplated for accumulating the use of weight indices across temporal images. An exemplary method can be formulated in a common equation:

[0074]

number

[0075] where w i is W L1 is a weight in Count jwhere (w) represents the occurrence of a certain weight value in the jth image of the temporal layer, n determines the number of recent images stored, and λ is a forgetting term. n and λ are encoder-specific parameters that can adapt themselves to various encoding conditions, such as n=0 for scene changes in each image, or smaller λ for motion video.

[0076] In some embodiments, the choice of codeword allocation method can be explicitly indicated using slice-level syntax elements. Therefore, the decoder does not need to maintain the use of weight indices over time, and thus analytical dependence on weight indices across temporal images can be completely avoided. Such a method also improves decoding robustness.

[0077] 4) CTU Adaptive Codeword Allocation. Switching between different methods for codeword allocation based solely on weight usage of previously coded images may not always match well with that of the current image. This may be due to a lack of consideration for the weight usage of the current image. In an exemplary embodiment using CTU adaptive codeword allocation, Prob k (w i ) can be updated based on the weight usage of coded blocks in the current CTU row and the CTU row immediately above. Assuming that the current image is the (k+1)th image in the temporal layer, Prob k (w i ) can be updated for each CTU as follows:

[0078]

number

[0079] where B represents the set of coded CTUs in the current CTU row and the CTU row immediately above, and Count' j(w) represents the occurrence of a weight value in the jth CTU collected in set B. k (w i ) is updated, it is applied to equation (10) and thus the best codeword allocation method can be determined.

[0080] Extension to advanced temporal prediction techniques Embodiments are described below for extending the application of generalized bi-prediction to use with other coding techniques, including local illumination compensation, weighted bi-prediction, merge mode, bi-predictive optical flow, affine motion prediction, overlapped block motion compensation, and frame rate up-conversion bi-prediction.

[0081] Local illumination compensation and weighted biprediction An exemplary generalized bi-prediction technique can be implemented on top of local illumination compensation (IC) and / or weighted bi-prediction, among other techniques. Both IC and weighted bi-prediction operate to compensate for illumination changes on reference blocks. One difference between them is that in the use of IC, the weights (c0 and c1) and offset values ​​(o0 and o1) are derived through block-by-block template matching, while in the use of weighted bi-prediction, these parameters are explicitly signaled per slice. Using these parameters (c0, c1, o0, o1), the prediction signal of GBi is

[0082]

number

[0083] where the weight scaling process described in the section "Weight Scaling Process" above is applied. When this scaling process is not applied, the predicted signal of GBi can be calculated as P[x]=(1-w1)*(c0*P0[x+v0]+o0)+w1*(c1*P1[x+v1]+o1) (14) It can be calculated as follows:

[0084] For example, the use of these combined prediction processes described in Equation (13) or Equation (14) can be signaled at the sequence level, image level, or slice level. Signaling can be performed separately for the combination of GBi and IC and the combination of GBi and weighted bi-prediction. In some embodiments, the combined prediction process of Equation (13) or Equation (14) is applied only when the weight value (w1) is not equal to 0, 0.5, or 1. In particular, when the use of the combined prediction process is active, the value of the block-level IC flag (used to indicate the use of IC) determines whether GBi (with w1 ≠ 0, 0.5, 1) is combined with IC. Otherwise, when the combined prediction process is not used, GBi and IC (with w1 ≠ 0, 0.5, 1) perform two independent prediction modes, and this block-level IC flag does not need to be signaled and is therefore inferred to be zero.

[0085] In some embodiments, whether GBi can be combined with IC or combined with weighted bi-prediction is signaled separately in the sequence parameter set (SPS), picture parameter set (PPS), or slice header using high-level syntax using flags such as gbi_ic_comb_flag and gbi_wb_comb_flag. In some embodiments, when gbi_ic_comb_flag is equal to 0, GBi and IC are not combined, and therefore, GBi weight values ​​(w1 ≠ 0, 0.5, 1) and IC flags do not coexist for any bi-predictive coding unit. For example, in some embodiments, if a GBi weight with w1 ≠ 0, 0.5, 1 is signaled for a coding unit, there is no signaled IC flag and the value of this flag is inferred; otherwise, the IC flag is explicitly signaled. In some embodiments, when gbi_ic_comb_flag is equal to 1, GBi and IC are combined and for one coding unit, both the GBi weight and the IC flag are signaled independently. The same semantics can be applied to gbi_wb_comb_flag.

[0086] Merge Mode In some embodiments, a merge mode is used to infer not only motion information from causally neighboring blocks but also the weight index of the block in the same time. The access order for causally neighboring blocks (as shown in FIG. 13) can be the same as that specified in HEVC, where spatial blocks are accessed in the order of left, top, top-right, bottom-left, and top-right blocks, while temporal blocks are accessed in the order of bottom-right and center blocks. In some embodiments, up to five merge candidates are constructed: four from spatial blocks and one from temporal blocks. Given the merge candidates, the GBi prediction process specified in Equation (3), Equation (8), Equation (13), or Equation (14) can be applied. Note that the weight index does not need to be signaled, since it is inferred from the weight information of the selected merge candidate.

[0087] In the JEM platform, an additional merge mode called advanced temporal motion vector prediction (ATMVP) is provided. In some embodiments of the present disclosure, ATMVP is combined with GBi prediction. In ATMVP, the motion information of each 4x4 unit in one CU is derived from the motion field of a temporal reference picture. In an exemplary embodiment using ATMVP, when the GBi prediction mode is enabled (e.g., when extra_number_of_weights is greater than 0), the weight index for each 4x4 unit can also be inferred from that of the corresponding temporal block in the temporal reference picture.

[0088] Bi-predictive optical flow In some embodiments, the weight values ​​of GBi can be applied to a bi-predictive optical flow (BIO) model. Based on the motion compensated prediction signals (P0[x+v0] and P1[x+v1]), BIO applies an offset value O to reduce the difference between two corresponding samples in L0 and L1 in terms of their spatial vertical and horizontal gradient values. BIO [x] can be estimated. To combine this offset value with the GBi prediction, equation (3) becomes P[x]=(1-w1)*P0[x+v0]+w1*P1[x+v1]+o BIO [x] (15) where w1 is the weight value used to perform GBi prediction. This offset value can also be applied to other GBi variations as an additive offset after the predicted signals in P0 and P1 are scaled, as in equation (8), equation (13), or equation (14).

[0089] Affine prediction In an exemplary embodiment, GBi prediction can be combined with affine prediction in a manner similar to an extension to conventional bi-prediction. However, there is a difference in the basic processing unit used to perform motion compensation. Affine prediction is a model-based motion field derivation technique for forming a fine-grained motion field representation of a PU, where the motion field representation of each 4x4 unit is derived based on uni-predictive or bi-predictive transform motion vectors and given model parameters. Because all motion vectors point to the same reference image, there is no need to adapt weight values ​​to each 4x4 unit. Therefore, weight values ​​can be shared across each unit, and only one weight index per PU can be signaled. Using the motion vectors and weight values ​​in the 4x4 unit, GBi can be performed on a unit-by-unit basis, so the same Equations (3), (8), (13), and (14) can be directly applied without modification.

[0090] Overlapped Block Motion Compensation Overlapped block motion compensation (OBMC) is a method for providing a prediction of a sample's intensity value based on a motion compensation signal derived from the sample's own motion vector and those of its causal neighbors. In an exemplary embodiment of GBi, weight values ​​can also be taken into account in motion compensation for OBMC. An example is shown in Figure 14, where a sub-block B1 in a current block has three motion compensation prediction blocks, each of which is formed by using motion information and weight values ​​from blocks N1, N5, or B1 itself, and the resulting prediction signal for B1 can be the weighted average of the three.

[0091] Frame rate upconversion bi-prediction In some embodiments, GBi can work together with frame rate up-conversion (FRUC). Two different modes can be used for FRUC. If the current image falls between the first reference image in L0 and the first reference image in L1, bi-prediction mode can be used. If the first reference image in L0 and the first reference image in L1 are both forward or backward reference images, uni-prediction mode can be used. The bi-prediction case in FRUC is described in detail below. In JEM, equal weights (i.e., 0.5) are used for FRUC bi-prediction. Although the quality of the two predictors in FRUC bi-prediction can be different, combining two predictors with unequal prediction quality using equal weights can be suboptimal. The use of GBi can improve the final bi-prediction quality due to the use of unequal weights. In an exemplary embodiment, the weight values ​​of GBi are derived for blocks coded using FRUC bi-prediction and therefore do not need to be signaled. L1Each weight value in is evaluated independently for each of the 4x4 sub-blocks in the PU using the FRUC bi-predictive MV derivation process. For a 4x4 block, the weight value that results in the minimal bilateral matching error (which is the sum of the absolute differences between the two uni-predictive motion compensation predictors associated with the two reference lists) is selected.

[0092] In an exemplary embodiment, FRUC bi-prediction is a decoder-side MV derivation technique that derives MVs by using bi-predictive matching. For each PU, a list of candidate MVs collected from causally related neighboring blocks is formed. Under the assumption of constant motion, each candidate MV is linearly projected onto the first reference image in the other reference list, and the scaling factor for the projection is the ratio of the reference image (e.g., at time t0 or t1) to the current image (t c ) is set to be proportional to the time distance between them. Take FIG. 15 as an example where v0 is a candidate MV associated with reference list L0. Then v1 is set to be v0*(t1-t c ) / (t0-t c ) Therefore, the bilateral matching error can still be calculated for each candidate MV, and the first MV that reaches the minimum bilateral matching error is selected from the candidate list. This first MV is called v0 INIT It is expressed as v0 INIT Starting from the first MV pointed to by, decoder-side motion estimation is performed to find an MV within a predetermined search range, and the MV that reaches the minimum bilateral matching error is selected as the PU-level MV. Assuming v1 is the projected MV, the optimization process is

[0093]

number

[0094] When FRUC bi-prediction is combined with GBi, the search process in equation (16) can be formulated as follows:L1 is reformulated using the weight value w in

[0095]

number

[0096] This PU level v0 can be further refined independently for each 4x4 sub-block within the PU using the same bilateral matching as in equation (17), as shown in equation (18).

[0097]

number

[0098] W L1 For each available weight value in , Equation (18) can be evaluated, and the weight value that minimizes the bilateral matching error is selected as the optimal weight. At the end of the evaluation process, each 4x4 sub-block in the PU has its own bi-prediction MV and weight value for performing generalized bi-prediction. The complexity of such an exhaustive search method may be high because the weights and motion vectors are searched in a joint manner. In another embodiment, the search for the optimal motion vector and the optimal weight can be performed in two steps. In the first step, the motion vector for each 4x4 block can be obtained using Equation (18) by setting w to an initial value, for example, w=0.5. In the second step, the optimal weight can be searched for given the optimal motion vector.

[0099] In yet another embodiment, three steps can be applied to improve the accuracy of the motion search. In the first step, the initial weights are applied to the initial motion vector v0. INIT The initial optimal weight is searched using w INIT In the second step, the motion vector for each 4x4 block is expressed as w INITIn the third step, given the optimal motion vector, the final optimal weights can be searched for.

[0100] From equations (17) and (18), the goal is to minimize the difference between the two weighted predictors associated with the two criteria lists, respectively. Negative weights may not be appropriate for this purpose. In one embodiment, the FRUC-based GBi mode only evaluates weight values ​​greater than zero. To reduce complexity, the calculation of the sum of absolute differences can be performed using partial samples within each sub-block. For example, the sum of absolute differences can be calculated using only samples located in even-numbered rows and columns (or alternatively, odd-numbered rows and columns).

[0101] GBi predictive search strategy Initial reference list for bipredictive search A method for improving the prediction performance of GBi by determining which of the two reference lists should be searched first in the motion estimation (ME) stage of bi-prediction is described below. As with conventional bi-prediction, the ME stage cost, i.e., Cost(t i ,U j )=Σ x |I[x]-P[x]|+λ*Bits(t i ,u j ,weight index) (19) There are two motion vectors associated with the reference list L0 and the reference list L1, respectively, that are determined to minimize I[x], where I[x] is the original signal of sample x located at x in the current image, P[x] is the predicted signal of GBi, and t i and u jare motion vectors pointing to the i-th reference image in L0 and the j-th reference image in L1, respectively; λ is the Lagrangian parameter used in the ME stage; and the Bits(●) function estimates the number of bits for encoding the input variables. Each of equations (3), (8), (13), and (14) can be applied to replace P[x] in equation (19). For simplicity, equation (3) is considered as an example for the following process. Therefore, the cost function in equation (19) is Cost(t i ,u j )=Σ x |I[x]-(1-w1)*P0[x+t i ]-w1*P1[x+u j ]|+λ*Bits(t i ,u j ,weight index) (20) can be rewritten as:

[0102] The two parameters to be determined (t i and u j ), an iterative procedure can be utilized. The first such procedure can proceed as follows: 1.{u j |∀j}, and then use the best move in t i ,∀i is optimized. 2.{t i Using the best move in |∀i}, u j ,∀j is optimized. 3.t i and u j Repeat steps 1 and 2 until no longer changes or the maximum number of iterations is reached.

[0103] A second exemplary iterative procedure can proceed as follows. 1.{t i Using the best move in |∀i}, u j ,∀j is optimized. 2.{u j|∀j}, and then use the best move in t i ,∀i is optimized. 3.u j and t i Repeat steps 1 and 2 until no longer changes or the maximum number of iterations is reached.

[0104] Which iteration procedure is selected depends on t i and u j can depend only on the ME stage cost, i.e.,

[0105]

number

[0106] where the ME stage cost function can be: Cost(t i )=Σ x |I[x]-P0[x+t i ]|+λ*Bits(t i ) (twenty two) Cost(u j )=Σ x |I[x]-P1[x+u j ]|+λ*Bits(u j ) (twenty three) However, this initialization process may not be optimal in cases where 1-w1 and w1 are not equal. One of the weight values ​​is very close to 0, e.g., w1=lim. w→0 A typical example is where w happens to have a lower ME stage cost for its associated move than the other. In this case, equation (20) becomes Cost(t i ,u j )=Σ x |I[x]-P0[x+t i ]|+λ*Bits(t i ,u j ,weight index) (24) degenerates into u jThe overhead spent for ∑ i = ...

[0107]

number

[0108] is.

[0109] Binary Search for Weight Index Because the number of weight values ​​evaluated can introduce additional complexity to the encoder, exemplary embodiments utilize a binary search method to prune less likely weight values ​​in the early stages of encoding. In one such search method, conventional uni-prediction (associated with weights of 0 and 1) and bi-prediction (associated with a weight of 0.5) are performed right from the start, with W L1 The weight values ​​in are divided into four groups, namely, A=[w min ,0], B=[0,0.5], C=[0.5,1], and D=[1,w max ] can be classified as w min and w max are respectively, W L1 represents the minimum and maximum weight values ​​in min < 0, and w max It is assumed that the weight is > 1. To determine the range of possible weight values, the following rules can be applied: ● If w=0 gives a better ME stage cost than w=1, the following rules apply. If w=0.5 gives a better ME stage cost than w=0 and w=1, then based on the weight values ​​in B, we select the weight set W (0) is formed. Otherwise, based on the weight values ​​in A, (0) is formed. ● Otherwise (if w=1 gives a better ME stage cost than w=0), the following rules apply. If w=0.5 gives a better ME stage cost than w=0 and w=1, then W (0) is formed. Otherwise, based on the weight values ​​in D, (0) is formed.

[0110] W (0) After the formation of W (0) according to the minimum and maximum values ​​in min and w max The value of W can be reset. (0) are associated with A and D, respectively, min and w in D max The ME stage cost can be calculated.

[0111] At the kth iteration, the iterative process continues until three or more weight values ​​remain in the set, and W (k) Assuming the process is at the kth iteration, the iterative process can be specified as follows: 1.(w min +w max ) / 2, which is the weight value closest to middle Execute GBi using 2.w middle But, w min and w max If you give a better ME stage cost than [w min ,w middle ] and [w middle ,w max ] independently, W (k+1) The recursive process for is invoked and the iterative process jumps to step 6. 3. No, w middle But, w min and w maxIf it gives a worse ME stage cost than , the iterative process terminates. 4. No, w min But, w max If you give a better ME stage cost than [w min ,w middle Based on the weight values ​​in (k+1) is formed and the iterative process jumps to step 6. 5.Otherwise (w min But, w max (If it gives a worse ME stage cost than [w middle ,w max Based on the weight values ​​in (k+1) is formed and the iterative process jumps to step 6. 6.W (k+1) If the number of weight values ​​remaining in W is three or more, (k+1) According to the minimum and maximum values ​​in min and w max is reset and the iterative process returns to step 1; otherwise, the iterative process ends.

[0112] After the iterative process stops, the weight value that achieved the lowest ME stage cost among all test values ​​can be selected to perform generalized bi-prediction.

[0113] Weight estimation for non-2N×2N partitions In some embodiments, after testing each weight value for a 2N×2N partition, the best-performing weight values ​​other than 0, 0.5, and 1 can serve as estimates for optimal weight values ​​for the non-2N×2N partition. In some embodiments, assuming n unique estimates exist, only weight values ​​equal to 0, 0.5, 1, and n estimates are evaluated for the non-2N×2N partition.

[0114] Partition size estimation for non-2N x 2N partitions In some embodiments, not all non-2N×2N partitions are tested by an exemplary video encoder. Non-2N×2N partitions can be classified into two subcategories: symmetric motion partitions (SMPs), which have 2N×N and N×2N partition types, and asymmetric motion partitions (AMPs), which have 2N×nU, 2N×nD, nL×2N, and nR×2N partition types. If the rate-distortion (RD) cost for a partition in SMP is smaller than that of 2N×2N, some of the partition types in AMP are evaluated in the encoder. The decision on which partition types in AMP to test can depend on which of 2N×N and N×2N shows better performance in terms of RD cost. If the rate-distortion cost of 2N×N is smaller, then partition types 2N×nU and 2N×nD can be further checked; otherwise (if the cost of N×2N is smaller), then partition types nL×2N and nR×2N can be further checked.

[0115] Fast Parameter Estimation for Multipass Encoding In an exemplary embodiment using a multi-pass encoder, optimized prediction parameters (such as block motion and weight values) from an earlier encoding pass can be adopted as initial parameter estimates in a subsequent encoding pass. In such an encoder, a coding block partitioned from an image can be predicted and encoded twice or even more times, resulting in a significant increase in encoding complexity. One technique for reducing this complexity is to buffer the optimized prediction parameters from the initial encoding pass and take them as initial parameter estimates for further refinement in subsequent encoding passes. For example, if an inter prediction mode happens to be the best mode in the initial pass, the encoder will only evaluate the inter prediction mode in the remainder of the encoding pass. In some embodiments, buffering is performed to reduce the W L1 The prediction parameters related to GBi are selected based on the weight value in the GBi, the bi-predictive MV associated with the selected weight value, the IC flag, the OBMC flag, the integer motion vector (IMV) flag, and the coded block flag (CBF). In such an embodiment, the values ​​of these buffered parameters can be reused or refined in subsequent encoding passes. In particular, when the above-mentioned bi-predictive MVs are taken, these MVs can serve as initial search positions for the bi-predictive search. They are then refined in the motion estimation stage and then taken as the initial search positions for the next encoding pass.

[0116] Exemplary Bitstream Communication Framework Figure 16 is a schematic diagram illustrating an example of a coded bitstream structure. The coded bitstream 1000 consists of multiple NAL (Network Abstraction Layer) units 1001. The NAL units may contain coded sample data, such as coded slices 1006, or high-level syntax metadata, such as parameter set data, slice header data 1005, or supplemental enhancement information data 1007 (sometimes referred to as SEI messages). Parameter sets are high-level syntax structures that contain basic syntax elements that can apply to multiple bitstream layers (e.g., video parameter sets 1002 (VPS)), or to coded video sequences within a single layer (e.g., sequence parameter sets 1003 (SPS)), or to multiple coded pictures within a single coded video sequence (e.g., picture parameter sets 1004 (PPS)). Parameter sets can be transmitted together with the coded pictures of the video bitstream or through other means (including out-of-band transmission using a reliable channel, hard coding, etc.). The slice header 1005 is also a high-level syntax structure that can contain some picture-related information that is relatively small or that is relevant only to a certain slice or picture type. The SEI message 1007 carries information that may not be needed by the decoding process, but can be used for various other purposes, such as picture output timing or display, and loss detection and concealment.

[0117] FIG. 17 is a schematic diagram illustrating an example of a communication system. The communication system 1300 may include an encoder 1302, a communication network 1304, and a decoder 1306. The encoder 1302 may communicate with the network 1304 via a connection 1308, which may be a wired or wireless connection. The encoder 1302 may be similar to the block-based video encoder of FIG. 1. The encoder 1302 may include a single-layer codec (e.g., FIG. 1) or a multi-layer codec. For example, the encoder 1302 may be a multi-layer (e.g., two-layer) scalable coding system with picture-level ILP support. The decoder 1306 may communicate with the network 1304 via a connection 1310, which may be a wired or wireless connection. The decoder 1306 may be similar to the block-based video decoder of FIG. 2. The decoder 1306 may include a single-layer codec (e.g., FIG. 2) or a multi-layer codec. For example, the decoder 1306 can be a multi-layer (eg, two-layer) scalable decoding system with picture-level ILP support.

[0118] The encoder 1302 and / or decoder 1306 may be incorporated within a wide variety of wired communication devices and / or wireless transmit / receive units (WTRUs), such as, but not limited to, digital televisions, wireless broadcast systems, network elements / terminals, servers such as content or web servers (e.g., Hypertext Transfer Protocol (HTTP) servers), personal digital assistants (PDAs), laptop or desktop computers, tablet computers, digital cameras, digital recording devices, video game devices, video game consoles, cellular or satellite wireless telephones, and / or digital media players.

[0119] The communication network 1304 can be any suitable type of communication network. For example, the communication network 1304 can be a multiple-access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communication network 1304 can enable multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication network 1304 can utilize one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), and / or single-carrier FDMA (SC-FDMA). The communication network 1304 can include multiple connected communication networks. The communication network 1304 can include the Internet and / or one or more private commercial networks, such as cellular networks, WiFi hotspots, and / or Internet Service Provider (ISP) networks.

[0120] 18 is a system diagram of an example WTRU in which the encoder or decoder described herein may be implemented. As shown, the example WTRU 1202 may include a processor 1218, a transceiver 1220, a transmit / receive element 1222, a speaker / microphone 1224, a keypad or keyboard 1226, a display / touchpad 1228, non-removable memory 1230, removable memory 1232, a power source 1234, a global positioning system (GPS) chipset 1236, and / or other peripherals 1238. It will be understood that the WTRU 1202 may include any subcombination of the above elements while remaining consistent with an embodiment. Furthermore, a terminal in which an encoder (e.g., encoder 100) and / or decoder (e.g., decoder 200) are incorporated may include some or all of the elements shown in the WTRU 1202 of FIG. 18 and described herein with reference thereto.

[0121] The processor 1218 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), a graphics processing unit (GPU), multiple microprocessors, one or more microprocessors in conjunction with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 1218 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 1202 to operate in a wired and / or wireless environment. The processor 1218 may be coupled to the transceiver 1220, which may be coupled to the transmit / receive element 1222. While FIG. 18 depicts the processor 1218 and the transceiver 1220 as separate components, it will be understood that the processor 1218 and the transceiver 1220 may be integrated together in an electronic package and / or chip.

[0122] The transmit / receive element 1222 can be configured to transmit signals to and / or receive signals from another base station over the air interface 1215. For example, in one or more embodiments, the transmit / receive element 1222 can be an antenna configured to transmit and / or receive RF signals. In one or more embodiments, the transmit / receive element 1222 can be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In one or more embodiments, the transmit / receive element 1222 can be configured to transmit and / or receive both RF and light signals. It will be understood that the transmit / receive element 1222 can be configured to transmit and / or receive any combination of wireless signals.

[0123] 18 as a single element, the WTRU 1202 may include any number of transmit / receive elements 1222. More specifically, the WTRU 1202 may utilize MIMO technology. Thus, in one embodiment, the WTRU 1202 may include two or more transmit / receive elements 1222 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 1215.

[0124] The transceiver 1220 may be configured to modulate signals to be transmitted by the transmit / receive element 1222 and / or demodulate signals received by the transmit / receive element 1222. As mentioned above, the WTRU 1202 may have multi-mode capabilities. Thus, the transceiver 1220 may include multiple transceivers to enable the WTRU 1202 to communicate via multiple RATs, such as, for example, UTRA and IEEE 802.11.

[0125] The processor 1218 of the WTRU 1202 may be coupled to and may receive user input data from a speaker / microphone 1224, a keypad 1226, and / or a display / touchpad 1228 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit). The processor 1218 may also output user data to the speaker / microphone 1224, the keypad 1226, and / or the display / touchpad 1228. Additionally, the processor 1218 may obtain information from and store data in any type of suitable memory, such as non-removable memory 1230 and / or removable memory 1232. The non-removable memory 1230 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 1232 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In one or more embodiments, the processor 1218 may obtain information from and store data in memory that is not physically located on the WTRU 1202, such as on a server or home computer (not shown).

[0126] The processor 1218 may receive power from the power source 1234 and may be configured to distribute and / or control the power to other components within the WTRU 1202. The power source 1234 may be any suitable device for powering the WTRU 1202. For example, the power source 1234 may include one or more dry batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0127] The processor 1218 may be coupled to a GPS chipset 1236, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 1202. In addition to or instead of information from the GPS chipset 1236, the WTRU 1202 may receive location information over the air interface 1215 from a terminal (e.g., a base station) and / or may determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 1202 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0128] The processor 1218 may further be coupled to other peripherals 1238, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 1238 may include an accelerometer, an orientation sensor, a motion sensor, a proximity sensor, an e-compass, a satellite transceiver, a digital camera and / or video recorder (for photos and / or videos), a Universal Serial Bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulation (FM) radio unit, and software modules such as digital music player, media player, video game player modules, an internet browser, etc.

[0129] By way of example, the WTRU 1202 may be configured to transmit and / or receive wireless signals and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a tablet computer, a personal computer, a wireless sensor, a home appliance, or any other terminal capable of receiving and processing compressed video communications.

[0130] The WTRU 1202 and / or the communication network (e.g., communication network 1304) may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 1215 using Wideband Code Division Multiple Access (WCDMA). WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink Packet Access (HSDPA) and / or High Speed ​​Uplink Packet Access (HSUPA). The WTRU 1202 and / or the communication network (e.g., communication network 1304) may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 1215 using Long Term Evolution (LTE) and / or LTE Advanced (LTE-A).

[0131] The WTRU 1202 and / or the communication network (e.g., communication network 1304) may implement a radio technology such as IEEE 802.16 (e.g., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), and GSM EDGE (GERAN). The WTRU 1202 and / or the communication network (e.g., communication network 804) may implement a radio technology such as IEEE 802.11 or IEEE 802.15.

[0132] It should be noted that the various hardware elements in one or more of the described embodiments are referred to as “modules,” and perform (i.e., execute, perform, etc.) various functions described herein associated with the respective modules. As used herein, a module includes hardware deemed appropriate by one of ordinary skill in the art for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices). It is noted that each described module may also include executable instructions for performing one or more functions described as being performed by the respective module, which may take the form of or include hardware (i.e., hard-wired) instructions, firmware instructions, and / or software instructions, etc., and may be stored in one or more suitable non-transitory computer-readable media, generally referred to as RAM, ROM, etc.

[0133] Although features and elements have been described above in particular combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein can be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted over wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor in conjunction with software can be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. 1. A video encoding method, comprising: determining that a merge mode is to be used to predict at least a current block in a current image; determining a current merging candidate from among a plurality of merging candidates, the current merging candidate being used to predict the current block; and deriving motion information of the current block from motion information of the current merging candidate; deriving a weight index of the current block from a weight index of the current merging candidate, the weight index of the current block indicating a first weight and a second weight used in weighted bi-prediction of the current block, and at least one of the first weight and the second weight having a value not equal to 0, 0.5, or 1; obtaining, from motion information of the current block, a prediction of the current block as a weighted sum of a first reference block of a first reference image and a second reference block of a second reference image, wherein the first reference block is weighted by the first weight and the second reference block is weighted by the second weight; encoding an indication that the merge mode is being used to predict the current block; and encoding an indication that the current merge candidate of the plurality of merge candidates is being used; and encoding the current block in the merge mode in response to the prediction of the current block; A method comprising:

2. 2. The method of claim 1, further comprising determining that an affine motion model is used to predict the current block, and that the first reference block and the second reference block used in the weighted sum are reference sub-blocks derived from the affine motion model.

3. The method of claim 1 , wherein at least one of the first weight and the second weight of the weighted bi-prediction of the current block is identified from a set of predetermined weights using the weight index.

4. The method of claim 1 , wherein the plurality of merging candidates includes at least one of spatially adjacent blocks or temporally adjacent blocks.

5. 1. A video decoding method comprising: decoding an indication that a merge mode is being used to predict at least a current block in a current image; obtaining an indication that a current merging candidate from a plurality of merging candidates is used to predict the current block; deriving motion information of the current block from motion information of the current merging candidate; deriving a weight index of the current block from a weight index of the current merging candidate, the weight index of the current block indicating a first weight and a second weight used in weighted bi-prediction of the current block, and at least one of the first weight and the second weight having a value not equal to 0, 0.5, or 1; obtaining a prediction of the current block from a weighted sum of a first reference block of a first reference picture and a second reference block of a second reference picture, the first reference block being weighted by the first weight and the second reference block being weighted by the second weight; decoding the current block in the merge mode in response to the prediction of the current block; A method comprising:

6. 6. The method of claim 5, further comprising decoding an indication that an affine motion model is used to predict the current block, and wherein the first reference block and the second reference block used in the weighted sum are reference sub-blocks derived from the affine motion model.

7. The method of claim 5 , wherein at least one of the first weight and the second weight of the weighted bi-prediction of the current block is identified from a set of predetermined weights using the weight index.

8. The method of claim 5 , wherein the plurality of merging candidates includes at least one of spatially adjacent blocks or temporally adjacent blocks.

9. 1. A video encoding device, comprising: determining that a merge mode is to be used to predict at least a current block in a current image; determining a current merging candidate from among a plurality of merging candidates, the current merging candidate being used to predict the current block; and deriving motion information of the current block from motion information of the current merging candidate; deriving a weight index of the current block from a weight index of the current merging candidate, the weight index of the current block indicating a first weight and a second weight used in weighted bi-prediction of the current block, and at least one of the first weight and the second weight having a value not equal to 0, 0.5, or 1; obtaining, from motion information of the current block, a prediction of the current block as a weighted sum of a first reference block of a first reference image and a second reference block of a second reference image, wherein the first reference block is weighted by the first weight and the second reference block is weighted by the second weight; encoding an indication that the merge mode is being used to predict the current block; and encoding an indication that the current merge candidate of the plurality of merge candidates is being used; and encoding the current block in the merge mode in response to the prediction of the current block; 1. An apparatus comprising a processor configured to perform at least the following:

10. 10. The apparatus of claim 9, wherein the processor is further configured to at least determine that an affine motion model is used to predict the current block, and that the first reference block and the second reference block used in the weighted sum are reference sub-blocks derived from the affine motion model.

11. The apparatus of claim 9 , wherein at least one of the first weight and the second weight for weighted bi-prediction of the current block is identified from a set of predetermined weights using the weight index.

12. The apparatus of claim 9 , wherein the plurality of merging candidates includes at least one of spatially adjacent blocks or temporally adjacent blocks.

13. 1. A video decoding device, comprising: decoding an indication that a merge mode is being used to predict at least a current block in a current image; obtaining an indication that a current merging candidate from a plurality of merging candidates is used to predict the current block; deriving motion information of the current block from motion information of the current merging candidate; deriving a weight index of the current block from a weight index of the current merging candidate, the weight index of the current block indicating a first weight and a second weight used in weighted bi-prediction of the current block, and at least one of the first weight and the second weight having a value not equal to 0, 0.5, or 1; obtaining a prediction of the current block from a weighted sum of a first reference block of a first reference picture and a second reference block of a second reference picture, the first reference block being weighted by the first weight and the second reference block being weighted by the second weight; decoding the current block in the merge mode in response to the prediction of the current block; 1. An apparatus comprising a processor configured to perform at least the following:

14. 14. The apparatus of claim 13, wherein the processor is further configured to at least perform decoding an indication that an affine motion model is being used to predict the current block, and the first reference block and the second reference block used in the weighted sum are reference sub-blocks derived from the affine motion model.

15. The apparatus of claim 13 , wherein at least one of the first weight and the second weight for weighted bi-prediction of the current block is identified from a set of predetermined weights using the weight index.

16. The apparatus of claim 13 , wherein the plurality of merging candidates includes at least one of spatially adjacent blocks or temporally adjacent blocks.

17. A non-transitory computer-readable medium containing instructions that, when executed, cause one or more processors to: decoding an indication that a merge mode is being used to predict at least a current block in a current image; obtaining an indication that a current merging candidate from a plurality of merging candidates is used to predict the current block; deriving motion information of the current block from motion information of the current merging candidate; deriving a weight index of the current block from a weight index of the current merging candidate, the weight index of the current block indicating a first weight and a second weight used in weighted bi-prediction of the current block, and at least one of the first weight and the second weight having a value not equal to 0, 0.5, or 1; obtaining a prediction of the current block from a weighted sum of a first reference block of a first reference picture and a second reference block of a second reference picture, the first reference block being weighted by the first weight and the second reference block being weighted by the second weight; decoding the current block in the merge mode in response to the prediction of the current block; A non-transitory computer-readable medium for performing a method comprising:

18. 20. The non-transitory computer-readable medium of claim 17, further comprising instructions that cause one or more processors to perform a method, further comprising decoding an indication that an affine motion model is being used to predict the current block, wherein the first reference block and the second reference block used in the weighted sum are reference sub-blocks derived from the affine motion model.

19. 20. The non-transitory computer-readable medium of claim 17, wherein at least one of the first weight and the second weight of the weighted bi-prediction of the current block is identified from a set of predetermined weights using the weight index.

20. 20. The non-transitory computer-readable medium of claim 17, wherein the plurality of merging candidates comprises at least one of spatially adjacent blocks or temporally adjacent blocks.

Images