Self-adaptive constraint method and device for bidirectional prediction under external boundary condition

Abstract

A method and apparatus for inter prediction performed by at least one processor. The method for inter prediction comprises: receiving an encoded video bitstream, the encoded video bitstream comprising a current image, and a first reference image and a second reference image associated with the current image, the current image comprising at least one block encoded according to an inter bidirectional prediction mode, the at least one block includes a first motion vector pointing to the first reference image and a second motion vector pointing to the second reference image; determining whether an out-of-boundary condition associated with the first motion vector and the second motion vector is satisfied; and for an inter-frame bi-prediction mode, adaptively prohibiting bi-prediction from having a motion vector (MV) pointing outside the reference image boundary.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] Under the provisions of Title 35 of the United States Code, Section 119 (35 U.S.C. §119), this application claims priority to U.S. Provisional Application No. 63 / 323,749, filed with the United States Patent and Trademark Office on March 25, 2022, and U.S. Provisional Application No. 17 / 945,679, filed with the United States Patent and Trademark Office on September 15, 2022, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0002] Methods and apparatus consistent with example embodiments of the present disclosure relate to motion vector (MV) constraints such that in an inter-frame bi-prediction mode, bi-prediction with MVs pointing outside a reference picture boundary is adaptively prohibited. Background Art

[0003] The Video Coding Experts Group (ITU-T VCEG (Q6 / 16)) and the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) Moving Picture Experts Group (ISO / IEC MPEG (JTC 1 / SC 29 / WG11)) released the H.265 / High Efficiency Video Coding (HEVC) standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4), respectively. In 2015, the two standards organizations jointly established the Joint Video Exploration Team (JVET) to explore the potential of developing the next generation video coding standard beyond HEVC. In October 2017, the two standards organizations issued a joint call for proposals (CfP) on video compression capabilities beyond HEVC. As of February 15, 2018, a total of 22 CfP responses on Standard Dynamic Range (SDR), 12 CfP responses on High Dynamic Range (HDR), and 12 CfP responses on 360 video categories were submitted. In April 2018, all received CfP responses were evaluated at the 122MPEG / 10th JVET meeting. As a result of this meeting, JVET officially launched the standardization process for the next generation of video coding beyond HEVC. The new standard was named Versatile Video Coding (VVC), and JVET was renamed the Joint Video Experts Group. In 2020, ITU-T VCEG (Q6 / 16) and ISO / IECMPEG (JTC 1 / SC 29 / WG 11) released the VVC video coding standard (version 1).

[0004] For each inter-frame prediction coding unit (CU), motion parameters consisting of motion vector, reference picture index and reference picture list usage index, as well as additional information required by the new coding features of VVC, are used for inter-frame prediction sample generation.

[0005] An image may have at least one block encoded according to a bidirectional prediction mode, wherein the at least one block includes a first motion vector (MV) pointing to a first reference image and a second MV pointing to a second reference image. When the first MV or the second MV points to a location outside a frame boundary, a constraint for the bidirectional prediction mode may be applied. For example, the constraint may change the coding mode of at least one block from a bidirectional prediction mode to another coding mode such as a unidirectional prediction mode. Current efforts to apply inter-frame bidirectional prediction constraints may not always be beneficial to coding efficiency when the motion vector (MV) exceeds a frame boundary. Summary of the invention

[0006] A simplified overview of one or more embodiments of the present disclosure is presented below to provide a basic understanding of these embodiments. This overview is not an extensive overview of all contemplated embodiments, and is intended to neither identify the key or core elements of all embodiments nor describe the scope of any or all embodiments. The sole purpose of this overview is to present some concepts of one or more embodiments of the present disclosure in a simplified form as a prelude to a more detailed description presented subsequently.

[0007] For inter-frame bi-prediction mode, methods, apparatuses, and non-transitory computer-readable media are disclosed for MV constraints to adaptively prohibit bi-prediction from having MVs pointing outside of reference picture boundaries.

[0008] According to an exemplary embodiment, a method for inter-frame prediction is performed by at least one processor. The method includes receiving an encoded video code stream, the encoded video code stream including a current image and a first reference image and a second reference image associated with the current image. The current image includes at least one block encoded according to an inter-frame bidirectional prediction mode, the at least one block including a first motion vector pointing to the first reference image and a second motion vector pointing to the second reference image. The method also includes determining whether an out of boundary condition associated with the first motion vector and the second motion vector is satisfied. The method also includes, in response to determining that the out of boundary condition is satisfied, determining whether a disabling condition for disabling the out of boundary condition is satisfied. The method also includes, in response to determining that the out of boundary condition is satisfied and the disabling condition is not satisfied, performing the following operations: (i) changing the encoding mode of at least one block from the inter-frame bidirectional prediction mode to another encoding mode, and (ii) decoding at least one block according to another encoding mode. The method also includes, in response to determining that the out of boundary condition is not satisfied or the disabling condition is satisfied, decoding at least one block according to the inter-frame bidirectional prediction mode.

[0009] According to an exemplary embodiment, an apparatus for inter-frame prediction includes at least one memory configured to store computer program code, and at least one processor, the at least one processor being configured to access the computer program code and run according to the instructions of the computer program code. The computer program code includes a receiving code, the receiving code is configured to cause at least one processor to receive an encoded video code stream, the encoded video code stream includes a current image, and a first reference image and a second reference image associated with the current image. The current image includes at least one block encoded according to an inter-frame bidirectional prediction mode, the at least one block includes a first motion vector pointing to the first reference image and a second motion vector pointing to the second reference image. The computer program code also includes a first determination code, the first determination code is configured to cause at least one processor to determine whether an out-of-boundary condition associated with the first motion vector and the second motion vector is satisfied. The computer program also includes a second determination code, the second determination code is configured to: in response to determining that the out-of-boundary condition is satisfied, cause at least one processor to determine whether a disabling condition for disabling the out-of-boundary condition is satisfied. The computer program code also includes a change code and a decoding code, and in response to determining that the out-of-boundary condition is satisfied and the disabling condition is not satisfied, (i) the change code causes the at least one processor to change the encoding mode of the at least one block from the inter-frame bidirectional prediction mode to another encoding mode, and (ii) the decoding code causes the at least one processor to decode the at least one block according to the other encoding mode. The computer program also includes, in response to determining that the out-of-boundary condition is not satisfied and the disabling condition is satisfied, the decoding code included by the computer program causes the at least one processor to decode the at least one block according to the inter-frame bidirectional prediction mode.

[0010] An exemplary embodiment provides a non-transitory computer-readable medium storing instructions, which, when executed by a processor, causes the processor to perform a method for inter-frame prediction. The method includes receiving an encoded video code stream, the encoded video code stream including a current image and a first reference image and a second reference image associated with the current image. The current image includes at least one block encoded according to an inter-frame bidirectional prediction mode, the at least one block including a first motion vector pointing to the first reference image and a second motion vector pointing to the second reference image. The method also includes determining whether an out-of-boundary condition associated with the first motion vector and the second motion vector is satisfied. The method also includes, in response to determining that the out-of-boundary condition is satisfied, determining whether a disabling condition for disabling the out-of-boundary condition is satisfied. The method also includes, in response to determining that the out-of-boundary condition is satisfied and the disabling condition is not satisfied, performing the following operations: (i) changing the encoding mode of at least one block from the inter-frame bidirectional prediction mode to another encoding mode, and (ii) decoding at least one block according to another encoding mode. The method also includes, in response to determining that the out-of-boundary condition is not satisfied and the disabling condition is satisfied, decoding at least one block according to the inter-frame bidirectional prediction mode.

[0011] Additional embodiments will be set forth in the description which follows and, in part, will be apparent from the description, and / or may be learned by practice of the embodiments presented in the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other aspects, features and aspects of the embodiments of the present disclosure will become apparent through the following description in conjunction with the accompanying drawings. In the accompanying drawings:

[0013] Figure 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0014] Figure 2 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0015] Figure 3 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

[0016] Figure 4 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0017] Figure 5 is an example of the location of spatial merging candidates according to an embodiment.

[0018] Figure 6 is an example of a candidate pair considering redundancy check for spatial merging candidates according to an embodiment.

[0019] Figure 7 is an example of motion vector scaling for temporal merging candidates according to an embodiment.

[0020] Figure 8 are examples of candidate positions C0 and C1 of the time domain merging candidates according to an embodiment.

[0021] Fig. 9A is an example of an affine motion field of a block according to an embodiment.

[0022] Fig. 9B is an example of an affine motion field of a block according to an embodiment.

[0023] Fig.10 is an example of an affine motion vector field of a block according to an embodiment.

[0024] Fig.11 is an example of the location of spatial merging candidates according to an embodiment.

[0025] Fig.12 is an example of control point motion vector inheritance according to an embodiment.

[0026] Fig.13is an example of positioning of candidate positions for constructing an affine merge pattern according to an embodiment.

[0027] Fig.14 is an example of a sub-block having a motion vector according to an embodiment.

[0028] Fig.15A It is an example of a spatial neighboring block used by Temporal Motion Vector Prediction (ATVMP).

[0029] Fig. 15B It is an example of a subblock-based temporal motion vector prediction (SbTMVP) process in VVC according to an embodiment.

[0030] Fig.16 is an example of an extended CU region used in Bi-Directional Optical Flow (BDOF) according to an embodiment.

[0031] Fig.17 is an example of decoding-side motion vector correction according to an embodiment.

[0032] Fig.18 is an example of GPM segmentation grouped by the same angle according to an embodiment.

[0033] Fig.19 is an example of unidirectional prediction motion vector selection for geometric partitioning mode according to an embodiment.

[0034] Fig. 20 is an example of generating warp weights using a geometric partitioning pattern according to an embodiment.

[0035] Fig.21 is a specification table of AmvrShift in VVC according to an embodiment.

[0036] Fig. 22 is an example of a sub-block level bi-directional prediction constraint according to an embodiment.

[0037] Fig.23 A flow chart of a method performed in accordance with an embodiment is depicted. DETAILED DESCRIPTION

[0038] The following detailed description of example embodiments refers to the accompanying drawings.The same reference numbers in different drawings may identify the same or similar elements.

[0039] The above disclosure provides illustration and description, but is not intended to be exhaustive or to limit the embodiments to the precise form disclosed. According to the above disclosure, modifications and variations are possible, or modifications and variations can be obtained from the practice of the embodiments. In addition, one or more features or components of some embodiments may be incorporated or combined into some embodiments (or one or more features of some embodiments). In addition, in the flowcharts and descriptions of the operations provided below, it should be understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed simultaneously (at least partially), and the order of one or more operations may be switched.

[0040] It is apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0041] Even though particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

[0042] Unless explicitly described, any element, behavior or instruction used herein should not be interpreted as critical or essential. In addition, as used herein, the article "a" is intended to include one or more items and can be used interchangeably with "one or more". If only one item is intended to be used, the term "one" or similar language is used. In addition, as used herein, the terms "having", "including", etc. are intended to be open terms. In addition, unless otherwise explicitly stated, the word "based on" means "based at least in part". In addition, expressions such as "at least one of A and B" or "at least one of A or B" should be understood to include only A, only B, or both A and B.

[0043] Throughout the specification, references to "some embodiments," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in connection with the indicated embodiment is included in at least some embodiments of the present solution. Thus, the phrases "in some embodiments," "in an embodiment," and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0044] In addition, the described features, advantages and characteristics of the present disclosure may be combined in any suitable manner in one or more embodiments. Based on the description herein, those skilled in the relevant art will recognize that the present disclosure may be implemented without one or more specific features or advantages of a particular embodiment. In other cases, additional features and advantages that may not be present in all embodiments of the present disclosure may be recognized in certain embodiments.

[0045] The disclosed methods may be used alone or in any order. In addition, the disclosed methods may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0046] Inter-frame prediction in VVC:

[0047] For each inter-prediction CU, motion parameters consisting of motion vector, reference picture index and reference picture list usage index, as well as additional information required for the new coding features of VVC, are used for inter-prediction sample generation. Motion parameters can be signaled explicitly or implicitly. When a CU is encoded using skip mode, the CU is associated with a prediction unit (PU) and has no significant residual coefficients, no encoded motion vector increments or reference picture indices. A merge mode is specified, whereby the motion parameters of the current CU (including spatial and temporal candidates) are obtained from neighboring CUs, as well as additional scheduling introduced in VVC. Merge mode can be applied to any inter-prediction CU, not just for skip mode. An alternative to merge mode is explicit transmission of motion parameters, where the motion vector, the corresponding reference picture index in each reference picture list and the reference picture list usage flag, as well as other required information, are explicitly signaled for each CU.

[0048] In VVC, the VTM reference software includes many new and improved inter-frame prediction coding tools, as described below:

[0049] extended combined forecasts;

[0050] Merge Motion Vector Difference (MMVD);

[0051] Advanced Motion Vector Prediction (AMVP) mode with symmetric Motion Vector Difference (MVD) signaling;

[0052] Affine motion compensated prediction;

[0053] Subblock-based Temporal Motion Vector Prediction (SbTMVP);

[0054] Adaptive Motion Vector Resolution (AMVR);

[0055] Motion field storage: 1 / 16 brightness sample MV storage and 8x8 motion field compression;

[0056] CU-level weighted bidirectional prediction (Bi-Prediction With Cu-Level Weight, BCW);

[0057] Bi-Directional Optical Flow (BDOF);

[0058] Decoder side motion vector refinement (DMVR);

[0059] Combined Inter And Intra Prediction (CIIP);

[0060] Geometric Partitioning Mode (GPM).

[0061] Details on inter prediction and related methods are provided below.

[0062] In VTM4, the merge candidate list is constructed by including the following five types of candidates in order:

[0063] The spatial MVP from the spatially adjacent CU;

[0064] Time domain MVP from the co-located CU;

[0065] History-based MVP from a First Input First Output (FIFO) table;

[0066] Average MVP in pairs;

[0067] Zero MV.

[0068] The size of the merge list is signaled in the slice header, and the maximum size of the merge list allowed in VTM4 is 6. For each CU code in merge mode, the index of the best merge candidate is encoded using a truncated unary binary code (TU). The first bin of the merge index is encoded using context, and bypass encoding is used for other bins.

[0069] The generation process of each type of merge candidates is provided in this section.

[0070] The derivation of spatial merge candidates in VVC is the same as that in HEVC. Figure 5 Up to four merge candidates are selected from the candidates at the positions shown. The order of derivation is B1, A1, B0, A0 and B2. Position B2 is considered only when any CU at position A0, B0, B1, A1 is not available (for example, because the CU belongs to another slice or tile) or is intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates. The redundancy check ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the redundancy check. Instead, only Figure 6 The pairs linked by arrows in , and the corresponding candidates are added to the list only when the candidates used for redundancy check have different motion information.

[0071] In this operation, only one candidate is added to the list. In particular, in the derivation of this temporal merge candidate, the scaled motion vector is derived based on the co-located CU belonging to the co-located reference picture. The reference picture list to be used for the derivation of the co-located CU is explicitly signaled in the slice header. Figure 7 The scaled motion vector for the temporal merge candidate is obtained by scaling the motion vector of the co-located CU using the picture order count (POC) distances tb and td, where tb is defined as the POC difference between the reference image of the current image and the current image, and td is defined as the POC difference between the reference image of the co-located image and the co-located image. The reference image index of the temporal merge candidate is set to zero.

[0072] like Figure 8 As shown, the position of the temporal candidate is selected between candidate C0 and candidate C1. If the CU at position C0 is not available, is intra-coded, or is outside the current row of the coding tree unit (CTU), position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate.

[0073] In HEVC, only the translational motion model is used for motion compensation prediction (MCP). In the real world, there are many kinds of motions, such as zooming in / out, rotation, perspective motion, and other irregular motions. In the current VTM, block-based affine transformation motion compensation prediction is applied. Fig. 9A and Fig. 9B As shown, the affine motion field of a block is described by the motion information of two control points (4 parameters) or three control point motion vectors (6 parameters).

[0074] For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0075]

[0076] The above motion vector can also be described as:

[0077]

[0078] For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0079]

[0080] The above motion vector can also be described as:

[0081]

[0082] Among them, (mv 0x , mv 0y ) is the motion vector of the upper left control point, (mv 1x , mv 1y ) is the motion vector of the upper right control point, (mv 2x , mv 2y ) is the motion vector of the lower left control point.

[0083] To simplify motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector for each 4×4 luminance sub-block, such as Fig.10 As shown, the motion vector of the center sample of each subblock is calculated according to the above equation and rounded to 1 / 16 fractional precision. Then, a motion compensation interpolation filter is applied to generate a prediction for each subblock using the derived motion vector. The subblock size of the chroma component is also set to 4×4. The MV of a 4×4 chroma subblock is calculated as the average of the MVs of the four corresponding 4×4 luminance subblocks.

[0084] Like translational motion inter prediction, there are two affine motion inter prediction modes: affine merge mode and affine AMVP mode.

[0085] AF_MERGE mode can be applied to CUs with width and height greater than or equal to 8. In this mode, the control point motion vector (CPMV) of the current CU is generated based on the motion information of the spatial neighboring CUs. There may be up to five control point motion vector prediction (CPMVP) candidates, and the index is written to the bitstream to indicate a CPMVP candidate to be used for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list:

[0086] Inherited affine merge candidates inferred from the CPMV of neighboring CUs;

[0087] The constructed affine merged candidate CPMVP derived using the translation MV of the adjacent CU;

[0088] Zero MV.

[0089] In VTM3, there are at most two inherited affine candidates, which are derived from the affine motion model of the neighboring blocks, one inherited affine candidate from the left neighboring CU and one inherited affine candidate from the upper neighboring CU. Fig.11 As shown in . For the left predictor, the scanning order is A0->A1, and for the upper predictor, the scanning order is B0->B1->B2. Only the first inheritance candidate from each side is selected. No pruning check is performed between two inheritance candidates. When a neighboring affine CU is identified, the control point motion vectors of the neighboring affine CU are used to derive the CPMVP candidate in the affine merge list of the current CU. Fig.12 As shown in the figure, if the lower left adjacent block A is encoded in affine mode, the motion vector v2 of the upper left corner, the motion vector v3 of the upper right corner, and the motion vector v4 of the lower left corner of the CU containing block A are obtained. When the 4-parameter affine model is used to encode block A, two CPMVs of the current CU are calculated based on v2 and v3. In the case of using a 6-parameter affine model to encode block A, three CPMVs of the current CU are calculated based on v2, v3, and v4.

[0090] The constructed affine candidate is constructed by combining the adjacent translation motion information of each control point. The motion information of the control point is obtained from Fig.13 The specified spatial and temporal neighbors are derived as shown. CPMV k (k=1, 2, 3, 4) represents the motion vector of the kth control point. For CPMV1, blocks are checked in the order B2->B3->A2, and the MV of the first available block is used. For CPMV2, blocks are checked in the order B1->B0. For CPMV3, blocks are checked in the order A1->A0. If TMVP is available, TMVP is used as CPMV4.

[0091] After obtaining the MVs of the four control points, an affine merge candidate is constructed based on the motion information. The following combinations of the control point MVs are used for construction in the following order:

[0092] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}.

[0093] The combination of 3 CPMVs constructs a 6-parameter affine merge candidate, and the combination of 2 CPMVs constructs a 4-parameter affine merge candidate. To avoid the motion scaling process, if the reference index of the control point is different, the relevant combination of control point MVs is discarded.

[0094] After checking inherited affine merge candidates and constructed affine merge candidates, if the list is still not full, a zero MV is inserted at the end of the list.

[0095] Affine AMVP mode can be applied to CUs with width and height greater than or equal to 16. The affine flag at the CU level is written to the bitstream to indicate whether the affine AMVP mode is used, and then another flag is signaled to indicate whether it is 4-parameter affine or 6-parameter affine. In this mode, the difference between the CPMV of the current CU and the predictor CPMVP of these CPMVs is written to the bitstream. The affine AVMP candidate list size is 2, and the affine AVMP candidate list is generated by using the following four types of CPVM candidates in order:

[0096] Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs;

[0097] The constructed affine AMVP candidate CPMVP derived using the translation MV of the adjacent CU;

[0098] Translational MV from neighboring CU;

[0099] Zero MV.

[0100] The order in which inherited affine AMVP candidates are checked is the same as that for inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as the current block are considered. When an inherited affine motion predictor is inserted into the candidate list, the pruning process is not applied.

[0101] The constructed AMVP candidates are Fig.13Derived from the specified spatial neighbors shown. Use the same check order as the affine merge candidate construction. In addition, the reference image index of the neighboring blocks is also checked. Use the first block in the check order that is inter-coded and has the same reference image as the current CU. There is only one. When the current CU is encoded using a 4-parameter affine mode and both mv0 and mv1 are available, mv0 and mv1 are added to the affine AMVP list as a candidate. When the current CU is encoded using a 6-parameter affine mode and all three CPMVs are available, the three CPMVs are added to the affine AMVP list as a candidate. Otherwise, the constructed AMVP candidate is set to unavailable.

[0102] If after checking the inherited affine AMVP candidates and the constructed AMVP candidates, the affine AMVP list candidates are still less than 2, mv0, mv1, and mv2 are added in order as translation MVs to predict all control point MVs of the current CU (if available). Finally, if the affine AMVP list is still not full, it is filled with zero MVs.

[0103] Compared with pixel-based motion compensation, sub-block-based affine motion compensation can save memory access bandwidth and reduce computational complexity, but at the expense of prediction accuracy. In order to achieve finer-grained motion compensation, Prediction Refinement With Optical Flow (PROF) is used to correct sub-block-based affine motion compensation prediction without increasing the memory access bandwidth for motion compensation. In VVC, after performing sub-block-based affine motion compensation, the brightness prediction samples are corrected by adding the difference derived from the optical flow equation. PROF is described as the following four operations:

[0104] Operation 1): Perform sub-block based affine motion compensation to generate a sub-block prediction I(i, j).

[0105] Operation 2): At each sample position, a 3-tap filter [-1, 0, 1] is used to calculate the spatial gradient g of the sub-block prediction x (i, j) and g y (i, j). The gradient calculation is exactly the same as the gradient calculation of BDOF

[0106] g x (i,j)=(I(i+1,j)>>shift1)-(I(i-1,j)>>shift1) (5)

[0107] g y (i,j)=(I(i,j+1)>>shift1)-(I(i,j-1)>>shift1) (6)

[0108] Shift1 is used to control the accuracy of the gradient. The sub-block (i.e., 4x4) prediction is extended by one sample on each side for gradient calculation. To avoid additional memory bandwidth and additional interpolation calculations, these extended samples on the extended boundary are copied from the nearest integer pixel position in the reference image.

[0109] Operation 3): Calculate the brightness prediction refinement through the following optical flow equation:

[0110] ΔI(i, j) = g x (i, j)*Δv x (i, j)+g y (i, j)*Δv y (i,j) (7)

[0111] Wherein, Δv(i, j) is the difference between the sample MV calculated for the sample position (i, j) (denoted as v(i, j)) and the sub-block MV of the sub-block to which the sample (i, j) belongs, as Fig.14 As shown, Δv(i, j) is quantized in units of 1 / 32 brightness sampling accuracy.

[0112] Since the affine model parameters and the sample positions relative to the sub-block center do not change between different sub-blocks, Δv(i, j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. dx(i, j) is the distance from the sample position (i, j) to the sub-block (x SB ,y SB ), and dy(i, j) is the distance from the sample position (i, j) to the sub-block (x SB ,y SB) The vertical offset of the center, Δv(i, j) can be derived by the following equation:

[0113]

[0114]

[0115] To maintain accuracy, the sub-block (x SB ,y SB ) is calculated as ((W SB -1) / 2, (H SB -1) / 2), where W SB and H SB are the width and height of the sub-block respectively.

[0116] For the 4-parameter affine model, we have the following equations:

[0117]

[0118] For the 6-parameter affine model, we have the following equations:

[0119]

[0120] Among them, (v 0x , v 0y ) is the motion vector of the upper left control point, (v 1x , v 1y ) is the upper right control point motion vector, and (v 2x , v 2y ) is the motion vector of the lower left control point, w and h are the width and height of the CU respectively.

[0121] Operation 4): Finally, the brightness prediction refinement ΔI(i, j) is added to the sub-block prediction I(i, j). The final prediction I' is generated by the following equation:

[0122] I′(i,j)=I(i,j)+AI(i,j)

[0123] PROF is not applicable to affine-coded CUs in two cases: 1) all control point MVs are the same, indicating that the CU has only translational motion; and 2) the affine motion parameters are larger than the specified limit, because the sub-block based affine MC is downgraded to CU based MC to avoid large memory access bandwidth requirements.

[0124] VVC supports the sub-block based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located image to improve the motion vector prediction and merge mode of the CU in the current image. The same co-located image used by TMVP is used for SbTVMP. SbTMVP differs from TMVP in the following two main aspects:

[0125] TMVP predicts motion at the CU level, while SbTMVP predicts motion at the sub-CU level;

[0126] TMVP obtains the temporal motion vector from the co-located block in the co-located image (the co-located block is the lower right or center block relative to the current CU), while SbTMVP applies motion translation before obtaining the temporal motion information from the co-located image, where the motion translation is obtained based on the motion vector of one of the spatial neighboring blocks of the current CU.

[0127] The SbTVMP process Fig.15A and Fig. 15B SbTMVP predicts the motion vector of a sub-CU within the current CU in two operations. In the first operation, check Fig.15AThe spatial neighbor A1 in . If A1 has a motion vector that uses the co-located image as its reference image, then that motion vector is selected as the motion shift to be applied. If no such motion is identified, then the motion shift is set to (0, 0).

[0128] In the second operation, the motion translation identified in operation 1 is applied (i.e., added to the coordinates of the current block) to obtain Fig. 15B The sub-CU level motion information (motion vector and reference index) is obtained from the co-located image shown in FIG. Fig. 15B In the example in , it is assumed that the motion translation is set to the motion of block A1. Then, for each sub-CU, the motion information of the block corresponding to the sub-CU in the co-located image (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After identifying the motion information of the co-located sub-CU, the motion information of the co-located sub-CU is converted into the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference image of the temporal motion vector and the reference image of the current CU.

[0129] In HEVC, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference images, and / or by using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow weighted averaging of two prediction signals.

[0130] P bi-pred =((8-w)*P0+w*P1+4)>>3 (3-1)

[0131] In weighted average bidirectional prediction, five weights w∈{-2, 3, 4, 5, 10} are allowed. For each bidirectionally predicted CU, the weight w is determined in one of the following two ways: 1) For non-merged CUs, the weight index is written to the bitstream / the weight index is signaled after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency images, all 5 weights are used. For non-low-latency images, only 3 weights (w∈{3, 4, 5}) are used.

[0132] In the encoder, fast search algorithms are applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized below. More details can be found in the VTM software and JVET-L0646 document. When combined with AMVR, unequal weights are only conditionally checked for 1-pixel and 4-pixel motion vector accuracy if the current picture is a low-latency picture.

[0133] When combined with affine, affine ME is performed on unequal weights if and only if the affine mode is selected as the current best mode.

[0134] Unequal weights are only conditionally checked when the two reference pictures in bidirectional prediction are the same.

[0135] Unequal weights are not searched when certain conditions are met, which depend on the POC distance between the current picture and its reference picture, the coding QP, and the temporal level.

[0136] The BCW weight index is encoded using one context coded bit immediately following the bypass coded bit. The first context coded bit indicates whether equal weights are used; and if unequal weights are used, bypass coding is used to write an additional bit into the codestream to indicate the unequal weights used.

[0137] Weighted Prediction (WP) is a coding tool supported by the H.264 / AVC and HEVC standards for efficiently encoding video content with fading. Support for WP has also been added to the VVC standard. For each reference image in each list of reference image lists L0 and L1, WP allows weighting parameters (weights and offsets) to be written into the codestream. Then, during motion compensation, the weights and offsets of the corresponding reference images are applied. WP and BCW are designed for different types of video content. In order to avoid interaction between WP and BCW (which will complicate the design of the VVC decoder), if the CU uses WP, the BCW weight index is not written into the codestream, and w is inferred to be 4 (ie, equal weights are applied). For the merged CU, the weight index is inferred from the adjacent blocks based on the merge candidate index. This can be applied to the conventional merge mode and the inherited affine merge mode. For the constructed affine merge mode, affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index of the CU using the constructed affine merge mode is simply set equal to the BCW index of the first control point MV.

[0138] In VVC, CIIP and BCW cannot be applied jointly to a CU. When a CU is encoded using CIIP mode, the BCW index of the current CU is set to 2, i.e., equal weight.

[0139] The Bidirectional Optical Flow (BDOF) tool is included in VVC. JEM includes BDOF, which was previously known as BIO. BDOF in VVC is a simpler version than the JEM version, which requires less computation, especially in terms of the number of multiplications and multiplier size.

[0140] BDOF is used to modify the bidirectional prediction signal of a CU at the 4×4 sub-block level. BDOF is applied to a CU if it meets all of the following conditions:

[0141] Encode the CU using a "true" bi-directional prediction mode, i.e. one of the two reference pictures precedes the current picture in display order and the other follows the current picture in display order;

[0142] The distances from the two reference images to the current image (i.e., POC difference) are the same;

[0143] Both reference images are short-term reference images;

[0144] Do not use affine mode or SbTMVP merge mode to encode the CU;

[0145] The CU has more than 64 luma samples;

[0146] CU height and CU width are both greater than or equal to 8 luma samples;

[0147] The BCW weight index indicates equal weights;

[0148] WP is not enabled for the current CU;

[0149] CIIP mode is not used for the current CU.

[0150] BDOF is only applied to the luma component. As the name suggests, BDOF mode is based on the concept of optical flow, which assumes that the motion of objects is smooth. For each 4×4 sub-block, the motion refinement (v x , v y ). Motion refinement is then used to adjust the bidirectional prediction sample values ​​in the 4x4 sub-block. The following operations are applied to the BDOF process.

[0151] First, the horizontal gradient of the two prediction signals is calculated by directly computing the difference between two adjacent samples and vertical gradient Among them, k=0,1.

[0152]

[0153]

[0154] Among them, I (k) (i, j) is the sample value at the prediction signal coordinate (i, j) in list k, k=0, 1, and shift1 is calculated based on the luma bit depth bitDepth, such as shift1=max(6, bitDepth-6).

[0155] Then, the autocorrelations and cross-correlations between the gradients S1, S2, S3, S5, and S6 are calculated as follows:

[0156] S1=∑ (i，j)∈Ω Abs(ψ x (i, j)), S3 = Σ (i，j)∈Ω θ(i, j)·Sign(ψ x (i, j))

[0157] S2=∑ (i，j)∈Ω ψ x (i,j)·Sign(ψ y (i,j)) (3-26)

[0158] S5=Σ (i，j)∈Ω Abs(ψ y (i, j)), S6 = Σ (i，j)∈Ω θ(i, j)·Sign(ψ y (i, j))

[0159] in,

[0160]

[0161]

[0162] θ(i, j)=(I (1) (i,j)>>n b )-(I (0) (i,j)>>n b )

[0163] where Ω is a 6×6 window around the 4×4 sub-block, n a The value of is set to min(1, bitDepth-11), and n b The value of is set to min(4, bitDepth-8).

[0164] Then the cross-correlation and autocorrelation terms are used to derive the motion refinement (v x , v y ):

[0165]

[0166]

[0167] in, th′ BIO =2 max(5，BD-7) . represents a floor operation, and

[0168] Based on the motion refinement and gradients, the following adjustments are computed for each sample in the 4×4 sub-block:

[0169]

[0170] Finally, the BDOF samples of the CU are calculated by adjusting the bidirectional prediction samples as follows:

[0171] pred BDOF (x, y) = (I (0) (x,y)+I (1) (x, y) + b(x, y) + o offset )>>shift(3-30)

[0172] These values ​​are chosen so that the multipliers in the BDOF process do not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process remains within 32 bits.

[0173] In order to obtain the gradient value, it is necessary to generate some prediction samples I in the list k (k = 0, 1) outside the current CU boundary (k) (i, j). Fig.16 As shown, BDOF in VVC uses an extended row / column around the boundary of the CU. In order to control the computational complexity of generating prediction samples outside the boundary, the predicted pixels in the extended area (white positions) are generated by directly obtaining reference pixels at nearby integer positions (using floor() to operate on the coordinates) without performing interpolation operations, and a conventional 8-tap motion compensated interpolation filter is used to generate prediction samples within the CU (gray positions). These extended sample values ​​are only used in gradient calculations. For the remaining steps in the BDOF process, if any samples and gradient values ​​outside the CU boundary are needed, the samples and gradient values ​​are filled (i.e., repeated) from their nearest pixels.

[0174] When the width and / or height of a CU is greater than 16 luma samples, the CU will be split into sub-blocks with a width and / or height equal to 16 luma samples, and the sub-block boundaries are considered as CU boundaries in the BDOF process. The maximum unit size of the BDOF process is limited to 16x16. The BDOF process can be skipped for each sub-block. The BDOF process is not applied to the sub-block when the sum of absolute differences (SAD) between the initial L0 and L1 prediction samples is less than a threshold. The threshold is set equal to (8*W*(H>>1)), where W represents the sub-block width and H represents the sub-block height. To avoid the additional complexity of SAD calculation, the SAD between the initial L0 and L1 prediction samples calculated in the DVMR process is reused here.

[0175] If BCW is enabled for the current block, that is, the BCW weight index indicates unequal weights, bidirectional optical flow is disabled. Similarly, if WP is enabled for the current block, that is, the luma_weight_lx_flag flag is 1 for either of the two reference images, BDOF is also disabled. BDOF is also disabled when the CU is encoded using symmetric MVD mode or CIIP mode.

[0176] In order to improve the accuracy of MV in merge mode, decoder-side motion vector correction based on bilateral matching (BM) is applied in VVC. In the bidirectional prediction operation, the refined MV is searched around the initial MV in the reference image list L0 and the reference image list L1. The BM method calculates the distortion between two candidate blocks in the reference image list L0 and the list L1. Fig.17 As shown, the SAD between the red blocks of each MV candidate around the initial MV is calculated. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bidirectional prediction signal.

[0177] In VVC, the application of DMVR is restricted and only applies to CUs coded with the following modes and features:

[0178] CU-level merge mode with bi-directional prediction MV;

[0179] Relative to the current image, one reference image is a previous image and the other reference image is a future image;

[0180] The distances from the two reference images to the current image (i.e., POC difference) are the same;

[0181] Both reference images are short-term reference images;

[0182] The CU has more than 64 luma samples;

[0183] CU height and CU width are both greater than or equal to 8 luma samples;

[0184] The BCW weight index indicates equal weighting;

[0185] WP is not enabled for the current block;

[0186] CIIP mode is not used for the current block.

[0187] The refined MV derived through the DMVR process is used to generate inter-frame prediction samples and is used for temporal motion vector prediction for future image coding, while the original MV is used for the deblocking process and also for spatial motion vector prediction for future CU coding.

[0188] Other features of DMVR are mentioned in the following sub-clauses.

[0189] In DVMR, the search point is around the initial MV and the MV offset follows the MV difference mirror rule. In other words, any point examined by DMVR represented by a candidate MV pair (MV0, MV1) follows the following two equations:

[0190] MV0′=MV0+MV_offset (3-2)

[0191] MV1′=MV1-MV_offset (3-3)

[0192] Wherein, MV_offset represents the refinement offset between the initial MV and the refined MV of a reference image in the reference image. The refinement search range is two integer brightness samples starting from the initial MV. The search includes an integer sample offset search phase and a fractional sample refinement phase.

[0193] The integer sample offset search uses a 25-point full search. First, the SAD of the initial MV pair is calculated. If the SAD of the initial MV pair is less than the threshold, the integer sampling stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. In order to reduce the uncertainty of DMVR refinement, the original MV is given priority in the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates is reduced by 1 / 4 of the SAD value.

[0194] The integer sample search is followed by fractional sample refinement. To reduce computational complexity, the parametric error surface equations (rather than additional searches with SAD comparisons) are used to derive the fractional sample refinement. The fractional sample refinement is conditionally called based on the output of the integer sample search stage. The fractional sample refinement is further applied when the integer sample search stage terminates with a center with a minimum SAD in either the first or second iteration search.

[0195] In the sub-pixel offset estimation based on the parameter error surface, the center position cost and the costs of the four neighboring positions from the center are used to fit a two-dimensional (2-D) parabolic error surface equation of the following form:

[0196] E(x, y) = A(x min ) 2 +B(yy min ) 2 +C (3-4)

[0197] Among them, (x min ,y min) corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost value. By solving the above equation using the cost values ​​of the five search points, (x min ,y min ) is calculated as follows:

[0198] x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) (3-5)

[0199] y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0)) (3-6)

[0200] Among them, x min and min The value of is automatically clamped between -8 and 8, because all cost values ​​are positive, and the minimum value is E(0,0). This corresponds to a half-pixel offset with 1 / 16 pixel MV precision in VVC. min ,y min ) is added to the integer distance refinement MV to obtain the sub-pixel accurate refinement variation MV.

[0201] In VVC, the resolution of MV is 1 / 16 luma sample. Samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, the search points are around the initial fractional pixel MV with integer sample offsets, so during the DMVR search process, samples at these fractional pixel positions need to be interpolated. In order to reduce the computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter, DVMR does not access more reference samples than the conventional motion compensation process within a search range of 2 samples. After the refined MV is obtained using the DMVR search process, a conventional 8-tap interpolation filter is applied to generate the final prediction. In order to avoid accessing more reference samples in the normal MC process, samples are filled from these available samples, which are not required for the interpolation process based on the original MV, but are required for the interpolation process based on the refined MV.

[0202] When the width and / or height of a CU is greater than 16 luma samples, the CU will be further split into sub-blocks with width and / or height equal to 16 luma samples. The maximum unit size of the DMVR search process is limited to 16x16.

[0203] In VVC, geometric partitioning mode is supported for inter-frame prediction. By using CU-level flags, the geometric partitioning mode is written into the bitstream as a merge mode. Other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. For each possible CU size of w×h=2 m ×2 n For CUs of size 8x64 and 64x8 (where m, n∈{3…6}, excluding CUs of size 8x64 and 64x8), the geometric partitioning mode supports a total of 64 partitions.

[0204] When this mode is used, the CU is divided into two parts by a geometrically positioned straight line ( Fig.18 ). The position of the partitioning line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; only unidirectional prediction is allowed for each partition, that is, each part has one motion vector and one reference index. Unidirectional prediction motion constraints are applied to ensure that, as with traditional bidirectional prediction, only two motion compensated predictions are required for each CU. The unidirectional prediction motion for each partition is derived using the process described in 2.1.6.1.

[0205] If the geometric partitioning mode is used for the current CU, the geometric partitioning index and two merge indices (one for each partition) are written to the bitstream, where the geometric partitioning index indicates the partitioning mode of the geometric partition (angle and offset). The number of maximum GPM candidate sizes is explicitly written to the bitstream in the SPS, and the syntax binary code of the GPM merge index is specified. After predicting each part of the geometric partition, the sample values ​​along the edges of the geometric partitions are adjusted using a blending process with adaptive weights as in 2.1.6.2. This is the prediction signal for the entire CU, and the transform and quantization process will be applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using the geometric partitioning mode is stored as described in 2.1.6.3.

[0206] The unidirectional prediction candidate list is derived directly from the merge candidate list constructed according to the extended merge prediction process. Denote n as the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector of the nth extended merge candidate, with X equal to the parity check of n, is used as the nth unidirectional prediction motion vector of the geometric partitioning mode. These motion vectors are Fig.19 In the case where there is no corresponding LX motion vector of the n-th extended merge candidate, the L(1-X) motion vector of the same candidate is used instead as the unidirectional prediction motion vector of the geometric partitioning mode.

[0207] After predicting each part of the geometric partition using its own motion, fusion is applied to the two prediction signals to derive samples around the edges of the geometric partition. The fusion weights for each position of the CU are derived based on the distance between each position and the partition edge.

[0208] The distance from position (x, y) to the edge of the partition is derived as:

[0209]

[0210]

[0211]

[0212]

[0213] where i, j are the indices of the angle and offset of the geometry partition, respectively, which depend on the geometry partition index sent by the signal. x，j and ρ y，j The sign of depends on the angle index i.

[0214] The weights for each part of the geometric partition are derived as follows:

[0215] wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (3-11)

[0216]

[0217] w1(x,y)=1-w0(x,y) (3-13)

[0218] partIdx depends on the angle index i. Fig. 20 An example of weight w0 is shown in FIG.

[0219] Mv1 from the first part of the geometric partition, Mv2 from the second part of the geometric partition, and the combination Mv of Mv1 and Mv2 are stored in the motion field of the CU encoded in the geometric partition mode.

[0220] For each position in the motion field, the type of motion vector stored is determined as:

[0221] sType=abs(motionIdx)<32?2: (motionIdx≤0?(1-partIdx):partIdx) (3-43)

[0222] Where motionIdx is equal to d(4x+2, 4y+2), which is recalculated according to equation (3-36). partIdx depends on the angle index i.

[0223] If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise, if sType is equal to 2, then the combined Mv from Mv0 and Mv2 is stored. The combined Mv is generated using the following process:

[0224] If Mv1 and Mv2 come from different reference picture lists (one from L0 and the other from L1), Mv1 and Mv2 are simply combined to form a bidirectional prediction motion vector.

[0225] Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored.

[0226] The video coding standards HEVC and AVC / H.264 use a fixed motion vector resolution of one quarter luma sample. However, it is well known that an optimal balance between the displacement vector rate and the prediction error rate must be chosen to achieve overall rate-distortion optimality. VVC allows the motion vector resolution to be selected at the coding block level, thus allowing a balance between the bitrate and fidelity used to signal motion parameters. This is enabled by the AMVR mode. The AMVR mode is signaled at the coding block level if at least one component of the MVD is not equal to zero. The motion vector predictor is rounded to a given resolution so that the resulting motion vector is guaranteed to fall on a grid of the given resolution. For each given resolution, the corresponding AmvrShift value in Table 1 is also defined to specify the resolution of the motion vector difference by a left shift operation of the AmvrShift-bit. When the AMVR mode is enabled, the given motion vector differences (denoted as MvdL0 and MvdL1 in AMVP mode and MvdCpL0 and MvdCpL1 in affine AMVP mode) are modified as follows.

[0227] Out of boundary constraint for inter-frame bidirectional prediction:

[0228] Constraints on the inter bi-prediction coding mode may apply to bi-prediction when some reference samples used in motion compensation are outside the picture boundary.

[0229] In an example, when a current pixel with a bidirectional prediction motion vector has a motion vector pointing to a position beyond a distance threshold outside the image boundary on one of the two reference lists, the motion vector of the reference list is considered to be out of bounds, and the inter-frame prediction is changed to unidirectional prediction. Only the motion vector of the other reference list that is not out of bounds will be used for unidirectional prediction. In some methods, when the MVs of both reference lists are out of bounds, bidirectional prediction is not constrained.

[0230] In another example, constraints on bidirectional prediction are applied at the sub-block level. Fig.21 As shown, for each of the NxN sub-blocks within a coding block with an inter-frame bidirectional prediction MV, if the motion vector on one of the reference lists exceeds the M-pixel threshold outside the boundary of the reference image, the sub-block can be changed to a unidirectional prediction mode, using only the MVs on the reference list that do not exceed the threshold outside the boundary of the corresponding reference image.

[0231] In some methods, when bidirectional prediction is changed to unidirectional prediction due to out-of-bounds conditions, bidirectional prediction related tools may be disabled or modified. In an example, when bidirectional prediction restrictions are applied and unidirectional prediction is used, BDOF may be disabled. Embodiments of the present disclosure adaptively apply constraints to bidirectional prediction modes to improve coding efficiency and / or quality.

[0232] Figure 1 A simplified block diagram of a communication system (100) according to an embodiment of the present disclosure is shown. The system (100) may include at least two terminals (110, 120) interconnected via a network (150). For unidirectional transmission of data, a first terminal (110) may encode video data at a local location for transmission to another terminal (120) via the network (150). A second terminal (120) may receive encoded video data of another terminal from the network (150), decode the encoded data, and display the recovered video data. Unidirectional data transmission may be common in media service applications, etc.

[0233] Figure 1 A second pair of terminals (130, 140) is shown, which are provided to support bidirectional transmission of encoded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal (130, 140) can encode video data captured at a local location for transmission to the other terminal via a network (150). Each terminal (130, 140) can also receive encoded video data sent by the other terminal, decode the encoded data, and can display the recovered video data on a local display device.

[0234] exist Figure 1In the embodiment of the present invention, the terminals (110-140) can be shown as servers, personal computers and smart phones, and / or any other type of terminals. For example, the terminals (110-140) can be laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network (150) represents any number of networks that transmit encoded video data between the terminals (110-140), including, for example, wired and / or wireless communication networks. The communication network (150) can exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purposes of this discussion, unless explained below, the architecture and topology of the network (150) may be unimportant to the steps of the present disclosure.

[0235] Figure 2 The placement of a video encoder and a video decoder in a streaming environment is shown as an example of application of the disclosed subject matter. The disclosed subject matter can be equally applicable to other video-enabled applications, including, for example, video conferencing, digital television, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0236] like Figure 2 As shown, a streaming system (200) may include a capture subsystem (213), which may include a video source (201) and an encoder (203). The video source (201) may be, for example, a digital camera, and may be configured to create an uncompressed video sample stream (202). The uncompressed video sample stream (202) may provide a high data volume when compared to an encoded video stream, and may be processed by an encoder (203) coupled to the camera (201). The encoder (203) may include hardware, software, or a combination of hardware and software to enable or implement various aspects of the disclosed subject matter described in more detail below. The encoded video stream (204) may include a lower data volume when compared to the sample stream, and may be stored on a streaming server (205) for future use. One or more streaming clients (206) may access the streaming server (205) to retrieve a video stream (209), which may be a copy of the encoded video stream (204).

[0237] In an embodiment, the streaming server (205) may also operate as a Media-Aware Network Element (MANE). For example, the streaming server (205) may be configured to prune the encoded video bitstream (204) to potentially customize different bitstreams for one or more streaming clients (206). In an embodiment, the MANE may be provided separately from the streaming server (205) in the streaming system (200).

[0238] The streaming client (206) may include a video decoder (210) and a display (212). The video decoder (210) may, for example, decode a video stream (209) that is an input copy of an encoded video stream (204), and the video decoder (210) may create an output video sample stream (211) that may be rendered on a display (212) or another rendering device (not shown). In some streaming systems, the video streams (204, 209) may be encoded according to certain video coding / compression standards. Examples of such standards include, but are not limited to, ITU-T Recommendation H.265. Under development is a video coding standard informally referred to as Versatile Video Coding (VVC). Embodiments of the present disclosure may be used in the context of VVC.

[0239] Figure 3 An example operational block diagram of a video decoder (210) attached to a display (212) is shown according to an embodiment of the present disclosure.

[0240] The video decoder (210) may include a channel (312), a receiver (310), a buffer memory (315), an entropy decoder / parser (320), a scaler / inverse transform unit (351), an intra-frame prediction unit (352), a motion compensated prediction unit (353), an aggregator (355), a loop filter unit (356), a reference image memory (357), and a current image memory (). In at least some embodiments, the video decoder (210) may include an integrated circuit, a series of integrated circuits, and / or other electronic circuits. The video decoder (210) may also be partially or fully embodied in software running on one or more CPUs with associated memory.

[0241] In this and other embodiments, the receiver (310) can receive one or more encoded video sequences in a manner that receives one encoded video sequence at a time, the encoded video sequences to be decoded by the decoder (210), wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequence can be received from a channel (312), which can be a hardware / software link to a storage device storing the encoded video data. The receiver (310) can receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective consuming entities (not shown). The receiver (310) can separate the encoded video sequence from the other data. To combat network jitter, a buffer memory (315) can be coupled between the receiver (310) and the entropy decoder / parser (320) (hereinafter referred to as "parser"). When the receiver (310) receives data from a storage / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer (315) can be not used, or the buffer (315) can be very small. For use over a best-effort packet network such as the Internet, a buffer (315) may be necessary, may be relatively large, and may have an adaptive size.

[0242] The video decoder (210) may include a parser (320) to reconstruct symbols (321) from an entropy coded video sequence. The types of symbols include, for example, information for managing the operation of the decoder (210) and potential information for controlling a rendering device such as a display (212) that may be coupled to the decoder, such as Figure 2As shown. The control information for the (one or more) rendering devices may be in the form of, for example, a supplemental enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not shown). The parser (320) may parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may be based on a video coding technique or standard and may follow principles known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (320) may extract a subgroup parameter set for at least one pixel subgroup in the pixel subgroup in the video decoder from the encoded video sequence based on at least one parameter corresponding to the group. The pixel subgroup may include a group of pictures (Groups of Pictures, GOP), a picture, a tile, a slice, a macroblock, a coding unit (Coding Unit, CU), a block, a transform unit (Transform Unit, TU), a prediction unit (Prediction Unit, PU), and the like. The parser (320) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, and the like from the encoded video sequence.

[0243] The parser (320) may perform entropy decoding / parsing operations on the video sequence received from the buffer (315), thereby creating symbols (321).

[0244] Depending on the type of coded video picture or part of coded video picture (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors, the reconstruction of the symbol (321) may involve multiple different units. Which units are involved and how they are involved may be controlled by subgroup control information parsed by the parser (320) from the coded video sequence. For clarity, such subgroup control information flow between the parser (520) and the multiple units below is not described.

[0245] In addition to the functional blocks already mentioned, the decoder (210) may be conceptually subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, the conceptual subdivision into the functional units below is appropriate.

[0246] One unit is a sealer / inverse transform unit (351). The sealer / inverse transform unit (351) receives quantized transform coefficients as symbols (321) from the parser (320) and control information including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (351) may output a block including sample values, which may be input into an aggregator (355).

[0247] In some cases, the output samples of the scaler / inverse transform unit (351) may belong to an intra-coded block, i.e., a block that does not use predictive information from a previously reconstructed image, but may use predictive information from a previously reconstructed portion of the current image. Such predictive information may be provided by an intra-image prediction unit (352). In some cases, the intra-image prediction unit (352) uses surrounding reconstructed information of the current (partially reconstructed) image extracted from the current image buffer (358) to generate a block of the same size and shape as the block being reconstructed. In some cases, the aggregator (355) adds the prediction information generated by the intra-frame prediction unit (352) to the output sample information provided by the scaler / inverse transform unit (351) on a per-sample basis.

[0248] In other cases, the output samples of the scaler / inverse transform unit (351) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (353) may access the reference image memory (357) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (321) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (351) (in this case referred to as residual samples or residual signals) by the aggregator (355) to generate output sample information. The address within the reference image memory (357) at which the motion compensated prediction unit (353) obtains the predicted samples may be controlled by a motion vector. The motion vector is provided to the motion compensated prediction unit (353) in the form of a symbol (321), which may have, for example, X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (357) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, and the like.

[0249] The output samples of the aggregator (355) may be subjected to various loop filtering techniques in a loop filter unit (356). The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video bitstream and available to the loop filter unit (356) as symbols (321) from the parser (320), however, the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of a coded image or coded video sequence, and to previously reconstructed and loop filtered sample values.

[0250] The output of the loop filter unit (356) may be a sample stream that may be output to a rendering device such as a display (212) and stored in a reference picture memory (357) for subsequent inter-picture prediction.

[0251] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. Once a coded picture is fully reconstructed, and the coded picture is identified (e.g., by the parser (320)) as a reference picture, the current reference picture may become part of the reference picture memory (357), and a new current picture memory may be reallocated before starting reconstruction of a subsequent coded picture.

[0252] The video decoder (210) may perform decoding operations according to a predetermined video compression technique in a standard such as ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the encoded video sequence follows the syntax of the video compression technique or standard specifically specified by the video compression technology document or standard and its configuration file. In order to conform to some video compression techniques or standards, the complexity of the encoded video sequence is within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, mega samples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.

[0253] In an embodiment, a receiver (310) may receive encoded video and additional (redundant) data. The additional data may be included as part of the encoded video sequence. The video decoder (210) may use the additional data to correctly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0254] Figure 4 An example block diagram of the operation of a video encoder (203) associated with a video source (201) according to an embodiment of the present disclosure is shown.

[0255] The video encoder (203) may include, for example, an encoder as a source encoder (430), a coding engine (432), a (local) decoder (433), a reference image memory (434), a predictor (435), a transmitter (440), an entropy encoder (445), a controller (450), and a channel (460).

[0256] The encoder (203) may receive video samples from a video source (201) (not part of the encoder), which may capture video images to be encoded by the encoder (203).

[0257] The video source (201) may provide a source video sequence to be encoded by the encoder (203) in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (201) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (203) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual images that impart motion when viewed sequentially. The image itself may be organized into a spatial array of pixels, where each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0258] According to an embodiment, the encoder (203) may encode and compress the images of the source video sequence into an encoded video sequence (443) in real time or under any other time constraints required by the application. Implementing the appropriate encoding speed is a function of the controller (450). In some embodiments, the controller (450) controls other functional units as described below and is functionally coupled to these functional units. For clarity, the coupling is not shown in the figure. The parameters set by the controller (450) may include rate control related parameters (image skipping, quantizer, lambda value of rate distortion optimization technology, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can easily recognize other functions of the controller (450) that are related to the video encoder (203) optimized for a specific system design.

[0259] As will be readily appreciated by the skilled person, some video encoders operate in a "coding loop". As a simplified description, the coding loop may include the encoding portion of a source encoder (430) (for creating symbols based on an input image to be encoded and a reference image) and a (local) decoder (433) embedded in the encoder (203), wherein the encoder (203) reconstructs the symbols to create sample data, and in some video compression techniques, when the compression between the symbols and the encoded video code stream is lossless, the (remote) decoder will also create sample data. The reconstructed sample stream is input to a reference image memory (434). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference image memory are also bit-accurate between the local encoder and the remote encoder. In other words, the prediction portion of the encoder "sees" as reference image samples having exactly the same sample values ​​as the decoder will "see" when using prediction during decoding. This basic principle of reference image synchronization (and the drift that results when synchronization cannot be maintained, for example due to channel errors) is known to those skilled in the art.

[0260] The operation of the "local" decoder (433) may be combined with, for example, Figure 3 The operation of the "remote" decoder (210) described in detail is the same. However, since the symbols are available and the entropy encoder (445) and parser (320) are able to losslessly encode / decode the symbols into the encoded video sequence, the entropy decoding portion of the decoder (210), including the channel (312), receiver (310), buffer (315) and parser (320), may not be fully implemented in the local decoder (433).

[0261] It can be observed at this point that, except for the parsing / entropy decoding present in the decoder, any decoder technology needs to be present in the corresponding encoder in substantially the same form of operation. For this reason, the disclosed subject matter focuses on the decoder operation. The description of the encoder technology can be simplified because the encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is only required in certain areas and is provided below.

[0262] As part of the encoder operation, the source encoder (430) may perform motion compensated predictive coding that predictively encodes an input frame with reference to one or more previously encoded frames in the video sequence designated as "reference frames." In this manner, the encoding engine (432) encodes the differences between pixel blocks of the input frame and pixel blocks of the reference frame, which may be selected as a prediction reference for the input frame.

[0263] The local video decoder (433) may decode the encoded video data of the frame that may be designated as the reference frame based on the symbol created by the source encoder (430). The operation of the encoding engine (432) may advantageously be a lossy process. When the video decoder ( Figure 4 When the encoded video data is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (433) replicates the decoding process that may be performed by the video decoder on the reference frame and may cause the reconstructed reference frame to be stored in the reference image memory (434). In this way, the encoder (203) may locally store a copy of the reconstructed reference frame that has common content (absent transmission errors) with the reconstructed reference frame that will be obtained by the remote video decoder.

[0264] The predictor (435) may perform a prediction search for the encoding engine (432). That is, for a new frame to be encoded, the predictor (435) may search the reference image memory (434) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new image. The predictor (435) may operate on a sample-by-sample-block-by-pixel-block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (435), the input image may have prediction references extracted from multiple reference images stored in the reference image memory (434).

[0265] The controller (450) may manage encoding operations of the source encoder (430), including, for example, setting parameters and subgroup parameters for encoding video data.

[0266] The outputs of all the above functional units may be entropy encoded in an entropy encoder (445). The entropy encoder performs lossless compression on the symbols generated by the various functional units according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0267] The transmitter (440) can buffer the encoded video sequence created by the entropy encoder (445) in preparation for transmission via a communication channel (460), which can be a hardware / software link to a storage device storing the encoded video data. The transmitter (440) can combine the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown).

[0268] The controller (450) may manage the operation of the encoder (203). During encoding, the controller (450) may assign a specific coded picture type to each coded picture, but this may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned as an intra picture (I picture), a predictive picture (P picture), or a bidirectional predictive picture (B picture).

[0269] The encoder (203) and the decoder (210) can adaptively apply the inter-frame bidirectional prediction constraints of the out-of-boundary MV. The disclosed method can be applied regardless of whether the bidirectional prediction constraints of the out-of-boundary MV are based on pixels (e.g., blocks) or sub-blocks. In some embodiments, the bidirectional prediction constraints are applied using predefined out-of-boundary conditions.

[0270] In some embodiments, the encoder (203) and the decoder (210) may be based on Fig.17 and Fig. 22 The bidirectional prediction constraint of the out-of-boundary MV is adaptively applied according to the temporal direction of the reference images (801) and (802) in the bidirectional prediction. When the two reference images (801) and (802) used for bidirectional prediction are from the same temporal direction, the bidirectional prediction constraint may not be applied regardless of the reference pixel position, and conventional bidirectional prediction may be applied; otherwise, the bidirectional prediction mode may be changed to a unidirectional prediction mode according to the out-of-boundary condition.

[0271] In some embodiments, when the MVs of reference list L0 and reference list L1 both point to the same reference image, the bidirectional prediction constraint of the out-of-boundary MV is not applied; otherwise, the bidirectional prediction mode can be changed to the unidirectional prediction mode according to the out-of-boundary condition.

[0272] In some embodiments, when the MVs of reference list L0 and reference list L1 both point to the same reference image, and the difference between the MV of L0 and the MV of L1 is within a threshold, the bidirectional prediction constraint for out-of-boundary MVs may not be applied; otherwise, the bidirectional prediction may be changed to unidirectional prediction based on out-of-boundary conditions.

[0273] In some embodiments, a threshold may be set to N luma samples, and when the difference between the horizontal components of the MV of L0 (denoted as MV0) and the MV of L1 (denoted as MV1) is less than or equal to the threshold, and / or the difference between the vertical components of MV0 and MV1 is less than or equal to the threshold, the bi-directional prediction constraint may not be applied. This condition may be represented by the following equation:

[0274]

[0275] Among them, MV0 and MV1 are motion vectors of reference lists L0 and L1 respectively, and x and y represent the horizontal component and vertical component of the motion vector respectively.

[0276] In an example, the threshold N is 1 / 4. In another example, the threshold N is 1 / 2. In another example, the threshold N is 1.

[0277] Embodiments of the present disclosure also relate to the application of adaptive bidirectional prediction constraints. In some embodiments, the disclosed adaptive bidirectional prediction constraints may be applied by default without signaling.

[0278] In some embodiments, the disclosed adaptive bi-directional prediction constraints may be applied according to a high-level syntax that may be signaled at a high level in the codestream (eg, sequence level, picture level, slice level, block level, block group level, etc.).

[0279] In some embodiments, the selection of inter bi-prediction constraints for out-of-boundary MVs may be performed at both the pixel (eg, block) level and the sub-block level.

[0280] In some embodiments, the selection of pixel-level and sub-block-level bi-prediction constraints can be signaled in the codestream at a high level (e.g., sequence level, picture level, slice level, block level, block group level, etc.). In some embodiments, the selection of pixel-level and sub-block-level bi-prediction constraints can be done adaptively at the block level. In some examples, the adaptive selection can be based on a distance threshold between a reference sample position and a picture boundary.

[0281] In some embodiments, the adaptive selection may be based on a threshold of the difference between the MV of reference list L0 and the MV of reference list L1. In some embodiments, the adaptive selection may be signaled at the block level. In some embodiments, block level signaling of the adaptive selection is allowed only when the block is at a picture (or slice, or block or sub-picture) boundary.

[0282] In some embodiments, the bi-prediction constraints for MVs outside of boundaries may be applied to boundaries outside of image boundaries. In some embodiments, the bi-prediction constraints may be applied to MVs outside of sub-image boundaries. In some embodiments, the bi-prediction constraints may be applied to MVs outside of virtual boundaries. In some embodiments, the bi-prediction constraints may be applied to MVs outside of slice boundaries. In some embodiments, the bi-prediction constraints may be applied to MVs outside of tile or tile group boundaries.

[0283] Fig.23A flow chart of an embodiment of a process for determining and applying inter-frame bidirectional prediction constraints is shown. The process may begin at operation S901, in which an encoded video code stream is received. The code stream may include a current image and a first image and a second reference image associated with the current image. The current image may include at least one block encoded according to an inter-frame bidirectional prediction mode, wherein the at least one block includes a first motion vector pointing to the first reference image and a second motion vector pointing to the second reference image.

[0284] The process proceeds to operation S902, in which it is determined whether an out-of-boundary condition associated with the first motion vector and the second motion vector is satisfied. The process proceeds to operation S903, in which it is determined whether a disabling condition for disabling the out-of-boundary condition is satisfied. The disabling condition may correspond to disabling a constraint on the bidirectional prediction mode when one of the first vector and the second motion vector points to a position beyond an image boundary on the corresponding reference image.

[0285] The process proceeds to operation S904, in which, if the out-of-boundary and disabled conditions are satisfied, the coding mode of the at least one block is changed from the inter-frame bidirectional prediction mode to another coding mode (e.g., unidirectional prediction mode), wherein the at least one block is decoded according to the other coding mode. The process proceeds to operation S905, in which, if the out-of-boundary condition is not satisfied or the disabled condition is satisfied, the at least one block is decoded according to the inter-frame bidirectional prediction mode.

[0286] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0287] It should be understood that the specific order or hierarchy of blocks in the process / flowchart disclosed herein is a diagram of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of blocks in the process / flowchart can be rearranged. In addition, some blocks can be combined or omitted. The attached method claims present the elements of various blocks in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

[0288] Some embodiments may involve systems, methods, and / or computer-readable media at any possible level of integrated technical detail. In addition, one or more of the above components may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or medium) having computer-readable program instructions thereon for causing a processor to perform operations.

[0289] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer floppy disk, a hard disk, a random access memory (Random Access Memory, RAM), a read-only memory (Read-Only Memory, ROM), an erasable programmable read-only memory (erasable Programmable Read-Only Memory, EPROM or flash memory), a static random access memory (Static Random Access Memory, SRAM), a portable compact disc read-only memory (Portable Compact Disc Read-Only Memory, CD-ROM), a digital versatile disk (Digital Versatile Disk, DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. Computer-readable storage media as used herein should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0290] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0291] The computer readable program code / instructions for performing the operations may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as "C" programming language or similar programming languages). The computer readable program instructions may be executed entirely on the user's computer as a stand-alone software package, or partially on the user's computer, or partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuits so as to perform aspects or operations.

[0292] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed via the processor of the computer or other programmable data processing device create a means for implementing the operations specified in the flowchart and / or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium, which may direct a computer, a programmable data processing device, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture, which includes instructions for implementing aspects of the operations specified in the flowchart and / or block diagram block or blocks.

[0293] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device to cause a series of operations to be performed on the computer, other programmable device, or other device, thereby producing a computer-implemented process, such that the instructions executed on the computer, other programmable device, or other device implement the operations specified in the flowchart and / or block diagram block or blocks.

[0294] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer-readable media according to various embodiments. In this regard, each block in the flowchart or block diagram may represent a module, segment or part of an instruction, which includes one or more executable instructions for implementing a specified logical operation. The method, computer system and computer-readable medium may include additional blocks, fewer blocks, different blocks or blocks of different arrangements relative to the blocks shown in the figure. In some alternative embodiments, the operations marked in the blocks may not occur in the order marked in the accompanying drawings. For example, two blocks shown in succession may actually be executed simultaneously or substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved in the blocks. It will also be noted that each block illustrated in the block diagram and / or flowchart and the combination of blocks in the block diagram and / or flowchart can be implemented by a system based on special-purpose hardware, which performs a specified operation or performs a combination of special-purpose hardware and computer instructions.

[0295] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods based on the descriptions herein.

[0296] The above disclosure also includes the following embodiments:

[0297] (1) A method for inter-frame prediction, executed by at least one processor, the method comprising: receiving an encoded video code stream, the encoded video code stream comprising a current image and a first reference image and a second reference image associated with the current image, the current image comprising at least one block encoded according to an inter-frame bidirectional prediction mode, the at least one block comprising a first motion vector pointing to the first reference image and a second motion vector pointing to the second reference image; determining whether an out-of-boundary condition associated with the first motion vector and the second motion vector is satisfied; in response to determining that the out-of-boundary condition is satisfied, determining whether a disabling condition for disabling the out-of-boundary condition is satisfied; in response to determining that the out-of-boundary condition is satisfied and the disabling condition is not satisfied, performing the following operations:

[0298] (i) changing the coding mode of at least one block from an inter-frame bidirectional prediction mode to another coding mode; and

[0299] (ii) decoding at least one block according to another coding mode; and

[0300] In response to determining that the out-of-bounds condition is not satisfied or the disable condition is satisfied, at least one block is decoded according to the inter bi-directional prediction mode.

[0301] (2) A method according to feature (1), wherein the out-of-bounds condition is satisfied in response to determining that: (i) a first motion vector points to a position in a first reference image that is outside a first image boundary of the first reference image and is at a distance from the first image boundary that is greater than a distance threshold, or (ii) a second motion vector points to a second position in a second reference image that is outside a second image boundary of the second reference image and is at a distance from the second image boundary that is greater than a distance threshold.

[0302] (3) A method according to feature (1) or (2), wherein the out-of-bounds condition is not satisfied in response to determining that: (i) the first motion vector points to a position in the first reference image, which is outside the first image boundary of the first reference image and the distance from the first image boundary is greater than a distance threshold, and (ii) the second motion vector points to a second position in the second reference image, which is outside the second image boundary of the second reference image and the distance from the second image boundary is greater than a distance threshold.

[0303] (4) A method according to any one of features (1) to (3), wherein the other encoding mode is a unidirectional prediction mode.

[0304] (5) A method according to any one of features (1) to (4), wherein the disabling condition is satisfied in response to determining that the first reference image and the second reference image are from the same temporal direction.

[0305] (6) The method of any one of features (1) to (5), wherein the disabling condition is satisfied in response to the first reference image and the second reference image being the same reference image.

[0306] (7) A method according to any one of features (1) to (6), wherein the disabling condition is satisfied in response to the first reference image and the second reference image being the same reference image and the difference between the first motion vector and the second motion vector being within a predetermined threshold.

[0307] (8) A method according to feature (7), wherein the predetermined threshold is N luminance samples and the disabling condition is satisfied in response to the following determination: (i) the difference between the horizontal component of the first motion vector and the horizontal component of the second motion vector is less than or equal to the predetermined threshold, or (ii) the difference between the vertical component of the first motion vector and the vertical component of the second motion vector is less than or equal to the predetermined threshold.

[0308] (9) A method according to any one of features (1) to (8), wherein a disabling condition is satisfied in response to determining that the first reference image and the second reference image are the same reference image and determining that the difference between the first reference position pointed to by the first motion vector and the second reference position pointed to by the second motion vector is within a predetermined threshold, wherein the predetermined threshold is N luminance samples.

[0309] (10) A method according to any one of features (1) to (9), wherein the encoded video code stream includes an indicator indicating whether the disabling condition is allowed for at least one block.

[0310] (11) An apparatus for inter-frame prediction, comprising: at least one memory configured to store computer program code; and at least one processor configured to access the computer program code and operate according to instructions of the computer program code. The computer program code includes: a receiving code configured to cause at least one processor to receive an encoded video code stream, the encoded video code stream includes a current image and a first reference image and a second reference image associated with the current image, the current image includes at least one block encoded according to an inter-frame bidirectional prediction mode, the at least one block includes a first motion vector pointing to the first reference image and a second motion vector pointing to the second reference image; a first determining code configured to cause the at least one processor to determine whether an out-of-boundary condition associated with the first motion vector and the second motion vector is satisfied; a second determining code configured to cause the at least one processor to determine whether a disabling condition for disabling the out-of-boundary condition is satisfied in response to determining that the out-of-boundary condition is satisfied; and a changing code and a decoding code, wherein, in response to determining that the out-of-boundary condition is satisfied and the disabling condition is not satisfied, (i) the changing code causes the at least one processor to change the encoding mode of the at least one block from the inter-frame bidirectional prediction mode to another encoding mode, and (ii) the decoding code causes the at least one processor to decode the at least one block according to the other encoding mode, and wherein, in response to determining that the out-of-boundary condition is not satisfied or the disabling condition is satisfied, the decoding code causes the at least one processor to decode the at least one block according to the inter-frame bidirectional prediction mode.

[0311] (12) An apparatus according to feature (11), wherein the out-of-bounds condition is satisfied in response to determining that: (i) a first motion vector points to a position in a first reference image that is outside a first image boundary of the first reference image and is at a distance from the first image boundary that is greater than a distance threshold, or (ii) a second motion vector points to a second position in a second reference image that is outside a second image boundary of the second reference image and is at a distance from the second image boundary that is greater than a distance threshold.

[0312] (13) An apparatus according to feature (11) or (12), wherein the out-of-bounds condition is not satisfied in response to determining that: (i) the first motion vector points to a position in the first reference image, which is outside the first image boundary of the first reference image and the distance from the first image boundary is greater than a distance threshold, and (ii) the second motion vector points to a second position in the second reference image, which is outside the second image boundary of the second reference image and the distance from the second image boundary is greater than a distance threshold.

[0313] (14) An apparatus according to any one of features (11) to (13), wherein the other encoding mode is a unidirectional prediction mode.

[0314] (15) An apparatus according to any one of features (11) to (14), wherein the disabling condition is satisfied in response to determining that the first reference image and the second reference image are from the same temporal direction.

[0315] (16) An apparatus according to any one of features (11) to (15), wherein the disabling condition is satisfied in response to determining that the first reference image and the second reference image are the same reference image.

[0316] (17) An apparatus according to any one of features (11) to (16), wherein the disabling condition is satisfied in response to determining that the first reference image and the second reference image are the same reference image and the difference between the first motion vector and the second motion vector is within a predetermined threshold.

[0317] (18) An apparatus according to feature (17), wherein the predetermined threshold is N luminance samples and the disabling condition is satisfied in response to a determination that: (i) the difference between the horizontal component of the first motion vector and the horizontal component of the second motion vector is less than or equal to the predetermined threshold, or (ii) the difference between the vertical component of the first motion vector and the vertical component of the second motion vector is less than or equal to the predetermined threshold.

[0318] (19) An apparatus according to any one of features (11) to (18), wherein a disabling condition is satisfied in response to determining that the first reference image and the second reference image are the same reference image, and determining that the difference between the first reference position pointed to by the first motion vector and the second reference position pointed to by the second motion vector is within a predetermined threshold, wherein the predetermined threshold is N luminance samples.

[0319] (21) A non-transitory computer-readable medium having instructions stored therein, which, when executed by a processor, cause the processor to perform a method for inter-frame prediction, the method comprising: receiving an encoded video stream, the encoded video stream comprising a current image and a first reference image and a second reference image associated with the current image, the current image comprising at least one block encoded according to an inter-frame bidirectional prediction mode, the at least one block comprising a first motion vector pointing to the first reference image and a second motion vector pointing to the second reference image; determining whether an out-of-boundary condition associated with the first motion vector and the second motion vector is satisfied; in response to determining that the out-of-boundary condition is satisfied, determining whether a disabling condition for disabling the out-of-boundary condition is satisfied; in response to determining that the out-of-boundary condition is satisfied and the disabling condition is not satisfied, performing the following operations: (i) changing the encoding mode of at least one block from the inter-frame bidirectional prediction mode to another encoding mode; and (ii) decoding at least one block according to the other encoding mode; and in response to determining that the out-of-boundary condition is not satisfied or the disabling condition is satisfied, decoding at least one block according to the inter-frame bidirectional prediction mode.

Claims

1. A method for inter-frame prediction, the method being executed by at least one processor and comprising: Receiving an encoded video stream, the encoded video stream comprising a current image and a first reference image and a second reference image associated with the current image, the current image comprising at least one block encoded according to an inter-frame bidirectional prediction mode, the at least one block comprising a first motion vector pointing to the first reference image and a second motion vector pointing to the second reference image; determining whether an out-of-boundary condition associated with the first motion vector and the second motion vector is satisfied; In response to determining that the out-of-boundary condition is satisfied, determining whether a disabling condition for disabling the out-of-boundary condition is satisfied; In response to determining that the out-of-bounds condition is satisfied and the disabling condition is not satisfied, performing the following operations: (i) changing the coding mode of the at least one block from the inter-frame bidirectional prediction mode to another coding mode; and (ii) decoding the at least one block according to the another coding mode; as well as In response to determining that the out-of-bounds condition is not satisfied or the disabling condition is satisfied, decoding the at least one block according to the inter bi-directional prediction mode.

2. The method according to claim 1, wherein: The out-of-bounds condition is satisfied in response to a determination that: (i) the first motion vector points to a position in the first reference image, the position being outside a first image boundary of the first reference image and at a distance from the first image boundary greater than a distance threshold, or (ii) the second motion vector points to a second position in the second reference image, the second position being outside a second image boundary of the second reference image and at a distance from the second image boundary greater than the distance threshold.

3. The method according to claim 1, wherein: The out-of-bounds condition is not satisfied in response to a determination that: (i) the first motion vector points to a location in the first reference image, the location being outside a first image boundary of the first reference image and at a distance from the first image boundary greater than a distance threshold, and (ii) the second motion vector points to a second location in the second reference image, the second location being outside a second image boundary of the second reference image and at a distance from the second image boundary greater than the distance threshold.

4. The method according to claim 1, wherein: The other coding mode is a unidirectional prediction mode.

5. The method according to claim 1, wherein: In response to determining that the first reference image and the second reference image are from the same temporal direction, the disabling condition is met.

6. The method according to claim 1, wherein: In response to determining that the first reference image and the second reference image are the same reference image, the disabling condition is satisfied.

7. The method according to claim 1, wherein: The disabling condition is satisfied in response to determining that the first reference image and the second reference image are the same reference image and a difference between the first motion vector and the second motion vector is within a predetermined threshold.

8. The method according to claim 7, wherein: The predetermined threshold is N luma samples, and the disabling condition is satisfied in response to a determination that: (i) a difference between a horizontal component of the first motion vector and a horizontal component of the second motion vector is less than or equal to the predetermined threshold, or (ii) a difference between a vertical component of the first motion vector and a vertical component of the second motion vector is less than or equal to the predetermined threshold.

9. The method according to claim 1, wherein: The disabling condition is met in response to determining that the first reference image and the second reference image are the same reference image, and a difference between a first reference position pointed to by the first motion vector and a second reference position pointed to by the second motion vector is within a predetermined threshold, wherein the predetermined threshold is N brightness samples.

10. The method according to claim 1, wherein: The encoded video code stream includes an indicator indicating whether the disabling condition is allowed for the at least one block.

11. An apparatus for inter-frame prediction, comprising: at least one memory configured to store computer program code; as well as at least one processor configured to access the computer program code and operate according to the instructions of the computer program code, the computer program code comprising: receiving code configured to cause the at least one processor to receive an encoded video stream, the encoded video stream comprising a current image and a first reference image and a second reference image associated with the current image, the current image comprising at least one block encoded according to an inter-frame bidirectional prediction mode, the at least one block comprising a first motion vector pointing to the first reference image and a second motion vector pointing to the second reference image; first determination code configured to cause the at least one processor to determine whether an out-of-boundary condition associated with the first motion vector and the second motion vector is satisfied; second determination code configured to cause the at least one processor to determine whether a disabling condition for disabling the out-of-boundary condition is satisfied in response to determining that the out-of-boundary condition is satisfied; and Change code and decode code, wherein, in response to the out-of-bounds condition being satisfied and the disabling condition not being satisfied, (i) changing the code to cause the at least one processor to change the coding mode of the at least one block from the inter-frame bidirectional prediction mode to another coding mode, and (ii) decoding code causing the at least one processor to decode the at least one block according to the other coding mode; and In response to determining that the out-of-bounds condition is not satisfied or the disabling condition is satisfied, the decoding code causes at least one processor to decode the at least one block according to the inter-frame bi-directional prediction mode.

12. The device according to claim 11, wherein The out-of-bounds condition is satisfied in response to a determination that: (i) a first motion vector points to a position in the first reference image, the position being outside a first image boundary of the first reference image and at a distance from the first image boundary greater than a distance threshold, or (ii) the second motion vector points to a second position in the second reference image, the second position being outside a second image boundary of the second reference image and at a distance from the second image boundary greater than the distance threshold.

13. The apparatus of claim 11 , wherein the out-of-bounds condition is not satisfied in response to a determination that: (i) the first motion vector points to a location in the first reference image that is outside a first image boundary of the first reference image and at a distance from the first image boundary that is greater than a distance threshold, and (ii) the second motion vector points to a second location in the second reference image that is outside a second image boundary of the second reference image and at a distance from the second image boundary that is greater than the distance threshold.

14. The device according to claim 11, wherein: The other coding mode is a unidirectional prediction mode.

15. The device according to claim 11, wherein In response to determining that the first reference image and the second reference image are from the same temporal direction, the disabling condition is met.

16. The device according to claim 11, wherein In response to determining that the first reference image and the second reference image are the same reference image, the disabling condition is satisfied.

17. The device according to claim 11, wherein: The disabling condition is satisfied in response to determining that the first reference image and the second reference image are the same reference image and a difference between the first motion vector and the second motion vector is within a predetermined threshold.

18. The device according to claim 17, wherein: The predetermined threshold is N luma samples, and the disabling condition is satisfied in response to a determination that: (i) a difference between a horizontal component of the first motion vector and a horizontal component of the second motion vector is less than or equal to the predetermined threshold, or (ii) a difference between a vertical component of the first motion vector and a vertical component of the second motion vector is less than or equal to the predetermined threshold.

19. The device according to claim 11, wherein: The disabling condition is met in response to determining that the first reference image and the second reference image are the same reference image, and a difference between a first reference position pointed to by the first motion vector and a second reference position pointed to by the second motion vector is within a predetermined threshold, wherein the predetermined threshold is N brightness samples.

20. A non-transitory computer readable medium having instructions stored therein, which, when executed by a processor, cause the processor to perform a method for inter-frame prediction, the method comprising: Receiving an encoded video stream, the encoded video stream comprising a current image and a first reference image and a second reference image associated with the current image, the current image comprising at least one block encoded according to an inter-frame bidirectional prediction mode, the at least one block comprising a first motion vector pointing to the first reference image and a second motion vector pointing to the second reference image; determining whether an out-of-boundary condition associated with the first motion vector and the second motion vector is satisfied; In response to determining that the out-of-boundary condition is satisfied, determining whether a disabling condition for disabling the out-of-boundary condition is satisfied; In response to determining that the outer condition of the boundary is satisfied and the disabling condition is not satisfied, performing the following operations: (i) changing the coding mode of the at least one block from the inter-frame bidirectional prediction mode to another coding mode; and (ii) decoding the at least one block according to the another coding mode; as well as In response to determining that the out-of-bounds condition is not satisfied or the disabling condition is satisfied, decoding the at least one block according to the inter bi-directional prediction mode.

Images