Methods and apparatus for video encoding or decoding, and computer programs

Abstract

This improves the motion vector derivation method for 4x4 color difference components corresponding to 4:4:4 and 4:2:2 color difference formats. [Solution] The method for encoding or decoding a video sequence is as follows: When encoding or decoding a video sequence using a 4:4:4 color difference format, an operation other than averaging is used to copy the affine motion vector of one 4x4 luminance block and associate the affine motion vector with a 4x4 color difference block at the same position. When encoding or decoding a video sequence using a 4:2:2 color difference format, each 4x4 color difference block is associated with two 4x4 luminance blocks at the same position so that the affine motion vector of one 4x4 color difference block becomes the average of the motion vectors of the two luminance blocks at the same position.

Description

[Technical Field]

[0001] [Cross-references to related applications] This application claims priority under 35 U.S.SC § 119 based on U.S. Provisional Application No. 62 / 817,517 filed with the U.S. Patent and Trademark Office on 12 March 2019, and the entirety of the disclosure herein is incorporated by reference.

[0002] [Technical field] The methods and apparatus according to the embodiments relate to video processing, and more particularly to encoding or decoding video sequences that can support different color difference formats (e.g., 4:4:4, 4:2:2) in VCC (Versatile Video Coding). [Background technology]

[0003] Recently, the Video Coding Experts Group (VCEG) of the ITU-T (ITU Telecommunication Standardization Sector), a division of the ITU (International Telecommunication Union), and the ISO / IEC MPEG (JTC 1 / SC 29 / WG 11), a standardization working group of the Joint Technical Committee ISO / IEC JTC 1 of the ISO (International Organization for Standardization) and IEC (International Electrotechnical Commission), published the H.265 / HEVC (High Efficiency Video Coding) standard (version 1) in 2013. This standard was updated to version 2 in 2014, to version 3 in 2015, and to version 4 in 2016.

[0004] Subsequently, research was conducted on the potential need for standardization of future video coding technologies with compression capabilities significantly exceeding those of the HEVC standard (including its extensions). In October 2017, a Call for Proposals (CfP) was issued for video compression with capabilities exceeding HEVC. By February 15, 2018, a total of 22 CfP responses had been submitted for standard dynamic range (SDR), 12 for high dynamic range (HDR), and 12 for 360 video categories. In April 2018, all received CfP responses were evaluated at the 122nd MPEG / 10th JVET (Joint Video Exploration Team - Joint Video Expert Team) meeting. After careful evaluation, JVET officially launched the standardization of next-generation video coding beyond HEVC, namely, the so-called VVC (Versatile Video Coding).

[0005] Here, we describe the HEVC block partitioning structure. In HEVC, a coding tree unit (CTU) may be partitioned into coding units (CUs) by using a quadtree structure, represented as a coding tree, to adapt to various local characteristics. The decision of whether or not to encode the picture region using interpicture (temporal) or intrapicture (spatial) prediction may be made at the CU level. Each CU can be further partitioned into one, two, or four prediction units (PUs) according to the PU partitioning type. Within a single PU, the same prediction process may be applied, and relevant information may be sent to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partitioning type, the CU can be partitioned into transform units (TUs) according to other quadtree structures, such as a coding tree for the CU. A characteristic of the HEVC structure is that it may include multiple partitioning concepts, including CUs, PUs, and TUs. In HEVC, CUs or TUs can generally only be square in shape, but PUs can be square or rectangular in shape for inter-prediction blocks. In HEVC, one encoded block may be further divided into four square subblocks, and the transformation may be performed on each subblock, i.e., the TU. Each TU can be recursively further divided into smaller TUs (using quadtree partitioning). This is called a Residual Quad-Tree (RQT).

[0006] At picture boundaries, HEVC uses implicit quadtree partitioning so that blocks maintain their quadtree partitioning until their size fits the picture boundary.

[0007] Here, we describe a block partitioning structure using a quad-tree (QT) + binary tree (BT). In HEVC, a CTU may be partitioned into CUs by using a quad-tree structure, which is shown as a coding tree, to adapt to various local characteristics. The decision of whether or not to encode the picture region using interpicture (temporal) or intrapicture (spatial) prediction may be made at the CU level. Each CU can be further partitioned into one, two, or four PUs according to the PU partitioning type. Within a single PU, the same prediction process may be applied, and relevant information may be sent to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partitioning type, the CU can be partitioned into transform units (TUs) according to other quad-tree structures, such as a coding tree for the CU. A feature of the HEVC structure is that it may include multiple partitioning concepts, including CUs, PUs, and TUs.

[0008] The QTBT structure eliminates the concept of multiple partition types, i.e., the separation of the concepts of CU, PU, ​​and TU, and supports greater flexibility in the shape of CU partitions. In the QTBT block structure, CU can have either a square or rectangular shape. As shown in Figure 1A, a coding tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes may then be further partitioned by a binary tree structure. There can be two types of binary tree partitions: symmetric horizontal partitioning and symmetric vertical partitioning. The binary tree leaf nodes are called coding units (CU), and their segmentation may be used for prediction and transformation processing without further partitioning. This means that CU, PU, ​​and TU have the same block size in the QTBT coding block structure. In JEM, a CU may, in some cases, consist of coding blocks (CB) of different color components. For example, one CU may contain one luminance CB and two color difference CBs in the case of P slices and B slices of a 4:2:0 color difference format, or it may contain a single-component CB. For example, a single CU may contain only one luminance CB or only two chrominance CBs in the case of an I-slice.

[0009] The following parameters are defined for the QTBT partitioning method. CTU size: The root node size of a quadtree, a concept similar to that of HEVC. MinQTSize: The minimum acceptable size of a quadtree leaf node. MaxBTSize: The maximum allowable binary tree root node size MaxBTDepth: The maximum allowed depth of a binary tree. MinBTSize: The minimum acceptable size of a binary tree leaf node.

[0010] In one example of a QTBT partitioning structure, the CTU size may be set as a 128x128 luminance sample with two corresponding 64x64 block color difference samples, MinQTSize may be set as 16x16, MaxBTSize may be set as 64x64, MinBTSize (both width and height) may be set as 4x4, and MaxBTDepth may be set as 4. A quadtree partition may first be applied to the CTU to generate a quadtree leaf node. The quadtree leaf node may have a size ranging from 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If the leaf quadtree node is 128x128, it is not further partitioned by a binary tree because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, the leaf quadtree node may be further partitioned by a binary tree. Therefore, a quadtree leaf node is also the root node of a binary tree and may have a binary tree depth of 0. If the binary tree depth reaches MaxBTDepth (i.e., 4), further partitioning is not considered. If a binary tree node has a width equal to MinBTSize (i.e., 4), further horizontal partitioning is not considered. Similarly, if a binary tree node has a height equal to MinBTSize, further vertical partitioning is not considered. Binary tree leaf nodes may be further processed by prediction and transformation processes without further partitioning. In JEM, the maximum CTU size may be 256 × 256 luminance samples.

[0011] Figure 1A shows an example of block partitioning using QTBT, and Figure 1B shows the corresponding tree representation. Solid lines represent quadtree partitioning, and dotted lines represent binary tree partitioning. At each partition (i.e., non-leaf) node of a binary tree, a flag may be propagated to indicate which partition type (i.e., horizontal or vertical) can be used, where 0 indicates horizontal partitioning and 1 indicates vertical partitioning. In quadtree partitioning, there is no need to indicate the partition type because the quadtree partition divides the block both horizontally and vertically to produce four subblocks of equal size.

[0012] Furthermore, the QTBT method supports the flexibility of having separate QTBT structures for luminance and chrominance. Currently, for P and B slices, the luminance and chrominance CTBs within a single CTU share the same QTBT structure. However, for I slices, the luminance CTB may be divided into CUs by a QTBT structure, and the chrominance CTB may be divided into chrominance CUs by another QTBT structure, namely a DualTree (DT) structure. This means that the CUs within I slices consist of an encoding block for the luminance component or an encoding block for two chrominance components, and the CUs within P or B slices consist of encoding blocks for all three chrominance components.

[0013] In HEVC, inter prediction may be restricted for small blocks to reduce motion-compensated memory access, bidirectional prediction is not supported for 4x8 and 8x4 blocks, and inter prediction is not supported for 4x4 blocks. In the QTBT scheme implemented in JEM-7.0, these restrictions may be removed.

[0014] Here, we will explain block partitioning using a ternary tree (TT). A multi-type-tree (MTT) structure has been proposed. MTT is a more flexible tree structure than QTBT. In MTT, as shown in Figures 2A and 2B, in addition to quadtrees and binary trees, horizontal and vertical central ternary trees are introduced.

[0015] Some of the advantages of ternary tree partitioning include: providing a supplement to quadtree and binary tree partitioning; capturing objects located at block centers, while quadtrees and binary trees can be partitioned at block centers; the width and height of a ternary tree partition can be powers of 2, so no further transformations are needed; and the design of two-level trees is primarily motivated by complexity reduction. Theoretically, the complexity of traversing a tree is T^D, where T is the number of partition types and D is the depth of the tree.

[0016] Here, we will explain the YUV format. Different YUV formats, or color difference formats, are shown in Figure 3. Different color difference formats define different downsampling grids for different color components.

[0017] Here, we will explain cross-component linear modeling (CCLM). In VTM, for the chromatic difference component of the intra-PU, the encoder selects the best chromatic difference prediction mode from eight modes, including planar, DC, horizontal, vertical, direct copy of intra-prediction mode from luminance components (DM), left and top cross-component linear mode (LT_CCLM), left cross-component linear mode (L_CCLM), and top cross-component linear mode (T_CCLM). Of these modes, LT_CCLM, L_CCLM, and T_CCLM can be classified as cross-component linear modes (CCLM). The difference between these three modes is that different regions of adjacent samples may be used to derive parameters α and β. In LT_CCLM, both the left and top adjacent samples may be used to derive parameters α and β. In L_CCLM, only the left-side neighboring sample is used to derive parameters α and β. In T_CCLM, only the upper-side neighboring sample is used to derive parameters α and β.

[0018] The cross-component linear model (CCLM) prediction mode may be used to reduce redundancy of chrominance components, and chrominance samples may be predicted based on restored luminance samples of the same CU by using the following linear model. Nod c (i,j)=αrec L’(i,j) + β (Equation 1)

[0019] Here, pred c (i,j) represents the predicted color difference sample within the CU, and rec L ’(i,j) represents the downsampled and restored luminance sample of the same CU. The parameters α and β may be derived by a linear equation, for example, the max-min method. This calculation process may be executed not only as an encoder search operation but also as part of the decoding process, and thus, no syntax is used to transmit the values of α and β.

[0020] For the color difference 4:2:0 format, for CCLM prediction, as shown in Figure 3, a 6-tap interpolation filter is applied to obtain the downsampled luminance samples corresponding to the color difference samples. Here, the downsampled luminance sample Rec’L[x,y] may be calculated from the restored luminance samples.

[0021] The downsampled luminance samples may be used to find the maximum and minimum sample points. Two points (the combination of luminance and color difference) (A,B) may be the minimum and maximum values within the set of adjacent luminance samples, as shown in Figure 4. Here, the linear model parameters α and β may be obtained according to the following equation.

Number

[0022] Here, division may be avoided and replaced by multiplication and shift. A look-up table (LUT) may be used to store pre-calculated values, and the absolute difference value between the maximum luminance sample and the minimum luminance sample may be used to specify the entry index of the LUT, and the size of the LUT may be 512.

[0023] In T_CCLM mode, only the adjacent samples shown in Figures 5A and 5B (including the 2*W sample) may be used to calculate the linear model coefficients.

[0024] In L_CCLM mode, as shown in Figures 6A and 6B, only the left-side adjacent samples (including the 2*H samples) may be used to calculate the linear model coefficients.

[0025] Furthermore, the CCLM prediction mode includes predictions between two chrominance components; that is, the Cr component may be predicted from the Cb component. Instead of using the reconstructed sample signal, the CCLM Cb-to-Cr prediction may be applied to the residual region. This may be done by adding the reconstructed Cb residual, weighted to the original Cr intra-prediction, to form the final Cr prediction. Nod cr *(i,j)=pred cr (i,j)+αresi cb '(i,j) (Formula 4)

[0026] The CCLM luma-to-chroma prediction mode may be added as one additional intra-color difference prediction mode. On the encoder side, one additional RD cost check may be added for the color difference component to select the intra-color difference prediction mode. If a cb intra-prediction mode other than the CCLM luma-to-chroma prediction mode can be used for the color difference component of CU, the CCLM cb-to-cr prediction may be used for predicting the Cr component.

[0027] Multiple Model CCLM (MMLM) is another extension of CCLM. As the name suggests, MMLM may have more than one model; for example, two models may be used. In MMLM, adjacent luminance samples and adjacent chrominance samples of the current block may be classified into two groups, each group of which may be used as a training set for deriving a linear model (i.e., specific α and β may be derived for a specific group). Furthermore, the samples of the current luminance block may be classified based on the same rules used for classifying adjacent luminance samples.

[0028] Figure 8 shows an example of classifying adjacent samples into two groups. The threshold may be calculated as the average value of adjacent reconstructed luminance samples. Adjacent samples with Rec'L[x,y] <= Threshold may be classified into group 1, and adjacent samples with Rec'L[x,y] > Threshold may be classified into group 2.

number

[0029] Here, we will discuss affine motion compensation prediction. In HEVC, a translational motion model may be applied to motion compensation prediction (MCP). However, many types of motion can exist, such as zoom in / out, rotation, strabismus, and other irregular motions. In VTM4, block-based affine transformation motion compensation prediction may be applied. As shown in Figure 9, the affine motion field of a block may be described by motion information of two control point motion vectors (4 parameters) or three control point motion vectors (6 parameters).

[0030] For a 4-parameter affine motion model, the motion vector at the sample position (x,y) within the block may be derived as follows.

Number

[0031] For the 6-parameter affine motion model, the motion vector at the sample position (x, y) within the block may be derived as follows.

Number

[0032] Here, mv 0x , mv 0y is the motion vector of the top-left control point, and mv 1x , mv 1y is the motion vector of the top-right control point, and mv 2x , mv 2y is the motion vector of the bottom-left control point.

[0033] To simplify motion compensation prediction, block-based affine transformation prediction may be applied. To derive the motion vector for each 4×4 luminance sub-block, as shown in FIG. 10, the motion vector of the central sample of each sub-block may be calculated according to the above formula and rounded to 1 / 16 fractional precision. Then, a motion compensation interpolation filter may be applied to generate a prediction for each sub-block having the derived motion vector. The sub-block size of the chrominance components may also be set to 4×4. The MV of the 4×4 chrominance sub-block may be calculated as the average of the MVs of the four corresponding 4×4 luminance sub-blocks. There may be two affine motion inter-prediction modes, the affine merge mode and the affine AMVP mode, as performed for translational motion inter-prediction.

[0034] Here, we will describe affine merge prediction. The AF_MERGE mode may be applied to CUs with a width and height of 8 or more. In this mode, the CPMV of the current CU may be generated based on the motion information of spatially adjacent CUs. There may be up to 5 CPMVP candidates, and one index may be propagated to indicate which one is used for the current CU. The following three types of CPVM candidates may be used to form the affine merge candidate list. (1) Inherited affine merge candidate extrapolated from the CPMV of the adjacent CU (2) Constructed affine merge candidate CPMP (Constructed Affine Merge Candidate) that can be derived using the translational MV of adjacent CUs. (3) Zero's music video.

[0035] In VTM4, there may be up to two inherited affine candidates, one from the left-side adjacent CU and one from the upper-side adjacent CU, which may be derived from the affine motion model of the adjacent block. The candidate blocks are shown in Figure 11. For the left-side predictor, the scan order may be A0->A1, and for the upper-side predictor, the scan order may be B0->B1->B2. The first inherited candidate may be selected from each edge. Pruning checks do not need to be performed between the two inherited candidates. If an adjacent affine CU is identified, its control point motion vector may be used to derive the CPMVP candidate in the affine merge list of the current CU. As shown in Figure 12, if the adjacent lower-left block A is encoded in affine mode, motion vectors v2, v3, and v4 for the upper-left, upper-right, and lower-left corners of the CU containing block A may be obtained. If block A is encoded with a 4-parameter affine model, the two CPMVs of the current CU may be calculated according to v2 and v3. If block A can be encoded with a 6-parameter affine model, the three CPMVs of the current CU may be calculated according to v2, v3, and v4.

[0036] The construction of an affine candidate means that the candidate may be constructed by combining the adjacent translational motion information of each control point. The motion information of the control points may be derived from the specified spatial and temporal adjacencies shown in Figure 13. CPMVk(k=1,2,3,4) represents the kth control point. For CPMV1, the block B2->B3->A2 may be examined, and the MV of the first available block may be used. For CPMV2, the block B1->B0 may be examined, and for CPMV3, the block A1->A0 may be examined. If TMVP is available, it is used as CPMV4.

[0037] After the motion MVs of the four control points are obtained, affine merge candidates may be constructed based on their motion information. The control point MV combinations {CPMV1,CPMV2,CPMV3}, {CPMV1,CPMV2,CPMV4}, {CPMV1,CPMV3,CPMV4}, {CPMV2,CPMV3,CPMV4}, {CPMV1,CPMV2}, and {CPMV1,CPMV3} may be used in order to construct.

[0038] A combination of three CPMV constructs a 6-parameter affine merge candidate, and a combination of two CPMV constructs a 4-parameter affine merge candidate. To avoid the motion scaling process, if the reference indices of the control points are different, the associated combinations of control point MVs may be discarded.

[0039] If the list is not yet complete after the inheritance affine merge candidates and construction affine merge candidates have been examined, a zero MV may be inserted at the end of the list.

[0040] Here, we describe affine AMVP prediction. The affine AMVP mode can be applied to CUs having both width and height of 16 or more. A CU-level affine flag may be transmitted in a bitstream to indicate whether the affine AMVP mode should be used, and other flags may be transmitted to indicate whether a 4-parameter affine or a 6-parameter affine should be used. In this mode, the difference between the current CU's CPMV and these predictor CPMVPs may be transmitted in a bitstream. The size of the affine AMVP candidate list may be 2, and it may be generated by sequentially using the following four types of CPVM candidates. (1) Inherited affine AMVP candidates that can be extrapolated from the CPMV of the adjacent CU (2) A candidate constructor of Affine AMVP that can be derived using the translational MV of adjacent CUs: CPMVP (3) Translational MV from adjacent CU (4) Zero's MV

[0041] The order in which inherited affine AMVP candidates are examined may be the same as the order in which inherited affine merge candidates are examined. The difference is that for AVMP candidates, affine CUs that have the same reference picture as the current block may be considered. When inserting inherited affine motion predictors into the candidate list, pruning does not need to be applied.

[0042] The constructed AMVP candidates may be derived from the specified spatial adjacencies shown in Figure 13. The same check order performed in the construction of affine merge candidates may be used. Furthermore, the reference picture indices of adjacent blocks may also be checked. The first block in the check order that has the same reference picture as the current CU and may be intercoded may be used. If the current CU is encoded in a 4-parameter affine mode and both mv0 and mv1 may be available, these may be added as one candidate in the affine AMVP list. If the current CU is encoded in a 6-parameter affine mode and all three CPMVs are available, these may be added as one candidate in the affine AMVP list. Otherwise, the constructed AMVP candidates are set as unavailable.

[0043] If the affine AMVP list candidate is still less than 2 after the inherited affine AMVP candidate and the constructed AMVP candidate have been examined, then mv0, mv1, and mv2 are added in order as translational MVs to predict all control point MVs of the current CU, if available. Finally, if the affine AMVP list is still not filled, zero MVs may be used to fill the affine AMVP list.

[0044] Here, we will describe the storage of affine motion information. In VTM4, the CPMV of an affine CU may be stored in a separate buffer. The stored CPMV may be used to generate the inherited CPMVP in affine merge mode and affine AMVP mode for the last encoded CU. The subblock MV derived from the CPMV may be used for motion compensation, merging translation MVs / MV derivation of AMVP lists, and deblocking.

[0045] To avoid picture line buffering for further CPMVs, affine motion data inheritance from a CU in an upper CTU may be treated differently from inheritance from a normal adjacent CU. If a candidate CU for affine motion data inheritance is on an upper CTU line, the lower-left and lower-right subblock MVs in the line buffer may be used for affine MVP derivation instead of the CPMV. Thus, the CPMV may be stored in a local buffer. If the candidate CU is 6-parameter affine encoded, the affine model may be reduced to a 4-parameter model. As shown in Figure 14, along the upper CTU boundary, the motion vectors of the lower-left and lower-right subblocks of the CU may be used for affine inheritance of the CU in the lower CTU.

[0046] Despite the advancements described above, in VTM-4.0, the MV of a 4x4 chrominance subblock in an affine-encoded encoding block is calculated as the average of the MVs of four corresponding 4x4 luminance subblocks. However, for chrominance 4:4:4 and 4:2:2 formats, each 4x4 chrominance subblock is associated with one or two 4x4 luminance subblocks, and the current method of MV derivation for 4x4 chrominance components has room for improvement to accommodate the 4:4:4 and 4:2:2 formats. [Overview of the project]

[0047] According to aspects of the present disclosure, a method for encoding or decoding a video sequence may include the steps of encoding or decoding a video sequence using a 4:4:4 color difference format, or encoding or decoding a video sequence using a 4:2:2 color difference format, wherein, if encoding or decoding a video sequence using a 4:4:4 color difference format, the method may further include the step of copying the affine motion vector of one 4x4 color difference block using an operation other than averaging, and if encoding or decoding a video sequence using a 4:2:2 color difference format, the method may further include the step of associating each 4x4 color difference block with two 4x4 color difference blocks at the same position so that the affine motion vector of one 4x4 color difference block is the average of the motion vectors of the two 4x4 color difference blocks at the same position.

[0048] According to aspects of the present disclosure, the method may further include, regardless of the color difference format, the steps of: dividing the current 4x4 color difference block into four 2x2 subblocks; deriving a first affine motion vector for the luminance block at the same position for the upper left 2x2 color difference subblock; deriving a second affine motion vector for the luminance block at the same position for the lower right 2x2 color difference block; and deriving an affine motion vector for the current 4x4 color difference block using the average of the first and second affine motion vectors.

[0049] According to aspects of this disclosure, the method may include the step of adjusting an interpolation filter used for motion compensation between luminance and chrominance components.

[0050] According to this aspect of the Disclosure, if a video sequence is input using a 4:2:0 color difference format, the method may further include the step of applying an 8-tap interpolation filter to the luminance and chrominance components.

[0051] According to aspects of the present disclosure, the method may further include the step of encoding components Y, Cb, and Cr as three separate trees, each of which encodes one of the components Y, Cb, and Cr.

[0052] According to this aspect of the disclosure, the encoding as three separate trees may be performed for an I-slice or an I-tile group.

[0053] According to aspects of this disclosure, the maximum permissible conversion size may be the same for different color components.

[0054] According to this aspect of the disclosure, when encoding or decoding a video sequence using a 4:2:2 color difference format, the maximum vertical size may be the same for different color components, and the maximum horizontal conversion size for color difference components may be half the maximum horizontal conversion size for luminance components.

[0055] According to aspects of this disclosure, at least one of the following may be applied to both the luminance component and the chrominance component: position-dependent predictor combination (PDPC), multiple transform selection (MTS), non-separable secondary transform (NSST), intra-sub partitioning (ISP), and multiple reference line (MRL) intra-prediction.

[0056] According to this aspect of the Disclosure, if multiple reference line (MRL) intra-prediction is applied to both luminance and chrominance components and the step of encoding or decoding a video sequence is performed using a 4:4:4 chrominance format, the method may further include the step of selecting an Nth reference for intra-prediction and using the same reference line for the chrominance components without explicit signaling; if intra-splitting (ISP) is applied to both luminance and chrominance components, the method may further include the step of applying intra-splitting (ISP) at the block level of the current block for components Y, Cb, and Cr; and if different trees are used for different color components, the method may further include the step of implicitly deriving encoding parameters for the U and V components from the Y component at the same location without signaling.

[0057] According to aspects of the present disclosure, a device for encoding or decoding a video sequence may include at least one memory configured to store program code, and at least one processor configured to read the program code and operate as instructed by the program code, the program code including a first encoding or decoding code configured to cause at least one processor to encode or decode a video sequence using a 4:4:4 color difference format, or to encode or decode a video sequence using a 4:2:2 color difference format, the first encoding or decoding code configured to cause at least one processor to encode or decode a video sequence using a 4:4:4 color difference format If so, the first encoding or decoding code may further include code configured to cause at least one processor to copy the affine motion vector of one 4x4 color difference block using an operation other than averaging, and if the first encoding or decoding code is configured to cause at least one processor to encode or decode a video sequence using a 4:2:2 color difference format, the first encoding or decoding code may further include code configured to cause at least one processor to associate each 4x4 color difference block with two 4x4 color difference blocks at the same position so that the affine motion vector of one 4x4 color difference block is the average of the motion vectors of the two 4x4 color difference blocks at the same position.

[0058] According to aspects of the present disclosure, the first encoding or decoding code may further include a code configured to cause at least one processor to divide the current 4x4 color difference block into four 2x2 subblocks, to derive a first affine motion vector for the luminance block at the same position for the upper left 2x2 color difference subblock, to derive a second affine motion vector for the luminance block at the same position for the lower right 2x2 color difference block, and to derive an affine motion vector for the current 4x4 color difference block using the average of the first and second affine motion vectors.

[0059] According to aspects of the present disclosure, the first encoding or decoding code may further include a code configured to cause at least one processor to adjust an interpolation filter used for motion compensation between luminance and chrominance components.

[0060] In this aspect of the Disclosure, if the first encoding or decoding code is configured to cause at least one processor to encode or decode a video sequence using a 4:2:2 color difference format, the first encoding or decoding code may further include a code configured to cause at least one processor to apply an 8-tap interpolation filter to the luminance and chrominance components.

[0061] According to aspects of the present disclosure, the first encoding or decoding code may further include a code configured to cause at least one processor to encode components Y, Cb, and Cr as three separate trees, each of which encodes one of the components Y, Cb, and Cr.

[0062] According to this aspect of the disclosure, the configuration for encoding as three separate trees may be configured to be performed for an I-slice or an I-tile group.

[0063] According to aspects of the present disclosure, the first encoding or decoding code may further include a code configured to cause at least one processor to allow the maximum conversion size to be the same for different color components.

[0064] In this aspect of the Disclosure, if the first encoding or decoding code is configured to cause at least one processor to encode or decode a video sequence using a 4:2:2 color difference format, the first encoding or decoding code may further include a code configured to cause at least one processor to set the maximum vertical size to be the same among different color components and the maximum horizontal conversion size for the color difference components to be half the maximum horizontal conversion size for the luminance components.

[0065] According to aspects of the present disclosure, the first encoding or decoding code may further include a code configured to apply to at least one processor at least one of the following: position-dependent predictor combination (PDPC), multiple transform selection (MTS), non-separable secondary transform (NSST), intra-sub partitioning (ISP), and multiple reference line (MRL) intra-prediction, to both the luminance component and the chrominance component.

[0066] According to aspects of the present disclosure, a non-temporary computer-readable medium storing instructions may be provided, and the instructions include one or more instructions that, when executed by one or more processors of the device, cause one or more processors to encode or decode a video sequence using a 4:4:4 color difference format, or to encode or decode a video sequence using a 4:2:2 color difference format, and when the instructions are executed by one or more processors of the device, cause one or more processors to encode or decode a video sequence using a 4:4:4 color difference format, the instructions include one or more instructions that, when executed by one or more processors of the device, cause one or more processors to encode or decode a video sequence using a 4:4:4 color difference format. If the instruction is executed by one or more processors in the device, and the instruction is executed by one or more processors, and the instruction is executed by at least one processor

[0067] The methods, devices, and non-transient computer-readable media described above are described separately, but these descriptions are not intended to imply any limitation on the scope of their use or functionality. In fact, these methods, devices, and non-transient computer-readable media may be combined in other embodiments of this disclosure. [Brief explanation of the drawing]

[0068] Further features, properties, and various advantages of the disclosed object will become clearer from the detailed description below and the attached drawings. [Figure 1A] This is a diagram of a divided coding tree unit according to one embodiment. [Figure 1B]This is a diagram of an encoding tree unit according to one embodiment. [Figure 2A] This is a diagram of an encoding tree unit according to one embodiment. [Figure 2B] This is a diagram of an encoding tree unit according to one embodiment. [Figure 3] This is a diagram of a different YUV format according to one embodiment. [Figure 4] This figure shows different luminance values ​​according to one embodiment. [Figure 5A] This is a sample diagram used in cross-component linear modeling according to one embodiment. [Figure 5B] This is a sample diagram used in cross-component linear modeling according to one embodiment. [Figure 6A] This is a sample diagram used in cross-component linear modeling according to one embodiment. [Figure 6B] This is a sample diagram used in cross-component linear modeling according to one embodiment. [Figure 7A] This is a sample diagram used in cross-component linear modeling according to one embodiment. [Figure 7B] This is a sample diagram used in cross-component linear modeling according to one embodiment. [Figure 8] This is an example of classification using a multi-model CCLM according to one embodiment. [Figure 9A] This is an example of an affine motion field of a block according to one embodiment. [Figure 9B] This is an example of an affine motion field of a block according to one embodiment. [Figure 10] This is an example of an affine motion vector field according to one embodiment. [Figure 11] This is an example of a candidate block for prediction according to one embodiment. [Figure 12] This is an example of a candidate block for prediction according to one embodiment. [Figure 13]This is an example of a candidate block for prediction according to one embodiment. [Figure 14] This is an example of using motion vectors according to one embodiment. [Figure 15] This is a simplified block diagram of a communication system according to one embodiment. [Figure 16] This is a diagram of a streaming environment according to one embodiment. [Figure 17] This is a block diagram of a video decoder according to one embodiment. [Figure 18] This is a block diagram of a video encoder according to one embodiment. [Figure 19] This is a flowchart illustrating an exemplary process for encoding or decoding a video sequence according to one embodiment. [Figure 20] This is a diagram of a computer system according to one embodiment. [Modes for carrying out the invention]

[0069] Figure 15 shows a simplified block diagram of a communication system (400) according to one embodiment of the present disclosure. The communication system (400) may include at least two terminals (410-420) interconnected via a network (450). For one-way data transmission, a first terminal (410) may encode video data at its local location for transmission to the other terminal (420) via the network (450). A second terminal (420) may receive the encoded video data from the other terminal via the network (450), decode the encoded data, and display the restored video data. One-way data transmission may be common in media delivery applications, etc.

[0070] Figure 15 shows a second pair of terminals (430, 440) provided to support the bidirectional transmission of encoded video that may occur, for example, during a video conference. For bidirectional transmission of data, each terminal (430, 440) may encode video data captured at its local location for transmission to the other terminal device over the network (450). Each terminal (430, 440) may also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the restored video data on a local display device.

[0071] In Figure 15, terminals (410-440) may be shown as servers, personal computers, and smartphones, but the principles of this disclosure are not limited to these. Embodiments of this disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (450) represents any number of networks that transmit encoded video data between terminals (410-440), including, for example, wired and / or wireless communication networks. Communication networks (450) may exchange data over circuit-switched channels and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of network (450) are not important to the operation of this disclosure unless described below herein.

[0072] Figure 16 shows an example of an application of the disclosed object, illustrating the arrangement of a video encoder and decoder in a streaming environment. The disclosed object is similarly applicable to other video-enabled applications, including, for example, video conferencing, digital TV, and the storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.).

[0073] The streaming system may include a capture subsystem (513), which may include, for example, a video source (501) (e.g., a digital camera) that generates an uncompressed video sample stream (502). The sample stream (502), which is drawn as a thick line to emphasize that it has a higher data volume compared to the encoded video bitstream, may be processed by an encoder (503) coupled to the camera (501). The encoder (503) may include hardware, software, or a combination thereof to enable or implement an aspect of the disclosed object, as will be described in more detail below. The encoded video bitstream (504), which is drawn as a thin line to emphasize that it has a lower data volume compared to the sample stream, may be stored in a streaming server (505) for future use. One or more streaming clients (506, 508) may access a streaming server (505) to obtain copies (507, 509) of the encoded video bitstream (504). A client (506) may include a video decoder (510). The video decoder (510) decodes the input copy (507) of the encoded video bitstream and generates an output video sample stream (511) that can be rendered on a display (512) or other rendering device (not shown). In some streaming systems, the video bitstreams (504, 507, 509) may be encoded according to a specific video encoding / compression standard. Examples of these standards include H.265 HEVC. Video encoding standards under development are informally known as VVC (Versatile Video Coding). The disclosures may be used in the background of VVC.

[0074] Figure 17 may also be a functional block diagram of a video decoder (510) according to one embodiment of the present disclosure.

[0075] The receiver (610) may receive one or more encoded video sequences to be decoded by the decoder (610), and in the same or other embodiments, may receive one encoded video sequence at a time, with the decoding of each encoded video sequence being independent of other encoded video sequences. The encoded video sequences may be received from a channel (612), which may be a hardware / software link to a storage device that stores encoded video data. The receiver (610) may receive the encoded video data together with other data (e.g., encoded audio data and / or auxiliary data streams), which may be transferred to their respective user entities (not shown). The receiver (610) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (615) may be coupled between the receiver (610) and the entropy decoder / parser (620) (hereinafter referred to as "parser (520)"). If the receiver (610) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isochronous network, the buffer (515) may not be necessary or can be small. For use in best-effort packet networks such as the Internet, the buffer (615) may be necessary and can be relatively large, or advantageously, adaptively sized.

[0076] The video decoder (510) may include a parser (620) for recovering symbols (621) from an entropy-encoded video sequence. These symbol categories include information used to manage the operation of the decoder (610) and potentially include information for controlling a rendering device such as a display (512). The rendering device (512), as shown in Figure 17, is not an integral part of the decoder but may be coupled to it. The rendering device control information may be in the form of Supplemental Enhancement Information (SEI messages) or Video Usability Information (VUI) parameter set fragments (not shown). The parser (620) may parse / entropy-decode the received encoded video sequence. The encoding of the encoded video sequence may follow video coding techniques or standards, and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (620) may extract a set of subgroup parameters for at least one of the pixel subgroups in the video decoder from the encoded video sequence, based on at least one parameter corresponding to a group. Subgroups may include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transformation units (TU), prediction units (PU), etc. The entropy decoder / parser may also extract information from the encoded video sequence such as transformation coefficients, quantizer parameter (QP) values, motion vectors, etc.

[0077] The parser (620) may perform an entropy decoding / analysis operation on the video sequence received from the buffer (615) to generate a symbol (621). The parser (620) may receive encoded data and selectively decode a particular symbol (621). Furthermore, the parser (620) may determine whether a particular symbol (621) should be provided to a motion compensation prediction unit (653), a scaler / inverse transform unit (651), an intra prediction unit (652), or a loop filter (656).

[0078] The reconstruction of symbol (621) may involve multiple different units, depending on the type of encoded video picture or portion thereof (e.g., interpicture and intrapicture, interblock and intrablock) and other factors. Which units are involved and how may be controlled by subgroup control information analyzed from the encoded video sequence by the parser (620). The flow of such subgroup control information between the parser (620) and the following multiple units is not illustrated for clarity.

[0079] In addition to the functional blocks described above, the decoder (510) may be conceptually subdivided into several functional units, as described below. In practical implementations operating under commercial constraints, many of these units may interact closely with each other and be at least partially integrated. However, for the purpose of illustrating the subject matter of the disclosure, it is appropriate to conceptually subdivide it into the following functional units.

[0080] The first unit is the scaler / inverse unit (651). The scaler / inverse unit (651) receives quantized transformation coefficients as symbols (621) from the parser (620), along with control information (including which transformation to use, block size, quantization coefficients, quantization scaling matrix, etc.). The scaler / inverse unit (651) may output a block containing sample values ​​that can be input to the aggregator (655).

[0081] In some cases, the output samples of the scaler / inverse transform (651) may relate to intra-encoded blocks, i.e., blocks that do not use prediction information from previously restored pictures but can use prediction information from previously restored portions of the current picture. Such prediction information may be provided by an intra-picture prediction unit (652). In some cases, the intra-picture prediction unit (652) generates blocks of the same size and shape as the block being restored using already restored surrounding information extracted from the (partially restored) current picture (656). In some cases, the aggregator (655) adds the prediction information generated by the intra-prediction unit (652) to the output sample information provided by the scaler / inverse transform unit (651) for each sample.

[0082] In other cases, the output samples of the scaler / inverse unit (651) may be associated with intercoded and potentially motion-compensated blocks. In such cases, the motion-compensated prediction unit (653) may access the reference picture memory (657) to retrieve samples to be used for prediction. After motion-compensating the retrieved samples according to the symbols (621) associated with the blocks, these samples may be added by the aggregator (655) to the output of the scaler / inverse unit (in this case, called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory from which the motion-compensated unit retrieves prediction samples, available to the motion-compensated unit, may be controlled by motion vectors, for example, in the form of symbols (621) which may have X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory when the exact motion vectors of the subsamples are used, motion vector prediction mechanisms, etc.

[0083] The output samples of the aggregator (655) may undergo various loop filtering techniques within the loop filtering unit (656). The video compression technique may include an in-loop filtering technique, which is controlled by parameters included in the encoded video bitstream and made available to the loop filtering unit (656) as symbols (621) from the parser (620), but may respond to metadata obtained during decoding of earlier portions (in decoding order) of the encoded picture or encoded video sequence, as well as to previously recovered and loop-filtered sample values.

[0084] The output of the loop filter unit (556) may also be a sample stream, which is output to the rendering device (512) and may also be stored in the reference picture memory (656) for use in future interpicture prediction.

[0085] A particular encoded picture, once fully restored, may be used as a reference picture for future predictions. For example, once an encoded picture is fully restored and identified as a reference picture (e.g., by a parser (620)), the current reference picture (656) may become part of the reference picture buffer (657), and new current picture memory may be reallocated before the restoration of subsequent encoded pictures begins.

[0086] The video decoder (510) may perform the decoding operation according to a predetermined video compression technique that may be documented in a standard, such as H.265 HEVC. The encoded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the encoded video sequence conforms to the syntax of the video compression technique or standard specified in the video compression technique documentation or standard, and in particular in the syntax specified in these profile documents. Also, for compliance purposes, the complexity of the encoded video sequence must be within the range defined by the level of the video compression technique or standard. In some cases, the level may limit the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the level may be further limited through the Hypothetical Reference Decoder (HRD) specification and metadata about HRD buffer management transmitted in the encoded video sequence.

[0087] In one embodiment, the receiver (610) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately restore the original video data. The additional data may be in the form of, for example, time, space, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0088] Figure 18 may also be a functional block diagram of a video encoder (503) according to one embodiment of the present disclosure.

[0089] The encoder (503) may receive video samples from a video source (501) (which is not part of the encoder), and the video source (501) may capture a video image to be encoded by the encoder (503).

[0090] The video source (501) may provide a source video sequence to be encoded by the encoder (503) in the form of a digital video sample stream, the digital video sample stream may be any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media delivery system, the video source (501) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (503) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that convey motion when viewed sequentially. The picture itself may be composed of a spatial array of pixels, and each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description will focus on samples.

[0091] According to one embodiment, the encoder (503) may encode and compress the pictures of a source video sequence into an encoded video sequence (743) in real time or under any other time constraints required by the application. Achieving an appropriate encoding speed is one function of the controller (750). The controller (750) controls and is functionally coupled to other functional units, as described below. The couplings are not shown for clarity. Parameters set by the controller may include rate control-related parameters (such as picture skip, quantization, lambda values ​​for rate distortion optimization techniques), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art will readily recognize other functions of the controller (750) that may be associated with a video encoder (703) optimized for a particular system design.

[0092] Some video encoders operate in a manner readily recognizable to those skilled in the art as a “coding loop.” In very simplified terms, the coding loop may consist of a coding portion of the encoder (730) (hereinafter referred to as the “source coder”) (responsible for generating symbols based on the input picture and reference picture to be encoded) and a (local) decoder (733) embedded in the encoder (503). The decoder (733) restores the symbols to generate sample data in the same manner as a (remote) decoder would generate (such that any compression between the symbols and the encoded video bitstream is reversible in the video compression techniques considered in the disclosure). The restored sample stream is input to the reference picture memory (734). Since decoding of the symbol stream yields bitwise-accurate results independent of the decoder's location (local or remote), the contents of the reference picture buffer are also bitwise-accurate between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder "sees" when using predictions during decoding. This basic principle of reference picture synchronization (including drift resulting from, for example, the inability to maintain synchronization due to channel errors) is well known to those skilled in the art.

[0093] The operation of the “local” decoder (733) may be the same as that of the “remote” decoder (510), as has already been explained in detail above in relation to Figure 16. However, referring briefly to Figure 16, since symbols are available and the encoding / decoding of symbols to the encoded video sequence by the entropy coder (745) and parser (620) may be reversible, the entropy decoding portion of the decoder (510), including the channel (612), receiver (610), buffer memory (615), and parser (620), does not have to be fully implemented in the local decoder (733).

[0094] At this point, one consideration is that any decoder technique, excluding analysis / entropy decoding, must necessarily exist in substantially the same functional form within the corresponding encoder. The description of encoder techniques is the reverse of the comprehensive description of decoder techniques and can therefore be omitted. More detailed explanations are only necessary in specific areas, which are provided below.

[0095] As part of its operation, the source coder (730) may perform motion-compensated predictive coding, which predictively codes the input frame by referencing one or more previously coded frames from a video sequence designated as “reference frames”. In this way, the coding engine (732) codes the difference between the pixel blocks of the input frame and the pixel blocks of the reference frame that may be selected as a predictive reference for the input frame.

[0096] The local video decoder (733) may decode the encoded video data of a frame that may be designated as a reference frame based on symbols generated by the source coder (730). The operation of the encoding engine (732) may, advantageously, be lossy. If the encoded video data can be decoded by a video decoder (not shown in Figure 17), the restored video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (733) may replicate the decoding process that may be performed by the video decoder on the reference frame and store the restored reference frame in a reference picture cache (734). Thus, the encoder (503) may locally store a copy of the restored reference frame having common content as a restored reference frame (without transmission errors) to be acquired by the far-end video decoder.

[0097] The predictor (735) may perform a predictive search for the coding engine (732). That is, for a new frame to be coded, the predictor (735) may search the reference picture memory (734) for sample data (as candidate reference pixel blocks) or specific metadata (reference picture motion vector, block shape, etc.). These may function as appropriate predictive references for the new picture. The predictor (735) may operate sample block-by-pixel block to find appropriate predictive references. In some cases, the input picture determined by the search results obtained by the predictor (735) may have predictive references drawn from multiple reference pictures stored in the reference picture memory (734).

[0098] The controller (750) may manage the encoding operation of the video coder (730), including, for example, setting parameters and subgroup parameters used to encode video data.

[0099] The outputs of all the above functional units may undergo entropy coding in an entropy coder (745). The entropy coder converts the symbols generated by the various functional units into coded video sequences by lossless compression of the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0100] The transmitter (740) may buffer the encoded video sequence generated by the entropy coder (745) and prepare it for transmission over a communication channel (760), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (740) may merge the encoded video data from the video coder (730) with other data to be transmitted (e.g., encoded audio data and / or auxiliary data streams (not shown)).

[0101] The controller (750) may manage the operation of the encoder (503). During encoding, the controller (750) may assign a specific encoded picture type to each encoded picture. This encoded picture type may affect the encoding technique that can be applied to each picture. For example, a picture may often be assigned as one of the following frame types:

[0102] An intra-picture (I-picture) may be one that can be encoded and decoded without using other pictures in the sequence as a source for prediction. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will recognize these variations of I-pictures and their respective uses and characteristics.

[0103] The prediction picture (P-picture) may be encoded and decoded using intra-prediction or inter-prediction, with up to one motion vector and reference index used to predict the sample values ​​of each block.

[0104] A bidirectional predictive picture (B-picture) may be encoded and decoded using intra-prediction or inter-prediction, using up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures may use two or more reference pictures and associated metadata to reconstruct a single block.

[0105] Generally, a source picture may be subdivided spatially into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively), and each block may be encoded. Blocks may be predictively encoded by referencing other (already encoded) blocks, as determined by the encoding assignment applied to each picture in the block. For example, blocks of picture I may be non-predictively encoded, or they may be predictively encoded by referencing already encoded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be non-predictively encoded via spatial prediction or temporal prediction by referencing a previously encoded reference picture. Blocks of picture B may be non-predictively encoded via spatial prediction or temporal prediction by referencing a reference picture encoded one or two steps prior.

[0106] The video coder (503) may perform encoding operations according to a predetermined video encoding technique or standard, such as H.265 HEVC. In such operations, the video coder (503) may perform various compression operations, including predictive encoding operations that take advantage of temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard being used.

[0107] In one embodiment, the transmitter (740) may transmit additional data along with the encoded video. The video coder (730) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data, such as time / space / SNR enhancement layers, redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0108] This disclosure relates to several block partitioning methods for video encoding in which motion information is taken into consideration during tree partitioning. More specifically, the techniques of this disclosure relate to tree partitioning methods for flexible tree structures based on motion field information. The techniques proposed in this disclosure are applicable to both uniformly derived motion fields and non-uniformly derived motion fields.

[0109] The derived motion field of a block is defined as uniform if the derived motion field is available to all subblocks within the block and all motion vectors within the derived motion field are similar (for example, if the motion vectors share the same reference frame and the absolute difference between the motion vectors is all less than a certain threshold). The threshold may be transmitted via a bitstream or may be predefined.

[0110] The derived motion field of a block is defined as heterogeneous if the derived motion field is available for all subblocks within the block and the motion vectors within the derived motion field are not similar (for example, if at least one motion vector refers to a reference frame not referenced by the other motion vectors, or if at least one absolute difference between two motion vectors in the field is greater than a transmitted threshold or a given threshold).

[0111] Figure 19 is a flowchart of an exemplary process (800) for encoding or decoding a video sequence. In some implementations, one or more processing blocks in Figure 19 may be performed by a decoder (510). In some implementations, one or more processing blocks in Figure 19 may be performed by other devices or groups of devices, such as an encoder (503), which may be separate from or include the decoder (510).

[0112] As shown in Figure 19, process (800) may include step (810) of encoding or decoding a video sequence using a 4:4:4 color difference format or a 4:2:2 color difference format.

[0113] If process (800) includes the step of encoding or decoding a video sequence using a 4:4:4 color difference format, process (800) may further include the step (820) of copying the affine motion vectors of one 4x4 color difference block using an operation other than averaging.

[0114] If process (800) includes the step of encoding or decoding a video sequence using a 4:2:2 color difference format, process (800) may further include the step (830) of associating each 4x4 color difference block with two 4x4 color difference blocks at the same location so that the affine motion vector of one 4x4 color difference block is the average of the motion vectors of the two 4x4 color difference blocks at the same location.

[0115] Figure 19 shows an exemplary block of process (800), but in some implementations, process (800) may contain more, fewer, different blocks, or blocks in different arrangements than those shown in Figure 19. Furthermore, or alternatively, two or more blocks of process (800) may be executed in parallel.

[0116] Furthermore, the proposed method may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-temporary computer-readable medium in order to perform one or more of the proposed methods.

[0117] The above techniques may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 20 shows a computer system (900) suitable for implementing a particular embodiment of the disclosed object.

[0118] Computer software may be encoded using any suitable machine code or computer language, which may undergo assembly, compilation, linking, or similar mechanisms to generate code containing instructions, which may be executed directly or through an interpreter, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0119] The instructions may be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game devices, and Internet of Things devices.

[0120] The components of the computer system (900) shown in Figure 20 are essentially illustrative and are not intended to imply any limitation on the scope or functionality of computer software implementing embodiments of the present disclosure. Furthermore, the configuration of the components should not be construed as having any dependency or requirement on any one or combination of components shown in the exemplary embodiments of the computer system (900).

[0121] The computer system (900) may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users, for example, through haptic input (keystrokes, swipes, data glove movements, etc.), audio input (voices, applause, etc.), visual input (gestures, etc.), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., conversations, music, ambient sounds), images (scanned images, photographic images taken from a still camera, etc.), or video (2D video, 3D video including stereoscopic pictures, etc.).

[0122] The input human interface device may include one or more of the following: keyboard (901), mouse (902), trackpad (903), touch screen (910), data glove (904), joystick (905), microphone (906), scanner (907), and camera (908).

[0123] The computer system (900) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such a human interface output device may include a haptic output device (e.g., haptic feedback via a touch screen (910), data glove (904), or joystick (905), although there may be haptic feedback devices that do not function as input devices), an audio output device (e.g., a speaker (909), headphones (not shown)), a visual output device (each having a touch screen input function or not, each having a haptic feedback function or not, and some of which may be capable of outputting three or more dimensions through means such as two-dimensional visual output or stereoscopic output, including a screen (910) including a cathode ray tube (CRT), liquid crystal display (LCD), plasma screen, or organic light-emitting diode (OLED) screen, virtual reality glasses (not shown), a holographic display, and a smoke tank (not shown)), and a printer (not shown).

[0124] The computer system (900) may also include human-accessible storage devices and related media such as optical media including CD / DVD ROM / RW (920) having a CD / DVD or similar medium (921), thumb drives (922), removable hard drives or solid-state drives (923), legacy magnetic media such as tapes and floppy disks (not shown), and special ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0125] Furthermore, those skilled in the art should understand that the term “computer-readable medium” as used herein in relation to the subject matter of this disclosure does not include a transmission medium, carrier wave, or other non-transient signal.

[0126] Furthermore, the computer system (900) may include an interface to one or more communication networks. The network may be, for example, wireless, wired, or optical. The network may be local, wide-area, metropolitan, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks include Ethernet, wireless LAN, cellular networks (including GSM (global systems for mobile communications), 3G (third generation), 4G (fourth generation), 5G (fifth generation), LTE (Long-Term Evolution), etc.), wired or wireless wide-area digital TV networks (including cable TV, satellite TV, and terrestrial TV), vehicle and industrial (including CANBus), etc. Certain networks generally require an external network interface adapter (e.g., a USB (universal serial bus) port on a computer system (900)) attached to a specific general-purpose data port or peripheral bus (949), while other network interface adapters are generally integrated into the core of the computer system (900) by being attached to the system buses described below (e.g., an Ethernet interface to a PC computer system or a cellular network to a smartphone computer system). Using any of these networks, the computer system (900) can communicate with other entities. Such communication may be one-way reception only (e.g., broadcast television), one-way transmission only (e.g., CANbus to a specific CANbus device), or bidirectional to other computer systems using local or wide-area digital networks, for example. Specific protocols and protocol stacks may be used in each of the networks and network interfaces described above.

[0127] The above-mentioned human interface devices, human-accessible memory devices, and network interfaces may be mounted on the core (940) of the computer system (900).

[0128] The core (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), special programmable processing units in the form of field programmable gate areas (FPGAs) (943), hardware accelerators for specific tasks (944), etc. These devices may be connected via a system bus (948) along with read-only memory (ROM) (945), random access memory (RAM) (946), and internal mass storage devices (internal, user-inaccessible hard drives, solid-state drives (SSDs), etc.) (947). In some computer systems, the system bus (948) may be accessible in the form of one or more physical plugs to allow for expansion with further CPUs, GPUs, etc. Peripheral devices may be directly connected to the core's system bus (948) or connected via a peripheral bus (949). The architecture of the peripheral bus includes PCI (peripheral component interconnect), USB, etc.

[0129] The CPU (941), GPU (942), FPGA (943), and accelerator (944) may execute specific instructions, and these specific instructions may, in combination, constitute the computer code described above. This computer code may be stored in ROM (945) or RAM (946). Temporary data may be stored in RAM (946), while persistent data may be stored, for example, in an internal mass storage device (947). High-speed storage and retrieval to any of the memory devices may be enabled by using a cache memory that may be closely associated with one or more CPUs (941), GPUs (942), mass storage devices (947), ROMs (945), RAM (946), etc.

[0130] A computer-readable medium may contain computer code for performing operations implemented on various computers. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be readily available to those skilled in the art in the computer software field.

[0131] As an example, and not an limitation, a computer system having an architecture (900), specifically a core (940), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media may be media related to user-accessible mass storage as described above, as well as non-transient storage of the core (940), such as a mass storage (947) or ROM (945) within the core. Software implementing various embodiments of this disclosure may be stored in such devices and executed by the core (940). The computer-readable media may include one or more memory devices or chips according to specific needs. The software may cause the core (940), specifically the processor (including a CPU, GPU, FPGA, etc.) therein, to perform specific processing or specific parts of specific processing as described herein, including defining data structures stored in RAM (946) and modifying such data structures according to processing defined by the software. Furthermore, or alternatively, a computer system may provide functionality as a result of logic wired within a circuit (e.g., an accelerator (944)) or otherwise embodied, such circuit may operate in place of or with software to perform specific processes or specific parts of specific processes as described herein. References to software include logic, and vice versa, as applicable. References to computer-readable media may include, as applicable, circuits that store software for execution (e.g., integrated circuits (ICs)), circuits that embody logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0132] While this disclosure describes several exemplary embodiments, there are many variations, substitutions, and alternative equivalents that fall within the scope of this disclosure. Therefore, those skilled in the art will recognize that numerous systems and methods, not expressly illustrated or described herein, can be devised that embody the principles of this disclosure and thus fall within the true intent and scope of this disclosure.

Claims

[Claim 1] A method by which a decoder or encoder encodes or decodes a video sequence, The steps include encoding or decoding the video sequence using one of the 4:4:4 color difference format and the 4:2:2 color difference format, When encoding or decoding the video sequence using the aforementioned 4:4:4 color difference format, the steps include: copying the affine motion vector of one 4x4 luminance block using an operation other than averaging, and associating the affine motion vector with a 4x4 color difference block at the same position; When encoding or decoding the video sequence using the aforementioned 4:2:2 color difference format, the steps include associating each 4x4 color difference block with two 4x4 luminance blocks at the same position so that the affine motion vector of one 4x4 color difference block becomes the average of the motion vectors of the two luminance blocks at the same position, and A method that includes this.

Images