Improved virtual time affine candidates

By creating and filtering virtual affine candidates in video encoding, the problem of insufficient compression efficiency in affine modeling motion compensation in existing technologies is solved, achieving more efficient video encoding and decoding.

CN112806011BActive Publication Date: 2024-11-26INTERDIGITAL VC HOLDINGS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980058909.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-09-13
Filing Date
2019-09-12
Publication Date
2024-11-26
Estimated Expiration
2039-09-12

AI Technical Summary

Technical Problem

Existing video coding technologies suffer from insufficient compression efficiency when utilizing the spatial and temporal redundancy of video content. This is especially true in motion compensation using affine modeling, where insufficient candidate list filtering and refinement leads to increased computational complexity and latency.

Method used

By creating more virtual affine candidates and filtering or refining them to maintain the size of the relevant candidate list, spatial and temporal candidates are interwoven, improving the efficiency of affine motion compensation.

Benefits of technology

It improves the compression efficiency of video encoding and decoding, reduces computational complexity and candidate list size, and enhances the effect of affine motion compensation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112806011B_ABST
    Figure CN112806011B_ABST
Patent Text Reader

Abstract

Method and apparatus for creating additional affine candidates. Virtual and temporal candidates are determined using adjacent spatial and temporal sub-blocks. Sub-blocks are checked in an order known to both the encoder and decoder. Affine models are computed using valid sub-blocks. Candidates may be conditionally filtered based on various criteria and added to a candidate list. Control point motion vectors may be determined using the candidates, and motion flow fields may be determined. Motion vectors for sub-blocks within a video coding block may be determined. Motion compensation may be performed using improved affine candidates and improved affine motion compensation based coding / decoding.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] At least one embodiment of the present invention is generally directed to a method or apparatus for video encoding or decoding, compression or decompression. Background Art

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction (including motion vector prediction), as well as transforms to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlations, and then the difference between the original image and the predicted image, which is usually represented as a prediction error or prediction residual, is transformed, quantized, and entropy encoded. To reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy encoding, quantization, transform, and prediction.

[0003] A recent addition to high compression techniques includes the use of motion models based on affine modeling. Affine modeling is used for motion compensation in encoding and decoding of video pictures. Typically, affine modeling is a model that uses at least two parameters, such as two control point motion vectors (CPMVs) representing the motion at the respective corners of a block of a picture, which allows the motion field for the entire block of the picture to be derived to simulate, for example, rotation and similarity (scaling). Summary of the invention

[0004] At least one embodiment of the present invention generally relates to a method or apparatus for video encoding or decoding, and more specifically, to a method or apparatus for selecting a predictor candidate from a predictor candidate set for motion compensation based on a motion model (e.g., an affine model) for a video encoder or a video decoder.

[0005] According to a first aspect, a method is provided. The method comprises the steps of: determining, for a subblock of a video block being encoded in a picture, a plurality of predictor candidates from neighboring subblocks and at least one reference picture; selecting a subset of the plurality of predictor candidates; determining one or more control point motion vectors for the video block based on the subset of predictor candidates; performing affine motion compensation on the subblock of the video block being encoded using the one or more control point motion vectors; and encoding the subblock of the video block.

[0006] According to a second aspect, a method is provided. The method comprises the following steps: for a subblock of a video block being decoded in a picture, determining a plurality of predictor candidates from neighboring subblocks and at least one reference picture; selecting a subset of the plurality of predictor candidates; determining one or more control point motion vectors for the video block based on the subset of predictor candidates; performing affine motion compensation on the subblock of the video block being decoded using the one or more control point motion vectors; and decoding the subblock of the video block.

[0007] According to another aspect, a device is provided. The device includes a processor. The processor can be configured to encode a block of a video or decode a bitstream by performing any of the above methods.

[0008] According to another general aspect of at least one embodiment, there is provided an apparatus comprising means according to any of the decoding embodiments; and at least one of (i) an antenna configured to receive a signal comprising a video block, (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block, or (iii) a display configured to display an output representing the video block.

[0009] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that contains data content generated according to any of the described encoding embodiments or variations.

[0010] According to another general aspect of at least one embodiment, there is provided a signal comprising video data generated according to any of the described encoding embodiments or variations.

[0011] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, there is provided a computer program product comprising instructions which, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0013] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of illustrative embodiments, which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 An example of a coding tree unit and a coding tree concept representing a compressed HEVC picture is shown.

[0015] Figure 2 An example of dividing a coding tree unit into a coding unit, a prediction unit, and a transform unit is shown.

[0016] Figure 3 An example of a simple affine model used in a joint exploration model is shown.

[0017] Figure 4 An example of an affine motion vector field based on a 4×4 sub-CU is shown.

[0018] Figure 5An example of a motion vector prediction process for an affine inter CU is shown.

[0019] Figure 6 An example flow chart of a process for creating new candidates in an affine AMVP process is shown.

[0020] Figure 7 An example of motion vector prediction candidates in affine merge mode is shown.

[0021] Figure 8 An example of the spatial derivation of affine motion field control points for affine merge mode is shown.

[0022] Fig. 9 An example of a flow chart for creating new affine candidates is shown.

[0023] Fig.10 A standard, common video compression scheme is shown.

[0024] Fig.11 A standard, common video decompression scheme is shown.

[0025] Fig.12 An example of a virtual temporal candidate sub-block is shown.

[0026] Fig.13 An example flow chart of a process for selecting a corner block is shown.

[0027] Fig.14 An example flow chart showing a process for affine model calculation.

[0028] Fig.15 An example flow diagram of a process for candidate validation is shown.

[0029] Fig.16 An example of additional motion vector data is shown.

[0030] Fig.17 An example of the location of potential motion vector data for a 48x16 block with a step size N=16 is shown.

[0031] Fig.18 An example flow chart showing additional affine model creation.

[0032] Fig.19 An example flow chart showing variations of additional model creation.

[0033] Fig. 20 Examples of motion vector data selection for corners in an iterative loop (left) and alternating iterations (right) are shown.

[0034] Fig.21 An example of control point motion vector model derivation is shown.

[0035] Fig. 22 One embodiment of a system for encoding or decoding using the general aspects described herein is shown.

[0036] Fig.23 One embodiment of a method of encoding using the general aspects described herein is shown.

[0037] Fig.24 One embodiment of a method of decoding using the general aspects described herein is shown.

[0038] Fig.25 One embodiment of an apparatus for encoding or decoding using the general aspects described herein is shown. DETAILED DESCRIPTION

[0039] The general aspects described herein are in the field of video compression. These aspects are directed to improving compression efficiency compared to existing video compression systems.

[0040] In the HEVC video compression standard (H.265 / HEVC High Efficiency Video Coding, ITU-T H.265 Telecommunication Standardization Sector of the ITU, "Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services-Coding of moving video, High efficiency video coding)), motion compensated temporal prediction is used to exploit the redundancy between consecutive pictures in a video.

[0041] To this end, a motion vector is associated with each prediction unit (PU) now introduced. Each CTU (coding tree unit) is represented by a coding tree in the compressed domain. This is a quadtree partitioning of the CTU, where each leaf is called a coding unit (CU), see Figure 1 .

[0042] Each CU is then given some intra or inter prediction parameters (prediction information). To this end, each CU is spatially split into one or more prediction units (PUs), each of which is assigned some prediction information. Figure 2 , intra-frame or inter-frame decoding mode is allocated at the CU level.

[0043] Specifically, one motion vector is assigned to each PU in HEVC. This motion vector is used for motion compensated temporal prediction of the considered PU. Therefore, in HEVC, the motion model of the linked prediction block and its reference blocks simply consists of translation.

[0044] In the Joint Exploration Model (JEM) and later VTM (Versatile Video Coding (VVC) Test Model) developed by the JVET (Joint Video Exploration Team) group, some richer motion models are supported to improve temporal prediction. To do this, a PU can be spatially divided into sub-PUs, and a richer model can be used to assign a dedicated motion vector to each sub-PU.

[0045] CU is no longer partitioned into PU or TU, but some motion data is directly allocated to each CU. In this new codec design, CU can be partitioned into sub-CUs, and motion vectors can be calculated for each sub-CU.

[0046] One of the new motion models introduced in JEM is the affine model, which basically consists in using an affine model to represent the motion vectors in a CU.

[0047] The motion model used is from Figure 3 It is shown that the affine motion field comprises the following motion vector component values ​​for each position (x, y) within the considered block:

[0048]

[0049] Equation 1: Affine model for generating motion fields within a CU for prediction

[0050] Coordinates (v 0x ,v 0y ) and (v 1x ,v 1y ) is the so-called control point motion vector used to generate the affine motion field. 0x ,v 0y ) is the upper left control point of the motion vector, and (v 1x ,v 1y ) is the upper right control point of the motion vector.

[0051] In fact, to keep the complexity reasonable, such as Figure 4 As shown, a motion vector is calculated for each 4×4 sub-block (sub-CU) of the considered CU. At the center position of each sub-block, an affine motion vector is calculated based on the control point motion vector. The obtained MV is represented with 1 / 16 pixel accuracy.

[0052] As a result, temporal decoding of a coding unit in affine mode consists in motion compensated prediction of each subblock using its own motion vector.

[0053] Note that models with three control points are also possible.

[0054] Affine motion compensation can be used in three ways in VTM: affine interframe (AF_INTER), affine merge and affine template. They are introduced as follows:

[0055] -Affine Inter (AF_INTER).

[0056] A CU in AMVP mode with a size greater than 8x8 can be predicted in affine inter mode. This is signaled by a flag in the bitstream. The generation of the affine motion field for the inter CU includes determining the control point motion vector (CPMV), which is obtained by the decoder by adding the motion vector difference to the control point motion vector prediction (CPMVP). The CPVMP is a pair of motion vector candidates, which are taken from the lists (A, B, C) and (D, E), respectively, as Figure 5 shown.

[0057] Up to 6 CPMVPs (3×2) can be obtained.

[0058] For a block of height H and width W, the validity of the first CPVMP is checked using Equation 2:

[0059]

[0060]

[0061]

[0062]

[0063] Equation 2: Effectiveness test for each CPMVP

[0064] Then, according to the third motion vector The valid CPVMPs are sorted by the value of (taken from position F or G). The closest approximation is given by the affine motion model for 4×4 sub-blocks with The vector given at the same position is preferably CPVMP.

[0065] For a block with a height of H and a width of W, the cost of each CPMVP is calculated using Equation 3, where X and Y are the horizontal and vertical components of the motion vector, respectively.

[0066]

[0067]

[0068]

[0069] Equation 3: Cost calculated for each CPMVP

[0070] -Affine Merge

[0071] In affine merge mode, the CU level flag indicates whether the merged CU uses affine motion compensation. If so, then Figure 7 The first available neighboring CU that has been coded in affine mode is selected from the ordered set of candidate positions (A, B, C, D, E).

[0072] Once the first neighboring CU in affine mode is obtained, the three motion vectors from the top left, top right and bottom left corners of the neighboring CU are retrieved (see Figure 8 ). Based on these three vectors, the two CPMVs of the upper left and upper right corners of the current CU are derived as follows:

[0073] ο

[0074] ο

[0075] Equation 4: Derivation of the CPMV of the current CU based on the three angular motion vectors of the neighboring CUs

[0076] When the control point motion vector of the current CU is obtained and When , the motion field within the current CU is calculated based on the 4×4 sub-CU through the model of Equation 1.

[0077] In the prior art, more candidates are added for affine merge mode, the best candidate is selected from a maximum of 7 candidates, and the index of the best candidate in the bitstream is decoded.

[0078] Another type of candidate is called temporal affine:

[0079] Similar to TMVP (Temporal Motion Vector Predictor) candidates, the affine CU is searched in the reference picture and added to the candidate list.

[0080] In another work, a new process for creating "virtual" candidates was added (see Fig. 9 ). The motivation is to create an affine candidate when no affine CU is available around the current CU. To do this, an affine model is created by taking the motion of each sub-block at the corner and creating an "affine" model.

[0081] During candidate list creation, there are 2 constraints to consider for complexity reasons:

[0082] -Total number of potential candidates: increases total computational requirements

[0083] - Final list size: increases latency at the decoder by increasing the number of comparisons required for each consecutive candidate.

[0084] Currently, virtual affine candidates are added to the list of potential candidates by taking the candidates at corners A, B, C and sorting the candidates using Eq. 2.

[0085] One problem addressed by the general aspects described herein is to enable the creation of more virtual temporal affine candidates, but filter each candidate to keep the final candidate list small and populated with relevant candidates.

[0086] In the current versions of JEM or VTM, there are no affine temporal candidates. In the prior art, virtual affine temporal candidates are proposed and they are currently being studied.

[0087] A basic idea of ​​the proposed aspects described is to enrich the set of virtual affine candidates (spatial and temporal), optionally replacing inherited affine candidates, and filtering or refining them to keep a list of relevant affine candidates. This process can produce a subset or the entire set of affine candidates.

[0088] Compared to the previous process of creating virtual affine candidates:

[0089] - Create more candidates

[0090] - To maintain a list of relevant candidates, filter each candidate, or refine the list

[0091] - Spatial and temporal candidates can be interleaved.

[0092] The affected codec modules are Fig.10 Motion estimation 175 and Fig.10 Motion compensation 170 and Fig.11 Motion compensation 275.

[0093] Fig.12 The sub-blocks for computing the virtual affine model are shown. For each angle A, B, C, D, apply Fig.13 The process described in : The sub-blocks are checked in order from 0 to 4. Once a valid sub-block is found (i.e., a sub-block with a motion vector), it is used to calculate the affine model. Note that in the case of spatial affine candidates, some sub-blocks are never available (e.g., block 0). The full affine model calculated from some corners is the same as the current virtual spatial candidate. Typically, the affine model can be 3, 4, or 6 parameters. Parameterization is usually done by giving the motion vectors associated with some corners of the current PU.

[0094] NOTE: This order can be different from A0, A1, ... A4, but is fixed between encoder and decoder.

[0095] Fig.14 An example process of the calculated affine model for a 4 parameter affine model is shown, where only corners A, B and C are picked. The model list is:

[0096] -Spatial model (A, B)

[0097] -Time model (A, B)

[0098] -Spatial model (A, C)

[0099] -Time model (A, C)

[0100] -Spatial Model (B, C)

[0101] - Time model (B, C),

[0102] Alternatively, adding angle D would add the following candidates:

[0103] -Time model (B, D)

[0104] -Time model (C, D)

[0105] - time model (A, D),

[0106] Or, for a 6 parameter affine model, up to 3 corners can be used together:

[0107] -Spatial model (A, B, C)

[0108] -Time Model (A, B, C)

[0109] -Time model (A, B, D)

[0110] - Time model (B, C, D),

[0111] Temporal candidates are interleaved with spatial candidates.

[0112] In all cases, when a corner is missing, if other corners are available, they can be reconstructed as described in PCT / US18 / 054318 ("Affine Model Candidates for Affine Motion Prediction").

[0113] exist Fig.14 In the example, for an affine model with 6 parameters, Fig.15 Model validity is defined as described in (note that it is also valid for the 4 parameter model).

[0114] exist Fig.15In , the translation model test is true when the motion vectors at A, B, and C for lists L0 and L1 are equal. If only one affine model (in either the L0 or L1 list) is translational, then the model is considered affine. If the PU is unidirectional, then the model is translational if the only affine model is translational.

[0115] The validation process first checks that the affine model does not introduce large motion differences between all corners (first 4 tests). In the test, A, B, and C represent Fig.12 The angles shown, Ax and Ay represent the motion vectors on the x and y axes, respectively. The values ​​h and w represent the PU height and width, and k is a fixed constant (e.g., for a PU with 1 / 16 (1 / 16 th ) high-precision motion vector with pixel accuracy, k=8).

[0116] This model is then compared to the models that are usually already in the list. If the model passes all tests, the model is added to the list.

[0117] To improve the diversity of the list of affine models, more virtual models can be created.

[0118] Fig.16 An example of additional motion vector data Ex and Fx for deriving a virtual affine model is shown.

[0119] The motion vectors located between the first and last rows are potential motion vectors used to create the affine model. Such a candidate is created if the block width (or height, respectively) is greater than a threshold D. Fig.16 In , the block width is assumed to be 32, and the block height is 16. In the case where both the height and width are greater than D, more potential motion vector data are added to the first / last row and the first row / last column.

[0120] Alternatively, additional candidate motion vectors are located every N pixels, where N is usually fixed to 16 pixels. In this case, for a block 48×16, Fig.17 The resulting motion vector sub-block positions are shown in .

[0121] For clarity, only the central motion vector data is shown. For the spatial affine model, the motion vector data is located outside the PU, while for the temporal affine model, the motion vector can be obtained inside the PU.

[0122] In this figure, the motion vector data are represented by their relative coordinates relative to the motion vector A0: V=A0+(u*N,v*N), where u is between 0 and w / N, and v is between 0 and h / N.

[0123] A list of derived motion models is formed by iterating over each potential motion vector data. Fig.18 and Fig.19 An example process is shown to extract a list of 4-parameter latent affine models (thus parameterized by control points). The process iterates over the motion vector data and forms angle pairs. In this variation, the x and y coordinates of the angle positions are iterated alternately.

[0124] When motion vector data is not available, or when the pair is on the same motion vector data, potential candidates are skipped. Note that in the spatial affine model, all motion vector data are taken from the boundaries of the PU, while in the temporal affine model, motion vector data can be taken from inside the PU.

[0125] Fig. 20 Shows the corresponding Fig.18 (Left: Raster scan) and Fig.19 (Right: zigzag scan) The path selected by the motion vector data.

[0126] The derivation of the motion model is done by applying a motion model transfer similar to the process described in Equation 4. For example, in Fig.21 In the example, the affine model represented by (v0, v1) is calculated by using the vector (v2, v3, v4) as described in Equation 4. First, vectors v3 and v4 are derived from the motion vector data v2 and v5, and then the model is transferred using the same principle considering that the adjacent PU is a PU formed by the angle of 2 motion vector data.

[0127] Fig.23 An embodiment of a method 2300 under the general aspects described herein is shown, the method starts at a start block 2301, and control proceeds to a block 2310 for determining a plurality of predictor candidates from neighboring subblocks and at least one reference picture for a subblock of a video block being encoded in a picture. Control proceeds from block 2310 to block 2320 to select a subset of the plurality of predictor candidates. Control proceeds from block 2320 to block 2330 to determine one or more control point motion vectors for the video block based on the subset of predictor candidates. Control proceeds from block 2330 to block 2340 to perform affine motion compensation on the subblock of the video block being encoded using the one or more control point motion vectors. The motion compensation process may include determining a motion vector field for the subblock based on the control point motion vectors. Control proceeds from block 2340 to block 2350 to encode the subblock of the video block.

[0128] Fig.24An embodiment of a method 2400 according to the general aspects described herein is shown, the method starts at a start block 2401, and control proceeds to a block 2410 for determining a plurality of predictor candidates from neighboring subblocks and at least one reference picture for a subblock of a video block being decoded in a picture. Control proceeds from block 2410 to block 2420 to select a subset of the plurality of predictor candidates. Control proceeds from block 2420 to block 2430 for determining one or more control point motion vectors for the video block from the subset of predictor candidates. Control proceeds from block 2430 to block 2340 for performing affine motion compensation on the subblock of the video block being decoded using the one or more control point motion vectors. The motion compensation process may include determining a motion vector field for the subblock based on the control point motion vectors. Control proceeds from block 2440 to block 2450 for decoding the subblock of the video block.

[0129] Fig.25 One embodiment of an apparatus 2500 for compressing, encoding or decoding video using improved virtual temporal affine candidates is shown. The apparatus includes a processor 2510 and may be interconnected to a memory 2520 through at least one port. The processor 2510 and the memory 2520 may also have one or more additional interconnections to external connections.

[0130] Processor 2510 is also configured to insert or receive information in a bitstream and use an interpolation filter for compression, encoding, or decoding.

[0131] Various aspects are described herein, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as having specificity, and at least in order to illustrate individual characteristics, are usually described in a manner that may sound restricted. However, this is for the purpose of describing clearly, and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide additional aspects. In addition, these aspects can also be combined and interchanged with the aspects described in earlier applications.

[0132] The aspects described and contemplated herein may be implemented in many different forms. Fig.10 , 11 22 provide some embodiments, but other embodiments are contemplated and are Fig.10 , 11The discussion of 22 does not limit the breadth of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0133] In this application, the terms "reconstruction" and "decoding" can be used interchangeably, the terms "pixel" and "sample" can be used interchangeably, and the terms "image", "picture" and "frame" can be used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0134] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0135] The various methods and other aspects described herein can be used to modify Fig.10 and Fig.11 Modules of the video encoder 100 and decoder 200 shown in, such as intra prediction, entropy coding and / or decoding modules (160, 360, 145, 330), in addition, aspects of the present invention are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations, whether pre-existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described herein can be used alone or in combination.

[0136] Various numerical values ​​are used in this document, for example, {{1,0}, {3,1}, {1,1}}. The specific values ​​are for example purposes, and the described aspects are not limited to these specific values.

[0137] Fig.10 An encoder 100 is shown. Variations of the encoder 100 are contemplated, but for the sake of clarity, the encoder 100 is described below without describing all contemplated variations.

[0138] Before being encoded, the video sequence may undergo a pre-encoding process (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0139] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is segmented (102) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) whether to encode the unit using one of intra mode or inter mode, and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (110) the predicted block from the original image block.

[0140] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass the transform and quantization, i.e., directly decode the residual without applying a transform or quantization process.

[0141] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed image to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0142] Fig.11 2 shows a block diagram of a video decoder 200. In the decoder 200, a bitstream is decoded by decoder elements as described below. The video decoder 200 generally performs the same Fig.10 The encoder 100 typically also performs video decoding as part of encoding the video data, as well as a decoding pass that is the inverse of the encoding pass described in .

[0143] The input to the decoder includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other decoding information. Picture segmentation information indicates how the picture is segmented. The decoder can therefore divide (235) the picture according to the decoded picture segmentation information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined (255) with the prediction block to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0144] The decoded picture may further undergo post-decoding processing (285), such as inverse color conversion (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which is a process that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0145] Fig. 22 A block diagram of an example of a system in which various aspects and embodiments are implemented is shown. System 1000 can be implemented as a device including various components described below, and is configured to perform one or more aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 1000 can be implemented individually or in combination in a single integrated circuit, multiple ICs and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed on multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to other similar systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 1000 is configured to implement one or more aspects described herein.

[0146] System 1000 includes at least one processor 1010, which is configured to execute instructions loaded therein, for implementing various aspects described herein, for example. Processor 1010 may include embedded memory, input and output interfaces, and various other circuits known in the art. System 1000 includes at least one memory 1020 (e.g., volatile memory device and / or non-volatile memory device). System 1000 includes storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drive, and / or optical drive. As a non-limiting example, storage device 1040 may include internal storage device, attached storage device, and / or network accessible storage device.

[0147] The system 1000 includes an encoder / decoder module 1030, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module (one or more) that may be included in the device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. In addition, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated into the processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0148] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of the various items during the execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0149] In several embodiments, memory within the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or a storage device 1040, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations, such as for MPEG-2, HEVC, or VVC (universal video coding).

[0150] As shown in block 1130, input to the elements of system 1000 may be provided through various input devices. Such input devices include, but are not limited to, (i) an RF section that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0151] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with the necessary elements for: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select (for example) a signal frequency band, which in some embodiments may be referred to as a channel, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband. In a set-top box embodiment, the RF part and its relevant input processing element receive the RF signal that sends by wired (for example, cable) medium, and filter to the frequency band of expectation again by filtering, down-conversion and perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) element, remove some in these elements, and / or add other elements that perform similar or different functions.Adding element can include and insert element between existing element, for example, insert amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.

[0152] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, in a separate input processing IC or processor 1010. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed in a separate interface IC or in processor 1010. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements (including, for example, processor 1010 and encoder / decoder 1030), which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on output devices.

[0153] The various components of the system 1000 may be disposed within an integrated housing. Within the integrated housing, the various components may be interconnected and transmit data therebetween using suitable connection means 1140 (eg, internal buses including I2C buses, wiring, and printed circuit boards known in the art).

[0154] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0155] In various embodiments, a wireless network such as IEEE 802.11 is used to transmit the data stream to the system 1000. The wireless signal of these embodiments is received by a communication channel 1060 and a communication interface 1050, for example, suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is usually connected to an access point or a router, which provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that transmits data through the HDMI connection of the input block 1130 to provide streaming data to the system 1000. Still other embodiments use the RF connection of the input block 1130 to provide streaming data to the system 1000.

[0156] The system 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripherals 1120. In various examples of various embodiments, the other peripherals 1120 include one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide functions based on the output of the system 1000. In various embodiments, control signals are transmitted between the system 1000 and the display 1100, the speaker 1110, or other peripherals 1120 using signaling such as AV. Link, CEC, or other communication protocols, which enables device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 1000 via dedicated connections through the corresponding interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to the system 1000 via the communication interface 1050 using the communication channel 1060. The display 1100 and the speaker 1110 can be integrated into a single unit in an electronic device (e.g., a television) along with the other components of the system 1000. In various embodiments, the display interface 1070 includes a display driver (eg, a timing controller (T Con) chip).

[0157] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0158] These embodiments may be implemented by computer software implemented by the processor 1010 or a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type suitable for the technical environment and may include, as a non-limiting example, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0159] Various implementations involve decoding. As used in the present application, "decoding" may include, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by a decoder of the various embodiments described in the present application, such as extracting indices of weights to be used for various intra-frame prediction reference arrays.

[0160] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations, or generally to a broader decoding process will be clear based on the context of the specific description, and is believed to be well understood by those skilled in the art.

[0161] Various implementations involve encoding. In a manner similar to the discussion above about "decoding", "encoding" as used in this application can include, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as segmentation, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by encoders of various implementations described in this application, such as weights of intra-frame prediction reference arrays.

[0162] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations, or generally to a broader encoding process will become clear based on the context of the specific description, and is believed to be well understood by those skilled in the art.

[0163] Note that the syntax elements used here are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0164] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0165] Various embodiments relate to rate-distortion calculation or rate-distortion optimization. During the encoding process, constraints on computational complexity are generally considered, and a balance or trade-off between rate and distortion is generally considered. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different methods to solve the rate-distortion optimization problem. For example, these methods can be based on extensive testing of all coding options, including all considered modes or decoding parameter values, and a complete evaluation of their decoding costs and the associated distortion of the reconstructed signal after decoding and decoding. Faster methods can also be used to save coding complexity, in particular, to calculate approximate distortion based on prediction or prediction residual signals rather than reconstructed signals. A mixture of these two methods can also be used, for example, by using approximate distortion only for some possible coding options and full distortion for other coding options. Other methods only evaluate a subset of possible coding options. More generally, many methods use any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of both decoding costs and associated distortions.

[0166] The implementations and aspects described herein can be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the discussed features can also be implemented in other forms (e.g., a device or program). For example, the device can be implemented with appropriate hardware, software, and firmware. For example, the method can be implemented in a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as computers, cellular phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate information communication between end users.

[0167] References to "one embodiment" or "an embodiment" or "one implementation" or "implementation" and other variations mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations in various places throughout this document are not necessarily all referring to the same embodiment.

[0168] Additionally, this document may refer to “determining” various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0169] Additionally, this document may refer to “accessing” various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0170] Additionally, this document may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often referred to in one way or another.

[0171] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," use of any of the following " / ," "and / or," and "at least one of..." is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to multiple items listed, as will be apparent to one of ordinary skill in this and related arts.

[0172] In addition, as used herein, the word "signaling" especially refers to indicating something to a corresponding decoder. For example, in some embodiments, the encoder signals a specific one of a plurality of weights to be used for an intra-frame prediction reference array. Thus, in one embodiment, the same parameters are used on the encoder side and the decoder side. Thus, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters as well as other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select specific parameters. Bit savings are achieved in various embodiments by avoiding the transmission of any actual function. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the foregoing relates to the verb form of the word "signaling", the word "signal" can also be used as a noun in this article.

[0173] As will be apparent to one of ordinary skill in the art, implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0174] Embodiments may include one or more of the following features or entities, alone or in combination, across a variety of claim categories and types:

[0175] Create additional virtual affine candidates.

[0176] Filtering of virtual affine candidates.

[0177] • Add the filtered virtual affine candidate to the candidate list.

[0178] • A bitstream or signal comprising one or more of the described syntax elements or variants thereof.

[0179] • Creating and / or sending and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0180] • A TV, set-top box, cell phone, tablet or other electronic device that performs in-loop filtering according to any of the described embodiments.

[0181] • A TV, set-top box, cell phone, tablet, or other electronic device performs in-loop filtering according to any of the described embodiments and displays the resulting image (eg, using a monitor, screen, or other type of display).

[0182] • A TV, set-top box, cell phone, tablet or other electronic device tunes a channel (eg, using a tuner) to receive a signal including an encoded image, and performs in-loop filtering according to any of the described embodiments.

[0183] • A TV, set-top box, cell phone, tablet or other electronic device receives a signal including an encoded image over the air (eg, using an antenna) and performs in-loop filtering according to any of the described embodiments.

[0184] Various other generalized and specific embodiments, aspects, and claims are also supported and contemplated throughout the present disclosure.

Claims

1. A method comprising: For a subblock of a video block being encoded in a picture, determining a plurality of predictor candidates from spatially neighboring subblocks and at least one temporal reference picture; selecting a subset of the plurality of predictor candidates after replacing the affine candidate with a refined virtual affine candidate to maintain a list of relevant affine candidates when the affine model used to determine the affine candidate does not cause motion differences between all corners of the video block and the affine model is not translational; determining one or more control point motion vectors for the video block based on the subset of predictor candidates; performing affine motion compensation for the sub-block of the video block being encoded using the one or more control point motion vectors; and The sub-block of the video block is encoded.

2. The method of claim 1 , wherein said selecting a subset of said plurality of predictor candidates comprises: performing a validation check on the plurality of predictor candidates to determine if an affine model does not cause motion differences between corners of the block, and At least one of the plurality of prediction candidates is conditionally added to a predictor candidate set based at least on the validation check. 3 . The method of claim 1 , further comprising encoding at least one index for the added predictor candidate from the predictor candidate set.

4. A device comprising: A processor configured to execute: For a subblock of a video block being encoded in a picture, determining a plurality of predictor candidates from spatially neighboring subblocks and at least one temporal reference picture; selecting a subset of the plurality of predictor candidates after replacing the affine candidate with a refined virtual affine candidate to maintain a list of relevant affine candidates when the affine model used to determine the affine candidate does not cause motion differences between all corners of the video block and the affine model is not translational; determining one or more control point motion vectors for the video block based on the subset of predictor candidates; performing affine motion compensation for the sub-block of the video block being encoded using the one or more control point motion vectors; and The sub-block of the video block is encoded. 5 . The apparatus of claim 4 , wherein the selecting a subset of the plurality of predictor candidates comprises replacing inherited affine candidates with virtual candidates. The apparatus of claim 4 , wherein the affine motion compensation is performed by using at least 3 parameters.

7. A method comprising: For a subblock of a video block being decoded in a picture, determining a plurality of predictor candidates from spatially neighboring subblocks and at least one temporal reference picture; selecting a subset of the plurality of predictor candidates after replacing the affine candidate with a refined virtual affine candidate to maintain a list of relevant affine candidates when the affine model used to determine the affine candidate does not cause motion differences between all corners of the video block and the affine model is not translational; determining one or more control point motion vectors for the video block based on the subset of predictor candidates; performing affine motion compensation for the sub-block of the video block being decoded using the one or more control point motion vectors; and The sub-blocks of the video block are decoded.

8. The method of claim 7, wherein motion vectors located in a first line and a last line of the video block are used in an affine model.

9. The method of claim 7, wherein motion vectors located at every N pixels of the video block are used in an affine model.

10. An apparatus comprising: A processor configured to execute: For a subblock of a video block being decoded in a picture, determining a plurality of predictor candidates from spatially neighboring subblocks and at least one temporal reference picture; selecting a subset of the plurality of predictor candidates after replacing the affine candidate with a refined virtual affine candidate to maintain a list of relevant affine candidates when the affine model used to determine the affine candidate does not cause motion differences between all corners of the video block and the affine model is not translational; determining one or more control point motion vectors for the video block based on the subset of predictor candidates; performing affine motion compensation for the sub-block of the video block being decoded using the one or more control point motion vectors; and The sub-blocks of the video block are decoded. The apparatus of claim 10 , wherein three corners of the video block are used for the affine model.

12. A device comprising: The device according to any one of claims 4 to 6 and claims 10 to 11; as well as At least one of: (i) an antenna configured to receive a signal, the signal comprising the video block, (ii) a frequency band limiter configured to limit the received signal to a frequency band comprising the video block, and (iii) a display configured to display an output representing the video block.

13. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 3 and claims 7 to 9.