Methods and apparatus for encoding samples at pixel positions in sub-blocks

Abstract

A method and apparatus for encoding a sample at a pixel position in a sub-block is provided. The method can include generating a plurality of affine motion compensated predictions at the pixel position in the sub-block and a plurality of neighboring pixel positions, determining a motion vector (MV) difference at at least one of the plurality of neighboring pixel positions, determining coefficients of a filter having a predetermined shape based on the motion vector (MV) difference, and obtaining a refined prediction of the sample at the pixel position using the filter based on the plurality of affine motion compensated predictions.

Description

[0001] This application is a divisional application of the invention patent application filed on July 30, 2021, with application number "202180004943.X" and titled "Method and Apparatus for Predictive Refinement of Affine Motion Compensation". Technical Field

[0002] This disclosure relates to video encoding and decoding and compression, and specifically, but not limited to, methods and apparatus for predictive refinement (AMPR) of affine motion compensation in video encoding and decoding. Background Technology

[0003] Various video codec technologies can be used to compress video data. Video codecs are performed according to one or more video codec standards. For example, some well-known video codec standards today include Universal Video Codec (VVC), High Efficiency Video Codec (HEVC, also known as H.265 or MPEG-H Part 2), and Advanced Video Codec (AVC, also known as H.264 or MPEG-4 Part 10), jointly developed by ISO / IEC MPEG and ITU-T VECG. AOMedia Video1 (AV1) was developed by the Alliance for Open Media (AOM) as the successor to its previous standard VP9. Audio and Video Codecs (AVS) are another series of video compression standards developed by the Chinese Audio and Video Codec Standards Working Group, referring to digital audio and digital video compression standards. Most existing video codec standards are built on the well-known hybrid video codec framework, which uses block-based prediction methods (e.g., inter-frame prediction, intra-frame prediction) to reduce redundancy present in video images or sequences, and uses transform codecs to compress the energy of prediction errors. A key goal of video encoding and decoding technology is to compress video data into a form that uses a lower bit rate, while avoiding or minimizing the degradation of video quality.

[0004] The first generation of AVS standards included the Chinese national standards "Information Technology, Advanced Audio and Video Coding, Part 2: Video" (referred to as AVS1) and "Information Technology, Advanced Audio and Video Coding, Part 16: Broadcast Television Video" (referred to as AVS+). Compared to the MPEG-2 standard, it offered approximately 50% bitrate savings at the same perceived quality. The video portion of the AVS1 standard was released as a Chinese national standard in February 2006. The second generation of AVS standards included the Chinese national standard series "Information Technology, High-Efficiency Multimedia Coding" (referred to as AVS2), primarily targeting the transmission of ultra-high-definition television programs. AVS2's coding and decoding efficiency was twice that of AVS+. In May 2016, AVS2 was released as a Chinese national standard. Simultaneously, the video portion of the AVS2 standard was submitted by the Institute of Electrical and Electronics Engineers (IEEE) as an international standard for applications. The AVS3 standard is a new generation of video coding and decoding standards for UHD video applications, aiming to surpass the coding and decoding efficiency of the latest international standard, HEVC. In March 2019, at the 68th AVS meeting, the AVS3-P2 baseline was completed, offering approximately 30% bit rate savings compared to the HEVC standard. Currently, a reference software called the High Performance Model (HPM) exists, maintained by the AVS group to demonstrate a reference implementation of the AVS3 standard. Summary of the Invention

[0005] This disclosure provides examples of techniques related to AMPR for the AVS3 standard.

[0006] According to a first aspect of this disclosure, a method is provided for encoding samples at pixel locations in a sub-block, comprising: generating a plurality of affine motion compensation predictions at the pixel location and a plurality of adjacent pixel locations in the sub-block; determining a motion vector MV difference at at least one of the plurality of adjacent pixel locations; determining coefficients of a filter having a predetermined shape based on the motion vector MV difference; and using the filter based on the plurality of affine motion compensation predictions to obtain a refined prediction of the samples at the pixel location.

[0007] According to a second aspect of this disclosure, an apparatus is provided for encoding samples at pixel locations in a sub-block. The apparatus includes one or more processors; and a memory configured to store instructions executable by the one or more processors. Upon execution of the instructions, the one or more processors are configured to: generate a plurality of affine motion compensation predictions at the pixel location and a plurality of adjacent pixel locations in the sub-block; determine a motion vector MV difference at at least one of the plurality of adjacent pixel locations; determine coefficients of a filter having a predetermined shape based on the motion vector MV difference; and use the filter based on the plurality of affine motion compensation predictions to obtain a refined prediction of the sample at the pixel location.

[0008] According to a third aspect of this disclosure, a non-transitory computer-readable storage medium is provided for encoding samples at pixel locations in a sub-block, storing computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform actions, including: generating a plurality of affine motion compensation predictions at the pixel location and a plurality of adjacent pixel locations in the sub-block; determining a motion vector MV difference at at least one of the plurality of adjacent pixel locations; determining coefficients of a filter having a predetermined shape based on the motion vector MV difference; and using the filter based on the plurality of affine motion compensation predictions to obtain a refined prediction of the sample at the pixel location.

[0009] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores a bit stream, wherein the bit stream is obtained by encoding by performing the method described above. Attached Figure Description

[0010] A more detailed description of the examples of this disclosure will be presented with reference to the specific examples shown in the accompanying drawings. Given that these drawings depict only a few examples and are therefore not intended to limit the scope, the examples will be described and explained using additional features and details through the use of the drawings.

[0011] Figure 1 This is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

[0012] Figure 2 This is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

[0013] Figures 3A to 3E This is a schematic diagram illustrating a multi-type tree partitioning pattern according to some embodiments of the present disclosure.

[0014] Figure 4This is a schematic diagram illustrating an example of a bidirectional optical flow (BIO) model according to some implementations of this disclosure.

[0015] Figures 5A to 5B This is a schematic diagram illustrating an example of a 4-parameter affine model according to some embodiments of the present disclosure.

[0016] Figure 6 This is a schematic diagram illustrating an example of a 6-parameter affine model according to some embodiments of the present disclosure.

[0017] Figure 7 The Predictive Refinement (PROF) process for affine modes using optical flow is illustrated according to some embodiments of the present disclosure.

[0018] Figure 8 Examples of calculating the horizontal and vertical offsets from the sample point location to a specific location of the sub-block from which the sub-block MV is derived, according to some embodiments of this disclosure, are shown.

[0019] Figure 9 An example of a sub-block within an affine CU is shown according to some embodiments of this disclosure.

[0020] Figures 10A to 10B Examples of diamond filters according to some embodiments of this disclosure are shown.

[0021] Figures 11A to 11B Examples of diamond filters scaled by MV difference in the horizontal and vertical directions according to some embodiments of this disclosure are shown.

[0022] Figure 12 This is a block diagram illustrating an exemplary apparatus for predicting sample points at pixel locations in a sub-block using AMPR according to some embodiments of the present disclosure.

[0023] Figure 13 This is a flowchart illustrating an exemplary process of predicting sample points at pixel locations in a sub-block using AMPR according to some embodiments of the present disclosure.

[0024] Figure 14 This is a flowchart illustrating steps in an exemplary process of predicting sample points at pixel locations in a sub-block using AMPR according to some embodiments of the present disclosure. Detailed Implementation

[0025] Referring now to the detailed description, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives may be used. For example, it will be apparent to those skilled in the art that the subject matter presented herein can be implemented on many types of electronic devices with digital video capabilities.

[0026] Throughout this specification, references to "an embodiment," "an embodiment," "an example," "some embodiments," "some examples," or similar language indicate that a particular feature, structure, or characteristic described is included in at least one embodiment or example. Unless otherwise expressly stated, the features, structures, elements, or characteristics described in connection with one or more embodiments also apply to other embodiments.

[0027] Throughout this disclosure, unless otherwise expressly stated, the terms “first,” “second,” “third,” etc., are used only to refer to related elements (e.g., equipment, components, compositions, steps, etc.) and do not indicate any spatial or temporal order. For example, “first equipment” and “second equipment” can refer to two separately formed devices, or two parts, components, or operating states of the same device, and can be named arbitrarily.

[0028] The terms "module," "submodule," "circuit," "subcircuit," "circuit system," "subcircuit system," "unit," or "subunit" may include memory (shared, dedicated, or combined) storing code or instructions executable by one or more processors. A module may include one or more circuits, with or without stored code or instructions. A module or circuit may include one or more components that are directly or indirectly connected. These components may or may not be physically attached to each other or located adjacent to each other.

[0029] As used herein, depending on the context, the terms "if" or "when..." may be understood to mean "at..." or "in response to...". If these terms appear in the claims, they do not necessarily indicate that the relevant limitation or feature is conditional or optional. For example, a method may include the steps of: i) performing a function or action X' when or if condition X exists, and ii) performing a function or action Y' when or if condition Y exists. The method may be implemented with both the ability to perform function or action X' and the ability to perform function or action Y'. Thus, both function X' and function Y' may be performed at different times during multiple executions of the method.

[0030] Units or modules can be implemented purely by software, purely by hardware, or by a combination of hardware and software. In a purely software implementation, for example, a unit or module may include functionally related code blocks or software components that are directly or indirectly linked together to perform a specific function.

[0031] Figure 1 A block diagram is shown illustrating an exemplary block-based hybrid video encoder 100 that can be used in conjunction with many video codec standards that employ block-based processing. In encoder 100, video frames are segmented into multiple video blocks for processing. For each given video block, a prediction is formed based on either an inter-frame prediction method or an intra-frame prediction method. In inter-frame prediction, one or more prediction factors are formed based on pixels from previously reconstructed frames, through motion estimation and motion compensation. In intra-frame prediction, prediction factors are formed based on reconstructed pixels in the current frame. Through mode decision, the optimal prediction factor can be selected to predict the current block.

[0032] Intra-frame prediction (also known as "spatial prediction") uses pixels from samples (called reference samples) of already encoded and decoded neighboring blocks in the same video frame and / or strip to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal.

[0033] Inter-frame prediction (also known as "temporal prediction") uses reconstructed pixels from already encoded and decoded video frames to predict the current video block. Temporal prediction reduces the inherent temporal redundancy in the video signal. A temporal prediction signal for a given coding unit (CU) or coding block is typically transmitted via one or more motion vectors (MVs) indicating the amount and direction of motion between the current CU and its temporal reference. Additionally, if multiple reference frames are supported, a separate reference frame index is sent, which identifies which reference frame in the reference frame repository the temporal prediction signal originates from.

[0034] After performing spatial and / or temporal prediction, the intra / inter-frame mode decision circuit 121 in encoder 100 selects the optimal prediction mode, for example, based on a rate-distortion optimization method. The block prediction factor 120 is then subtracted from the current video block; and the resulting prediction residual is decorrelated using transform circuit 102 and quantization circuit 104. The resulting quantized residual coefficients are dequantized by inverse quantization circuit 116 and inverse transformed by inverse transform circuit 118 to form the reconstruction residual, which is then added back to the prediction block to form the reconstructed signal of the CU. Furthermore, before placing the reconstructed CU into the reference image repository of image buffer 117 for encoding subsequent video blocks, loop filtering 115, such as a deblocking filter, sample adaptive offset (SAO), and / or adaptive loop filter (ALF), can be applied to the reconstructed CU. To form the output video bitstream 114, the encoding mode (inter-frame or intra-frame), prediction mode information, motion information, and quantized residual coefficients are all sent to entropy coding unit 106 for further compression and packing to form the bitstream.

[0035] For example, deblocking filters are available in current versions of AVC, HEVC, and VVC. In HEVC, an additional loop filter called Sample Adaptive Offset (SAO) is defined to further improve encoding and decoding efficiency. In the current version of the VVC standard, another loop filter called Adaptive Loop Filter (ALF) is under active investigation and is very likely to be included in the final standard.

[0036] These loop filter operations are optional. Performing these operations helps improve encoding / decoding efficiency and visual quality. They can also be turned off based on decisions made by encoder 100 to save computational complexity.

[0037] It should be noted that when these filter options are enabled in encoder 100, intra-frame prediction is typically based on unfiltered reconstructed pixels, while inter-frame prediction is based on filtered reconstructed pixels.

[0038] Figure 2 This is a block diagram illustrating an exemplary block-based video decoder 200 that can be used in conjunction with many video codec standards. The decoder 200 is similar to [the one residing in...]. Figure 1The reconstruction-related part is located in the encoder 100. In the decoder 200, the input video bitstream 201 is first decoded by entropy decoding 202 to derive the quantized coefficient levels and prediction-related information. Then, the quantized coefficient levels are processed by inverse quantization 204 and inverse transform 206 to obtain the reconstructed prediction residuals. The block predictor mechanism implemented in the intra / inter-frame mode selector 212 is configured to perform intra-frame prediction 208 or motion compensation 210 based on the decoded prediction information. A set of unfiltered reconstructed pixels is obtained by summing the reconstructed prediction residuals from inverse transform 206 and the prediction output generated by the block predictor mechanism using summer 214.

[0039] Before storing the reconstructed blocks in image buffer 213, which serves as a reference image repository, the reconstructed blocks can be further passed through loop filter 209. The reconstructed video in image buffer 213 can be sent to drive a display device and to predict subsequent video blocks. With loop filter 209 open, filtering operations are performed on these reconstructed pixels to produce the final reconstructed video output 222.

[0040] The video encoding / decoding standards mentioned above (such as HEVC and AV3) are conceptually similar. For example, they all use a block-based hybrid video encoding / decoding framework. The following details some of the block segmentation schemes in these standards.

[0041] HEVC uses only quadtrees to partition blocks. The basic unit used for compression is called a coding tree unit (CTU). Each CTU can contain one coding unit (CU) or be recursively divided into four smaller CUs until a predefined minimum CU size is reached. Each CU (also called a leaf CU) contains one or more prediction units (PUs) and transform units (TUs) trees.

[0042] In AVS3, a coding tree unit (CTU) is divided into multiple CUs based on a quadtree / binary tree / extended quadtree to accommodate varying local characteristics. Furthermore, the concept of multiple partitioning unit types in HEVC is removed; that is, there is no distinction between CU, PU, ​​and TU in AVS3. Instead, each CU is always used as the basic unit for both prediction and transformation without further partitioning. In AVS3's tree partitioning structure, a CTU is first partitioned based on a quadtree structure. Then, each quadtree leaf node can be further partitioned based on binary tree and extended quadtree structures.

[0043] Figures 3A to 3E This is a schematic diagram illustrating multiple tree partitioning patterns according to some embodiments of the present disclosure. For example... Figures 3A to 3EAs shown, there are five partitioning types in the multi-type tree structure: quadruple partition 301, vertical binary partition 302, horizontal binary partition 303, vertical extended quadruple partition 304, and horizontal extended quadruple partition 305.

[0044] In the current VVC and AVS3 standards, block-based motion compensation can be applied to achieve a trade-off between encoding / decoding efficiency, complexity, and memory access bandwidth. Average prediction accuracy is lower than pixel-based prediction because all pixels within a block or sub-block share the same block-level motion vector. To improve prediction accuracy at each pixel, PROF for affine modes is adopted as the encoding / decoding tool in the current VVC standard. A similar tool does not exist in AVS3.

[0045] Some examples in this disclosure provide alternative optical flow-based methods to improve affine mode longitude prediction, as described below.

[0046] Affine mode

[0047] In HEVC, only the translational motion model is applied to motion compensation prediction. However, in the real world, many types of motion exist, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC and AVS3, affine motion compensation prediction is applied by sending a flag to each inter-frame codec block to indicate whether a translational or affine motion model is applied to inter-frame prediction. In current VVC and AVS3 designs, two affine modes are supported for an affine codec block: a 4-parameter affine mode and a 6-parameter affine mode.

[0048] A 4-parameter affine model can have the following parameters: two parameters for translational motion in the horizontal and vertical directions, one parameter for scaling motion in both directions, and one parameter for rotational motion in both directions. In a 4-parameter affine model, the horizontal scaling parameter can be equal to the vertical scaling parameter, and the horizontal rotation parameter can be equal to the vertical rotation parameter. To achieve better reconciliation between the motion vectors and affine parameters, those affine parameters can be derived from the two MVs (also known as control point motion vectors (CPMVs)) located at the top left and top right corners of the current block.

[0049] Figures 5A to 5B This is a diagram illustrating an example of a four-parameter affine model according to some embodiments of the present disclosure. For example... Figures 5A to 5B As shown, the affine motion field of a block is described by two control points MVs(V0, V1). Based on the motion of the control points, the motion field (v0, V1) of a block that has been affine encoded and decoded is... x ,v y It is described by the following equation (1):

[0050]

[0051] A 6-parameter affine model can have the following parameters: two parameters for translational motion in the horizontal and vertical directions, two parameters for scaling and rotational motion in the horizontal direction, and two additional parameters for scaling and rotational motion in the vertical direction. The 6-parameter affine motion model is encoded and decoded using three CPMVs.

[0052] Figure 6 This is a diagram illustrating an example of a 6-parameter affine model according to some embodiments of the present disclosure. For example... Figure 6 As shown, the three control points of a 6-parameter affine block are located at the top left, top right, and bottom left corners of the block. The motion at the top left control point is related to translational motion, the motion at the top right control point is related to rotational and scaling motion in the horizontal direction, and the motion at the bottom left control point is related to rotational and scaling motion in the vertical direction. Compared to the 4-parameter affine motion model, the horizontal rotational and scaling motions of the 6-parameter affine motion model can differ from the vertical rotational and scaling motions.

[0053] In some examples, when (V0, V1, V2) is Figure 6 If the top-left, top-right, and bottom-left corners of the current block are MV values, then each sub-block (v...) x ,v y The motion vector of ) is derived using the three MVs at the control points as shown in the following equation (2):

[0054]

[0055] Predictive refinement (PROF) using optical flow for affine modes

[0056] To improve the accuracy of affine motion compensation, PROF is used in VVC, which refines sub-block-based affine motion compensation based on an optical flow model. Specifically, after performing sub-block-based affine motion compensation, each brightness prediction sample of an affine block is modified by a sample refinement value derived based on the optical flow equation. In some examples, the operation of PROF can be summarized in the following four steps:

[0057] In the first step, sub-block-based affine motion compensation is performed using the sub-block MV derived from equation (1) above for the 4-parameter affine model or the sub-block MV derived from equation (2) above for the 6-parameter affine model to generate sub-block prediction I(i,j).

[0058] Furthermore, in the second step, the spatial gradient g of each predicted sample point... x (i, j) and g y (i, j) is calculated as shown in the following equation (3):

[0059]

[0060] Therefore, to compute the gradient, an additional row and / or column of predicted samples needs to be generated on each of the four sides of a sub-block, expanding the 4×4 sub-block to a 6×6 sub-block. To reduce memory bandwidth and complexity, the samples on the expanded boundaries are copied from the nearest integer pixel position in the reference image to avoid additional interpolation processes.

[0061] In addition, in the third step, the brightness prediction refinement value is calculated using the following equation (4):

[0062] ΔI(i,j)=g x (i, j)*Δv x (i, j) + g y (i, j)*Δv y (i, j) (4)

[0063] Where Δv(i,j) is the difference between the pixel MV calculated for the sample point position (i,j) and the sub-block MV of the sub-block where pixel (i,j) is located, represented by v(i,j).

[0064] Figure 7 The PROF process for affine modes according to some embodiments of this disclosure is illustrated. In PROF, after the prediction refinement is added to the original prediction sample, a clipping operation "clip3" is performed to clip the value of the refined prediction sample to within 15 bits, as shown in the following equation:

[0065] I r (i, j) = I(i, j) + ΔI(i, j)

[0066] I r (i,j)=clip3(-dILimit,dILimit-1,I r (i, j))

[0067] dILimit=(1<<max(13,BitDepth+1))

[0068] where I(i, j) and I r (i, j) represent the original predicted sample value and the refined predicted sample value at position (i, j), respectively. The function clip3(min, max, val) restricts the given value "val" to the range [min, max].

[0069] Since the affine model parameters and the pixel position relative to the center of the sub-block do not change between sub-blocks, Δv(i,j) can be calculated for the first sub-block and then applied to other sub-blocks in the same CU. When Δx and Δy are the horizontal and vertical offsets from the sample position (i,j) to the center of the sub-block to which the sample belongs, Δv(i,j) can be derived as shown in the following equation (5):

[0070]

[0071] Based on the affine subblock MV, equations (1) and (2) are derived, and the parameters c, d, e, and f in equation (5) above can be derived. Specifically, for a 4-parameter affine model, the parameters c, d, e, and f can be derived as shown in the following equation:

[0072]

[0073] Furthermore, for a 6-parameter affine model, the parameters c, d, e, and f can be derived as shown in the following equation:

[0074]

[0075] Where (v 0x v 0y ), (v 1x v 1y ), (v 2x v 2y ) represents the top-left control point MV, top-right control point MV, and bottom-left control point MV of the current coded block, while w and h are the width and height of the block. In PROF, MV differs from Δv. x and Δv y It is always exported with a resolution of 1 / 32 of a pixel.

[0076] Finally, in the fourth step, the brightness prediction refinement ΔI(i,j) is added to the sub-block prediction I(i,j). The final prediction I′(i,j) for the sample at position (i,j) is generated as shown in the following equation (6):

[0077] I′(i,j)=I(i,j)+ΔI(i,j) (6)

[0078] Bidirectional optical flow (BIO)

[0079] Bidirectional prediction in video encoding and decoding is a simple combination of two temporal prediction blocks obtained from a reference image. However, due to the signaling cost and accuracy trade-offs in motion vectors, the motion vectors received at the decoder may not be very accurate. As a result, there may still be some minor motion remaining between the two prediction blocks, which can reduce the efficiency of motion-compensated prediction. To address this issue, both the VVC and AVS3 standards employ the BIO tool to compensate for this motion for each sample within a block. Specifically, BIO is a sample-by-sample motion refinement performed on top of block-based motion-compensated prediction when using bidirectional prediction.

[0080] In BIO design, the derivation of the refined motion vector for each sample point in a block is based on the classical optical flow model. Let I (k) (x, y) represents the sample value at coordinates (x, y) of the predicted block derived from the reference image list k (k = 0, 1), and and Let be the horizontal and vertical gradients of the sample point. Assuming the optical flow model is valid, then the motion refinement (v) at (x, y) is... x v y The following optical flow equation (7) can be derived:

[0081]

[0082] Using a combination of optical flow equation (7) and interpolation of the predicted block along the motion trajectory, such as Figure 4 As shown, BIO prediction can be obtained, as shown in the following equation (8):

[0083]

[0084] Figure 4 This is a schematic diagram illustrating examples of BIO models according to some embodiments of this disclosure. For example... Figure 4 As shown, (MV) x0 MV y0 ) and (MV x1 MV yl The instruction is used to generate two prediction blocks I. (0) and I (1) The block-level motion vector. Furthermore, by minimizing motion refinement compensation (i.e., Figure 4 The difference Δ between the sample values ​​after A and B in the original text is used to calculate the motion refinement (v) at the sample location (x, y). x v y As shown in equation (9) below:

[0085]

[0086] Furthermore, to ensure the uniformity of the derived motion refinement, it is assumed that the motion refinement is consistent within the local surrounding region centered at (x, y); therefore, in the BIO design in AVS3, (v) is derived by minimizing Δ within a 4×4 window Ω around the current sample point at (x, y). x v y The value of ) is shown in equation (10) below:

[0087]

[0088] As shown in equations (8) and (10), in addition to the block-level MC, it is also necessary to perform motion compensation for each motion compensation block (i.e., I) in the BIO. (0) and I (1) The gradient is derived for each sample point to refine the local motion and produce the final prediction at that sample point location. In AVS3, the gradient is computed via a two-dimensional (2D) separable finite impulse response (FIR) filtering process that defines a set of 8-tap filters and applies different filters according to the block-level motion vector (e.g., ...). Figure 4 (MV) x0 ,MV y0 ) and (MV x1 ,MV y1 The accuracy of the gradient is used to derive the horizontal and vertical gradients. Table 1 shows the coefficients of the gradient filter used by BIO.

[0089] Table 1

[0090] Fraction Position Gradient Filter 0 {-4，11，-39，-1，41，-14，8，-2} 1 / 4 {-2，6，-19，-31，53，-12，7，-2} 1 / 2 {0，-1，0，-50，50，0，1，0} 3 / 4 {2，-7，12，-53，31，19，-6，2}

[0091] Finally, BIO is applied only to bidirectional prediction blocks predicted by two reference blocks from temporally adjacent images. Furthermore, BIO is enabled without sending additional information from the encoder to the decoder. Specifically, BIO is applied to all bidirectional prediction blocks that have both forward and backward prediction signals.

[0092] Final Motion Vector Expression (UMVE)

[0093] The UMVE mode in the AVS3 standard is the same tool as the merge mode named with motion vector difference (MMVD) in the VVC standard. In addition to the regular merge mode which derives the motion information of the current block from its spatial / temporal neighboring blocks, the MMVD / UMVE mode is introduced as a special merge mode in both the VVC and AVS standards.

[0094] Specifically, in both VVC and AVS3, this is signaled at the code block level via an MMVD flag. In MMVD mode, two basic merge candidates are first generated as the first two candidates in the regular merge mode. After selecting and signaling a basic merge candidate, additional syntax elements are signaled to indicate the MVD of motion added to the selected merge candidate. The MMVD syntax elements include a merge candidate flag for selecting the basic merge candidate, a distance index for specifying the MVD size, and a direction index for indicating the MVD direction.

[0095] In the AVS3 standard, sub-block-based affine motion compensation (affine mode), similar to that in the VVC standard, is used to generate inter-frame predicted pixel values. This sub-block-based prediction is a trade-off between coding efficiency, complexity, and memory access bandwidth. The average prediction accuracy is lower than pixel-based prediction because all pixels within each sub-block share the same motion vector. Unlike the VVC standard, the AVS3 standard does not provide pixel-level refinement after sub-block-based motion compensation in affine mode.

[0096] This disclosure provides a novel method for improving prediction accuracy in affine modes. Following conventional sub-block-based affine motion compensation, the prediction value for each pixel is refined by adding a difference value derived from the optical flow equation. The proposed method may be referred to as AMPR. AMPR achieves pixel-level prediction accuracy without significantly increasing complexity, while also maintaining worst-case memory access bandwidth comparable to conventional sub-block-based motion compensation in affine modes. Although AMPR is also based on optical flow, it differs significantly from PROF in the VVC standard in the following aspects.

[0097] First, gradient calculation is performed at each pixel. Unlike PROF, which expands the sub-block prediction by one pixel on each side, AMPR utilizes interpolation-based filtering for gradient calculation at each pixel, allowing for a unified design between AMPR and BIO workflows in AVS3.

[0098] Second, MV difference is calculated at each pixel. Unlike PROF, which always calculates the MV difference based on the pixel position relative to the center of the sub-block, AMPR can calculate the MV difference based on the pixel position relative to different positions within the sub-block.

[0099] Third is early termination. Unlike the PROF process, which is always called on the decoder side when predicting coded blocks via affine modes, AMPR can be adaptively skipped on the decoder side based on certain defined conditions indicating that applying AMPR is not a good performance and / or complexity tradeoff.

[0100] In some examples, this early termination method can also be used to simplify encoder-side operations. Examples disclosed here also present some encoder-side optimization methods for the AMPR process to reduce its latency and energy consumption, such as skipping AMPR for affine UMVEs, checking the optimal mode selection at the parent CU before applying AMPR, skipping AMPR for motion estimation of certain block sizes, checking the magnitude of the pixel MV difference before applying AMPR, and skipping AMPR for certain picture types (e.g., low-latency pictures or non-low-latency pictures), etc.

[0101] AMPR's exemplary workflow

[0102] The AMPR method may include the five steps described below. In the first step, conventional sub-block-based affine motion compensation is performed to generate a sub-block prediction I(i,j) at each pixel location (i,j).

[0103] In the second step, the horizontal spatial gradient g of the sub-block prediction is calculated at each pixel location using interpolation-based filtering. x (i, j) and vertical spatial gradient g y (i, j). In some examples, both the horizontal and vertical gradients of the affine prediction samples are calculated directly from reference samples at integer sample positions in the time reference image. One advantage of this is that for each affine sub-block, its gradient values ​​can be generated simultaneously with the generation of its prediction samples. Another design benefit of this gradient calculation method is that it is consistent with the gradient calculation process used by other codec tools in the AVS standard, such as BIO. Sharing the same process between different modules in the standard is friendly to pipelined and / or parallel designs in actual hardware codec implementations.

[0104] Specifically, the input to the gradient derivation process is the same reference samples used for motion compensation of the affine sub-blocks and the input motion (MV) of the sub-blocks. x MV y The same fractional components (fracX, fracY) are used to derive the gradient values ​​at each sample location, in addition to the default 8-tap FIR filter h used for affine prediction. L In addition, another set of new FIR filters h is introduced into the proposed method. G To calculate the gradient value.

[0105] Additionally, depending on the direction of the derived gradient, the filter h is applied. G and h L The order is different. In deriving the horizontal gradient g... x In the case of (i, j), first apply the gradient filter h in the horizontal direction. GTo derive the horizontal gradient value at the horizontal fractional sample location fracX; then, apply the interpolation filter h vertically. L The gradient value is interpolated at the vertical fractional sample point location fracY.

[0106] Conversely, when deriving the vertical gradient g y When (i, j), first apply the interpolation filter h horizontally. L Interpolate intermediate samples at the horizontal sample location fracX, and then apply a gradient filter h in the vertical direction. G The vertical gradient value at the vertical fractional sample point position fracY is derived from the intermediate interpolation sample point.

[0107] In some examples, gradient filters can be generated with different filter coefficient accuracies and different numbers of taps, which can provide various trade-offs between gradient computation accuracy and computational complexity. For example, gradient filters with more filter taps and / or higher filter coefficient accuracy generally lead to better encoding / decoding efficiency, but at the cost of more computational operations (e.g., multiple additions, multiplications, and shifts) due to the gradient computation process. In one example, the following 8-tap filter is proposed for the horizontal and / or vertical gradient computation of AMPR, as shown in Table 2.

[0108] Table 2 lists the predefined 8-tap interpolation filter coefficients f used to generate spatial gradients with 1 / 16 pixel precision based on input sample values. grad An example table for [p].

[0109] Table 2

[0110]

[0111] In another example, to reduce the complexity of gradient computation, the following 4-tap FIR filters, as shown in Table 3, are used for gradient generation in the proposed AMPR method. Table 3 lists the predefined 4-tap interpolation filter coefficients f for generating spatial gradients with 1 / 16 pixel precision based on input sample values. grad An example table for [p].

[0112] Table 3

[0113]

[0114] In the third step, at each pixel position (i, j), the MV difference Δv(i, j) between the MV of each pixel and the MV of the sub-block to which the pixel belongs is calculated. Figure 8 An example is shown that calculates the horizontal and vertical offsets from the sample point location to a specific location in the derived sub-block MV. For example... Figure 8As shown, the horizontal offset Δx and vertical offset Δy are calculated from the sample point position (i, j) to a specific position (i′, j′) in the sub-block from which the sub-block MV is derived. In some examples, the specific position (i′, j′) may not always be the center of the sub-block. Figure 8 As shown, Δv(i,j) is calculated based on the pixel position relative to a specific position within the sub-block using equation (5).

[0115] In one example, for the affine pattern, let (i, j) be the pixel position / coordinate within the sub-block to which the pixel belongs, and w and h be the width and height of the sub-block (e.g., for a 4×4 sub-block, w = h = 4, for an 8×8 sub-block, w = h = 8). The horizontal offset Δx and vertical offset Δy (Δx and Δy are defined in equation (5)) for each pixel (i, j) can be derived as follows, where i = 0…(w-1) and j = 0…(h-1).

[0116] In one example, if the sub-block MV is derived from the position at the center of the sub-block at an integer position, then Δx and Δy can be calculated using the equation shown below:

[0117]

[0118] Alternatively, if the sub-block MV is derived from the position at the center of the sub-block at the fractional position, then Δx and Δy can be calculated using the equation shown below:

[0119]

[0120] In another example, if the sub-block MV is derived from the position at the top left corner of the sub-block, then Δx and Δy can be calculated using the equation shown below:

[0121]

[0122] In another example, if the sub-block MV is derived from the position at the top right corner within the sub-block, then Δx and Δy can be calculated using the equation shown below:

[0123]

[0124] Alternatively, if the sub-block MV is derived from the position at the top right corner outside the sub-block, then Δx and Δy can be calculated using the equation shown below:

[0125]

[0126] In another example, if the sub-block MV is derived from the position at the bottom left corner within the sub-block, then Δx and Δy can be calculated using the equation shown below:

[0127]

[0128] Alternatively, if the sub-block MV is derived from the position at the bottom left corner outside the sub-block, then Δx and Δy can be calculated as:

[0129]

[0130] In another example, Δv(i,j) can be calculated using equation (5), where Δx and Δy are the horizontal and vertical offsets from the sample location (i,j) to the pilot sample location of the subblock to which the sample belongs. The pilot sample location refers to the sample location within a subblock, which is used to derive the MV of the subblock-based prediction samples used to generate the subblock. In one example, the values ​​of Δx and Δy are derived based on the location of the subblock within the current CU as follows.

[0131] Figure 9 An example of a sub-block within an affine CU according to some embodiments of this disclosure is shown. For the upper left sub-block, i.e. Figure 9 In the sub-block A, Δx = i, Δy = j. For the upper right sub-block, i.e. Figure 9 For sub-block B, Δx = (i - w + 1), Δy = j. For the lower left sub-block, i.e. Figure 9 For sub-block C, when applying the 6-parameter affine model, Δx = i, Δy = (j - h + 1); and when applying the 4-parameter affine model, Δx = (i - (w >> 1) - 0.5), Δy = (j - (h >> 1) - 0.5). For other sub-blocks, Δx = (i - (w >> 1) - 0.5), Δy = (j - (h >> 1) - 0.5).

[0132] Once the horizontal offset Δx and the vertical offset Δy have been calculated, Δv(i,j) can be derived using the following equation (11):

[0133]

[0134] Where c, d, e, and f are affine model parameters, which are known because the current block is an affine mode coded block. Equation (11) is similar to equation (5) of the PROF tool in the VVC standard.

[0135] In the fourth step, the predicted refinement value is calculated using equation (4).

[0136] In the fifth step, the prediction refinement is added to the sub-block prediction I(i,j). The final prediction I'(i,j) is generated as shown in equation (6).

[0137] In this disclosure, the proposed AMPR workflow can be applied to the luminance component and / or chrominance component.

[0138] In one example, to achieve a good performance / complexity tradeoff, the proposed AMPR is only applied to the affine prediction samples for refining the luminance component, while the chrominance prediction samples are still generated based on the existing sub-block-based affine motion compensation.

[0139] In another example, to refine the alignment, both the luminance and chrominance components are refined using the proposed AMPR process. In this case, the sample-by-sample MV difference Δv(i,j) can be derived in a different manner.

[0140] In one example, when calculating the predicted refinement value in step four above, the sample-by-sample MV difference Δv(i,j) can always be derived only once based on the luma sub-block and then reused for the chroma component. In this case, the value of Δv(i,j) used by the chroma component can be scaled according to the sampling grid ratio between the co-located luma-coded block and the chroma-coded block. For example, for 4:2:0 video, the value of the reused Δv(i,j) can be halved before being used by the chroma component, while for 4:4:4 video, the same value of the reused Δv(i,j) can be used by the chroma component. For 4:2:2 video, the horizontal offset of Δv(i,j) can be halved before being used by the chroma component, while the vertical offset of Δv(i,j) can remain unchanged.

[0141] In another example, the sample-by-sample MV difference Δv(i,j) can be derived separately for the luminance and chrominance components, where the derivation process can be the same as the third step described above.

[0142] In another example, an adaptive switching is proposed between a method that repeatedly applies the luma motion refinement Δv(i,j) to chroma and a method that separately derives the luma and chroma motion refinements based on the chroma sample format of the input video. For example, for 4:2:0 and 4:2:2 videos, assuming the sampling grids of the luma and chroma components are not aligned, separate extraction of the motion refinement for the luma and chroma components can be applied. On the other hand, when the input video is in a 4:4:4 chroma sample format, it only needs to derive the motion refinement once (i.e., for luma) and then repeat it for the other two color components because the sampling grids of the three color components are perfectly aligned.

[0143] In another example, a flag is signaled to indicate whether AMPR is applied to the chroma components at various codec levels (e.g., sequence level, picture level, stripe level, etc.). Furthermore, if the aforementioned enable / disable flag is true, another flag can be signaled from the encoder to the decoder to indicate whether the chroma motion refinement is recalculated from the corresponding control point motion vector or directly borrowed from the corresponding motion refinement of the luminance component.

[0144] AMPR's Alternative Workflow

[0145] Another alternative implementation of AMPR approximates the product based on the optical flow equation through a filtering process. Specifically, it proposes replacing the product of the gradient value and the motion vector difference at each sample location with a filtering process applied to conventional sub-block-based affine motion prediction. This can be formulated as follows:

[0146]

[0147] Where P AFF (x, y) are motion compensation prediction samples based on sub-blocks, f k These are filter coefficients, and P AMPR (i, j) are filtered affine prediction samples. In practice, different numbers of filter taps and filter shapes can be applied to achieve different trade-offs between complexity and encoding / decoding performance.

[0148] In one or more examples, the filtering process can be performed by a cross-shaped filter, which may also be called a diamond filter. For example, a diamond filter can be a combination of vertical and horizontal shapes of a 3-tap filter [-1, 1, 1] or a 5-tap filter [-1, -2, 4, 2, 1], such as... Figures 10A to 10B As shown. The cross-shaped filter can be an approximation of the gradient calculation process described in the previous optical flow-based refinement process.

[0149] To capture the motion vector (MV) difference Δv(i,j) between the MV of each pixel and the MV of the sub-block to which that pixel belongs, the filter coefficients in the selected diamond filter can be calculated based on the values ​​of Δv(i,j) in the horizontal and vertical directions. In other words, a scaled diamond filter can be used to compensate for the motion vector difference at each sample location. Figures 11A to 11B Examples of diamond filters scaled by the difference in MV in the horizontal and vertical directions are shown according to some embodiments of this disclosure. Figure 11A A diamond filter with 5 taps scaling is shown. Figure 11B A diamond filter with 9 taps scaling is shown.

[0150] In another example, the filtering process can be performed by a square filter. For instance, the square filter can be a 3×3 or 5×5 shaped filter, where the importance of each coefficient of the square filter can depend on the distance between the position of each coefficient and the center of the filter, meaning the center coefficient can have a maximum value in the filter. Similar to the diamond-shaped filter described above, the filter coefficients in the selected square filter can be scaled by the values ​​of Δv(i,j) in both the horizontal and vertical directions.

[0151] Once a specific type of filter is selected (e.g., rhombus or square shape), the corresponding scaled filter is computed at each sample location. The scaling value is the difference Δv(i,j) in motion vectors (MV) between the MV of each pixel and the MV of the sub-block to which that pixel belongs at each sample location. The computation process is the same as AMPR's optical flow-based implementation, where the value of Δv(i,j) is determined based on whether the associated sub-block MV is derived from the position at the center of the sub-block at an integer location, from the position at the top left corner of the sub-block, from the position at the top right corner of the sub-block, or from the position at the bottom left corner of the sub-block.

[0152] In a specific example, when a 3-tap cross-shaped filter is applied to the proposed scheme, the corresponding filtered affine prediction samples can be calculated as follows:

[0153] P AMPR (i, j) = (M·P) AFF (i,j)+N·(P) AFF (i, j+1)·Δ x (i, j+1)-P AFF (i, j-1)·Δxi, j-1+N·Δy·PAFFi+1, j·Δyi+1, j-PAFFi-1, j·Δyi-1, j÷M (13)

[0154] Where M and N are the initialization coefficients of the constant values, and Δ x and Δ y It is a scaling factor that can be used to adjust the importance of neighboring samples to a filtered sample at a current location. In a particular example, a setting of M=2N is proposed, for example, M=16 and N=8.

[0155] Once the scaled filter coefficients are calculated, filtering can be performed on regular sub-block-based affine motion prediction samples. For adjacent sample locations outside the current block / sub-block, padding may be required. In one or more examples, the padding sample values ​​may be copied from the nearest reference sample at an integer position. In another example, the padding sample values ​​may be repeating values ​​of the reference sample at an integer position used by the current block / sub-block. In one example, the integer sample closest to the current boundary sample (which can be a fraction) of the current CU is used to pad the extended region of the current CU. In another example, integer samples with positions smaller than the corresponding boundary sample of the current CU (i.e., floor()) are used to pad samples in the extended region of the current CU.

[0156] Selective enabling of AMPR

[0157] The refinement of predictions derived by applying AMPR may not always be beneficial or / and necessary. According to equation (4), the importance of the derived ΔI(i,j) is determined by the accuracy and magnitude of the derived Δv(i,j) and g(i,j).

[0158] In some examples, AMPR operation can be conditionally applied based on specific conditions. This can be achieved by signaling a flag for each block to indicate whether AMPR mode is applied. Alternatively, the same conditions can be used to enable AMPR operation on both the encoder and decoder sides without requiring additional signaling.

[0159] The motivation for this conditional application of the AMPR operation is that if the CPMV of the CU is inaccurate, or the derived affine model (e.g., a 2-parameter, 4-parameter, or 6-parameter affine mode) is inaccurate, then the Δv(i,j) subsequently derived via equation (11) may also be inaccurate. In such cases, the AMPR operation may not help or even harm the encoding / decoding performance, so it is better to skip the AMPR operation for blocks. Another motivation for this conditional application of the AMPR operation is that in some cases, the benefit of applying AMPR may be negligible, and from a computational complexity perspective, it is better to disable the operation.

[0160] In one or more examples, AMPR operations can be applied depending on whether the CMPV is explicitly signaled. In affine merging mode where the CMPV is not explicitly signaled but implicitly derived from spatially adjacent CUs, AMPR can be skipped for the current CU because the CMPV in this mode may be inaccurate.

[0161] In another example, AMPR can be skipped if, for example, the derived Δv(i,j) and / or g(i,j) are smaller compared to some predefined or dynamically determined threshold. Such thresholds can be determined based on various factors, such as CU aspect ratio and / or sub-block size. Such examples can be implemented in different ways as described below.

[0162] In one example, if the absolute value of Δv(i,j) derived for all pixels within a sub-block is less than a threshold, the AMPR for that sub-block can be skipped. This condition can be implemented in various ways. For example, the check for the absolute value of Δv(i,j) derived for all pixels can be simplified by examining only the four corners of the current sub-block, where the maximum absolute value of Δv(i,j) derived for all pixels within the sub-block can be found as shown in the following equation (14):

[0163]

[0164] The pixel position (i, j) can be any pixel coordinate in the sub-block, or it can come from the four corners (0, 0), (w-1, 0), (0, h-1), (w-1, h-1).

[0165] In another example, the maximum absolute value of all Δv(i,j) can be calculated using the following equation:

[0166]

[0167] Where the sample point position (i, j) is the sub-block in CU excluding the upper left sub-block (i.e., Figure 9 Sub-block A in the middle), the upper right sub-block (i.e., Figure 9 Sub-block B in the middle) and the sub-block to the lower left (i.e., Figure 9 The four corners of the sub-blocks outside of sub-block C). The coordinates of the four corner pixels within the sub-block are: (0, 0), (w-1, 0), (0, h-1), (w-1, h-1). |x| is a function that takes the absolute value of x.

[0168] In another example, the check of Δv(i,j) derived for all pixels within a sub-block can be performed together as in equation (15) or separately in the horizontal and vertical directions as in equation (16).

[0169] if Or if Then ΔI(i,j)=0 (15)

[0170]

[0171] In equations (15) and (16) above, different similar forms of the expression for ΔI(i,j) represent different simplification methods. For example, in ΔI(i,j) = g y (i, j)*Δv y In the case of (i, j), g can be skipped. x and Δv x The calculation.

[0172] In another example, the examination of the derived Δv(i,j) can be combined with a non-reduced AMPR operation. In this case, equation (14) can be combined with equation (4), and the predicted refined value can then be calculated using the following equation:

[0173]

[0174] In some examples, threshv x or threshv y They can have different or the same values.

[0175] In some examples, when deriving Δv(i,j) for a sub-block, the threshold v may be determined depending on which position is used to derive the sub-block MV. x and threshv y The value of threshv. In other words, if the MV of two sub-blocks is derived using different positions, then the threshv can be determined for these two sub-blocks. x and threshv y A pair of identical values ​​or a pair of different values. For example, for a sub-block whose sub-block-level MV is derived based on the sub-block center, its threshv x and threshv y A pair of values ​​can be used to derive the threshold of the sub-block-level MV based on the position of the top-left corner of the sub-block. x and threshv y A pair of values ​​may be the same or different.

[0176] In some examples, threshv x and threshv y The value can be defined in pixels within the range of [1 / 32, 1 / 16]. For example, values ​​of (1 / 16)×(10 / 16), (1 / 16)×(12 / 16), or (1 / 16)×(14 / 16) can be used as thresholds. In this case, the threshold is a floating-point number in 1 / 16 of a pixel, such as 0.625, 0.75, or 0.875 in 1 / 16 of a pixel.

[0177] In some examples, threshv can be defined based on the image type. x and threshv y The value of the threshold. For low-latency images, the derived affine model parameters can have smaller amplitudes than for other non-low-latency images, because low-latency images tend to have smaller and / or smoother motion, so smaller values ​​are preferred for those thresholds.

[0178] In some examples, threshv x and threshv y The values ​​can be the same, regardless of the image type.

[0179] In some examples, AMPR can be skipped for a sub-block if the absolute values ​​of most of the derived g(i,j) for all pixels within the sub-block are less than a threshold. An example of this approach is when the sub-block contains a smooth surface that can be composed of flat textures (e.g., with little or no high-frequency detail).

[0180] In some examples, the importance of Δv(i,j) and g(i,j) can be considered together or used in a mixed manner to determine whether AMPR should be skipped for the current sub-block or CU.

[0181] In the case of approximating AMPR by using the above filtering process, selective activation of AMPR can also be performed. For example, based on equation (15), if and / or Less than the predefined threshold threshv x and threshv y If this is the case, the corresponding scaled coefficients in the selected filter can become 0, meaning the filtering process can be simplified from a 2D filtering process to a one-dimensional (1D) filtering process. Figure 10A Taking the 5-tap filter in the example, when only Less than threshv x When only a 1D filter [1, 2, 1] is applied in the vertical direction. Less than threshv y When, only a 1D filter [1, 2, 1] is applied in the horizontal direction. and None of them are less than the corresponding threshold (i.e., threshv) x and threshv y When ), 2D filters are applied in both the horizontal and vertical directions. and When all values ​​are less than the corresponding threshold, no filter is applied at all; that is, AMPR is not applied.

[0182] Encoder-side optimization

[0183] In the AVS3 standard, the affine UMVE mode is computationally intensive for the encoder because it involves selecting the optimal distance index for each merging mode candidate. When calculating the sum of absolute transform differences (SATD) cost for each candidate distance index, conventional affine motion compensation is always applied. Applying AMPR on top of affine motion compensation can significantly increase the computational load.

[0184] In one example, on the encoder side, the AMPR operation is skipped during SATD-based cost computation for affine UMVE patterns. Experiments show that while the optimal index is selected based on the best SATD cost, whether or not AMPR is applied during SATD computation generally does not change the ranking of the best SATD costs. Therefore, using the proposed method, enabling AMPR mode does not lead to significant encoder complexity for affine UMVE patterns.

[0185] Motion estimation is another major overhead on the encoder side. In another example, the AMPR process can be skipped depending on specific conditions. These conditions indicate that the optimal encoding mode for the CU is unlikely to be an affine mode after the mode selection process.

[0186] An example of this condition is whether the current CU has a parent CU that has been determined to be encoded and decoded using either explicit affine mode or affine merge mode. This is due to the strong correlation between the encoding / decoding mode selection of a CU and its parent CU, and if the above condition is true, the optimal encoding / decoding mode for the current CU is also likely to be explicit affine mode.

[0187] Another exemplary condition for enabling AMPR is whether the parent CU of the current CU is determined to be performing inter-frame prediction using explicit affine mode. If true, AMPR is applied during affine motion estimation of the current CU; otherwise, AMPR is skipped during affine motion estimation of the current CU.

[0188] Compared to large block-size CUs such as 64×64 CUs, small-size CUs such as 16×16 CUs have a significantly higher average per-pixel computational cost when applying AMPR. To effectively save computational complexity, in another example of this disclosure, AMPR can be skipped for small-size CUs during the motion estimation process. The size of the CU can be defined as the total number of pixels. A pixel number threshold can be defined, such as 16×16, 16×32, or 32×32, and for blocks smaller than the defined threshold, AMPR can be skipped during the affine motion estimation process for that block.

[0189] When AMPR is approximated using the filtering process described above, encoder-side optimizations can also be performed. For example, the filtering process may not be performed for affine UMVE mode. Another encoder optimization for enabling filter-based AMPR is whether the parent CU of the current CU is determined to be performing inter-frame prediction using explicit affine mode. If true, AMPR filtering is applied during affine motion estimation of the current CU. Otherwise, the filtering process is skipped during affine motion estimation of the current CU.

[0190] Figure 12 This is a block diagram illustrating an apparatus for predicting sample points at pixel locations in a sub-block using AMPR according to some embodiments of the present disclosure. The apparatus 1200 may be a terminal, such as a mobile phone, tablet computer, digital broadcasting terminal, tablet device, or personal digital assistant.

[0191] like Figure 12As shown, device 1200 may include one or more of the following components: processing component 1202, memory 1204, power supply component 1206, multimedia component 1208, audio component 1210, input / output (I / O) interface 1212, sensor component 1214, and communication component 1216.

[0192] Processing component 1202 typically controls the overall operation of device 1200, such as operations related to display, telephone calls, data communication, camera operation, and recording. Processing component 1202 may include one or more processors 1220 for executing instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1202 may include one or more modules for facilitating interaction between processing component 1202 and other components. For example, processing component 1202 may include a multimedia module for facilitating interaction between multimedia component 1208 and processing component 1202.

[0193] Memory 1204 is configured to store different types of data to support the operation of device 1200. Examples of such data include instructions for any application or method operating on device 1200, contact data, phonebook data, messages, pictures, videos, etc. Memory 1204 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, and memory 1204 may be static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0194] Power supply assembly 1206 supplies power to various components of device 1200. Power supply assembly 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 1200.

[0195] Multimedia component 1208 includes a screen that provides an output interface between device 1200 and the user. In some examples, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen that receives input signals from the user. The touch panel may include one or more touch sensors for sensing touches, swipes, and gestures on the touch panel. The touch sensors may not only sense the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some examples, multimedia component 1208 may include a front-facing camera and / or a rear-facing camera. When device 1200 is in an operating mode (such as a shooting mode or a video mode), the front-facing camera and / or the rear-facing camera may receive external multimedia data.

[0196] Audio component 1210 is configured to output and / or input audio signals. For example, audio component 1210 includes a microphone (MIC). When device 1200 is in an operating mode (such as call mode, recording mode, and voice recognition mode), the microphone is configured to receive external audio signals. The received audio signals may be further stored in memory 1204 or transmitted via communication component 1216. In some examples, audio component 1210 also includes a speaker for outputting audio signals.

[0197] I / O interface 1212 provides an interface between processing component 1202 and peripheral interface modules. These peripheral interface modules can be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0198] Sensor assembly 1214 includes one or more sensors for providing state assessment in various aspects of device 1200. For example, sensor assembly 1214 may detect the on / off state of device 1200 and the relative position of components. For example, components may be the display and keyboard of device 1200. Sensor assembly 1214 may also detect changes in position of device 1200 or its components, the presence or absence of user contact on device 1200, the orientation or acceleration / deceleration of device 1200, and temperature changes of device 1200. Sensor assembly 1214 may include a proximity sensor configured to detect the presence of nearby objects without any physical touch. Sensor assembly 1214 may also include optical sensors, such as CMOS or CCD image sensors used in imaging applications. In some examples, sensor assembly 1214 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0199] Communication component 1216 is configured to facilitate wired or wireless communication between device 1200 and other devices. Device 1200 may access a wireless network based on communication standards such as WiFi, 4G, or combinations thereof. In the example, communication component 1216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In the example, communication component 1216 may also include a near-field communication (NFC) module for facilitating short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0200] In the example, device 1200 may be implemented by one or more of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components to perform the methods described above.

[0201] Non-transitory computer-readable storage media can be, for example, hard disk drives (HDDs), solid-state drives (SSDs), flash memory, hybrid drives or solid-state hybrid drives (SSHDs), read-only memory (ROMs), optical disc read-only memory (CD-ROMs), magnetic tapes, floppy disks, etc.

[0202] Figure 13 This is a flowchart illustrating an exemplary process of predicting sample points at pixel locations in a sub-block using AMPR according to some embodiments of the present disclosure.

[0203] In step 1302, the processor 1220 generates multiple affine motion compensation predictions at pixel locations and multiple adjacent pixel locations within the sub-block.

[0204] In step 1304, the processor 1220 uses a filter with a predetermined shape to obtain a refined prediction of the sample at the pixel location based on multiple affine motion compensation predictions.

[0205] In some examples, processor 1220 can generate multiple affine motion compensation predictions by performing sub-block-based affine motion compensation on video images comprising multiple sub-blocks.

[0206] In some examples, the predetermined shape can cover a pixel location and multiple adjacent pixel locations.

[0207] In some examples, MV difference can include horizontal MV difference and vertical MV difference.

[0208] Figure 14 This is a flowchart illustrating steps in an exemplary process of predicting sample points at pixel locations in a sub-block using AMPR according to some embodiments of the present disclosure. Figure 13 Step 1304 in the process can be achieved through Figure 14 The steps shown are used to achieve this.

[0209] In step 1401, processor 1220 initializes the filter coefficients to a set of constant values ​​to approximate the gradient calculation process described in the previous optical flow-based refinement process. As shown in equation (13), M and N are the initialization coefficients of constant values.

[0210] In step 1403, processor 1220 scales the initialized coefficients using one or more scaling factors to obtain a scaled filter. As shown in equation (13), Δ x and Δ y It can be a scaling factor.

[0211] In step 1405, the processor 1220 obtains a refined prediction at the pixel location using a scaled filter with a predetermined shape based on multiple affine motion compensation predictions.

[0212] In some examples, processor 1220 may determine one or more scaling factors based on the MV difference at each of a plurality of adjacent pixel locations.

[0213] In some examples, processor 1220 may determine one or more scaling factors based on the horizontal, vertical, or horizontal and vertical values ​​of the MV difference at each of a plurality of adjacent pixel locations.

[0214] In some examples, the predetermined shape is a cross or a square. For example, the predetermined shape is a 3×3 square.

[0215] In some examples, each coefficient in the filter is determined based on the distance between the coefficient's location and the center of the filter.

[0216] In some examples, the coefficients at the center of the filter have a maximum value.

[0217] In some examples, processor 1220 may copy fill samples to the adjacent pixel location in response to determining that one of a plurality of adjacent pixel locations is outside the sub-block.

[0218] In some examples, processor 1220 may fill adjacent pixel locations with reference neighbor samples from a reference image, wherein the reference neighbor sample is the reference sample at the integer position closest to the predicted sample corresponding to the pixel location.

[0219] In some examples, processor 1220 can fill adjacent pixel locations with samples at integer positions that are closest to the sample points located on the boundaries of the sub-blocks.

[0220] In some examples, an apparatus is provided for predicting sample points at pixel locations in a sub-block by implementing AMPR. The apparatus includes one or more processors 1220; and a memory 1204 configured to store instructions executable by the one or more processors; wherein the processors, when executing the instructions, are configured to perform actions such as... Figures 13 to 14 The method shown.

[0221] In some other examples, a non-transitory computer-readable storage medium 1204 is provided, having instructions stored therein. When executed by one or more processors 1220, the instructions cause the processor to perform actions such as Figures 13 to 14 The method shown.

[0222] The description in this disclosure has been presented for illustrative purposes and is not intended to be exhaustive or limited thereto. Many modifications, variations, and alternative embodiments will be apparent to those skilled in the art from the teachings presented in the foregoing description and the associated drawings.

[0223] The examples were chosen and described to explain the principles of this disclosure and to enable others skilled in the art to understand the various embodiments of this disclosure and, best of all, to utilize the basic principles and the various embodiments with modifications suitable for the intended particular purpose. Therefore, it will be understood that the scope of this disclosure is not limited to the specific examples of the disclosed embodiments, and that modifications and other embodiments are intended to be included within the scope of this disclosure.

Claims

1. A method for encoding samples at pixel locations in a sub-block, comprising: Multiple affine motion compensation predictions are generated at the pixel location and multiple adjacent pixel locations in the sub-block; Determine the MV difference between the motion vector MV of the corresponding neighboring pixel at at least one of the plurality of neighboring pixel positions and the MV of the sub-block; The coefficients of a filter with a predetermined shape are determined based on the MV difference; The filter is used to obtain a refined prediction of the sample point at the pixel location based on the multiple affine motion compensation predictions.

2. The method according to claim 1, further comprising: The plurality of affine motion compensation predictions are generated by performing sub-block-based affine motion compensation on video images comprising multiple sub-blocks.

3. The method of claim 1, wherein the step of using the filter to obtain a refined prediction of the sample point at the pixel location based on the plurality of affine motion compensation predictions comprises: The filter is used to filter each of the plurality of affine motion compensation predictions.

4. The method according to claim 1, further comprising: The initial coefficients of the filter are obtained by initializing the coefficients of the filter to a set of constant values.

5. The method according to claim 4, further comprising: The initial coefficients of the filter are scaled using one or more scaling factors; The step of using the filter to obtain a refined prediction of the sample point at the pixel location based on the multiple affine motion compensation predictions includes: At the pixel location, the refined prediction is obtained using scaled coefficients of the filter based on the plurality of affine motion compensation predictions.

6. The method according to claim 5, further comprising: At least one scaling factor among the one or more scaling factors is determined based on the MV difference at at least one of the plurality of adjacent pixel locations.

7. The method according to claim 6, further comprising: Based on the horizontal and vertical values ​​of the MV difference, or the horizontal and vertical values, determine at least one scaling factor among the one or more scaling factors.

8. The method according to claim 1, wherein the predetermined shape is cross-shaped.

9. The method according to claim 1, wherein the predetermined shape is a square shape.

10. The method of claim 4, wherein each initialization coefficient in the filter is determined based on the distance between the position of the initialization coefficient in the filter and the center of the filter.

11. The method of claim 10, wherein the initialization coefficient at the center of the filter has a maximum value.

12. The method of claim 1, wherein the step of generating multiple affine motion compensation predictions at the pixel location and multiple adjacent pixel locations in the sub-block comprises: For at least one of the plurality of adjacent pixel locations outside the sub-block, a fill sample is copied to the adjacent pixel location.

13. The method of claim 12, wherein the step of copying the fill sample to the adjacent pixel location comprises: The reference neighboring sample points in the reference image are filled into the adjacent pixel positions, wherein the reference neighboring sample point is the reference sample point at the integer position closest to the predicted sample point corresponding to the pixel position.

14. The method of claim 12, wherein the step of copying the fill sample to the adjacent pixel location comprises: Fill the adjacent pixel position with the sample point at the integer position closest to the sample point located on the boundary of the sub-block.

15. An apparatus for encoding samples at pixel locations in a sub-block, comprising: One or more processors; as well as The memory is configured to store instructions executable by the one or more processors. The one or more processors wherein the instructions are executed are configured to perform the method according to any one of claims 1 to 14.

16. A non-transitory computer-readable storage medium for encoding samples at pixel locations in a sub-block, storing computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform the method according to any one of claims 1 to 14.

17. A method for storing a bit stream, comprising: A bit stream is generated by performing the method as described in any one of claims 1 to 14; and Store the bit stream.

Images