Method and apparatus for affine motion compensation prediction refinement

Abstract

Methods and apparatuses for affine motion compensation prediction refinement (AMPR) are provided. The method includes generating a sub-block prediction at a pixel position of a sub-block by performing sub-block based affine motion compensation on a video picture including a plurality of sub-blocks, obtaining a horizontal spatial gradient and a vertical spatial gradient for the sub-block prediction at the pixel position using an interpolation filter, and obtaining a motion vector (MV) difference between a first MV and a second MV at the pixel position based on the pixel position relative to a position within the sub-block. The first MV is a MV of a pixel located at the pixel position, and the second MV is a MV of the sub-block.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Provisional Application No. 62 / 992,897, filed March 20, 2020, entitled “Refinement of Affine Motion Compensation Prediction” and Provisional Application No. 62 / 993,654, filed March 23, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to video encoding and decoding and compression. More specifically, this application relates to, but is not limited to, methods and apparatus for affine motion compensation prediction refinement (AMPR) in video encoding and decoding. Background Technology

[0004] Various video codec technologies can be used to compress video data. Video codecs are performed according to one or more video codec standards. For example, some well-known video codec standards today include Universal Video Codec (VVC), High Efficiency Video Codec (HEVC, also known as H.265 or MPEG-H Part 2), and Advanced Video Codec (AVC, also known as H.264 or MPEG-4 Part 10), jointly developed by ISO / IEC MPEG and ITU-T VECG. AOMedia Video1 (AV1) was developed by the Alliance for Open Media (AOM) as the successor to its previous standard VP9. Audio and Video Codecs (AVS) are another series of video compression standards developed by the Chinese Audio and Video Codec Standards Working Group, referring to digital audio and digital video compression standards. Most existing video codec standards are built on the well-known hybrid video codec framework, which uses block-based prediction methods (e.g., inter-frame prediction, intra-frame prediction) to reduce redundancy present in video images or sequences, and uses transform codecs to compress the energy of prediction errors. A key goal of video encoding and decoding technology is to compress video data into a form that uses a lower bit rate, while avoiding or minimizing the degradation of video quality.

[0005] The first generation of AVS standards included the Chinese national standards "Information Technology, Advanced Audio and Video Coding, Part 2: Video" (referred to as AVS1) and "Information Technology, Advanced Audio and Video Coding, Part 16: Broadcast Television Video" (referred to as AVS+). Compared to the MPEG-2 standard, it offered approximately 50% bitrate savings at the same perceived quality. The video portion of the AVS1 standard was released as a Chinese national standard in February 2006. The second generation of AVS standards included the Chinese national standard series "Information Technology, High-Efficiency Multimedia Coding" (referred to as AVS2), primarily targeting the transmission of ultra-high-definition television programs. AVS2's coding and decoding efficiency was twice that of AVS+. In May 2016, AVS2 was released as a Chinese national standard. Simultaneously, the video portion of the AVS2 standard was submitted by the Institute of Electrical and Electronics Engineers (IEEE) as an international standard for applications. The AVS3 standard is a new generation of video coding and decoding standards for UHD video applications, aiming to surpass the coding and decoding efficiency of the latest international standard, HEVC. In March 2019, at the 68th AVS meeting, the AVS3-P2 baseline was completed, offering approximately 30% bit rate savings compared to the HEVC standard. Currently, a reference software called the High Performance Model (HPM) exists, maintained by the AVS group to demonstrate a reference implementation of the AVS3 standard. Summary of the Invention

[0006] This disclosure provides examples of techniques related to AMPR for the AVS3 standard.

[0007] According to a first aspect of this disclosure, a method for AMPR is provided. The method includes generating a sub-block prediction at a pixel location within a sub-block by performing sub-block-based affine motion compensation on a video image comprising multiple sub-blocks. Additionally, the method includes obtaining a horizontal spatial gradient and a vertical spatial gradient for the sub-block prediction at the pixel location using an interpolation filter. Furthermore, the method includes obtaining a MV difference between a first motion vector (MV) and a second MV at the pixel location and based on the pixel location relative to the position within the sub-block. The first MV is the MV of the pixel located at that pixel location, and the second MV is the MV of the sub-block.

[0008] According to a second aspect of this disclosure, an apparatus for AMPR is provided. The apparatus includes one or more processors and a memory configured to store instructions executable by the one or more processors. Upon execution of the instructions, the one or more processors are configured to: generate a sub-block prediction at a pixel location within a sub-block by performing sub-block-based affine motion compensation on a video image comprising a plurality of sub-blocks; obtain a horizontal spatial gradient and a vertical spatial gradient for the sub-block prediction at the pixel location using an interpolation filter; and obtain a MV difference between a first motion vector (MV) and a second MV at the pixel location and based on the pixel location relative to the position within the sub-block. The first MV is the MV of the pixel located at the pixel location, and the second MV is the MV of the sub-block.

[0009] According to a third aspect of this disclosure, a non-transitory computer-readable storage medium for AMPR is provided, the non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform actions, the actions including: generating a sub-block prediction at a pixel location within a sub-block by performing sub-block-based affine motion compensation on a video image comprising a plurality of sub-blocks; obtaining a horizontal spatial gradient and a vertical spatial gradient for the sub-block prediction at the pixel location using an interpolation filter; and obtaining a MV difference between a first motion vector (MV) and a second MV at the pixel location and based on the pixel location relative to the location within the sub-block. The first MV is the MV of the pixel located at the pixel location, and the second MV is the MV of the sub-block. Attached Figure Description

[0010] A more detailed description of the examples of this disclosure will be presented with reference to the specific examples shown in the accompanying drawings. Given that these drawings depict only a few examples and are therefore not intended to limit the scope, the examples will be described and explained using additional features and details through the use of the drawings.

[0011] Figure 1 This is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

[0012] Figure 2 This is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

[0013] Figures 3A to 3E This is a schematic diagram illustrating a multi-type tree partitioning pattern according to some embodiments of the present disclosure.

[0014] Figure 4 This is a schematic diagram illustrating an example of a bidirectional optical flow (BIO) model according to some implementations of this disclosure.

[0015] Figures 5A to 5B This is a schematic diagram illustrating an example of a 4-parameter affine model according to some embodiments of the present disclosure.

[0016] Figure 6 This is a schematic diagram illustrating an example of a 6-parameter affine model according to some embodiments of the present disclosure.

[0017] Figure 7 The Predictive Refinement (PROF) process for affine modes using optical flow is illustrated according to some embodiments of the present disclosure.

[0018] Figure 8 Examples of calculating the horizontal and vertical offsets from the sample point location to a specific location of the sub-block from which the sub-block MV is derived, according to some embodiments of this disclosure, are shown.

[0019] Figure 9 An example of a sub-block within an affine CU is shown according to some embodiments of this disclosure.

[0020] Figure 10 This is a block diagram illustrating an exemplary apparatus for AMPR according to some embodiments of the present disclosure.

[0021] Figure 11 This is a flowchart illustrating an exemplary method for AMPR according to some embodiments of the present disclosure. Detailed Implementation

[0022] Referring now to the detailed description, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives may be used. For example, it will be apparent to those skilled in the art that the subject matter presented herein can be implemented on many types of electronic devices with digital video capabilities.

[0023] Throughout this specification, references to "an embodiment," "an embodiment," "an example," "some embodiments," "some examples," or similar language indicate that a particular feature, structure, or characteristic described is included in at least one embodiment or example. Unless otherwise expressly stated, the features, structures, elements, or characteristics described in connection with one or more embodiments also apply to other embodiments.

[0024] Throughout this disclosure, unless otherwise expressly stated, the terms “first,” “second,” “third,” etc., are used only to refer to related elements (e.g., equipment, components, compositions, steps, etc.) and do not indicate any spatial or temporal order. For example, “first equipment” and “second equipment” can refer to two separately formed devices, or two parts, components, or operating states of the same device, and can be named arbitrarily.

[0025] The terms "module," "submodule," "circuit," "subcircuit," "circuit system," "subcircuit system," "unit," or "subunit" may include memory (shared, dedicated, or combined) storing code or instructions executable by one or more processors. A module may include one or more circuits, with or without stored code or instructions. A module or circuit may include one or more components, directly or indirectly connected. These components may or may not be physically attached to each other or positioned adjacent to each other.

[0026] As used herein, depending on the context, the terms "if" or "when..." may be understood to mean "at..." or "in response to...". If these terms appear in the claims, they do not necessarily indicate that the relevant limitation or feature is conditional or optional. For example, a method may include the steps of: i) performing a function or action X' when or if condition X exists, and ii) performing a function or action Y' when or if condition Y exists. The method may be implemented with both the ability to perform function or action X' and the ability to perform function or action Y'. Thus, both function X' and function Y' may be performed at different times during multiple executions of the method.

[0027] Units or modules can be implemented purely by software, purely by hardware, or by a combination of hardware and software. In a purely software implementation, for example, a unit or module may include functionally related code blocks or software components that are directly or indirectly linked together to perform a specific function.

[0028] Figure 1 A block diagram of an exemplary block-based hybrid video encoder 100, which can be used in conjunction with many video codec standards that employ block-based processing, is shown. In encoder 100, video frames are segmented into multiple video blocks for processing. For each given video block, a prediction is formed based on either an inter-frame prediction method or an intra-frame prediction method. In inter-frame prediction, one or more prediction factors are formed based on pixels from previously reconstructed frames, through motion estimation and motion compensation. In intra-frame prediction, prediction factors are formed based on reconstructed pixels in the current frame. Through mode decision, the optimal prediction factor can be selected to predict the current block.

[0029] Intra-frame prediction (also known as "spatial prediction") uses pixels from samples (called reference samples) of already encoded and decoded neighboring blocks in the same video frame and / or strip to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal.

[0030] Inter-frame prediction (also known as "temporal prediction") uses reconstructed pixels from already encoded and decoded video frames to predict the current video block. Temporal prediction reduces the inherent temporal redundancy in the video signal. A temporal prediction signal for a given coding unit (CU) or coding block is typically transmitted in signaling form by one or more motion vectors (MVs) indicating the amount and direction of motion between the current CU and its temporal reference. Additionally, if multiple reference frames are supported, a separate reference frame index is sent, which is used to identify which reference frame in the reference frame repository the temporal prediction signal originates from.

[0031] After performing spatial and / or temporal prediction, the intra / inter-frame mode decision circuit 121 in encoder 100 selects the optimal prediction mode, for example, based on a rate-distortion optimization method. The block prediction factor 120 is then subtracted from the current video block; and the resulting prediction residual is decorrelated using transform circuit 102 and quantization circuit 104. The resulting quantized residual coefficients are dequantized by inverse quantization circuit 116 and inverse transformed by inverse transform circuit 118 to form the reconstruction residual, which is then added back to the prediction block to form the reconstructed signal of the CU. Furthermore, before placing the reconstructed CU into the reference image repository of image buffer 117 for encoding subsequent video blocks, loop filtering 115, such as a deblocking filter, sample adaptive offset (SAO), and / or adaptive loop filter (ALF), can be applied to the reconstructed CU. To form the output video bitstream 114, the encoding mode (inter-frame or intra-frame), prediction mode information, motion information, and quantized residual coefficients are all sent to entropy coding unit 106 for further compression and packing to form the bitstream.

[0032] For example, deblocking filters are available in current versions of AVC, HEVC, and VVC. In HEVC, an additional loop filter called Sample Adaptive Offset (SAO) is defined to further improve encoding and decoding efficiency. In the current version of the VVC standard, another loop filter called Adaptive Loop Filter (ALF) is under active investigation and is very likely to be included in the final standard.

[0033] These loop filter operations are optional. Performing these operations helps improve encoding / decoding efficiency and visual quality. They can also be turned off based on decisions made by encoder 100 to save computational complexity.

[0034] It should be noted that when these filter options are enabled in encoder 100, intra-frame prediction is typically based on unfiltered reconstructed pixels, while inter-frame prediction is based on filtered reconstructed pixels.

[0035] Figure 2 This is a block diagram illustrating an exemplary block-based video decoder 200 that can be used in conjunction with many video codec standards. The decoder 200 is similar to [the one residing in...]. Figure 1 The reconstruction-related part is located in the encoder 100. In the decoder 200, the input video bitstream 201 is first decoded by entropy decoding 202 to derive the quantized coefficient levels and prediction-related information. Then, the quantized coefficient levels are processed by inverse quantization 204 and inverse transform 206 to obtain the reconstructed prediction residuals. The block predictor mechanism implemented in the intra / inter-frame mode selector 212 is configured to perform intra-frame prediction 208 or motion compensation 210 based on the decoded prediction information. A set of unfiltered reconstructed pixels is obtained by summing the reconstructed prediction residuals from inverse transform 206 and the prediction output generated by the block predictor mechanism using summer 214.

[0036] Before storing the reconstructed blocks in image buffer 213, which serves as a reference image repository, the reconstructed blocks can be further passed through loop filter 209. The reconstructed video in image buffer 213 can be sent to drive a display device and to predict subsequent video blocks. With loop filter 209 open, filtering operations are performed on these reconstructed pixels to produce the final reconstructed video output 222.

[0037] The video encoding / decoding standards mentioned above (such as HEVC and AV3) are conceptually similar. For example, they all use a block-based hybrid video encoding / decoding framework. The following details some of the block segmentation schemes in these standards.

[0038] HEVC uses only quadtrees to partition blocks. The basic unit used for compression is called a coding tree unit (CTU). Each CTU can contain one coding unit (CU) or be recursively divided into four smaller CUs until a predefined minimum CU size is reached. Each CU (also called a leaf CU) contains one or more prediction unit (PU) and transform unit (TU) trees.

[0039] In AVS3, a coding tree unit (CTU) is divided into multiple CUs based on a quadtree / binary tree / extended quadtree to accommodate varying local characteristics. Furthermore, the concept of multiple partitioning unit types in HEVC is removed; that is, there is no distinction between CU, PU, ​​and TU in AVS3. Instead, each CU is always used as the basic unit for both prediction and transformation without further partitioning. In AVS3's tree partitioning structure, a CTU is first partitioned based on a quadtree structure. Then, each quadtree leaf node can be further partitioned based on binary tree and extended quadtree structures.

[0040] Figures 3A to 3E This is a schematic diagram illustrating multiple tree partitioning patterns according to some embodiments of the present disclosure. For example... Figures 3A to 3E As shown, there are five partitioning types in the multi-type tree structure: quadruple partition 301, vertical binary partition 302, horizontal binary partition 303, vertical extended quadruple partition 304, and horizontal extended quadruple partition 305.

[0041] In the current VVC and AVS3 standards, block-based motion compensation can be applied to achieve a trade-off between encoding / decoding efficiency, complexity, and memory access bandwidth. Average prediction accuracy is lower than pixel-based prediction because all pixels within a block or sub-block share the same block-level motion vector. To improve prediction accuracy at each pixel, PROF for affine modes is adopted as the encoding / decoding tool in the current VVC standard. A similar tool does not exist in AVS3.

[0042] Some examples in this disclosure provide alternative optical flow-based methods to improve affine mode longitude prediction, as described below.

[0043] Affine mode

[0044] In HEVC, only the translational motion model is applied to motion compensation prediction. However, in the real world, many types of motion exist, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC and AVS3, affine motion compensation prediction is applied by sending a flag to each inter-frame codec block to indicate whether a translational or affine motion model is applied to inter-frame prediction. In current VVC and AVS3 designs, two affine modes are supported for an affine codec block: a 4-parameter affine mode and a 6-parameter affine mode.

[0045] A 4-parameter affine model can have the following parameters: two parameters for translational motion in the horizontal and vertical directions, one parameter for scaling motion in both directions, and one parameter for rotational motion in both directions. In a 4-parameter affine model, the horizontal scaling parameter can be equal to the vertical scaling parameter, and the horizontal rotation parameter can be equal to the vertical rotation parameter. To achieve better reconciliation between the motion vectors and affine parameters, those affine parameters can be derived from the two MVs (also known as control point motion vectors (CPMVs)) located at the top left and top right corners of the current block.

[0046] Figures 5A to 5B This is a diagram illustrating an example of a 4-parameter affine model according to some embodiments of the present disclosure. For example... Figures 5A to 5B As shown, the affine motion field of a block is described by two control points MV(V0, V1). Based on the motion of the control points, the motion field (v0, V1) of a block that has been affinely encoded and decoded is... x ,v y It is described by the following equation (1):

[0047]

[0048] A 6-parameter affine model can have the following parameters: two parameters for translational motion in the horizontal and vertical directions, two parameters for scaling and rotational motion in the horizontal direction, and two additional parameters for scaling and rotational motion in the vertical direction. The 6-parameter affine motion model is encoded and decoded using three CPMVs.

[0049] Figure 6 This is a diagram illustrating an example of a 6-parameter affine model according to some embodiments of the present disclosure. For example... Figure 6 As shown, the three control points of a 6-parameter affine block are located at the top left, top right, and bottom left corners of the block. The motion at the top left control point is related to translational motion, the motion at the top right control point is related to rotational and scaling motion in the horizontal direction, and the motion at the bottom left control point is related to rotational and scaling motion in the vertical direction. Compared to the 4-parameter affine motion model, the horizontal rotational and scaling motions of the 6-parameter affine motion model can differ from the vertical rotational and scaling motions.

[0050] In some examples, when (V0, V1, V2) is Figure 6 If the top-left, top-right, and bottom-left corners of the current block are MV values, then each sub-block (v...) x ,v y The motion vector of ) is derived using the three MVs at the control points as shown in the following equation (2):

[0051]

[0052] Predictive refinement (PROF) using optical flow for affine modes

[0053] To improve the accuracy of affine motion compensation, PROF is used in VVC, which refines sub-block-based affine motion compensation based on an optical flow model. Specifically, after performing sub-block-based affine motion compensation, each brightness prediction sample of an affine block is modified by a sample refinement value derived based on the optical flow equation. In some examples, the operation of PROF can be summarized in the following four steps:

[0054] In the first step, sub-block-based affine motion compensation is performed using the sub-block MV derived from equation (1) above for the 4-parameter affine model or the sub-block MV derived from equation (2) above for the 6-parameter affine model to generate sub-block prediction I(i,j).

[0055] Furthermore, in the second step, the spatial gradient g of each predicted sample point... x (i,j) and g y (i,j) is calculated as shown in the following equation (3):

[0056] g x (i,j)=(I(i+1,j)-I(i-1,j)) (1)

[0057] g y (i,j)=(I(i,j+1)-I(i,j-1))

[0058] Therefore, to compute the gradient, an additional row and / or column of predicted samples needs to be generated on each of the four sides of a sub-block, expanding the 4×4 sub-block to a 6×6 sub-block. To reduce memory bandwidth and complexity, the samples on the expanded boundaries are copied from the nearest integer pixel position in the reference image to avoid additional interpolation processes.

[0059] In addition, in the third step, the brightness prediction refinement value is calculated using the following equation (4):

[0060] ΔI(i,j)=g x (i,j)*Δv x (i,j)+g y (i,j)*Δv y (i,j) (2)

[0061] Where Δv(i,j) is the difference between the pixel MV calculated for the sample point position (i,j) and the sub-block MV of the sub-block where pixel (i,j) is located, represented by v(i,j).

[0062] Figure 7The PROF process for affine modes according to some embodiments of this disclosure is illustrated. In PROF, after the prediction refinement is added to the original prediction samples, a clipping operation "clip3" is performed to clip the values ​​of the refined prediction samples to within 15 bits, as shown in the following equation:

[0063] I r (i,j)=I(i,j)+ΔI(i,j)

[0064] I r (i,j)=clip3(-dILimit,dILimit-1,I r (i,j))

[0065] dILimit=(1< <max(13,BitDepth+1))

[0066] Among them, I(i,j) and I r (i,j) represent the original predicted sample value and the refined predicted sample value at position (i,j), respectively. The function clip3(min,max,val) restricts the given value "val" to the range [min,max].

[0067] Since the affine model parameters and the pixel position relative to the center of the sub-block do not change between sub-blocks, Δv(i,j) can be calculated for the first sub-block and then applied to other sub-blocks in the same CU. When Δx and Δy are the horizontal and vertical offsets from the sample position (i,j) to the center of the sub-block to which the sample belongs, Δv(i,j) can be derived as shown in the following equation (5):

[0068]

[0069] Based on the affine subblock MV, equations (1) and (2) are derived, and the parameters c, d, e, and f in equation (5) above can be derived.

[0070] Specifically, for a 4-parameter affine model, the parameters c, d, e, and f can be derived as shown in the following equation:

[0071]

[0072] Furthermore, for a 6-parameter affine model, the parameters c, d, e, and f can be derived as shown in the following equation:

[0073]

[0074] Among them, (v 0x ,v 0y ), (v 1x ,v 1y ), (v2x ,v 2y ) represents the top-left control point MV, top-right control point MV, and bottom-left control point MV of the current coded block, while w and h are the width and height of the block. In PROF, MV differs from Δv. x and Δv y It is always exported with a resolution of 1 / 32 of a pixel.

[0075] Finally, in the fourth step, the brightness prediction refinement ΔI(i,j) is added to the sub-block prediction I(i,j). The final prediction I′(i,j) for the sample at position (i,j) is generated as shown in the following equation (6):

[0076] I'(i,j)=I(i,j)+ΔI(i,j) (6)

[0077] Bidirectional optical flow (BIO)

[0078] Bidirectional prediction in video encoding and decoding is a simple combination of two temporal prediction blocks obtained from a reference image. However, due to the signaling cost and accuracy trade-offs in motion vectors, the motion vectors received at the decoder may not be very accurate. As a result, there may still be some minor motion remaining between the two prediction blocks, which can reduce the efficiency of motion-compensated prediction. To address this issue, both the VVC and AVS3 standards employ the BIO tool to compensate for this motion for each sample within a block. Specifically, BIO is a sample-by-sample motion refinement performed on top of block-based motion-compensated prediction when using bidirectional prediction.

[0079] In BIO design, the derivation of the refined motion vector for each sample point in a block is based on the classical optical flow model. Let I (k) (x, y) represents the sample value at coordinates (x, y) of the predicted block derived from the reference image list k (k = 0, 1), and and Let be the horizontal and vertical gradients of the sample point. Assuming the optical flow model is valid, then the motion refinement (v) at (x,y) is... x ,v y The following optical flow equation (7) can be derived:

[0080]

[0081] Using a combination of optical flow equation (7) and interpolation of the predicted block along the motion trajectory, such as Figure 4 As shown, it can be obtained

[0082] BIO prediction is shown in the following equation (8):

[0083]

[0084] Figure 4 This is a schematic diagram illustrating an example of a BIO model according to some embodiments of the present disclosure. For example... Figure 4 As shown, (MV) x0 ,MV y0 ) and (MV x1 ,MV y1 The instruction is used to generate two prediction blocks I. (0) and I (1) The block-level motion vector. Furthermore, by minimizing motion refinement compensation (i.e., Figure 4 The difference Δ between the sample values ​​after A and B in the original text is used to calculate the motion refinement (v) at the sample location (x,y). x ,v y As shown in equation (9) below:

[0085]

[0086] Furthermore, to ensure the uniformity of the derived motion refinement, it is assumed that the motion refinement is consistent within the local surrounding region centered at (x,y); therefore, in the BIO design in AVS3, (v) is derived by minimizing Δ within a 4×4 window Ω around the current sample point at (x,y). x ,v y The value of ) is shown in equation (10) below:

[0087]

[0088] As shown in equations (8) and (10), in addition to the block-level MC, it is also necessary to perform motion compensation for each motion compensation block (i.e., I) in the BIO. (0) and I (1) The gradient is derived for each sample point to refine the local motion and produce the final prediction at that sample point location. In AVS3, the gradient is computed via a two-dimensional (2D) separable finite impulse response (FIR) filtering process that defines a set of 8-tap filters and applies different filters according to the block-level motion vector (e.g., ...). Figure 4 (MV) x0 ,MV y0 ) and (MV x1 ,MV y1 The accuracy of the gradient is used to derive the horizontal and vertical gradients. Table 1 shows the coefficients of the gradient filter used by BIO.

[0089] Table 1

[0090] 0 {-4,11,-39,-1,41,-14,8,-2} 1 / 4 {-2,6,-19,-31,53,-12,7,-2} 1 / 2 {0,-1,0,-50,50,0,1,0} 3 / 4 {2,-7,12,-53,31,19,-6,2}

[0091] Finally, BIO is applied only to bidirectional prediction blocks predicted by two reference blocks from temporally adjacent images. Furthermore, BIO is enabled without sending additional information from the encoder to the decoder. Specifically, BIO is applied to all bidirectional prediction blocks that have both forward and backward prediction signals.

[0092] Final Motion Vector Expression (UMVE)

[0093] The UMVE mode in the AVS3 standard is the same tool as the merge mode named with motion vector difference (MMVD) in the VVC standard. In addition to the regular merge mode which derives the motion information of the current block from its spatial / temporal neighboring blocks, the MMVD / UMVE mode is introduced as a special merge mode in both the VVC and AVS standards.

[0094] Specifically, in both VVC and AVS3, this is signaled at the code block level via an MMVD flag. In MMVD mode, two basic merge candidates are first generated as the first two candidates in the regular merge mode. After selecting and signaling a basic merge candidate, additional syntax elements are signaled to indicate the MVD of motion added to the selected merge candidate. The MMVD syntax elements include a merge candidate flag for selecting the basic merge candidate, a distance index for specifying the MVD size, and a direction index for indicating the MVD direction.

[0095] In the AVS3 standard, sub-block-based affine motion compensation (affine mode), similar to that in the VVC standard, is used to generate inter-frame predicted pixel values. This sub-block-based prediction is a trade-off between coding efficiency, complexity, and memory access bandwidth. The average prediction accuracy is lower than pixel-based prediction because all pixels within each sub-block share the same motion vector. Unlike the VVC standard, the AVS3 standard does not offer pixel-level refinement after sub-block-based motion compensation in affine mode.

[0096] This disclosure provides a novel method for improving prediction accuracy in affine modes. Following conventional sub-block-based affine motion compensation, the prediction value for each pixel is refined by adding a difference value derived from the optical flow equation. The proposed method may be referred to as Affine Motion Compensated Prediction Refinement (AMPR). AMPR achieves pixel-level prediction accuracy without significantly increasing complexity, while also maintaining worst-case memory access bandwidth comparable to conventional sub-block-based motion compensation in affine modes. Although AMPR is also based on optical flow, it differs significantly from PROF in the VVC standard in the following aspects.

[0097] First, gradient calculation is performed at each pixel. Unlike PROF, which expands the sub-block prediction by one pixel on each side, AMPR utilizes interpolation-based filtering for gradient calculation at each pixel, allowing for a unified design between AMPR and BIO workflows in AVS3.

[0098] Second, the MV difference is calculated at each pixel. Unlike PROF, which always calculates the MV difference based on the pixel position relative to the center of the sub-block, AMPR can calculate the MV difference based on the pixel position relative to different positions within the sub-block.

[0099] Third is early termination. Unlike the PROF process, which is always called on the decoder side when predicting coded blocks via affine modes, AMPR can be adaptively skipped on the decoder side based on certain defined conditions indicating that applying AMPR is not a good performance and / or complexity tradeoff.

[0100] In some examples, this early termination method can also be used to simplify encoder-side operations. Examples disclosed here also present some encoder-side optimization methods for the AMPR process to reduce its latency and energy consumption, such as skipping AMPR for affine UMVEs, checking the optimal mode selection at the parent CU before applying AMPR, skipping AMPR for motion estimation of certain block sizes, checking the magnitude of the pixel MV difference before applying AMPR, and skipping AMPR for certain image types (such as low-latency images or non-low-latency images).

[0101] AMPR's exemplary workflow

[0102] The AMPR method may include the five steps described below. In the first step, conventional sub-block-based affine motion compensation is performed to generate a sub-block prediction I(i,j) at each pixel location (i,j).

[0103] In the second step, the horizontal spatial gradient g of the sub-block prediction is calculated at each pixel location using interpolation-based filtering. x (i,j) and vertical spatial gradient g y (i,j). In some examples, both the horizontal and vertical gradients of the affine prediction samples are calculated directly from reference samples at integer sample positions in the time reference image. One advantage of this is that for each affine sub-block, its gradient values ​​can be generated simultaneously with the generation of its prediction samples. Another design benefit of this gradient calculation method is that it is consistent with the gradient calculation process used by other codec tools in the AVS standard, such as BIO. Sharing the same process between different modules in the standard is friendly to pipelined and / or parallel designs in actual hardware codec implementations.

[0104] Specifically, the input to the gradient derivation process is the same reference samples used for motion compensation of the affine sub-blocks and the input motion (MV) of the sub-blocks. x ,MV y The same fractional components (fracX, fracY) are used to derive the gradient value at each sample location, in addition to the default 8-tap FIR filter h used for affine prediction. L In addition, another set of new FIR filters h is introduced into the proposed method. G To calculate the gradient value.

[0105] Additionally, depending on the direction of the derived gradient, the filter h is applied. G and h L The order is different. In deriving the horizontal gradient g... x In the case of (i,j), first apply the gradient filter h in the horizontal direction. G To derive the horizontal gradient value at the horizontal fractional sample location fracX; then, apply the interpolation filter h vertically. L The gradient value is interpolated at the vertical fractional sample point location fracY.

[0106] Conversely, when deriving the vertical gradient g y When (i,j), first apply the interpolation filter h horizontally. L Interpolate intermediate samples at the horizontal sample location fracX, and then apply a gradient filter h in the vertical direction. G The vertical gradient value at the vertical fractional sample point position fracY is derived from the intermediate interpolation sample point.

[0107] In some examples, gradient filters can be generated with different filter coefficient accuracies and different numbers of taps, which provides various trade-offs between gradient computation accuracy and computational complexity. For example, gradient filters with more filter taps and / or higher filter coefficient accuracy generally lead to better encoding / decoding efficiency, but at the cost of more computational operations (e.g., multiple additions, multiplications, and shifts) due to the gradient computation process. In one example, the following 8-tap filter is proposed for horizontal and / or vertical gradient computation for AMPR, as shown in Table 2. Table 2 lists the predefined 8-tap interpolation filter coefficients f for generating spatial gradients based on 1 / 16 pixel accuracy of input sample values. grad An example table for [p].

[0108] Table 2

[0109]

[0110] In another example, to reduce the complexity of gradient computation, the following 4-tap FIR filters, as shown in Table 3, are used for gradient generation in the proposed AMPR method. Table 3 lists the predefined 4-tap interpolation filter coefficients f for generating spatial gradients with 1 / 16 pixel precision based on input sample values. grad An example table for [p].

[0111] Table 3

[0112]

[0113] In the third step, at each pixel position (i,j), the MV difference Δv(i,j) between the MV of each pixel and the MV of the sub-block to which the pixel belongs is calculated. Figure 8 An example is shown that calculates the horizontal and vertical offsets from the sample point location to a specific location in the derived sub-block MV. For example... Figure 8 As shown, the horizontal offset Δx and vertical offset Δy are calculated from the sample point position (i,j) to a specific position (i′,j′) of the sub-block from which the sub-block MV is derived. In some examples, the specific position (i′,j′) may not always be the center of the sub-block. For example... Figure 8 As shown, Δv(i,j) is calculated based on the pixel position relative to a specific position within the sub-block using equation (5).

[0114] In one example, for the affine pattern, let (i,j) be the pixel position / coordinate within the sub-block to which the pixel belongs, and w and h be the width and height of the sub-block (e.g., for a 4×4 sub-block, w = h = 4, for an 8×8 sub-block, w = h = 8). The horizontal offset Δx and vertical offset Δy (Δx and Δy are defined in equation (5)) for each pixel (i,j) can be derived as follows, where i = 0…(w-1) and j = 0…(h-1).

[0115] In one example, if the sub-block MV is derived from the position at the center of the sub-block at an integer position, then Δx and Δy can be calculated using the equation shown below:

[0116]

[0117] Alternatively, if the sub-block MV is derived from the position at the center of the sub-block at the fractional position, then Δx and Δy can be calculated using the equation shown below:

[0118]

[0119] In another example, if the sub-block MV is derived from the position at the top left corner of the sub-block, then Δx and Δy can be calculated using the equation shown below:

[0120]

[0121] In another example, if the sub-block MV is derived from the position at the top right corner within the sub-block, then Δx and Δy can be calculated using the equation shown below:

[0122]

[0123] Alternatively, if the sub-block MV is derived from the position at the top right corner outside the sub-block, then Δx and Δy can be calculated using the equation shown below:

[0124]

[0125] In another example, if the sub-block MV is derived from the position at the bottom left corner within the sub-block, then Δx and Δy can be calculated using the equation shown below:

[0126]

[0127] Alternatively, if the sub-block MV is derived from the position at the bottom left corner outside the sub-block, then Δx and Δy can be calculated as:

[0128]

[0129] In another example, Δv(i,j) can be calculated using equation (5), where Δx and Δy are the horizontal and vertical offsets from the sample location (i,j) to the pilot sample location within the subblock to which the sample belongs. The pilot sample location refers to the sample location within a subblock, which is used to derive the MV of the subblock-based prediction samples used to generate the subblock. In one example, the values ​​of Δx and Δy are derived based on the location of the subblock within the current CU as follows.

[0130] Figure 9 An example of a sub-block within an affine CU according to some embodiments of this disclosure is shown. For the upper left sub-block (i.e....) Figure 9 In the sub-block A), Δx = i, Δy = j. For the upper right sub-block, (i.e. Figure 9 For the sub-block B in the middle, Δx = (i - w + 1), Δy = j. For the lower left sub-block (i.e. Figure 9 For sub-block C), when applying the 6-parameter affine model, Δx = i, Δy = (j - h + 1); and when applying the 4-parameter affine model, Δx = (i - (w >> 1) - 0.5), Δy = (j - (h >> 1) - 0.5). For other sub-blocks, Δx = (i - (w >> 1) - 0.5), Δy = (j - (h >> 1) - 0.5).

[0131] Once the horizontal offset Δx and the vertical offset Δy have been calculated, Δv(i,j) can be derived using the following equation (11):

[0132]

[0133] Where c, d, e, and f are affine model parameters, which are known because the current block is an affine mode coded block. Equation (11) is similar to equation (5) of the PROF tool in the VVC standard.

[0134] In the fourth step, the predicted refinement value is calculated using equation (4).

[0135] In the fifth step, prediction refinement is added to the sub-block prediction I(i,j). The final prediction I′(i,j) is generated as shown in equation (6).

[0136] In this disclosure, the proposed AMPR workflow can be applied to the luminance component and / or chrominance component.

[0137] In one example, to achieve a good performance / complexity tradeoff, the proposed AMPR is only applied to the affine prediction samples for refining the luminance component, while the chrominance prediction samples are still generated based on the existing sub-block-based affine motion compensation.

[0138] In another example, to refine the alignment, both the luminance and chrominance components are refined using the proposed AMPR process. In this case, the sample-by-sample MV difference Δv(i,j) can be derived in a different manner.

[0139] In one example, when calculating the predicted refinement value in step four above, the sample-by-sample MV difference Δv(i,j) can always be derived only once based on the luma sub-block and then reused for the chroma component. In this case, the value of Δv(i,j) used by the chroma component can be scaled according to the sampling grid ratio between the co-located luma-coded block and the chroma-coded block. For example, for 4:2:0 video, the value of the reused Δv(i,j) can be halved before being used by the chroma component, while for 4:4:4 video, the same value of the reused Δv(i,j) can be used by the chroma component. For 4:2:2 video, the horizontal offset of Δv(i,j) can be halved before being used by the chroma component, while the vertical offset of Δv(i,j) can remain unchanged.

[0140] In another example, the sample-by-sample MV difference Δv(i,j) can be derived separately for the luminance and chrominance components, where the derivation process can be the same as the third step described above.

[0141] In another example, an adaptive switching is proposed between a method that repeatedly applies the luma motion refinement Δv(i,j) to chroma and a method that separately derives the luma and chroma motion refinements based on the chroma sample format of the input video. For example, for 4:2:0 and 4:2:2 videos, assuming the sampling grids of the luma and chroma components are not aligned, separate derivation of the motion refinement for the luma and chroma components can be applied. On the other hand, when the input video is in a 4:4:4 chroma sample format, it only needs to derive the motion refinement once (i.e., for luma) and then repeat it for the other two color components because the sampling grids of the three color components are perfectly aligned.

[0142] In another example, a flag is signaled to indicate whether AMPR is applied to the chroma components at various codec levels (e.g., sequence level, picture level, stripe level, etc.). Furthermore, if the aforementioned enable / disable flag is true, another flag can be signaled from the encoder to the decoder to indicate whether the chroma motion refinement is recalculated from the corresponding control point motion vector or directly borrowed from the corresponding motion refinement of the luminance component.

[0143] Selective enabling of AMPR

[0144] The refinement of predictions derived by applying AMPR may not always be beneficial or / and necessary. According to equation (4), the importance of the derived ΔI(i,j) is determined by the accuracy and magnitude of the derived Δv(i,j) and g(i,j).

[0145] In some examples, AMPR operation can be conditionally applied based on specific conditions. This can be achieved by signaling a flag for each block to indicate whether AMPR mode is applied. Alternatively, the same conditions can be used to enable AMPR operation on both the encoder and decoder sides without requiring additional signaling.

[0146] The motivation for this conditional application of the AMPR operation is that if the control point motion vector (CPMV) of the CU is inaccurate, or the derived affine model (e.g., a 2-parameter, 4-parameter, or 6-parameter affine mode) is inaccurate, then the Δv(i,j) subsequently derived by equation (11) may also be inaccurate. In this case, the AMPR operation may not help or even harm the encoding / decoding performance, so it is better to skip the AMPR operation for blocks. Another motivation for this conditional application of the AMPR operation is that in some cases, the benefit of applying AMPR may be negligible, and from a computational complexity perspective, it is better to disable the operation.

[0147] In one example, the AMPR operation can be applied depending on whether the CMPV is explicitly signaled. In affine merging mode where the CMPV is not explicitly signaled but implicitly derived from spatially adjacent CUs, AMPR can be skipped for the current CU because the CMPV in this mode may be inaccurate.

[0148] In another example, AMPR can be skipped if, for example, the derived Δv(i,j) and / or g(i,j) are smaller compared to some predefined or dynamically determined threshold. Such thresholds can be determined based on various factors, such as CU aspect ratio and / or sub-block size. Such examples can be implemented in different ways as described below.

[0149] In one example, if the absolute value of Δv(i,j) derived for all pixels within a sub-block is less than a threshold, the AMPR for that sub-block can be skipped. This condition can be implemented in various ways. For example, the check for the absolute value of Δv(i,j) derived for all pixels can be simplified by examining only the four corners of the current sub-block, where the maximum absolute value of Δv(i,j) derived for all pixels within the sub-block can be found as shown in the following equation (12):

[0150]

[0151] The pixel position (i,j) can be any pixel coordinate in the sub-block, or it can come from the four corners (0,0), (w-1,0), (0,h-1), (w-1,h-1).

[0152] In another example, the maximum absolute value of all Δv(i,j) can be calculated using the following equation:

[0153]

[0154] Where the sample point position (i,j) is the sub-block in CU excluding the upper left sub-block (i.e., Figure 9 Sub-block A in the middle), the upper right sub-block (i.e., Figure 9 Sub-block B in the middle) and the sub-block to the lower left (i.e., Figure 9 The four corners of the sub-blocks outside of sub-block C). The coordinates of the four corner pixels within the sub-block are: (0,0), (w-1,0), (0,h-1), (w-1,h-1). |x| is a function taking the absolute value of x.

[0155] In another example, the check of Δv(i,j) derived for all pixels within a sub-block can be performed together as in equation (13) or separately in the horizontal and vertical directions as in equation (14).

[0156] like Or if

[0157] In equations (13) and (14) above, different similar forms of expression for ΔI(i,j) represent different simplification methods. For example, in ΔI(i,j)=g y (i,j)*Δv y In the case of (i,j), g can be skipped. x and Δv x The calculation.

[0158] In another example, the examination of the derived Δv(i,j) can be combined with a non-reduced AMPR operation. In this case, equation (14) can be combined with equation (4), and then the predicted refined value can be calculated using the following equation:

[0159]

[0160] In some examples, threshv x or threshv y They can have different or the same values.

[0161] In some examples, when deriving Δv(i,j) for a sub-block, the threshold v may depend on which position is used to derive the sub-block MV. x and threshv y The value of threshv. In other words, if the MV of two sub-blocks is derived using different positions, then the threshv can be determined for these two sub-blocks. x and threshv y A pair of identical values ​​or a pair of different values. For example, for a sub-block whose sub-block-level MV is derived based on the sub-block center, its threshv x and threshv y A pair of values ​​can be used to derive the threshold of the sub-block-level MV based on the position of the top-left corner of the sub-block. x and threshv y A pair of values ​​may be the same or different.

[0162] In some examples, threshv x and threshv y The value can be defined in pixels within the range of [1 / 32, 1 / 16]. For example, values ​​of (1 / 16)×(10 / 16), (1 / 16)×(12 / 16), or (1 / 16)×(14 / 16) can be used as thresholds. In this case, the threshold is a floating-point number in 1 / 16 of a pixel, such as 0.625, 0.75, or 0.875 in 1 / 16 of a pixel.

[0163] In some examples, threshv can be defined based on the image type. x and threshv y The value of the threshold. For low-latency images, the derived affine model parameters can have smaller amplitudes than for other non-low-latency images, because low-latency images tend to have smaller and / or smoother motion, so smaller values ​​are preferred for those thresholds.

[0164] In some examples, threshv x and threshv y The values ​​can be the same, regardless of the image type.

[0165] In some examples, AMPR can be skipped for a sub-block if the absolute values ​​of most of the derived g(i,j) for all pixels within the sub-block are less than a threshold. An example of this approach is when the sub-block contains a smooth surface that can be composed of flat textures (e.g., with little or no high-frequency detail).

[0166] In some examples, the importance of Δv(i,j) and g(i,j) can be considered together or used in a mixed manner to determine whether AMPR should be skipped for the current sub-block or CU.

[0167] Encoder-side optimization

[0168] In the AVS3 standard, the affine UMVE mode is computationally intensive for the encoder because it involves selecting the optimal distance index for each merging mode candidate. When calculating the sum of absolute transform differences (SATD) cost for each candidate distance index, conventional affine motion compensation is always applied. Applying AMPR on top of affine motion compensation can significantly increase the computational load.

[0169] In one example, on the encoder side, the AMPR operation is skipped during SATD-based cost computation for affine UMVE patterns. Experiments show that while the optimal index is selected based on the best SATD cost, whether or not AMPR is applied during SATD computation generally does not change the ranking of the best SATD costs. Therefore, using the proposed method, enabling AMPR mode does not lead to significant encoder complexity for affine UMVE patterns.

[0170] Motion estimation is another major overhead on the encoder side. In another example, the AMPR process can be skipped depending on specific conditions. These conditions indicate that the optimal encoding mode for the CU is unlikely to be an affine mode after the mode selection process.

[0171] An example of this condition is whether the current CU has a parent CU that has been determined to be encoded and decoded using either explicit affine mode or affine merge mode. This is due to the strong correlation between the encoding / decoding mode selection of a CU and its parent CU, and if the above condition is true, the optimal encoding / decoding mode for the current CU is also likely to be explicit affine mode.

[0172] Another exemplary condition for enabling AMPR is whether the parent CU of the current CU is determined to be performing inter-frame prediction using explicit affine mode. If true, AMPR is applied during affine motion estimation of the current CU; otherwise, AMPR is skipped during affine motion estimation of the current CU.

[0173] Compared to large block-size CUs such as 64×64CUs, small-size CUs such as 16×16CUs have a significantly higher average per-pixel computational cost when applying AMPR. To effectively save computational complexity, in another example of this disclosure, AMPR can be skipped for small-size CUs during the motion estimation process. The size of the CU can be defined as the total number of pixels. A pixel number threshold can be defined, such as 16×16, 16×32, or 32×32, and for blocks smaller than the defined threshold, AMPR can be skipped during the affine motion estimation process for that block.

[0174] Figure 10 This is a block diagram illustrating an apparatus for AMPR according to some embodiments of the present disclosure. The apparatus 1000 may be a terminal, such as a mobile phone, tablet computer, digital broadcasting terminal, tablet device, or personal digital assistant.

[0175] like Figure 10 As shown, the device 1000 may include one or more of the following components: processing component 1002, memory 1004, power supply component 1006, multimedia component 1008, audio component 1010, input / output (I / O) interface 1012, sensor component 1014, and communication component 1016.

[0176] Processing component 1002 typically controls the overall operation of device 1000, such as operations related to display, telephone calls, data communication, camera operation, and recording. Processing component 1002 may include one or more processors 1020 for executing instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 1002 may include one or more modules for facilitating interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module for facilitating interaction between multimedia component 1008 and processing component 1002.

[0177] Memory 1004 is configured to store different types of data to support the operation of device 1000. Examples of such data include instructions for any application or method operating on device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, and memory 1004 may be static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0178] Power supply assembly 1006 supplies power to various components of device 1000. Power supply assembly 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 1000.

[0179] Multimedia component 1008 includes a screen that provides an output interface between device 1000 and the user. In some examples, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen that receives input signals from the user. The touch panel may include one or more touch sensors for sensing touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some examples, multimedia component 1008 may include a front-facing camera and / or a rear-facing camera. When device 1000 is in an operating mode (such as a shooting mode or a video mode), the front-facing camera and / or the rear-facing camera can receive external multimedia data.

[0180] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC). When device 1000 is in an operating mode (such as call mode, recording mode, and voice recognition mode), the microphone is configured to receive external audio signals. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some examples, audio component 1010 also includes a speaker for outputting audio signals.

[0181] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules. These peripheral interface modules can be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0182] Sensor assembly 1014 includes one or more sensors for providing state assessment in various aspects of device 1000. For example, sensor assembly 1014 may detect the on / off state of device 1000 and the relative position of components. Components, for example, are the display and keyboard of device 1000. Sensor assembly 1014 may also detect changes in position of device 1000 or its components, the presence or absence of user contact on device 1000, the orientation or acceleration / deceleration of device 1000, and temperature changes of device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical touch. Sensor assembly 1014 may also include optical sensors, such as CMOS or CCD image sensors used in imaging applications. In some examples, sensor assembly 1014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0183] Communication component 1016 is configured to facilitate wired or wireless communication between device 1000 and other devices. Device 1000 may access a wireless network based on communication standards such as WiFi, 4G, or combinations thereof. In the example, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In the example, communication component 1016 may also include a near-field communication (NFC) module for facilitating short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0184] In the example, device 1000 may be implemented by one or more of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components to perform the methods described above.

[0185] Non-transitory computer-readable storage media can be, for example, hard disk drives (HDDs), solid-state drives (SSDs), flash memory, hybrid drives or solid-state hybrid drives (SSHDs), read-only memory (ROMs), optical disc read-only memory (CD-ROMs), magnetic tapes, floppy disks, etc.

[0186] Figure 11 This is a flowchart illustrating an exemplary AMPR process according to some embodiments of the present disclosure.

[0187] In step 1102, the processor 1020 generates sub-block predictions at pixel locations within the sub-blocks by performing sub-block-based affine motion compensation on a video image comprising multiple sub-blocks.

[0188] In step 1104, the processor 1020 uses an interpolation filter at the pixel location to obtain the horizontal and vertical spatial gradients for sub-block prediction.

[0189] In step 1106, the processor 1020 obtains the MV difference between the first MV and the second MV at the pixel location based on the pixel position relative to the position within the sub-block.

[0190] In some examples, the first MV can be the MV of the pixel at the pixel location, and the second MV can be the MV of the sub-block.

[0191] The processor 1020 can obtain prediction refinement at the pixel location based on the horizontal spatial gradient, the vertical spatial gradient, the horizontal MV difference, and the vertical MV difference, and generate the final prediction at the pixel location by adding the prediction refinement to the sub-block prediction.

[0192] In some examples, MV difference can include horizontal MV difference and vertical MV difference.

[0193] The processor 1020 can determine whether AMPR is applied to the sub-block, enable AMPR of the sub-block when it is determined that AMPR is applied, and skip AMPR of the sub-block when it is determined that AMPR is not applied.

[0194] The processor 1020 can determine whether one or more CPMVs of a sub-block are explicitly signaled with a flag indicating the application of the AMPR, and skip the AMPR of the sub-block when it is determined that one or more CPMVs are not explicitly signaled with a flag indicating the application of the AMPR and are implicitly derived from spatially adjacent sub-blocks.

[0195] The processor 1020 can determine whether the magnitude of the horizontal MV difference, vertical MV difference, horizontal spatial gradient, or vertical spatial gradient is less than a threshold, and skip the AMPR of the sub-block when it is determined that the magnitude is less than the threshold.

[0196] The processor 1020 can determine whether the magnitude of the horizontal MV difference or the magnitude of the horizontal spatial gradient is less than a horizontal threshold, and skip the AMPR of the sub-block in the horizontal direction when it determines that the magnitude of the horizontal MV difference or the magnitude of the horizontal spatial gradient is less than the horizontal threshold.

[0197] The processor 1020 can determine whether the magnitude of the vertical MV difference or the magnitude of the vertical spatial gradient is less than a vertical threshold, and skip the AMPR of the sub-block in the vertical direction when it is determined that the magnitude of the vertical MV difference or the magnitude of the vertical spatial gradient is less than the vertical threshold.

[0198] The processor 1020 can determine whether the magnitude of the horizontal MV difference or the magnitude of the horizontal spatial gradient is less than a horizontal threshold, determine whether the magnitude of the vertical MV difference or the magnitude of the vertical spatial gradient is less than a vertical threshold, and in response to determining that the magnitude of the horizontal MV difference or the magnitude of the horizontal spatial gradient is less than the horizontal threshold and determining that the magnitude of the vertical MV difference or the magnitude of the vertical spatial gradient is less than the vertical threshold, skip the AMPR of the sub-block in the horizontal and vertical directions.

[0199] While generating prediction samples for the sub-block at integer sample locations in the time reference image associated with the sub-block, the processor 1020 can obtain the horizontal spatial gradient and the vertical spatial gradient used for the prediction of the sub-block.

[0200] The processor 1020 can obtain the horizontal spatial gradient at the pixel location by applying a gradient filter horizontally to derive the horizontal spatial gradient at the horizontal fractional sample location and applying an interpolation filter vertically to interpolate the horizontal spatial gradient at the vertical fractional sample location.

[0201] The processor 1020 can obtain the vertical spatial gradient at the pixel location by applying an interpolation filter horizontally to interpolate an intermediate interpolation sample at a horizontal fractional sample location and applying a gradient filter vertically to derive the vertical spatial gradient at a vertical fractional sample location from the intermediate interpolation sample.

[0202] The processor 1020 can determine the maximum horizontal absolute value and the maximum vertical absolute value of the MV difference of a plurality of pixels within the sub-block, and determine the MV difference based on the maximum horizontal absolute value and the maximum vertical absolute value by one of the following actions: determining the horizontal MV difference of the plurality of pixels to be zero when the maximum horizontal absolute value is determined to be less than a horizontal threshold; determining the vertical MV difference of the plurality of pixels to be zero when the maximum vertical absolute value is determined to be less than a vertical threshold; and determining the horizontal MV difference and the vertical MV difference of the plurality of pixels to be zero when both the maximum horizontal absolute value and the maximum vertical absolute value are determined to be less than the horizontal threshold and the maximum vertical absolute value is determined to be less than the vertical threshold.

[0203] The processor 1020 can obtain the MV difference based on the luma sub-block and obtain the prediction refinement of the chroma sub-block based on the MV difference obtained based on the luma sub-block.

[0204] The processor 1020 can respond to a chroma sub-block transmitted via a signal indicating the AMPR application to obtain a predictive refinement of the chroma sub-block based on the MV difference obtained from the luminance sub-block.

[0205] In some examples, an apparatus for video encoding and decoding is provided. The apparatus includes one or more processors 1020; and a memory 1004 configured to store instructions executable by one or more processors; wherein, when executing these instructions, the processor is configured to perform actions such as... Figure 11 The method shown.

[0206] In some other examples, a non-transitory computer-readable storage medium 1004 is provided, in which instructions are stored. When these instructions are executed by one or more processors 1020, the instructions cause the processor to perform actions such as Figure 11 The method shown.

[0207] The description in this disclosure has been presented for illustrative purposes and is not intended to be exhaustive or limited thereto. Many modifications, variations, and alternative embodiments will be apparent to those skilled in the art from the teachings presented in the foregoing description and the associated drawings.

[0208] The examples were chosen and described to explain the principles of this disclosure and to enable others skilled in the art to understand the various embodiments of this disclosure and, best of all, to utilize the basic principles and the various embodiments with modifications suitable for the intended particular purpose. Therefore, it will be understood that the scope of this disclosure is not limited to the specific examples of the disclosed embodiments, and that modifications and other embodiments are intended to be included within the scope of this disclosure.

Claims

1. A method for video encoding and decoding, comprising: At each pixel location in the sub-block, perform sub-block-based affine motion compensation from the video image to generate an initial prediction; At each pixel location, a motion vector MV difference is obtained, wherein the MV difference includes a horizontal MV difference and a vertical MV difference. Obtaining the MV difference at each pixel location includes: obtaining the MV difference based on the horizontal and vertical offsets of each pixel location relative to a specific location within the sub-block; and the specific location is a non-center location; and Based on the initial prediction, the horizontal MV difference, and the vertical MV difference, a final prediction is obtained at each pixel location.

2. The method according to claim 1, further comprising: Determine whether the AMPR (Affine Motion Compensation Prediction Refinement) enable flag for the sub-block is true; In response to determining that the AMPR enable flag is true, the AMPR of the sub-block is enabled; as well as In response to determining that the AMPR enable flag is false, AMPR of the sub-block is not enabled.

3. The method according to claim 1, further comprising: At each pixel location, the horizontal and vertical spatial gradients for predicting the sub-block are obtained using an interpolation filter; The final prediction at each pixel location, based on the initial prediction, the horizontal MV difference, and the vertical MV difference, includes: The final prediction is obtained at each pixel location based on the initial prediction, the horizontal spatial gradient, the vertical spatial gradient, the horizontal MV difference, and the vertical MV difference.

4. The method of claim 2, further comprising: Determine whether one or more control points MV CPMV of the sub-block are explicitly transmitted via signaling using a flag indicating that the AMPR is enabled; as well as In response to determining that one or more CPMVs do not explicitly transmit a flag indicating that the AMPR is enabled via signaling and are implicitly derived from spatially adjacent sub-blocks, the AMPR of the sub-block is not enabled.

5. The method of claim 3, further comprising: Determine whether the magnitude of the horizontal MV difference, the vertical MV difference, the horizontal spatial gradient, or the vertical spatial gradient is less than a threshold. as well as In response to determining that the size is less than the threshold, AMPR of the sub-block is not enabled.

6. The method of claim 3, further comprising: Determine whether the magnitude of the horizontal MV difference or the magnitude of the horizontal spatial gradient is less than a horizontal threshold. as well as In response to determining that the magnitude of the horizontal MV difference or the magnitude of the horizontal spatial gradient is less than the horizontal threshold, AMPR of the sub-block in the horizontal direction is not enabled.

7. The method of claim 3, further comprising: Determine whether the magnitude of the vertical MV difference or the magnitude of the vertical spatial gradient is less than a vertical threshold; as well as In response to determining that the magnitude of the vertical MV difference or the magnitude of the vertical spatial gradient is less than the vertical threshold, AMPR of the sub-block in the vertical direction is not enabled.

8. The method of claim 3, further comprising: Determine whether the magnitude of the horizontal MV difference or the magnitude of the horizontal spatial gradient is less than a horizontal threshold, and determine whether the magnitude of the vertical MV difference or the magnitude of the vertical spatial gradient is less than a vertical threshold; and In response to determining that the magnitude of the horizontal MV difference or the magnitude of the horizontal spatial gradient is less than the horizontal threshold and determining that the magnitude of the vertical MV difference or the magnitude of the vertical spatial gradient is less than the vertical threshold, AMPR of the sub-block in the horizontal and vertical directions is not enabled.

9. The method according to claim 3, wherein, While generating prediction samples for the sub-block at integer sample locations in the time reference image associated with the sub-block, the horizontal spatial gradient and the vertical spatial gradient for predicting the sub-block are obtained.

10. The method according to claim 3, wherein, Obtaining the horizontal and vertical spatial gradients for sub-block prediction using an interpolation filter at each pixel location includes: The horizontal spatial gradient at each pixel location is obtained in the following manner: A gradient filter is applied horizontally to derive the horizontal spatial gradient at the horizontal fractional sample location; and The interpolation filter is applied vertically to interpolate the horizontal spatial gradient at the vertical fractional sample location.

11. The method according to claim 10, wherein, The interpolation filter is a 4-tap filter or an 8-tap filter.

12. The method according to claim 3, wherein, Obtaining the horizontal spatial gradient and the vertical spatial gradient for the sub-block prediction using the interpolation filter at each pixel location includes: The vertical spatial gradient at each pixel location is obtained in the following manner: The interpolation filter is applied horizontally to interpolate intermediate interpolated samples at the horizontal fractional sample locations; and A gradient filter is applied vertically to derive the vertical spatial gradient at the vertical fractional sample location from the intermediate interpolated sample.

13. The method according to claim 12, wherein, The interpolation filter is a 4-tap filter or an 8-tap filter.

14. The method according to claim 1, wherein, Obtaining the motion vector MV difference at a pixel location includes: The MV difference is calculated based on the horizontal and vertical offsets, where, The horizontal offset and the vertical offset are calculated using the following equations: in, This indicates the horizontal offset. This indicates the vertical offset. This indicates the position of a single pixel.

15. The method according to claim 1, wherein, Obtaining the motion vector MV difference at a pixel location includes: The MV difference is calculated based on the horizontal and vertical offsets, where, The horizontal offset and the vertical offset are calculated using the following equations: in, This indicates the horizontal offset. This indicates the vertical offset. This indicates the position of a single pixel. w It is the width of the sub-block.

16. The method according to claim 1, wherein, Obtaining the motion vector MV difference at a pixel location includes: The MV difference is calculated based on the horizontal and vertical offsets, where, The horizontal offset and the vertical offset are calculated using the following equations: in, This indicates the horizontal offset. This indicates the vertical offset. This indicates the position of a single pixel. h It is the height of the sub-block.

17. The method of claim 1, further comprising: Determine the maximum absolute value of the horizontal MV difference and the maximum absolute value of the vertical MV difference of multiple pixels within the sub-block; In response to determining that the maximum absolute value of the level is less than the level threshold, AMPR of the sub-block is not enabled in the horizontal direction; In response to determining that the maximum vertical absolute value is less than a vertical threshold, AMPR of the sub-block is not enabled in the vertical direction; as well as In response to determining that the maximum horizontal absolute value is less than the horizontal threshold and the maximum vertical absolute value is less than the vertical threshold, AMPR of the sub-block is not enabled in the horizontal direction and AMPR of the sub-block is not enabled in the vertical direction.

18. The method according to claim 17, wherein, The horizontal threshold and the vertical threshold are determined based on the image type, and the values ​​of the horizontal threshold and the vertical threshold are respectively in the range of [1 / 32, 1 / 16].

19. The method of claim 17, wherein, The plurality of pixels within the sub-block include the four corners of the sub-block.

20. The method according to claim 1, wherein, The MV difference is obtained based on luminance sub-blocks, and the method further includes: Based on the MV difference obtained from the luminance sub-block, a prediction refinement for the chrominance sub-block is obtained.

21. The method of claim 20, further comprising: In response to a chroma sub-block transmitted via a signal indicating AMPR activation, a prediction refinement for the chroma sub-block is obtained based on the MV difference obtained from the luminance sub-block.

22. An apparatus for video encoding and decoding, comprising: One or more processors; as well as A memory configured to store instructions executable by the one or more processors; wherein, when the instructions are executed, the one or more processors are configured to perform the method according to any one of claims 1-21.

23. A non-transitory computer-readable storage medium for video encoding and decoding, storing computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform the method according to any one of claims 1-21.

24. A computer program product comprising a plurality of programs, said plurality of programs being executed by an electronic device having one or more processors, wherein, When the plurality of programs are executed by the one or more processors, the electronic device performs the method according to any one of claims 1-21.

Images