Phase-based motion compensation for predictive video coding of chroma content

Abstract

Apparatuses and methods are disclosed, including techniques that address luma-chroma phase inconsistency in a video stream that contributes to prediction errors associated with predictive coding of chroma contents. The disclosed techniques include predictively encoding and decoding chroma blocks. The coding and decoding of a chroma block are performed by motion compensation based on a chroma phase, representing a phase between corresponding samples from the chroma block and its reference chroma block.

Description

[0001] PHASE-BASED MOTION COMPENSATION FOR PREDICTIVE VIDEO CODING OF CHROMA CONTENT CROSS REFERENCE TO RELATED APPLICATIONS [1] This application claims the benefit of European Application No. 23305427.9, filed on March 28, 2023, which is incorporated herein by reference in its entirety. BACKGROUND [2] Predictive video coding employs prediction to leverage spatial and temporal redundancy in the video’s content. In predictive coding, a video block can be predicted based on motion compensation where a reference block, that is spatially related to the video block by a motion vector, is used to generate a prediction block that matches the video block. Typically, a motion vector used for a luma block’s prediction is also used for the prediction of a corresponding chroma block, as the contents of both are expected to be correlated. And so, accurate prediction can be obtained assuming that the relative position of corresponding chroma and luma samples – namely, the luma-chroma phase – is constant across the video frames and throughout each frame. However, in practice, luma-chroma phase inconsistency introduces a phase between corresponding samples in a chroma block (currently encoded) and in the chroma block used for its prediction (the reference chroma block). This chroma phase, if not addressed, leads to a prediction error associated with the chroma block predictive coding. SUMMARY [3] Aspects disclosed in the present disclosure describe methods for encoding video data. The methods include obtaining a video block from a video frame of the video data and coding a chroma block of the video block. The coding of the chroma block comprises performing motion compensation based on a chroma phase, where a chroma phase is a phase between corresponding samples from the chroma block and a reference chroma block. Aspects disclosed in the present disclosure also describe methods for decoding the video data. The methods include obtaining a bitstream that codes a video block from a video frame of the video data and decoding a chroma block of the video block. The decoding of the chroma block comprises performing motion compensation based on the chroma phase. [4] Aspects disclosed in the present disclosure describe an apparatus for encoding video data. The apparatus comprises at least one processor and memory storing instructions. The instructions, when executed by the at least one processor, cause the apparatus to obtain a video block from a video frame of the video data and to code a chroma block of the video block. The coding of the chroma block comprises performing motion compensation based on a chroma phase, where a chroma phase is a phase between corresponding samples from the chroma block and a reference chroma block. Aspects disclosed in the present disclosure also describe an apparatus for decoding video data. The apparatus comprises at least one processor and memory storing instructions. The instructions, when executed by the at least one processor, cause the apparatus to obtain a bitstream that codes a video block from a video frame of the video data and to decode a chroma block of the video block. The decoding of the chroma block comprises performing motion compensation based on the chroma phase. [5] Further aspects disclosed in the present disclosure describe a non-transitory computer- readable medium comprising instructions executable by at least one processor to perform methods for encoding video data. The methods include obtaining a video block from a video frame of the video data and coding a chroma block of the video block. The coding of the chroma block comprises performing motion compensation based on a chroma phase, where a chroma phase is a phase between corresponding samples from the chroma block and a reference chroma block. Aspects disclosed in the present disclosure also describe a non-transitory computer- readable medium comprising instructions executable by at least one processor to perform methods for decoding video data. The methods include obtaining a bitstream that codes a video block from a video frame of the video data and decoding a chroma block of the video block. The decoding of the chroma block comprises performing motion compensation based on the chroma phase. [6] This Summary is provided to introduce a selection of concepts in a simplified form that is further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to limitations that solve any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS [7] FIG. 1 is a block diagram of an example system, according to which aspects of the present embodiments can be implemented. [8] FIG.2 is a functional block diagram of an example video encoder, according to which aspects of the present embodiments can be implemented. [9] FIG.3 is a functional block diagram of an example video decoder, according to which aspects of the present embodiments can be implemented.

[0010] FIG. 4 is a functional block diagram of an example encoder using reference picture resampling, according to which aspects of the present embodiments can be implemented.

[0011] FIG. 5 is a functional block diagram of an example decoder using reference picture resampling, according to which aspects of the present embodiments can be implemented.

[0012] FIG.6 is a block diagram illustrating the application of adaptive resolution coding to a video stream, according to which aspects of the present embodiments can be implemented.

[0013] FIG.7 is a diagram illustrating luma-chroma phase and chromatic aberration, according to which aspects of the present embodiments can be implemented.

[0014] FIG.8 is a diagram illustrating phase-based motion compensation, according to which aspects of the present embodiments can be implemented.

[0015] FIG. 9 is a diagram illustrating template-base and phase-based motion compensation, according to which aspects of the present embodiments can be implemented.

[0016] FIG. 10 is a flowchart illustrating template-based and phase-based motion compensation using optical flow, according to which aspects of the present embodiments can be implemented.

[0017] FIG. 11 is a flowchart illustrating chroma phase derivation based on luma-chroma alignment, according to which aspects of the present embodiments can be implemented.

[0018] FIG. 12 is a flowchart of an example encoding method, utilizing phase-based motion compensation, according to which aspects of the present embodiments can be implemented.

[0019] FIG. 13 is a flowchart of an example decoding method, utilizing phase-based motion compensation, according to which aspects of the present embodiments can be implemented. DETAILED DESCRIPTION

[0020] Apparatuses and methods are disclosed herein for video encoding and decoding. Aspects of the present disclosure describe techniques for addressing luma-chroma phase inconsistency, thereby reducing prediction errors associated with predictive coding and decoding of chroma blocks. Traditional systems and methods for predictive video coding are described next in reference to FIGS. 1-3, followed by a description of aspects of the present disclosure, described in reference to FIGS.4-13.

[0021] FIG. 1 illustrates a block diagram of an example system 100. System 100 can be embodied as a device and can be configured to perform one or more of the aspects described in this application. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100, singly or in combination, can be embodied in an integrated circuit, multiple integrated circuits, and / or discrete components. For example, in at least one embodiment, the processing 110 and encoder / decoder 130 elements of system 100 are distributed across multiple integrated circuits and / or discrete components. In various embodiments, the system 100 is communicatively coupled to other systems, or to other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports.

[0022] The system 100 includes at least one processor 110 that can be configured to execute instructions loaded therein for implementing, for example, the various aspects described in this application. Processor 110 can include embedded memory, input and output interfaces, and various other circuitries as known in the art. The system 100 includes at least one memory 120, such as a volatile memory device and / or a non-volatile memory device. System 100 includes a storage device 140, which can include non-volatile memory and / or volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. The storage device 140 can be an internal storage device, an attached storage device, and / or a network accessible storage device, for example.

[0023] System 100 includes an encoder / decoder module 130 configured to process data to provide encoded video data or decoded video data. The encoder / decoder module 130 can include its own processor and memory. The encoder / decoder module 130 can be implemented as a separate element of system 100 or can be incorporated within processor 110 as a combination of hardware and / or software as known to those skilled in the art. Additionally, the encoder / decoder module 130 represents module(s) that can be implemented in a separate device to perform encoding and / or decoding functions.

[0024] Program code that is to be loaded into processor 110 or into encoder / decoder 130 to perform the various aspects described in this application can be stored in a storage device 140 and subsequently loaded into memory 120 for execution by processor 110. In accordance with various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 can store one or more of various items during the performance of the processes described in this application. Such stored items can include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, operational logic, and intermediate or final results from the processing of equations, formulas, operations.

[0025] In several embodiments, memory inside of the processor 110 and / or the encoder / decoder module 130 is used to store instructions and to provide working memory for processing functions that are needed during encoding or decoding. In other embodiments, however, memory external to the processing device (where, for example, the processing device can be either the processor 110 or the encoder / decoder module 130) can be used for one or more of these functions. The external memory can be the memory 120 and / or the storage device 140 that may comprise, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations.

[0026] The input to the elements of system 100 can be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to, (i) an RF portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Composite input terminal (COMP), (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0027] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select, for example, a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the down converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements that perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs some of these functions, including, for example, down-converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to a baseband. In one set-top box embodiment, the RF portion and its associated input processing element receive an RF signal transmitted over a wired (for example, cable) medium, and perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Added elements can include inserting elements in between existing elements, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[0028] Additionally, the USB and / or HDMI terminals can include respective interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed- Solomon error correction, can be implemented, for example, within a separate input processing integrated circuit or within processor 110 as necessary. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface integrated circuits or within processor 110 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110, and encoder / decoder 130 operating in combination with the memory and storage elements to process the data stream as necessary for presentation on an output device.

[0029] Various elements of system 100 can be provided within an integrated housing. Within the integrated housing, the various elements can be interconnected and transmit data therebetween using a suitable connection arrangement 115, for example, an internal bus as known in the art, including the I2C bus, wiring, and printed circuit boards.

[0030] The system 100 includes communication interface 150 that enables communication with other devices via communication channel 190. The communication interface 150 can include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 190. The communication interface 150 can include, but is not limited to, a modem or network card. The communication channel 190 can be implemented, for example, within a wired and / or a wireless medium.

[0031] Data can be streamed to the system 100, in various embodiments, using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signal of these embodiments is received over the communication channel 190 and the communication interface 150 which can be adapted for Wi-Fi communications. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to outside networks including the Internet for allowing streaming applications and other over-the-top communications. In other embodiments, data can be streamed to the system 100 using a set-top box that delivers the data over the HDMI connection of the input block 105 or data can be streamed to the system 100 using the RF connection of the input block 105.

[0032] The system 100 can provide an output signal to various output devices, including a display device 165, an audio device (e.g., speaker(s)) 175, and other peripheral devices 185. The other peripheral devices 185 include, in various examples of embodiments, one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide a function based on the output of the system 100. In various embodiments, control signals are communicated between the system 100 and the display device 165, the audio device 175, or the other peripheral devices 185 using signaling such as AV.link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output devices can be connected to system 100 using the communication channel 190 via the communication interface 150. The display device 165 and the audio device 175 can be integrated in a single unit with the other components of system 100 in an electronic device, for example, a television. In various embodiments, the display interface 160 includes a display driver, for example, a timing controller (T Con) chip.

[0033] Alternatively, the display device 165 and the audio device 175 can be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which the display device 165 and the audio device 175 are external components, the output signal can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0034] FIG. 2 illustrates a functional block diagram of an example video encoder 200. The video encoder 200 can be employed by the system 100 described in reference to FIG. 1. For example, the video encoder 200 can be an encoder that operates according to coding standards such as Advanced Video Coding (AVC, H.264 / MPEG-4 | ISO / IEC 14496-10), High Efficiency Video Coding (HEVC, ITU-T H.265 | ISO / IEC 23008-2), or Versatile Video Coding (VVC, Standard ITU-T H.266, ISO / IEC 23090-3, 2020).

[0035] Prior to undergoing encoding, the video data can be pre-processed by a precoding processor (not shown). Such pre-processing can include applying a color model transform to the color components of the input video frames (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0) or mapping the color components of the input video frames to obtain a signal distribution that is more resilient to compression (for instance, applying a histogram equalizer and / or a denoising filter to one or more of the video frames’ color components). The pre-processing can also include associating metadata with the video data that can be attached to the coded video bitstream.

[0036] In the encoder 200, a video frame is encoded by the encoder elements as generally described below. A picture (referred to herein also as a frame) of the original video to be encoded is partitioned into coding units (namely, original blocks) by an image partitioner 202. Typically, a coding unit (CU) contains a luminance block and respective chroma blocks, and so, generally, operations described herein as applied to a CU are applied to the luminance block and to the respective chroma blocks. Following partition 202, each CU can be encoded using an intra-prediction mode or an inter-prediction mode. In an intra-prediction mode, a prediction of the CU is performed by an intra-predictor 260. In the intra-prediction mode, the content of a CU in a frame is predicted based on content from one or more other CUs of the same frame, using the other CUs’ reconstructed version (available from the adder 255 output). In an inter- prediction mode, motion estimation and motion compensation are performed by a motion estimator 275 and a motion compensator 270, respectively.

[0037] In the inter-prediction mode, the content of a CU in a frame is predicted based on content from one or more other CUs of neighboring frames, using the other CUs’ reconstructed versions (available from the reference picture buffer 280). The encoder decides 205 which prediction result (one obtained through operations in the intra-prediction mode 260 or one obtained through operations in the inter-prediction mode 270, 275) to use for encoding a CU, and indicates the selected prediction mode by a prediction mode flag, for example. The selected prediction result may then be enhanced (e.g., filtered) by a prediction enhancer 285, outputting a respective prediction block. Once a prediction block is generated for each CU, a respective residual block is calculated, for example, by subtracting 210 the predicted CU (i.e., prediction block) from the CU (i.e., original block).

[0038] A CU’s respective residual block or a partition thereof (i.e., a transform block) is then transformed into a coefficient block by a transformer 220 – that is, residual samples of the transform block are transformed into transform coefficients of the coefficient block. The resulting coefficient block is quantized by a quantizer 230. An entropy encoder 245 is next employed to entropy-encode the quantized coefficient block and respective coding parameters (e.g., syntax elements including motion vectors and other control data). Hence, the entropy- encoded quantized coefficient blocks and respective encoding parameters associated with each video frame of the original video are packed into the bitstream of the coded video data.

[0039] Along with the coding of original blocks (CUs), as described above, the encoder 200 reconstructs the coded original blocks to provide references for future predictions. Accordingly, quantized coefficient blocks (provided by the quantizer 230) are de-quantized, by an inverse quantizer 240, and then inverse transformed, by an inverse transformer 250, to reconstruct (decode) the residual blocks of respective original blocks. Adding 255 the reconstructed residual blocks to respective prediction blocks results in respective reconstructed original blocks. In-loop filters 265 can then be applied to the reconstructed picture (formed by the reconstructed original blocks), performing, for example, deblocking filtering and / or sample adaptive offset (SAO) filtering to reduce encoding artifacts. The filtered reconstructed picture can then be stored in the reference picture buffer 280, available for future predictions in an inter-prediction mode. Thus, the encoder 200 also performs decoding operations 240, 250 through which the encoded pictures (frames) are reconstructed. The reconstructed pictures can then be stored in the reference picture buffer 280 and be used to facilitate motion estimation 275 and compensation 270, as explained above. In an aspect, when the reference picture is not of the same dimension as the current picture, the motion compensator 270 may perform a combination of motion compensation and re-sampling operations, as further described in reference to FIG.4.

[0040] FIG. 3 illustrates a functional block diagram of an example video decoder 300. The video decoder 300 can be employed by the system 100 described in reference to FIG. 1. Generally, operational aspects of the video decoder 300 are reciprocal to operational aspects of the video encoder 200. In the decoder 300, the bitstream of coded video data, generated by the video encoder 200, is first entropy-decoded by an entropy decoder 330, decoding from the bitstream the quantized coefficient blocks and various coding parameters. The quantized coefficient blocks are de-quantized, by an inverse quantizer 340, and then are inverse transformed, by an inverse transformer 350, to decode (reconstruct) respective residual blocks. Adding 355 the reconstructed residual blocks to respective prediction blocks results in respective reconstructed original blocks. Depending on the selected prediction mode, a predicted original block can be obtained 370 from an intra-predictor 360 or from a motion compensator 375. The predicted original block may then be enhanced (e.g., filtered) by a prediction enhancer 390, generating a prediction block. In-loop filters 365 can be applied to the reconstructed picture (formed by the reconstructed original blocks), outputting a reconstructed (decoded) video frame. The filtered reconstructed picture is also stored in a reference picture buffer 380 to facilitate motion compensation 375. In an aspect, when the reference picture is not of the same dimension as the current picture, the motion compensator 375 may perform a combination of motion compensation and re-sampling operations, as further described in reference to FIG.5.

[0041] A post-decoding processor (not shown) can further process the reconstructed video. For example, post-decoding processing can include an inverse color model transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or an inverse mapping to reverse the mapping process performed by the pre-encoding processor. The post-decoding processor can use metadata that were derived by the pre-encoding processor and / or were signaled in the video bitstream.

[0042] Aspects disclosed herein are described in reference to a CU, however, the described aspects are similarly applicable to any region in the video frame that prediction can be applied to by an encoder 200 or a decoder 300. Generally, aspects described herein may be applied to a video data region, formed by a video partition, of any shape or size (e.g., a video block). A CU can include a luma component, Y, and chroma components, Cb and Cr (also denoted as “U” and “V”), either one of which is referred to herein also by C. Typically, the chroma component C is subsampled, and, so, has a reduced resolution relative to the corresponding luma component Y. Other color models may be used to represent the image content associated with a CU. For example, a CU can include three color components, red (R), green (G), and blue (B), which are generally of the same resolution. The principles described herein may be extended to such a case where, for example, the G component plays the same role as the Y component and where the R and the B components play the same role as the Cr and Cb components.

[0043] In the VVC (i.e., H.266) standard, a picture-based (or a block-based) re-scaling feature is defined, namely, reference picture resampling (RPR). RPR applies when adaptive resolution coding is used, that is, when successive pictures may be coded at different picture resolutions. The RPR feature can be utilized, for example, for adaptive streaming to continuously adapt the coded video’s bitrate to network bandwidth constraints. Accordingly, given a video sequence input composed of picture (or frame) size of width ^^ and height ^^, the encoder may choose for each frame which resolution (picture size) to use for coding that frame. The RPR feature thus enables inter-prediction based on reference frames of various resolutions, as further explained in reference to FIG.4 and FIG.5.

[0044] FIG. 4 is a functional block diagram of an example encoder using RPR 400. The encoder’s 400 components – including a down-sampler 420, a main encoder 430, a re-sampler 440 (a component of motion compensator 445), and a reference picture buffer 450 – are employed to encode an original video 410 into a coded video bitstream 460. In an aspect, the main encoder 430 represents the encoder 200 of FIG.2, except for the reference picture buffer 280 and the motion compensator 270 that are shown in FIG. 4 as reference picture buffer 450 and motion compensator 445, respectively. In the example of FIG. 4, for each input frame of the original video 410, the encoder 400 can select whether to encode the frame at the original frame size or at a reduced frame size. If the encoder selects to encode at a reduced frame size, the input frame is down-sampled, by the down-sampler 420, before the frame is encoded bythe main encoder 430. For example, the frame may be sub-sampled 420 by a resolution rate of1⁄ 2 , resulting in frame size of width ^^ / 2 and of height ^^ / 2. The decision whether to encodea frame at the original frame size or at a reduced frame size can be done by the encoder 400, for example, based on comparing coding results of the input frame at various resolutions or based on spatial and temporal activity related to the coding of input frames of the original video 410. Consequently, when image content from a currently encoded frame and a corresponding content from a reference frame (needed for inter-prediction of the content from the currently encoded frame) are not at the same resolution, a resampling operation is required. In VVC, the resampling operation and the motion compensation are combined into one single step. This can be done by the re-sampler 440 that can be configured to change the resolution of image content from reference frames to a resolution that matches the resolution of content from the current frame.

[0045] FIG. 5 is a functional block diagram of an example decoder using RPR 500. The decoder’s 500 components – including a main decoder 520, a re-sampler 530 (a component of motion compensator 535), a reference picture buffer 540, and an up-sampler 550 – are employed to decode the original video 410 from a given coded video bitstream 510, resulting in the output video 560. In an aspect, the main decoder 520 represents the decoder 300 of FIG. 3, except for the reference picture buffer 380 and the motion compensator 375 that are shown in FIG. 5 as reference picture buffer 540 and motion compensator 535, respectively. In the example of FIG. 5, image content from one or more frames (stored in the reference picture buffer 540) that is needed as reference to inter-predict image content from a currently decoded frame is re-sampled, by the re-sampler 530, into a resolution that matches the resolution of the currently decoded image content. To output the video, frames that are decoded 500 in a reduced resolution are up-sampled to the resolution of the original video 410, by the up-sampler 550, resulting in the output video 560. Note that the down-sampler 420 of FIG.4 (e.g., employed in a pre-processing stage) and the up-sampler 550 of FIG.5 (e.g., employed in a post-processing stage) are not specified by the VVC standard.

[0046] FIG.6 is a block diagram illustrating the application of adaptive resolution coding to a video stream using RPR 600. In the example of FIG. 6, RPR is applied to a video stream including successive frames 610-660 that are encoded at different picture resolutions. As shown, frames 610, 640, and 660 are encoded at the resolution of the original video 410, while frames 620, 630, and 650 are encoded at a reduced resolution. The bolded arrows 622, 642, 662 indicate reference to a previous frame that was encoded at a different resolution, representing temporal prediction with RPR. The un-bolded arrows 632, 644, 664 indicate reference to a previous frame that was encoded at the same resolution, representing temporal prediction without RPR. For example, frame 640 refers to frame 630 that was encoded at a reduced resolution relative to frame 640, and so, prediction of image content (e.g., luma samples of a CU from frame 640) based on corresponding image content from frame 630 requires up-sampling (by the re-sampler 440 during encoding 400 or by the re-sampler 530 during decoding 500). Additionally, frame 640 refers to frame 610 that was encoded at the same resolution as frame 640, and so, prediction of image content (e.g., luma samples of a CU from frame 640) based on corresponding image content from frame 610 does not require up- sampling. As described above, the down-sampled frames 620, 630, and 650 are up-sampled 550 back to the resolution of the original video 410 to generate the output video 560 for display or for further processing and / or streaming.

[0047] As discussed above, a video block of a currently encoded CU of a frame (e.g., samples from a luma block Y or a chroma block C) may be predicted based on one or more reference video blocks of the same frame (intra-prediction mode) or of neighboring frames (inter- prediction mode), using the reference video blocks’ reconstructed versions. Generally, one or more motion vectors are used to spatially relate the video block to the reference video blocks used for its prediction. Typically, motion vectors used to predict luma blocks are used for the prediction of the corresponding chroma blocks, adjusted to the chroma blocks’ resolution. For instance, when a 4:2:0 format is used, the motion vectors used for luma blocks are scaled down by two when used for the corresponding chroma blocks. The down-scaled motion vectors may also be rounded, and so, these scaled and rounded motion vectors, used for predicting chroma blocks, provide different accuracy than that provided by the motion vectors used to predict the luma blocks. For example, in VVC / H.266, the motion vectors (used for inter-prediction of luma and chroma blocks) provide inter-prediction accuracy of 1 / 16 pixel (i.e., pel).

[0048] The prediction of a video block results in a prediction block through a process of motion compensation ^^ ^^. The operation of motion compensation can be formulated as follows: ^^^ൌ ^^ ^^^ ^^^^^, ^^ ^^^, (1) where ^^^represents a prediction block – that is, a motion compensated reference block ^^^^^of a reference frame ^^^^^that is used to predict a video block ^^^௨^of a currently encoded frame ^^^௨^. Thus, a prediction block ^^^is provided by a motion compensation ^^ ^^ operation based on a motion vector ^^ ^^ and a reference frame ^^^^^. Typically, the motion vector is a two- dimensional displacement vector, ^^ ^^ ൌ ^ ^^ ^^ ^^, ^^ ^^ ^^^, that points to a reference block ^^^^^within a respective reference frame ^^^^^. The pointed to reference block ^^^^^is then used to predict the video block ^^^௨^, either directly (i.e., ^^^ൌ ^^^^^) or indirectly (i.e., ^^^ൌ ^^^ ^^^^^^).

[0049] Note that the reference frame can be the same frame as the current frame, ^^^^^ൌ ^^^௨^. For example, a chroma block of a current frame can be predicted by a motion compensation operation that is applied internally to the current frame – for example, when using an intra- prediction mode that involves a displacement vector, such as intra block copy (IBC) or intra template matching (intraTMP). On the other hand, a chroma block of a current frame can be predicted by a motion compensation operation that is applied to another frame, ^^^^^് ^^^௨^– for example, when using an inter-prediction mode.

[0050] In VVC / H.266 the motion compensation ^^ ^^ operation includes a two-dimensional interpolation operation, using separable interpolation filters applied horizontally and then vertically. When the horizontal motion component ^^ ^^ ^^ and / or the vertical motion component ^^ ^^ ^^ are provided with sub-pixel accuracy, fractional sample interpolation is required. In such a case, first, the motion vector components are rounded into integer values, denoted as ^^ ^^ ^^ ^^ ^^ ൌ ^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ ^^ ^^ ^^ ^^^, and then these integer values are used to identify the position of the reference block ^^^^^in the reference frame ^^^^^. The remaining fractional parts of the motion vector components, denoted as ^^ ^^ ^^ ^^ ^^ ^^ ൌ ^ ^^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ ^^ ^^ ^^ ^^ ^^^, are obtained by ^^ ^^ ^^ ^^ ^^ ^^ ^^ ൌ ^^ ^^ ^^ – ^^ ^^ ^^ ^^ ^^ ^^ and ^^ ^^ ^^ ^^ ^^ ^^ ^^ ൌ ^^ ^^ ^^ – ^^ ^^ ^^ ^^ ^^ ^^ . Here, a floating-point representation is considered. However, in practice, all computations are generally performed with integer-precision. For instance, if 1 / 32thpel accuracy is used, a motion vector of ^^ ^^ ൌ ^1.5, 2.09375^ is represented by ^32 ∗ 1.5, 32 ∗ 2.09375^ ൌ ^48, 67^ . Using 1 / 32thpel accuracy, ^^ ^^ ^^ ^^ ^^ ൌ ^32 ∗ 1,32 ∗ 2^ ൌ ^32, 64^ and ^^ ^^ ^^ ^^ ^^ ^^ ൌ ^32 ∗ 0.5, 32 ∗ 0.09375^ ൌ ^16, 3^.

[0051] A reference sample from ^^^^^is interpolated by polyphase filters, based on a phase corresponding to ^^ ^^ ^^ ^^ ^^ ^^ . Examples of one-dimensional polyphase filters defined in VVC / H.266 for chroma samples interpolation are provided in the following Tables. Table 1 applies when picture size ratio (current picture / reference picture) is lower than or equal to 1.25. Table 2 applies when picture size ratio (current picture / reference picture) is greater than 1.25 and lower than or equal to 1.75. And Table 3 applies when picture size ratio (current picture / reference picture) is greater than 1.75. In Tables 1-3 the parameter ^^ corresponds to ^^ ^^ ^^ ^^ ^^ ^^ ^^ or ^^ ^^ ^^ ^^ ^^ ^^ ^^ (multiplied by 32 to get an integer value).

[0052] Table 1. Chroma polyphase filter coefficients fc^ ^^^ for each 1 / 32 fractional sample position p. p fC[p][0] fC[p][1] fC[p][2] fC[p][3] p fC[p][0] fC[p][1] fC[p][2] fC[p][3] 0 0 64 0 0 16 −4 36 36 −4 1 −1 63 2 0 17 −4 33 39 −4 2 −2 62 4 0 18 −4 30 42 −4 3 -2 60 7 −1 19 −4 29 44 −5 4 −2 58 10 −2 20 −4 28 46 −6 5 −3 57 12 −2 21 −3 24 49 −6 6 −4 56 14 −2 22 −2 20 52 −6 7 −4 55 15 −2 23 −2 18 53 −5 8 −4 54 16 −2 24 −2 16 54 −4 9 −5 53 18 −2 25 −2 15 55 −4 10 −6 52 20 −2 26 −2 14 56 −4 11 −6 49 24 −3 27 −2 12 57 −3 12 −6 46 28 −4 28 −2 10 58 −2 13 −5 44 29 −4 29 −1 7 60 −2 14 −4 42 30 −4 30 0 4 62 −2 15 −4 39 33 −4 31 0 2 63 −1

[0053] Table 2. Chroma polyphase filter coefficients fc^ ^^^ for each 1 / 32 fractional sample position p for scaling factors of around 1.5x. p fC[p][0] fC[p][1] fC[p][2] fC[p][3] p fC[p][0] fC[p][1] fC[p][2] fC[p][3] 0 12 40 12 0 16 1 31 31 1 1 11 40 13 0 17 0 30 33 1 2 10 40 15 −1 18 0 29 33 2 3 9 40 16 −1 19 0 28 34 2 4 8 40 17 −1 20 0 26 35 3 5 8 39 18 −1 21 0 25 36 3 6 7 39 19 −1 22 −1 24 37 4 7 6 38 21 −1 23 −1 23 38 4 8 5 38 22 −1 24 −1 22 38 5 9 4 38 23 −1 25 −1 21 38 6 10 4 37 24 −1 26 −1 19 39 7 11 3 36 25 0 27 −1 18 39 8 12 3 35 26 0 28 −1 17 40 8 13 2 34 28 0 29 −1 16 40 9 14 2 33 29 0 30 −1 15 40 10 15 1 33 30 0 31 0 13 40 11

[0054] Table 3. Chroma polyphase filter coefficients fc^ ^^^ for each 1 / 32 fractional sample position p for scaling factors of around 2x. p fC[p][0] fC[p][1] fC[p][2] fC[p][3] p fC[p][0] fC[p][1] fC[p][2] fC[p][3] 0 17 30 17 0 16 7 25 25 7 1 17 30 18 −1 17 6 25 26 7 2 16 30 18 0 18 6 24 26 8 3 16 30 18 0 19 5 24 26 9 4 15 30 18 1 20 5 23 27 9 5 14 30 18 2 21 5 22 27 10 6 13 29 19 3 22 4 22 28 10 7 13 29 19 3 23 4 21 28 11 8 12 29 20 3 24 3 20 29 12 9 11 28 21 4 25 3 19 29 13 10 10 28 22 4 26 3 19 29 13 11 10 27 22 5 27 2 18 30 14 12 9 27 23 5 28 1 18 30 15 13 9 26 24 5 29 0 18 30 16 14 8 26 24 6 30 0 18 30 16 15 7 26 25 6 31 −1 18 30 17

[0055] Usually, the color format used for video compression is YCbCr 4:2:0, indicating that the chroma components ^^ ^^ and ^^ ^^ of a frame have half the width and half the height of the luma component of the frame. The relative positions of chroma samples and luma samples – namely, the luma-chroma phase – may have different configurations, as illustrated in FIG. 7 by the patterns in 700A and in 700B, where a square represents a luma sample and a circle represents a chroma sample.

[0056] Ideally, the luma-chroma phase of a video content should satisfy the following conditions: 1) it has a predefined pattern (e.g., 700A or 700B); 2) it is the same for all the pictures of a video stream; and 3) it is the same within each picture of the video stream. However, in practice, these conditions may not be satisfied or verified. For instance, in a case of chromatic aberration, chroma samples may not be consistently located relative to corresponding luma samples across frames and / or the locations of the chroma samples relative to the locations of corresponding luma samples may vary throughout the same frame. This is illustrated in the two pictures 700C shown in FIG. 7 where chromatic aberration is shown in the bottom picture. Note that when chromatic aberration occurs, the luma-chroma phase is generally different for the ^^ ^^ and the ^^ ^^ components. Such deviation in phase may be larger than one pixel.

[0057] An inconsistent luma-chroma phase may generate an erroneous prediction of chroma samples, for instance, when a current block ^^^௨^and the reference block ^^^^^used for its prediction do not have the same luma-chroma phase. This issue may also arise when the RPR feature is in used, that is, when the reference frame ^^^^^is not of the same dimension (resolution) as the current frame ^^^௨^, and when the down-sampling 420 (performed before encoding 430) has not taken into account the inconsistency of the luma-chroma phase.

[0058] Aspects disclosed herein address luma-chroma phase inconsistency. To that end, the operation of motion compensation takes into account temporal and spatial changes in the luma- chroma phase in a video stream. According to aspects, motion compensation applied to a chroma block (in the encoder 400 and in the decoder 500) is performed based on a respective chroma phase. The respective chroma phase can be used to adjust the motion vector and / or to interpolate a polyphase filter used to resample the reference chroma block. The chroma phase associated with a chroma block can be directly computed by comparing corresponding template samples around the chroma block and its reference block or can be derived from luma-chroma phases associated with the chroma block and its reference block. In an aspect, a parametric model can be computed to represent the luma-chroma phase associated with a video frame and can be signaled in the bitstream of the coded video. Alternatively, or in combination, changes in the luma-chroma phase in an area of the video frame can be signaled and may be used to locally refine the parametric model. Aspects of phase-based motion compensation are further described below with respect to FIGS.8-13.

[0059] The phase-based motion compensation ^^ ^^ operation disclosed herein takes into consideration a chroma phase difference, denoted ^^ ^^ ൌ ^ ^^ ^^ ^^, ^^ ^^ ^^^ . The chroma phase difference (i.e., the chroma phase) represents the phase between corresponding chroma samples from a currently encoded chroma block and a respective reference chroma block. Using a chroma phase ^^ ^^ and a motion vector ^^ ^^ to carry out the motion compensation operation is demonstrated in FIG.8.

[0060] FIG. 8 is a diagram illustrating phase-based motion compensation. In the example of FIG.8, chroma blocks 820, 870 are predicted by motion compensation operations – at the top 800A, the motion compensation is applied within the same frame 810, and at the bottom 800B, motion compensation is applied relative to a reference frame 860 at a different resolution. In both cases, the motion compensation operation is a function of the motion vector MV and the chroma phase vector PH. The chroma blocks 820, 870 can be either the Cb or the Cr components, and the chroma phase vector PH can be derived for both components, or independently for each component. For example, in an intra-prediction mode, a chroma block 820 to be encoded may be predicted based on a reference chroma block 830 that is located in the same (current) frame 810. The reference chroma block 830 is identified by a motion compensation operation 840 that is a function of a motion vector ^^ ^^ and a chroma phase vector ^^ ^^. In another example, in an inter-prediction mode, a chroma block 870 to be encoded may be predicted based on a reference chroma block 880 that is located in another (reference) frame 860. In this example, the reference frame 860 has been down-sampled 420 by the encoder 400, operating in an RPR mode. The reference chroma block 880 is identified by a motion compensation operation 890 that is a function of a motion vector ^^ ^^ and a chroma phase vector ^^ ^^.

[0061] According to aspects described herein, modified motion compensation is performed with respect to chroma blocks. Thus, the motion compensation operation MC (of equation (1)) is modified as follows: ^^^ൌ ^^^ ^^൫^^^^^, ^^ ^^, ^^ ^^൯, (2) where ^^^represents a prediction chroma block – that is, a motion compensated reference chroma block ^^^^^(of a reference frame ^^^^^) that is used to predict a chroma block ^^^௨^of a currently encoded frame ^^^௨^. Thus, in an aspect, a prediction chroma block ^^^is provided by a modified motion compensation operation, denoted ^^^ ^^ , based on a motion vector ^^ ^^, a chroma phase vector ^^ ^^, and a reference frame ^^^^^. As explained above, the reference frame ^^^^^may be the same as the current frame (in an intra-prediction mode) or another frame (in aninter-prediction mode). The modified motion compensation operation ^^^ ^^ identifies areference chroma block ^^^^^that can be used to predict the chroma block ^^^௨^, either directly (i.e., ^^^ൌ ^^^^^) or indirectly (i.e., ^^^ൌ ^^^ ^^^^^^).

[0062] In a variant, the modified motion compensation operation can be carried out by first adding the chroma phase vector ^^ ^^ to the motion vector ^^ ^^, resulting in an adjusted motion vector ^^ ^^ᇱ: ^^ ^^ᇱൌ ^^ ^^ ^ ^^ ^^ (3) Then, modified motion compensation is applied based on the reference image and the adjusted motion vector as follows: ^^^ ൌ ^^^ ^^ ൫ ^^^^^, ^^ ^^′൯ (4)

[0063] In another variant, the chroma phase vector ^^ ^^ can be used to interpolate the polyphase filter coefficients (shown in Tables 1-3). The interpolated coefficients can be used to resample the reference chroma block. Such interpolation can be performed when the chroma phase vector ^^ ^^ has a higher accuracy than that of the motion vector ^^ ^^. For example, interpolation of the polyphase filter coefficients can be performed when the accuracy, denoted K, for ^^ ^^ is K=1 / 64thpel while the accuracy for ^^ ^^ is K=1 / 32th.

[0064] To that end, considering the x component of the adjusted motion vector ^^ ^^’, ^^ ^^’ ^^ can be formulated as follows: ^^ ^^’ ^^ ൌ ^^ ^^ ^^ ^^ ^^’ ^^ ^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^ ^ ^^ ^^ ^^ ^^ ^^ ^^ ^^, (5) where ^^ ^^ ^^ ^^ ^^’ ^^ is the integer part of ^^ ^^’ ^^ , ^^ ^^ ^^ ^^ ^^ ^^’ ^^ is the fractional part of ^^ ^^’ ^^ that corresponds to the accuracy level of the motion vector (e.g., 1 / 32thpel), and ^^ ^^ ^^ ^^ ^^ ^^ ^^ is the fractional part of ^^ ^^’ ^^ that corresponds to the accuracy level of the chroma phase vector (e.g.,1 / 64th pel). For example, if ^^ ^^ ^^ ൌ 16 ^ 5 / 32 and ^^ ^^ ^^ ൌ 2 ^ 3 / 64, then ^^ ^^ ^^ ^^ ^^’ ^^ ൌ 16 ^2, ^^ ^^ ^^ ^^ ^^ ^^’ ^^ ൌ 6 / 32, and ^^ ^^ ^^ ^^ ^^ ^^ ^^ ൌ 1 / 64. As mentioned above, for integer-computations,these values should be multiplied according to the considered accuracy, for instance multiplied by 32 for 1 / 32th pel accuracy.

[0065] Thus, the filter coefficients for ^^ ^^ ^^ ^^ ^^ ^^ ^^ can be derived as an interpolation between the coefficients ^^ ^^^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^^^ ^^^ and ^^ ^^^^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^ ^ 1^%32^^ ^^^ , for ^^ ൌ 0.. ^^ െ 1 , ^^ being the number of coefficients of the filters, and “%” being the modulo operator. For instance, if ^^ ^^ ^^ ^^ ^^ ^^ ^^ is represented as fractional value (e.g., 1 / 64), the following formula can be applied: ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^௫^ ^^^ ൌ ^ ^1– ^^ ∙ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ∙ ^^ ^^^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^^^ ^^^ ^^^^ ∙ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ∙ ^^ ^^^^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^ ^ 1^%32^^ ^^^ ^ . (6)The formula of equation (6) can be adapted to employ fixed-point or integer computations. Similarly, horizontal interpolation can be employed as follows:^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^௬^ ^^^ ൌ ^ ^1– ^^ ∙ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ∙ ^^ ^^^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^^^ ^^^ ^^^^ ∙ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ∙ ^^ ^^^^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^ ^ 1^%32^^ ^^^ ^. (7)

[0066] To avoid the interpolation computations (as shown in equations (6) and (7)), a predefined set of interpolated polyphase filters can be precomputed and stored. Based on the chroma phase used for motion compensation, one of the interpolated polyphase filters can then be extracted (by the encoder 400 and / or the decoder 500) from the stored set to be used for resampling of the reference chroma block.

[0067] Alternatively, when both ^^ ^^ ^^ ^^ ^^ ^^ ^^ and ^^ ^^ ^^ ^^ ^^ ^^ ^^ are non-zero, chroma block prediction can be computed by employing four motion compensation operations for the x- fractional and y-fractional positions. Accordingly, deriving: ^^^^with ^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^, ^^ ^^ ^^ ^^ ^^ ^^’ ^^^, ^^^ଶwith ^^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^ ^ 1^%32, ^^ ^^ ^^ ^^ ^^ ^^’ ^^^, ^^^ଷwith ^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^, ^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^ ^ 1^%32^, and ^^^ସwith ^^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^ ^ 1^%32, ^ ^^ ^^ ^^ ^^ ^^ ^^’ ^^ ^ 1^%32^ , the final interpolated block prediction ^^^is derived from these four predictions using, for instance, bilinear interpolation, as illustrated by the following steps: ^^^^ ^^^ ൌ ^ ^1 – ^^ ∙ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ∙ ^^^^^ ^^^ ^ ^ ^^ ∙ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ∙ ^^^ଶ^ ^^^ (8) ^^ଶ^ ^^^ ൌ ^ ^1 – ^^ ∙ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ∙ ^^^ଷ^ ^^^ ^ ^ ^^ ∙ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ∙ ^^^ସ^ ^^^ (9) ^^^^ ^^^ ൌ ^ ^1 – ^^ ∙ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ∙ ^^^^ ^^^ ^ ^ ^^ ∙ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ∙ ^^ଶ^ ^^^ (10) The above steps should be applied for all locations ^^ in the block.

[0068] In an aspect, the interpolated polyphase filters can also be obtained by DCT Interpolation Filter (DCT-IF) approach, that was used for deriving the filters in Tables 1 to 3 (see Kim MJ. et al., Discrete Sine Transform-Based Interpolation Filter for Video Compression, Symmetry 2017, 9, 257). The DCT-IF approach allows to directly derive the filter coefficients as follows. Assume that the values ^ ^^^ ^^^^, ^^ ൌ 0 … ^ ^^ െ 1^ are integer points values of a smooth function x(r). The forward DCT-II provides the transform coefficients set X(k): భ ^^^ ^^^ ൌ ^ଶ ∑ேି^^^ ^^^^^^c^^ାమ^గ^ே^ୀ^ ^ os ^ ே ^ , (11)and the inverse ^^^ ^^^ ൌ ^ଶ ∑ேି^^^ ^^^^^^ ^^ାభమ^గ^ே^ୀ^ ^ cos ^ ே ^ . (12)Combining both ^^^ ^^^ ൌଶே∑ே^ିୀ^^ ^^^ ^^^ ∑ே^ୀି^^^^^ଶcos ^^^ାభమ^గ^ே ^cos ^^^ାభమ^గ^ே ^ , (13) where ckis value p, the N filter coefficients ^^^ ^^^, ^^ ൌ 0.. ^^ െ 1, are therefore directly obtained as: ^^ାభ^గ^ భ^^^ ^^^ ൌଶ∑ேି^^^ଶcos ^మ ^^ାమ^గ^ே^ୀ^ ^ ே^cos ^ே^ . (14)

[0069] In an aspect, a reference block ^^^^^can be computed based on the difference between the luma-chroma phase of the current block, denoted ^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ ^^ ^^ ^^ ^^^, and the luma-chroma phase of the reference block, denoted ^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ ^^ ^^ ^^ ^^^, as follows: ^^ ^^ ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ – ^^ ^^ ^^ ^^ ^^ ^^ (15) ^^ ^^ ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ – ^^ ^^ ^^ ^^ ^^ ^^ (16) The luma-chroma phase of the current block ^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ ^^ ^^ ^^ ^^^ and the luma-chroma phase of the reference block ^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ ^^ ^^ ^^ ^^^ can be provided to a decoder, for example, by signaling in the coded video data bitstream the parameters of a parametric model representative of the luma-chroma phase associated with a video frame or by signaling the luma-chroma phases associated with respective areas within the video frame, as further disclosed below.

[0070] In an aspect, the luma-chroma phase associated with a video frame can be represented by a parametric function whose parameters can be signaled in the bitstream (e.g., at the coded video frame level). For example, the luma-chroma phase of a video frame may be modelled by a linear function ^^^^ ^^, ^^^ or by a quadratic function ^^ொ^ ^^, ^^^, as follows: ^^^^ ^^, ^^^ ൌ ^^ ^ ^^ ∙ ^ ^^ – ^^ ^^^ ^ ^^ ∙ ^ ^^ – ^^ ^^^, or (17)^^ ^ ^^, ^^^ ൌ ^^ ^ ^^ ∙ ^ ^^ – ^^ ^^^ ^ ^^ ∙ ^ ^^ – ^^ ^^^ ^ ^^ ∙ ^ ^^ – ^ ଶ ଶொ ^ ^^^ ^ ^^ ∙ ^ ^^ – ^^ ^^^ ^ ^^ ∙ ^ ^^ – ^^ ^^^ ∙^ ^^ – ^^ ^^^, (18) where ^ ^^ ^^, ^^ ^^^ is a reference point in the frame, for example, the picture’s center location. Note that two parametric functions may be used to model separately the ^^ and the ^^ dimensions of the luma-chroma phase. Furthermore, different parametric functions may be used to model the phase of different color components (e.g., Cb and Cr) relative to the luma component; or the same parametric functions can be used to model the phase for all color components. The parameters of the used model(s) (e.g., parameters a, b, and c of ^^^^ ^^, ^^^ or parameters a, b, c, d, e, and f of ^^ொ^ ^^, ^^^) can be signaled for each picture, intra picture, Rand-Access Point (RAP) picture, or sequence of the encoded video frames. For example, the parameters can be signaled in high level syntax (HLS) structures, such as sequence parameter sets (SPS), picture parameter sets (PPS), Slice header, as well as in adaptation parameter sets (APS), per a tile or per a sub- picture. The parameters can be signaled in the SPS and updated at a lower level (e.g., in the PPS) when necessary (when the parameters are not signaled at a lower level than the parameters signaled in the SPS apply). Alternatively, the latest signaled parameters (in decoding order or in display order) may be applied as long as no new parameters are signaled.

[0071] Alternatively, or in combination, the luma-chroma phase can be regionally signaled, that is, signaled with respect to various areas of a frame. For instance, the luma-chroma phase can be signaled per CTU, per VDPU, per tile, per sub-picture, and / or per slice. As the amplitude of the luma-chroma phase is expected to be small, the coding cost of this information may be limited. Regional signaling of the luma-chroma phase can be performed for each picture, intra picture, RAP picture, or sequence of the encoded video frames. For example, regional signaling can be performed in HLS structures, such as SPS, PPS, Slice header, as well as in APS per a tile or per a sub-picture. Regional signaling can be performed in the SPS and updated at a lower level (e.g., in the PPS) when necessary (when regional parameters are not signaled at a lower level than the parameters signaled in the SPS apply). Alternatively, the latest signaled parameters (in decoding order or in display order) may be applied as long as no new parameters are signaled.

[0072] In an aspect, signaling of luma-chroma phase information can be differential. That is, the luma-chroma phase information can be predictively encoded. Accordingly, the phase information of a current area (e.g., the current block) can be predicted based on phase information derived for a previously coded area from the same frame. The phase information of the current area can also be temporally predicted based on phase information derived for a collocated area from a reference frame or based on phase information derived for an area from the reference frame that is identified by a motion vector obtained for the current area.

[0073] In a variant, a parametric model can be used to initialize the luma-chroma phase information of a frame (or a partition thereof). Thus, the parameters of that model are signaled with respect to the frame (or the frame partition). Then, this initial phase information can be refined – that is, the phase information can be adjusted by a regional (local) phase ^^ ^^ ^^ that is signaled locally (e.g., signaled at a slice or a CTU level). In this variant, the luma-chroma phase can be derived as follows: ^^ ^^ ൌ ^^^^^^, ^^^^ ^^ ^^ ^^ ൌ ^^ ^ ^^ ∙^^^ – ^^ ^^^^ ^^ ∙^^^ – ^^ ^^^^ ^^ ^^ ^^, (19)where the parametric model ^^^^ ^^, ^^^ represents a global part of the phase information and the regional phase dPH represents a local part of the phase information. Similarly, the luma-chroma phase can be derived by ^^ ^^ ൌ ^^ொ^ ^^, ^^^ ^ ^^ ^^ ^^.

[0074] Instead of signaling the luma-chroma phase associated with frames and / or regions of frames (from which the chroma phase can be derived according to equation (15) and (16)), in an aspect, chroma phase information can be inferred as described below with respect to FIG. 9-10.

[0075] FIG. 9 is a diagram illustrating template-based and phase-based motion compensation 900. The motion compensation illustrated in this diagram 900 is similar to the one described with respect to FIG.8. However, in the example of FIG.9, the chroma phase may be computed based on templates (gray areas in FIG. 9). Thus, in an aspect, the chroma phase PH can be computed based on a template (925 or 975) of the current chroma block (respectively, 920 or 970), denoted ^^ ^^ ^^ ^^ ^^ and based on a template (935 or 985) of the reference chroma block (respectively, 930 or 980), denoted ^^ ^^ ^^ ^^ ^^. Both the ^^ ^^ ^^ ^^ ^^ and the ^^ ^^ ^^ ^^ ^^ templates may contain prediction samples or reconstructed samples.

[0076] In a first variant, the chroma phase ^^ ^^ that minimizes a distance metric defined over corresponding template samples is determined. For instance, the sum of absolute difference (SAD) metric can be minimized as follows: ^^ ^^ ^^^ ^^ ^^^ ൌ ^^ ∈ ்ெ | ^^ ^^ ^^ ^^ ^^^^^^െ ^^ ^^ ^^ ^^ ^^^ ^^, ^^ ^^, ^^ ^^^| , (20)where ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^, ^^ ^^, ^^ ^^^ is obtained by applying the displacement vector MV+PH from location p – that is, ^^ ^^ ^^ ^^ ^^^ ^^, ^^ ^^, ^^ ^^^ ൌ ^^^^^^ ^^ ^ ^^ ^^ ^ ^^ ^^^. The optimization process involves a search for the PH that minimizes the distance metric. The searching range is typically small as the phase ^^ ^^ is not expected to be large. For instance, the searching range for ^^ ^^ can be centered around the motion vector MV, extending horizontally and vertically up to a few pixels (e.g., a range from -2 to +2 in pixel units). Other metrics, instead of the SAD metric, may be used to determine ^^ ^^ , such as mean square error (MSE) or sum of absolute transform difference (SATD) metrics. Note that the solution obtained by equation (20) may correspond to a process including: initializing the chroma motion vector ^^ ^^ with the luma motion vector and then refining the chroma motion vector ^^ ^^ using template-matching.

[0077] In a case where the chroma phase is common to all color components, the optimization can take into account the template samples from all components. For instance, optimization may be performed across template samples from the ^^ ^^ and Cr components: ^^ ^^ ^^^ ^^ ^^^ ൌ ^| ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^^ െ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^, ^^ ^^, ^^ ^^^| ^^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^ ^^ , (21) ^^ ^^ ^^ ^^ ^^ ^^ ^^ . And, where ^^ ^^ ^^ ^^ ^^ ^^ ^^ is the template of a current ^^ ^^ block, TMCurCr is the template of a current ^^ ^^ block, TMRefCb is the template of a reference ^^ ^^ block, and ^^ ^^ ^^ ^^ ^^ ^^ ^^ is the template of a reference ^^ ^^ block. In another variant, the chroma phase can be determined based on an optical flow. In this case a chroma phase ^^ ^^ is determined that minimizes an optical flow based metric defined over corresponding template samples. For example, the optical flow ^^ ^^^ ^^^ can be defined as follows: ^^ ^^^^^^ൌ ^^ ^^^^^^∙ ^^ ^^ ^^ ^ ^^ ^^^^^^∙ ^^ ^^ ^^ െ ^^ ^^^ ^^^; (22)^^ ^^^ ^^^ ൌ ^^ ^^ ^^ ^^ ^^^ ^^^ – ^^ ^^ ^^ ^^ ^^^ ^^^, (23) ^^ ^^^ ^^^ ൌ ൫ ^^ ^^ ^^ ^^ ^^^ ^^, ^^^– ^^ ^^ ^^ ^^ ^^^ ^^ െ 1, ^^^൯ ^^ ^^ (24)^^^ ^^ ^^ ^^ ^^^ ^^ ^ 1, ^^^ – ^^ ^^ ^^ ^^ ^^^ ^^ െ 1, ^^^ ^ / 2,^^ ^^^ ^^^ ൌ ൫ ^^ ^^ ^^ ^^ ^^^ ^^, ^^^– ^^ ^^ ^^ ^^ ^^^ ^^, ^^ െ 1^൯ ^^ ^^ (25) where ^ ^^, ^^^ are the coordinates of a sample p. And where the location of a sample p in ^^ ^^ ^^ ^^ ^^ that corresponds to a sample ^^ in ^^ ^^ ^^ ^^ ^^ is obtained by applying a displacement vector MV to the location of the sample ^^ in ^^ ^^ ^^ ^^ ^^ . ^^ ^^^ ^^^ is the difference at location ^^ between ^^ ^^ ^^ ^^ ^^ and ^^ ^^ ^^ ^^ ^^ . And ^^ ^^^ ^^^ and ^^ ^^^ ^^^ are the horizontal and vertical gradients at location ^^ , computed either in the current template ^^ ^^ ^^ ^^ ^^ or in the reference template ^^ ^^ ^^ ^^ ^^ (i.e., in equations (24) and (25), TMCur may be replaced by TMRef). In equation (23), TMCur and TMRef may be pre-filtered for instance using a low-pass filter.

[0078] When there are low brightness variations between the reference samples and the current samples, OF(p) should be very close to zero. Using an optical flow function (as that formulated in equations (22)-(25) for instance) ^^ ^^ ^^ and ^^ ^^ ^^ can be determined as those that minimize an optical flow based metric, such as an MSE metric: ^^ ^^ ^^^ ^^ ^^ ^^, ^^ ^^ ^^^ ൌ ^^ ∈்ெ| ^^ ^^^ ^^^|ଶ, (26) where ^^ ^^is the set of corresponding samples from ^^ ^^ ^^ ^^ ^^ and ^^ ^^ ^^ ^^ ^^. The solution of this minimization can be achieved by solving a two dimensional linear system with PHx and PHy as output of this system. Other metrics can be used instead of the MSE metric, such as the SAD metric.

[0079] In another variant, ^^ ^^ ^^ and ^^ ^^ ^^ can be computed as follows: ^^ ^^ ^^ ൌ ^^ ∙∑ ^ ∈ ^ಾ ௗூ^^^∑^ ^^^ ^^ ^^ ^^^ ^^^ ് 0, ^^ ^^ℎ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ൌ 0, and (27) ^∈ ^ಾ ீ௫ ^ phase is common to both color components, ^^ ^^ and ^^ ^^, the optimization can take into account the samples from the two components. Specifically, the gradients ^^ ^^ and ^^ ^^ , and the difference ^^ ^^ can be computed for template samples from both ^^ ^^ and ^^ ^^.

[0080] In the case where the reference picture ^^^^^and the current picture ^^^௨^^are not of the same resolution, the reference template ^^ ^^ ^^ ^^ ^^ can be first resampled to the resolution of the current template ^^ ^^ ^^ ^^ ^^, before computing the gradients, ^^ ^^ and ^^ ^^, and the difference ^^ ^^ (see equations (23)-(25)), as shown in FIG.10

[0081] FIG. 10 is a flowchart illustrating template-based and phase-based motion compensation using optical flow 1000. Accordingly, in step 1010, a motion vector ^^ ^^ is obtained for a currently encoded chroma block ^^^௨^from ^^^௨^. The obtained motion vector ^^ ^^ identifies a reference chroma block ^^^^^from ^^^^^. When the reference picture ^^^^^and the current picture ^^^௨^^are not of the same resolution (as illustrated in FIG. 6), in step 1020, a reference template ^^ ^^ ^^ ^^ ^^ surrounding the reference chroma block is resampled to match the resolution of a current template ^^ ^^ ^^ ^^ ^^ surrounding the current chroma block. In step 1030, an optical flow is computed. That is, an optical flow ^^ ^^^ ^^^ can be computed according to equations (22)-(25), including deriving parameters ^^ ^^^ ^^^, ^^ ^^^ ^^^, and ^^ ^^^ ^^^ that are obtained with respect to corresponding template samples from ^^ ^^ ^^ ^^ ^^ and ^^ ^^ ^^ ^^ ^^. Then, in step 1040, a chroma phase is determined based on an optical flow based metric – that is, the chroma phase ^^ ^^ that minimizes the optical flow based metric is determined. The minimization of the optical flow based metric can be performed as described above with respect to equation (26).

[0082] In an aspect, the chroma phase used for motion compensation of a current image region can be inferred from the luma-chroma phases of neighboring image regions that were already reconstructed. In this aspect, a luma-chroma phase of a reconstructed region can be determined based on alignment of corresponding luma and chroma samples from the reconstructed region, as further described in reference to FIG.11.

[0083] FIG. 11 is a flowchart illustrating chroma phase derivation based on luma-chroma alignment 1100. In the example of FIG. 11, in step 1110, the chroma phase of a current CTU is derived based on luma-chroma phases computed for previously encoded and reconstructed CTUs, as further described below. Then, in step 1120, the current CTU is encoded and reconstructed, including the application of phase-based motion compensation using the chroma phase derived in step 1110. That is, the derived chroma phase is used to perform motion compensation when encoding content of the current CTU, according to aspects described herein. Next, in step 1130, the luma-chroma phase for the reconstructed current CTU is obtained based on luma-chroma alignment. That is, corresponding chroma samples and luma samples are aligned to obtain a luma-chroma phase for the current CTU, as further described below. The obtained luma-chroma phase for the current CTU is then stored, in step 1140, to be used by CTUs (in the current frame or other frames) to be next encoded that may refer to the current CTU.

[0084] In an aspect, the chroma phase, ^^ ^^ ൌ ^ ^^ ^^ ^^, ^^ ^^ ^^^ , associated with a currently encoded chroma block from a current CTU, can be computed (in step 1110) based on the difference between the luma-chroma phase of reconstructed regions in the vicinity of the current block (e.g., CTUs on top and / or on left of the current CTU), denoted ^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ ^^ ^^ ^^ ^^^ , and the luma-chroma phase of reconstructed reference regions (e.g., CTUs on top and / or on left of the reference CTU), denoted ^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ ^^ ^^ ^^ ^^^, as follows: ^^ ^^ ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ – ^^ ^^ ^^ ^^ ^^ ^^ (29) ^^ ^^ ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ – ^^ ^^ ^^ ^^ ^^ ^^ (30) The luma-chroma phases of the reconstructed regions, ^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ ^^ ^^ ^^ ^^^ and ^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ ^^ ^^ ^^ ^^^, can be obtained based on a luma-chroma alignment, as described next.

[0085] In an aspect, a luma-chroma phase for a reconstructed image area ^^ (e.g., a reconstructed CTU) can be obtained (e.g., in step 1130) by aligning chroma samples and luma samples from that reconstructed area ^^, as follows: ^^ ^^ ^^^ ^^, ^^, ^^ ^^, ^^ ^^^ ൌ ^^௫,௬^ ^^ ^| ^^^ ^^, ^^^ െ ^^ െ ^^ ∙ ^ ^^^^௧^^^^ ^^ ^ ^^ ^^, ^^ ^ ^^ ^^^|ଶ, (31) chroma samples’ values. And where, ^^ ^^ and ^^ ^^ are phase parameters. The luma samples are interpolated (e.g., sup-sampled by two) to match the resolution of the chroma samples and displaced by the phase parameters as expressed by ^^^^௧^^^^ ^^ ^ ^^ ^^, ^^ ^ ^^ ^^^. In an aspect, ^^^^௧^^^^ ^^ ^ ^^ ^^, ^^ ^ ^^ ^^^ can be computed using filters defined for the motion compensation. In practice, the optimization can be done by performing a search over a range of ^^ ^^ and ^^ ^^ values (e.g., in a range from -2 to +2 in pixel units) with a sub-pel step size corresponding to the phase accuracy (for instance 1 / 16thpel). For each of the considered ^^ ^^ and ^^ ^^ values, alignment parameters ^^ and ^^ that minimize ^^ ^^ ^^^ ^^, ^^, ^^ ^^, ^^ ^^^ can be searched within a predetermined range using a predetermined step size within that range. The values for ^^ ^^ and ^^ ^^ that minimize ^^ ^^ ^^^ ^^, ^^, ^^ ^^, ^^ ^^^ are determined as the luma-chroma phase for the image area ^^. The computational complexity of the search process can be reduced by performing a multi-stage search. For example, in an initial stage, an initial range and an initial step size within that range can be used for parameter searching; next, in any successive stage, the range can be centered around the optimal parameters found in the previous stage, and the centered range and / or the step size can be reduced.

[0086] FIG. 12 is a flowchart of an example encoding method, utilizing phase-based motion compensation 1200. The method 1200 encodes video data, including the processing of video blocks from video frames of the video data. The method 1200 includes, in step 1210, obtaining a video block from a video frame of the video data. The method 1200 further includes, in step 1220, coding a chroma block of the video block. The coding of the chroma block involves, in step 1230, performing motion compensation based on a chroma phase, where a chroma phase is a phase between corresponding samples from the chroma block and from a reference chroma block. In an aspect, the phase-based motion compensation is performed by obtaining a chroma motion vector (e.g., derived from a luma motion vector) that identifies the reference chroma block, and then adjusting the chroma motion vector based on the chroma phase (e.g., equations (2)-(4)). In another aspect, the phase-based motion compensation can be performed by resampling the reference chroma block using a polyphase filter that is interpolated based on the chroma phase (e.g., equations (5)-(10)). Alternatively, the resampling of the reference chroma block can be done based on a polyphase filter that is extracted, based on the chroma phase, from a predefined set of interpolated polyphase filters.

[0087] The chroma phase (based on which motion compensation is performed 1230) can be determined based on corresponding template samples from a template surrounding the chroma block and from a template surrounding the reference block (see FIG. 9). The determined chroma phase can be the one that minimizes a distance metric defined over the corresponding template samples (e.g., equations (20)-(21)) or the one that minimizes an optical flow based metric defined over the corresponding template samples (e.g., equations (22)-(28)).

[0088] The chroma phase can also be determined based on a luma-chroma phase associated with the chroma block and a luma-chroma phase associated with the reference chroma block, where a luma-chroma phase is a phase between luma samples and corresponding chroma samples (e.g., equations (15)-(16) or (29)-(30)). In an aspect, the encoder can be configured to signal, in the bitstream of the encoded video data, the luma-chroma phase associated with the chroma block. In another aspect, the luma-chroma phase associated with the video frame (or a partition thereof) can be represented by a parametric model (e.g., equations (17)-(18)). Such parametric representation can be used to locally determine the luma-chroma phase associated with the coded chroma block (e.g., equation (19)). Hence, the parameters of the parametric model can be signaled, by the encoder 400, in a bitstream of the encoded video data, making them available at the decoder 500 end. The encoder can also signal, in the bitstream, a local luma-chroma phase (associated with an area in the video frame including the chroma block) that can be used to refine the luma-chroma phase provided by the parametric model.

[0089] In yet another aspect, the luma-chroma phase associated with the chroma block can be derived, by the encoder 400, based on luma-chroma phase alignment (e.g., equation (31)). The derived luma-chroma phase associated with the chroma block can be stored to be available for the coding of chroma blocks of other video blocks (e.g., as explained in reference to FIG.11).

[0090] FIG. 13 is a flowchart of an example decoding method, utilizing phase-based motion compensation 1300. The decoding method 1300 generally reverses the operation of the encoding method 1200 described above. Hence, the method 1300 decodes video data, including the processing of video blocks from video frames of the video data. The method 1300 includes, in step 1310, obtaining a bitstream that codes a video block from the video frame of the video data. The method 1300 further includes, in step 1320, decoding a chroma block of the video block. The decoding of the chroma block, in step 1330, includes performing motion compensation based on a chroma phase, where a chroma phase is a phase between corresponding samples from the chroma block and from a reference chroma block.

[0091] In an aspect, the phase-based motion compensation is performed by obtaining a chroma motion vector (e.g., derived from a luma motion vector decoded from the bitstream) that identifies the reference chroma block, and then adjusting the chroma motion vector based on the chroma phase (e.g., equations (2)-(4)). In another aspect, the phase-based motion compensation can be performed by resampling the reference chroma block using a polyphase filter that is interpolated based on the chroma phase (e.g., equations (5)-(10)). Alternatively, as mentioned before, the resampling of the reference chroma block can be done based on a polyphase filter that is extracted, based on the chroma phase, from a predefined set of interpolated polyphase filters.

[0092] The chroma phase (based on which motion compensation is performed 1330) can be determined based on corresponding template samples from a template surrounding the chroma block and from a template surrounding the reference block (see FIG. 9). The determined chroma phase can be the one that minimizes a distance metric defined over the corresponding template samples (e.g., equations (20)-(21)) or the one that minimizes an optical flow based metric defined over the corresponding template samples (e.g., equations (22)-(28)).

[0093] The chroma phase can also be determined based on the luma-chroma phase associated with the chroma block and the luma-chroma phase associated with the reference chroma block (e.g., equations (15)-(16) or (29)-(30)). In an aspect, a luma-chroma phase associated with an area in the video frame can be decoded, by the decoder 500, from the bitstream and can be used to determine the luma-chroma phase associated with the chroma block. In another aspect, parameters of a parametric model representing a luma-chroma phase associated with the video frame (or a partition thereof) can be decoded from the bitstream and can be used to determine the luma-chroma phase associated with the chroma block. For example, a local luma-chroma phase (associated with an area in the video frame including the chroma block) can be decoded from the bitstream and can be used to refine the luma-chroma phase provided by the parametric model.

[0094] In yet another aspect, the luma-chroma phase associated with the chroma block can be derived, by the decoder 500, based on luma-chroma phase alignment (e.g., equation (31)). The derived luma-chroma phase associated with the chroma block can be stored to be available for the coding of chroma blocks of other video blocks (e.g., as explained in reference to FIG.11).

[0095] We have described several aspects and embodiments in the present disclosure. These aspects and embodiments provide at least the following outputs and results, including all combinations, across different claim categories and types: ^ Encoding, into coded video data, syntax elements that can enable the decoder to decode the coded video data, according to any of the aspects described herein. ^ A bitstream that includes one or more of the described syntax elements, or variations thereof. A bitstream can be any set of data whether transmitted, stored, or otherwise made available. ^ Creating, transmitting, receiving, and / or decoding of the bitstream. ^ An electronic device (e.g., a TV, a set-top box, a cell phone, or a tablet) that tunes (e.g., using a tuner) a channel to receive the bitstream or that receives (e.g., using an antenna) the bitstream over the air. The electronic device decodes the syntax elements from the bitstream, and, optionally, displays (e.g., using a monitor, screen, or any other type of display) a resulting image. Various other generalized, as well as particularized, outputs, results, implementations, and claims are also supported and contemplated throughout this disclosure.

[0096] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for the proper operation of the method, the order and / or use of specific steps and / or actions can be modified or combined. Additionally, terms such as “first”, “second”, etc. can be used in various embodiments to modify an element, component, step, operation, etc., for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and can occur, for example, before, during, or in an overlapping time period with the second decoding.

[0097] Various methods and other aspects described in this application can be used to modify modules, for example, the modules of the video encoder 200 and the video decoder 300 as shown in FIG.2 and FIG.3. Moreover, the present aspects are not limited to a specific standard (such as VVC or HEVC) and can be applied, for example, to other standards and recommendations, as well as extensions of any such standards and recommendations. Unless indicated otherwise, or technically precluded, the aspects described in this application can be used individually or in combination.

[0098] Various numeric values are used in the present application. The specific values are for example purposes and the aspects described are not limited to these specific values.

[0099] Various implementations involve decoding. “Decoding,” as used in this application, can encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.

[0100] Various implementations involve encoding. In an analogous way to the above discussion about “decoding”, “encoding” as used in this application can encompass all or part of the processes performed, for example, on an input video data in order to produce an encoded bitstream. Additionally, the terms “reconstructed” and “decoded” can be used interchangeably, the terms “encoded” or “coded” can be used interchangeably, and the terms “image,” “picture,” and “frame” can be used interchangeably. Usually, but not necessarily, the term “reconstructed” is used on the encoder side while the term “decoded” is used on the decoder side.

[0101] Note that the syntax elements as used herein are descriptive terms. As such, they do not preclude the use of other syntax element names.

[0102] This disclosure has described various pieces of information, such as for example syntax, that can be transmitted or stored, for example. This information can be packaged or arranged in a variety of manners, including, for example, manners that are common in video standards such as putting the information into an SPS, a PPS, a NAL unit, a header (for example, a NAL unit header, or a slice header), or an SEI message. Other manners are also available, including, for example, manners common for system level or application level standards such as signaling the information into one or more of the following: a. SDP (session description protocol), a format for describing multimedia communication sessions for the purposes of session announcement and session invitation, for example, as described in RFCs and used in conjunction with RTP (Real-time Transport Protocol) transmission. b. DASH MPD (Media Presentation Description) Descriptors, for example, as used in DASH and transmitted over HTTP. A descriptor is associated with a Representation or collection of Representations to provide additional characteristics to the content Representation. c. RTP header extensions, for example, as used during RTP streaming. d. ISO Base Media File Format, for example, as used in OMAF and using boxes which are object-oriented building blocks defined by a unique type identifier and length (also known as 'atoms' in some specifications). e. HLS (HTTP live Streaming) manifest transmitted over HTTP. A manifest can be associated, for example, with a version or collection of versions of content to provide the characteristics of the version or collection of versions.

[0103] The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, an apparatus, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, for example, computers, cell phones, portable / personal digital assistants (PDAs), and other devices that facilitate communication of information between end-users.

[0104] Reference to “one / an aspect” or “one / an embodiment” or “one / an implementation,” as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the aspect / embodiment / implementation is included in at least one embodiment. Thus, the appearances of the phrase “in one / an aspect” or “in one / an embodiment” or “in one / an implementation,” as well any other variations, appearing in various places throughout this application, are not necessarily all referring to the same embodiment.

[0105] Additionally, this application can refer to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0106] Further, this application may refer to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0107] Additionally, this application may refer to “receiving” various pieces of information. Receiving is, as with “accessing,” intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0108] It is to be appreciated that the use of any of the following “ / ”, “and / or”, and “at least one of”, for example, in the cases of “A / B”, “A and / or B” and least one of A and B,” is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This can be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.

[0109] Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments the encoder signals a quantization parameter for de-quantization. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual data, a bit savings is realized in various embodiments. It is to be appreciated that signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0110] As will be evident to one of ordinary skill in the art, implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

Claims

CLAIMS 1. A method for encoding video data, comprising: obtaining a video block from a video frame of the video data; and coding a chroma block of the video block, the coding comprises: performing motion compensation based on a chroma phase, wherein the chroma phase is a phase between corresponding samples from the chroma block and a reference chroma block.

2. The method of claim 1, wherein the performing of the motion compensation further comprising: obtaining a chroma motion vector, identifying the reference chroma block, wherein the chroma motion vector is derived from a luma motion vector.

3. The method of claim 2, wherein the performing of the motion compensation further comprising: adjusting the chroma motion vector based on the chroma phase.

4. The method of claim 2, wherein the performing of the motion compensation further comprising: resampling the reference chroma block using a polyphase filter, interpolated based on the chroma phase.

5. The method of claim 2, wherein the performing of the motion compensation further comprising: resampling the reference chroma block based on a polyphase filter, the poly phase filter is extracted, based on the chroma phase, from a predefined set of interpolated polyphase filters.

6. The method of claim 1, further comprising: determining the chroma phase based on corresponding template samples from a template surrounding the chroma block and from a template surrounding the reference block, wherein the determined chroma phase is the one that minimizes a distance metric defined over the corresponding template samples.

7. The method of claim 1, further comprising: determining the chroma phase based on corresponding template samples from a template surrounding the chroma block and from a template surrounding the reference block, wherein the determined chroma phase is the one that minimizes an optical flow based metric defined over the corresponding template samples.

8. The method of claim 1, further comprising: determining the chroma phase based on a luma-chroma phase associated with the chroma block and a luma-chroma phase associated with the reference chroma block, wherein a luma-chroma phase is a phase between luma samples and corresponding chroma samples.

9. The method of claim 8, further comprising: signaling, in a bitstream of the encoded video data, the luma-chroma phase associated with the chroma block.

10. The method of claim 8, further comprising: representing, by a parametric model, a luma-chroma phase associated with the video frame; and using the parametric representation to determine the luma-chroma phase associated with the chroma block.

11. The method of claim 8, further comprising: signaling, in a bitstream of the encoded video data, parameters of a parametric model representing a luma-chroma phase associated with the video frame.

12. The method of claim 11, further comprising: signaling, in the bitstream, a local luma-chroma phase, wherein the local luma-chroma phase is used to refine the luma-chroma phase associated with the video frame in an area of the video frame including the chroma block.

13. The method of claim 8, further comprising: deriving the luma-chroma phase associated with the chroma block based on a luma-chroma phase alignment, wherein corresponding chroma samples and luma samples from a reconstructed version of the video block are spatially aligned.

14. The method of claim 13, further comprising: storing the derived luma-chroma phase associated with the chroma block to be available for the coding of a chroma block of another video block.

15. A method for decoding video data, comprising: obtaining a bitstream, coding a video block from a video frame of the video data; and decoding a chroma block of the video block, the decoding comprises: performing motion compensation based on a chroma phase, wherein the chroma phase is a phase between corresponding samples from the chroma block and a reference chroma block.

16. The method of claim 15, wherein the performing of the motion compensation further comprising: obtaining a chroma motion vector, identifying the reference chroma block, wherein the chroma motion vector is derived from a luma motion vector decoded from the bitstream.

17. The method of claim 16, wherein the performing of the motion compensation further comprising: adjusting the chroma motion vector based on the chroma phase.

18. The method of claim 16, wherein the performing of the motion compensation further comprising: resampling the reference chroma block using a polyphase filter, interpolated based on the chroma phase.

19. The method of claim 16, wherein the performing of the motion compensation further comprising: resampling the reference chroma block based on a polyphase filter, the poly phase filter is extracted, based on the chroma phase, from a predefined set of interpolated polyphase filters.

20. The method of claim 15, further comprising: determining the chroma phase based on corresponding template samples from a template surrounding the chroma block and from a template surrounding the reference block, wherein the determined chroma phase is the one that minimizes a distance metric defined over the corresponding template samples.

21. The method of claim 15, further comprising: determining the chroma phase based on corresponding template samples from a template surrounding the chroma block and from a template surrounding the reference block, wherein the determined chroma phase is the one that minimizes an optical flow based metric defined over the corresponding template samples.

22. The method of claim 15, further comprising: determining the chroma phase based on a luma-chroma phase associated with the chroma block and a luma-chroma phase associated with the reference chroma block, wherein a luma-chroma phase is a phase between luma samples and corresponding chroma samples.

23. The method of claim 22, further comprising: decoding, from the bitstream, the luma-chroma phase associated with the chroma block.

24. The method of claim 22, further comprising: representing, by a parametric model, a luma-chroma phase associated with the video frame; and using the parametric representation to determine the luma-chroma phase associated with the chroma block.

25. The method of claim 22, further comprising: decoding, from the bitstream, parameters of a parametric model representing a luma- chroma phase associated with the video frame.

26. The method of claim 25, further comprising:decoding, from the bitstream, a local luma-chroma phase, wherein the local luma- chroma phase is used to refine the luma-chroma phase associated with the video frame in an area of the video frame including the chroma block.

27. The method of claim 22, further comprising: deriving the luma-chroma phase associated with the chroma block based on a luma- chroma phase alignment, wherein corresponding chroma samples and luma samples from a reconstructed version of the video block are spatially aligned.

28. The method of claim 27, further comprising: storing the derived luma-chroma phase associated with the chroma block to be available for the coding of a chroma block of another video block.

29. An apparatus for encoding video data, comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the apparatus to: obtain a video block from a video frame of the video data, and code a chroma block of the video block, the coding comprises: performing motion compensation based on a chroma phase, wherein the chroma phase is a phase between corresponding samples from the chroma block and a reference chroma block.

30. The apparatus of claim 29, wherein the performing of the motion compensation further comprising: obtaining a chroma motion vector, identifying the reference chroma block, wherein the chroma motion vector is derived from a luma motion vector, and adjusting the chroma motion vector based on the chroma phase.

31. An apparatus for decoding video data, comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the apparatus to:obtain a bitstream, coding a video block from a video frame of the video data; and decode a chroma block of the video block, the decoding comprises: performing motion compensation based on a chroma phase, wherein the chroma phase is a phase between corresponding samples from the chroma block and a reference chroma block.

32. The apparatus of claim 31, wherein the performing of the motion compensation further comprising: obtaining a chroma motion vector, identifying the reference chroma block, wherein the chroma motion vector is derived from a luma motion vector, and adjusting the chroma motion vector based on the chroma phase.

33. A non-transitory computer-readable medium comprising instructions executable by at least one processor to perform a method for encoding video data, the method comprising: obtaining a video block from a video frame of the video data; and coding a chroma block of the video block, the coding comprises: performing motion compensation based on a chroma phase, wherein the chroma phase is a phase between corresponding samples from the chroma block and a reference chroma block.

34. A non-transitory computer-readable medium comprising instructions executable by at least one processor to perform a method for decoding video data, the method comprising: obtaining a bitstream, coding a video block from a video frame of the video data; and decoding a chroma block of the video block, the decoding comprises: performing motion compensation based on a chroma phase, wherein the chroma phase is a phase between corresponding samples from the chroma block and a reference chroma block.

Images