VIDEO DECODER, METHOD PERFORMED BY A VIDEO DECODER, COMPUTER PROGRAM, AND VIDEO ENCODER - Patent application

Abstract

A method performed by a video decoder includes receiving a coded video bitstream including a current picture, a first reference picture, a second reference picture, and a joint motion vector differential (JMVD) associated with at least one block in the current picture. The method includes determining whether one or more scaling factors are used for a JMVD based on adaptive motion vector differential decomposition (AMVD) based on a predetermined condition. The method includes scaling the JMVD based on the one or more scaling factors in response to determining that one or more scaling factors are used for the JMVD based on the AMVD. The method includes deriving a first MVD of the first reference picture based on the scaled JMVD. The method includes deriving a second MVD of the second reference picture based on the derived first MVD. The method includes reconstructing at least one block based on the derived first and second MVDs.

Description

[Technical field]

[0001] FIELD This disclosure relates generally to communication systems, and more particularly to methods and apparatus for inter-prediction mode dependent scaling factors. [Background technology]

[0002] AOMedia Video 1 (AV1) is an open video coding format designed for video transmission over the Internet. The coding format was developed as a successor to VP9 by the Alliance for Open Media (AOMedia), a consortium founded in 2015 that includes semiconductor companies, video-on-demand providers, video content producers, software developers, and web browser vendors. Many of the components of the AV1 project were sourced from previous research efforts by Alliance members. Individual contributors have been initiating experimental technology platforms for years, with Xipha and Mozilla's Daala having already released their code in 2010, Google's experimental VP9 evolution project, VP10, announced on September 12, 2014, and Cisco's Thor on August 11, 2015. Built on the VP9 code base, AV1 incorporates additional technologies, some of which were developed in these experimental formats. The first version 0.1.0 of the AV1 reference codec was released on April 7, 2016. The Alliance announced the release of the AV1 bitstream specification on March 28, 2018, along with a reference software-based encoder and decoder. The validated version 1.0.0 of the specification was released on June 25, 2018. The validated version 1.0.0, including Errata 1, of the specification was released on January 8, 2019. The AV1 bitstream specification includes a reference video codec. Current coding standards do not properly take into account nonlinear motion between two frames when selecting a scaling factor for the motion vector difference (MVD) applied to the two frames. Summary of the Invention

[0003] The following presents a brief summary of one or more embodiments of the present disclosure in order to provide a basic understanding of such embodiments. This summary is not an exhaustive overview of all contemplated embodiments, and is not intended to identify key or critical elements of all embodiments or to delineate the scope of any and all embodiments. Its sole purpose is to present some concepts of one or more embodiments of the present disclosure in a simplified form as a prelude to the more detailed description that is presented later.

[0004] SUMMARY OF THE DISCLOSURE A method, apparatus, and non-transitory computer-readable medium for inter-prediction mode dependent scaling factors are disclosed in accordance with the present disclosure.

[0005] According to an exemplary embodiment, a method performed by at least one processor of a video decoder includes receiving a coded video bitstream including a current picture, a first reference picture, a second reference picture, and a joint motion vector difference (JMVD) associated with at least one block in the current picture. The method further includes determining whether one or more scaling factors are used for a JMVD based on an adaptive motion vector difference resolution (AMVD) based on a predetermined condition. The method further includes scaling the JMVD based on the one or more scaling factors in response to determining that one or more scaling factors are used for a JMVD based on an AMVD. The method further includes deriving a first MVD of the first reference picture based on the scaled JMVD. The method further includes deriving a second MVD of the second reference picture based on the derived first MVD. The method further includes reconstructing at least one block based on the derived first MVD and the derived second MVD.

[0006] According to an exemplary embodiment, a video decoder includes at least one memory configured to store computer program code and at least one processor configured to access the computer program code and operate as instructed by the computer program code. The computer program code includes a receiving code configured to cause the at least one processor to receive a coded video bitstream including a current picture, a first reference picture, a second reference picture, and a joint motion vector differential (JMVD) associated with at least one block in the current picture. The computer program code further includes a decision code configured to cause the at least one processor to determine whether one or more scaling factors are used in the JMVD based on adaptive motion vector differential decomposition (AMVD) based on a predetermined condition. The computer program code further includes a scaling code configured to cause the at least one processor to scale the JMVD based on the one or more scaling factors in response to determining that one or more scaling factors are used in the JMVD based on the AMVD. The computer program code further includes a first derivation code configured to cause the at least one processor to derive a first MVD of the first reference picture based on the scaled JMVD. The computer program code further includes second derivation code configured to cause the at least one processor to derive a second MVD of the second reference picture based on the derived first MVD. The computer program code further includes reconstruction code configured to cause the at least one processor to reconstruct at least one block based on the derived first MVD and the derived second MVD.

[0007] According to an exemplary embodiment, a non-transitory computer-readable medium stores instructions that, when executed by a processor of a video decoder, cause the processor to perform a method including receiving a coded video bitstream including a current picture, a first reference picture, a second reference picture, and a joint motion vector differential (JMVD) associated with at least one block in the current picture. The method further includes determining whether one or more scaling factors are used for a JMVD based on an adaptive motion vector differential decomposition (AMVD) based on a predetermined condition. The method further includes scaling the JMVD based on the one or more scaling factors in response to determining that one or more scaling factors are used for the JMVD based on the AMVD. The method further includes deriving a first MVD of the first reference picture based on the scaled JMVD. The method further includes deriving a second MVD of the second reference picture based on the derived first MVD. The method further includes reconstructing at least one block based on the derived first MVD and the derived second MVD.

[0008] Additional embodiments will be set forth in the description that follows, and in part will be obvious from the description, and / or may be learned by practice of presented embodiments of the present disclosure.

[0009] The above and other aspects, features, and aspects of embodiments of the present disclosure will become apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic block diagram of a communication system according to various embodiments of the present disclosure. [Diagram 2] FIG. 1 is a schematic block diagram of a communication system according to various embodiments of the present disclosure. [Diagram 3] FIG. 2 is a schematic block diagram of a decoder according to various embodiments of the present disclosure. [Figure 4]FIG. 2 is a block diagram of an encoder according to various embodiments of the present disclosure. [Diagram 5] 1 illustrates an example partition tree according to various embodiments of the present disclosure. [Figure 6] 1A and 1B depict an example quadtree-binary tree (QTBT) structure according to various embodiments of the present disclosure. [Figure 7] 1A and 1B represent a ternary tree (TT) structure according to various embodiments of the present disclosure. [Figure 8] 4 illustrates an example merge mode motion vector difference (MMVD) search point according to various embodiments of the present disclosure. [Figure 9] 1 illustrates a flowchart of an embodiment of a process for determining an aggregate motion vector differential (MVD) scaling factor, according to various embodiments of the present disclosure. [Figure 10] 1 illustrates an example computer system according to various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The following detailed description of the example embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements.

[0012] The above disclosure provides examples and descriptions, and is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. Moreover, one or more features or components of one embodiment may be incorporated or combined with other embodiments (or one or more features of the other embodiments). Moreover, in the flow charts and operational descriptions provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) simultaneously, and the order of one or more operations may be interchanged.

[0013] The systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not intended to limit implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code. It is understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0014] Even if particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim within the scope of the claim.

[0015] No element, act, or instruction used herein should be construed as critical or essential unless expressly described as such. Also, as used herein, the indefinite article ("a" or "an") is intended to include one or more items and may be used synonymously with "one or more." Where only one item is intended, the term "one" or similar language is used. Also, as used herein, terms such as "has," "have," "having," "include," "including," and the like are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood as including only A, only B, or both A and B.

[0016] References herein to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in connection with the indicated embodiment is included in at least one embodiment of the solution. Thus, appearances of the phrases "in one embodiment," "in an embodiment," and similar language herein may, but do not necessarily, all refer to the same embodiment.

[0017] Furthermore, the described features, advantages, and characteristics of the present disclosure may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize in light of the description herein that the present disclosure can be practiced without relying on one or more of the specific features or advantages of a particular embodiment. In other cases, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments of the present disclosure.

[0018] FIG. 1 illustrates a simplified block diagram of a communication system (100) according to an embodiment of the present disclosure. The system (100) may include at least two terminals (110, 120) interconnected via a network (150). In the case of a one-way transmission of data, a first terminal (110) may code video data at a local location for transmission to another terminal (120) via the network (150). The second terminal (120) may receive the coded video data of the other terminal from the network (150), decode the coded data, and display the recovered video data. One-way data transmission may be common in media serving applications, etc.

[0019] 1 illustrates a second pair of terminals (130, 140) provided to support bidirectional transmission of coded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal device (130, 140) may code video data captured at a local location for transmission to the other terminal over the network (150). Each terminal (130, 140) may also receive coded video data transmitted by the other terminal, decode the coded video data, and display the recovered video data on a local display device.

[0020] In FIG. 1, the terminals (110-140) may be represented as servers, personal computers, and smartphones, and / or any other type of terminal. For example, the terminals (110-140) may be laptop computers, tablet computers, media players, and / or dedicated video conferencing devices. The network (150) represents any number of networks that carry coded video data between the terminals (110-140), including, for example, wireline and / or wireless communication networks. The communication network (150) may exchange data in circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of the network (150) may be irrelevant to the operation of the present disclosure, unless otherwise described hereinafter.

[0021] 2 depicts the arrangement of a video encoder and decoder in a streaming environment as an example of an application of the disclosed subject matter, which may be similarly applicable to other video-enabled applications including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0022] As depicted in FIG. 2, the streaming system (200) may include a video source (201) and a capture subsystem (213) that may include an encoder (203). The video source (201) may be, for example, a digital camera and may be configured to generate an uncompressed video sample stream (202). The uncompressed video sample stream (202) may provide a high data volume compared to an encoded video bitstream and may be processed by an encoder (203) coupled to the camera (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (204) may include a lower data volume compared to the sample stream and may be stored in a streaming server (205) for future use. One or more streaming clients (206) may access the streaming server (205) to retrieve a video bitstream (209), which may be a copy of the encoded video bitstream (204).

[0023] In an embodiment, the streaming server (205) may also function as a Media-Aware Network Element (MANE). For example, the streaming server (205) may be configured to prune the encoded video bitstream (204) in order to tailor potentially different bitstreams to one or more streaming clients (206). In an embodiment, a MANE may be separate from the streaming server (205) in the streaming system (200).

[0024] The streaming client (206) may include a video decoder (210) and a display (212). The video decoder (210) may, for example, decode a video bitstream (209), which may be an incoming copy of the encoded video bitstream (204), and generate an outgoing video sample stream (211) that may be rendered on a display (212) or other rendering device (not shown). In some streaming systems, the video bitstreams (204, 209) may be encoded according to a particular video coding / compression standard. An example of such a standard includes, but is not limited to, ITU-T Recommendation H.265. A video coding standard commonly known as Versatile Video Coding (VVC) is under development. Embodiments of the present disclosure may be used in conjunction with VVC.

[0025] 3 illustrates an example functional block diagram of a video decoder (210) attached to a display (212) according to an embodiment of the present disclosure. The video decoder (210) may include a channel (312), a receiver (310), a buffer memory (315), an entropy decoder / parser (320), a scaler / inverse transform unit (351), an intra prediction unit (352), a motion compensated prediction unit (353), an aggregator (355), a loop filter unit (356), a reference picture memory (357), and a current picture memory (358). In at least one embodiment, the video decoder (210) may include an integrated circuit, a series of integrated circuits, and / or other electronic circuitry. The video decoder (210) may also be embodied, in part or in whole, in software running on one or more CPUs with associated memory.

[0026] In this and other embodiments, the receiver (310) can receive one or more coded video sequences to be decoded by the video decoder (210), one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences can be received from a channel (312), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (310) may receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective use entities (not shown). The receiver (310) may separate the coded video sequences from the other data. To combat network jitter, a buffer memory (315) may be coupled between the receiver (310) and the entropy decoder / parser (320) (hereinafter the "parser"). When the receiver 310 is receiving data from a store-and-forward device of sufficient bandwidth and controllability, or from an isosynchronous network, the buffer 315 may not be used or may be small. For use with best-effort packet networks such as the Internet, the buffer 315 may be required and can be relatively large and of adaptive size.

[0027] The video decoder (210) may include a parser (320) that reconstructs symbols (321) from the entropy coded video sequence. These categories of symbols include, for example, information used to manage the operation of the decoder (210) and potentially information for controlling a rendering device, such as a display (212), which may be coupled to the decoder (210) as shown in FIG. 2. The control information for the rendering device may take the form of, for example, a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (320) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow any video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, context-dependent or non-context-dependent arithmetic coding, etc. The parser (320) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (320) may also extract information from the coded video sequence, such as transform coefficients, quantization parameter values, motion vectors, etc.

[0028] The parser (320) may perform an entropy decoding / parsing operation on the video sequence received from the buffer (315) to generate symbols (321). The reconstruction of the symbols (321) may have a number of different units, depending on the type of coded video picture or portion thereof (e.g., inter and intra pictures, inter and intra blocks), and other factors. Which units are included and how may be controlled by subgroup control information parsed by the parser (320) from the coded video sequence. The flow of such subgroup control information between the parser (320) and the following units is not shown for clarity.

[0029] Beyond the functional blocks already mentioned, the decoder (210) may be conceptually subdivided into a number of functional units, which are described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate.

[0030] One unit can be a scalar / inverse transform unit (351), which may receive quantized transform coefficients as symbols (321) from the parser (320) along with control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The scalar / inverse transform unit (351) may output a block containing sample values ​​that can be input to an aggregator (355).

[0031] In some cases, the output samples of the scaler / inverse transformer (351) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from the current (partially reconstructed) picture from the current picture memory (358). The aggregator (355) adds, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (352) to the output sample information provided by the scaler / inverse transformer unit (351).

[0032] In other cases, the output samples of the scaler / inverse transform unit (351) may relate to an inter-coded and potentially motion-compensated block. In such cases, the motion compensated prediction unit (353) may access the reference picture memory (357) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (321) related to the block, these samples may be added by the aggregator (355) to the output of the scaler / inverse transform unit (351) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (357) from which the motion compensated prediction unit (353) fetches the prediction samples may be controlled by a motion vector. The motion vector may be available to the motion compensated prediction unit (353) in the form of a symbol (321), which may have, for example, X, Y, and reference picture components. Motion compensation can also include interpolation of sample values ​​fetched from the reference picture memory (357) when sub-sample accurate motion vectors are in use, as well as motion vector prediction mechanisms.

[0033] The output samples of the aggregator (355) may be subjected to various loop filtering techniques in a loop filter unit (356). Video compression techniques may include in-loop filter techniques that are controlled by parameters included in the coded video bitstream and made available to the loop filter unit (356) as symbols (321) from the parser (320). Note that the video compression may also be responsive to meta-information obtained during the decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and may even be responsive to previously constructed loop filtered sample values.

[0034] The output of the loop filter unit (356) can be a sample stream that can be output to the display (212) or further stored in a reference picture memory (357) for use in future inter-picture prediction.

[0035] Once a particular coded picture is fully reconstructed, it may be used as a reference picture for future prediction. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (320)), the current reference picture may become part of the reference picture memory (357), and any unused current picture memory may be reallocated before beginning reconstruction of a subsequent coded picture.

[0036] The video decoder (210) may perform decoding operations according to a given video compression technique, which may be documented in a standard, such as ITU-T Recommendation H.265. The coded video sequence may conform to a syntax defined by the video compression technique or standard in use, in the sense that the coded video sequence conforms to the syntax of the video compression technique or standard as defined in the video compression technique document or standard, and specifically in a profile document therein. Also, for compliance with some video compression techniques or standards, the complexity of the coded video sequence may be within a boundary defined by a level of the video compression technique or standard. In some cases, the level limits a maximum picture size, a maximum frame rate, a maximum reconstruction sample rate (e.g., measured in megasamples / second), a maximum reference picture size, etc. The limits set by the level may be further limited in some cases through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0037] In an embodiment, the receiver (310) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0038] 4 illustrates an example functional block diagram of a video encoder (203) associated with a video source (201) according to an embodiment of the present disclosure. The video encoder (203) may include an encoder, e.g., a source coder (430), a coding engine (432), a (local) decoder (433), a reference picture memory (434), a predictor (435), a transmitter (440), an entropy coder (445), a controller (450), and a channel (460).

[0039] The encoder (203) may receive video samples from a video source (201) (not part of the encoder) that may capture video images to be coded by the encoder (203). The video source (201) may provide a source video sequence to be coded by the encoder (203) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCB, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (201) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (201) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that impart motion when viewed in sequence. The picture itself may be organized as a spatial array of pixels, and each pixel may contain one or more samples, depending on the sampling structure, color space, etc. in use. Those skilled in the art can readily understand the relationship between pixels and samples. The following description focuses on samples.

[0040] According to an embodiment, the encoder (203) may code and compress pictures of a source video sequence into a coded video sequence (443) in real-time or under any other time constraint required by the application. Imposing an appropriate coding rate is one function of the controller (450). The controller (450) may control and be operatively coupled to other functional units as described below. Coupling is not shown for clarity. Parameters set by the controller (450) may include parameters related to rate control (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. One skilled in the art can readily identify the functions of the controller (450) as they may relate to a video encoder (303) optimized for a particular system design.

[0041] Some video encoders operate in what one skilled in the art would easily recognize as a "coding loop." As an oversimplified description, the coding loop may consist of a source coder (430) (responsible for generating symbols based on an input picture to be coded and reference pictures) and a (local) decoder (433) embedded in the encoder (203). The decoder (433) reconstructs the symbols to generate sample data that a (remote) decoder would also generate when the compression between the symbols and the coded video bitstream is lossless with a particular video compression technique. That reconstructed sample stream may be input to a reference picture memory (434). Since the decoding of the symbol stream produces bit-exact results independent of the location of the decoder (local or remote), the contents of the reference picture memory are also bit-perfect between the local and remote encoders. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder would "see" when using prediction during decoding. This basic principle of reference picture synchronicity (and the resulting drift when synchronicity cannot be maintained, for example due to channel errors) is known to those skilled in the art.

[0042] The operation of the "local" decoder (433) can be the same as the "remote" decoder (210), already described in detail above in conjunction with Figure 3. However, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (445) and parser (320) can be lossless, the entropy decoding portion of the decoder (210), including the channel (312), receiver (310), buffer (315), and parser (320), may not be fully implemented in the local decoder (433).

[0043] An observation that can be made at this point is that any decoder technique, other than parsing / entropy decoding, that is present in the decoder may need to be present in roughly the same functional form in the corresponding encoder. For this reason, the disclosed subject matter focuses on the operation of the decoder. Descriptions of the encoder techniques may be omitted, in that they may be the inverse of the decoder techniques that are generically described. Only to certain extents are more detailed descriptions required and are given below.

[0044] As part of its operation, the source coder (430) may perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from the video sequence designated as "reference frames." In this manner, the coding engine (432) codes differences between pixel blocks of the input frame and pixel blocks of the reference frames that may be selected as predictive references for the input frame.

[0045] The local video decoder (433) may decode the coded video data of frames that may be designated as reference frames based on the symbols generated by the source coder (430). The operation of the coding engine (432) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence is usually a copy of the source video sequence with some errors. The local video decoder (433) may replicate the decoding process that may be performed by the video decoder on the reference frames, such that the reconstructed reference frames are stored in the reference picture memory (434). In this way, the encoder (403) may locally store copies of reconstructed reference frames that have common content with the reconstructed reference frames that would be obtained by the far-end video decoder (without transmission errors).

[0046] The predictor (435) may perform a prediction search for the coding engine (432). That is, for a new frame to be coded, the predictor (435) may look for certain metadata, such as reference picture motion vectors, block shapes, or sample data (as candidate reference pixel blocks) from the reference picture memory (434) that may be suitable prediction references for the new picture. The predictor (435) may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (435), the input picture may have prediction references derived from multiple reference pictures stored in the reference picture memory (434).

[0047] The controller (450) may manage the coding operations of the video coder (430), including, for example, setting parameters and subgroup parameters used to encode the video data. The output of all of the above functional units may undergo entropy coding in the entropy coder (445), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art as Huffman coding, variable length coding, arithmetic coding, etc.

[0048] The transmitter (440) may buffer the coded video sequence generated by the entropy coder (445) to prepare it for transmission over the communication channel (460), which can be a hardware / software link to a storage device that stores the coded video data. The transmitter (440) may merge the coded video data from the video coder (430) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown). The controller (450) may manage the operation of the encoder (303). During coding, the controller (450) may assign a particular coding picture type to each coded picture, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as Intra Picture (I-picture), Predictive Picture (P-picture), or Bi-directionally Predictive Picture (B-picture).

[0049] An Intra picture (I-picture) can be a picture that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow various types of Intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of these variations of I-pictures and their respective applications and characteristics.

[0050] A predictive picture (P-picture) can be a picture that can be encoded and decoded by intra-prediction or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0051] Bidirectionally predictive pictures (B-pictures) can be pictures that can be coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple-predictive picture(s) can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0052] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to each picture of the block. For example, blocks of I pictures may be coded non-predictively or they may be coded predictively with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P pictures may be coded non-predictively or with spatial prediction with reference to one previously coded reference picture or with temporal prediction. Blocks of B pictures may be coded non-predictively or with spatial prediction with reference to one or two previously coded reference pictures or with temporal prediction.

[0053] The video coder (203) may perform coding operations according to a given video coding technique or standard, such as ITU-T Recommendation H.265. During its operation, the video coder (203) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax defined by the video coding technique or standard being used.

[0054] In an embodiment, the transmitter (440) may transmit additional data along with the encoded video. The video coder (430) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0055] Before describing certain aspects of embodiments of the present disclosure in more detail, some terminology that will be referred to in the remainder of the specification is introduced below.

[0056] Hereinafter, a "subpicture" refers to a rectangular arrangement of samples, blocks, macroblocks, coding units, or similar entities that may, in some cases, be semantically grouped and coded independently at a changed resolution. One or more subpictures may form a picture. One or more coded subpictures may form a coded picture. One or more subpictures may be assembled into a picture, and one or more subpictures may be extracted from a picture. In certain circumstances, one or more coded subpictures may be assembled into a coded picture in the compressed domain without transcoding to the sample level, and in the same or certain other cases, one or more coded subpictures may be extracted from a coded picture in the compressed domain.

[0057] "Adaptive Resolution Change" (ARC) hereinafter refers to a mechanism that enables changing the resolution of pictures or sub-pictures in a coded video sequence, e.g., by reference picture resampling. "ARC parameters" hereinafter refer to the control information needed to perform adaptive resolution change, which may include, e.g., filter parameters, scaling factors, output and / or reference picture resolutions, various control flags, etc.

[0058] VP9 uses a 4-way partition tree from the 64x64 level down to the 4x4 level, with some additional restrictions for blocks 8x8 and below, as shown in the top half of Figure 5. Figure 5 shows the partitioning of a 64x64 block (500). The partitioning designated as R refers to recursive partitioning, where the same partition tree is repeated at lower scales until the lowest 4x4 level is reached.

[0059] AV1 not only extends the partition tree to the 10-way structure shown in Figure 5, but also increases the maximum size (called superblocks in VP9 / AV1) to start with 128x128 blocks (502). This partitioning includes 4:1 / 1:4 rectangular partitions, which did not exist in VP9. None of the rectangular partitions can be further subdivided. Moreover, AV1 adds more flexibility in the use of partitions below the 8x8 level, in the sense that 2x2 chroma inter prediction is now possible in certain cases.

[0060] In HEVC, a coding tree unit (CTU) may be divided into coding units (CUs) by using a quadtree structure represented as a coding tree to adapt to various local characteristics. A decision on whether to code a picture area using inter-picture (temporal) prediction or intra-picture (spatial) prediction may be made at the CU level. Each CU may be further divided into one, two, or four prediction units (PUs) according to a PU partition type. Within one PU, the same prediction process may be applied, and related information may be transmitted to the decoder on a PU-by-PU basis. After obtaining a residual block by applying a prediction process based on a PU partition type, the CU may be partitioned into transform units (TUs) according to another quadtree structure, such as a coding tree of a CU. One of the important features of the HEVC structure is that it has a multi-partition concept, including CUs, PUs, and TUs. In HEVC, a CU or TU can only have a square shape, while a PU can be square or rectangular in case of an inter-predicted block. In HEVC, a coding block may be further divided into four sub-blocks, and a transform may be performed on each sub-block (e.g., TU). Each TU may be further divided recursively (e.g., by quad-tree partitioning) into smaller TUs called Residual Quad-Tree (RQT). At picture boundaries, HEVC may use implicit quad-tree partitioning, whereby a block continues to be quad-tree partitioned until its size fits the picture boundary.

[0061] In HEVC, CTUs may be divided into CUs by using a quadtree structure represented as a coding tree to adapt to different local characteristics. The decision on whether to code a picture area using inter-picture (temporal) prediction or intra-picture (spatial) prediction may be made at the CU level. Each CU may be further divided into one, two, or four PUs according to the PU partition type. Within one PU, the same prediction process may be applied, and related information may be transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partition type, the CU may be partitioned into transform units (TUs) according to another quadtree structure, such as the coding tree of the CU. One of the important features of the HEVC structure is that it has a multi-partition concept, including CUs, PUs, and TUs.

[0062] The QTBT structure can eliminate the concept of multiple partition types (e.g., the QTBT structure removes the separation of the concepts of CU, PU, ​​and TU), and supports more flexibility in CU partition shapes. In the QTBT block structure, CUs can be either square-shaped or rectangular. As shown in Fig. 6 (A) and (B), a coding tree unit (CTU) can be first partitioned by a quadtree structure. The quadtree leaf nodes can be further partitioned by a binary tree structure. There are two partition types in the binary tree partition: symmetric horizontal partition and symmetric vertical partition. The binary tree leaf nodes can be called coding units (CUs), and their segmentation can be used for prediction and transform processes without any further partitioning. This means that CUs, PUs, and TUs have the same block size in the QTBT coding block structure. In JEM, a CU sometimes consists of coding blocks (CBs) of different color components. For example, a CU includes one luma CB and two chroma CBs in the case of P slices and B slices in 4:2:0 chroma format, and sometimes consists of a single component CB. For example, a CU includes only one luma CB or only two chroma CBs in the case of an I slice.

[0063] The following parameters are defined for the QTBT partitioning scheme: CTU size: Root node size of the quadtree, same concept as HEVC MinQTSize: The minimum allowable quadtree leaf node size. MaxBTSize: Maximum allowed binary tree root node size MaxBTDepth: Maximum allowed binary tree depth MinBTSize: The minimum allowable binary tree leaf node size.

[0064] In one example of a QTBT partitioning structure, the CTU size may be set as 128×128 luma samples with two corresponding 64×64 blocks of chroma samples, MinQTSize may be set as 16×16, MaxBTSize may be set as 64×64, MinBTSize (for both width and height) may be set as 4×4, and MaxBTDepth may be set as 4. Quad-tree partitioning may first be applied to the CTU to generate quad-tree leaf nodes. The quad-tree leaf nodes may have a size from 16×16 (e.g., MinQTSize) to 128×128 (e.g., CTU size). If the leaf quad-tree node is 128×128, the node cannot be further divided by the binary tree because the size exceeds MaxBTSize (e.g., 64×64). If not, the leaf quad-tree node may be further partitioned by the binary tree. Thus, the quadtree leaf node becomes the root node of the binary tree with binary tree depth 0. When the binary tree depth reaches MaxBTDepth (e.g., 4), no further splits are considered. If a binary tree node has a width equal to MinBTSize (e.g., 4), no further horizontal splits are considered. Similarly, if a binary tree node has a height equal to MinBTSize, no further vertical splits are considered. The leaf node of the binary tree is further processed by the prediction and transform process without any further partitioning. In JEM, the maximum CTU size is 256x256 luma samples.

[0065] Fig. 6(A) shows an example of block partitioning by using QTBT, and Fig. 6(B) shows the corresponding tree representation. Solid lines indicate quadtree partitioning, and dotted lines indicate binary tree partitioning. At each partition (e.g., non-leaf) node of the binary tree, one flag may be signaled to indicate which partition type (e.g., horizontal or vertical) is used. 0 indicates horizontal partitioning, and 1 indicates vertical partitioning. In the case of quadtree partitioning, there is no need to indicate the partition type, since the quadtree partitioning always partitions a block both horizontally and vertically to generate four sub-blocks of equal size.

[0066] Furthermore, the QTBT scheme may support the flexibility for luma and chroma to have separate QTBT structures. Currently, for P slices and B slices, the luma CTB and chroma CTB in one CTU share the same QTBT structure. However, for I slices, the luma CTB may be partitioned into CUs by a QTBT structure, and the chroma CTB may be partitioned into chroma CUs by another QTBT structure. This means that a CU in an I slice may consist of a coding block of a luma component or a coding block of two chroma components, and a CU in a P slice or B slice may consist of coding blocks of all three color components.

[0067] In HEVC, inter prediction of small blocks may be restricted to reduce memory accesses for motion compensation, so that bi-prediction is not supported for 4x8 and 8x4 blocks, and inter prediction is not supported for 4x4 blocks. In QTBT implemented in JEM-7.0, these restrictions may be removed.

[0068] In VVC, a Multi-type-tree (MTT) structure is included, which further adds horizontal and vertical center-side ternary trees in addition to the QTBT, as shown in Figures 7(A) and (B).

[0069] The important advantages of ternary tree partitioning are: (i) the complementarity of quadtree and binary tree partitioning: ternary tree partitioning can capture objects located at block centers, while quadtree and binary trees always split along block centers, and (ii) the width and height of the proposed ternary tree partitions are always powers of two, so no additional transformations are required. The design of the two-level tree is primarily motivated by reduced complexity. The complexity of traversing the tree is T D where T represents the number of split types and D is the tree depth.

[0070] In addition to the merge mode in which the implicitly derived motion information is directly used to generate the prediction samples of the current CU, a merge mode with motion vector differences (MMVD) is introduced in VVC. The MMVD flag can be signaled immediately after sending the skip flag and the merge flag to specify whether the MMVD mode is used for the CU. In MMVD, after a merge candidate is selected, the merge candidate is further refined by the signaled MVD information. This further information can include a candidate flag, an index specifying the magnitude of the motion, and an index indicating the direction of the motion. In the MMVD mode, one of the first two candidates in the merge list can be selected to be used as the MV basis. A merge candidate flag can be signaled to specify which one is used.

[0071] The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. As shown in Figure 8, the offset is added to either the horizontal or vertical component of the starting MV in the L0 reference picture (800) and the L1 reference picture (802). The relationship between the distance index and the predefined offset is specified in Table 1. [Table 1]

[0072] The direction index can represent the direction of the MVD relative to the starting point. The direction index can represent one of four directions shown in Table 2, which indicates the sign of the MV offset specified by the direction index. [Table 2]

[0073] The meaning of the MVD code can be various according to the information of the start MV. When the start MV is a uni-predictive MV or a bi-predictive MV with both lists pointing to the same side of the current picture (e.g., the POC of two references are both larger than the POC of the current picture, or both smaller than the POC of the current picture), the code in Table 2 may specify the sign of the MV offset added to the start MV. When the start MV is a bi-predictive MV with two MVs pointing to different sides of the current picture (e.g., the POC of one reference is larger than the POC of the current picture, and the POC of the other reference is smaller than the POC of the current picture), and the difference of the POC in list 0 is larger than the POC in list 1, the code in Table 2 can specify the sign of the MV offset added to the list 0 MV component of the start MV, and the code of the list 1 MV has the opposite value. Otherwise, if the difference in POC in List 1 is greater than List 0, then the code in Table 2 may specify the encoding of the MV offset added to the List 1 MV component of the starting MV, and the code of the List 0 MV has the opposite value.

[0074] The MVD may be scaled according to the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling is necessary. Otherwise, if the difference in POC in list 0 is greater than that of list 1, the MVD of list 1 may be scaled. If the POC difference of L1 is greater than that of L0, the MVD of list 0 may be scaled in the same way. If the starting MV is one-way predicted, the MVD may be added to the available MV.

[0075] In VVC, in addition to the normal unidirectional and bidirectional predictive mode MVD signaling, a symmetric MVD mode for bidirectional MVD signaling may be applied. In the symmetric MVD mode, the motion information including the reference picture indexes of both list 0 and list 1 and the MVD of list 1 is not signaled, but may be derived instead. The decoding process of the symmetric MVD mode is as follows:

[0076] First, at the slice level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 may be derived as follows: If mvd_l1_zero_flag is 1, BiDirPredFlag is set equal to 0. Otherwise, if the closest reference picture in list 0 and the closest reference picture in list 1 form a forward and backward pair of reference pictures or a backward and forward pair of reference pictures, BiDirPredFlag is set to 1 and both reference pictures in list 0 and list 1 are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0.

[0077] Second, at the CU level, a symmetric mode flag indicating whether symmetric mode is used or not is explicitly signaled if the CU is bi-predictively coded and BiDirPredFlag is equal to 1.

[0078] When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD0 may be explicitly signaled. The reference indexes of list 0 and list 1 may be set equal to a pair of reference pictures, respectively. MVD1 may be set equal to (-MVD0).

[0079] In AV1, for each coded block in an inter frame, another flag may be signaled to indicate whether a single or mixed reference mode is used for the current block if the mode of the current block is an inter coding mode rather than a skip mode. In a single reference mode, a predictive block may be generated by one motion vector, while in a mixed reference mode, a predictive block may be generated by a weighted average of two predictive blocks derived from two motion vectors.

[0080] In the case of a single reference, one or more of the following modes may be signaled: Use one of the motion vector predictors (MVP) in the list indicated by the NEARMV-DRL (Dynamic Reference List) index. · Use one of the motion vector predictors (MVPs) in the list signaled by the NEWMV-DRL index as a reference and apply the delta to the MVP. · GLOBALMV - Use motion vectors based on frame-level global motion parameters.

[0081] In case of mixed reference mode, the following modes can be signaled: · NEAR_NEARMV-Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index. · NEAR_NEWMV-Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference to transmit the delta MV of the second MV. · NEW_NEARMV-Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference and send the delta MV of the first MV. · NEW_NEWMV-Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference and send the delta MV for both MVs. · GLOBAL_GLOBALMV - Use the MV from each reference based on their frame-level global motion vectors.

[0082] AV1 allows for 1 / 8 pixel motion vector precision (or accuracy), and the following syntax can be used to signal motion vector differentials within reference frame list 0 or list 1: mv_joint specifies which components of the motion vector differential are non-zero. 0 indicates that there is no non-zero MVD along either the horizontal or vertical direction. 1 indicates that there is non-zero MVD only along the horizontal direction. 2 indicates that there is non-zero MVD only along the vertical direction. 3 indicates there is non-zero MVD along both the horizontal and vertical directions. · mv_sign specifies whether the motion vector differential is positive or negative. · mv_class specifies the class of the motion vector differential. · mv_bit specifies the integer part of the offset between the motion vector differential and the starting magnitude of each MV class. · mv_fr specifies the first two fractional bits of the motion vector differential. · mv_hp specifies the third fractional bit of the motion vector differential.

[0083] As shown in Table 3, Table 3 indicates the magnitude classes of the MVD, the higher the class, the larger the magnitude the motion vector differential will have. [Table 3]

[0084] For NEW_NEARMV and NEAR_NEWMV modes, the accuracy of the MVD may depend on the associated class and the magnitude of the MVD. First, fractional MVD is only allowed if the magnitude of the MVD is less than or equal to 1 pixel. Second, only one MVD value is allowed if the value of the associated MV class is MV_CLASS_1 or greater, and the MVD value at each MV class is derived as 4, 8, 16, 32, 64 for MV classes 1 (MV_CLASS_1), 2 (MV_CLASS_2), 3 (MV_CLASS_3), 4 (MV_CLASS_4), or 5 (MV_CLASS_5). Furthermore, one context may be used to signal mv_joint or mv_class if the current block is coded as NEW_NEWMV or NEAR_NEWMV mode. Otherwise, the other context may be used to signal mv_joint or mv_class.

[0085] The permitted MVD values ​​for each MV class are given in Table 4. [Table 4]

[0086] A new inter-coding mode called JOINT_NEWMV may be applied to indicate whether the MVDs of two reference lists are signaled together. When the inter-prediction mode is equal to JOINT_NEWMV mode, the MVDs of reference list 0 and reference list 1 may be signaled together. Therefore, under this mode, only one MVD called joint_mvd is signaled and transmitted to the decoder, and the delta MVs of reference list 0 and reference list 1 may be derived from joint_mvd. The JOINT_NEWMV mode may be signaled together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes. No additional context is added.

[0087] When JOINT_NEWMV mode is signaled and the POC distances between the two reference frames and the current frame are different, the MVD may be scaled for reference list 0 or reference list 1 based on the POC distances. Specifically, the distance between reference frame list 0 and the current frame is denoted as td0, and the distance between reference frame list 1 and the current frame is denoted as td1. If td0 is greater than or equal to td1, joint_mvd may be used directly for reference list 0, and the mvd of reference list 1 may be derived from joint_mvd as follows:

number

[0088] Otherwise, if td1 is greater than or equal to td0, then joint_mvd is used directly for reference list 1, and the mvd for reference list 0 is derived from joint_mvd as follows:

number

[0089] A new inter-coding mode called AMVDMV is added to the single reference case. When the AMVDMV mode is selected, this mode may indicate that the AMVD is applied to the single MVD. One flag called amvd_flag may be added under the JOINT_NEWMV mode to indicate whether the AMVD is applied to the joint MVD coding mode. When the adaptive MVD decomposition is applied to the joint MVD coding mode, also called the joint AMVD coding mode, the MVDs of the two reference frames may be signaled together, and the accuracy of the MVD may be implicitly determined by the size of the MVD. Otherwise, the MVDs of the two (or more than two) reference frames may be signaled together, and conventional MVD coding is applied.

[0090] AMVR was first proposed in CWG-C012, where a total of seven MV precisions (8, 4, 2, 1, 1 / 2, 1 / 4, 1 / 8) are supported. For each precision block, the AVM encoder explores all supported precision values ​​and signals the best precision to the decoder. To reduce the encoder runtime, two precision sets are supported. Each precision set may contain four predefined precisions. The precision set may be adaptively selected at the frame level based on the maximum precision value of the frame. As with AV1, the maximum precision may be signaled in the frame header. Table 5 summarizes the supported precision values ​​based on the frame-level maximum precision. [Table 5]

[0091] In the current AMV software (similar to AV1), there is a frame level flag to indicate whether the MV of a frame contains sub-pel precision or not. AMVR is only enabled if the cur_frame_force_integer_mv flag has a value of 0. In AMVR, if the precision of a block is less than the maximum precision, the motion model and the interpolation filter are not signaled. If the precision of a block is less than the maximum precision, the motion mode can be inferred as translation motion and the interpolation filter can be inferred as a REGULAR interpolation filter. Similarly, if the precision of a block is either 4-pel or 8-pel, the inter-intra mode is not signaled and can be inferred as 0.

[0092] For joint MVD coding mode (JMVD), there is an assumption that there is linear motion between the backward reference frame and the forward reference frame. When joint MVD coding mode is selected for one block, one joint MVD can be signaled for both reference frames, and the MVDs of the two reference frames can be derived from the joint MVD based on the distance between the reference frame and the current frame. However, the motion between the two reference frames may not always be linear. For example, the motion may be slower or faster from the backward reference frame to the forward reference frame.

[0093] For AMVD-based JMVD, MVD can only be applied along either the horizontal or vertical direction. However, for conventional JMVD, MVD can be along both the horizontal and vertical directions. Therefore, sharing the same scaling factor between AMVD-based JMVD and conventional JMVD is suboptimal.

[0094] The embodiments of the present disclosure are directed to a set of advanced image and video coding techniques. More specifically, the embodiments of the present disclosure are directed to inter-prediction mode dependent scaling factors. The embodiments of the present disclosure may be used separately or combined in any order. Furthermore, each of the embodiments utilizing an encoder or a decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-transitory computer-readable medium. Hereinafter, the term block may be interpreted as a prediction block, a coding block, or a coding unit (i.e., CU). The term block may also be used to refer to a transform block.

[0095] The orientation of the reference frame may be determined by whether the reference frame is before the current frame in standard order or after the current frame in standard order. The terms x-axis and y-axis may refer to the horizontal and vertical components of a two-dimensional value, and may be replaced by two other axes along two predefined directions that are orthogonal to each other, and the same embodiment is applicable to these other axes. For example, the x-axis and y-axis may be replaced by 45 degree and 135 degree axes, respectively. Conventional JMVD may refer to JMVD by normal full MV decomposition, or JMVD by AMVR.

[0096] In some embodiments, if MVDs are jointly signaled for multiple reference frames in one block, flexible scaling factors are used to derive MVDs for reference frame lists 0 and / or 1 from the signaled joint MVD based on the signaled scaling factors in the bitstream or the coded information of the current block (or neighboring blocks). The allowed / supported scaling factors may be different for JMVD based on AMVD and conventional JMVD with different MV precision.

[0097] In some embodiments, the values ​​of the scaling factors are restricted to powers of two, such as 1, 2, 4, 1 / 2, 1 / 4, etc.

[0098] In some embodiments, for JMVD modes based on AMVD, the same scaling factor is applied to both the vertical and horizontal components of the MVD of reference frame lists 0 and / or 1.

[0099] In some embodiments, one predefined lookup table may be used to store the supported / allowed scaling factors, and the associated entry index for the selected scaling factor in the lookup table is signaled in the bitstream. Table 6 represents an example of a lookup table that specifies the scaling factor for each index. [Table 6]

[0100] In some embodiments, the AMVD JMVD scaling factors may be derived from the conventional JMVD scaling factors, i.e., if AMVD is applied along the x-axis, the conventional JMVD x-axis scaling factor is used, otherwise, if AMVD is applied along the y-axis, the conventional JMVD y-axis scaling factor is used.

[0101] In some embodiments, the AMVD JMVD scaling coefficients may be a subset of scaling coefficients extracted from conventional JMVD. For example, if conventional JMVD mode is selected for a block, different scaling coefficients may be applied separately to the vertical or horizontal components of the MVD of reference frame lists 0 and / or 1.

[0102] In some embodiments, one predefined lookup table may be used to store supported / allowed scaling factor pairs for the vertical and horizontal components of MVD in conventional JMVD mode, and the associated entry index for the selected scaling factor pair in the lookup table may be signaled in the bitstream. Table 7 shows an example of a table where separate scaling factors are specified for each of the x-axis and y-axis. [Table 7]

[0103] In some embodiments, the scaling factor of one component of the MVD (either the vertical or horizontal component) may be restricted to be 1, while the scaling factor of the other component of the MVD can be other values ​​such as 2 or ½. The supported scaling factor pairs are also stored in one lookup table, and the associated entry index for the selected scaling factor pair in the lookup table is signaled in the bitstream. Table 8 represents an example of a lookup table where the scaling factor for one of the x-axis and y-axis is restricted to 1. [Table 8]

[0104] In some embodiments, the context for signaling the scaling factor may depend on the inter-prediction mode information of the current block. For example, if the current block is coded as a JMVD mode based on AMVD, one context may be used to signal the scaling factor of the current block. Otherwise, another context may be used to signal the scaling factor of the current block. In another example, if the current block is coded as a conventional JMVD mode, one context may be used to signal the scaling factor of the current block when AMVR is applied, and another context is used when AMVR is not applied.

[0105] In some embodiments, the context for signaling the scaling factor depends on the distance between the current frame and the reference frame of the current frame. For example, if the distance between the reference frame and the current frame is equal to or greater than a threshold, one context may be used. Otherwise, another context may be used. As an example, the threshold is set to 4 or 8.

[0106] In some embodiments, the MV precision of the current block may be used as a context for signaling the scaling factor. For example, one context may be used to signal the index of the scaling factor if the MV precision is equal to or greater than a threshold value. If not, another context may be used instead. As an example, the threshold value may be set to 1 pel.

[0107] In some embodiments, the indices of the scaling factors may be signaled only if the current block is coded as a JMVD mode based on AMVD. In some embodiments, the indices of the scaling factors are signaled only if optical flow refinement is applied to the current block. In some embodiments, if the indices of the scaling factors indicate that at least one of the scaling factors is not equal to 1, the interpolation filter type is not signaled in the bitstream and is derived as one default interpolation filter type. For example, the default interpolation filter type may be one of the REGULAR, SMOOTH, or SHARP filters defined in AV1.

[0108] In some embodiments, if the index of the scaling factor indicates that at least one of the scaling factor is not equal to 1, then only a subset of the MV precisions may be allowed or signaled in the bitstream. For example, if the index of the scaling factor indicates that at least one of the scaling factor is not equal to 1, then the MV precision of the current block is finer than one threshold, such as 2 pels. In other examples, if the index of the scaling factor indicates that at least one of the scaling factor is not equal to 1, then the MV precision of the current block is not signaled and is derived as one default value. In other examples, if the index of the scaling factor indicates that at least one of the scaling factor is not equal to 1, then the MV precision of the current block is not signaled and may be derived as a frame-level MV precision or a superblock-level MV precision. In some embodiments, the index of the scaling factor may be used as a context for signaling the MV precision of the current block when the current block is coded as a joint MVD mode.

[0109] 9 depicts a flowchart of an embodiment of a process (900) for determining a scaling factor for a joint MVD. The process (900) may be performed by a decoder, such as the decoder (210). The process may begin with an operation (902) in which a coded video bitstream is received. The bitstream may include a current picture, a first reference picture, a second reference picture, and a joint MVD associated with at least one block in the current picture.

[0110] The process may proceed to operation (904) where it is determined whether one or more scaling factors are to be used for the AMVD-based JMVD based on a predetermined condition, such as a flag included in the received bitstream. The process may proceed to operation (906) where, in response to determining that one or more scaling factors are to be used for the AMVD JMVD, the JMVD is scaled based on the one or more scaling factors. As an example, the scaling factors are determined based on an index value included in the bitstream that indexes into a lookup table. The lookup table may specify scaling factors to be applied to both the x-axis and the y-axis, or the lookup table may specify separate scaling factors to be applied to both the x-axis and the y-axis.

[0111] The process proceeds to operation (908), where a first MVD of the first reference picture is derived based on the scaled joint MVD. As an example, the first MVD may be determined as shown in equation (1). The process proceeds to operation (910), where a second MVD of the second reference picture is derived based on the derived first MVD. As an example, the second MVD may be determined as shown in equation (2). The process proceeds to operation (912), where at least one block is decoded based on the derived first MVD and the derived second MVD.

[0112] The techniques of the embodiments of the present disclosure described above may be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 10 shows a computer system (1000) suitable for implementing embodiments of the disclosed subject matter.

[0113] The computer software may be coded in any suitable machine code or computer language that may be subject to mechanisms such as assembly, compilation, linking, etc. to generate code containing instructions that may be executed by a central processing unit (CPU), graphics processing unit (GPU), etc. directly, or through interpretation, microcode execution, etc.

[0114] The instructions may be executed in various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming consoles, Internet of Things devices, and the like.

[0115] 10 for computer system (1000) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components described in the exemplary embodiment of computer system (1000).

[0116] The computer system (1000) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more users through, for example, tactile input (e.g., keyboard, swipes, dataglove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0117] The input human interface devices may include one or more of a keyboard (1001), a mouse (1002), a trackpad (1003), a touch screen (1010), a data glove, a joystick (1005), a microphone (1006), a scanner (1007), and a camera (1008) (only one of each is shown).

[0118] The computer system (1000) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the user's senses through, for example, haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1010), data gloves, or joystick (1005), although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (1009), headphones (not shown)), visual output devices (e.g., screens (1010) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may provide two-dimensional visual output or output in more than three dimensions by means of stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0119] The computer system (1000) may also include human accessible storage devices and their associated media, such as CD / DVD or similar media (1021), including CD / DVD ROM / RW (1020), thumb drives (1022), removable hard disks or solid state drives (1023), legacy magnetic media, such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices, such as security dongles (not shown), and the like.

[0120] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0121] The computer system (1000) may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wireline, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and factory networks including CAN bus, etc. Certain networks generally require an external network interface adapter attached to a particular general purpose digital port or peripheral bus (1049) (e.g., a USB port of the computer system (1000)). Others are generally built into the core of the computer system 1000 by attachment to a system bus as described below (e.g., an Ethernet network to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1000) may communicate with other entities. Such communications may be one-way receive-only (e.g., broadcast TV) or one-way transmit-only (e.g., a CAN bus to a particular CAN bus device), or may be two-way, for example, to other computer systems using local or wide area digital networks. Such communications may include communications to a cloud computing environment (1055). Specific protocols or protocol stacks may be used with each of the networks and network interfaces as described above.

[0122] The above-mentioned human interface devices, human accessible storage devices, and network interface (1054) may be attached to the core (1040) of the computer system (1000).

[0123] The cores (1040) may include one or more central processing units (CPUs) (1041), graphics processing units (GPUs) (1042), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1043), hardware accelerators for specific tasks (1044), etc. These devices may be connected through a system bus (1048) along with read only memory (ROM) (1045), random access memory (RAM) (1046), internal mass storage such as an internal non-user accessible hard drive, SSD, etc. (1047). In some computer systems, the system bus (1048) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals may be attached directly to the core's system bus (1048) or through a peripheral bus (1049). Architectures for peripheral buses include PCI, USB, etc. A graphics adapter (1050) may be included in the core (1040).

[0124] The CPU (1041), GPU (1042), FPGA (1043), and accelerator (1044) can execute certain instructions that, in combination, may constitute the above computer code. The computer code may be stored in a ROM (1045) or a RAM (1046). Temporary data may also be stored in the RAM (1046), while persistent data may be stored, for example, in an internal mass storage device (1047). Rapid storage and retrieval from any of the memory devices may be made possible by the use of a cache memory. A cache memory may be closely associated with one or more of the CPU (1041), GPU (1042), mass storage device (1047), ROM (1045), RAM (1046), etc.

[0125] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those having skill in the computer software arts.

[0126] By way of example, and not by way of limitation, a computer system having the architecture (1000), and in particular the core (1040), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with the user-accessible mass storage devices introduced above, in addition to specific storage devices of the core (1040) that are non-transitory in nature, such as the core's internal mass storage device (1047) or ROM (1045). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1040). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (1040), and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (1046) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system can provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerators (1044)) that can operate in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. References to software can encompass logic, where appropriate, and vice versa. References to computer-readable media can encompass circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both, where appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0127] The above disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0128] It is understood that the particular order or hierarchy of blocks in the processes / flowcharts disclosed herein is an illustration of an example approach. Based on design preferences, it is understood that the particular order or hierarchy of blocks in the processes / flowcharts may be rearranged. Further, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order, and are not meant to be limited to the particular order or hierarchy presented.

[0129] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail of integration. Furthermore, one or more of the above components may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or medium) having computer-readable program instructions for causing a processor to perform operations.

[0130] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk-read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or raised structures in grooves with instructions recorded thereon, and any suitable combination thereof. A computer readable storage medium as used herein should not be construed as being a transitory signal per se, such as an electric signal transmitted through a wire, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse traveling through a fiber optic cable), or an electric signal transmitted through a wire.

[0131] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0132] The computer readable program code / instructions for performing the operations may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages ​​including object oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network such as a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform an aspect or operation.

[0133] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, generate means for implementing the functions / acts specified in the flowcharts and / or one or more blocks of the block diagrams. These computer readable program instructions can be stored on a computer readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer readable storage medium on which the instructions are stored constitutes an article of manufacture including instructions that implement aspects of the functions / acts specified in the flowcharts and / or one or more blocks of the block diagrams.

[0134] The computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / operations specified in the flowcharts and / or one or more blocks of the block diagrams.

[0135] The flowcharts and block diagrams in the figures represent the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical functions. The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks compared to those shown in the figures. In some alternative implementations, the functions shown in the blocks may be performed in a different order than that shown in the figures. For example, two blocks shown in succession may in fact be executed simultaneously or nearly simultaneously, or the blocks may be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, as well as combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that executes the specified functions or operations, or a combination of dedicated hardware and computer programs.

[0136] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, although the operation and behavior of the systems and / or methods are described herein without reference to specific software code, it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0137] Appendix A: Acronyms AV1:AOMedia Video 1 HEVC:High Efficiency Video Coding MVD: Motion vector difference MVP: Motion vector prediction VVC: Versatile Video Coding CfL: Chroma from Luma SDT: Semi Decoupled Tree SDP: Semi Decoupled Partitioning SST: Semi Separate Tree SB: Super Block IBC(or IntraBC): Intra Block Copy CDF: Cumulative Density Function SCC: Screen Content Coding GBI: Generalized Bi-prediction BCW: Bi-prediction with CU-level weights CIIP: Combined intra-inter prediction POC: Picture Order Count DRL: Dynamic Reference List OBMC: Overlapped block motion compensation AMVD: adaptive MVD resolution AMVR: adaptive motion vector resolution JMVD: Joint coding of motion vector difference(or joint MVD coding) MVP0: motion vector prediction for reference frame list 0 MVP1: motion vector prediction for reference frame list 1

[0138] The above disclosure also includes the embodiments listed below.

[0139] (1) A method performed by at least one processor of a video decoder, comprising: receiving a coded video bitstream including a current picture, a first reference picture, a second reference picture, and a joint motion vector differential (JMVD) associated with at least one block in the current picture; determining whether one or more scaling factors are used in a JMVD based on an adaptive motion vector differential decomposition (AMVD) based on a predetermined condition; in response to determining that the one or more scaling factors are to be used for a JMVD based on the AMVD, scaling the JMVD based on the one or more scaling factors; deriving a first MVD for the first reference picture based on the scaled JMVD; deriving a second MVD of the second reference picture based on the derived first MVD; reconstructing the at least one block based on the derived first MVD and the derived second MVD; The method according to claim 1,

[0140] (2) in response to determining that the one or more scaling factors are to be used in a JMVD based on the AMVD, the one or more scaling factors are limited to powers of two; The method according to feature (1).

[0141] (3) in response to determining that the one or more scaling factors are to be used in a JMVD based on the AMVD, the one or more scaling factors include a same scaling factor for a vertical component and a horizontal component of the JMVD. The method according to feature (1).

[0142] (4) the coded video bitstream further includes an index value; the one or more scaling factors are determined by using a look-up table that specifies one or more predetermined scaling factors based on the index value. The method according to feature (3).

[0143] (5) the one or more predetermined scaling factors include a first scaling factor for a horizontal component of the JMVD and a second scaling factor for a vertical component of the JMVD; The method according to feature (4).

[0144] (6) the first scaling factor is different from the second scaling factor; The method according to feature (5).

[0145] (7) one of the first scaling coefficient and the second scaling coefficient is limited to 1, and the other of the first scaling coefficient and the second scaling coefficient is a value other than 1. The method according to feature (6).

[0146] (8) determining that the one or more scaling factors are to be used in a JMVD based on the AMVD based on a flag in the coded video bitstream indicating that the one or more scaling factors are to be used in a JMVD based on the AMVD; in response to determining that the one or more scaling factors are to be used in a JMVD based on the AMVD, applying the same one or more predetermined scaling factors to both a horizontal component and a vertical component of the JMVD. The method according to feature (5).

[0147] (9) determining that the one or more scaling factors are to be used in the AMVD-based JMVD is based on a flag in the coded video bitstream indicating that the AMVD-based JMVD is applied to a horizontal component of the JMVD; the AMVD-based JMVD utilizes the one or more scaling factors applied to the horizontal component of the JMVD; The method according to feature (1).

[0148] (10) determining that the one or more scaling factors are to be used in the AMVD-based JMVD is based on a flag in the coded video bitstream indicating that the AMVD-based JMVD is applied to a vertical component of the JMVD; the AMVD-based JMVD utilizes the one or more scaling factors applied to the vertical component of the JMVD; The method according to feature (1).

[0149] (11) A video decoder comprising: at least one memory configured to store computer program code; at least one processor configured to access the computer program code and to operate as directed by the computer program code; having The computer program code comprises: receiving code configured to cause the at least one processor to receive a coded video bitstream including a current picture, a first reference picture, a second reference picture, and a joint motion vector differential (JMVD) associated with at least one block in the current picture; decision code configured to cause the at least one processor to determine whether one or more scaling factors are used in a JMVD based on adaptive motion vector differential decomposition (AMVD) based on a predetermined condition; scaling code configured to cause the at least one processor to scale the JMVD based on the AMVD based on the one or more scaling factors in response to determining that the one or more scaling factors are to be used for the JMVD; first derivation code configured to cause the at least one processor to derive a first MVD for the first reference picture based on the scaled JMVD; second derivation code configured to cause the at least one processor to derive a second MVD for the second reference picture based on the derived first MVD; reconstruction code configured to cause the at least one processor to reconstruct the at least one block based on the derived first MVD and the derived second MVD; a video decoder.

[0150] (12) in response to determining that the one or more scaling factors are to be used in a JMVD based on the AMVD, the one or more scaling factors are limited to powers of two. A video decoder according to feature (11).

[0151] (13) in response to determining that the one or more scaling factors are to be used in a JMVD based on the AMVD, the one or more scaling factors include a same scaling factor for a vertical component and a horizontal component of the JMVD. A video decoder according to feature (11).

[0152] (14) the coded video bitstream further comprises an index value; the one or more scaling factors are determined by using a look-up table that specifies one or more predetermined scaling factors based on the index value. A video decoder according to feature (13).

[0153] (15) The one or more predetermined scaling factors include a first scaling factor for a horizontal component of the JMVD and a second scaling factor for a vertical component of the JMVD. A video decoder according to feature (14).

[0154] (16) The first scaling factor is different from the second scaling factor. A video decoder according to feature (15).

[0155] (17) One of the first scaling coefficient and the second scaling coefficient is limited to 1, and the other of the first scaling coefficient and the second scaling coefficient is a value other than 1. 17. A video decoder according to claim 16.

[0156] (18) determining that the one or more scaling factors are to be used in the JMVD based on the AMVD is based on a flag in the coded video bitstream indicating that the one or more scaling factors are to be used in the JMVD based on the AMVD; in response to determining that the one or more scaling factors are to be used in a JMVD based on the AMVD, applying the same one or more predetermined scaling factors to both a horizontal component and a vertical component of the JMVD. A video decoder according to feature (15).

[0157] (19) determining that the one or more scaling factors are to be used in the AMVD-based JMVD is based on a flag in the coded video bitstream indicating that the AMVD-based JMVD is applied to a horizontal component of the JMVD; the AMVD-based JMVD utilizes the one or more scaling factors applied to the horizontal component of the JMVD; 12. A video decoder according to feature 11.

[0158] (20) A non-transitory computer-readable medium storing instructions, comprising: The instructions, when executed by a processor of a video decoder, cause the processor to: receiving a coded video bitstream including a current picture, a first reference picture, a second reference picture, and a joint motion vector differential (JMVD) associated with at least one block in the current picture; determining whether one or more scaling factors are used in a JMVD based on an adaptive motion vector differential decomposition (AMVD) based on a predetermined condition; in response to determining that the one or more scaling factors are to be used for a JMVD based on the AMVD, scaling the JMVD based on the one or more scaling factors; deriving a first MVD for the first reference picture based on the scaled JMVD; deriving a second MVD of the second reference picture based on the derived first MVD; reconstructing the at least one block based on the derived first MVD and the derived second MVD; A non-transitory computer readable medium for carrying out a method having the steps of:

[0159] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Patent Application No. 63 / 390,910, filed July 20, 2022, and U.S. Patent Application No. 17 / 982,946, filed November 8, 2022. The disclosures of these prior applications are incorporated herein by reference in their entireties.

Claims

1. 1. A method performed by at least one processor of a video decoder, comprising: receiving a coded video bitstream including a current picture, a first reference picture, a second reference picture, and a joint motion vector differential (JMVD) associated with at least one block in the current picture; determining whether one or more scaling factors are used in a JMVD based on adaptive motion vector differential decomposition (AMVD) based on a predetermined condition; in response to determining that the one or more scaling factors are to be used in a JMVD based on the AMVD, scaling the JMVD based on the one or more scaling factors; deriving a first MVD for the first reference picture based on the scaled JMVD; deriving a second MVD of the second reference picture based on the derived first MVD; reconstructing the at least one block based on the derived first MVD and the derived second MVD; The method according to claim 1,

2. in response to determining that the one or more scaling factors are to be used in a JMVD based on the AMVD, the one or more scaling factors are limited to powers of two. The method of claim 1.

3. in response to determining that the one or more scaling factors are to be used in a JMVD based on the AMVD, the one or more scaling factors include a same scaling factor for a vertical component and a horizontal component of the JMVD. The method of claim 1.

4. the coded video bitstream further comprises index values; the one or more scaling factors are determined by using a look-up table that specifies one or more predetermined scaling factors based on the index value. The method according to claim 3.

5. the one or more predetermined scaling factors include a first scaling factor for a horizontal component of the JMVD and a second scaling factor for a vertical component of the JMVD. The method according to claim 4.

6. the first scaling factor is different from the second scaling factor; The method according to claim 5.

7. one of the first scaling factor and the second scaling factor is limited to 1, and the other of the first scaling factor and the second scaling factor is a value other than 1. The method according to claim 6.

8. determining that the one or more scaling factors are to be used in the AMVD-based JMVD is based on a flag in the coded video bitstream indicating that the one or more scaling factors are to be used in the AMVD-based JMVD; in response to determining that the one or more scaling factors are to be used in a JMVD based on the AMVD, applying the same one or more predetermined scaling factors to both horizontal and vertical components of the JMVD. The method according to claim 5.

9. determining that the one or more scaling factors are to be used for the AMVD-based JMVD is based on a flag in the coded video bitstream indicating that the AMVD-based JMVD is applied to a horizontal component of the JMVD; the AMVD-based JMVD utilizing the one or more scaling factors applied to the horizontal component of the JMVD; The method of claim 1.

10. determining that the one or more scaling factors are to be used for the AMVD-based JMVD is based on a flag in the coded video bitstream indicating that the AMVD-based JMVD is applied to a vertical component of the JMVD; the AMVD-based JMVD utilizes the one or more scaling factors applied to the vertical component of the JMVD; The method of claim 1.

11. at least one memory configured to store computer program code; at least one processor configured to access the computer program code and to operate as directed by the computer program code; having 11. A video decoder, the computer program code, when executed by the at least one processor, causing the at least one processor to perform a method according to any one of claims 1 to 10.

12. A computer program which, when executed by a processor of a video decoder, causes said processor to perform the method of any one of claims 1 to 10.

13. 1. A method performed by at least one processor of a video encoder, comprising: determining, based on a predetermined condition, that one or more scaling factors are used for a joint motion vector differential (JMVD) based adaptive motion vector differential decomposition (AMVD); scaling a JMVD associated with at least one block in the current picture based on the one or more scaling factors; deriving a first MVD of a first reference picture of the current picture based on the scaled JMVD; deriving a second MVD of a second reference picture of the current picture based on the derived first MVD; encoding the at least one block based on the derived first MVD and the derived second MVD; The method according to claim 1,

Images