System and method for joint coding of motion vector differentials using template matching based scaling factor derivation - Patents.com

Abstract

A system and method for joint coding of motion vector differentials using template matching-based scaling factor derivation includes receiving a current frame in a video bitstream, determining that a current block in the current frame is coded in joint motion vector differential (JMVD) mode, selecting a first adjacent reconstructed sample and a second adjacent reconstructed sample of the current block as template regions to be used to predict the current block in the JVMD mode, determining a predictive block from a reference frame based on a scaling factor derived from the selected template region and applied to a motion vector differential (MVD) associated with the current block, and reconstructing the current block in the JVMD mode based on at least the predictive block.

Description

[Technical field]

[0001] This disclosure relates to advanced image and video coding techniques, and more particularly to a system and method for joint coding of motion vector difference using template matching based scaling factor derivation.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 328,508, filed in the U.S. Patent and Trademark Office on April 7, 2022, and U.S. Patent Application No. 17 / 982,991, filed on November 8, 2022, the disclosures of which are incorporated by reference in their entireties herein. [Background technology]

[0003] Streaming of audiovisual content is gaining popularity. To facilitate the increase in the amount and quality of such streaming content, substantial bandwidth is required. Thus, there is a need for efficient coding and decoding schemes to stream content using less bandwidth while maintaining high quality. For example, H.265 / HEVC, VP9, ​​and AOMedia Video1 (AV1) are some of the coding and decoding schemes developed for this purpose.

[0004] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC1 / SC29 / WG11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4).

[0005] AOMedia Video1 (AV1) is an open video coding format designed for video transmission over the Internet. It was developed as a successor to VP9 by AOMedia (Alliance for Open Media), a consortium founded in 2015. The consortium includes semiconductor companies, video-on-demand providers, video content producers, software developers, and web browser vendors. Many of the components of the AV1 project were sourced from previous research efforts by Alliance members. Individual contributors started experimental technology platforms several years ago: Xiph / Mozilla's Daala released its code already in 2010, Google's experimental VP9 evolution project VP10 was announced on September 12, 2014, and Cisco's Thor on August 11, 2015. Building on the VP9 code base, AV1 incorporates additional technologies, some of which were developed in these experimental formats. The first version 0.1.0 of the AV1 reference codec was published on April 7, 2016. The Alliance announced the release of the AV1 bitstream specification on March 28, 2018, together with references to software-based encoders and decoders. Validated version 1.0.0 of the specification was released on June 25, 2018. Verified version 1.0.0 with Errata 1 of the specification was released on January 8, 2019. The AV1 bitstream specification contains the reference video codec.

[0006] Research and development into next-generation video codecs is also underway. For example, AOMedia has officially begun standardization of a next-generation video codec called Versatile Video Coding (VVC).

[0007] VVC includes some improvements such as the introduction of joint motion vector difference (JMVD) coding. It is named JOINT_NEWMV. This new inter prediction coding mode is applied to indicate whether the motion vector difference (MVD) for two reference lists is signaled together or not. When the inter prediction mode is equal to JOINT_NEWMV mode, the MVD for reference list 0 and reference list 1 are signaled together, and only one MVD named joint_mvd is signaled and transmitted to the decoder that derives the MVD for reference list 0 and reference list 1 from joint_mvd. In general, the JOINT_NEWMV mode is signaled together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes. No additional context is added.

[0008] When JOINT_NEWMV mode is signaled and the Picture Order Count (POC) distance between two reference frames and the current frame is different, the MVD is scaled for reference list 0 or reference list 1 based on the POC distance. For example, if the distance (td0) between reference frame list 0 and the current frame is greater than or equal to the distance (td1) between reference frame list 1 and the current frame, joint_mvd is used directly for reference list 0, and the MVD for reference list 1 is derived from joint_mvd based on equation (1).

[0009]

number

[0010] Otherwise, if td1 is greater than or equal to td0, then joint_mvd is used directly for reference list 1, and the MVD for reference list 0 is derived from joint_mvd based on equation (2).

[0011]

number

[0012] When the JMVD mode is selected for a block, the JMVDs for two reference frames are signaled, and the MVDs for the two reference frames are derived from the JMVDs based on the distances between the reference frames and the current frame. This assumes that there is linear motion between the backward and forward reference frames with respect to the current frame. However, the motion between the two reference frames is not necessarily linear motion. For example, the motion may be slower or faster from the backward reference frame to the forward reference frame. Summary of the Invention

[0013] According to one aspect of the disclosure, a method is provided, the method including: receiving a current frame in a video bitstream including a current block and a plurality of neighboring blocks, determining that the current block is coded in a joint motion vector differential (JMVD) mode, in response to the current block being coded in the JMVD mode, selecting a first neighboring reconstructed sample as a first template area and a second neighboring reconstructed sample of the current block as a second template area used to predict the current block in the JVMD mode, determining a predictive block from a reference frame based on a scaling factor derived from the selected first and second template areas and applied to a motion vector differential (MVD) associated with the current block, and reconstructing the current block in the JVMD mode based on at least the predictive block.

[0014] According to other aspects of the present disclosure, an apparatus and computer readable medium consistent with the present method are also provided. [Brief description of the drawings]

[0015] The above and other aspects, features, and advantages of certain embodiments according to the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings. [Figure 1] FIG. 1 is a diagram illustrating components of one or more devices according to various embodiments. [Diagram 2] FIG. 2 is a diagram illustrating template matching based scaling factor derivation for joint coding of motion vector differentials in accordance with various embodiments. [Diagram 3] FIG. 3 is a flow diagram illustrating a method for joint coding of motion vector differentials in a video bitstream in accordance with various embodiments. [Figure 4]FIG. 4 is a flow chart illustrating a method for determining scaling factors using template matching based scaling factor derivation, according to various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] According to one aspect of the disclosure, a method is provided for joint coding of motion vector differential (JMVD) in a video bitstream, which may include: defining a first template region and a second template region for a current block of a current frame in the video bitstream based on a joint motion vector differential (JMVD) mode of the current block, identifying a plurality of candidate scaling factors from among a plurality of predefined scaling factors, determining a scaling factor for a motion vector differential (MVD) associated with the current block from among the plurality of candidate scaling factors based on the first template region for the current block, the second template region for the current block, and the plurality of candidate scaling factors, and fetching a block from a reference frame based on the determined scaling factor.

[0017] The first template region for the current block may include reconstructed samples of an adjacent block to the left of the current block, and the second template region for the current block may include reconstructed samples of an adjacent block to the top of the current block.

[0018] The method may include predicting a first motion vector prediction (MVP) corresponding to a backward reference frame that is before the current frame in display order, and predicting a second MVP corresponding to a forward reference frame that is after the current frame in display order.

[0019] The step of determining a scaling factor from among a plurality of candidate scaling factors for the MVD associated with the current block may include a step of generating a plurality of first predictive blocks, each of the plurality of first predictive blocks corresponding to a first candidate scaling factor among the plurality of candidate scaling factors, and a step of generating a plurality of second predictive blocks, each of the plurality of second predictive blocks corresponding to a second candidate scaling factor among the plurality of candidate scaling factors.

[0020] The step of generating a plurality of first predictive blocks may include a step of determining a plurality of first MVDs based on the first MVP and a plurality of first reference frames, a step of scaling each of the plurality of first MVDs based on a plurality of candidate scaling factors, obtaining a plurality of scaled first MVDs for each of the plurality of scaling factors, and a step of generating the first predictive block using a motion vector for each candidate scaling factor among the plurality of candidate scaling factors that is equal to the sum of the first MVP and the plurality of scaled first MVDs corresponding to each candidate scaling factor.

[0021] The step of generating a plurality of second predictive blocks may include a step of determining a plurality of second MVDs based on the second MVP and a plurality of second reference frames, a step of scaling each of the plurality of second MVDs based on a plurality of candidate scaling factors, obtaining a plurality of scaled second MVDs for each of the plurality of scaling factors, and a step of generating the second predictive block using a motion vector for each candidate scaling factor among the plurality of candidate scaling factors that is equal to the sum of the second MVP and the plurality of scaled second MVDs corresponding to each candidate scaling factor.

[0022] The method may further include determining a first template region in each of the plurality of first reference frames based on the plurality of first predictive blocks, where the first template region in each first reference frame corresponds to the first template region of a current block in the current frame; determining a second template region in each of the plurality of first reference frames based on the plurality of first predictive blocks, where the second template region in each first reference frame corresponds to the second template region of the current block in the current frame; determining a first template region in each of the plurality of second reference frames based on the plurality of second predictive blocks, where the first template region in each second reference frame corresponds to the first template region of the current block in the current frame; and determining a second template region in each of the plurality of second reference frames based on the plurality of second predictive blocks, where the second template region in each second reference frame corresponds to the second template region of the current block in the current frame.

[0023] The method may further include generating a plurality of first template predictions associated with the current frame based on a weighted average of a first template region associated with each first predictive block and a first template region associated with each second predictive block, and generating a plurality of second template predictions associated with the current frame based on a weighted average of a second template region associated with each first predictive block and a second template region associated with each second predictive block.

[0024] The method may further include determining a first difference between each first template prediction and a first template region for the current frame, determining a second difference between each second template prediction and a second template region for the current frame, and evaluating the first and second differences based on a cost criterion, where a scaling factor associated with a lowest cost based on the cost criterion is determined as the scaling factor for the MVD associated with the current block.

[0025] The cost metric may include the sum of absolute differences (SAD), the sum of squared errors (SSE), or the sum of absolute transformed differences (SATD).

[0026] According to one aspect of the present disclosure, there is provided an apparatus comprising: a memory storing program code; and at least one processor configured to execute the program code and to act as instructed by the program code. The program code may include receiving code configured to cause at least one of the at least one processor to receive a current frame in a video bitstream including a current block and a plurality of neighboring blocks; decision code configured to cause at least one of the at least one processor to determine that the current block is coded in a joint motion vector differential (JMVD) mode; selection code configured to, in response to the current block being coded in the JMVD mode, cause at least one of the at least one processor to select a first neighboring reconstructed sample as a first template region and select a second neighboring reconstructed sample of the current block as a second template region to be used for predicting the current block in the JVMD mode; decision code configured to cause at least one of the at least one processor to determine a prediction block from a reference frame based on a scaling factor derived from the selected first and second template regions and applied to a motion vector differential (MVD) associated with the current block; and reconstruction decision code configured to cause at least one of the at least one processor to reconstruct the current block in the JVMD mode based on at least the prediction block.

[0027] The program code may further include prediction code configured to cause at least one of the at least one processor to predict a first motion vector prediction (MVP), corresponding to a backward reference frame that is before the current frame in display order, and prediction code configured to cause at least one of the at least one processor to predict a second MVP, corresponding to a forward reference frame that is after the current frame in display order.

[0028] The decision code may include: generation code configured to cause at least one of the at least one processor to generate a plurality of first prediction blocks, each of the plurality of first prediction blocks corresponding to a first candidate scaling factor among a plurality of candidate scaling factors; and generation code configured to cause at least one of the at least one processor to generate a plurality of second prediction blocks, each of the plurality of second prediction blocks corresponding to a second candidate scaling factor among the plurality of candidate scaling factors.

[0029] The program code may further include generation code configured to cause at least one of the at least one processor to generate a plurality of first template predictions associated with the current frame based on a weighted average of a first template region in each of a plurality of first reference frames based on the plurality of first prediction blocks and a first template region in each of a plurality of second reference frames based on the plurality of second prediction blocks; and generation code configured to cause at least one of the at least one processor to generate a plurality of second template predictions associated with the current frame based on a weighted average of a second template region in each of the first reference frames based on the plurality of first prediction blocks and a second template region in each of the second reference frames based on the plurality of second prediction blocks.

[0030] The program code may further include decision code configured to cause at least one of the at least one processor to determine a first difference between each first template prediction and a first template region for the current frame; decision code configured to cause at least one of the at least one processor to determine a second difference between each second template prediction and a second template region for the current frame; and evaluation code configured to cause the at least one of the at least one processor to evaluate the first and second differences based on a cost criterion, where a scaling factor associated with a lowest cost based on the cost criterion is determined as a scaling factor for the MVD associated with the current block.

[0031] According to one aspect of the disclosure, a non-transitory computer-readable storage medium having computer-readable program code stored thereon is provided, which, when executed by a processor, causes the processor to define a first template region and a second template region for a current block of a current frame in a video bitstream based on at least a joint motion vector differential (JMVD) mode of the current block, identify a plurality of candidate scaling factors from among a plurality of predefined scaling factors, determine a scaling factor for a motion vector differential (MVD) associated with the current block from among the plurality of candidate scaling factors based on the first template region for the current block, the second template region for the current block, and the plurality of candidate scaling factors, and fetch a block from a reference frame based on the determined scaling factor.

[0032] The program code further causes the processor to predict at least a first motion vector prediction (MVP) corresponding to a backward reference frame that is before the current frame in display order, and to predict a second MVP corresponding to a forward reference frame that is after the current frame in display order.

[0033] The program code causes the processor to determine a scaling factor and further causes the processor to generate at least a plurality of first prediction blocks, where each of the plurality of first prediction blocks corresponds to a first candidate scaling factor among a plurality of candidate scaling factors, and generate a plurality of second prediction blocks, where each of the plurality of second prediction blocks corresponds to a second candidate scaling factor among the plurality of candidate scaling factors.

[0034] The program code further causes the processor to generate a plurality of first template predictions associated with the current frame based on at least a weighted average of a first template region in each of the plurality of first reference frames based on the plurality of first prediction blocks and a first template region in each of the plurality of second reference frames based on the plurality of second prediction blocks, and to generate a plurality of second template predictions associated with the current frame based on a weighted average of a second template region in each of the first reference frames based on the plurality of first prediction blocks and a second template region in each of the second reference frames based on the plurality of second prediction blocks.

[0035] The program code further causes the processor to determine at least a first difference between each first template prediction and a first template region for the current frame, determine a second difference between each second template prediction and a second template region for the current frame, and evaluate the first and second differences based on a cost criterion, where a scaling factor associated with a lowest cost based on the cost criterion is determined as a scaling factor for the MVD associated with the current block.

[0036] The following detailed description of the exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements.

[0037] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. Moreover, one or more features or components of one embodiment may be incorporated or combined with another embodiment (or one or more features of another embodiment). In addition, in the flowcharts and descriptions of operations provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed simultaneously (at least in part), and the order of one or more operations may be rearranged.

[0038] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limiting of the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code. It will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0039] Although particular combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features can be combined in ways not specifically recited in the claims and / or specifically disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.

[0040] No element, act, or instruction used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Also, as used herein, the terms "has," "have," "having," "include," "including," etc. are intended to be open-ended terms. Further, the phrase "based on" is intended to mean "based, at least in part, on," unless expressly specified otherwise. Further, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood as including only A, only B, or both A and B.

[0041] As described above, when MVD is jointly signaled to perform joint coding of motion vector differentials, VVC assumes that there is linear motion between the backward reference frame and the forward reference frame. However, the motion between two reference frames is not necessarily linear. For example, the motion may be slower or faster from the backward reference frame to the forward reference frame.

[0042] Various embodiments according to the present disclosure provide a system and method for template matching based scaling factor derivation for joint coding of motion vector differentials. When a joint motion vector differential (JMVD) mode is selected for a current block in a current frame, template regions are defined in a current frame and in multiple reference frames. The template regions in the multiple reference frames are based on a predicted first motion vector prediction (MVP) or a predicted second MVP, and these template regions are used to generate a template prediction for the current block. The template prediction for the current block is compared to the defined template region for the current block based on a criterion for selecting a scaling factor, and the selected scaling factor is used to fetch a corresponding predicted block.

[0043] 1 is a diagram illustrating components of one or more devices according to various embodiments. With reference to FIG. 1, device 100 may include a bus 110, one or more processors 120, a memory 130, a storage component 140, and a communication interface 150. It is understood that one or more of the components may be omitted and / or one or more additional components may be included.

[0044] Bus 110 includes components that enable communication between components of device 100. Processor 120 may be implemented in hardware, firmware, or a combination of hardware and software. Processor 120 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a sparse tensor core, or another type of processing component. Processor 120 may include one or more processors. For example, processor 120 may include one or more CPUs, APUs, FPGAs, ASICs, sparse tensor cores, or another type of processing component. One or more processors of processor 120 may be capable of being programmed to perform functions.

[0045] Memory 130 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 120.

[0046] Storage component 140 stores information and / or software related to the operation and use of device 100. For example, storage component 140 may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact disks (CDs), digital versatile disks (DVDs), floppy disks, cartridges, magnetic tapes, and / or another type of non-transitory computer-readable medium along with corresponding drives.

[0047] Communications interface 150 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable device 100 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communications interface 150 may enable device 100 to receive information from another device and / or provide information to another device. For example, communications interface 150 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0048] Device 100 may perform one or more processes or functions described herein. Device 100 may perform operations based on processor 120 executing software instructions stored by a non-transitory computer-readable medium, such as memory 130 and / or storage component 140. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[0049] The software instructions may be loaded into memory 130 and / or storage component 140 from another computer-readable medium or from another device via communications interface 150. When executed, the software instructions stored in memory 130 and / or storage component 140 may cause processor 120 to perform one or more of the processes described herein.

[0050] Additionally, or alternatively, hardwired circuitry may be used in place of, or in combination with, software instructions to implement one or more processes described herein. Thus, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0051] The number and arrangement of components shown in Figure 1 is provided as an example. In practice, device 100 may include additional, fewer, different, or differently arranged components than those shown in Figure 1. Additionally or alternatively, a set of components (e.g., one or more components) of device 100 may perform one or more functions that are described as being performed by another set of components of device 100.

[0052] Any one of the operations or processes described below (eg, FIGS. 2-4) may be performed by or using any one of the elements shown in FIG.

[0053] 2 is a diagram illustrating a template matching based scaling factor derivation according to various embodiments. As shown in FIG. 2, a JMVD mode may be selected for a current block 200. The current block 200 may belong to a current frame in a bitstream being processed in a coding / decoding scheme, such as Versatile Video Coding (VVC).

[0054] When the JMVD mode is selected for the current block 200, a first template region 203 and a second template region 204 are defined. The first template region 203 and the second template region 204 may be reconstructed samples of adjacent blocks. For example, the first template region 203 may be a reconstructed sample of a block adjacent to the left side of the current block 200, and the second template region 204 may be a reconstructed sample of a block adjacent to the top side of the current block.

[0055] Additionally, when the JMVD mode is selected for the current block 200, a first motion vector prediction (MVP) 201 and a second MVP 202 may be predicted. The first MVP 201 corresponds to a backward reference frame that is before the current frame in the display order of frames, and the second MVP 202 corresponds to a forward reference frame that is after the current frame in the display order of frames.

[0056] A plurality of first motion vector differentials (MVDs) may be calculated based on the first MVP 201. Each of the first MVDs may correspond to a reference frame in a first list of reference frames. The first list of reference frames may include a plurality of backward reference frames.

[0057] A plurality of second MVDs may be calculated based on the second MVP 202. Each second MVD corresponds to a reference frame in a second list of reference frames. The second list of reference frames may include a plurality of forward reference frames.

[0058] According to various embodiments, a plurality of candidate scaling factors may be identified from a group of predefined scaling factors. Each of the plurality of candidate scaling factors may be applied to a plurality of first MVDs to generate a plurality of scaled first MVDs corresponding to each candidate scaling factor. For example, a plurality of scaled first MVDs may be generated for a first candidate scaling factor by applying a first candidate scaling factor to the plurality of first MVDs. And a plurality of scaled first MVDs may be generated for a second candidate scaling factor by applying a second candidate scaling factor to the plurality of first MVDs. Additionally, each of the plurality of candidate scaling factors may be applied to a plurality of second MVDs to generate a plurality of scaled second MVDs corresponding to each candidate scaling factor. For example, a plurality of scaled second MVDs may be generated for a first candidate scaling factor by applying a first candidate scaling factor to the plurality of second MVDs. And a plurality of scaled second MVDs may be generated for a second candidate scaling factor by applying a second candidate scaling factor to the plurality of first MVDs.

[0059] A plurality of first predictive blocks may be generated for each candidate scaling factor among the plurality of candidate scaling factors based on the first MVP and a corresponding plurality of scaled first MVDs. For example, a first predictive block may be generated for a first candidate scaling factor based on the first MVP and a plurality of scaled first MVDs corresponding to the first candidate scaling factor. And a first predictive block may be generated for a second candidate scaling factor based on the first MVP and a plurality of scaled first MVDs corresponding to the second candidate scaling factor. As shown in FIG. 2, a first predictive block 210 may be generated for a candidate scaling factor among the plurality of candidate scaling factors. The first predictive block 210 may be generated using a motion vector equal to the sum of the first MVP 201 and a plurality of scaled first MVDs corresponding to the candidate scaling factor.

[0060] A plurality of second predictive blocks may be generated for each candidate scaling factor among the plurality of candidate scaling factors based on the second MVP and the corresponding plurality of scaled second MVDs. For example, the second predictive block may be generated for a first candidate scaling factor based on the second MVP and the plurality of scaled second MVDs corresponding to the first candidate scaling factor. And, the second predictive block may be generated for a second candidate scaling factor based on the second MVP and the plurality of scaled second MVDs corresponding to the second candidate scaling factor. As shown in FIG. 2, the second predictive block 220 may be generated for one candidate scaling factor among the plurality of candidate scaling factors. The second predictive block 220 may be generated using a motion vector equal to the sum of the second MVP 202 and the plurality of scaled second MVDs corresponding to the candidate scaling factor.

[0061] A first template region may be defined for a plurality of first predictive blocks and a plurality of second predictive blocks corresponding to the first template region 203. For example, a first template region 212 may be defined for the first predictive block 210, and a first template region 222 may be defined for the second predictive block 220. The first template region 212 and the first template region 222 may correspond to the first template region 203.

[0062] A second template region may be defined for the first and second predictive blocks corresponding to the second template region 204. For example, a second template region 214 may be defined for the first predictive block 210, and a second template region 224 may be defined for the second predictive block 220. The second template region 214 and the second template region 224 may correspond to the second template region 204.

[0063] The first template regions associated with the plurality of first predictive blocks and the first template regions associated with the plurality of second predictive blocks may be weight-averaged to generate a plurality of predicted first template areas associated with the current block 200. Each predicted first template region among the plurality of predicted first template regions may correspond to one candidate scaling factor among the plurality of candidate scaling factors. For example, among the plurality of predicted first template regions associated with the current block 200, the first template region 212 and the first template region 222 may be weight-averaged to generate the predicted first template region associated with the current block 200.

[0064] The second template regions associated with the plurality of first predictive blocks and the second template regions associated with the plurality of second predictive blocks may be weighted averaged to generate a plurality of predicted second template regions associated with the current block 200. Each predicted second template region among the plurality of predicted second template regions may correspond to a candidate scaling factor among the plurality of candidate scaling factors. For example, among the plurality of predicted second template regions associated with the current block 200, second template region 214 and second template region 224 may be weighted averaged to generate a predicted second template region associated with the current block 200.

[0065] A difference between each predicted first template region and the first template region 203 may be determined, and a difference between each predicted second template region and the second template region 204 may be determined. The differences may be evaluated based on a cost criterion, and the candidate scaling factor corresponding to the lowest cost may be used as the predicted scaling factor for the MVD of the current block 200.

[0066] According to various embodiments, the cost metric may include, but is not limited to, the sum of absolute difference (SAD), the sum of a squared error (SSE), or the sum of absolute transform difference (SATD).

[0067] According to various embodiments, an index or flag associated with each of a number of candidate scaling factors may be signaled, and the predictive scaling factor may be used as a context for entropy coding of the index or flag.

[0068] In accordance with various embodiments, instead of explicitly signaling an index or flag associated with each of multiple candidate scaling factors, the predicted scaling factor is used to reorder the candidate scaling factors, and then the index or flag of the reordered candidate scaling factor is signaled.

[0069] According to various embodiments, instead of explicitly signaling an index or flag associated with each of multiple candidate scaling factors, the predicted scaling factor may be used directly as a scaling factor for deriving the MVD used to generate the first predictive block 210 or the second predictive block 220.

[0070] According to various embodiments, the indexes or flags of the candidate scaling factors may be either explicitly signaled in the bitstream or implicitly derived based on template matching as described above. The choice between explicit signaling and implicit derivation may be signaled.

[0071] According to various embodiments, the predictive scaling factors may be derived from a number of candidate scaling factors that are different from the number of candidate scaling factors that are explicitly signaled.

[0072] According to various embodiments, the multiple candidate scaling factors that are explicitly signaled may include a limited number of candidates (e.g., 1, 1 / 2, 2), and the predicted scaling factor may be derived from a larger set of candidates (e.g., 1, 1 / 8, 2 / 8, 3 / 8, 4 / 8, ..., 15 / 8, 16 / 8, ..., 32 / 8).

[0073] FIG. 3 is a flow diagram illustrating a method 300 for joint coding of motion vector differentials in a video bitstream, in accordance with various embodiments.

[0074] At 301, the method 300 includes receiving a current frame including a current block and a neighboring block. For example, the device 100 may receive a current frame in a video bitstream including the current block 200 and the neighboring blocks.

[0075] At 302, method 300 includes selecting a first template region and a second template region for the current block based on a determination that the current block is coded in JMVD mode. For example, device 100 may determine that the current block is coded in JMVD mode, and in response to the current block being coded in JMVD mode, device 100 may select a first-neighboring reconstructed sample of the current block as first template region 203 and a second-neighboring reconstructed sample of the current block as second template region 204 to be used to predict the current block in JMVD mode.

[0076] At 303, the method 300 includes determining a predictive block from a reference frame based on a scaling factor. The scaling factor may be derived based on the first and second template regions and the MVD. For example, the device 100 may determine a predictive block from a reference frame based on a scaling factor derived from the selected first template region 203 and second template region 204. The device 100 may apply the scaling factor to a motion vector difference (MVD) associated with the current block 200 to determine the predictive block. Determining a scaling factor for the MVD associated with the current block is described in more detail in connection with FIG. 4.

[0077] At 304, the method 300 includes reconstructing the current block. For example, the device 100 may reconstruct the current block 200 based on the determined predictive block.

[0078] FIG. 4 is a flow chart illustrating a method 400 for determining scaling factors using template matching based scaling factor derivation, according to various embodiments.

[0079] At 401, the method 400 includes predicting multiple MVPs. For example, the device 100 may predict a first MVP 201 that corresponds to a backward reference frame that is before the current frame in display order, and the device 100 may predict a second MVP 202 that corresponds to a forward reference frame that is after the current frame in display order.

[0080] At 402, the method 400 includes generating a plurality of predictive blocks. For example, the device 100 may generate a plurality of first predictive blocks (e.g., the first predictive block 210). Each of the plurality of first predictive blocks corresponds to a candidate scaling factor among the plurality of candidate scaling factors. The device 100 may generate a plurality of second predictive blocks (e.g., the second predictive block 220). Each of the plurality of second predictive blocks corresponds to a candidate scaling factor among the plurality of candidate scaling factors.

[0081] The device 100 may generate a plurality of first prediction blocks by determining a plurality of first MVDs based on the first MVP 201 and a plurality of first reference frames. The device 100 may scale each of the plurality of first MVDs based on a plurality of candidate scaling factors to obtain a plurality of scaled first MVDs for each of the plurality of candidate factors. The device 100 may generate a first prediction block for each candidate scaling factor among the plurality of candidate scaling factors using a motion vector equal to the sum of the first MVP 201 and the plurality of scaled first MVDs corresponding to each candidate scaling factor.

[0082] The device 100 may generate a plurality of second predictive blocks by determining a plurality of second MVDs based on the second MVP 202 and a plurality of second reference frames. The device 100 may scale each of the plurality of second MVDs based on a plurality of candidate scaling factors to obtain a plurality of scaled second MVDs for each of the plurality of scaling factors. The device 100 may generate a second predictive block for each candidate scaling factor among the plurality of candidate scaling factors using a motion vector equal to the sum of the second MVP 202 and the plurality of scaled second MVDs corresponding to each candidate scaling factor.

[0083] At 403, the method 400 may determine a template region in each of the first reference frames among the plurality of first reference frames and in each of the second reference frames among the plurality of second reference frames based on the plurality of predictive blocks. For example, the device 100 may determine a first template region for each of the plurality of first predictive blocks corresponding to the first template region 203 for the current block 200. The first template region for each of the plurality of first predictive blocks may include a reconstructed sample of a block adjacent to the left of the block corresponding to the first predictive block in each of the plurality of first reference frames. The device 10 may determine a second template region for each of the plurality of first predictive blocks corresponding to the second template region 204 for the current block 200. The second template region for each of the plurality of first predictive blocks may include a reconstructed sample of a block adjacent to the top of the block corresponding to the first predictive block in each of the plurality of first reference frames. The device 10 may determine a first template region for each of the plurality of second predictive blocks corresponding to the first template region 203 for the current block 200. The first template region for each of the plurality of second predictive blocks may include a reconstructed sample of a block adjacent to the left of the block corresponding to the second predictive block in each of the plurality of second reference frames. The device 100 may determine a second template region for each of the plurality of second predictive blocks that corresponds to the second template region 204 for the current block 200. The second template region for each of the plurality of second predictive blocks may include a reconstructed sample of a block adjacent to the top of the block corresponding to the second predictive block in each of the plurality of second reference frames.

[0084] At 404, the method 400 may include generating a plurality of template predictions. For example, the device 100 may generate a plurality of first template predictions associated with the current frame based on a weighted average of a first template region associated with the plurality of first predictive blocks and a first template region associated with the plurality of second predictive blocks. The device 100 generates a plurality of second template predictions associated with the current frame based on a weighted average of a second template region associated with the plurality of first predictive blocks and a second template region associated with the plurality of second predictive blocks.

[0085] At 405, the method 400 may include determining a scaling factor for the MVD associated with the current block from among a plurality of candidate scaling factors. For example, the device 100 may determine a difference between each first template prediction and a first template region for the current frame, and a difference between each second predicted template region and a second template region for the current frame. The device 100 may evaluate the determined differences based on a cost criterion. The device 100 may determine a scaling factor associated with the lowest cost based on the cost criterion as a scaling factor for the MVD associated with the current block 200. The cost criterion may include a sum of absolute differences (SAD), a sum of squared errors (SSE), or a sum of absolute transformed differences (SATD).

[0086] According to various exemplary embodiments, joint coding of motion vector differentials may be performed using template matching based scaling factor derivation. The joint coding operation may be used even when there is non-linear motion between backward and forward reference frames with respect to a current block of a current frame. Systems and methods are provided for performing joint coding of motion vector differentials by defining a template region for a current block, identifying candidate scaling factors, generating predictive blocks based on the candidate scaling factors, defining a template region for each generated predictive block, and determining a scaling factor for the MVD of the current block.

[0087] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0088] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail of integration. Furthermore, one or more of the above components described above may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or media) having computer-readable program instructions for causing a processor to perform operations.

[0089] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves with instructions recorded thereon, and any suitable combination of the above. Computer-readable storage media, as used herein, should not be construed as being a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0090] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0091] The computer readable program code / instructions for performing operations may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or a connection to an external computer may be made (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects or operations.

[0092] These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine. As a result, the instructions, executed via the processor of the computer or other programmable data processing apparatus, create means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner. As a result, the computer-readable storage medium on which the instructions are stored includes an article of manufacture containing instructions that perform aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0093] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0094] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing the specified logical functions. The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks than those shown in the figures. In some alternative implementations, the functions described in the blocks may occur in a different order than that described in the figures. For example, two blocks shown in succession may in fact be executed simultaneously or substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0095] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

Claims

1. It is a method receiving a current frame in a video bitstream, the current frame including a current block and a number of adjacent blocks; determining that the current block is coded in a joint motion vector differential (JMVD) mode; in response to the current block being coded in JVMD mode, selecting a first adjacent reconstructed sample of the current block as a first template region and selecting a second adjacent reconstructed sample of the current block as a second template region, which are used to predict the current block in the JVMD mode; determining a prediction block from a reference frame based on a scaling factor derived from the selected first and second template regions and applied to a motion vector differential (MVD) associated with the current block; reconstructing the current block in the JVMD mode based on at least the predicted block; A method comprising:

2. The step of determining a prediction block from a reference frame based on the scaling factor comprises: identifying a plurality of candidate scaling factors from among a plurality of predefined scaling factors; determining a scaling factor for a motion vector difference (MVD) associated with the current block from among the plurality of candidate scaling factors based on the first template region for the current block, the second template region for the current block, and the plurality of candidate scaling factors; The method of claim 1 , comprising:

3. The method further comprises: predicting a first motion vector prediction (MVP) corresponding to a backward reference frame preceding the current frame in display order; predicting a second MVP corresponding to a forward reference frame that is after the current frame in the display order; The method of claim 1 , comprising:

4. The step of determining a scaling factor comprises: generating a plurality of first predicted blocks, each of the plurality of first predicted blocks corresponding to a first candidate scaling factor among the plurality of candidate scaling factors; generating a plurality of second predicted blocks, each of the plurality of second predicted blocks corresponding to a second candidate scaling factor among the plurality of candidate scaling factors; The method of claim 3 , comprising:

5. The step of generating a plurality of first predicted blocks comprises: determining a plurality of first MVDs based on the first MVP and a plurality of first reference frames; scaling each of the plurality of first MVDs based on a plurality of candidate scaling factors to obtain a plurality of scaled first MVDs for each of the plurality of scaling factors; generating a first predictive block using, for each candidate scaling factor among the plurality of candidate scaling factors, a motion vector equal to the sum of the first MVP and the plurality of scaled first MVDs corresponding to each candidate scaling factor; The method of claim 4 , comprising:

6. The step of generating a plurality of second predicted blocks comprises: determining a plurality of second MVDs based on the second MVP and a plurality of second reference frames; scaling each of the plurality of second MVDs based on a plurality of candidate scaling factors to obtain a plurality of scaled second MVDs for each of the plurality of scaling factors; generating a second predictive block using, for each candidate scaling factor among the plurality of candidate scaling factors, a motion vector equal to the sum of the second MVP and the plurality of scaled second MVDs corresponding to each candidate scaling factor; The method of claim 4 , comprising:

7. The method further comprises: determining a first template region in each of a plurality of first reference frames based on the plurality of first predicted blocks; the first template region in each first reference frame corresponds to the first template region of the current block in the current frame; Steps and determining, based on the plurality of first predicted blocks, a second template region in each of the plurality of first reference frames, the second template region corresponding to the second template region of the current block; the second template region in each first reference frame corresponds to the second template region of the current block in the current frame; Steps and determining a first template region in each of a plurality of second reference frames based on the plurality of second predicted blocks; the first template region in each second reference frame corresponds to the first template region of the current block in the current frame; Steps and determining a second template region in each of the second reference frames based on the second predicted blocks; the second template region in each second reference frame corresponds to the second template region of the current block in the current frame; Steps and The method of claim 4 , comprising:

8. The method further comprises: generating a plurality of first template predictions associated with the current frame based on a weighted average of the first template region associated with each first predicted block and the first template region associated with each second predicted block; generating a plurality of second template predictions associated with the current frame based on a weighted average of the second template region associated with each first predicted block and the second template region associated with each second predicted block; The method of claim 7, comprising:

9. The method further comprises: determining a first difference between each first template prediction and the first template region for the current frame; determining a second difference between each second template prediction and the second template region for the current frame; evaluating the first difference and the second difference based on a cost criterion; a scaling factor associated with a lowest cost based on the cost criteria is determined as the scaling factor for the MVD associated with the current block. Steps and The method of claim 8 , comprising:

10. The cost metric includes sum of absolute differences (SAD), sum of squared errors (SSE), or sum of absolute transformed differences (SATD).

10. The method of claim 9.

11. A device A memory for storing program code; at least one processor configured to execute said program code and to act as instructed by said program code; The program code comprises: receiving code configured to cause at least one of the at least one processor to receive a current frame in a video bitstream including a current block and a number of neighboring blocks; decision code configured to cause at least one of the at least one processor to determine that the current block is coded in a joint motion vector differential (JMVD) mode; selection code configured to, in response to the current block being coded in a JVMD mode, cause at least one of the at least one processor to select a first adjacent reconstructed sample of the current block as a first template region and to select a second adjacent reconstructed sample of the current block as a second template region to be used for predicting the current block in the JVMD mode; decision code configured to cause at least one of the at least one processor to determine a prediction block from a reference frame based on a scaling factor derived from the selected first and second template regions and applied to a motion vector differential (MVD) associated with the current block; reconstructing the current block in the JVMD mode; and 13. An apparatus comprising:

12. The program code further comprises: prediction code configured to cause at least one of the at least one processor to predict a first motion vector prediction (MVP) corresponding to a backward reference frame that precedes the current frame in display order; prediction code configured to cause at least one of the at least one processor to predict a second MVP corresponding to a forward reference frame that is after the current frame in the display order; The apparatus of claim 11 , comprising:

13. The decision code is: generating code configured to cause at least one of the at least one processor to generate a plurality of first prediction blocks, each of the plurality of first prediction blocks corresponding to a first candidate scaling factor among the plurality of candidate scaling factors; generation code configured to cause at least one of the at least one processor to generate a plurality of second prediction blocks, each of the plurality of second prediction blocks corresponding to a second candidate scaling factor among the plurality of candidate scaling factors; and The apparatus of claim 12 , comprising:

14. The program code further comprises: generation code configured to cause at least one of the at least one processor to generate a plurality of first template predictions associated with the current frame based on a weighted average of a first template region in each of a plurality of first reference frames based on the plurality of first prediction blocks and a first template region in each of a plurality of second reference frames based on the plurality of second prediction blocks; generation code configured to cause at least one of the at least one processor to generate a plurality of second template predictions associated with the current frame based on a weighted average of a second template region in each first reference frame based on the plurality of first prediction blocks and a second template region in each second reference frame based on the plurality of second prediction blocks; The apparatus of claim 13 , comprising:

15. The program code further comprises: decision code configured to cause at least one of the at least one processor to determine a first difference between each first template prediction and the first template region for the current frame; decision code configured to cause at least one of the at least one processor to determine a second difference between each second template prediction and the second template region for the current frame; evaluation code configured to cause at least one of the at least one processor to evaluate the first and second differences based on a cost criterion, wherein a scaling factor associated with a lowest cost based on the cost criterion is determined as the scaling factor for the MVD associated with the current block; and The apparatus of claim 14 , comprising:

16. A non-transitory computer readable storage medium storing computer readable program code, the program code, when executed by a processor, causing the processor to perform at least: receiving a current frame in a video bitstream including a current block and a plurality of adjacent blocks; determining that the current block is coded in a joint motion vector differential (JMVD) mode; in response to the current block being coded in a JVMD mode, selecting a first adjacent reconstructed sample of the current block as a first template region and selecting a second adjacent reconstructed sample of the current block as a second template region to be used for predicting the current block in the JVMD mode; determining a prediction block from a reference frame based on a scaling factor derived from the selected first and second template regions and applied to a motion vector differential (MVD) associated with the current block; reconstructing the current block in the JVMD mode based on at least the predicted block; A non-transitory computer-readable storage medium.

17. The program code further causes the processor to: predicting a first motion vector prediction (MVP) corresponding to a backward reference frame that precedes the current frame in display order; predicting a second MVP corresponding to a forward reference frame that is after the current frame in the display order; 20. The non-transitory computer-readable storage medium of claim 16.

18. The program code causes the processor to determine the scaling factor and further includes at least generating a plurality of first predicted blocks, each of the plurality of first predicted blocks corresponding to a first candidate scaling factor among the plurality of candidate scaling factors; and generating a plurality of second predicted blocks, each of the plurality of second predicted blocks corresponding to a second candidate scaling factor among the plurality of candidate scaling factors; 20. The non-transitory computer-readable storage medium of claim 17.

19. The program code further causes the processor to: generating a plurality of first template predictions associated with the current frame based on a weighted average of a first template region in each of a plurality of first reference frames based on the plurality of first predicted blocks and a first template region in each of a plurality of second reference frames based on the plurality of second predicted blocks; generating a plurality of second template predictions associated with the current frame based on a weighted average of a second template region in each first reference frame based on the plurality of first prediction blocks and a second template region in each second reference frame based on the plurality of second prediction blocks; 20. The non-transitory computer-readable storage medium of claim 18.

20. The program code further causes the processor to: determining a first difference between each first template prediction and the first template region for the current frame; determining a second difference between each second template prediction and the second template region for the current frame; and evaluating the first difference and the second difference based on a cost criterion; a scaling factor associated with a lowest cost based on the cost criteria is determined as the scaling factor for the MVD associated with the current block.

20. The non-transitory computer-readable storage medium of claim 19.

Images