Method, electronic device and computer program for sub-block motion vector prediction

Abstract

This disclosure relates generally to video coding, and more particularly to smooth sub-block motion compensation, and discloses methods and systems for determining sub-block motion vectors of a video block based on motion information of its spatial neighboring blocks, temporal blocks spatially co-located with the neighboring blocks, or temporal blocks co-located with the video block.

Description

[Technical Field]

[0001] Incorporation by Reference This PCT International Patent Application is based on and claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 443,016, entitled "Smooth Sub-Block Motion Vector Prediction," filed February 2, 2023, and U.S. Non-Provisional Patent Application No. 18 / 455,309, entitled "Smooth Sub-Block Motion Vector Prediction," filed August 24, 2023, which is incorporated herein by reference in its entirety.

[0002] Technical Field FIELD OF THE DISCLOSURE This disclosure relates generally to video coding, and more particularly to methods and systems for sub-block motion compensation. [Background technology]

[0003] Uncompressed digital video can contain a series of pictures and may have specific bitrate requirements for storage, data processing, and transmission bandwidth in streaming applications. One goal of video coding and decoding is to reduce redundancy in the uncompressed input video signal through various compression techniques. Summary of the Invention [Problem to be solved by the invention]

[0004] This disclosure relates generally to video coding, and more particularly to smooth sub-block motion compensation, and discloses methods and systems for determining sub-block motion vectors of a video block based on motion information of its spatial neighboring blocks, temporal blocks spatially co-located with the neighboring blocks, or temporal blocks co-located with the video block. [Means for solving the problem]

[0005] In an example implementation, a method for decoding a video block including multiple sub-blocks of a current frame in a video bitstream is disclosed. The method may include receiving a syntax element signaled in the video bitstream indicating whether the video block is inter-predicted under a sub-block motion compensation mode, determining, based on a value of the received syntax element, that the video block is inter-predicted under the sub-block motion compensation mode, determining, based on available candidate motion vectors including motion information of one or more neighboring blocks of the video block in the current frame and temporal blocks of the video block, a set of reference motion vectors corresponding to a set of neighboring blocks or sub-block positions of the video block, and deriving, based on the reference motion vectors, a sub-block motion vector for each of the multiple sub-blocks.

[0006] Aspects of the present disclosure also provide an electronic device or apparatus including a circuit or processor configured to perform any of the above method implementations.

[0007] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by an electronic device, cause the electronic device to perform any one of the implementations of the above methods. [Brief explanation of the drawings]

[0008] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings.

[0009] [Figure 1] 1 is a simplified block diagram schematic of a communication system (100) according to an exemplary embodiment.

[0010] [Figure 2] 1 is a simplified block diagram schematic of a communication system (200) according to an exemplary embodiment.

[0011] [Figure 3] FIG. 2 is a schematic diagram of a simplified block diagram of a video decoder according to an example embodiment.

[0012] [Figure 4] FIG. 2 is a schematic diagram of a simplified block diagram of a video encoder according to an example embodiment;

[0013] [Figure 5] FIG. 2 is a block diagram of a video encoder according to another example embodiment.

[0014] [Figure 6] FIG. 4 is a block diagram of a video decoder according to another example embodiment.

[0015] [Figure 7] FIG. 1 illustrates a coding block partitioning scheme according to an exemplary embodiment of the present disclosure.

[0016] [Figure 8] FIG. 10 illustrates another scheme for coding block partitioning according to an exemplary embodiment of the present disclosure.

[0017] [Figure 9] FIG. 10 illustrates another scheme for coding block partitioning according to an exemplary embodiment of the present disclosure.

[0018] [Figure 10] FIG. 1 illustrates composite motion compensation.

[0019] [Figure 11] FIG. 2 illustrates an exemplary interpolated reference frame for motion compensation.

[0020] [Figure 12]FIG. 10 illustrates an example of using neighboring motion vectors to generate motion vectors for sub-blocks.

[0021] [Figure 13] FIG. 10 illustrates pixel-level motion vector suppression for exemplary sub-block motion compensation.

[0022] [Figure 14] FIG. 10 illustrates example neighboring virtual sub-blocks for motion vector attributes for sub-block motion compensation.

[0023] [Figure 15] FIG. 10 illustrates an exemplary sub-block motion vector generation based on motion vectors attributed to neighboring virtual sub-blocks.

[0024] [Figure 16] FIG. 10 illustrates a next neighboring sub-block for use in sub-block motion compensation.

[0025] [Figure 17] FIG. 1 illustrates an exemplary logic flow of a method for sub-block motion compensation.

[0026] [Figure 18] FIG. 1 is a schematic diagram of a computer system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0027] Throughout this specification and the claims, terms may have nuanced meanings that are suggested or implied in context beyond their explicitly stated meaning. As used herein, the phrases "in one embodiment / implementation" or "in some embodiments / implementations" do not necessarily refer to the same embodiment / implementation, and as used herein, the phrases "in another embodiment / implementation" or "in other embodiments" do not necessarily refer to different embodiments. For example, claimed subject matter is intended to include combinations of example embodiments / implementations, in whole or in part.

[0028] In general, terms may be understood, at least in part, from their usage in context. For example, as used herein, terms such as "and," "or," or "and / or" may have various context-dependent meanings. Typically, when "or" is used to relate a list such as A, B, or C, it is intended to mean A, B, and C, which are used herein in an inclusive sense, as well as A, B, or C, which are used herein in an exclusive sense. Additionally, as used herein, the terms "one or more," "at least one," "a," "an," or "the" may be used in a singular or plural sense, depending at least in part on the context. Additionally, the terms "based on" or "determined by" may be understood as not necessarily intended to convey an exclusive set of factors, but instead may allow for the presence of additional factors not necessarily explicitly recited, depending at least in part on the context.

[0029] Figure 1 is a simplified block diagram of a communication system (100) according to an embodiment of the present invention. The communication system (100) includes multiple terminal devices, e.g., 110, 120, 130, and 140, that can communicate with each other, e.g., via a network (150). In the example of Figure 1, a first pair of terminal devices (110) and (120) may perform unidirectional data transmission. For example, the terminal device (110) may code video data in the form of one or more coded bitstreams (e.g., streams of video pictures captured by the terminal device (110)) for transmission over the network (150). The terminal device (120) may receive the coded video data from the network (150), decode the coded video data to recover the video pictures, and display the video pictures according to the recovered video data. The unidirectional data transmission may be implemented in a media service application, etc.

[0030] In another example, the second pair of terminal devices (130) and (140) may perform bidirectional transmission of coded video data, such as during a video conferencing application. For bidirectional transmission of data, in one example, each of the terminal devices (130) and (140) may code video data for transmission (e.g., of a stream of video pictures captured by the terminal device) and may also receive coded video data from another of the terminal devices (130) and (140) to reconstruct and display the video pictures.

[0031] In the example of FIG. 1 , the terminal devices may be implemented as servers, personal computers, and smartphones, although the applicability of the principles underlying the present disclosure need not be so limited. Embodiments of the present disclosure may also be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and the like. Network 150 represents any number or type of network that carries coded video data between terminal devices, including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data over circuit-switched, packet-switched, and / or other types of channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet.

[0032] 2 shows the arrangement of a video encoder and a video decoder in a video streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter may be equally applicable to other video applications, including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0033] As shown in Figure 2, a video streaming system may include a video capture subsystem (213), which may include a video source (201), such as a digital camera, to create a stream of uncompressed video pictures or images (202). In one example, the stream of video pictures (202) includes samples recorded by the digital camera of the video source (201). The stream of video pictures (202), depicted as a thick line to emphasize its high data volume when compared to the encoded video data (204) (or coded video bitstream), may be processed by an electronic device (220) that includes a video encoder (203) coupled to the video source (201). The video encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The coded video data 204 (or coded video bitstream 204), depicted as a thin line to emphasize its lower data volume when compared to the stream of uncompressed video pictures 202, may be stored on the streaming server 205 for future use or directly on a downstream video device (not shown). One or more streaming client subsystems, such as the client subsystems 206 and 208 of FIG. 2, can access the streaming server 205 to retrieve copies 207 and 209 of the coded video data 204. The client subsystem 206 may include a video decoder 210, for example, within the electronic device 230. The video decoder 210 decodes the input copy 207 of the coded video data to create an output stream of video pictures 211 that is uncompressed and can be rendered on a display 212 (e.g., a display screen) or other rendering device (not shown).

[0034] 3 shows a block diagram of a video decoder (310) of an electronic device (330) according to any of the following embodiments of the present disclosure. The electronic device (330) may include a receiver (331) (e.g., a receiving circuit). The video decoder (310) may be used in place of the video decoder (210) in the example of FIG. 2.

[0035] As shown in FIG. 3, the receiver (331) may receive one or more coded video sequences from the channel (301). To combat network jitter and / or handle playback timing, a buffer memory (315) may be disposed between the receiver (331) and an entropy decoder / parser (320) (hereinafter, "parser (320)"). The parser (320) may reconstruct symbols (321) from the coded video sequences. Categories of these symbols include information used to manage the operation of the video decoder (310) and potential information for controlling a rendering device such as a display (312) (e.g., a display screen). The parser (320) may parse / entropy decode the coded video sequences. The parser (320) may extract from the coded video sequences a set of subgroup parameters for at least one of the subgroups of pixels within the video decoder. Subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (320) may also extract information from the coded video sequence, such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc. Reconstruction of the symbols (321) may involve multiple different processing or functional units. The involved units and how they participate may be controlled by subgroup control information parsed from the coded video sequence by the parser (320).

[0036] The first unit may include a scalar / inverse transform unit (351), which may receive control information (321) from the parser (320) including information indicating the type of inverse transform to use, block size, quantization coefficients / parameters, quantization scaling matrix, and position as a symbol, along with quantized transform coefficients. The scalar / inverse transform unit (351) may output blocks containing sample values ​​that may be input to an aggregator (355).

[0037] In some cases, the output samples of the scaler / inverse transform (351) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) may generate blocks of the same size and shape as the block being reconstructed using surrounding block information that has already been reconstructed and stored in the current picture buffer (358). The current picture buffer (358) may, for example, buffer a partially reconstructed and / or a fully reconstructed current picture. In some implementations, the aggregator (355) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (352) to the output sample information provided by the scaler / inverse transform unit (351).

[0038] In other cases, the output samples of the scaler / inverse transform unit (351) may relate to an inter-coded, potentially motion-compensated, block. In such cases, the motion-compensated prediction unit (353) can access a reference picture memory (357) based on the motion vector to fetch samples used for inter-picture prediction. After the fetched reference samples are motion-compensated according to the symbols (321) associated with the block, these samples can be added by an aggregator (355) to the output of the scaler / inverse transform unit (351) (the output of unit 351 may be referred to as residual samples or a residual signal) to generate output sample information.

[0039] The output samples of the aggregator (355) may be subjected to various loop filtering techniques in a loop filter unit (356), which may include several types of loop filters. The output of the loop filter unit (356) may be a sample stream that may be output to a rendering device (312) and stored in a reference picture memory (357) for use in future inter-picture prediction.

[0040] 4 shows a block diagram of a video encoder (403) according to an exemplary embodiment of the present disclosure. The video encoder (403) may be included in an electronic device (420). The electronic device (420) may further include a transmitter (440) (e.g., a transmitting circuit). The video encoder (403) may be used in place of the video encoder (203) in the example of FIG. 2.

[0041] The video encoder (403) may receive video samples from a video source (401). According to some exemplary embodiments, the video encoder (403) may code and compress pictures of a source video sequence into a coded video sequence (443) in real time or under any other time constraint required by the application. Achieving an appropriate coding rate constitutes one function of the controller (450). In some embodiments, the controller (450) may be operatively coupled to and control other functional units, as described below. Parameters set by the controller (450) may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, ...), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc.

[0042] In some exemplary embodiments, the video encoder (403) may be configured to operate in a coding loop. The coding loop may include a source coder (430) and a (local) decoder (433) embedded in the video encoder (403). Even if the embedded decoder 433 processes the coded video stream by the source coder 430 without entropy coding (since any compression between the symbols in entropy coding and the coded video bitstream may be lossless in the video compression techniques contemplated by the disclosed subject matter), the decoder (433) reconstructs symbols to generate sample data in a manner similar to that generated by a (remote) decoder. It can be observed at this point that any decoder technology, with the exception of analysis / entropy decoding, which may only exist in the decoder, must exist in substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter may focus on decoder operations related to the decoding portion of the encoder. Therefore, descriptions of the encoder technology may be omitted, as it is the reverse of the decoder technology, which is described generically. Only in certain areas or aspects will a more detailed description of the encoder be provided below.

[0043] During operation in some example implementations, the source coder (430) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as "reference pictures."

[0044] The local video decoder (433) may decode coded video data of pictures that may be designated as reference pictures. The local video decoder (433) may replicate the decoding process that may be performed by a video decoder on the reference pictures and store the reconstructed reference pictures in a reference picture cache (434). In this way, the video encoder (403) may locally store copies of reconstructed reference pictures that have content in common with reconstructed reference pictures obtained by a far-end (remote) video decoder (in the absence of transmission errors).

[0045] The predictor (435) may perform the predictive search for the coding engine (432). That is, for a new picture to be coded, the predictor (435) may search the reference picture memory (434) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors, block shapes, etc., that can serve as suitable predictive references for the new picture.

[0046] The controller (450) may manage the coding operations of the source coder (430), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0047] The output of all of the above functional units may be subject to entropy coding in an entropy coder (445). The transmitter (440) may buffer the coded video sequence produced by the entropy coder (445) and prepare it for transmission over a communication channel (460), which may be a hardware / software link to a storage device that stores the coded video data. The transmitter (440) may merge the coded video data from the video coder (403) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0048] The controller (450) may manage the operation of the video encoder (403). During coding, the controller (450) may assign a particular coding picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types: intra picture (I picture), predicted picture (P picture), bidirectionally predicted picture (B picture), or multi-predicted picture. A source picture may generally be spatially subdivided into multiple sample coding blocks, as described in more detail below.

[0049] 5 shows a diagram of a video encoder (503) according to another exemplary embodiment of this disclosure. The video encoder (503) is configured to receive a processed block (e.g., a predictive block) of sample values ​​in a current video picture in a sequence of video pictures and encode the processed block into a coded picture that is part of a coded video sequence. The video encoder (503) may be used in place of the video encoder (403) in the example of FIG. 4.

[0050] For example, the video encoder (503) receives a matrix of sample values ​​for a processing block. The video encoder (503) then determines, for example, using rate-distortion optimization (RDO), whether the processing block is best coded using an intra-mode, an inter-mode, or a bi-predictive mode.

[0051] In the example of Figure 5, the video encoder (503) includes an inter-encoder (530), an intra-encoder (522), a residual calculator (523), a switch (526), ​​a residual encoder (524), a general controller (521), and an entropy encoder (525), which are coupled together as shown in the exemplary configuration of Figure 5.

[0052] The inter-encoder (530) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in previous and subsequent pictures in display order), generate inter-prediction information (e.g., a description of redundant information according to an inter-coding technique, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique.

[0053] The intra encoder (522) is configured to receive samples of a current block (e.g., a processing block), compare the block with previously coded blocks in the same picture, generate transformed and quantized coefficients, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques).

[0054] The general controller (521) may be configured to determine general control data and control other components of the video encoder (503) based on the general control data, for example, to determine a prediction mode for a block and provide a control signal to the switch (526) based on the prediction mode.

[0055] The residual calculator (523) may be configured to calculate the difference (residual data) between a received block and a prediction result for a block selected from the intra-encoder (522) or inter-encoder (530). The residual encoder (524) may be configured to encode the residual data to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various exemplary embodiments, the video encoder (503) also includes a residual decoder (528). The residual decoder (528) is configured to perform an inverse transform to generate decoded residual data. The entropy encoder (525) may be configured to format a bitstream to include the coded blocks and perform entropy coding.

[0056] 6 shows a diagram of an exemplary video decoder (610) according to another embodiment of the present disclosure. The video decoder (610) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (610) may be used in place of the video decoder (410) in the example of FIG. 4.

[0057] In the example of Figure 6, the video decoder (610) includes an entropy decoder (671), an inter-decoder (680), a residual decoder (673), a reconstruction module (674), and an intra-decoder (672), which are coupled together as shown in the example configuration of Figure 6.

[0058] The entropy decoder (671) may be configured to reconstruct, from a coded picture, certain symbols that represent the syntax elements of which the coded picture is composed. The inter decoder (680) may be configured to receive inter prediction information and generate inter prediction results based on the inter prediction information. The intra decoder (672) may be configured to receive intra prediction information and generate prediction results based on the intra prediction information. The residual decoder (673) may be configured to perform inverse quantization, extract inverse quantized transform coefficients, and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The reconstruction module (674) may be configured to combine, in the spatial domain, the residual output by the residual decoder (673) and the prediction results (possibly output by the inter or intra prediction module) to form reconstructed blocks that form part of the reconstructed picture as part of the reconstructed video.

[0059] It should be noted that the video encoders (203), (403), and (503) and the video decoders (210), (310), and (610) can be implemented using any suitable technology. In some exemplary embodiments, the video encoders (203), (403), and (503) and the video decoders (210), (310), and (610) can be implemented using one or more integrated circuits. In other embodiments, the video encoders (203), (403), and (503) and the video decoders (210), (310), and (610) can be implemented using one or more processors executing software instructions.

[0060] Turning to block partitioning for coding and decoding, general partitioning may start from a base block and may follow a predetermined set of rules, a specific pattern, a partition tree, or any partition structure or scheme. The partitioning may be hierarchical and recursive. After dividing or partitioning the base block according to any of the exemplary partitioning procedures or other procedures described below, or a combination thereof, a final set of partitions or coding blocks may be obtained. Each of these partitions may be at one of various partitioning levels in the partitioning hierarchy and may have various shapes. Each partition may be referred to as a coding block (CB). In various exemplary partitioning implementations described further below, each resulting CB may be of any of the allowed sizes and partitioning levels. Such partitions are referred to as coding blocks because they may form a unit around which some basic coding / decoding decisions are made, coding / decoding parameters are optimized and determined, and signaled within the coded video bitstream. The highest or deepest level of the final partition represents the depth of the tree's coding block partitioning structure. A coding block may be a luma coding block or a chroma coding block. The CB tree structure for each color may be referred to as a coding block tree (CBT). The coding blocks for all color channels may be collectively referred to as a coding unit (CU). The hierarchical structures for all color channels may be collectively referred to as a coding tree unit (CTU). The partitioning pattern or structure of various color channels within a CTU may or may not be the same.

[0061] In some implementations, the partition tree scheme or structure used for the luma channel and the chroma channel may not need to be the same. In other words, the luma channel and the chroma channel may have separate coding tree structures or patterns. Furthermore, whether the luma channel and the chroma channel use the same or different coding partition tree structures, and the actual coding partition tree structure used, may depend on whether the slice being coded is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma channel and the luma channel may have separate coding partition tree structures or coding partition tree structure modes, while for a P or B slice, the luma channel and the chroma channel may share the same coding partition tree scheme. When separate coding partition tree structures or modes are applied, the luma channel may be partitioned into CBs by one coding partition tree structure, and the chroma channel may be partitioned into chroma CBs by another coding partition tree structure.

[0062] FIG. 7 shows ten exemplary predefined partitioning structures / patterns that allow recursive partitioning to form a partitioning tree. The root block may start at a predefined level (e.g., from a base block at the 128x128 or 64x64 level). The exemplary partitioning structure of FIG. 7 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. In some exemplary implementations, none of the rectangular partitions in FIG. 7 are allowed to be further subdivided. A coding tree depth may be further defined to indicate the partitioning depth from the root node or root block. For example, the coding tree depth of the root node or root block may be set to 0, and after one further division of the root block according to FIG. 7, the coding tree depth is increased by 1. In some implementations, only all square partitions within 710 may be allowed for recursive partitioning to the next level of the partitioning tree according to the pattern of FIG. 7.

[0063] In some other exemplary implementations for coding block partitioning, a quadtree structure may be used. Such quadtree division may be applied hierarchically and recursively to any square-shaped partition. Whether a base block or an intermediate block or partition is further quadtree divided can be adapted to various local characteristics of the base block or intermediate block / partition.

[0064] In yet some other examples, a ternary partitioning scheme may be used to partition the base block or any intermediate blocks, as shown in FIG. 8. The ternary pattern may be implemented vertically, as shown at 802, or horizontally, as shown at 804. While the example division ratio in FIG. 8 is shown as 1:2:1, other ratios may be predefined. In some implementations, two or more different ratios may be predefined. In some implementations, the width and height of the partitions of the example triple tree are always powers of two to avoid additional transformations.

[0065] The above partitioning schemes may be combined in any manner at different partitioning levels. As an example, the above-described quadtree and binary partitioning schemes may be combined to partition the base block into a quadtree-binary-tree (QTBT) structure. In such a scheme, the base block or intermediate blocks / partitions may be quadtree-partitioned or binary-partitioned, if specified, according to a set of predefined conditions. A specific example is illustrated in FIG. 9, where the base block is first quadtree-partitioned into four partitions indicated by 902, 904, 906, and 908. Each of the resulting partitions is then either quadtree-partitioned into four further partitions (e.g., 908) or binary-partitioned at the next level into two further partitions (e.g., both symmetric, either horizontally or vertically, as in 902 or 906), or not partitioned (e.g., 904). As shown by the overall example partition pattern in 910 and the corresponding tree structure / representation in 920, binary or quadtree splitting may be recursively allowed for square-shaped partitions, where solid lines represent quadtree splitting and dashed lines represent binary splitting. A flag may be used to indicate whether the binary split is horizontal or vertical for each binary split node (non-leaf binary partition). For example, as shown in 920, a flag "0" may represent a horizontal binary split and a flag "1" may represent a vertical binary split, consistent with the partitioning structure of 910. For quadtree-partitioned partitions, there is no need to indicate the split type, since quadtree splitting always splits a block or partition both horizontally and vertically to generate four sub-blocks / partitions of equal size. In some implementations, a flag "1" may represent a horizontal binary split and a flag "0" may represent a vertical binary split.

[0066] In some example implementations of QTBT, the quadtree and binary splitting rule sets may be represented by the following predefined parameters and their associated corresponding functions: - CTU size: Quad tree root node size (base block size) - MinQTSize: Minimum allowed quad tree leaf node size - MaxBTSize: Maximum allowed binary tree root node size - MaxBTDepth: Maximum allowed binary tree depth - MinBTSize: Minimum allowed binary tree leaf node size In some exemplary implementations of the QTBT partitioning structure, the CTU size may be set as 128x128 luma samples with two corresponding 64x64 blocks of chroma samples (when exemplary chroma subsampling is considered and used), MinQTSize may be set as 16x16, MaxBTSize may be set as 64x64, MinBTSize (for both width and height) may be set as 4x4, and MaxBTDepth may be set as 4. Quad-tree partitioning may be first applied to the CTU to generate quad-tree leaf nodes. The quad-tree leaf nodes may have sizes from the minimum allowed size of 16x16 (i.e., MinQTSize) to 128x128 (i.e., the CTU size). If a node is 128x128, it will not be initially partitioned by the binary tree because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, nodes that do not exceed MaxBTSize may be partitioned by a binary tree. In the example of Figure 9, the base block is 128x128. The base block can only be quadtree partitioned according to a predefined set of rules. The base block has a partition depth of 0. Each of the resulting four partitions is 64x64, does not exceed MaxBTSize, and may be further quadtree or binary tree partitioned at level 1. The process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), further partitions may not be considered. When a binary tree node has a width equal to MinBTSize (i.e., 4), further horizontal partitions may not be considered. Similarly, when a binary tree node has a height equal to MinBTSize, further vertical partitions may not be considered.

[0067] In some example implementations, the above-described QTBT scheme may be configured to support the flexibility of luma and chroma having the same or separate QTBT structures. For example, for P slices and B slices, the luma CTB and chroma CTB within one CTU may share the same QTBT structure. However, for I slices, the luma CTB may be partitioned into CBs by a QTBT structure, and the chroma CTB may be partitioned into chroma CBs by a different QTBT structure. This means that a CU may be used to refer to different color channels within an I slice; for example, an I slice may consist of a coding block for a luma component or a coding block for two chroma components, and a CU in a P or B slice may consist of coding blocks for all three color components.

[0068] The various CB partitioning schemes described above and further partitioning of the CB into PBs may be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0069] Inter prediction may be implemented, for example, in single reference mode or mixed reference mode. In some implementations, a skip flag may be initially included in the bitstream of the current block (or at a higher level) to indicate whether the current block is inter-coded and not skipped. If the current block is inter-coded, another flag may also be included in the bitstream as a signal to indicate whether single reference mode or mixed reference mode is used to predict the current block. In single reference mode, one reference block may be used to generate the prediction block of the current block. In mixed reference mode, two or more reference blocks may be used to generate the prediction block, for example, by weighted averaging. One reference block or multiple reference blocks may be identified using one reference frame index or multiple reference frame indexes, additionally using a corresponding one or multiple motion vectors indicating a shift between the reference block and the current block in position relative to the frame, for example, in horizontal and vertical pixels. For example, in single reference mode, the inter-predicted block of the current block may be generated from a single reference block identified as the predictive block by one motion vector in a reference frame, while in mixed reference mode, the predictive block may be generated by a weighted average of two reference blocks in two reference frames indicated by two reference frame indices and two corresponding motion vectors. The motion vectors may be coded and included in the bitstream in various ways.

[0070] In some example implementations, one or more reference picture lists including identification of short-term and long-term reference frames for inter prediction may be formed based on information in a reference picture set (RPS). For example, a single picture reference list may be formed for unidirectional inter prediction, denoted as L0 reference (or reference list 0), while two picture reference lists may be formed for bidirectional inter prediction, denoted as L0 (or reference list 0) and L1 (or reference list 1), for each of the two prediction directions. The reference frames included in the L0 and L1 lists may be ordered in various predetermined manners. The lengths of the L0 and L1 lists may be signaled in the video bitstream. Unidirectional inter prediction may be a single reference mode or a mixed reference mode when multiple references for generating a prediction block by weighted averaging in a mixed prediction mode are on the same side of the frame in which the predicted block is located. Bidirectional inter prediction may be only a mixed mode in that bidirectional inter prediction includes at least two reference blocks.

[0071] In some implementations, a merge mode (MM) for inter prediction may be implemented. Generally, in the merge mode, one or more of the motion vectors in single reference prediction or mixed reference prediction for the current PB may be derived from other motion vectors rather than being calculated and signaled independently. For example, in an encoding system, the current motion vector of the current PB may be represented by a difference between the current motion vector and one or more other already coded motion vectors (called reference motion vectors). Such a difference in the motion vector, rather than the entire current motion vector, may be coded and included in the bitstream, and may be linked to the reference motion vector. Correspondingly, in a decoding system, the motion vector corresponding to the current PB may be derived based on the decoded motion vector difference and the decoded reference motion vector linked thereto. As a specific form of the general merge mode (MM) inter prediction, such inter prediction based on the motion vector difference may be referred to as a merge mode with motion vector difference (MMVD). Therefore, MM in general, and MMVD in particular, can be implemented to exploit correlations between motion vectors associated with different PBs to improve coding efficiency. For example, neighboring PBs can have similar motion vectors, so that the MVD can be small and can be coded efficiently. In another example, motion vectors can be temporally correlated (between frames) with respect to similarly positioned / located blocks in space.

[0072] In some example implementations of MMVD, a list of reference motion vector (RMV) or MV predictor candidates for motion vector prediction may be formed for a block being predicted. The list of RMV candidates may include a predetermined number (e.g., two) of MV predictor candidate blocks whose motion vectors can be used to predict the current motion vector. The RMV candidate blocks may include blocks selected from neighboring blocks in the same frame and / or temporal block (e.g., blocks at the same location in a frame preceding or following the current frame). These options represent blocks at spatial or temporal locations relative to the current block that are likely to have similar or identical motion vectors to the current block. The size of the list of MV predictor candidates may be predetermined. For example, the list may include two or more candidates. To be on the list of RMV candidates, a candidate block may be required, for example, to have the same reference frame (or multiple reference frames) as the current block, must exist (e.g., a boundary check needs to be performed when the current block is near an edge of a frame), and must have been previously coded during the coding process and / or decoded during the decoding process. In some implementations, the list of merge candidates may first be populated with spatial neighboring blocks (scanned in a certain predefined order) if they are available and satisfy the above conditions, and then with temporal blocks if space is still available in the list. Neighboring RMV candidate blocks may be selected, for example, from the blocks to the left and above the current block. The list of RMV predictor candidates may be dynamically formed at various levels (sequence, picture, frame, slice, superblock, etc.) as a Dynamic Reference List (DRL). The DRL may be signaled in the bitstream.

[0073] In some implementations, the actual MV predictor candidate used as the reference motion vector for predicting the motion vector of the current block may be signaled. In the case where the RMV candidate list includes two candidates, a one-bit flag called a merge candidate flag may be used to indicate the selection of the reference merge candidate. For the current block being predicted in combined mode, each of the multiple motion vectors predicted using the MV predictor may be associated with a reference motion vector from the merge candidate list. The encoder may determine which of the RMV candidates more closely predicts the MV of the current coding block and signal the selection as an index to the DRL.

[0074] In some example implementations of MMVD, after an RMV candidate is selected and used as a base motion vector predictor for a predicted motion vector, a motion vector difference (MVD or delta MV representing the difference between the predicted motion vector and a reference candidate motion vector) may be calculated within the coding system. Such MVD may include information representing the magnitude of the MV difference and the direction of the MV difference, both of which may be signaled within the bitstream in various ways.

[0075] In some exemplary embodiments of MMVD, a distance index may be used to specify motion vector difference magnitude information and indicate one of a set of predefined offsets representing a predefined motion vector difference from a starting point (reference motion vector). The MV offset according to the signaled index may then be added to either the horizontal or vertical component of the starting (reference) motion vector. An exemplary predefined relationship between the distance index and the predefined offset is specified in Table 1. [Table 1]

[0076] In some exemplary implementations of MMVD, a direction index may be further signaled and used to indicate the direction of the MVD relative to the reference motion vector. In some implementations, the direction may be limited to either the horizontal or vertical direction. An exemplary 2-bit direction index is shown in Table 2. In the example of Table 2, the interpretation of the MVD may change depending on the information of the start / reference MV. For example, when the start / reference MV corresponds to a uni-predictive block or a bi-predictive block where both reference frame lists point to the same side of the current picture (i.e., the POCs of the two reference pictures are both greater than or both less than the POC of the current picture), the code in Table 2 may specify the sign (direction) of the MV offset added to the start / reference MV. When the start / reference MV corresponds to a bi-predictive block with two reference pictures on different sides of the current picture (i.e., the POC of one reference picture is larger than the POC of the current picture and the POC of the other reference picture is smaller than the POC of the current picture) and the difference between the reference POC of picture reference list 0 and the current frame is larger than the difference between the reference POC of picture reference list 1 and the current frame, the code in Table 2 may specify the sign of the MV offset added to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture in picture reference list 1 may have the opposite value (opposite sign to the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is larger than the difference between the reference POC in picture reference list 0 and the current frame, the code in Table 2 may specify the sign of the MV offset added to the reference MV associated with picture reference list 1, and the sign of the offset to the reference MV associated with picture reference list 0 has the opposite value. [Table 2]

[0077] In some example implementations, the MVD may be scaled according to the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling is required. Otherwise, if the difference in POC in reference list 0 is larger than that in reference list 1, the MVD in reference list 1 is scaled. If the difference in POC in reference list 1 is larger than that in list 0, the MVD in list 0 may be scaled in the same way. If the starting MV is uni-predicted, the MVD is added to the available MV or reference MV.

[0078] In some example implementations of MVD coding and signaling for bidirectional mixed prediction, in addition to or instead of separately coding and signaling two MVDs, symmetric MVD coding may be implemented so that only one MVD needs to be signaled and the other MVD can be derived from the signaled MVD. In such implementations, motion information including reference picture indices for list 0 and list 1 are not signaled. Specifically, at the slice level, a flag called "mvd_l1_zero_flag" may be included in the bitstream to indicate whether reference list 1 is not signaled in the bitstream. If this flag is 1, indicating that reference list 1 is equal to zero (and therefore not signaled), a bidirectional prediction flag called "BiDirPredFlag" may be set to 0, meaning there is no bidirectional prediction. Otherwise, if mvd_l1_zero_flag is zero, then BiDirPredFlag may be set to 1 if the closest reference picture in list 0 and the closest reference picture in list 1 form a forward and backward pair of reference pictures or a backward and forward pair of reference pictures, and both reference pictures in list 0 and list 1 are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. BiDirPredFlag being 1 may indicate that a symmetric mode flag is additionally signaled in the bitstream. A decoder may extract the symmetric mode flag from the bitstream when BiDirPredFlag is 1. The symmetric mode flag may be signaled, for example, at the CU level (if needed), and may indicate whether symmetric MVD coding mode is used for the corresponding CU. When the symmetric mode flag is 1, this indicates the use of symmetric MVD coding mode, and indicates that only the reference picture indices for both list 0 and list 1 (called "mvp_l0_flag" and "mvp_l1_flag") are signaled in the MVD associated with list 0 (called "MVD0"), and the other motion vector difference "MVD1" is derived rather than signaled.For example, MVD1 may be derived as −MVD0. Thus, in the exemplary symmetric MVD mode, only one MVD is signaled.

[0079] In some other example implementations for MV prediction, a harmonized scheme may be used to implement general merge mode, MMVD, and some other types of MV prediction, for both single-reference mode MV prediction and mixed-reference mode MV prediction. Various syntax elements may be used to signal how the MV of the current block is predicted. For example, for single-reference mode, the following MV prediction modes may be signaled:

[0080] NEARMV - bypasses MVD and directly uses one of the motion vector predictors (MVP) in the list indicated by the DRL (Dynamic Reference List) index.

[0081] NEWMV - Use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference and apply a delta to the MVP (e.g., using MVD).

[0082] GLOBALMV - Use motion vectors based on frame-level global motion parameters.

[0083] Similarly, for a mixed reference inter prediction mode using two reference frames corresponding to the two MVs to be predicted, the following MV prediction modes may be signaled:

[0084] NEAR_NEARMV - For each of the two MVs to be predicted, use one of the motion vector predictors (MVPs) in the list signaled by the DRL index, without using the MVD.

[0085] NEAR_NEWMV - To predict the first of two motion vectors, one of the motion vector predictors (MVPs) in the list signaled by the DRL index is used as a reference MV without MVD, and to predict the second of two motion vectors, one of the motion vector predictors (MVPs) in the list signaled by the DRL index is used as a reference MV with an additionally signaled delta MV (MVD).

[0086] NEW_NEARMV - Use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV without MVD to predict the second of two motion vectors, and use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV with the additionally signaled delta MV (MVD) to predict the first of two motion vectors.

[0087] NEW_NEWMV - Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference MV, and use this together with the additionally signaled delta MV (MVD) to predict for each of the two MVs.

[0088] GLOBAL_GLOBALMV - Use the MV from each reference based on their frame-level global motion parameters.

[0089] Thus, the term "NEAR" above refers to MV prediction using a reference MV without an MVD, as a general merge mode, while the term "NEW" refers to MV prediction using a reference MV and offsetting it with a signaled or derived MVD, as in the MMVD mode. In the case of hybrid inter prediction, both the reference-based motion vector and the above-mentioned motion vector delta may generally be different or independent between the two references or two MVDs, even if the two MVDs are, for example, correlated, and such correlation is utilized to reduce the amount of information required to signal the two motion vector deltas. To utilize such correlation, joint signaling of the two MVDs may be implemented and indicated in the bitstream, as described in more detail below.

[0090] In some exemplary implementations of MVD, a predefined pixel resolution for the MVD may be allowed. For example, a motion vector accuracy (or precision) of 1 / 8 pixel may be allowed. The MVDs described above for various MV prediction modes may be constructed and signaled in various ways. In some implementations, various syntax elements may be used to signal the above motion vector difference in reference frame list 0 or list 1.

[0091] For example, a syntax element called "mv_joint" may specify which components of its associated motion vector difference are non-zero. An mv_joint with a value of 0 may indicate that there is no non-zero MVD in either the horizontal or vertical direction; An mv_joint with a value of 1 may indicate that there is a non-zero MVD only in the horizontal direction, An mv_joint with a value of 2 may indicate that there is non-zero MVD only in the vertical direction, A mv_joint with a value of 3 may indicate that there is a non-zero MVD in both the horizontal and vertical directions.

[0092] For MVD, when the "mv_joint" syntax element signals that there are no non-zero MVD components, no further MVD information can be signaled. However, if the "mv_joint" syntax signals that there are one or two non-zero components, additional syntax elements can be further signaled for each non-zero MVD component, as described below.

[0093] For example, a syntax element called "mv_sign" may be used to additionally specify whether the corresponding motion vector difference component is positive or negative.

[0094] In another example, a syntax element called "mv_class" may be used to specify the class of a motion vector difference from among a set of predefined classes for the corresponding non-zero MVD component. For example, the predefined classes of motion vector differences may be used to divide the continuous magnitude space of motion vector differences into non-overlapping ranges of classes. Thus, the signaled MVD class indicates the magnitude range of the corresponding MVD component. In the example implementation shown in Table 3 below, higher classes correspond to motion vector differences with a larger magnitude range. The symbol (n,m) represents a range of motion vector differences that is greater than n pixels and less than or equal to m pixels. [Table 3]

[0095] In some other examples, a syntax element called “mv_bit” may be further used to specify the integer portion of the offset between a non-zero motion vector difference component and the starting magnitude of the magnitude range of the correspondingly signaled MV class. In some other examples, a syntax element called “mv_fr” may be further used to specify the first two fractional bits of the motion vector difference of the corresponding non-zero MVD component, while a syntax element called “mv_hp” may be used to specify the third fractional bit (high-resolution bit) of the motion vector difference of the corresponding non-zero MVD component. The two “mv_fr” bits essentially provide 1 / 4 pixel MVD resolution, while the “mv_hp” bits may further provide 1 / 8 pixel resolution. In some other implementations, two or more “mv_hp” bits may be used to provide MVD pixel resolution finer than 1 / 8 pixel. In some example implementations, additional flags may be signaled at one or more of various levels to indicate whether MVD resolution of 1 / 8 pixel or finer is supported. If an MVD resolution does not apply to a particular coding unit, the above syntax elements for the corresponding unsupported MVD resolution may not be signaled.

[0096] However, in some other exemplary implementations, the resolution of motion vector differences in various MVD magnitude classes may be differentiated or adaptive. Specifically, a high-resolution MVD for a large MVD magnitude of a higher MVD class may not provide a statistically significant improvement in compression efficiency or coding gain. Therefore, the MVD may be coded at a reduced or non-increasing resolution (integer pixel resolution or fractional pixel resolution) for a larger MVD magnitude range corresponding to a higher MVD magnitude class. The term "resolution" may also be referred to as "pixel resolution."

[0097] In some example implementations, each MVD class may be associated with a single allowed resolution. In some other implementations, one or more MVD classes may each be associated with two or more optional MVD pixel resolutions. For example, adaptively allowed MVD pixel resolutions may include, but are not limited to, 1 / 64 pel (pixel), 1 / 32 pel, 1 / 16 pel, 1 / 8 pel, 1-4 pel, 1 / 2 pel, 1 pel, 2 pel, 4 pel... (in descending order of resolution).

[0098] In some other example implementations, only a single MVD value may be allowed for MV classes above a threshold MV class. For example, such a threshold MV class may be MV_CLASS 2. Thus, MV_CLASS_2 and above may only be allowed to have a single MVD value and do not have fractional pixel resolution.

[0099] Turning to various composite inter-prediction modes in which each MV is predicted by a reference motion vector and can be coded by an MVD, the two MVDs may be signaled separately or jointly in a bitstream, as described above. Thus, in some exemplary embodiments, in addition to the above-described NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes, another inter-prediction mode called JOINT_NEWMV may be introduced into the mode of MVD for joint signaling of reference lists 0 and 1. Specifically, when the inter-prediction mode is indicated as NEW_NEWMV mode, the MVDs of reference lists 0 and 1 are signaled separately, while when the inter-prediction mode is indicated as JOINT_NEWMV mode, the MVDs of reference lists 0 and 1 are signaled jointly. In particular, for joint MVD, only one MVD, called joint_delta_mv, needs to be signaled and transmitted in the bitstream, and the MVDs of reference lists 0 and 1 may be derived from joint_delta_mv. The derived MVD may then be combined with the reference motion vectors in reference lists 0 or 1 to generate two motion vectors for locating reference blocks for hybrid inter prediction.

[0100] In some implementations of composite inter prediction, the JOINT_NEWMV mode may be signaled along with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes. In such implementations, syntax may be included in the bitstream for indicating any one of these alternative composite inter prediction modes at any one of various signal levels (e.g., sequence level, picture level, frame level, slice level, tile level, superblock level, etc.). Alternatively, the JOINT_NEWMV mode may be implemented as a sub-mode of the NEW_NEWMV mode. In other words, under the NEW_NEWMV mode, the two MVDs of the two reference blocks are either jointly signaled (hence, the JOINT_NEWMV sub-mode) or not (another sub-mode of the NEW_NEWMV mode). In such an implementation, a first syntax element may be included in the bitstream to indicate any one of NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes, and when the first syntax element indicates that NEW_NEWMV mode is selected for a coding block, a second syntax element may further be included in the bitstream and may be extractable by a decoder to indicate whether the MVDs of the coding block are signaled separately or jointly.

[0101] In some example implementations, when JOINT_NEWMV mode is signaled and the POC distances between two reference frames and the current frame are different, the MVD may be scaled for reference list 0 or reference list 1 based on the POC distances. Specifically, the distance between reference frame list 0 and the current frame may be denoted as td0, and the distance between reference frame list 1 and the current frame may be denoted as td1. If td0 is greater than or equal to td1, joint_mvd may be used directly for reference list 0, and the MVD of reference list 1 may be derived from joint_mvd based on equation (1).

number

[0102] Otherwise, if td1 is greater than or equal to td0, then joint_mvd for reference list 1 is used directly, and the MVD of reference list 0 may be derived from joint_mvd based on equation (2).

number

[0103] In some example implementations, another inter-coding mode named AMVDMV may be added to the single-reference case. When the AMVDMV mode is selected, this indicates that AMVD (Adaptive Motion Vector Difference) is applied to signal MVD. For example, a flag named amvd_flag may be added under the JOINT_NEWMV mode to indicate whether AMVD is applied to the joint MVD coding mode. When adaptive MVD resolution is applied to the joint MVD coding mode, the MVDs for the two reference frames are jointly signaled, and the accuracy of the MVD may be implicitly determined by the size of the MVD. Otherwise, the MVDs for the two (or more) reference frames are jointly signaled, and conventional MVD coding without adaptive MVD resolution may be applied.

[0104] Moving on to composite inter modes, as shown in Figure 10, these modes are based on two different reference frames F i-1 and F i+1 By combining two hypotheses for the motion vectors MV0 and MV1 from the current frame F i In this way, two motion information components (e.g., motion vectors) can be signaled in the bitstream for each block.

[0105] Alternatively, an interpolation process can be used to find the two reference frames F i-1 and F i+1 Combine the information in the current frame F i The interpolated frame may be projected to the same time instance as the current frame F to generate an interpolated frame. Multiple TIP modes may be supported. In one TIP mode, the interpolated frame may be used as an additional reference frame. i The coding blocks of F can directly reference the interpolated frame and take advantage of the information coming from two different references with only the overhead cost of a single inter prediction mode. Another TIP mode uses the interpolated frame as a reference to the current frame F while skipping other conventional coding steps. i This mode can have significant coding and complexity advantages, especially for low bitrate applications.

[0106] In some implementations, adaptive motion vector resolutions (AMVR) may be supported. In a particular exemplary implementation, a total of seven MV precisions (8, 4, 2, 1, 1 / 2, 1 / 4, 1 / 8) may be supported. For each prediction block, the AVM encoder may search all supported precision values ​​and signal the highest precision to the decoder.

[0107] To reduce the runtime of the encoder, two MV precision sets are supported. Each precision set may include, for example, four predefined precisions. The precision set may be adaptively selected at the frame level based on the maximum precision value of the frame. The maximum precision may be signaled in the frame header. Table 4 summarizes exemplary supported precision values ​​based on an exemplary frame-level maximum precision. [Table 4]

[0108] In some example implementations of AMVR, there may be a frame-level flag that indicates whether the MV of a frame includes sub-pel precision. AMVR may be enabled only if the value of the cur_frame_force_integer_mv flag is 0. In AMVR, if the precision of a block is less than the maximum precision, the motion model and interpolation filter may not be signaled. If the precision of a block is less than the maximum precision, the motion mode is estimated to translation motion, and the interpolation filter is estimated to a regular interpolation filter. Similarly, if the precision of a block is either 4-pel or 8-pel, the inter-intra mode may not be signaled and may be estimated to be 0.

[0109] In some example implementations, warped motion compensation may be applied. Unlike traditional motion compensation, which assumes a translational motion model between a reference block and a target block, warped motion may utilize an affine model, as conceptually shown in Figure 12. For example, an affine motion model can be expressed in equation (3):

number

[0110] In some example implementations, global motion information may be signaled for each inter reference frame, including a global motion type and several motion parameters. Example global motion types and associated parameter numbers are listed in Table 5. [Table 5]

[0111] After signaling the reference frame index, if global motion is selected for use, the global motion type and parameters associated with the given reference frame may be used for the current coding block.

[0112] Warp motion compensation may be applied locally at the block level. In some example implementations, for inter-coding blocks, local warp motion is allowed when the following conditions are met: (1) the current block uses single reference prediction; (2) the width or height of the coding block is greater than or equal to a threshold (e.g., 8); and (3) at least one of the adjacent neighboring blocks uses the same reference frame as the current block.

[0113] When such a local warp motion is used for the current block, the affine model parameters can be estimated by mean-squared minimization of the difference between the reference and modeled projections based on the MVs of the current block and its adjacent neighboring blocks. To estimate the parameters of the local warp motion, for example, if the neighboring block uses the same reference frame as the current block, a projection sample pair is obtained: the center sample of the neighboring block and its corresponding sample in the reference frame. Then, three extra samples can be created by shifting the center position by a quarter sample in one or both dimensions. These extra samples can also be considered as projection sample pairs to ensure the stability of the model parameter estimation process.

[0114] The MVs of neighboring blocks used to derive motion parameters may be called motion samples. The motion samples may be selected from neighboring blocks that use the same reference frame as the current block. The warped motion prediction mode may only be enabled for blocks that use a single reference frame. For example, in FIG. 12, the MVs of neighboring blocks B0, B1, and B2 are called MV0, MV1, and MV2, respectively. As an example, the current block may be predicted using single prediction with reference frame Ref0; neighboring block B0 may be predicted using mixed prediction with reference frames Ref0 and Ref1; neighboring block B1 may be predicted using single prediction with reference frame Ref0; and neighboring block B2 may be predicted using mixed prediction with reference frames Ref0 and Ref2. Motion vector MV0 of B0 Ref0 , B1's motion vector MV1 Ref0 and B2's motion vector MV2 Ref0 may be used as the motion sample for deriving the affine motion parameters of the current block.

[0115] In some exemplary implementations, both global and local warp motion models may be supported. For example, a global warp model may be associated with each reference frame, as described above, where each of the four non-translational parameters may be given a predefined bit (e.g., 12-bit) precision, and the translational motion vectors may be coded with another predefined bit (e.g., 15-bit) precision. A coding block may choose to use the global warp directly and provide a reference frame index. The global warp model captures frame-level scaling and rotation and thus focuses primarily on rigid motion across the frame. In a local warp model at the coding block level, e.g., in WARPED_CAUSAL mode, the warp parameters of the current block may be derived by fitting the model to nearby motion vectors using a least-squares method.

[0116] Another example for extending warp motion compensation, such as WARP_EXTEND, allows the motion of neighboring blocks to be smoothly extended into the current block, but with some ability to modify the warp parameters. This allows complex warping motion to be represented distributed across multiple blocks while minimizing blocking artifacts. Such a mode, applied to, for example, the NEWMV block, can be constructed based on two exemplary constraints: (1) the per-pixel motion vectors generated by this model must be contiguous with the per-pixel motion vectors in neighboring blocks (smoothness); and (2) the pixel at the center of the current block must have a per-pixel motion vector that matches the motion vector signaled for the entire block.

[0117] For example, if the block to the left of the current block is warped, the model that matches the motion vectors shown in FIG. 13 may be used as the warp model. The above two example constraints imply certain equations that include the warp parameters of the neighboring blocks and the current block. These equations can then be solved to calculate the warp model for the current block. For example, mathematically, if (A,...,F) represents the neighboring warp model and (A',...,F') represents the extended warp model, then the above constraint (1) implies that at each point along the common edge:

number

[0118] Points along the edge may have different y values, but they all have the same x value. This means that the y coefficients must be the same on both sides (e.g. B'=B and D'=D). The x coefficients, on the other hand, give two equations that relate them to the other coefficients:

number

[0119] Constraint (2) above specifies that the motion vector of the center of the block must be equal to the one signaled using the NEWMV mechanism. This gives two more equations, resulting in a system of six equations in six variables with a unique solution.

[0120] These equations can be solved efficiently in both software and hardware, and the solution amounts to basic addition / subtraction / multiplication and division by powers of 2. Therefore, this mode is significantly less complex than the least-squares based local warp mode.

[0121] In the above example implementation, there may be multiple neighboring blocks that can be used for extension, in which case a choice may need to be made as to which block to extend from. This may be a problem similar to that encountered with motion vector prediction (e.g., there may be several possible motion vectors from nearby blocks to choose from as the basis for NEWMV coding). The method used for motion vector prediction may be extended to handle the need for WARP_EXTEND described above. This may be done by tracking the source of each motion vector prediction. WARP_EXTEND may then be enabled only if the selected motion vector prediction is obtained from a directly adjacent block. That block may then be used as the single "neighboring block" in the remainder of the algorithm.

[0122] In some situations, the neighbor's warp model may be very good as is, without requiring any further modification. To make this case cheaper to code, WARP_EXTEND may be allowed to be used for the NEWMV block. In this case, the neighbor selection may be the same as for NEWMV, except that the neighbor may need to be warped rather than just translated. However, if this condition is true and WARP_EXTEND is selected, the neighbor's warp model parameters may be copied to the current block.

[0123] In some example implementations, a motion mode called WARP_DELTA may be used. In this mode, the warp model of a block may be coded as a delta from a predicted warp model, similar to how motion vectors are coded as deltas from predicted motion vectors. The prediction may be obtained either from a global motion model (if present) or from neighboring blocks.

[0124] To avoid having multiple ways of coding the same prediction warp model, the following exemplary constraints may be applied: if the mode is NEAR MV or NEW MV, the same neighborhood selection logic as described for WARP_EXTEND may be used; if this results in a neighboring block being warped, then that neighboring block's model may be used as the prediction (without applying the rest of the WARP_EXTEND logic); otherwise, the global warp model may be used as the base. Deltas for each of the non-translational parameters may then be coded. Finally, the translational part of the model may be adjusted so that the per-pixel motion vector at the center of the block matches the motion vector for the entire block.

[0125] Because the above example implementation explicitly codes the delta for each warp parameter, it may use more bits to encode than other warp modes. As such, WARP_DELTA may be disabled for blocks smaller than 16x16. However, while the decoding logic is relatively simple, it can express more complex motion that other warp modes cannot.

[0126] In some of the other example implementations described above, a video block may be divided into sub-blocks, and warped motion compensation may be applied at the sub-block level. For example, the positions of the sub-blocks within the current block may be warped (e.g., translated, rotated, scaled). Warped motion compensation may be performed by warping each sub-block as a whole, while performing regular motion compensation for each sub-block, with or without Prediction Refinement with Optical Flow (PROF). In such implementations, the motion vector associated with each sub-block may be derived based on motion information of neighboring blocks. Such sub-block motion compensation may be applied only for a single inter-prediction mode, and temporal motion information for the current block may not be used to derive the sub-block motion vector. Such implementations effectively capture sub-block-level warping, as sub-blocks of the current block may be associated with different motion vectors.

[0127] In some further enhanced exemplary implementations, the derivation of the sub-block motion vector may be based not only on the motion vectors of neighboring blocks, but also on the temporal motion vector predictor of the current block and / or on motion vectors from reference pictures, as described in more detail below.

[0128] More specifically, for sub-block motion compensation, which may capture sub-block-level warping, an inter-predicted video block may be divided into sub-blocks, with each sub-block associated with a motion vector. The motion vector for each of the sub-blocks with respect to a particular reference frame may be determined / derived according to motion vectors associated with one or more of the inter-predicted video block's spatial neighboring blocks in the current frame, motion vectors from a reference picture of the current frame, or a temporal motion vector predictor for the current block.

[0129] In some example implementations, the decision that motion vectors of various sub-blocks within a current block are inter-predicted may be performed in two stages. In the first stage, motion vectors (referred to as inter-motion vectors, for reasons that will become clear below) may be attributed to a set of virtual sub-blocks surrounding the current block. Each of these virtual sub-blocks is associated with a neighboring sub-block position relative to the current block. The virtual blocks may alternatively be referred to as placeholder neighboring sub-blocks of the current block (as placeholders for the attributed position-sensitive inter-motion vectors). In the second stage, the position-sensitive inter-motion vectors and one or more of their virtual sub-block positions relative to the current block are used to determine / calculate / derive motion vectors for each of the sub-blocks within the current block.

[0130] An implementation of the above example virtual or placeholder sub-blocks is shown in FIG. 14. As shown in FIG. 14, a current video block 1410 in a current frame, represented by a thick border, may be predicted under sub-block motion compensation. Accordingly, the current video block 1410 may be divided into a plurality of sub-blocks having a particular predefined or adaptively determined sub-block size. In the example of FIG. 14, the current block 1410 may be divided into 36 equal sub-blocks, as defined by the dashed lines shown within the current block 1410. Correspondingly, virtual or placeholder neighboring sub-blocks may be established surrounding the current block 1410, as shown by the textured region area in FIG. 14. As described in more detail below, each or some of these virtual sub-blocks may be attributed with a motion vector. These attributed motion vectors may be selectively used to derive sub-block-level motion vectors for the sub-blocks within the current block 1410.

[0131] The virtual sub-blocks surrounding the current block 1410 may be specified by their neighboring positions relative to the current block 1410. Some of the virtual sub-blocks 1410 may be associated with positions corresponding to spatially neighboring (already reconstructed) sub-blocks of the current block 1410 in the current frame, such as the virtual sub-blocks located above and to the left of the current block 1410, as indicated by the diagonal line texture. If the neighboring sub-blocks of the current block 1410 corresponding to these virtual sub-blocks or placeholder sub-blocks in place are coded in inter-prediction mode, their motion vectors will be identified and known during their reconstruction. However, other virtual sub-blocks, such as the sub-blocks below the bottom and to the right of the current block 1410, as indicated by the virtual sub-blocks with the vertical line texture in FIG. 14 and shown as 1420 and 1440, respectively, may positionally correspond to the (unreconstructed) neighboring sub-blocks of the current block 1410 in the current frame. Therefore, the neighboring sub-blocks of the current block 1410 corresponding to the in-place virtual sub-blocks 1420 and 1440 have not been reconstructed when the current block 1410 is being processed. The motion vectors of these neighboring sub-blocks are not yet known if they are inter-predicted.

[0132] Each of the already reconstructed neighboring blocks or sub-blocks of the current block 1410 in the current frame, whose positions correspond to the virtual sub-blocks of Figure 14 having a diagonal texture, may be either inter-predicted or intra-predicted, and therefore may or may not be associated with a known motion vector. In addition, each of these neighboring blocks or sub-blocks of the current block 1410 may be associated with a reference block or sub-block at the same location in a reference frame, or may be associated with a temporal block or sub-block in a temporal reference frame (e.g., a temporally predicted frame interpolated from a reference frame), which may or may not be associated with any motion vector (depending on whether these reference sub-blocks or temporal sub-blocks are inter-predicted or intra-predicted).

[0133] Each of the not-yet-reconstructed neighboring blocks or sub-blocks of the current block 1410 in the current frame, whose positions correspond to the virtual or placeholder sub-blocks 1420 and 1440 of Figure 14 having vertical texture, is not associated with an already reconstructed motion vector of the current frame, but may nevertheless be associated with an already reconstructed co-located block or sub-block in a reference frame or in a temporal reference frame (e.g., a temporally predicted frame interpolated from the reference frame), which may or may not be associated with an already reconstructed motion vector (depending on whether these temporal sub-blocks are inter-predicted or intra-predicted).

[0134] 14 may correspond in position to inter-predicted spatial neighboring blocks or sub-blocks of the current block 1410, or to co-located temporal blocks or sub-blocks that may belong to prediction units larger than the size of the sub-block. In this way, two or more adjacent virtual sub-blocks may correspond to the same neighboring or temporal prediction unit, and therefore the same reconstructed motion vector.

[0135] As generally described above, a first stage may attribute motion vectors to virtual sub-blocks at different locations surrounding the current block 1410 of FIG. 14 , and then these attributed motion vectors may be used to determine / calculate / derive motion vectors for sub-blocks within the current block 1410. In some example implementations, the x-component (horizontal component) of the sub-block motion vector in the current block 1410 may be calculated or derived based on the x-components (horizontal components) of the motion vectors of neighboring blocks and / or motion vectors from the current block's temporal motion vector predictor and / or reference picture. Specifically, the x-component (horizontal component) of the sub-block motion vector in the current block 1410 may be calculated or derived based on the x-components (horizontal components) of one or more of the motion vectors attributed to the virtual sub-blocks. Similarly, the y-component of the sub-block motion vector of the current block 1410 may be calculated or derived based on the motion vectors of neighboring blocks and / or motion vectors from the current block's temporal motion vector predictor and / or reference picture. Specifically, the y-component (horizontal component) of the sub-block motion vector of the current block 1410 may be calculated or derived based on the y-components (vertical components) of one or more of the motion vectors attributed to the virtual sub-block.

[0136] Each of these virtual sub-blocks may be assigned or attributed to an already reconstructed motion vector, if available, from one of the following: a correspondingly located spatial neighboring block / sub-block of the current block 1410 in the current frame; a co-located block / sub-block from a reference frame; a co-located block / sub-block from a temporal reference frame, for example, interpolated from the reference frame of the current frame; or a temporal motion vector predictor for the current block. The decision of which motion vector to select from these candidate motion vectors (if available) for attribution to the virtual sub-block position may follow a predefined availability check order of these potential candidate motion vector options. The selection from available candidate motion vectors for a virtual sub-block may be referred to as a motion vector collection process for the virtual sub-block. In other words, motion vectors are collected from the candidate motion vectors and selectively attributed to these virtual sub-blocks. If no collectable motion vector is available for a virtual sub-block, the motion vector for such virtual sub-block may be padded or calculated according to one of the exemplary methods described in more detail below. The motion vectors for the virtual sub-blocks, collected or otherwise padded or calculated, along with their positions, can then be used in a second stage to derive sub-block motion vectors for the sub-blocks of the current block 1410.

[0137] In some example implementations, when collecting / selecting motion vectors to attribute to virtual sub-blocks from candidate motion vectors, the candidate motion vectors (current neighboring or temporal motion vectors already reconstructed) may be fetched directly according to a predefined availability check order.

[0138] In some example implementations described above, a motion vector of a virtual sub-block may be padded / derived from one of its neighboring virtual sub-blocks if its own candidate motion vector for collection or attribution is not otherwise available from a spatially co-located block / sub-block in the current frame or temporal block / sub-block. For example, in FIG. 14, if a candidate motion vector for virtual sub-block 1432 is not available from the corresponding co-located block / sub-block or any temporal block / sub-block in the current frame (e.g., these blocks / sub-blocks are intra-predicted rather than inter-predicted), the candidate motion vector of neighboring virtual sub-block 1434 or 1436 (adjacent to virtual sub-block 1432 in the set of virtual sub-blocks) may be used to pad or derive the motion vector attributed to virtual sub-block 1432.

[0139] In some example implementations, if a motion vector for a block / sub-block in a current frame corresponding to a virtual sub-block at a given location is not available, the temporal motion vectors of the co-located sub-blocks in the reference frame may be first checked as candidate motion vectors for attributing to the virtual sub-block. If such co-located temporal motion vectors are available, they are collected in a predetermined availability check order and used to attribute to the virtual sub-block. Otherwise, the motion vectors attributed to its neighboring virtual sub-blocks may be used for padding or deriving the motion vector.

[0140] In some example implementations, if a motion vector for a block / sub-block in the current frame corresponding to a virtual sub-block at a predetermined location is not available, motion vectors for second-tier spatial neighboring blocks / sub-blocks of the current block 1410 may be searched for candidate motion vectors to attribute to the virtual sub-block. For example, the second-tier neighboring block / sub-block may refer to a block / sub-block immediately outside the virtual sub-block at a predetermined location in the current frame. If motion vectors for such second-tier neighbors in the current frame are also not available, the motion vector to be attributed to the virtual neighboring sub-block may be padded or derived as described above. Figure 16 illustrates the locations of second-tier neighboring sub-blocks relative to the current block and the virtual sub-block as an outer sub-block surrounding the virtual sub-block.

[0141] In some example implementations, the order of checking the availability of candidate motion vectors may follow spatially neighboring sub-blocks in the current frame, spatially neighboring sub-blocks of the next hierarchy in the current frame, and temporal sub-blocks in one or more reference frames.

[0142] In some example implementations, the order of checking the availability of candidate motion vectors may follow spatially neighboring sub-blocks in the current frame, temporal sub-blocks in one or more reference frames, and spatially neighboring sub-blocks of the next hierarchy in the current frame, or any other combination and order.

[0143] In some other example implementations, if a motion vector for a block / sub-block in the current frame corresponding to a virtual sub-block at a given location is not available, the motion vector attributed to the virtual sub-block may be set to a default value, such as a zero motion vector value.

[0144] In some example implementations, for a bottom-right virtual sub-block 1460 (shown as the intersection of the bottom sub-block 1420 and the right sub-block 1440 in FIG. 14 ) that is not associated with a co-located sub-block of the current frame that already has a reconstructed motion vector, the temporal motion vector of the sub-block co-located with this bottom-right virtual sub-block 1460 in the reference frame may be directly fetched and attributed to this virtual sub-block if available. Otherwise, if a co-located temporal motion vector is not available, a weighted average of (1) the attributed motion vectors of the bottom-left and top-right virtual sub-blocks, or (2) the temporal motion vectors of the sub-blocks in the reference frame that correspond in spatial position to the bottom-left and top-right virtual sub-blocks, may be used as the motion vector predictor for attributing the bottom-right virtual sub-block. In other implementations, if a co-located temporal motion vector is not available, the temporal motion vector predictor of the bottom-right virtual sub-block may be considered non-existent and may not be used to attribute the motion vector of the bottom-right virtual sub-block.

[0145] In some example implementations, only the motion vectors of the virtual sub-blocks at the upper-left, upper-right, lower-left, and lower-right corners of FIG. 14 are collected from spatial / temporal neighboring sub-blocks as described above, and the motion vectors attributed to other virtual sub-blocks of FIG. 14 between these corner virtual sub-blocks may be interpolated using, for example, bilinear interpolation or other predefined nonlinear weighting interpolation of the motion vectors attributed to the corner virtual sub-blocks. For example, the virtual sub-block between the upper-left corner virtual sub-block and the lower-left corner virtual sub-block may be attributed by interpolation or nonlinear weighting of the motion vectors attributed to the upper-left corner virtual sub-block and the lower-left corner virtual sub-block. Similarly, the virtual sub-block between the upper-left corner virtual sub-block and the upper-right corner virtual sub-block may be attributed by interpolation or nonlinear weighting of the motion vectors attributed to the upper-left corner virtual sub-block and the upper-right corner virtual sub-block. Similarly, a virtual sub-block between the top right corner virtual sub-block and the bottom right corner virtual sub-block may be attributed by interpolation or non-linear weighting of the motion vectors attributed to the top right corner virtual sub-block and the bottom right corner virtual sub-block. Similarly, a virtual sub-block between the bottom right corner virtual sub-block and the bottom left corner virtual sub-block may be attributed by interpolation or non-linear weighting of the motion vectors attributed to the bottom right corner virtual sub-block and the bottom left corner virtual sub-block.

[0146] In some example implementations, the virtual sub-blocks surrounding the current block may be segmented by virtual sub-blocks that have collected candidate motion vectors available, as described above, and the attributed motion vectors of virtual sub-block segments between virtual sub-block segments that do not have collectable candidate motion vectors may be derived by linear or non-linear interpolation of the collected motion vectors at both ends of each of these virtual sub-block segments.

[0147] 15, moving to the second stage for determining a motion vector for a sub-block of the current block, e.g., sub-block 1510, based on the attributed motion vectors in surrounding virtual sub-blocks, in some example implementations, the motion vector for sub-block 1510 may be calculated as a weighted average of one or more attributed motion vectors selected from the attributed motion vectors of the above virtual sub-block 1520, left virtual sub-block 1530, right virtual sub-block 1540, and bottom virtual sub-block 1550. Which of the virtual sub-blocks to select from a set of virtual sub-block positions (e.g., a set of top, left, right, and bottom) for such calculation may be predetermined or may be dynamically signaled at various levels (e.g., block, picture, frame, etc.).

[0148] In the example implementation described above with respect to FIG. 15, a bilinear equation may be used to generate the motion vector for each sub-block of the current block.

[0149] In some example implementations, multiple sets of weighting factors may be predefined for calculating motion vectors for sub-blocks of the current block. For example, the selection of a set of weighting factors from the predefined sets of weighting factors for calculating the motion vector of a particular sub-block may depend on the relative position of the sub-block within the current block.

[0150] In another example, the selection of a set of weighting factors from a set of predefined weighting factors for the calculation of a motion vector for a particular sub-block may depend on the relative position of the sub-block within the current block and / or the difference between neighboring motion vectors, e.g., the difference between the selected attributed motion vector of the virtual sub-block to be used and the attributed motion vectors of those neighboring virtual sub-blocks.

[0151] For example, if the attributed motion vectors selected to calculate the motion vector of a sub-block are close (small difference, e.g., less than a threshold difference), these motion vectors may be averaged to obtain the motion vector of the sub-block regardless of the position of the sub-block. Otherwise, the distance between the sub-block and the selected virtual sub-block (the length of the arrow in FIG. 15) may be used as a weight for calculating the motion vector of the sub-block based on the selected attributed motion vector of the selected virtual sub-block (e.g., virtual sub-blocks 1520, 1530, 1540, 1550 in FIG. 15).

[0152] In some other example implementations, the motion vectors attributed to the virtual sub-blocks may be further adjusted with a motion vector difference (MVD) before being used to calculate the motion vectors of the sub-blocks in the current block. Such motion vector differences may be determined and generated by the encoder and signaled in the bitstream. The decoder may then correspondingly use the signaled MVD to adjust the attributed motion vectors (collected or derived in the decoder according to the various example implementations above) with the signaled MVD and use the adjusted attributed motion vectors of the virtual sub-blocks for calculating the motion vectors of the sub-blocks of the current block.

[0153] In some example implementations, for a block containing multiple sub-blocks, zero or one MVD may be signaled. If one MVD is signaled, it may be applied to adjusting the attributed motion vectors of the virtual sub-blocks at a predetermined location. For example, the predetermined location may be the bottom right. In some alternative implementations, the location at which the signaled MVD applies may be additionally signaled among a set of locations (e.g., bottom right, bottom left, top left, and top right, or any combination thereof). For example, the set of locations may be predefined and indexed, and the signaling of the location may be achieved via signaling one of the indexes.

[0154] In some particular example implementations, two MVDs may be signaled, where a first MVD may be applied to the attributed motion vector of the bottom-left virtual sub-block and a second MVD may be applied to the attributed motion vector of the top-right virtual sub-block.

[0155] In some other example implementations, a fixed or variable number of MVDs up to a predetermined number may be signaled for a block and applied to adjust the attributed motion vectors of virtual sub-blocks at a set of predefined ordered positions, e.g., first, second, ... The signaled MVDs may be mapped to an ordered list of virtual sub-block positions. Alternatively, the virtual sub-block positions of multiple signaled MVDs may be signaled additionally rather than predefined.

[0156] Adjustments to the attributed motion vectors of some selected virtual sub-block positions via the signaled MVD can help provide more accurate sub-block motion vectors for the current block, potentially leading to improved coding gain. The exemplary positions are selected above for MVD signaling, taking into consideration that these positions may be used for interpolation of attributed motion vectors of other virtual sub-blocks. Thus, such interpolation may use the adjusted attributed motion vectors rather than the unadjusted attributed motion vectors at these virtual sub-block positions.

[0157] In some example implementations, the signaling of MVD for the purpose of adjusting the attributed motion vectors of the above virtual sub-blocks, including, but not limited to, the associated syntax and context, may be the same as the signaling for the NEWMV inter prediction mode.

[0158] In some example implementations, the signaling of MVD for the purpose of adjusting the attributed motion vectors of the above virtual sub-blocks, including, but not limited to, the associated syntax and context, may be the same as the signaling for the AMVDNEWMV mode in which adaptive MVD resolution is applied.

[0159] In some example implementations, the signaling of MVDs for the purpose of adjusting the attributed motion vectors of the above virtual sub-blocks, including but not limited to, associated syntax and context, may be based on the inter-prediction mode of the current block. In other words, the signaling syntax or context for these MVDs may vary from block to block depending on the inter-prediction mode of the block.

[0160] In some example implementations of sub-block motion compensation, the MVP / DRL index of the current block may not be signaled to the bitstream and may be derived as zero at both the encoder and decoder sides. In other words, under the above sub-block motion compensation, the motion vector derivation of a sub-block depends on the motion vectors and temporal vectors of neighboring blocks, and therefore may not need to refer to a specific signaled motion vector predictor or DRL index.

[0161] In some exemplary implementations, the above smooth sub-block motion vector or compensation modes may be signaled as one of the inter prediction modes, in parallel with inter prediction modes such as NEARMV, NEWMV, GLOBALMV, NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV and GLOBAL_GLOBALMV modes.

[0162] In some example implementations, the above smooth sub-block motion vectors or compensation modes may be signaled as one of multiple motion modes, such as overlapped block motion compensation (OBMC) or warped motion modes, all under various inter-prediction modes.

[0163] In some example implementations, one high-level syntax may be signaled at the sequence level / frame level / slice level / superblock level to indicate whether smooth sub-block motion compensation is enabled or not.

[0164] In some example implementations, the above smooth sub-block motion vectors or compensation modes may be signaled only for skip mode. Example decoding code for parsing information related to skip mode and smooth sub-block motion modes is shown below. [Table 6]

[0165] In the above reference syntax, the bitstream is parsed to determine whether skip mode is enabled for the block. If it is enabled, a flag "subblock" is read from the bitstream, which indicates whether the block is predicted under subblock mode (e.g., smooth subblock mode).

[0166] Figure 17 shows an example logic flow 1700 according to the above implementation. Logic flow 1700 starts at S1701. At S1710, a syntax element signaled in a video bitstream is received, the syntax element indicating whether a video block is to be inter-predicted in sub-block motion compensation mode. At S1720, it is determined whether the video block is to be inter-predicted in sub-block motion compensation mode based on the value of the received syntax element. At S1730, a set of reference motion vectors corresponding to a set of neighboring blocks or sub-block positions of the video block is determined based on available candidate motion vectors including motion information of one or more neighboring blocks of the video block and temporal blocks of the video block in a current frame. At S1740, a sub-block motion vector for each of a plurality of sub-blocks is derived based on the reference motion vectors. Logic flow 1700 stops at S1799.

[0167] The above operations may be combined or arranged in any quantity or order as desired. Two or more of the steps and / or operations may be performed in parallel. The embodiments and implementations of the present disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may apply to luma blocks or chroma blocks. The term block may be interpreted as a prediction block, a coding block, or a coding unit, i.e., a CU. The term block in this specification may also be used to refer to a transform block. In the following sections, references to block size may refer to the width or height of the block, the maximum value of the width and height, the minimum value of the width and height, or the area size (width*height) or aspect ratio (width:height, height:width) of the block.

[0168] The techniques described above can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 18 illustrates a computer system (1800) suitable for implementing certain embodiments of the disclosed subject matter.

[0169] Computer software can be coded using any suitable machine code or computer language that may follow assembly, compilation, linking, or similar mechanisms to create code containing instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., either directly, or via translation, microcode execution, etc.

[0170] The instructions may be executed in various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0171] 18 for computer system 1800 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system 1800.

[0172] The computer system (1800) may include certain human interface input devices, which may include one or more (only one of each is shown) of a keyboard (1801), a mouse (1802), a trackpad (1803), a touchscreen (1810), a data glove (not shown), a joystick (1805), a microphone (1806), a scanner (1807), and a camera (1808).

[0173] The computer system (1800) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1810), data gloves (not shown), or joystick (1805), although haptic feedback devices that do not function as input devices may also be present), audio output devices (e.g., speakers (1809), headphones (not shown), etc.), visual output devices (e.g., screens (1810), including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capability and each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or greater than three-dimensional output via means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0174] The computer system (1800) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW with CD / DVD or similar media (1821), thumb drives (1822), removable hard drives or solid state drives (1823), legacy magnetic media such as tape and floppy disks (not shown), and special ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0175] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0176] The computer system 1800 may also include interfaces 1854 to one or more communications networks 1855. The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, metropolitan, vehicular and industrial, real-time, delay-tolerant, etc. Examples of networks include Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial networks including CAN bus, etc.

[0177] The human interface devices, human-accessible storage devices, and network interfaces described above can be attached to the core (1840) of the computer system (1800).

[0178] The core (1840) may include one or more central processing units (CPUs) (1841), graphics processing units (GPUs) (1842), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (1843), hardware accelerators for specific tasks (1844), graphics adapters (1850), etc. These devices may be connected via a system bus (1848), along with read-only memory (ROM) (1845), random access memory (1846), and internal mass storage devices (1847) such as internal non-user-accessible hard drives, SSDs, etc. In some computer systems, the system bus (1848) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1848) or via a peripheral bus (1849). In one example, a screen (1810) may be connected to the graphics adapter (1850). Peripheral bus architectures include PCI, USB, and the like.

[0179] The computer-readable medium may bear computer code for performing various computer-implemented operations, and the media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0180] While this disclosure describes several exemplary embodiments, alterations, permutations, and various substitute equivalents exist, and these are within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

Claims

1. 1. A method for decoding a video block comprising a plurality of sub-blocks of a current frame in a video bitstream, the method being performed by an electronic device and comprising: receiving a syntax element signaled within the video bitstream that indicates whether the video block is inter predicted under a sub-block motion compensation mode; determining, based on a value of the received syntax element, that the video block is inter predicted under a sub-block motion compensation mode; determining a set of reference motion vectors corresponding to a set of neighboring block or sub-block positions of the video block based on available candidate motion vectors including motion information of one or more neighboring blocks of the video block and temporal blocks of the video block within the current frame; deriving a sub-block motion vector for each of the plurality of sub-blocks based on the reference motion vector; A method comprising:

2. The step of deriving a sub-block motion vector for each of the plurality of sub-blocks based on the reference motion vector includes: deriving a horizontal component of the sub-block motion vector based on a horizontal component of the reference motion vector; deriving a vertical component of the sub-block motion vector based on a vertical component of the reference motion vector; The method of claim 1 , comprising:

3. For each of at least one of the set of neighboring block or sub-block positions, the reference motion vector is selected, in a predetermined availability check order, from: a first motion vector determined from a neighboring block of the video block in the current frame, the neighboring block being spatially co-located with the neighboring block or sub-block position; a second motion vector derived based on a temporal block spatially co-located with the neighboring block or sub-block position; The method of claim 1 , wherein the candidate motion vectors are determined by selecting from the available candidate motion vectors including:

4. For each of at least one of the set of neighboring block or sub-block positions, the reference motion vector is selected, in a predetermined availability check order, from: a first motion vector determined from a neighboring block of the video block in the current frame, the neighboring block being spatially co-located with the neighboring block or sub-block position; a second motion vector determined from a second neighboring block of the video block in the current frame adjacent the neighboring block or sub-block position; The method of claim 1 , wherein the candidate motion vectors are determined by selecting from the available candidate motion vectors including:

5. For a neighboring block or sub-block position for which no candidate motion vector is available, the corresponding reference motion vector is padded from reference motion vectors associated with its adjacent neighboring block or sub-block position.

5. The method according to any one of claims 1 to 4.

6. The reference motion vector associated with the lower right neighboring block or sub-block position is If available, the temporal motion vector is fetched from the co-located location. otherwise, derived as a weighted average of the reference motion vectors associated with the upper right neighboring block or sub-block position and the lower left neighboring block or sub-block position; 5. The method according to any one of claims 1 to 4.

7. The reference motion vector associated with the lower right neighboring block or sub-block position is If available, the temporal motion vector is fetched from the co-located location. otherwise, it is not used to derive the sub-block motion vector for each of the plurality of sub-blocks.

5. The method according to any one of claims 1 to 4.

8. the set of neighboring block or sub-block locations includes corner locations including a top left corner location, a top right corner location, and a bottom right corner location; a reference motion vector associated with the corner position is selected from the available candidate motion vectors; other reference motion vectors associated with neighboring blocks or sub-block positions other than the corner positions are interpolated from the reference motion vectors associated with the corner positions; 5. The method according to any one of claims 1 to 4.

9. deriving the sub-block motion vector for a sub-block of the video block based on the reference motion vector, selecting, from the set of reference motion vectors, a subset of reference motion vectors for neighboring blocks or sub-block positions to the left, above, right and below the sub-block; deriving the sub-block motion vector for the sub-block based on the subset of reference motion vectors; 5. The method of claim 1, comprising:

10. the sub-block motion vector for the sub-block is derived from the subset of reference motion vectors using a bilinear equation; 10. The method of claim 9.

11. the sub-block motion vector for the sub-block is derived from a weighted average of a subset of the reference motion vectors.

10. The method of claim 9.

12. a set of weighting factors for deriving the sub-block motion vector from the subset of reference motion vectors is selected from among a plurality of sets of predetermined or signaled weighting factors; The method of claim 11.

13. the set of weighting factors is selected based on the position of the sub-block or the difference between the subset of reference motion vectors. The method of claim 12.

14. parsing the video bitstream to extract one or more motion vector deltas; applying the motion vector delta to adjust at least one of the set of reference motion vectors, and then deriving a sub-block motion vector therefrom; The method of claim 1 , further comprising:

15. the one or more motion vector deltas are attributed to and applied to neighboring block or sub-block positions via a predefined order; 15. The method of claim 14.

16. and parsing the video bitstream to extract signaling identifying neighboring block or sub-block positions corresponding to at least one of the set of reference motion vectors to which the one or more motion vector deltas are applied.

15. The method of claim 14.

17. the one or more motion vector deltas are signaled within the video bitstream using the same syntax and context as motion vector difference (MVD) signaling or adaptive resolution MVD signaling.

15. The method of claim 14.

18. a motion vector predictor or a dynamic reference list index for the video block is not signaled within the video bitstream.

5. The method according to any one of claims 1 to 4.

19. the sub-block motion compensation mode is signaled as one of a plurality of motion modes, the plurality of motion modes further comprising at least one of a warped motion mode and an overlapped block motion compensation (OBMC) mode.

5. The method according to any one of claims 1 to 4.

20. the sub-block motion compensation mode is signaled only when skip mode is enabled for the video block.

5. The method according to any one of claims 1 to 4.

21. the sub-block motion compensation mode is signaled at one of the sequence, frame, slice, or superblock level; 5. The method according to any one of claims 1 to 4.

22. the sub-block motion compensation mode includes a smooth sub-block motion compensation mode; 5. The method according to any one of claims 1 to 4.

23. a memory for storing instructions; and executing the stored instructions Analyzing the video bitstream to determine that video blocks in a current frame are inter-predicted under a sub-block motion compensation mode; determining a plurality of sub-blocks of the video block; determining, based on available candidate motion vectors including motion information of one or more neighboring blocks of the video block and temporal blocks of the video block within the current frame, a set of reference motion vectors corresponding to a set of neighboring block or sub-block positions of the video block; deriving a sub-block motion vector for each of the sub-blocks based on the reference motion vector; An electronic device comprising: a processor.

24. A computer program which, when executed by an electronic device, causes the electronic device to carry out the method of any one of claims 1 to 4.

Images