Smooth subblock motion vector prediction

Abstract

The present disclosure relates generally to video coding, and in particular to smooth sub-block motion compensation, and discloses methods and systems for determining sub-block motion vectors for a video block based on motion information for spatially neighboring blocks of the video block, temporal blocks spatially co-located with neighboring blocks, or co-located temporal blocks of the video block.

Description

Cross-references

[0001] This PCT international patent application is based on and claims priority to U.S. non-provisional patent application No. 18 / 455,309, filed on August 24, 2023, entitled “Smooth Sub-Block Motion Vector Prediction,” which is based on and claims priority to U.S. provisional patent application No. 63 / 443,016, filed on February 2, 2023, entitled “Smooth Sub-Block Motion Vector Prediction,” which is incorporated by reference in its entirety into this application. Technical Field

[0002] The present disclosure relates generally to video coding and, in particular, to methods and systems for sub-block motion compensation. Background Art

[0003] Uncompressed digital video may include a series of pictures and may be a specific bit rate requirement for storage, data processing, and transmission bandwidth for streaming applications. One purpose of video encoding and decoding may be to reduce redundancy in the uncompressed input video signal through various compression techniques. Summary of the invention

[0004] The present disclosure relates generally to video coding and decoding, and in particular to smooth sub-block motion compensation. A method and system for determining a sub-block motion vector of a video block based on motion information of spatial neighboring blocks of the video block, a temporal block spatially co-located with the neighboring blocks, or a co-located temporal block of the video block is disclosed.

[0005] In an example embodiment, a method for decoding a video block including multiple sub-blocks of a current frame in a video bitstream is disclosed. The method may include receiving a syntax element signaled in the video bitstream, the syntax element indicating whether the video block is inter-frame predicted in a sub-block motion compensation mode; determining, based on the value of the received syntax element, that the video block is to be inter-frame predicted in the sub-block motion compensation mode; determining, based on available candidate motion vectors, a reference motion vector set corresponding to a set of neighboring blocks or sub-block positions of the video block, the available candidate motion vectors including motion information of one or more neighboring blocks of the video block in the current frame and a temporal block of the video block; and deriving a sub-block motion vector for each of the multiple sub-blocks based on the reference motion vector.

[0006] Aspects of the present disclosure also provide an electronic device or apparatus, which includes a circuit or processor configured to execute any one of the above method embodiments.

[0007] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions, which, when executed by an electronic device, cause the electronic device to perform any one of the above method embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which:

[0009] Figure 1 A schematic diagram showing a simplified block diagram of a communication system (100) according to an example embodiment of the present disclosure;

[0010] Figure 2 A schematic diagram showing a simplified block diagram of a communication system (200) according to an example embodiment of the present disclosure;

[0011] Figure 3 A schematic diagram showing a simplified block diagram of a video decoder according to an example embodiment of the present disclosure;

[0012] Figure 4 A schematic diagram showing a simplified block diagram of a video encoder according to an example embodiment of the present disclosure;

[0013] Figure 5 A schematic diagram showing a simplified block diagram of a video encoder according to another example embodiment of the present disclosure;

[0014] Figure 6 A schematic diagram showing a simplified block diagram of a video encoder according to another example embodiment of the present disclosure;

[0015] Figure 7 A scheme for encoding block partitioning according to an exemplary embodiment of the present disclosure is shown;

[0016] Figure 8 Another scheme of coding block partitioning according to an exemplary embodiment of the present disclosure is shown;

[0017] Fig. 9 Another scheme of coding block partitioning according to an exemplary embodiment of the present disclosure is shown;

[0018] Fig.10 Compound motion compensation is illustrated.

[0019] Fig.11 An example interpolated reference frame for motion compensation is illustrated.

[0020] Fig.12 An example of generating a motion vector of a sub-block using neighboring motion vectors is shown.

[0021] Fig.13Pixel-level motion vector constraints for example sub-block motion compensation are illustrated.

[0022] Fig.14 Example neighboring virtual sub-blocks of motion vector attributes for sub-block motion compensation are illustrated.

[0023] Fig.15 An example sub-block motion vector generation based on motion vectors attributed to neighboring virtual sub-blocks is illustrated.

[0024] Fig.16 The next neighboring sub-block used for sub-block motion compensation is illustrated.

[0025] Fig.17 An example logic flow of a method for sub-block motion compensation is shown.

[0026] Fig.18 A schematic diagram of a computer system according to an example embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] Throughout the specification and claims, terms may have subtle meanings that are suggested or implied by the context beyond the explicitly stated meaning. The phrases "in one embodiment / implementation" or "in some embodiments / implementations" as used herein do not necessarily refer to the same embodiment / implementation, and the phrases "in another embodiment / implementation" or "in other embodiments / implementations" as used herein do not necessarily refer to different embodiments. For example, the claimed subject matter is intended to include all or part of the combination of the exemplary embodiments / implementations.

[0028] In general, terms may be understood, at least in part, based on their use in context. For example, terms such as "and", "or", "and / or", as used herein, may include a variety of context-dependent meanings. Typically, "or", if used in an associative list, such as A, B, or C, is intended to mean A, B, and C, used herein in an inclusive sense, and A, B, or C, used herein in an exclusive sense. Additionally, the terms "one or more", "at least one", "one", "an", or "the", as used herein, may be used in a singular sense or in a plural sense, depending, at least in part, on the context. Additionally, the terms "based on" or "determined by" may be understood as not necessarily intended to express an exclusive set of factors, but may allow for the presence of additional factors that are not necessarily explicitly described again, depending, at least in part, on the context.

[0029] Figure 1A simplified block diagram of a communication system (100) according to an embodiment of the present disclosure is illustrated. The communication system (100) includes a plurality of terminal devices, such as 110, 120, 130, and 140, which can communicate with each other via, for example, a network (150). Figure 1 In the example of , a first pair of terminal devices (110) and (120) can perform unidirectional transmission of data. For example, the terminal device (110) can encode video data in the form of one or more encoded code streams (e.g., video picture streams captured by the terminal device (110)) for transmission via the network (150). The terminal device (120) can receive the encoded video data from the network (150), decode the encoded video data to restore the video picture, and display the video picture based on the restored video data. The unidirectional data transmission can be implemented in a media service application, etc.

[0030] In another example, the second pair of terminal devices (130) and (140) can perform bidirectional transmission of encoded video data, for example, during a video conferencing application. For bidirectional transmission of data, in the example, each of the terminal devices (130) and (140) can encode video data (e.g., video data of a video picture stream captured by the terminal device) for transmission to the other of the terminal devices (130) and (140), and can also receive encoded video data from the other of the terminal devices (130) and (140) to restore and display the video picture.

[0031] exist Figure 1 In the example of, the terminal device can be implemented as a server, a personal computer and a smart phone, but the applicability of the basic principles of the present disclosure may not be limited to this. The embodiments of the present disclosure can be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, special video conferencing equipment, etc. The network (150) represents any number or type of network that transmits encoded video data between terminal devices, including, for example, wired (wired) and / or wireless communication networks. The communication network (150) can exchange data in circuit switching, packet switching and / or other types of channels. Representative networks include telecommunications networks, local area networks, wide area networks and / or the Internet.

[0032] Figure 2 The arrangement of a video encoder and a video decoder in a video streaming environment as an example of an application of the disclosed subject matter is illustrated. The disclosed subject matter can be equally applied to other video applications, including, for example, video conferencing, digital television broadcasting, gaming, virtual reality, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0033] like Figure 2As shown, a video streaming system may include a video capture subsystem (213) that may include a video source (201), such as a digital camera, for creating an uncompressed video picture or image stream (202). In an example, the video picture stream (202) includes samples recorded by the digital camera of the video source 201. The video picture stream (202), which is depicted as a thick line to emphasize the high amount of data when compared to the encoded video data (204) (or encoded video bitstream), may be processed by an electronic device (220) including a video encoder (203) coupled to the video source (201). The video encoder (203) may include hardware, software, or a combination thereof to implement or embody aspects of the disclosed subject matter described in more detail below. The encoded video data (204) (or encoded video bitstream (204)), depicted as thin lines to emphasize the lower amount of data when compared to the stream of uncompressed video pictures (202), can be stored on a streaming server (205) for future use or directly to a downstream video device (not shown). Figure 2 One or more streaming client subsystems (206) and (208) in the video processing apparatus can access the streaming server (205) to retrieve copies (207) and (209) of the encoded video data (204). The client subsystem (206) can include, for example, a video decoder (210) in an electronic device (230). The video decoder (210) decodes the input copy (207) of the encoded video data and creates an output stream of video pictures (211) that are uncompressed and can be rendered on a display (212) (e.g., a display screen) or other rendering device (not depicted).

[0034] Figure 3 A block diagram of a video decoder (310) of an electronic device (330) according to any embodiment of the present disclosure below is shown. The electronic device (330) may include a receiver (331) (e.g., a receiving circuit). The video decoder (310) may be used to replace Figure 2 A video decoder (210) in an example of FIG.

[0035] like Figure 3As shown, a receiver (331) can receive one or more encoded video sequences from a channel (301). In order to combat network jitter and / or handle playback timing, a buffer memory (315) can be set between the receiver (331) and the entropy decoder / parser (320) (hereinafter referred to as "parser (320)"). The parser (320) can reconstruct symbols (321) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (310), and potentially information for controlling a rendering device such as a display (312) (e.g., a display screen). The parser (320) can parse / entropy decode the encoded video sequence. The parser (320) can extract a subgroup parameter set for at least one pixel subgroup in the video decoder from the encoded video sequence. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (320) may also extract information such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc. from the encoded video sequence. The reconstruction of the symbols (321) may involve a number of different processing or functional units. The units involved and how they are involved may be controlled by subgroup control information parsed by the parser (320) from the encoded video sequence.

[0036] The first unit may include a scalar / inverse transform unit (351). The scalar / inverse transform unit (351) may receive quantized transform coefficients and control information from the parser (320), including information indicating which type of inverse transform to use, block size, quantization factor / parameter, quantization scaling matrix, etc. as one or more symbols (321). The scalar / inverse transform unit (351) may output a block including sample values ​​that may be input into the aggregator (355).

[0037] In some cases, the output samples of the scalar / inverse transform (351) may belong to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) may use surrounding block information that has already been reconstructed and stored in a current picture buffer (358) to generate a block of the same size and shape as the block in the reconstruction. The current picture buffer (358) may, for example, buffer a partially reconstructed current picture and / or a fully reconstructed current picture. In some embodiments, the aggregator (355) may add the prediction information that has been generated by the intra-prediction unit (352) to the output sample information provided by the scalar / inverse transform unit (351) on a per-sample basis.

[0038] In other cases, the output samples of the scalar / inverse transform unit (351) may belong to an inter-coded and possibly motion-compensated block. In this case, the motion-compensated prediction unit (353) may access the reference picture memory (357) based on the motion vector to obtain samples for inter-prediction. After motion compensation of the obtained samples according to the symbols (321) belonging to the block, these samples may be added to the output of the scalar / inverse transform unit (351) by an aggregator (355) (the output of unit 351 may be referred to as residual samples or residual signal) to generate output sample information.

[0039] The output samples of the aggregator (355) can be subjected to various loop filtering techniques, including several types of loop filters, in the loop filter unit (356). The output of the loop filter unit (356) can be a sample stream that can be output to the rendering device (312) and stored in the reference picture memory (357) for future use in inter-frame prediction.

[0040] Figure 4 A block diagram of a video encoder (403) according to an example embodiment of the present disclosure is shown. The video encoder (403) may be included in an electronic device (420). The electronic device (420) may further include a transmitter (440) (e.g., a transmitting circuit). The video encoder (403) may be used to replace Figure 4 A video encoder (403) in an example.

[0041] The video encoder (403) may receive video samples from a video source (401). According to some example embodiments, the video encoder (403) may encode and compress pictures of a source video sequence into an encoded video sequence (443) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed constitutes a function of the controller (450). In some embodiments, the controller (450) may be functionally coupled to and control other functional units as described below. The parameters set by the controller (450) may include parameters related to rate control (picture skipping, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc.

[0042] In some example embodiments, the video encoder (403) may be configured to operate in a coding loop. The coding loop may include a source encoder (430) and a (local) decoder (433) embedded in the video encoder (403). The decoder (433) reconstructs the symbols to create sample data in a similar manner to that created by the (remote) decoder, even if the embedded decoder 433 processes the video stream encoded by the source encoder 430 without entropy coding (because any compression between the symbols and the encoded video code stream in entropy coding may be lossless in the video compression technology considered in the disclosed subject matter). An observation that can be made at this point is that any decoder technology other than parsing / entropy decoding that may only exist in the decoder may also necessarily need to exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter may sometimes focus on the decoder operation, which is related to the decoding portion of the encoder. Therefore, the description of the encoder technology can be abbreviated as they are the inverse of the decoder technology described comprehensively. A more detailed description of the encoder is provided below only in certain areas or aspects.

[0043] During operation in some example embodiments, the source encoder (430) may perform motion compensated predictive coding, which predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as "reference pictures."

[0044] The local video decoder (433) can decode the encoded video data of the picture that can be designated as the reference picture. The local video decoder (433) replicates the decoding process that can be performed by the video decoder on the reference picture, and can cause the reconstructed reference picture to be stored in the reference picture cache (434). In this way, the video encoder (403) can locally store copies of the reconstructed reference pictures that have the same content (absent transmission errors) as the reconstructed reference pictures that will be obtained by the far-end (remote) video decoder.

[0045] The predictor (435) can perform prediction searches on the encoding engine (432). That is, for a new picture to be encoded, the predictor (435) can search the reference picture memory (434) for sample data (as candidate reference pixel blocks) or certain metadata such as reference picture motion vectors, block shapes, etc. that can be used as appropriate prediction references for the new picture.

[0046] The controller (450) can manage encoding operations of the source encoder (430), including, for example, setting parameters and sub-group parameters for encoding video data.

[0047] The outputs of all of the above functional units may be entropy encoded in an entropy encoder (445). The transmitter (440) may buffer one or more encoded video sequences created by the entropy encoder (445) in preparation for transmission via a communication channel (460), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (440) may combine the encoded video data from the video encoder (403) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0048] The controller (450) may manage the operation of the video encoder (403). During encoding, the controller (450) may assign a particular coded picture type to each coded picture, which may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned one of the following picture types: intra picture (I picture), predicted picture (P picture), bidirectional predicted picture (B picture), multi-predicted picture. A source picture may generally be spatially subdivided into a plurality of sample coding blocks, as described in further detail below.

[0049] Figure 5 A diagram of a video encoder (503) according to another example embodiment of the present disclosure is shown. The video encoder (503) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures, and encode the processed block into an encoded picture that is part of an encoded video sequence. The example video encoder (503) may be used instead of Figure 4 The video encoder (403) in the example.

[0050] For example, the video encoder (503) receives a matrix of sample values ​​for a processing block.The video encoder (503) then determines whether the processing block is best encoded using intra mode, inter mode, or a bi-directional prediction mode such as rate-distortion optimization (RDO).

[0051] exist Figure 5 In the example of , the video encoder (503) includes an inter-frame encoder (530), an intra-frame encoder (522), a residual calculator (523), a switch (526), ​​a residual encoder (524), a general controller (521) and an entropy encoder (525) coupled together, such as Figure 5 The example arrangement is shown in .

[0052] The inter-frame encoder (530) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in previous and later pictures in display order), generate inter-frame prediction information (e.g., a description of redundant information according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a prediction block) based on the inter-frame prediction information using any suitable technique.

[0053] The intra encoder (522) is configured to receive samples of a current block (e.g., a processing block), compare the block to blocks already encoded in the same picture, and generate quantized coefficients after transformation, and in some cases also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques).

[0054] The general controller (521) may be configured to determine general control data and control other components of the video encoder (503) based on the general control data, for example to determine a prediction mode for a block and provide a control signal to a switch (526) based on the prediction mode.

[0055] The residual calculator (523) may be configured to calculate the difference (residual data) between the received block and the prediction result of the block selected from the intra encoder (522) or the inter encoder (530). The residual encoder (524) may be configured to encode the residual data to generate a transform coefficient. The transform coefficient is then quantized to obtain a quantized transform coefficient. In various example embodiments, the video encoder (503) further includes a residual decoder (528). The residual decoder (528) is configured to perform an inverse transform and generate decoded residual data. The entropy encoder (525) may be configured to format the bitstream to include the encoded block and perform entropy encoding.

[0056] Figure 6 A diagram of an example video decoder (610) according to another embodiment of the present disclosure is shown. The video decoder (610) is configured to receive an encoded picture as part of an encoded video sequence and decode the encoded picture to generate a reconstructed picture. In an example, the video decoder (610) may be used instead of Figure 4 A video decoder (410) in an example of FIG.

[0057] exist Figure 6 In the example of , the video decoder (610) includes an entropy decoder (671), an inter-frame decoder (680), a residual decoder (673), a reconstruction module (674) and an intra-frame decoder (672) coupled together, as shown in FIG. Figure 6 An example arrangement is shown in FIG.

[0058] The entropy decoder (671) may be configured to reconstruct certain symbols representing syntax elements constituting the encoded picture from the encoded picture. The inter-frame decoder (680) may be configured to receive inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information. The intra-frame decoder (672) may be configured to receive intra-frame prediction information and generate a prediction result based on the intra-frame prediction information. The residual decoder (673) may be configured to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The reconstruction module (674) may be configured to combine the residual output by the residual decoder (673) and the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module, depending on the specific situation) in the spatial domain to form a reconstructed block, which forms a part of the reconstructed picture as a part of the reconstructed video.

[0059] Note that the video encoders (203), (403) and (503) and the video decoders (210), (310) and (610) may be implemented using any suitable technology. In some example embodiments, the video encoders (203), (403) and (503) and the video decoders (210), (310) and (610) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (203), (403) and (503) and the video decoders (210), (310) and (610) may be implemented using one or more processors that execute software instructions.

[0060] Turning to block partitioning for encoding and decoding, general partitioning can start from basic blocks and can follow a predefined set of rules, a specific pattern, a partition tree, or any partition structure or scheme. Partitioning can be hierarchical and recursive. After following any of the example partitioning procedures described below or other procedures or combinations thereof to divide or partition basic blocks, a final set of partitions or coding blocks can be obtained. Each of these partitions can be in one of the various partition levels in the partition hierarchy and can have various shapes. Each of the partitions can be referred to as a coding block (CB, coding block). For the various example partitioning implementations further described below, each generated CB can have any allowed size and partition level. Such partitions are called coding blocks because they can form units to which some basic encoding / decoding decisions can be made, and the encoding / decoding parameters can be optimized, determined, and signaled in the encoded video code stream. The highest or deepest level in the final partition represents the depth of the coding block partition structure of the tree. The coding block can be a luminance coding block or a chrominance coding block. The CB tree structure for each color can be referred to as a coding block tree (CBT, coding block tree). The coding blocks of all color channels can be collectively referred to as coding units (CU, coding unit). The hierarchical structure of all color channels may be collectively referred to as a coding tree unit (CTU). The partition patterns or structures of various color channels in a CTU may be the same or different.

[0061] In some embodiments, the partition tree schemes or structures for the luma and chroma channels may not have to be the same. In other words, the luma and chroma channels may have separate coding tree structures or patterns. In addition, whether the luma and chroma channels use the same or different coding partition tree structures and the actual coding partition tree structure to be used may depend on whether the stripe being encoded is a P, B or I stripe. For example, for an I stripe, the chroma channel and the luma channel may have separate coding partition tree structures or coding partition tree structure patterns, while for a P or B stripe, the luma and chroma channels may share the same coding partition tree scheme. When separate coding partition tree structures or patterns are applied, the luma channel may be partitioned into CBs by one coding partition tree structure, and the chroma channel may be partitioned into chroma CBs by another coding partition tree structure.

[0062] Figure 7 An exemplary predefined 10-way partition structure / pattern is shown that allows recursive partitioning to form a partition tree. A root block may start at a predefined level (eg, starting from a basic block at a 128x128 or 64x64 level). Figure 7 Example partition structures include various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. In some exemplary embodiments, further subdivision is not allowed. Figure 7A coding tree depth may be further defined to indicate the partition depth from a root node or a root block. For example, the coding tree depth of a root node or a root block may be set to 0, and the root block may follow Figure 7 After being further split once, the coding tree depth is increased by 1. In some embodiments, all square partitions in 710 may only be allowed to follow Figure 7 The pattern is used to recursively partition to the next level of the partition tree.

[0063] In some other example implementations for coding block partitioning, a quadtree structure may be used. This quadtree segmentation may be applied hierarchically and recursively to any square-shaped partition. Whether a basic block or an intermediate block or partition is further quadtree segmented may be adapted to various local characteristics of the basic block or intermediate block / partition.

[0064] In yet other examples, a ternary partitioning scheme may be used to partition a basic block or any intermediate block, such as Figure 8 The ternary mode can be implemented vertically as shown in 802, or horizontally as shown in 804. Figure 8 The example split ratio in is shown as 1:2:1, but other ratios may be predefined. In some embodiments, two or more different ratios may be predefined. In some embodiments, the width and height of the partitions of the example ternary tree are always powers of 2 to avoid extra transformations.

[0065] The above partitioning schemes can be combined in any manner at different partitioning levels. As an example, the above quadtree and binary partitioning schemes can be combined to partition basic blocks into a quadtree-binary-tree (QTBT) structure. In this scheme, basic blocks or intermediate blocks / partitions can be quadtree partitioned or binary partitioned, if specified, subject to a predefined set of conditions. Fig. 9A specific example is illustrated in , where the basic block is a first quadtree partitioned into four partitions, as shown in 902, 904, 906 and 908. Thereafter, each of the resulting partitions is either quadtree partitioned into four further partitions (such as 908), or binary partitioned into two further partitions (horizontally or vertically, such as 902 or 906, for example, both are symmetrical) at the next level, or is non-partitioned (such as 904). For square-shaped partitions, binary or quadtree partitioning can be recursively allowed, as shown in the entire example partitioning pattern of 910 and the corresponding tree structure / representation in 920, where the solid lines represent quadtree partitioning and the dotted lines represent binary partitioning. A flag can be used for each binary partition node (non-leaf binary partition) to indicate whether the binary partitioning is horizontal or vertical. For example, as shown in 920, consistent with the partitioning structure of 910, the flag "0" can represent horizontal binary partitioning, and the flag "1" can represent vertical binary partitioning. For quadtree partitioning, there is no need to indicate the partitioning type, because quadtree partitioning always partitions blocks or partitions horizontally and vertically to produce 4 sub-blocks / partitions of equal size. In some embodiments, a flag "1" may represent horizontal binary partitioning, and a flag "0" may represent vertical binary partitioning.

[0066] In some example implementations of QTBT, the quadtree and binary segmentation rule set may be represented by the following predefined parameters and corresponding functions associated therewith: -CTU size: the root node size of the quadtree (the size of the basic block) -MinQTSize: Minimum allowed quadtree leaf node size -MaxBTSize: Maximum allowed binary tree root node size -MaxBTDepth: Maximum allowed binary tree depth -MinBTSize: The minimum allowed binary tree leaf node size In some example implementations of the QTBT partition structure, the CTU size can be set to 128×128 luma samples with two corresponding 64×64 chroma sample blocks (when example chroma subsampling is considered and used), MinQTSize can be set to 16×16, MaxBTSize can be set to 64×64, MinBTSize (for both width and height) can be set to 4×4, and MaxBTDepth can be set to 4. Quadtree partitioning can be first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes from their minimum allowed size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., the CTU size). If the node is 128×128, it will not be split by the binary tree first because the size exceeds the MaxBTSize (i.e., 64×64). Otherwise, nodes that do not exceed MaxBTSize can be partitioned by the binary tree. Fig. 9 In the example of , the basic block is 128×128. According to a predefined set of rules, the basic block can only be a quadtree partition. The basic block has a partition depth of 0. Each of the four resulting partitions is 64×64, not exceeding MaxBTSize, and can be further quadtree or binary tree partitioned at level 1. The process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), further partitions can be disregarded. When the binary tree node has a width equal to MinBTSize (i.e., 4), further horizontal partitions can be disregarded. Similarly, when the binary tree node has a height equal to MinBTSize, further vertical partitions are disregarded.

[0067] In some example embodiments, the above QTBT scheme can be configured to support the flexibility of having the same QTBT structure or separate QTBT structures for luma and chroma. For example, for P and B slices, the luma and chroma CTBs in one CTU can share the same QTBT structure. However, for I slices, the luma CTB can be partitioned into CBs by the QTBT structure, and the chroma CTB can be partitioned into chroma CBs by another QTBT structure. This means that a CU can be used to refer to different color channels in an I slice, for example, an I slice can consist of coding blocks of a luma component or coding blocks of two chroma components, and a CU in a P or B slice can consist of coding blocks of all three color components.

[0068] The above various CB partitioning schemes and further partitioning of CB to PB can be combined in any manner.The following specific implementations are provided as non-limiting examples.

[0069] Inter prediction can be implemented, for example, in a single reference mode or a composite reference mode. In some embodiments, a skip flag may first be included in the code stream of the current block (or at a higher level) to indicate whether the current block is inter-coded and not skipped. If the current block is inter-coded, another flag may be further included in the code stream as a signal indicating whether a single reference mode or a composite reference mode is used for prediction of the current block. For a single reference mode, one reference block may be used to generate a prediction block for the current block. For a composite reference mode, two or more reference blocks may be used to generate a prediction block (for example) by weighted averaging. One or more reference blocks may be identified using one or more reference frame indexes and additionally using corresponding one or more motion vectors, which indicate one or more shifts between one or more reference blocks and the current block in a position relative to the frame (for example, in horizontal and vertical pixels). For example, an inter-frame prediction block for a current block can be generated from a single reference block as a prediction block in a single reference mode, the single reference block being identified by a motion vector in a reference frame, while for a composite reference mode, the prediction block can be generated by a weighted average of two reference blocks in two reference frames, the two reference frames being indicated by two reference frame indices and two corresponding motion vectors. One or more motion vectors can be encoded in various ways and included in the bitstream.

[0070] In some example embodiments, one or more reference picture lists containing identifications of short-term and long-term reference frames for inter prediction may be formed based on information in a reference picture set (RPS). For example, a single picture reference list for unidirectional inter prediction may be formed, denoted as L0 reference (or reference list 0), while two picture reference lists for bidirectional inter prediction may be formed, denoted as L0 (or reference list 0) and L1 (or reference list 1) for each of the two prediction directions. The reference frames included in the L0 and L1 lists may be ordered in various predetermined ways. The lengths of the L0 and L1 lists may be signaled in the video bitstream. When multiple references used to generate a prediction block by weighted averaging in a composite prediction mode are on the same side of the frame where the block to be predicted is located, the unidirectional inter prediction may be in a single reference mode or in a composite reference mode. Bidirectional inter prediction may be only a composite mode because bidirectional inter prediction involves at least two reference blocks.

[0071] In some embodiments, a merge mode (MM) for inter-frame prediction may be implemented. Typically, for merge mode, one or more motion vectors in a motion vector in a single reference prediction or a motion vector in a composite reference prediction for a current PB may be derived from one or more other motion vectors, rather than being calculated and signaled independently. For example, in an encoding system, one or more current motion vectors for a current PB may be represented by one or more differences between one or more current motion vectors and one or more other already encoded motion vectors (referred to as reference motion vectors). Such one or more differences in one or more motion vectors, rather than the entirety of one or more current motion vectors, may be encoded and included in a bitstream, and may be linked to one or more reference motion vectors. Correspondingly, in a decoding system, one or more motion vectors corresponding to the current PB may be derived based on one or more decoded motion vector differences and one or more decoded reference motion vectors linked thereto. As a specific form of general merge mode (MM) inter-frame prediction, such inter-frame prediction based on one or more motion vector differences may be referred to as a merge mode with motion vector difference (MMVD). Therefore, MM in general or MMVD in particular may be implemented to exploit the correlation between motion vectors associated with different PBs to improve coding efficiency. For example, adjacent PBs may have similar motion vectors, and thus the MVD may be small and may be efficiently encoded. For another example, for blocks of similar position / location in space, the motion vectors may be correlated in time (between frames).

[0072] In some example embodiments of the MMVD, a list of reference motion vectors (RMVs) or MV predictor candidates for motion vector prediction may be formed for the block being predicted. The RMV candidate list may contain a predetermined number (e.g., 2) of MV predictor candidate blocks whose motion vectors may be used to predict the current motion vector. The RMV candidate blocks may include blocks selected from neighboring blocks and / or temporal blocks in the same frame (e.g., blocks located in the same position in a continuation or subsequent frame of the current frame). These options represent blocks at a spatial or temporal position relative to the current block that may have a motion vector similar to or the same as the current block. The size of the MV predictor candidate list may be predetermined. For example, the list may contain two or more candidates. In order to be on the RMV candidate list, for example, the candidate block may need to have the same one or more reference frames as the current block, and must exist (e.g., when the current block is close to the edge of the frame, a boundary check needs to be performed), and must have been encoded during the encoding process, and / or decoded during the decoding process. In some embodiments, if spatial neighboring blocks are available and the above conditions are met, the merge candidate list can be first filled with spatial neighboring blocks (scanned in a specific predefined order), and then filled with temporal blocks if space is still available in the list. For example, adjacent RMV candidate blocks can be selected from the left block and the top block of the current block. The RMV predictor candidate list can be dynamically formed as a dynamic reference list (DRL) at various levels (sequence, picture, frame, stripe, super block, etc.). The DRL can be signaled in the bitstream.

[0073] In some embodiments, the actual MV predictor candidate used as a reference motion vector for predicting the motion vector of the current block can be signaled. In the case where the RMV candidate list contains two candidates, a one-bit flag called a merge candidate flag can be used to indicate the selection of the reference merge candidate. For the current block predicted in composite mode, each of the multiple motion vectors predicted using the MV predictor can be associated with a reference motion vector from the merge candidate list. The encoder can determine which RMV candidate more closely predicts the MV of the current coding block and signal the selection as an index to the DRL.

[0074] In some example embodiments of MMVD, after selecting an RMV candidate and using it as a basic motion vector predictor for a motion vector to be predicted, a motion vector difference (MVD or delta MV, which represents the difference between the motion vector to be predicted and the reference candidate motion vector) may be calculated in the encoding system. Such an MVD may include information representing the magnitude of the MV difference and the direction of the MV difference, both of which may be signaled in the bitstream in various ways.

[0075] In some example implementations of the MMVD, a distance index may be used to specify the magnitude information of the motion vector difference and indicate one of a set of predefined offsets representing a predefined motion vector difference from a starting point (reference motion vector). The MV offset may then be added to the horizontal or vertical component of the starting (reference) motion vector according to the signaled index. An example predefined relationship between the distance index and the predefined offsets is specified in Table 1. Table 1 - Example relationship between distance index and predefined MV offsets

[0076] In some example embodiments of the MMVD, a direction index may be further signaled and used to indicate the direction of the MVD relative to a reference motion vector. In some embodiments, the direction may be limited to either a horizontal direction or a vertical direction. An exemplary 2-bit direction index is shown in Table 2. In the example of Table 2, the interpretation of the MVD may vary depending on the information of the start / reference MV. For example, when the start / reference MV corresponds to a unidirectional prediction block or to a bidirectional prediction block, where both reference frame lists point to the same side of the current picture (i.e., the POCs of both reference pictures are greater than the POC of the current picture, or are less than the POC of the current picture), the symbol in Table 2 may specify the symbol (direction) of the MV offset added to the start / reference MV. When the start / reference MV corresponds to a bidirectional prediction block, in which two reference pictures are on different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture, and the POC of the other reference picture is less than the POC of the current picture), and the difference between the reference POC in picture reference list 0 and the current frame is greater than the difference between the reference POC in picture reference list 1 and the current frame, the symbols in Table 2 may specify the sign of the MV offset added to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture in picture reference list 1 may have an opposite value (opposite sign of the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is greater than the difference between the reference POC in picture reference list 0 and the current frame, the symbols in Table 2 may indicate the sign of the MV offset added to the reference MV associated with picture reference list 1, and the sign of the offset of the reference MV associated with picture reference list 0 has an opposite value. Table 2 - Example implementation of the sign of the MV offset specified by the direction index Directions to IDX 00 01 10 11 x-axis (horizontal) + - not applicable not applicable y axis (vertical) not applicable not applicable + -

[0077] In some example embodiments, the MVD may be scaled according to the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling is required. Otherwise, if the difference in POC in reference list 0 is greater than the difference in POC in reference list 1, the MVD of reference list 1 is scaled. If the POC difference of reference list 1 is greater than list 0, the MVD of list 0 may be scaled in the same manner. If the starting MV is unidirectionally predicted, the MVD is added to the available or reference MV.

[0078] In some example embodiments of MVD encoding and signaling for bidirectional composite prediction, in addition to or as an alternative to separately encoding and signaling two MVDs, symmetric MVD encoding can be implemented so that only one MVD needs to be signaled, and the other MVD can be derived from the signaled MVD. In such embodiments, motion information including reference picture indices of list-0 and list-1 are not all signaled. Specifically, at the slice level, a flag called "mvd_11_zero_flag" can be included in the codestream to indicate whether reference list-1 is not signaled in the codestream. If this flag is 1, indicating that reference list-1 is equal to zero (and therefore not signaled), the bidirectional prediction flag (called "BiDirPredFlag") can be set to 0, meaning that there is no bidirectional prediction. Otherwise, if mvd_11_zero_flag is zero, if the nearest reference picture in list-0 and the nearest reference picture in list-1 form a front-to-back pair of reference pictures or a back-to-front pair of reference pictures, then BiDirPredFlag is set to 1, and both list-0 and list-1 reference pictures are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. BiDirPredFlag of 1 may indicate that the symmetry mode flag is additionally signaled in the bitstream. When BiDirPredFlag is 1, the decoder may extract the symmetry mode flag from the bitstream. For example, the symmetry mode flag may signal the symmetry mode flag at the CU level (if necessary), and it may indicate whether the symmetric MVD coding mode is being used for the corresponding CU. When the symmetric mode flag is 1, it indicates that the symmetric MVD coding mode is used, and only the reference picture indexes of list-0 and list-1 (referred to as "mvp_10_flag" and "mvp_11_flag") are signaled together with the MVD associated with list-0 (referred to as "MVD0"), and the other motion vector difference "MVD1" will be derived instead of signaled. For example, MVD1 can be derived as -MVD0. In this way, only one MVD is signaled in the exemplary symmetric MVD mode.

[0079] In some other example implementations of MV prediction, a coordination scheme may be used to implement general merge mode, MMVD, and some other types of MV prediction for single reference mode and composite reference mode MV prediction. Various syntax elements may be used to signal the manner in which the MV of the current block is predicted. For example, for a single reference mode, the following MV prediction modes may be signaled:

[0080] NEARMV - directly uses one of the motion vector predictors (MVP) in the list indicated by the Dynamic Reference List (DRL) index without any MVD.

[0081] NEWMV - uses one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference, and applies the delta to the MVP (eg, using MVD).

[0082] GLOBALMV - Use motion vectors based on frame-level global motion parameters.

[0083] Likewise, for the composite reference inter prediction mode using two reference frames corresponding to the two MVs to be predicted, the following MV prediction modes may be signaled:

[0084] NEAR_NEARMV-For each of the two MVs to be predicted, in the absence of MVD, use one of the motion vector predictors (MVP) in the list signaled by the DRL index.

[0085] NEAR_NEWMV - To predict the first of the two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as a reference MV without MVD; to predict the second of the two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as a reference MV in combination with a delta MV (MVD) additionally signaled.

[0086] NEW_NEARMV - To predict the second of the two motion vectors, one of the motion vector predictors (MVPs) in the list signaled by the DRL index is used as a reference MV without MVD; to predict the first of the two motion vectors, one of the motion vector predictors (MVPs) in the list signaled by the DRL index is used as a reference MV in combination with an additionally signaled delta MV (MVD).

[0087] NEW_NEWMV - uses one of the motion vector predictors (MVPs) in the list signaled by the DRL index as the reference MV, and uses it in conjunction with the additionally signaled delta MV to predict each of the two MVs.

[0088] GLOBAL_GLOBALMV - Use the MV from each reference based on its frame-level global motion parameters.

[0089] Therefore, the term "NEAR" above refers to MV prediction using a reference MV as a general merge mode without any MVD, while the term "NEW" refers to MV prediction involving the use of a reference MV and the MVD signaled or derived as in MMVD mode to offset it. For composite inter-frame prediction, the above reference basic motion vector and motion vector increment may generally be different or independent between two references or two MVDs, even though the two MVDs may be correlated, for example, and this correlation may be exploited to reduce the amount of information required to signal the two motion vector increments. In order to exploit this correlation, joint signaling of the two MVDs may be implemented and indicated in the codestream, as described in further detail below.

[0090] In some example implementations of the MVD, a predefined pixel resolution of the MVD may be allowed. For example, a motion vector precision (or accuracy) of 1 / 8 pixel may be allowed. The MVD in the various MV prediction modes described above may be constructed and signaled in various ways. In some implementations, various syntax elements may be used to signal the above one or more motion vector differences in reference frame list 0 or list 1.

[0091] For example, a syntax element called "mv_joint" may specify which components of the motion vector difference associated with it are non-zero. For example, mv_joint has the following values: 0, which may indicate that there is no non-zero MVD in the horizontal or vertical direction; 1, which may indicate that there is non-zero MVD only along the horizontal direction; 2, which may indicate that there is non-zero MVD only in the vertical direction; 3, may indicate the presence of non-zero MVD in both horizontal and vertical directions.

[0092] When the "mv_joint" syntax element of the MVD signals the absence of non-zero MVD components, no further MVD information is signaled. However, if the "mv_joint" syntax signals the presence of one or two non-zero components, additional syntax elements may be further signaled for each of the non-zero MVD components as described below.

[0093] For example, a syntax element called "mv_sign" may be used to additionally specify whether the corresponding motion vector difference amount is positive or negative.

[0094] For another example, a syntax element called "mv_class" can be used to specify the class of a motion vector difference in a predefined set of classes for a corresponding non-zero MVD component. For example, the predefined classes for motion vector differences can be used to divide the continuous amplitude space of motion vector differences into non-overlapping ranges of classes. Thus, the signaled MVD class indicates the amplitude range of the corresponding MVD component. In the example implementation shown in Table 3 below, a higher class corresponds to a motion vector difference with a larger amplitude range. The symbol (n, m] is used to represent a range of motion vector differences greater than n pixels and less than or equal to m pixels. Table 3: Magnitude classes of motion vector differences Music Videos Magnitude of MVD MV_CLASS_0 (0，2] MV_CLASS1 (2，4] MV_CLASS2 (4，8] MV_CLASS_3 (8，16] MV_CLASS_4 (16，32] MV_CLASS_5 (32，64] MV_CLASS_6 (64，128] MV_CLASS_7 (128，256] MV_CLASS_8 (256，512] MV_CLASS_9 (512，1024] MV_CLASS10 (1024，2048]

[0095] In some other examples, a syntax element called "mv_bit" may be further used to specify the integer part of the offset between the non-zero motion vector difference component and the start amplitude of the corresponding signaled MV class amplitude range. In some other examples, a syntax element called "mv_fr" may be further used to specify the first 2 fractional bits of the motion vector difference of the corresponding non-zero MVD component, and a syntax element called "mv_hp" may be used to specify the third fractional bit (high resolution bit) of the motion vector difference of the corresponding non-zero MVD component. The two bits "mv_fr" essentially provide a 1 / 4 pixel MVD resolution, while the "mv_hp" bit may further provide a 1 / 8 pixel resolution. In some other embodiments, more than one "mv_hp" bit may be used to provide an MVD pixel resolution finer than 1 / 8 pixel. In some example embodiments, additional flags may be signaled at one or more of the various levels to indicate whether 1 / 8 pixel or higher MVD resolution is supported. If the MVD resolution is not applied to a particular coding unit, the above syntax elements for the corresponding unsupported MVD resolution may not be signaled.

[0096] However, in some other example embodiments, the resolution of motion vector differences in various MVD magnitude classes is differentiated or adaptively adjustable. Specifically, a high resolution MVD for large MVD magnitudes of higher MVD classes may not provide a statistically significant improvement in compression efficiency or coding gain. Thus, for larger MVD magnitude ranges corresponding to higher MVD magnitude classes, the MVD may be encoded with a reduced or non-increased resolution (integer pixel resolution or fractional pixel resolution). The term "resolution" may be further referred to as "pixel resolution".

[0097] In some example embodiments, each MVD class may be associated with a single allowed resolution. In some other embodiments, one or more MVD classes may each be associated with two or more optional MVD pixel resolutions. For example, adaptively allowed MVD pixel resolutions may include, but are not limited to, 1 / 64 pixel (pixel), 1 / 32 pixel, 1 / 16 pixel, 1 / 8 pixel, 1-4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels, 4 pixels ... (arranged in descending order of resolution).

[0098] In some other example embodiments, only a single MVD value may be allowed for MV classes equal to or above a threshold MV class. For example, such a threshold MV class may be MV_CLASS 2. Thus, MV_CLASS 2 and above are only allowed to have a single MVD value and no fractional pixel resolution.

[0099] Turning to various composite inter prediction modes in which each MV can be predicted by a reference motion vector and encoded by an MVD, as described above, two MVDs can be signaled separately or jointly in the bitstream. Thus, in some example embodiments, in addition to the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes described above, another inter prediction mode called JOINT_NEWMV can be introduced for the mode in which the MVD for joint signaling of reference lists 0 and 1 is used. Specifically, if the inter prediction mode is indicated as NEW_NEWMV, the MVDs of reference lists 0 and 1 are signaled separately, and when the inter prediction mode is indicated as JOINT_NEWMV mode, the MVDs of reference lists 0 and 1 are signaled jointly. In particular, for the joint MVD, it may be necessary to signal and transmit only one MVD called JOINT_DELTA_MV in the bitstream, and the MVDs of reference lists 0 and 1 can be derived from JOINT_DELTA_MV. The derived MVD may then be combined with the reference motion vectors in reference list 0 or 1 to generate two motion vectors for locating reference blocks for composite inter prediction.

[0100] In some embodiments of composite inter prediction, the JOINT_NEWMV mode may be signaled together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes. In such embodiments, syntax may be included in the bitstream for indicating any of these alternative composite inter prediction modes at any of the various signaling levels (e.g., sequence level, picture level, frame level, slice level, tile level, super block level, etc.). Alternatively, the JOINT_NEWMV mode may be implemented as a sub-mode of the NEW_NEWMV mode. In other words, in the NEW_NEWMV mode, the two MVDs for the two reference blocks are jointly signaled (thus the JOINT_NEWMV sub-mode) or not signaled (another sub-mode of the NEW_NEWMV mode). In this embodiment, a first syntax element may be included in the codestream to indicate any one of the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes, and when the first syntax element indicates that the NEW_NEWMV mode is selected for the coding block, a second syntax element may be further included in the codestream and may be extracted by the decoder to indicate whether the MVD for the coding block is signaled separately or jointly.

[0101] In some example embodiments, when JOINT_NEWMV mode is signaled and the POC distances between two reference frames and the current frame are different, the MVD may be scaled for reference list 0 or reference list 1 based on the POC distance. Specifically, the distance between reference frame list 0 and the current frame may be denoted as td0, and the distance between reference frame list 1 and the current frame may be denoted as td1. If td0 is equal to or greater than td1, joint_mvd may be used directly for reference list 0, and the MVD of reference list 1 may be derived from joint_mvd based on equation (1).

[0102] Otherwise, if td1 is equal to or greater than td0, joint_mvd is used directly for reference list 1, and the MVD for reference list 0 is derived from joint_mvd based on equation (2).

[0103] In some example embodiments, another inter-frame coding mode called AMVDMV may be added to the single reference case. When the AMVDMV mode is selected, it indicates that AMVD (Adaptive Motion Vector Difference) is applied to signal the MVD. A flag, such as amvd_flag, may be added in the JOINT_NEWMV mode to indicate whether AMVD is applied to the joint MVD coding mode. When adaptive MVD resolution is applied to the joint MVD coding mode, the MVD of the two reference frames is jointly signaled, and the accuracy of the MVD may be implicitly determined by the MVD amplitude. Otherwise, the MVD of the two (or more than two) reference frames is jointly signaled, and conventional MVD coding without adaptive MVD resolution may be applied.

[0104] Switching to composite inter-frame mode, such as Fig.10 As shown, these patterns are obtained by combining the patterns from two different reference frames F i-1 and F i+1 Two assumptions about the motion vectors MV0 and MV1 are used to create the current frame F i In this way, two motion information components (eg, motion vectors) may be signaled in the bitstream for each block.

[0105] Alternatively, if Fig.11 As shown, the two reference frames F i-1 and F i+1 The information in the frame can be combined and projected to the same frame as the current frame F using an interpolation process. i The same time instance is used to generate interpolated frames. Multiple TIP modes can be supported. In one TIP mode, the interpolated frame can be used as an additional reference frame. The current frame F i The coding block of can directly reference the interpolated frame and utilize information from two different references at only the overhead cost of a single inter prediction mode. In another TIP mode, the interpolated frame can be directly assigned as the current frame F i The output of the decoding process of , while skipping any other traditional encoding steps. This mode can have considerable coding and complexity benefits, especially for low bit rate applications.

[0106] In some embodiments, adaptive motion vector resolutions (AMVR) may be supported. In a specific example embodiment, a total of 7 MV precisions (8, 4, 2, 1, 1 / 2, 1 / 4, 1 / 8) may be supported. For each prediction block, the AMVR encoder may search through all supported precision values ​​and signal the best precision to the decoder.

[0107] To reduce encoder run time, two MV precision sets are supported. Each precision set may contain, for example, 4 predefined precisions. The precision set may be adaptively selected at the frame level based on the maximum precision value of the frame. The maximum precision may be signaled in the frame header. Table 4 summarizes the precision values ​​supported by the example based on the exemplary frame level maximum precision. Table 4: MV accuracy supported in two sets Maximum frame-level accuracy Supported MV accuracy 1 / 8 1 / 8、1 / 2、1、4 1 / 4 1 / 4、1、4、8

[0108] In some example implementations of AMVR, there may be a frame-level flag to indicate whether the MV of a frame contains sub-pixel precision. AMVR may be enabled only when the value of the cur_frame_force_integer_mv flag is 0. In AMVR, if the precision of a block is lower than the maximum precision, the motion model and interpolation filter may not be signaled. If the precision of a block is lower than the maximum precision, the motion mode is inferred to be translational motion, and the interpolation filter is inferred to be a REGULAR interpolation filter. Similarly, if the precision of a block is 4 pixels or 8 pixels, the inter-intra mode may not be signaled and may be inferred to be 0.

[0109] In some example embodiments, warping motion compensation may be applied. Unlike conventional motion compensation that assumes a translational motion model between a reference block and a target block, warping motion may utilize an affine model, such as Fig.12 For example, the affine motion model can be expressed by equation (3): Where [x, y] is the coordinate of the original pixel, and [x', y'] is the warped coordinate of the reference block. From equation (1), it can be seen that up to six parameters are used to specify the warping motion: a3 and b3 indicate the traditional translation MV; a1 and b2 indicate the scaling along the MV; and a2 and b1 indicate the rotation.

[0110] In some example embodiments, global motion information may be signaled for each inter-reference frame, the global motion information comprising a global motion type and several motion parameters. Table 5 lists example global motion types and the number of associated parameters. Table 5: Global motion types with associated number of parameters Global motion type Number of parameters Identification (zero motion) 0 Pan 2 Rotation 4 enlarge 4 General Affine 6

[0111] After signaling the reference frame index, if global motion is chosen to be used, the global motion type and parameters associated with the given reference frame may be used for the current coding block.

[0112] Warp motion compensation may be applied locally at the block level. In some example embodiments, for an inter-coded block, local warp motion may be allowed when the following conditions are met: (1) the current block uses a single reference prediction; (2) the width or height of the coded block is greater than or equal to a threshold (e.g., 8); and (3) at least one of the adjacent neighboring blocks uses the same reference frame as the current block.

[0113] If such a local warping motion is used for the current block, the affine model parameters can be estimated based on the MVs of the current block and its adjacent neighboring blocks by minimizing the mean square of the difference between the reference projection and the modeling projection. In order to estimate the parameters of the local warping motion, for example, if the neighboring block uses the same reference frame as the current block, a projection sample pair of the center sample in the neighboring block and its corresponding sample in the reference frame is obtained. Thereafter, three additional samples can be created by shifting the center position by a quarter sample in one or two dimensions. These additional samples can also be considered as projection sample pairs to ensure the stability of the model parameter estimation process.

[0114] The MVs of neighboring blocks used to derive motion parameters may be referred to as motion samples. Motion samples may be selected from neighboring blocks that use the same reference frame as the current block. Warp motion prediction mode may be enabled only for blocks that use a single reference frame. For example, in Fig.12 In the example, the MVs of neighboring blocks B0, B1, and B2 of the current block are referred to as MV0, MV1, and MV2, respectively. As an example, the current block can be predicted using a unidirectional prediction with reference frame Ref0; the neighboring block B0 can be predicted using a composite prediction with reference frames Ref0 and Ref1; the neighboring block B1 can be predicted using a unidirectional prediction with reference frame Ref0; and the neighboring block B2 can be predicted using a composite prediction with reference frames Ref0 and Ref2. The motion vector MV0 of B0 Ref0 、MV1 of B1 Ref0 and B2's MV2 R ef0 can be used as a motion sample to derive affine motion parameters for the current block.

[0115] In some example embodiments, global and local warping motion models may be supported. As described above, a global warping model may be associated with each reference frame, where each of the four non-translational parameters may have a predefined bit (e.g., 12 bits) of precision, and the translational motion vector may be encoded with another predefined bit (e.g., 15 bits) of precision. The coding block may choose to use global warping directly and provide a reference frame index. The global warping model captures frame-level scaling and rotation, and therefore focuses primarily on rigid body motion across the frame. In a local warping model at the coding block level (e.g., WARPED_CAUSAL mode), the warping parameters for the current block may be derived by fitting the model to nearby motion vectors using least squares.

[0116] In another example for extended warp motion compensation (e.g., in WARP_EXTEND mode), the motion of neighboring blocks can be smoothly extended into the current block, but with some ability to modify the warping parameters. This allows for the representation of complex warping motions, spread over multiple blocks, while minimizing blocking artifacts. Such a mode, applied to, for example, NEWMV blocks, can be built on two example constraints: (1) the per-pixel motion vectors generated by the model should be continuous (smoothness) with the per-pixel motion vectors in neighboring blocks; and (2) the pixel at the center of the current block should have a per-pixel motion vector that matches the signaled motion vector of the block as a whole.

[0117] For example, if the block to the left of the current block is warped, then we can create Fig.13 The illustrated model is suitable for motion vectors. The above two example constraints imply certain equations involving the warping parameters of neighboring blocks and the current block. These equations can then be solved to calculate the warping model of the current block. For example, mathematically, if (A, ..., F) represents the warping model of the neighbor and (A', ..., F') represents the extended warping model, then the above constraint (1) implies that at each point along the common edge:

[0118] Points along an edge can have different y values, but they all have the same x value. This means that the y coefficient must be the same on both sides (so B'=B and D'=D). At the same time, the x coefficient gives two equations related to the other coefficients: where, in the last two equations of (5), x is the horizontal position of the vertical column of pixels and is therefore effectively a constant.

[0119] Constraint (2) above states that the motion vector of the block center must be equal to the motion vector signaled using the NEWMV mechanism. This gives two more equations, resulting in a set of six equations for six variables with unique solutions.

[0120] These equations can be solved in both software and hardware, and the solution boils down to basic addition / subtraction / multiplication, and division by powers of 2. As such, the model can be much more complex than least squares based local warping motion compensation.

[0121] In the above example implementation, there may be multiple neighboring blocks that can be used to extend from, and in this case, it may be necessary to select which block to extend from. This may be a similar problem to that encountered in motion vector prediction (for example, there may be several possible motion vectors from nearby blocks to select as the basis for NEWMV encoding). The method for motion vector prediction can be extended to handle the needs of the above WARP_EXTEND. This can be done by tracking the source of each motion vector prediction. WARP_EXTEND can then be enabled only if the selected motion vector prediction is taken from a directly neighboring block. This block can then be used as a single "neighboring block" in the rest of the algorithm.

[0122] In some cases, the neighbor's warp model may be already very good as is, without any further modification. To make encoding of such cases cheaper, the WARP_EXTEND mode may be allowed for NEARMV chunks. The neighbor selection may be the same as for NEWMV, except that in this case it may be required that the neighbor be warped, not just translated. But if true, and the WARP_EXTEND mode is selected, the neighbor's warp model parameters may be copied for the current chunk.

[0123] In some example embodiments, a motion mode called WARP_DELTA may be used. In this mode, the warping model of a block may be encoded as a delta from a predicted warping model, similar to how a motion vector is encoded as a delta from a predicted motion vector. The prediction may be derived from a global motion model (if any) or from neighboring blocks.

[0124] To avoid having multiple ways to encode the same predicted warping model, the following example restrictions may be applied: if the mode is NEARMV or NEWMV, the same neighbor selection logic as described for WARP_EXTEND may be used; if this results in a neighboring block being warped, the model for that block (without applying the rest of the WARP_EXTEND logic) may be used as the prediction; and otherwise, the global warping model may be used as the basis. Then, the deltas for each of the non-translational parameters may be encoded. Finally, the translational portion of the model may be adjusted so that the per-pixel motion vector at the center of the block matches the total motion vector for the block.

[0125] Since the above example implementation involves explicitly encoding the delta per warp parameter, it can be encoded using more bits than other warping modes. Therefore, for blocks smaller than a pre-trained threshold (e.g., 16×16), WARP_DELTA can be disabled. However, the decoding logic is relatively simple, but more complex motions that other warping modes cannot represent can be represented.

[0126] In some other example embodiments, a video block may be divided into sub-blocks, and warp motion compensation may be applied at the sub-block level. For example, the position of a sub-block in a current block may be warped (e.g., translated, rotated, scaled). Warp motion compensation may be performed by warping each sub-block as a whole, but performing conventional motion compensation on each sub-block with or without prediction refinement with optical flow (PROF). In such an embodiment, a motion vector associated with each of the sub-blocks may be derived based on neighboring block motion information. Such sub-block motion compensation may be applied only to a single inter-frame prediction mode, and temporal motion information about the current block may not be used to derive the sub-block motion vector. Because the sub-blocks of the current block may be associated with different motion vectors, such an embodiment effectively captures sub-block level warping.

[0127] In some further extended example embodiments, the derivation of the sub-block motion vector may be based not only on the motion vectors of neighboring blocks, but also on the temporal motion vector prediction value of the current block and / or the motion vector from the reference picture, as described in further detail below.

[0128] In more detail, for sub-block motion compensation that can capture sub-block level warping, a video block to be inter-predicted may be divided into sub-blocks, each of which is associated with a motion vector. A motion vector for each of the sub-blocks relative to a particular reference frame may be determined / derived based on motion vectors associated with one or more of the spatial neighboring blocks of the video block to be inter-predicted in the current frame, or motion vectors from a reference picture of the current frame, or a temporal motion vector predictor of the current block.

[0129] In some example embodiments, determining motion vectors for various sub-blocks in a current block to be inter-predicted may be performed in two stages. In a first stage, a set of virtual sub-blocks surrounding the current block may be assigned motion vectors (referred to as intermediate motion vectors for reasons that will become clear below). Each of these virtual sub-blocks is associated with a neighboring sub-block position relative to the current block. Alternatively, the virtual blocks may be referred to as placeholder neighboring sub-blocks of the current block (as placeholders for the belonging position-sensitive intermediate motion vectors). In a second stage, one or more of the position-sensitive intermediate motion vectors and their virtual sub-block positions relative to the current block are used to determine / calculate / derive motion vectors for each of the sub-blocks in the current block.

[0130] The above example dummy or placeholder sub-block implementation is Fig.14 As shown in the figure. Fig.14As shown, the current video block 1410 in the current frame indicated by the bold boundary line can be predicted under sub-block motion compensation. Therefore, the current video block 1410 can be divided into a plurality of sub-blocks having a specific predefined or adaptively determined sub-block size. Fig.14 In the example of FIG. 14 , the current block 1410 may be divided into 36 equal sub-blocks, as depicted by the dashed lines shown within the current block 1410. Accordingly, virtual sub-blocks or placeholder neighboring sub-blocks may be established around the current block 1410, such as Fig.14 Each or some of these virtual sub-blocks may then be assigned a motion vector, as described in detail below. These assigned motion vectors may then be selectively used to derive sub-block-level motion vectors for sub-blocks within the current block 1410.

[0131] The virtual sub-blocks surrounding the current block 1410 may be specified by their relative neighboring positions with respect to the current block 1410. For example, some of the virtual sub-blocks 1410 may be associated with positions corresponding to spatially neighboring sub-blocks of the current block 1410 that have already been reconstructed in the current frame, such as virtual sub-blocks located above and to the left of the current block 1410, as indicated by the diagonal texture. If the neighboring sub-blocks of the current block 1410 corresponding to these virtual or placeholder sub-blocks in the appropriate positions are encoded in inter-prediction mode, their motion vectors will be identified and known during their reconstruction. However, other virtual sub-blocks, such as virtual sub-blocks below the bottom side and to the right of the current block 1410, may be identified and known during their reconstruction. Fig.14 The virtual sub-blocks indicated by the vertical line texture in the image, and respectively represented as 1420 and 1440, may correspond in position to the neighboring sub-blocks of the current block 1410 in the current frame that have not yet been reconstructed. Therefore, when the current block 1410 is being processed, the neighboring sub-blocks of the current block 1410 corresponding to the virtual sub-blocks 1420 and 1440 in the appropriate positions will not be reconstructed. If the motion vectors of these neighboring sub-blocks are inter-frame predicted, they will also be unknown.

[0132] The current frame corresponds to the Fig.14 Each of the reconstructed neighboring blocks or sub-blocks of the current block 1410 of the virtual sub-block with diagonal texture may be inter-predicted or intra-predicted, and therefore may or may not be associated with a known motion vector. In addition, each of these neighboring blocks or sub-blocks of the current block 1410 may be associated with a co-located reference block or sub-block in a reference frame, or with a temporal block or sub-block in a temporal reference frame (e.g., a temporal prediction frame interpolated from a reference frame), which may or may not be associated with any motion vector (depending on whether these reference sub-blocks or temporal sub-blocks are inter-predicted or intra-predicted).

[0133] The current frame corresponds to the Fig.14 Each of the yet-to-be-reconstructed neighboring blocks or sub-blocks of the current block 1410 of the virtual or placeholder sub-blocks 1420 and 1440 with vertical texture, although not associated with any reconstructed motion vector in the current frame, may be associated with a co-located block or sub-block that has been reconstructed in a reference frame or a temporal reference frame (e.g., a temporal prediction frame interpolated from a reference frame), and these co-located blocks or sub-blocks may or may not be associated with a reconstructed motion vector (depending on whether these temporal sub-blocks are inter-frame predicted or intra-frame predicted).

[0134] Fig.14 Some of the virtual subblocks in may correspond to spatial neighboring blocks or subblocks or co-located temporal blocks or subblocks of the inter-frame prediction of the current block 1410 at the appropriate position, which may belong to a prediction unit larger than the size of the subblock. In this way, more than one neighboring virtual subblock may correspond to the same neighboring prediction unit or temporal prediction unit, and therefore to the same reconstructed motion vector.

[0135] Generally as described above, in the first stage, Fig.14 The virtual sub-blocks around the current block 1410 at different positions in the image may first be assigned motion vectors, and then these attributed motion vectors may be used to determine / calculate / derive the motion vectors of the sub-blocks within the current block 1410. In some example embodiments, the x-component (horizontal component) of the sub-block motion vector in the current block 1410 may be calculated or derived based on the x-component (horizontal component), the motion vectors of the neighboring blocks and / or the temporal motion vector prediction value of the current block and / or the motion vector from the reference picture. In particular, the x-component (horizontal component) of the sub-block motion vector in the current block 1410 may be calculated or derived based on the x-component (horizontal component) of one or more of the motion vectors attributed to the virtual sub-blocks. Similarly, the y-component of the sub-block motion vector of the current block 1410 may be calculated or derived based on the motion vectors of the neighboring blocks and / or the temporal motion vector prediction value of the current block and / or the motion vector from the reference picture. In particular, the y-component (vertical component) of the sub-block motion vector in the current block 1410 may be calculated or derived based on the y-component (vertical component) of one or more of the motion vectors attributed to the virtual sub-blocks.

[0136] For example, each of these virtual sub-blocks may be assigned or given (if available) a correspondingly positioned spatial neighboring block / sub-block of the current block 1410 in the current frame, a co-located block / sub-block from a reference frame, or a co-located block / sub-block from a temporal reference frame interpolated, for example, from a reference frame of the current frame, or an already reconstructed motion vector of a temporal motion vector prediction value of the current block. Determining which motion vector to select from these candidate motion vectors (if available) to attribute to the virtual sub-block position may follow a predefined availability check order for these potential candidate motion vector options. Selecting from the available candidate motion vectors of the virtual sub-block may be referred to as a motion vector collection process for the virtual sub-block. In other words, motion vectors are collected from the candidate motion vectors and selectively attributed to these virtual sub-blocks. If no collectable motion vectors are available for the virtual sub-block, the motion vector of such a virtual sub-block may be filled or calculated following one of the example methods described in further detail below. Then, the collected or otherwise filled or calculated motion vectors for the virtual sub-blocks are used together with their positions to derive the sub-block motion vectors of the sub-blocks of the current block 1410 in the second stage.

[0137] In some example embodiments, when collecting / selecting a motion vector for attributing to a virtual subblock from candidate motion vectors, the candidate motion vectors (already reconstructed current neighboring motion vectors or temporal motion vectors) may be directly acquired according to a predefined availability check order.

[0138] In some of the above example embodiments, if one or more candidate motion vectors of a virtual sub-block to be collected or attributed are not available from its spatial co-located block / sub-block or temporal block / sub-block in the current frame, the motion vector of the virtual sub-block may be filled / derived from one of its neighboring virtual sub-blocks in other ways. Fig.14 In the present invention, if the candidate motion vector for the virtual sub-block 1432 is not available from the corresponding co-located block / sub-block or any temporal block / sub-block in the current frame (for example, these blocks / sub-blocks are intra-predicted rather than inter-predicted), the candidate motion vector of the adjacent virtual sub-block 1434 or 1436 (adjacent to the virtual sub-block 1432 in the virtual sub-block set) can be used to fill in or derive the motion vector belonging to the virtual sub-block 1432.

[0139] In some example embodiments, if the motion vector of the block / sub-block corresponding to the virtual sub-block in the appropriate position in the current frame is not available, the temporal motion vector of the co-located sub-block in the reference frame may be checked first as a candidate motion vector attributed to the virtual sub-block. If such co-located temporal motion vectors are available, they are collected in a predetermined availability check order and used to attribute to the virtual sub-block. Otherwise, the motion vectors attributed to its neighboring virtual sub-blocks may be used to fill in or derive its motion vector.

[0140] In some example embodiments, if the motion vector of the block / subblock corresponding to the virtual subblock at the position in the current frame is not available, the motion vector of the second layer spatial neighboring block / subblock of the current block 1410 may be searched for a candidate motion vector attributable to the virtual subblock. For example, the second layer neighboring block / subblock may refer to an immediately adjacent external block / neighboring subblock of the virtual subblock at the appropriate position in the current frame. If such a second layer neighboring motion vector in the current frame is also not available, the motion vector attributable to the virtual neighboring subblock may be filled or derived as described above. Fig.16 Second layer neighboring sub-block positions relative to the current block and the virtual sub-block are shown as external sub-blocks surrounding the virtual sub-block.

[0141] In some example embodiments, the order of availability check of candidate motion vectors may follow spatial neighboring subblocks in the current frame, next layer spatial neighboring subblocks in the current frame, and temporal subblocks in one or more reference frames.

[0142] In some example embodiments, the order of availability checks of candidate motion vectors may follow spatial neighboring subblocks in the current frame, temporal subblocks in one or more reference frames, and next-layer spatial neighboring subblocks in the current frame, or any other combination and order.

[0143] In some other example embodiments, if a motion vector of a block / subblock corresponding to an appropriately located virtual subblock in the current frame is not available, the motion vector attributed to the virtual subblock may be set to a default value, such as a zero motion vector value.

[0144] In some example embodiments, for the lower right virtual sub-block 1460 (eg, Fig.14 ), if the temporal motion vectors of the sub-blocks co-located with the lower right virtual sub-block 1490 in the reference frame are available, they can be directly obtained and attributed to the virtual sub-block. Otherwise, if the co-located temporal motion vectors are not available, then (1) a weighted average of the attributed motion vectors of the lower left virtual sub-block and the upper right virtual sub-block, or (2) a weighted average of the temporal motion vectors of the sub-blocks corresponding to the lower left virtual sub-block and the upper right virtual sub-block in the reference frame in the spatial position can be used as the motion vector prediction value attributed to the lower right virtual sub-block. In some other embodiments, if the co-located temporal motion vectors are not available, the temporal motion vector prediction value of the lower right virtual sub-block can be considered to be non-existent and may not be used for the motion vector attributed to the lower right virtual sub-block.

[0145] In some example embodiments, as described above, only the spatial / temporal neighboring sub-blocks or blocks are collected. Fig.14 The motion vectors of the virtual sub-blocks at the upper left corner, upper right corner, lower left corner and lower right corner of the middle, and are attributed to Fig.14 The motion vectors of other virtual sub-blocks between these corner virtual sub-blocks can be interpolated using, for example, bilinear interpolation or other predefined nonlinear weighted interpolation of the motion vectors of the corner virtual sub-blocks. For example, the virtual sub-blocks between the upper left corner and the lower left corner virtual sub-blocks can be attributed by interpolation or nonlinear weighting of the motion vectors of the upper left corner and the lower left corner virtual sub-blocks. Similarly, the virtual sub-blocks between the upper left corner and the upper right corner virtual sub-blocks can be attributed by interpolation or nonlinear weighting of the motion vectors of the upper left corner and the upper right corner virtual sub-blocks. Similarly, the virtual sub-blocks between the upper right corner and the lower right corner virtual sub-blocks can be attributed by interpolation or nonlinear weighting of the motion vectors of the upper right corner and the lower right corner virtual sub-blocks. Similarly, the virtual sub-blocks between the lower right corner and the lower left corner virtual sub-blocks can be attributed by interpolation or nonlinear weighting of the motion vectors of the lower right corner and the lower left corner virtual sub-blocks.

[0146] In some example embodiments, the virtual sub-blocks surrounding the current block may be segmented by virtual sub-blocks having available collected candidate motion vectors as described above, and the belonging motion vectors of the virtual sub-block segments therebetween without collectable candidate motion vectors may be derived via linear or non-linear interpolation of the collected motion vectors at both ends of each of these virtual sub-block segments.

[0147] like Fig.15 As shown, and turning to the second stage for determining the motion vector of a sub-block (e.g., sub-block 1510) of the current block based on the belonging motion vectors in the surrounding virtual sub-blocks, in some example embodiments, one or more belonging motion vectors selected from the belonging motion vectors of the upper virtual sub-block 1520, the left virtual sub-block 1530, the right virtual sub-block 1540, and the bottom virtual sub-block 1550 may be weighted averaged to calculate the motion vector of the sub-block 1510. Which virtual sub-blocks are selected from the set of virtual sub-block positions (e.g., the set of upper, left, right, and bottom) for such calculation may be predetermined or dynamically signaled at various levels (block, picture, frame, etc.).

[0148] In the above about Fig.15 In an example implementation of FIG. 1 , a bilinear equation may be used to generate a motion vector for each sub-block of the current block.

[0149] In some example embodiments, a plurality of weighting factor sets may be predefined for calculating motion vectors of sub-blocks of the current block. For example, from a predefined set of weighting factors for calculating motion vectors of a particular sub-block, selecting a set of weighting factors may depend on the relative position of the sub-block within the current block.

[0150] For another example, selecting a set of weighting factors from a predefined set of weighting factors for calculating a motion vector for a particular sub-block may depend on the relative position of the sub-block within the current block and / or the difference between adjacent motion vectors, e.g., the difference between the motion vector belonging to the selected virtual sub-block to be used and the motion vector belonging to its adjacent virtual sub-block.

[0151] For example, if the selected belonging motion vectors used to calculate the motion vector of the sub-block are close (small difference, such as less than a threshold difference), these motion vectors can be averaged to obtain the motion vector of the sub-block regardless of the position of the sub-block. Otherwise, the distance between the sub-block and the selected virtual sub-block ( Fig.15 The length of the arrow in the figure) can be used as a weight for selecting a virtual sub-block based on the selected virtual sub-block (e.g., Fig.15 The motion vector of the sub-block is calculated by using the selected belonging motion vector of the virtual sub-block 1520, 1530, 1540 and 1550).

[0152] In some other example embodiments, the belonging motion vector of the virtual sub-block may be further adjusted by a motion vector difference (MVD) before being used to calculate the motion vector of the sub-block in the current block. Such a motion vector difference may be determined and generated by the encoder and signaled in the bitstream. Correspondingly, the decoder may then use the signaled MVD to adjust the belonging motion vector with the signaled MVD (as collected or derived in the decoder according to the various example embodiments above), and use the adjusted belonging motion vector of the virtual sub-block to calculate the motion vector of the sub-block of the current block.

[0153] In some example embodiments, zero or one MVD may be signaled for a block containing multiple sub-blocks. If one MVD is signaled, it may be applied to the adjustment of the attribution motion vector of the virtual sub-block at a predetermined position. For example, the predetermined position may be the bottom right. In some alternative embodiments, the position at which the signaled MVD is applied may be additionally signaled in a set of positions (e.g., bottom right, bottom left, top left, and top right, or any combination thereof). For example, the set of positions may be predefined and indexed, and the signaling of the position may be implemented via signaling of one of the indexes.

[0154] In some specific example embodiments, two MVDs may be signaled, a first MVD may be applied to the home motion vector of the bottom left virtual sub-block, and a second MVD may be applied to the home motion vector of the top right virtual sub-block.

[0155] In some other example embodiments, a fixed or variable number up to a predefined number of MVDs may be signaled for a block, and the fixed or variable number may be applied to adjust the belonging motion vectors of virtual sub-blocks at a predefined set of ordered positions, e.g., the first, second, and ... signaled MVDs may be mapped to an ordered list of virtual sub-block positions. Alternatively, virtual sub-block positions may be signaled additionally instead of predefining a plurality of signaled MVDs.

[0156] Adjustment of the belonging motion vectors of some selected virtual sub-block positions by the signaled MVD can help provide a more accurate sub-block motion vector for the current block, thereby improving the coding gain. Considering that these positions can be used for interpolation of the belonging motion vectors of other virtual sub-blocks, the example positions for the signaling of the MVD are selected above. Therefore, for such interpolation, the adjusted belonging motion vectors at these virtual sub-block positions can be used instead of the unadjusted belonging motion vectors.

[0157] In some example embodiments, the signaling of the MVD for the purpose of adjusting the belonging motion vector of the above virtual sub-block (including but not limited to the related syntax and context) may be the same as the signaling of the NEWMV inter-prediction mode.

[0158] In some example embodiments, the signaling of MVD for the purpose of adjusting the belonging motion vectors of the above virtual sub-blocks (including but not limited to the relevant syntax and context) may be the same as the signaling of AMVDNEWMV mode, where adaptive MVD resolution is applied.

[0159] In some example embodiments, the signaling of the MVD for the purpose of adjusting the belonging motion vector of the above virtual sub-block (including but not limited to the relevant syntax and context) can be based on the inter-frame prediction mode of the current block. In other words, the signaling syntax or context of these MVDs can be changed block by block according to the inter-frame prediction mode of the block.

[0160] In some example implementations of sub-block motion compensation, the MVP / DRL index of the current block may not be signaled to the codestream and may be derived as zero at the encoder and decoder sides. In other words, under the above sub-block motion compensation, the motion vector derivation of the sub-block depends on the neighboring block motion vector and the temporal vector, and therefore may not need to refer to a specific signaled motion vector prediction value or DRL index.

[0161] In some example embodiments, the above smooth sub-block motion vector or compensation mode may be signaled as one of the inter prediction modes, in parallel with the inter prediction modes, e.g., in parallel with the NEARMV, NEWMV, GLOBALMV, NEAR__NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes.

[0162] In some example embodiments, the above smooth sub-block motion vector or compensation mode may be signaled as one of multiple motion modes, such as Overlapped Block Motion Compensation (OBMC) or Warped Motion Mode, both in various inter prediction modes.

[0163] In some example embodiments, a high-level syntax may be signaled in sequence level / frame level / slice level / super-block level to indicate whether smooth sub-block motion compensation is enabled.

[0164] In some example embodiments, the above smooth sub-block motion vector or compensation mode may be signaled only for skip mode.Example decoding code for parsing information related to skip mode and smooth sub-block motion mode is shown below.

[0165] In the above reference syntax, the codestream is parsed to determine whether the skip mode for a block is enabled. If it is enabled, a flag "subblock" is read from the codestream, which indicates whether the block is to be predicted in subblock mode (e.g., smooth subblock mode).

[0166] Fig.17 An example logic flow 1700 according to the above embodiment is illustrated. The logic flow 1700 starts at S1701. In S1710, a syntax element signaled in a video bitstream is received, the syntax element indicating whether the video block is to be inter-predicted in a sub-block motion compensation mode. In S1720, based on the value of the received syntax element, it is determined whether the video block is to be inter-predicted in the sub-block motion compensation mode. In S1730, based on the available candidate motion vectors including motion information of one or more neighboring blocks of the video block in the current frame and the temporal block of the video block, a set of reference motion vectors corresponding to a set of neighboring blocks or sub-block positions of the video block is determined. In S1740, a sub-block motion vector of each of a plurality of sub-blocks is derived based on the reference motion vector. The logic flow 1700 stops at S1799.

[0167] The above operations can be combined or arranged in any number or order as needed. Two or more steps and / or operations can be performed in parallel. The embodiments and implementation methods in the present disclosure can be used alone or in combination in any order. In addition, each of the method (or embodiment), encoder and decoder can be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer-readable medium. The embodiments in the present disclosure can be applied to luminance blocks or chrominance blocks. The term block can be interpreted as a prediction block, a coding block, or a coding unit, i.e., a CU. The term "block" can also be used here to refer to a transform block. In the following items, when a block size is mentioned, it can refer to a block width or height, or a maximum value of width and height, or a minimum value of width and height, or an area size (width*height), or an aspect ratio of a block (width:height, or height:width).

[0168] The above techniques may be implemented as computer software via computer-readable instructions and physically stored in one or more computer-readable media. Fig.18 A computer system (1800) is shown that is suitable for implementing certain embodiments of the disclosed subject matter.

[0169] The computer software may be encoded in any suitable machine code or computer language, and may be assembled, compiled, linked, or the like to create a code comprising instructions, which may be directly executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed by decoding, microcode, or the like.

[0170] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, etc.

[0171] Fig.18 The components shown for the computer system (1800) are exemplary in nature and are not intended to limit the scope of use or functionality of computer software implementing the embodiments of the present application. The configuration of the components should not be interpreted as having any dependency or requirement on any component or combination of components shown in the exemplary embodiment of the computer system (1800).

[0172] The computer system (1800) may include certain human-computer interface input devices. The human-computer interface input devices may include one or more of the following (only one of which is depicted): keyboard (1801), mouse (1802), touch pad (1803), touch screen (1810), data gloves (not shown), joystick (1805), microphone (1806), scanner (1807), camera (1808).

[0173] The computer system (1800) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (1810), a data glove (not shown), or a joystick (1805), but there may also be tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (1809), headphones (not shown)), visual output devices (e.g., screens (1810) including cathode ray tube screens, liquid crystal screens, plasma screens, organic light emitting diode screens, each of which has or does not have a touch screen input function, each of which has or does not have a tactile feedback function-some of which may output two-dimensional visual outputs or outputs of more than three dimensions by means such as stereoscopic picture output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)) and printers (not shown).

[0174] The computer system (1800) may also include human-accessible storage devices and their associated media, such as optical media including high-density read-only / rewritable compact disks (CD / DVD ROM / RW) (1820) with CD / DVD or similar media (1821), thumb drives (1822), removable hard disk drives or solid state drives (1823), traditional magnetic media such as tapes and floppy disks (not shown), special-purpose ROM / ASIC / PLD-based devices such as security software protectors (not shown), and the like.

[0175] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0176] The computer system (1800) may also include an interface (1855) to one or more communication networks (1854). The network may be wireless, wired, or optical. The network may also be a local area network, a wide area network, a metropolitan area network, an in-vehicle network, an industrial network, a real-time network, a delay-tolerant network, and the like. Examples of networks may include local area networks such as Ethernet, wireless local area networks, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), in-vehicle and industrial networks (including CANBus), and the like.

[0177] The above-mentioned human-machine interface devices, human-accessible storage devices, and network interfaces may be connected to the core (1840) of the computer system (1800).

[0178] The core (1840) may include one or more central processing units (CPUs) (1841), graphics processing units (GPUs) (1842), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1843), hardware accelerators for specific tasks (1844), etc. These devices, as well as read-only memory (ROM) (1845), random access memory (1846), internal mass storage (e.g., internal non-user accessible hard disk drives, solid-state drives, etc.) (1847), etc., may be connected via a system bus (1848). In some computer systems, the system bus (1848) may be accessed in the form of one or more physical plugs so that it may be expanded by additional central processing units, graphics processing units, etc. Peripheral devices may be directly attached to the core's system bus (1848) or connected via a peripheral bus (1849). The architecture of the peripheral bus includes PCI, USB, etc. In one example, the screen (1810) may be connected to a graphics adapter (1850). The architecture of the peripheral bus includes PCI, USB, etc.

[0179] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purpose of this application, or may be medium and code well known and available to those skilled in the art of computer software.

[0180] Although the present application has described a number of exemplary embodiments, various changes, arrangements and various equivalent substitutions of the embodiments are within the scope of the present application. Therefore, it should be understood that those skilled in the art can design a variety of systems and methods, which, although not explicitly shown or described herein, embody the principles of the present application and are therefore within the spirit and scope of the present application.

Claims

1. A method for decoding a video block including a plurality of sub-blocks of a current frame in a video code stream, characterized in that: The method comprises: receiving a syntax element signaled in the video bitstream, the syntax element indicating whether the video block is inter-predicted in a sub-block motion compensation mode; Determining, based on the received value of the syntax element, that the video block is to be inter-predicted in a sub-block motion compensation mode; determining a set of reference motion vectors corresponding to a set of neighboring block or sub-block positions of the video block based on available candidate motion vectors, the available candidate motion vectors including motion information of one or more neighboring blocks of the video block in the current frame and a temporal block of the video block, and Based on the reference motion vector, a sub-block motion vector for each of the plurality of sub-blocks is derived.

2. The method according to claim 1, characterized in that Based on the reference motion vector, deriving the sub-block motion vector of each of the plurality of sub-blocks comprises: deriving a horizontal component of the sub-block motion vector based on the horizontal component of the reference motion vector; and Based on the vertical component of the reference motion vector, the vertical component of the sub-block motion vector is derived.

3. The method according to claim 1, characterized in that For each of at least one of the sets of neighboring blocks or sub-block positions, a reference motion vector is determined by selecting from the available candidate motion vectors comprising and in a predetermined availability checking order of: a first motion vector determined from a neighboring block of the video block in the current frame, the neighboring block being spatially co-located with the neighboring block or sub-block; as well as A second motion vector is derived based on a temporal block that is spatially co-located with the neighboring block or sub-block.

4. The method according to claim 1, characterized in that: For each of at least one of the sets of neighboring blocks or sub-block positions, a reference motion vector is determined by selecting from the available candidate motion vectors comprising and in a predetermined availability checking order of: a first motion vector determined from a neighboring block of the video block in the current frame, the neighboring block being spatially co-located with the neighboring block or sub-block; as well as A second motion vector is determined from a secondary neighboring block of the video block adjacent to the neighboring block or sub-block location in the current frame.

5. The method according to any one of claims 1 to 4, characterized in that For neighboring block or sub-block positions without available candidate motion vectors, the corresponding reference motion vectors are filled in from the reference motion vectors associated with its adjacent neighboring block or sub-block positions.

6. The method according to any one of claims 1 to 4, characterized in that Reference motion vector associated with the lower right neighboring block or sub-block position: if a co-located temporal motion vector is available, deriving from the co-located temporal motion vector; and Otherwise, it is derived as a weighted average of the reference motion vectors associated with the upper right neighboring block or sub-block position and the lower left neighboring block or sub-block position.

7. The method according to any one of claims 1 to 4, characterized in that Reference motion vector associated with the lower right neighboring block or sub-block position: if a co-located temporal motion vector is available, deriving from the co-located temporal motion vector; and Otherwise, it is not used to derive the sub-block motion vector of each sub-block in the plurality of sub-blocks.

8. The method according to any one of claims 1 to 4, characterized in that: The adjacent block or sub-block position set includes corner positions, and the corner positions include an upper left corner position, an upper right corner position, and a lower right corner position; selecting the reference motion vector associated with the angular position from the available candidate motion vectors; as well as Other reference motion vectors associated with neighboring block or sub-block positions other than the angular position are interpolated from the reference motion vector associated with the angular position.

9. The method according to any one of claims 1 to 4, characterized in that Based on the reference motion vector, deriving the sub-block motion vector of the sub-block of the video block comprises: selecting a subset of reference motion vectors for neighboring block or sub-block positions from the set of reference motion vectors, the neighboring block or sub-block positions being the left, top, right and bottom of the sub-block; and The sub-block motion vector of the sub-block is derived based on a subset of the reference motion vectors.

10. The method according to claim 9, characterized in that The sub-block motion vectors for the sub-blocks are derived from a subset of the reference motion vectors using a bilinear equation.

11. The method according to claim 9, characterized in that The sub-block motion vector of the sub-block is derived from a weighted average of a subset of the reference motion vectors.

12. The method according to claim 11, characterized in that A set of weighting factors used to derive the sub-block motion vector from the subset of reference motion vectors is selected from a predetermined or signaled plurality of sets of weighting factors.

13. The method according to claim 12, characterized in that A set of weighting factors is selected based on the position of the sub-block or the difference in a subset of the reference motion vectors.

14. The method according to any one of claims 1 to 4, characterized in that Further including: Parsing the video stream to extract one or more motion vector increments; as well as The motion vector increment is applied to adjust at least one of the set of reference motion vectors before deriving the sub-block motion vector based on the set of reference motion vectors.

15. The method according to claim 14, characterized in that Via a predetermined order, the one or more motion vector increments are attributed and applied to neighboring block or sub-block locations.

16. The method according to claim 14, characterized in that Further comprising parsing the video bitstream to extract signaling identifying neighboring block or sub-block positions corresponding to the at least one of the reference motion vector sets to which the one or more motion vector increments are to be applied.

17. The method according to claim 14, characterized in that The one or more motion vector increments are signaled in the video code stream using the same syntax and context as motion vector difference (MVD) signaling or adaptive resolution MVD signaling.

18. The method according to any one of claims 1 to 4, characterized in that No motion vector prediction value or dynamic reference list index for the video block is signaled in the video code stream.

19. The method according to any one of claims 1 to 4, characterized in that The sub-block motion compensation mode is signaled as one of a plurality of motion modes, the plurality of motion modes additionally including at least one of a warped motion mode and an overlapped block motion compensation (OBMC) mode.

20. The method according to any one of claims 1 to 4, characterized in that The sub-block motion compensation mode is signaled only when skip mode is enabled for the video block.

21. The method according to any one of claims 1 to 4, characterized in that The sub-block motion compensation mode is signaled at one of the sequence, frame, slice or super-block levels.

22. The method according to any one of claims 1 to 4, characterized in that The sub-block motion compensation mode includes a smooth sub-block motion compensation mode.

23. An electronic device, characterized in that: A computer program product comprising a memory for storing instructions, and a processor for executing the stored instructions to: Parse the video bitstream to determine if the video blocks in the current frame are inter-frame predicted in a sub-block motion compensation mode; determining a plurality of sub-blocks of the video block; determining a set of reference motion vectors corresponding to a set of neighboring block or sub-block positions of the video block based on available candidate motion vectors, the available candidate motion vectors including motion information of one or more neighboring blocks of the video block in the current frame and a temporal block of the video block, and A sub-block motion vector for each of the sub-blocks is derived based on the reference motion vector.

24. A non-volatile computer-readable storage medium storing instructions, which, when executed by a processor of an electronic device, cause the electronic device to: receiving a syntax element signaled in the video bitstream, the syntax element indicating whether the video block is inter-predicted in a sub-block motion compensation mode; Determining, based on the received value of the syntax element, that the video block is to be inter-predicted in a sub-block motion compensation mode; determining a set of reference motion vectors corresponding to a set of neighboring block or sub-block positions of the video block based on available candidate motion vectors, the available candidate motion vectors including motion information of one or more neighboring blocks of the video block in the current frame and a temporal block of the video block, and Based on the reference motion vector, a sub-block motion vector for each of the plurality of sub-blocks is derived.

Images