Derivation of motion vector range for enhanced interpolation filter

Abstract

The present invention relates to a method of coding video data using an affine motion model, thereby avoiding a significant reduction of the motion vector range near the image boundaries. The method comprises the steps of obtaining a center motion vector (mv_center) of a coding block, deriving a first motion vector range of the coding block depending on the center motion vector and a motion vector spread, wherein the motion vector spread depends on a size of the coding block, updating the first motion vector range to point inside a first area comprising a reference image, if the first motion vector range at least partially points outside the first area, such that a minimum value and / or a maximum value of the updated first motion vector range points to a boundary of the first area, wherein a difference between the maximum value and the minimum value of the updated first motion vector range is equal to a smaller value of a double value of the motion vector spread and a size of the first area, and performing pixel-based motion compensation depending on the updated first motion vector range.

Description

[0001] Cross-referencing related applications

[0002] This patent application claims priority to U.S. Provisional Application No. 62 / 958,291, filed January 7, 2020. The entire disclosure of the aforementioned patent application is incorporated herein by reference. Technical Field

[0003] Embodiments of this application (the present invention) generally relate to the field of image processing, and more specifically, to inter-frame prediction, such as the derivation of the motion vector range of an enhanced interpolation filter. Background Technology

[0004] Video decoding (video encoding and decoding) is widely used in digital video applications such as broadcast digital television (TV), video transmission based on the Internet and mobile networks, real-time conversational applications such as video chat and video conferencing, DVD and Blu-ray discs, video content capture and editing systems, and portable cameras for security applications.

[0005] Even relatively short videos require a large amount of video data to describe, which can cause difficulties when streaming or otherwise transmitting data over bandwidth-constrained communication networks. Therefore, video data typically needs to be compressed before transmission over modern telecommunications networks. The size of the video can also be an issue when storing it on storage devices due to potentially limited memory resources. Video compression devices typically decode the video data on the source side using software and / or hardware before transmission or storage, thereby reducing the amount of data required to represent the digital video image. Video decompression devices then receive the compressed data at the destination side. Given limited network resources and the growing demand for higher video quality, there is a need to improve compression and decompression techniques to increase compression ratios with minimal impact on image quality.

[0006] In Essential Video Coding (EVC), in case of using affine prediction mode, pixel-based affine transform motion compensation can be applied. Specifically, when an enhanced bi-linear interpolation filter (EIF) is applied, the filtering process includes deriving a pixel-based motion vector field from control point motion vectors, obtaining interpolated samples from the derived motion vectors, and filtering the results using a high-pass filter. In order to meet the memory limit requirement, motion vector clipping can be necessary. However, in the prior art, the motion vector range is significantly reduced (sometimes to one point) if the center motion vector is close to or at the picture boundary. Furthermore, the boundary motion vector can not be clipped to the range corresponding to the EVC motion vector storage format. SUMMARY

[0007] Aspects of the present application provide apparatuses and methods for encoding and decoding as recited in the independent claims.

[0008] The above and other objects can be achieved by subject matters recited in the independent claims. Other implementations are apparent from the dependent claims, the description, and the accompanying drawings.

[0009] Particular embodiments are set forth in the attached dependent claims, other embodiments are set forth in the independent claims.

[0010] In a first aspect, the present application relates to a method for coding video data, comprising the steps of: obtaining a center motion vector (mv_center) of a coding block; deriving a first motion vector range of the coding block depending on the center motion vector and a motion vector spread, wherein the motion vector spread depends on a size of the coding block; if the first motion vector range at least partially points outside a first area comprising a reference picture, updating the first motion vector range to point inside the first area such that a minimum value and / or a maximum value of the updated first motion vector range points to a boundary of the first area, wherein a difference between the maximum value and the minimum value of the updated first motion vector range is equal to a smaller value of a double value of the motion vector spread and a size of the first area; performing pixel-based motion compensation depending on the updated first motion vector range.

[0011] A motion vector is a two-dimensional vector used for inter prediction, providing an offset between coordinates of a decoded picture and coordinates of a reference picture. A reference picture comprises samples that can be used for inter prediction in the decoding process of decoding subsequent pictures in decoding order.

[0012] Min(horMaxPic - horMinPic, 2*deviationMV[log2CbWidth-3]) = Min(width of the first region, double value of the horizontal motion vector dispersion), where the width of the first region can be represented as horMaxPic - horMinPic.

[0013] The difference between the maximum and minimum values of the first MV horizontal component range of the updated first motion vector range can be represented as horMax - horMin.

[0014] Min(horMaxPic - horMinPic, 2*deviationMV[log2CbWidth-3]) = Min(width of the first region, double value of the horizontal motion vector dispersion), where the width of the first region can be represented as horMaxPic - horMinPic.

[0015] Similarly, defining verMax - verMin = Min(verMaxPic, verMinPic + 2*deviationMV[log2CbHeight-3]) - verMinPic = Min(verMaxPic - verMinPic, verMinPic + 2*deviationMV[log2CbHeight-3]) - verMinPic = Min(verMaxPic - verMinPic, 2*deviationMV[log2CbHeight-3]), then

[0016] The difference between the maximum and minimum values of the first MV vertical component range of the updated first motion vector range can be represented as verMax - verMin.

[0017] Min(verMaxPic - verMinPic, 2*deviationMV[log2CbHeight-3]) = Min(height of the first region, double value of the vertical motion vector dispersion), where the height of the first region can be represented as verMaxPic - verMinPic.

[0018] FIG. 12 and 13 An example of a condition where the motion vector range at least partially points outside the boundary of a region comprising a reference picture is shown in Fig. 15, where the motion vector range (extending to the left and right of the center motion vector) partially lies outside the left boundary of the picture. Similarly, the example also applies to the right boundary, the lower boundary and the upper boundary.

[0019] The method of the first aspect avoids a significant reduction of the motion vector range near the picture boundaries.

[0020] In an implementation, the motion vector dispersion is represented by a horizontal motion vector dispersion and / or a vertical motion vector dispersion, and the horizontal motion vector dispersion is derived from a width of the coding block, and the vertical motion vector dispersion is derived from a height of the coding block.

[0021] In another implementation, the horizontal motion vector dispersion is represented as deviationMV[log2CbWidth-3] and the vertical motion vector dispersion is represented as deviationMV[log2CbHeight-3], where cbWidth and cbHeight represent the width and height of the coding block.

[0022] In another implementation, the array deviationMV is set equal to {128, 256, 544, 1120, 2272}.

[0023] In another implementation, the first motion vector range is represented by a first MV horizontal component range and / or a first MV vertical component range, where the first MV horizontal component range comprises a first minimum MV horizontal component value hor_min and a first maximum MV horizontal component value hor_max, and the first MV vertical component range comprises a first minimum MV vertical component value ver_min and a first maximum MV vertical component value ver_max.

[0024] In another implementation, the method further comprises deriving a second motion vector range from a size of a picture comprising the coding block, where the second motion vector range is represented by a second MV horizontal component range and / or a second MV vertical component range, where the second MV horizontal component range comprises a second minimum MV horizontal component value hor_min_pic and a second maximum MV horizontal component value hor_max_pic, and the second MV vertical component range comprises a second minimum MV vertical component value ver_min_pic and a second maximum MV vertical component value ver_max_pic.

[0025] In another implementation, the deriving the second motion vector range according to the size of the picture comprises deriving the second motion vector range according to the size of the picture, the size of the extended region, the position of the coding block within the picture and the size of the coding block.

[0026] In another implementation, the updating the first motion vector range to point inside the first region if the first motion vector range at least partially points outside the first region comprising the reference picture comprises setting an updated value of the first minimum MV horizontal component value hor min equal to the second minimum MV horizontal component value hor min pic if the first minimum MV horizontal component value hor min is smaller than the second minimum MV horizontal component value hor min pic, and deriving an updated value of the first maximum MV horizontal component value hor max according to a sum of the second minimum MV horizontal component value hor min pic and a double value of the horizontal motion vector spread.

[0027] Thus, the second minimum MV horizontal component value hor min pic corresponds to a left border of the first region.

[0028] In another implementation, the updating the first motion vector range to point inside the first region if the first motion vector range at least partially points outside the first region comprising the reference picture comprises setting an updated value of the first maximum MV horizontal component value hor max equal to the second maximum MV horizontal component value hor max pic if the first maximum MV horizontal component value hor max is larger than the second maximum MV horizontal component value hor max pic, and deriving an updated value of the first minimum MV horizontal component value hor min according to a difference between the second maximum MV horizontal component value hor max pic and a double value of the horizontal motion vector spread.

[0029] Thus, the second maximum MV horizontal component value hor max pic corresponds to a right border of the first region.

[0030] In another implementation, the updating the first motion vector range to point inside the first region if the first motion vector range at least partially points outside the first region comprising a reference picture includes: if the first minimum MV vertical component value ver_min is smaller than the second minimum MV vertical component value ver_min_pic, setting an updated value of the first minimum MV vertical component value ver_min equal to the second minimum MV vertical component value ver_min_pic, deriving an updated value of the first maximum MV vertical component value ver_max from a sum of the second minimum MV vertical component value ver_min_pic and a double value of the vertical motion vector spread.

[0031] Thus, the second minimum MV vertical component value ver_min_pic corresponds to an upper boundary of the first region.

[0032] In another implementation, the updating the first motion vector range to point inside the first region if the first motion vector range at least partially points outside the first region comprising a reference picture includes: if the first maximum MV vertical component value ver_max is larger than the second maximum MV vertical component value ver_max_pic, setting an updated value of the first maximum MV vertical component value ver_max equal to the second maximum MV vertical component value ver_max_pic, deriving an updated value of the first minimum MV vertical component value ver_min from a difference between the second maximum MV vertical component value ver_max_pic and a double value of the vertical motion vector spread.

[0033] Thus, the second maximum MV vertical component value ver_max_pic corresponds to a lower boundary of the first region.

[0034] In another implementation, if the first motion vector range at least partially points outside a left boundary of the first region, updating variables hor_min and hor_max representing the first MV horizontal component range of the first motion vector range to:

[0035] hor_min = hor_min_pic,

[0036] hor_max = min(hor_max_pic, hor_min_pic + 2*horizontal motion vector spread), wherein,

[0037] hor_min denotes the updated first minimum MV horizontal component, hor_max denotes the updated first maximum MV horizontal component; hor_min_pic and hor_max_pic denote the second MV horizontal component range of the second motion vector range, hor_min_pic denotes the second minimum MV horizontal component of the second MV horizontal component range, hor_max_pic denotes the second maximum MV horizontal component of the second MV horizontal component range; the second motion vector range depends on the size of the picture comprising the coding block.

[0038] Here, hor_min_pic corresponds to the left boundary of the first region if the first minimum MV horizontal component value hor_min is smaller than the second minimum MV horizontal component value.

[0039] In another implementation, if the first motion vector range at least partially points outside the right boundary of the first region, the variables hor_min and hor_max denoting the first MV horizontal component range of the first motion vector range are updated as follows:

[0040] hor_min = max(hor_min_pic, hor_max_pic - 2 * horizontal motion vector spread),

[0041] hor_max = hor_max_pic, wherein

[0042] hor_min denotes the updated first minimum MV horizontal component, hor_max denotes the updated first maximum MV horizontal component; hor_min_pic and hor_max_pic denote the second MV horizontal component range of the second motion vector range, hor_min_pic denotes the second minimum MV horizontal component of the second MV horizontal component range, hor_max_pic denotes the second maximum MV horizontal component of the second MV horizontal component range; the second motion vector range depends on the size of the picture comprising the coding block.

[0043] Here, hor_max_pic corresponds to the right boundary of the first region if the first maximum MV horizontal component value hor_max is larger than the second maximum MV horizontal component value.

[0044] In another implementation, if the first motion vector range at least partially points outside the upper boundary of the first region, the variables ver_min and ver_max denoting the first MV vertical component range of the first motion vector range are updated as follows:

[0045] ver_min = ver_min_pic,

[0046] ver_max = min(ver_max_pic, ver_min_pic + 2 * vertical motion vector dispersion),

[0047] ver_min denotes an updated first minimum MV vertical component, ver_max denotes an updated first maximum MV vertical component; ver_min_pic and ver_max_pic denote the second MV vertical component range of the second motion vector range, ver_min_pic denotes the second minimum MV vertical component of the second MV vertical component range, ver_max_pic denotes the second maximum MV vertical component of the second MV vertical component range; the second motion vector range depends on a size of an image comprising the coding block.

[0048] Here, if the first minimum MV vertical component value ver_min is smaller than the second minimum MV horizontal component value, ver_min_pic corresponds to an upper boundary of the first region.

[0049] In another implementation, if the first motion vector range is at least partially located outside a lower boundary of the first region, variables ver_min and ver_max denoting the first MV vertical component range of the first motion vector range are updated to:

[0050] ver_min = max(ver_min_pic, ver_max_pic - 2 * vertical motion vector dispersion),

[0051] ver_max = ver_max_pic, wherein

[0052] ver_min denotes an updated first minimum MV vertical component of the first MV vertical component range, ver_max denotes an updated first maximum MV vertical component of the first MV vertical component range; ver_min_pic and ver_max_pic denote the second MV vertical component range of the second motion vector range, ver_min_pic denotes the second minimum MV vertical component of the second MV vertical component range, ver_max_pic denotes the second maximum MV vertical component of the second MV vertical component range; the second motion vector range depends on a size of an image comprising the coding block.

[0053] Here, if the first maximum MV vertical component value ver_max is greater than the second maximum MV vertical component value, ver_max_pic corresponds to a lower boundary of the first region. The size of the picture can be substantially the same as the size of the reference picture.

[0054] In another implementation, the minimum or maximum of the horizontal component of the updated first motion vector range points to a left or right boundary of the first region, respectively, and / or the minimum or maximum of the vertical component of the updated first motion vector range points to an upper or lower boundary of the first region, respectively.

[0055] In another implementation, the first region includes the reference picture and an extended region around the reference picture.

[0056] In another implementation, the size of the extended region depends on the size of a coding tree unit (CTU). A margin can correspond to a region around the reference picture, such that the size of the margin can correspond to the size of the extended region.

[0057] In another implementation, the size of the extended region is 128 pixels.

[0058] In another implementation, the method further includes performing a clipping operation on the updated first motion vector range such that it is within the range of [-2 17 ,2 17 -1]; and performing the pixel-based motion compensation according to the updated first motion vector range includes performing the pixel-based motion compensation according to the updated and clipped first motion vector range.

[0059] In another implementation, variables hor_min, ver_min, hor_max and ver_max represent the first minimum MV horizontal component value, the first minimum MV vertical component value, the first maximum MV horizontal component value and the first maximum MV vertical component value of the updated first motion vector range, respectively, and are clipped as follows:

[0060] hor_max = Clip3(-2 17 ,2 17 -1, hor_max),

[0061] ver_max = Clip3(-2 17 ,2 17 -1, ver_max),

[0062] hor_min = Clip3(-217 ,2 17 -1,hor_min),

[0063] ver_min=Clip3(-2 17 ,2 17 -1,ver_min)

[0064] In another implementation, the performing pixel-based motion compensation according to the updated first motion vector range comprises: performing a clipping operation on the motion vector of a pixel of the coding block to be within a range to obtain a clipped motion vector, wherein the range depends on the updated first motion vector range; and performing pixel-based motion compensation according to the clipped motion vector.

[0065] The motion vector of each pixel of the coding block is obtained according to an affine motion model, and the motion vector of a pixel of an extended area (a margin area) around the coding block is obtained according to the affine motion model.

[0066] In another implementation, the center motion vector of the coding block is obtained by the following equation:

[0067] mv_center[0]=(mvBaseScaled[0]+dX[0]*(cbWidth>>1)+dY[0]*(cbHeight>>1)),

[0068] mv_center[1]=(mvBaseScaled[1]+dX[1]*(cbWidth>>1)+dY[1]*(cbHeight>>1)), wherein

[0069] dX is a horizontal change of the motion vector according to the affine motion model,

[0070] dY is a vertical change of the motion vector according to the affine motion model,

[0071] mvBaseScaled is a motion vector corresponding to the upper left corner of the coding block according to the affine motion model of the coding block,

[0072] cbWidth and cbHeight are two variables representing the width and height of the coding block.

[0073] In another implementation, the first motion vector range of the coding block is derived according to the center motion vector and the motion vector dispersion by the following equation:

[0074] hor_min = mv_center[0] - deviationMV[log2CbWidth - 3],

[0075] ver_min = mv_center[1] - deviationMV[log2CbHeight - 3],

[0076] hor_max = mv_center[0] + deviationMV[log2CbWidth - 3],

[0077] ver_max = mv_center[1] + deviationMV[log2CbHeight - 3],

[0078] wherein

[0079] hor_min is the first minimum MV horizontal component value,

[0080] ver_min is the first minimum MV vertical component value,

[0081] hor_max is the first maximum MV horizontal component value,

[0082] ver_max is the first maximum MV vertical component value,

[0083] mv_center is the center motion vector,

[0084] mv_center[0] corresponds to the horizontal component of the center motion vector, mv_center[1] corresponds to the vertical component of the center motion vector.

[0085] In another implementation, the second motion vector range is derived from the size of the picture by the following equations:

[0086] hor_max_pic = (pic_width + 128 - xCb - cbWidth - 1) « 5,

[0087] ver_max_pic = (pic_height + 128 - yCb - cbHeight - 1) « 5,

[0088] hor_min_pic = (-128 - xCb) « 5,

[0089] ver_min_pic = (-128 - yCb) « 5,

[0090] wherein

[0091] hor_min_pic is the second minimum MV horizontal component value,

[0092] ver_min_pic is the second minimum MV vertical component value,

[0093] hor_max_pic is the second maximum MV horizontal component value,

[0094] ver_max_pic is the second maximum MV vertical component value,

[0095] (xCb, yCb) denotes the position of the coding block in full samples

[0096] cbWidth and cbHeight are two variables denoting the width and height of the coding block,

[0097] pic_width is the width of the picture in samples,

[0098] pic_height is the height of the picture in samples.

[0099] In another implementation, the pixel-based motion compensation is performed using an enhanced interpolation filter (EIF). Thus, the EIF can be used for an affine motion model.

[0100] In a second aspect, the present application relates to a decoder comprising processing circuitry configured to perform the method according to the first aspect or any implementation thereof.

[0101] In one implementation, the decoder comprises an encoder or a decoder.

[0102] In a third aspect, the present application relates to a computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the first aspect or any implementation thereof.

[0103] In a fourth aspect, the present application relates to a decoder comprising: one or more processors; and a non-transitory computer-readable storage medium coupled to the one or more processors and storing instructions for execution by the one or more processors, wherein the instructions, when executed by the one or more processors, cause the decoder to carry out the method according to the first aspect or any implementation thereof.

[0104] In a fifth aspect, the present application relates to an encoder, comprising: one or more processors; and a non-transitory computer-readable medium coupled to the one or more processors and storing programming for execution by the one or more processors, wherein the programming, when executed by the one or more processors, causes the encoder to carry out the method according to the first aspect or any implementation thereof.

[0105] The method according to the first aspect can be performed by the decoder according to the second aspect. Other features and implementation modes of the method according to the first aspect correspond to the features and implementation modes of the decoder according to the second aspect.

[0106] The method according to the first aspect can be performed by the decoder according to the second aspect. Other features and implementation modes of the method according to the first aspect correspond to the features and implementation modes of the decoder according to the second aspect.

[0107] The method according to the first aspect can be performed by the decoder according to the second aspect. Other features and implementation modes of the method according to the first aspect correspond to the features and implementation modes of the decoder according to the second aspect.

[0108] The decoder, the encoder and the method according to the third aspect, the fourth aspect and the fifth aspect have the same advantages as the method according to the first aspect and the corresponding implementation modes.

[0109] The following drawings and detailed description are specifically set forth to provide further appreciation for one or more embodiments. Other features, objects, and / or advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0110] The embodiments of the present application will be described in detail with reference to the drawings, wherein:

[0111] FIG. 1A a block diagram of an example of a video coding system for implementing embodiments of the present application;

[0112] FIG. 1B a block diagram of another example of a video coding system for implementing embodiments of the present application;

[0113] FIG. 2 a block diagram of an example of a video encoder for implementing embodiments of the present application;

[0114] FIG. 3 a block diagram of an example of a video decoder for implementing embodiments of the present application;

[0115] FIG. 4 a block diagram of an example of an encoding device or a decoding device;

[0116] FIG. 5 Block diagram of another example of an encoding device or a decoding device;

[0117] FIG. 6 shows one example of control point based affine motion model (4-parameter affine model and 6-parameter affine model);

[0118] FIG. 7 One example of affine sub-block motion vector field is shown;

[0119] FIG. 8 One example of the angular coordinates of affine block (sub-block) and intermediate EIF block (sub-block) is shown;

[0120] FIG. 9 One example of the position of the transformed block (sub-block) in the reference picture and the corresponding bounding box is shown;

[0121] FIG. 10 One example of updating the motion vector range in case the center motion vector points outside the reference picture is shown;

[0122] FIG. 11 Another example of updating the motion vector range by drawing the real (uncropped) reference region in case the center motion vector points outside the reference picture is shown;

[0123] FIG. 12 One example of reducing the motion vector range is shown;

[0124] FIG. 13 Another example of reducing the motion vector range is shown;

[0125] FIG. 14 A comparison between the design in prior art and the design provided by the present application is shown, where in the design in prior art the motion vector range is reduced to a point near the image boundary.

[0126] FIG. 15 Block diagram of one example structure of a content providing system 3100 for implementing a content distribution service;

[0127] FIG. 16 Block diagram of one example structure of a terminal device;

[0128] FIG. 17 Block diagram of one method provided by the present application;

[0129] FIG. 18 Block diagram of one decoder provided by the present application;

[0130] FIG. 19 Block diagram of one encoder provided by the present application.

[0131] In the following description, identical reference signs will refer to identical or at least functionally equivalent features, if not explicitly stated otherwise. DETAILED DESCRIPTION

[0132] In the following description, reference is made to the accompanying drawings which form a part hereof, and in which are shown by way of illustration specific aspects or specific aspects that can be used in implementing embodiments of the application. It is to be understood that other aspects can be used and structural or logical changes can be made without departing from the scope of the present application. The following detailed description, therefore, is not to be taken in a limiting sense, as the scope of the present application is defined by the appended claims.

[0133] For example, it is to be understood that the disclosure in relation to a described method can equally apply to a corresponding device or system for carrying out the described method, and vice versa. For example, if one or more specific method steps are described, a corresponding device can include one or more units (for example, functional units) to perform the described one or more method steps (for example, one unit performing the one or more steps, or several units each performing one or more of the steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, for example, if a specific apparatus is described based on one or more units (for example, functional units), a corresponding method can include one or more steps to perform the features of the one or more units (for example, one step performing the features of the one or more units, or several steps each performing the features of one or more of the units), even if such one or more steps are not explicitly described or illustrated in the figures. Further, it is understood that the features of the example embodiments and / or aspects described herein can be combined with each other, unless explicitly stated otherwise.

[0134] Video coding generally refers to the processing of a sequence of pictures forming a video or video sequence. In the field of video coding, the terms "frame" and "picture / image" can be used as synonyms. Video coding (or generally coding) comprises both video encoding and video decoding. Video encoding is performed at the source side and generally comprises processing (e.g. by compression) of original video pictures to reduce the amount of data required to represent the video pictures (thus enabling more efficient storage and / or transmission). Video decoding is performed at the destination side and generally comprises an inverse process with respect to the encoder for reconstructing the video pictures. Embodiments relate to the "coding" of video pictures (or generally pictures), which is to be understood as "encoding" or "decoding" of video pictures or corresponding video sequences. The encoding part and the decoding part are also referred to as CODEC (COde DECoder).

[0135] In case of lossless video coding, the original video pictures can be reconstructed, i.e. the reconstructed video pictures have the same quality as the original video pictures (assuming no transmission losses or other data loss during storage or transmission). In case of lossy video coding, further compression is performed by quantization or the like to reduce the amount of data required to represent the video pictures, and the video pictures cannot be completely reconstructed at the decoder side, i.e. the reconstructed video pictures have a lower or worse quality than the original video pictures.

[0136] Several video coding standards belong to the group of "lossy hybrid video codecs" (i.e. spatial and temporal prediction in the sample domain is combined with 2D transform coding for quantization in the transform domain). Each picture of a video sequence is typically partitioned into a set of non-overlapping blocks, and coding is typically performed at the block level. In other words, the encoder side typically processes, i.e. encodes, a video at the block (video block) level, e.g. by generating a prediction block by spatial (intra) prediction and / or temporal (inter) prediction, subtracting the prediction block from the current block (block currently processed / pending for processing) to obtain a residual block, transforming the residual block in the transform domain and quantizing the residual block to reduce the amount of data to be transmitted (compressed); and the decoder side applies an inverse processing procedure with respect to the encoder to the encoded or compressed block to reconstruct the current block for representation. Furthermore, the processing loop of the encoder is identical to the processing loop of the decoder, such that the encoder and the decoder generate identical predictions (e.g. intra and inter predictions) and / or reconstructions for processing, i.e. coding, subsequent blocks.

[0137] In the following embodiments of a video coding system 10, a video encoder 20 and a video decoder 30 are based on FIGS. 1A-3 are described.

[0138] FIG. 1A As a schematic block diagram of an exemplary coding system 10, e.g. a video coding system 10 (or simply coding system 10), can make use of the techniques of the present application. The video encoder 20 (or simply encoder 20) and the video decoder 30 (or simply decoder 30) in the video coding system 10 represent examples of devices that can be used to perform various techniques in accordance with the various examples described in this application.

[0139] As shown in FIG. 1A The coding system 10 comprises a source device 12 for providing encoded picture data 21 to a destination device 14 or the like for decoding the encoded picture data 21.

[0140] The source device 12 comprises the encoder 20 and can additionally, i.e. optionally, comprise a picture source 16, a pre-processor (or pre-processing unit) 18, e.g. a picture pre-processor 18, and a communication interface or communication unit 22.

[0141] The image source 16 can comprise or be any type of image capturing device, such as a camera for capturing real-world images; and / or any type of image generating device, such as a computer graphics processor for generating computer-animated images; or any other device for obtaining and / or providing real-world images, computer-animated images (such as screen content, virtual reality (VR) images) and / or any combination thereof (such as augmented reality (AR) images). The image source can be any type of memory (storage) storing any of the above images.

[0142] For distinguishing the processing performed by the pre-processor 18 and the pre-processing unit 18, the image or image data 17 can also be referred to as raw image or raw image data 17.

[0143] The pre-processor 18 is configured to receive the (raw) image data 17 and to pre-process the image data 17 to obtain pre-processed image data 19 or pre-processed image 19. The pre-processing performed by the pre-processor 18 can include trimming, color format conversion (such as from RGB to YCbCr), color grading or de-noising, etc. It should be appreciated that the pre-processing unit 18 can be an optional component.

[0144] The video encoder 20 is configured to receive the pre-processed image data 19 and to provide encoded image data 21 (more details will be described below in connection with the description of the decoder 30). FIG. 2

[0145] The communication interface 22 in the source device 12 can be configured to receive the encoded image data 21 and to transmit the encoded image data 21 (or data derived by further processing of the encoded image data 21) to another device (such as the destination device 14) or any other device via the communication channel 13 for storage or direct reconstruction.

[0146] The destination device 14 comprises a decoder 30 (such as a video decoder 30) and can additionally (i.e. optionally) comprise a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32) and a display device 34.

[0147] The communication interface 28 in the destination device 14 is configured to receive the encoded image data 21 (or data derived by further processing of the encoded image data 21) directly from the source device 12 or from a storage device, such as an encoded image data storage device or any other source, and to provide the encoded image data 21 to the decoder 30.

[0148] ​The communication interface 22 and the communication interface 28 can be configured to transmit or receive the encoded image data 21 or the encoded data 21 via a direct communication link (e.g., a direct wired or wireless connection) between the source device 12 and the destination device 14 or via any type of network (e.g., a wired network, a wireless network, or any combination thereof, or any type of private and public network or any type of combination thereof).

[0149] For example, the communication interface 22 can be configured to encapsulate the encoded image data 21 into a suitable format (e.g., a data packet) and / or to process the encoded image data by any type of transmission encoding or processing in order to transmit it over the communication link or the communication network.

[0150] For example, the communication interface 28 corresponding to the communication interface 22 can be configured to receive the transmitted data and to process it by any type of corresponding transmission decoding or processing and / or de-encapsulation to obtain the encoded image data 21.

[0151] The communication interface 22 and the communication interface 28 can both be configured as FIG. 1A The communication interface 22 and the communication interface 28 can both be configured as

[0152] The decoder 30 is configured to receive the encoded image data 21 and to provide decoded image data 31 or a decoded image 31 (more details will be described below in connection with the description of the FIG. 3 or FIG. 5 etc.).

[0153] The post-processor 32 of the destination device 14 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data) (e.g., the decoded image 31) to obtain post-processed image data 33 (e.g., a post-processed image 33). For example, the post-processing performed by the post-processing unit 32 can include a color format conversion (e.g., from YCbCr to RGB), toning, pruning, resampling, or any other processing for providing the decoded image data 31 for display by the display device 34 or the like, etc.

[0154] The display device 34 in the destination device 14 is used to receive the post-processed image data 33 in order to display the image to a user or viewer. The display device 34 can be or can include any type of display for representing the reconstructed image, such as an integrated or external display or screen. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display.

[0155] Although FIG. 1A In this embodiment, source device 12 and destination device 14 are shown as separate devices. However, device embodiments may also include both devices or the functions of both devices simultaneously, that is, simultaneously including source device 12 or its corresponding functions and destination device 14 or its corresponding functions. In these embodiments, the same hardware and / or software may be used, or separate hardware and / or software or any combination thereof may be used to implement source device 12 or its corresponding functions and destination device 14 or its corresponding functions.

[0156] It is obvious to those skilled in the art from the above description that FIG. 1A The presence and (precise) functional division of the different units or functions of the source device 12 and / or destination device 14 shown may vary depending on the actual device and application.

[0157] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30, can be transmitted via, for example, FIG. 1B The processing circuitry shown can be implemented using, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding-specific processors, or any combination thereof. Encoder 20 can be implemented using processing circuitry 46 to reflect the reference. FIG. 2 The encoder 20 describes various modules and / or any other encoder system or subsystem described herein. The decoder 30 may be implemented via processing circuitry 46 to reflect the reference... FIG. 3the various modules described with respect to decoder 30 and / or any other decoder system or subsystem described herein. The processing circuitry can be used to perform the various operations discussed below. As FIG. 5 If the techniques are implemented partially in software, a device can store instructions for the software in a suitable, non- transitory computer-readable storage medium and can execute the instructions in hardware using one or more processors to perform the techniques of the present application. Either video encoder 20 or video decoder 30 can be integrated in a single device as part of a combined video encoder / decoder (CODEC), as FIG. 1B is shown.

[0158] Source device 12 and destination device 14 can comprise any of a variety of devices including handheld or fixed devices, e.g., notebook or laptop computers, mobile phones, smart phones, tablets, tablet computers, camcorders, desktop computers, set-top boxes, televisions, display devices, digital media players, video gaming consoles, video streaming devices (e.g., content serving servers or content distribution servers), broadcast receiver devices, broadcast transmitter devices, etc., and can use or not use any type of operating system. In some cases, source device 12 and destination device 14 can be equipped for wireless communication. Thus, source device 12 and destination device 14 can be wireless communication devices.

[0159] In some cases, FIG. 1A Video coding system 10 as shown is merely exemplary and the techniques of this application can be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily involve any data communication between an encoding device and a decoding device. In other examples, data is retrieved from local storage, streamed across a network, etc. A video encoding device can encode data and store the data to memory, and / or a video decoding device can retrieve data from memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with one another, but simply encode data to memory and / or retrieve data from memory and decode the data.

[0160] For ease of description, this document refers to the High-Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC) (next-generation video coding standards) reference software developed by the Joint Collaboration Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). Those skilled in the art will understand that the embodiments of this invention are not limited to HEVC or VVC.

[0161] Encoders and Encoding Methods

[0162] FIG. 2 This is a schematic block diagram of an exemplary video encoder 20 used to implement the technology of this application. FIG. 2 In the example, the video encoder 20 includes an input terminal 201 (or input interface 201), a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter unit 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal 272 (or output interface 272). The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). FIG. 2 The video encoder 20 shown can also be called a hybrid video encoder or a hybrid video codec-based video encoder.

[0163] The residual calculation unit 204, transform processing unit 206, quantization unit 208, and mode selection unit 260 can form the forward signal path of the encoder 20, while the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, buffer 216, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 can form the backward signal path of the video encoder 20. The backward signal path of the video encoder 20 corresponds to the decoder (see [link to decoder]). FIG. 3The signal path of the illustrated video decoder 30) is shown. The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction unit 244 and the intra prediction unit 254 also constitute an "in-built decoder" of the video encoder 20.

[0164] Image and image partitioning (images and blocks)

[0165] For example, the encoder 20 can be configured to receive an image 17 (or image data 17) via the input 201, e.g. an image of a sequence of images forming a video or video sequence. The received image or image data can also be a pre-processed image 19 (or pre-processed image data 19). For simplicity, both are described as image 17 in the following. The image 17 can also be referred to as current image or picture to be coded (especially in video coding to distinguish the current image from other images (e.g. previously encoded images and / or decoded images) in the same video sequence, i.e. a video sequence that also comprises the current image).

[0166] A (digital) image is or can be regarded as a two-dimensional array or matrix of samples with intensity values. A sample in the array can also be referred to as a pixel (short for picture element). The number of samples in the array or image in horizontal and vertical direction (or axis) defines the size and / or resolution of the image. For representing colors, usually three color components are employed, i.e. an image can be represented as or can comprise three arrays of samples. In RGB format or color space, an image comprises corresponding arrays of red, green, blue samples. However, in video coding, each pixel is usually represented in a luma and chroma format or color space, e.g. YCbCr, comprising one luma component (sometimes also denoted as L) and two chroma components denoted as Cb and Cr. The luma component Y represents the luminance or gray scale intensity (e.g. as in a gray scale image), while the two chroma components Cb and Cr represent the chrominance or color information components. Correspondingly, a YCbCr formatted image comprises one array of luma samples composed of luma sample values (Y) and two arrays of chroma samples composed of chroma values (Cb and Cr). An RGB formatted image can be converted or transformed into YCbCr format and vice versa, a process also referred to as color conversion or color transform. If the image is black and white, the image can comprise only one array of luma samples. Correspondingly, for example, an image can be one array of luma samples in black and white format or one array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2 and 4:4:4 color format.

[0167] Embodiments of the video encoder 20 may include an image segmentation unit ( FIG. 2 (Not shown in the image) is used to segment image 17 into multiple (typically non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (in the H.264 / AVC standard), coding tree blocks (CTBs), or coding tree units (CTUs) (in the H.265 / HEVC and VVC standards). Image segmentation units can be used to apply the same block size to all images in a video sequence using a corresponding grid with defined block sizes, or to vary the block size between images, subsets of images, or groups of images, dividing each image into multiple corresponding blocks.

[0168] In other embodiments, the video encoder may be used to directly receive blocks 203 of image 17, such as one, several, or all of the blocks that make up image 17. Image block 203 may also be referred to as the current image block or the image block to be decoded.

[0169] Similar to image 17, image block 203 is also a two-dimensional array or matrix composed of pixels with intensity values ​​(sample values), but the size of image block 203 is smaller than that of image 17. In other words, depending on the applied color format, block 203 may include, for example, a sample array (e.g., a luminance array when image 17 is a black and white image, or a luminance array or a chrominance array when image 17 is a color image), three sample arrays (e.g., a luminance array and two chrominance arrays when image 17 is a color image), or any other number and / or type of array. The number of samples in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, a block may be an M×N (M columns × N rows) sample array or an M×N transform coefficient array, etc.

[0170] exist FIG. 2 In the embodiment of the video encoder 20 shown, the video encoder 20 can be used to encode the image 17 block by block, for example, to encode and predict each block 203.

[0171] exist FIG. 2 In the embodiment of the video encoder 20 shown, the video encoder 20 can also be used to segment and / or encode an image using slices (also known as video slices), wherein one or more slices (typically non-overlapping slices) can be used to segment or encode an image, and each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., tiles (H.265 / HEVC and VVC) or bricks (VVC)).

[0172] exist FIG. 2In the illustrated embodiment of video encoder 20, video encoder 20 can also be configured to partition and / or encode pictures using slices / tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles), where a picture can be partitioned or encoded using one or more slices / tile groups (typically non-overlapping slices / tile groups), each of which can include one or more blocks (e.g., CTUs) or one or more tiles, each of which can be, for example, rectangular and can include one or more blocks (e.g., CTUs), e.g., complete blocks or partial blocks.

[0173] Residual calculation

[0174] Residual calculation unit 204 can be configured to calculate a residual block 205 (also referred to as residual 205) from picture block 203 and prediction block 265 (prediction block 265 will be described in more detail below) by, for example, subtracting sample values of prediction block 265 from sample values of picture block 203 on a sample-by-sample (pixel-by-pixel) basis to obtain residual block 205 in the sample domain.

[0175] Transform

[0176] Transform processing unit 206 can be configured to apply a transform, such as a discrete cosine transform (DCT) or a discrete sine transform (DST), to sample values of residual block 205 to obtain transform coefficients 207 in the transform domain. Transform coefficients 207 can also be referred to as transform residual coefficients and represent residual block 205 in the transform domain.

[0177] Transform processing unit 206 can be configured to apply an integer approximation of a DCT / DST, such as the transform specified by H.265 / HEVC. In comparison to an orthogonal DCT transform, such an integer approximation is typically scaled by a certain factor. In order to maintain the norm of a residual block that is processed by the forward and inverse transform, other scaling factors are applied as part of the transform process. The scaling factors are typically chosen according to certain constraints, such as the scaling factors being powers of two for shift operations, the bit depth of the transform coefficients, a tradeoff between precision and implementation cost, etc. The specific scaling factors are specified for the inverse transform (and, on the video decoder 30 side, for the corresponding inverse transform) by, for example, inverse transform processing unit 212; correspondingly, the corresponding scaling factors can be specified for the forward transform on the encoder 20 side by, for example, transform processing unit 206.

[0178] In an embodiment of the video encoder 20, the video encoder 20 (respectively, the transform processing unit 206) can be configured to output, e.g., directly or after encoding by the entropy encoding unit 270, one or more transform parameters of the one or more transforms such that the video decoder 30 can receive and use the transform parameters for decoding.

[0179] quantization

[0180] The quantization unit 208 can be configured to quantize the transform coefficients 207 by applying scalar quantization or vector quantization, etc., to obtain quantized coefficients 209. The quantized coefficients 209 can also be referred to as quantized transform coefficients 209 or quantized residual coefficients 209.

[0181] The quantization process can reduce the bit depth of some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m, and the degree of quantization can be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, different scaling can be performed to achieve finer or coarser quantization. The smaller the quantization step size, the finer the corresponding quantization; and the larger the quantization step size, the coarser the corresponding quantization. A suitable quantization step size can be represented by a quantization parameter (QP). For example, the quantization parameter can be an index to a predefined set of suitable quantization step sizes. For example, a smaller quantization parameter can correspond to finer quantization (smaller quantization step size), a larger quantization parameter can correspond to coarser quantization (larger quantization step size), and vice versa. Quantization can include division by the quantization step size and a corresponding dequantization or inverse quantization performed by the inverse quantization unit 210, or can include multiplication by the quantization step size. In some embodiments, according to some standards such as HEVC, a quantization parameter can be used to determine the quantization step size. Typically, the quantization step size can be calculated using a fixed-point approximation of an equation depending on the quantization parameter. An additional scaling factor can be introduced for quantization and dequantization to recover the norm of the residual block that can have been modified due to the scaling used in the fixed-point approximation of the equation for the quantization step size and the quantization parameter. In an example implementation, the scaling of the inverse transform and the dequantization can be combined. Alternatively, a custom quantization table can be used and signaled to the decoder by the encoder, e.g., in a bitstream. Quantization is a lossy operation, where the larger the quantization step size, the larger the loss.

[0182] In an embodiment of the video encoder 20, the video encoder 20 (respectively, the quantization unit 208) can be configured to output, e.g., directly or after encoding by the entropy encoding unit 270, a quantization parameter (QP) such that the video decoder 30 can receive and use the quantization parameter for decoding.

[0183] Dequantization

[0184] The dequantization unit 210 is configured to perform dequantization of the quantization performed by the quantization unit 208 on the quantized coefficients to obtain dequantized coefficients 211, e.g., according to or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 can also be referred to as dequantized residual coefficients 211, corresponding to the transform coefficients 207, but typically not identical to the transform coefficients due to the loss caused by the quantization.

[0185] Inverse transform

[0186] The inverse transform processing unit 212 is configured to perform an inverse transform of the transform performed by the transform processing unit 206, e.g., an inverse discrete cosine transform (DCT), an inverse discrete sine transform (DST), or another inverse transform, to obtain a reconstructed residual block 213 (or corresponding dequantized coefficients 211) in the sample domain. The reconstructed residual block 213 can also be referred to as a transform block 213.

[0187] Reconstruction

[0188] The reconstruction unit 214, e.g., a summer or adder 214, is configured to add sample values of the reconstructed residual block 213 and sample values of the prediction block 265 sample by sample, e.g., by adding the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265, to obtain a reconstructed block 215 in the sample domain.

[0189] Filtering

[0190] The in-loop filter unit 220 (or simply "in-loop filter" 220) is configured to filter the reconstructed block 215 to obtain a filtered block 221, or generally to filter reconstructed samples to obtain filtered sample values. For example, the in-loop filter unit is configured to smooth pixel transitions or to improve video quality. The in-loop filter unit 220 can include one or more in-loop filters, such as a deblocking filter, a sample adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, the in-loop filter unit 220 can include a deblocking filter, a SAO filter, and an ALF filter. The filtering processes can be performed in the order of the deblocking filter, the SAO filter, and the ALF filter. For another example, a process called luma mapping with chroma scaling (LMCS), i.e., an adaptive in-loop reshaper, is added. This process is performed before deblocking filtering. For yet another example, the deblocking filter process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the in-loop filter unit 220 is shown as an in-loop filter in FIG. 2, in other configurations, the in-loop filter unit 220 can be implemented as a post-loop filter. The filtered block 221 can also be referred to as a filtered reconstructed block 221. FIG. 2 In an embodiment of the video encoder 20, the video encoder 20 (correspondingly, the in-loop filter unit 220) can be configured to output, e.g., directly or after encoding by the entropy encoding unit 270, in-loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters) so that the decoder 30 can receive and use the same or different in-loop filter parameters for decoding.

[0191] In an embodiment of the video encoder 20, the video encoder 20 (correspondingly, the in-loop filter unit 220) can be configured to output, e.g., directly or after encoding by the entropy encoding unit 270, in-loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters) so that the decoder 30 can receive and use the same or different in-loop filter parameters for decoding.

[0192] The decoded picture buffer

[0193] The decoded picture buffer (DPB) 230 can be a memory that stores reference pictures or generally stores reference picture data for use in encoding video data by the video encoder 20. The DPB 230 can be composed of any of a plurality of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer (DPB) 230 can be used to store one or more filtered blocks 221. The decoded picture buffer 230 can also be used to store other previously filtered blocks (e.g., previously reconstructed and filtered blocks 221) in the same current picture or different pictures (e.g., previously reconstructed pictures) and can provide complete previously reconstructed (i.e., decoded) pictures (and corresponding reference blocks and samples) and / or partial reconstructed current pictures (and corresponding reference blocks and samples) for inter prediction, etc. The decoded picture buffer (DPB) 230 can also be used to store one or more unfiltered reconstructed blocks 215, or generally store unfiltered reconstructed samples, e.g., if the reconstructed blocks 215 are not filtered by the loop filter unit 220, or store reconstructed blocks or reconstructed samples that are not subject to any other processing.

[0194] Mode selection (partitioning and prediction)

[0195] The mode selection unit 260 includes a partitioning unit 262, an inter prediction unit 244, and an intra prediction unit 254 to receive or obtain original picture data, such as original blocks 203 (current blocks 203 of a current picture 17) and reconstructed picture data, e.g., filtered and / or unfiltered reconstructed samples or blocks of the same (current) picture and / or one or more previously decoded pictures, from the decoded picture buffer 230 or other buffers (e.g., line buffers, not shown). The reconstructed picture data is used as reference picture data in prediction, e.g., inter prediction or intra prediction, to derive predicted blocks 265 or prediction values 265.

[0196] The mode selection unit 260 can be used to determine or select one partitioning and one prediction mode (e.g., an intra prediction mode or an inter prediction mode) for a current block prediction mode (including the no partitioning case), generate a corresponding predicted block 265, to calculate a residual block 205 and reconstruct a reconstructed block 215.

[0197] In some embodiments, the mode selection unit 260 can be configured to select the partitioning and prediction modes (e.g., from those modes supported or available to the mode selection unit 260). The partitioning and prediction modes provide the best match or, in other words, the smallest residual (meaning better compression for transmission or storage), or provide the smallest signaling overhead (meaning better compression for transmission or storage), or consider or balance both. The mode selection unit 260 can be configured to determine the partitioning and prediction modes according to rate distortion optimization (RDO), i.e., to select the prediction mode that provides the smallest rate distortion. Herein, the terms “best”, “smallest”, “optimal” and the like do not necessarily refer to an overall “best”, “smallest”, “optimal” and the like, but can also refer to a situation that meets a termination or selection criterion, e.g., a certain value exceeds or is below a threshold or other limit, which can lead to a “suboptimal selection”, but reduces complexity and processing time.

[0198] In other words, the partitioning unit 262 can be configured to partition the pictures in the video sequence into a sequence of coding tree units (CTUs) by iteratively using quad-tree-partitioning (QT), binary partitioning (BT) or triple-tree-partitioning (TT) or any combination thereof, for example, to further partition the CTUs 203 into smaller partitions or sub-blocks (which form blocks again), and to perform prediction, e.g., for each of the partitions or sub-blocks, wherein the mode selection comprises selecting the tree structure of the partitions 203 and selecting the prediction mode applied to each partition or sub-block.

[0199] The partitioning (e.g., performed by the partitioning unit 260) and the prediction processing (performed by the inter prediction unit 244 and the intra prediction unit 254) performed by the exemplary video encoder 20 will be described in detail below.

[0200] Partitioning

[0201] The partitioning unit 262 can be configured to partition a picture in a video sequence into a series of coding tree units (CTUs), which can be partitioned (or divided) into smaller blocks, e.g., smaller blocks of square or rectangular size, by the coding tree unit (CTU) 203. For a picture with three sample arrays, a CTU consists of one NxN block of luma samples and two corresponding blocks of chroma samples. In the developing Versatile Video Coding (VVC) standard, the maximum allowed size of the luma block in a CTU is specified as 128x128, but can be specified as a value other than 128x128 in the future, e.g., 256x256. The CTUs of a picture can be clustered / grouped into slices / tile groups, tiles, or bricks. A tile covers a rectangular region of a picture and can be divided into one or more bricks. A brick consists of multiple CTU rows in a tile. A tile that is not partitioned into multiple bricks can be referred to as a brick. However, a brick is a proper subset of a tile, and thus cannot be referred to as a tile. VVC supports two tile group modes, i.e., a raster-scan slice / tile group mode and a rectangular slice mode. In the raster-scan tile group mode, a slice / tile group contains a sequence of tiles in a raster-scan of the tiles of a picture. In the rectangular slice mode, a slice contains multiple bricks of a picture that together form a rectangular region of the picture. The bricks within a rectangular slice are arranged in the order of brick raster-scan of the slice. These smaller blocks (which can also be referred to as sub-blocks) can be further partitioned into even smaller partition blocks. This is also referred to as tree partitioning or hierarchical tree partitioning, in which, for example, a root block at root tree level 0 (hierarchical level 0, depth 0) can be recursively partitioned into two or more blocks at the next lower level, e.g., nodes at tree level 1 (hierarchical level 1, depth 1). These blocks can be again partitioned into two or more blocks at the next lower level, e.g., blocks at tree level 2 (hierarchical level 2, depth 2), until the partitioning ends (because an end criterion is reached, e.g., a maximum tree depth is reached or a minimum block size is reached). Blocks that are not further partitioned are also referred to as leaf blocks or leaf nodes of the tree. A tree that is partitioned into two partition blocks is referred to as a binary-tree (BT), a tree that is partitioned into three partition blocks is referred to as a ternary-tree (TT), and a tree that is partitioned into four partition blocks is referred to as a quad-tree (QT).

[0202] For example, a coding tree unit (CTU) can be or can include one CTB of luma samples and two corresponding CTBs of chroma samples in a picture having three sample arrays, or can be or can include one CTB of samples in a monochrome picture or a picture coded using three separate color planes and syntax structures used to code the samples. Correspondingly, a coding tree block (CTB) can be an NxN block of samples, where N can be set to a value such that one component is divided into multiple CTBs, which is one way of partitioning. A coding unit (CU) can be or can include one coding block of luma samples and two corresponding coding blocks of chroma samples in a picture having three sample arrays, or can be or can include one coding block of samples in a monochrome picture or a picture coded using three separate color planes and syntax structures used to code the samples. Correspondingly, a coding block (CB) can be an MxN block of samples, where M and N can be set to a value such that one CTB is divided into multiple coding blocks, which is one way of partitioning.

[0203] In an embodiment, for example according to HEVC, a coding tree unit (CTU) can be divided into CUs by a quad-tree structure denoted as coding tree. Whether a picture region is coded by inter (temporal) prediction or intra (spatial) prediction is decided at the leaf CU level. Each leaf CU can be further divided into one, two or four PUs according to the PU partition type. The same prediction process is performed within one PU and related information is sent to the decoder in units of PUs. After the prediction process is performed according to the PU partition type to obtain a residual block, the leaf CU can be partitioned into transform units (TUs) according to other quad-tree structure similar to the coding tree of the CU.

[0204] In some embodiments, for example, according to the latest video coding standard currently under development, called Versatile Video Coding (VVC), a combined quadtree of nested multi-type trees (e.g., binary and ternary trees) is used to partition the segment structure, e.g., for splitting coding tree units. In the coding tree structure within a coding tree unit, a CU can be square or rectangular. For example, a coding tree unit (CTU) is first split by a quadtree structure. Then, a quadtree leaf node can be further split by a multi-type tree structure. There are four types of partitioning in the multi-type tree structure: vertical binary tree split (SPLIT_BT_VER), horizontal binary tree split (SPLIT_BT_HOR), vertical ternary tree split (SPLIT_TT_VER), and horizontal ternary tree split (SPLIT_TT_HOR). A multi-type tree leaf node is called a coding unit (CU). Unless a CU is too large for the maximum transform length, such a segmentation will be used for prediction and transform processing without any further splitting. This means that, in most cases, the CU, PU, and TU have the same block size in the coding block structure of quadtree-nested multi-type tree. This exceptional case occurs when the supported maximum transform length is smaller than the width or height of a CU for a color component. VVC specifies a unique indication mechanism for splitting partitioning information in the quadtree-nested multi-type tree coding tree structure. In this indication mechanism, a coding tree unit (CTU) is first split by a quadtree structure as the root of the quadtree. Then, each quadtree leaf node (when it needs to be split due to being too large) is further split by a multi-type tree structure. In the multi-type tree structure, a first flag (mtt_split_cu_flag) is signaled to indicate whether a node is further split; when a node is further split, a second flag (mtt_split_cu_vertical_flag) is signaled to indicate the split direction, and a third flag (mtt_split_cu_binary_flag) is signaled to indicate whether the split is a binary tree split or a ternary tree split. Based on the values of mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the multi-type tree split mode (MttSplitMode) of a CU can be derived by a decoder according to a pre-defined rule or table. It is noted that for certain designs, e.g., 64x64 luma block and 32x32 chroma pipeline design in VVC hardware decoder, TT split is not allowed when the width or height of a luma coding block is larger than 64. TT split is also not allowed when the width or height of a chroma coding block is larger than 32.Pipelined designs divide the picture into multiple virtual pipeline data units (VPDUs), which are defined as non-overlapping units in the picture. In hardware decoders, consecutive VPDUs are processed simultaneously by multiple pipeline stages. In most pipeline stages, the size of a VPDU is roughly proportional to the size of a buffer, so it is important to keep VPDUs small. In most hardware decoders, the size of a VPDU can be set to the size of a maximum transform block (TB). However, in VVC, ternary tree (TT) and binary tree (BT) splits can increase the size of a VPDU.

[0205] In addition, it is noted that when a portion of a tree node block exceeds the lower or right boundary of the picture, the tree node block is forced to be split until all samples of each coded CU are within the picture boundary.

[0206] For example, an intra sub-partition (ISP) tool can divide an intra-predicted luma block vertically or horizontally into 2 or 4 sub-partition blocks depending on the size of the block.

[0207] In one example, the mode selection unit 260 in video encoder 20 can be configured to perform any combination of the partitioning techniques described herein.

[0208] As described above, video encoder 20 is configured to determine or select a best or optimal prediction mode from a set of (e.g., predetermined) prediction modes. For example, the set of prediction modes can include intra-prediction modes and / or inter-prediction modes, etc.

[0209] Intra prediction

[0210] The set of intra-prediction modes can include 35 different intra-prediction modes, e.g., non-directional modes such as DC (or mean) mode and planar mode, or directional modes as defined in HEVC, or 67 different intra-prediction modes, e.g., non-directional modes such as DC (or mean) mode and planar mode, or directional modes as defined in VVC. For example, several traditional angular intra-prediction modes are replaced adaptively by wide-angle intra-prediction modes for non-square blocks as defined in VVC. For another example, to avoid division operations for DC prediction, only the longer side is used to calculate the mean value for non-square blocks. Also, the intra-prediction result of planar mode can be modified by a position-dependent intra-prediction combination (PDPC) method.

[0211] The intra prediction unit 254 is configured to generate, for a current block of the current picture, an intra predicted block 265 using reconstructed samples of neighboring blocks in the same current picture according to an intra prediction mode out of a set of intra prediction modes.

[0212] The intra prediction unit 254 (or generally the mode selection unit 260) is further configured to output the intra prediction parameters (or generally information representative of the selected intra prediction mode for the block) in the form of syntax elements 266 to the entropy encoding unit 270 for inclusion into the encoded picture data 21 such that the video decoder 30 can receive and use the prediction parameters for the decoding, for example.

[0213] Inter prediction

[0214] The (possible) set of inter prediction modes depends on the available reference pictures (i.e. at least some of the decoded pictures stored in the DPB 230 as described above, for example) and other inter prediction parameters, e.g. whether the entire reference picture or only a part of the reference picture (e.g. a search window region around the region of the current block) is used for searching the best matching reference block, and / or e.g. whether pixel interpolation is performed or not, e.g. half-pel, quarter-pel and / or 1 / 16-pel interpolation.

[0215] In addition to the above-mentioned prediction modes, a skip mode, a direct mode and / or other inter prediction modes can be applied as well.

[0216] For example, extended merge prediction, the merge candidate list in this mode consists of the following 5 candidate types in order: spatial MVP from spatial neighboring CUs, temporal MVP from collocated CUs, history MVP from FIFO table, pair-wise average MVP, and zero MV. Also, decoder side motion vector refinement (DMVR) method based on bilateral matching can be used to improve the precision of MV in merge mode. Merge mode with MVD (MMVD) is derived from merge mode with motion vector difference. MMVD flag is signaled immediately after the skip flag and the merge flag to indicate whether the CU is coded in MMVD mode, and CU-level adaptive motion vector resolution (AMVR) scheme can be applied. AMVR scheme supports coding MVD of a CU at different precisions. According to the prediction mode of the current CU, the MVD of the current CU can be adaptively selected. When the CU is coded in merge mode, combined inter / intra prediction (CIIP) mode can be applied to the current CU. The CIIP prediction is obtained by weighted average of the inter prediction signal and the intra prediction signal. In affine motion compensation prediction, the affine motion field of a block is described by motion information of two control point motion vectors (4 parameters) or three control point motion vectors (6 parameters). Subblock-based temporal motion vector prediction (SbTMVP) is similar to temporal motion vector prediction (TMVP) in HEVC, but it predicts the motion vector of a sub-CU in the current CU. Bi-directional optical flow (BDOF), formerly known as BIO, is a simplified version with significantly reduced computation, especially in terms of number of multiplications and size of the multiplier. Triangular partition mode: in this mode, a CU is uniformly partitioned into two triangular partition blocks by diagonal partition and anti-diagonal partition. In addition, bi-prediction mode is extended on the basis of simple average to support weighted average of two prediction signals.

[0217] The inter prediction unit 244 can include a motion estimation (ME) unit and a motion compensation (MC) unit (both not shown in FIG. 2B), which can be located in the video encoder 20 and the video decoder 30, respectively, or included in separate encoders and decoders. The inter prediction unit 244 can be configured to generate motion vectors for a block of samples based on a search for similar samples in another image of the video data. FIG. 3The motion estimation unit can be configured to receive or obtain the image block 203 (current image block 203 of the current image 17) and the decoded image 231, or at least one or more previously reconstructed blocks (e.g. reconstructed blocks of one or more other / different previously decoded images 231 ) for motion estimation. For example, the video sequence can comprise the current image and the previously decoded image 231, in other words, the current image and the previously decoded image 231 can be part of or constitute a series of images, which constitute the video sequence.

[0218] For example, the encoder 20 can be configured to select one reference block from a plurality of reference blocks of the same or different images of the plurality of other images and provide the reference image (or reference image index) and / or an offset (spatial offset) between the position of the reference block and the position of the current block (x-coordinate, y-coordinate) as inter prediction parameters to the motion estimation unit. Such an offset is also referred to as motion vector (MV).

[0219] The motion compensation unit is configured to obtain (e.g. receive) the inter prediction parameters and perform inter prediction according to or using the inter prediction parameters, resulting in the inter predicted block 265. The motion compensation performed by the motion compensation unit can comprise obtaining or generating the prediction block according to the motion / block vectors determined by the motion estimation and can further comprise performing interpolation on sub-pixel accuracy. Interpolation filtering can generate further pixel samples from known pixel samples, thereby possibly increasing the number of candidate prediction blocks that can be used for coding an image block. Upon receiving the motion vector corresponding to the PU of the current image block, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0220] The motion compensation unit can further generate syntax elements related to the blocks and the video slice for use by the video decoder 30 when decoding the image blocks of the video slice. In addition or as an alternative to the slice and the corresponding syntax elements, tile groups and / or tiles and corresponding syntax elements can be generated or used.

[0221] Entropy encoding

[0222] Entropy encoding unit 270 is configured to apply or not apply (no compression) an entropy encoding algorithm or scheme, such as a variable length coding (VLC) scheme, a context adaptive VLC (CAVLC) scheme, an arithmetic coding scheme, binarization, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy encoding methods or techniques, to quantized coefficients 209, inter prediction parameters, intra prediction parameters, loop filter parameters, and / or other syntax elements, to obtain encoded image data 21 that can be output via output terminal 272 in the form of an encoded bitstream 21, such that video decoder 30 can receive and use these parameters for decoding, etc. Encoded bitstream 21 can be transmitted to video decoder 30, or stored in a memory for later transmission or retrieval by video decoder 30.

[0223] Other structural variations of video encoder 20 can be used to encode a video stream. For example, a non-transform-based encoder 20 can directly quantize the residual signal of certain blocks or frames without transform processing unit 206. In another implementation, encoder 20 can include quantization unit 208 and inverse quantization unit 210 combined into a single unit.

[0224] Decoder and decoding method

[0225] FIG. 3 An example of a video decoder 30 for implementing the techniques of this application is shown. Video decoder 30 is configured to receive encoded image data 21 (e.g., an encoded bitstream 21) encoded, for example, by encoder 20, to obtain decoded images 331. The encoded image data or bitstream includes information for decoding the encoded image data, such as data representing image blocks of encoded video slices (and / or tile groups or tiles) and related syntax elements.

[0226] In FIG. 2In the example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded picture buffer (DPB) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. The inter-frame prediction unit 344 may be or may include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same functions as the reference unit. FIG. 3 The video encoder 20 in the text describes the encoding process as the inverse of the decoding process.

[0227] As described with reference to encoder 20, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 344, and intra-frame prediction unit 354 also constitute the "built-in decoder" of video encoder 20. Accordingly, the function of inverse quantization unit 310 can be the same as that of inverse quantization unit 210, the function of inverse transform processing unit 312 can be the same as that of inverse transform processing unit 212, the function of reconstruction unit 314 can be the same as that of reconstruction unit 214, the function of loop filter 320 can be the same as that of loop filter 220, and the function of decoded picture buffer 330 can be the same as that of decoded picture buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 is correspondingly applicable to the corresponding units and functions of video decoder 30.

[0228] Entropy Decoding

[0229] The entropy decoding unit 304 is used to parse the bitstream 21 (or commonly referred to as encoded image data 21) and perform entropy decoding on the encoded image data 21 to obtain quantization coefficients 309 and / or decoded encoding parameters. FIG. 3 (Not shown in the image), such as inter-frame prediction parameters (e.g., reference image index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and / or any one or all of other syntax elements. The entropy decoding unit 304 can be used to apply a decoding algorithm or scheme corresponding to the encoding scheme described by the entropy coding unit 270 in the reference encoder 20. The entropy decoding unit 304 can also be used to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 can receive video strip-level and / or video block-level syntax elements. In addition to stripes and corresponding syntax elements, or as a substitute for stripes and corresponding syntax elements, it can also receive and / or use chunk groups and / or chunks and corresponding syntax elements.

[0230] Dequantization

[0231] The dequantization unit 310 can be configured to receive quantization parameters (QPs) (or generally information related to dequantization) and quantized coefficients from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304, etc.), and dequantize the decoded quantized coefficients 309 according to these quantization parameters to obtain dequantized coefficients 311, which can also be referred to as transform coefficients 311. The dequantization process can include determining a degree of quantization using the quantization parameters determined by the video encoder 20 for each video block in a video slice (or tile or tile group), and likewise determining a degree of dequantization to apply.

[0232] Inverse transform

[0233] The inverse transform processing unit 312 can be configured to receive the dequantized coefficients 311 (also referred to as transform coefficients 311), and apply a transform to the dequantized coefficients 311 to obtain a reconstructed residual block 313 in the sample domain. The reconstructed residual block 313 can also be referred to as a transform block 313. The transform can be an inverse transform, such as an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 can also be configured to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304, etc.) to determine the transform to be applied to the dequantized coefficients 311.

[0234] Reconstruction

[0235] The reconstruction unit 314 (e.g., summer or adder 314) can be configured to add the sample values of the reconstructed residual block 313 to the sample values of the prediction block 365, e.g., by adding the reconstructed residual block 313 to the prediction block 365, to obtain a reconstructed block 315 in the sample domain.

[0236] Filtering

[0237] A loop filter unit 320 (in-loop or post-loop) can be used to filter the reconstructed block 315 to produce a filtered block 321, e.g., to smooth pixel transitions or otherwise improve video quality, etc. The loop filter unit 320 can include one or more loop filters, such as a deblocking filter, a sample adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, the loop filter unit 220 can include a deblocking filter, a SAO filter, and an ALF filter. The filtering processes can be performed in the order of the deblocking filter, the SAO filter, and the ALF filter. For another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking filtering. For yet another example, the deblocking filter process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 320 is shown as an in-loop filter in FIG. 3, in other configurations, the loop filter unit 320 can be implemented as a post-loop filter. FIG. 3

[0238] decoded picture buffer

[0239] The decoded video blocks 321 of the picture are then stored in a decoded picture buffer 330, where the decoded picture buffer 330 stores the decoded picture 331 as a reference picture for subsequent motion compensation of other pictures and / or for output and display.

[0240] The decoder 30 is configured to output the decoded picture 311 via an output terminal 312, etc., for presentation to or viewing by a user.

[0241] prediction

[0242] ​The functions of the inter prediction unit 344 can be the same as those of the inter prediction unit 244 (in particular the motion compensation unit), and the functions of the intra prediction unit 354 can be the same as those of the intra prediction unit 254, and perform partitioning or segmentation decisions and predictions according to the partitioning and / or prediction parameters or respective information received from the coded picture data 21 (e.g. parsed and / or decoded by the entropy decoding unit 304 or the like). The mode application unit 360 can be configured to perform a prediction (intra prediction or inter prediction) for each block according to the reconstructed picture, block or respective samples (filtered or unfiltered) resulting in predicted blocks 365.

[0243] When a video slice is coded as an intra coded (I) slice, the intra prediction unit 354 in the mode application unit 360 is used to generate the predicted blocks 365 for the blocks of the picture of the current video slice from the signaled intra prediction mode and data of previously decoded blocks from the current picture. When a video picture is coded as an inter coded (e.g. B or P) slice, the inter prediction unit 344 (e.g. motion compensation unit) of the mode application unit 360 is used to generate the predicted blocks 365 for the blocks of the video slice of the current video picture from the motion vectors and other syntax elements received from the entropy decoding unit 304. For inter prediction, these predicted blocks can be generated from one of the reference pictures within one of the reference picture lists. The video decoder 30 can construct the reference picture lists 0 and 1 using default construction techniques from the reference pictures stored in the DPB 330. The same or similar processes can be applied in addition to or instead of slices (e.g. video slices) to embodiments using tile groups (e.g. video tile groups) and / or tiles (e.g. video tiles), e.g. a video can be coded using I, P or B tile groups and / or tiles.

[0244] The mode application unit 360 is configured to determine prediction information for video blocks of a current video slice by parsing motion vectors or related information and other syntax elements, and to use the prediction information to generate prediction blocks for the current video blocks being decoded. For example, the mode application unit 360 uses some of the syntax elements received to determine a prediction mode (e.g., intra prediction or inter prediction) used to code the video blocks of a video slice, an inter prediction slice type (e.g., B slice, P slice, or GPB slice), construction information for one or more lists of reference pictures used for the slice, motion vectors for each inter coded video block of the slice, inter prediction status for each inter coded video block of the slice, and other information to decode the video blocks in the current video slice. In addition to or instead of slices (e.g., video slices), the same or similar processes can be applied to or by embodiments that use tile groups (e.g., video tile groups) and / or tiles (e.g., video tiles), e.g., a video can be coded using I, P, or B tile groups and / or tiles.

[0245] In some embodiments, FIG. 3 The illustrated video decoder 30 can be configured to partition and / or decode an image using slices (also referred to as video slices), where an image can be partitioned or decoded using one or more slices (typically non-overlapping slices), and each slice can include one or more blocks (e.g., CTUs), one or more tile groups (e.g., tiles (H.265 / HEVC and VVC) or bricks (VVC)).

[0246] In some embodiments, FIG. 4 The illustrated video decoder 30 can be configured to partition and / or decode an image using slices / tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles), where an image can be partitioned or decoded using one or more slices / tile groups (typically non-overlapping slices / tile groups), each slice / tile group can include one or more blocks (e.g., CTUs) or one or more tiles, etc., each tile can be, e.g., rectangular, and can include one or more blocks (e.g., CTUs), e.g., full blocks or partial blocks.

[0247] Other variations of the video decoder 30 can be configured to decode the encoded image data 21. For example, the decoder 30 can be capable of producing an output video stream without the loop filter unit 320. For example, a non-transform-based decoder 30 can directly dequantize residual signals for certain blocks or frames without the inverse transform processing unit 312. In another implementation, the video decoder 30 can include the inverse quantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0248] It should be understood that in the encoder 20 and the decoder 30, the processing result of the current step can be further processed and then output to the next step. For example, after the interpolation filtering, the motion vector derivation or the loop filtering, the processing result of the interpolation filtering, the motion vector derivation or the loop filtering can be further operated, for example, the clip or shift operation.

[0249] It should be noted that the derived motion vector of the current block (including but not limited to the control point motion vector in the affine mode, the sub-block motion vector in the affine mode, the planar mode, the ATMVP mode, the temporal motion vector, etc.) can be further operated. For example, the value of the motion vector is limited in a predefined range according to the representation bit of the motion vector. If the representation bit of the motion vector is bitDepth, the value range of the motion vector is -2^(bitDepth-1) to 2^(bitDepth-1)-1, wherein the "^" symbol represents the power. For example, if the bitDepth is set to be equal to 16, the range is -32768~32767; if the bitDepth is set to be equal to 18, the range is -131072~131071. For example, the value of the derived motion vector (for example, the MV of four 4x4 sub-blocks in one 8x8 block) is limited so that the maximum difference between the integer parts of the MV of the four 4x4 sub-blocks does not exceed N pixels, for example, does not exceed 1 pixel. Here, two methods of limiting the motion vector according to the bitDepth are provided.

[0250] FIG. 1A A schematic diagram of a video coding device 400 is provided for embodiments of the present application. The video coding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video coding device 400 can be a decoder (e.g., the video decoder 30 in FIG. 1) or an encoder (e.g., the video encoder 20 in FIG. 1). FIG. 1A FIG. 5

[0251] ​​The video coding device 400 includes an ingress port 410 (or input port 410) and a receiving unit (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430 for processing data, a transmitting unit (Tx) 440 and an egress port 450 (or output port 450) for transmitting data, and a memory 460 for storing data. The video coding device 400 can further include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled with the ingress port 410, the receiving unit 420, the transmitting unit 440, and the egress port 450, serving as an outlet or an inlet for optical signals or electrical signals.

[0252] The processor 430 can be implemented by hardware and software. The processor 430 can be implemented as one or more CPU chips, one or more cores (e.g., a multi-core processor), one or more FPGAs, one or more ASICs, and one or more DSPs. The processor 430 is in communication with the ingress port 410, the receiving unit 420, the transmitting unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470. The coding module 470 implements the disclosed embodiments described above. For example, the coding module 470 performs, processes, prepares, or provides various coding operations. Thus, inclusion of the coding module 470 can substantially improve the function of the video coding device 400 and affect the transition of the video coding device 400 to different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.

[0253] The memory 460 can include one or more disks, one or more tape drives, and one or more solid-state drives, and can serve as an overflow data storage device to store programs selected for execution and to store instructions and data read during execution of the programs. For example, the memory 460 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0254] FIG. 1A A simplified block diagram of an apparatus 500 is provided for exemplary embodiments, where the apparatus 500 can be used as FIG. 7either or both of the source device 12 and the destination device 14 in FIG. 1.

[0255] The processor 502 in the apparatus 500 can be a central processing unit. Alternatively, the processor 502 can be any other suitable type of device, or multiple devices, capable of manipulating or processing information now-existing or hereafter developed. Although a single processor 502 as shown can be used to implement the disclosed implementations, speed and efficiency can be increased by using multiple processors.

[0256] In one implementation, the memory 504 in the apparatus 500 can be a read only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device can be used as the memory 504. The memory 504 can include code and data 506 that is accessed by the processor 502 via the bus 512. The memory 504 can also include an operating system 508 and an application program 510 including at least one program that, when executed by the processor 502, performs the methods described herein. For example, the application program 510 can include applications 1 through N, and can also include a video coding application that performs the methods described herein.

[0257] The apparatus 500 can also include one or more output devices, such as a display 518. In one example, the display 518 can be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch inputs. The display 518 can be coupled to the processor 502 via the bus 512.

[0258] Although the bus 512 in the apparatus 500 is described herein as a single bus, the bus 512 can include multiple buses. In addition, the secondary storage 514 can be directly coupled to the other components in the apparatus 500 or can be accessed via a network, and can include a single integrated unit such as one memory card or multiple units such as multiple memory cards. Thus, the apparatus 500 can be implemented in a variety of configurations. 1BACKGROUND

[0260] Background information for the present application is presented below.

[0261] 1.1Affine motion compensated prediction

[0262] In ITU-T H.265, only translational motion model is applied for motion compensation prediction (MCP). In the real world, there are many kinds of motions, such as zoom-in / zoom-out, rotation, perspective motion, and other irregular motions. In EVC, block-based affine transform motion compensation prediction is applied. As shown in FIG. 6, the affine motion field of a block is described by the motion information of two (4-parameter) or three (6-parameter) control point motion vectors (CPMV).

[0263] 1.1.1 General equations for motion vector derivation

[0264] The general equation for calculating the motion vector at sample position (x, y) is:

[0265]

[0266] For the 4-parameter affine motion model, the motion vector at sample position (x, y) is derived as follows:

[0267]

[0268] For the 6-parameter affine motion model, the motion vector at sample position (x, y) is derived as follows:

[0269]

[0270] where (mv 0x , mv 0y ) is the motion vector of the top-left control point, (mv 1x , mv 1y ) is the motion vector of the top-right control point, and (mv 2x , mv 2y ) is the motion vector of the bottom-left control point.

[0271] In the case of using the 6-parameter affine motion model,

[0272]

[0273]

[0274]

[0275]

[0276] In the case of using the 4-parameter affine motion model,

[0277]

[0278]

[0279] dVerX = -dHorY, (1-10)

[0280] dVerY = dHorX (1-11)

[0281] As for translational motion inter prediction, there are also two affine motion inter prediction modes, namely affine merge mode and affine AMVP mode.

[0282] 1.1.2 Memory bandwidth calculation

[0283] The memory bandwidth calculation is the area ratio of the reference block to the current block. For example, for an 8x8 bi-predicted block, the area value of the reference block Sr is equal to 2(8+T-1)(8+T-1) in case an interpolation filter with T taps is used, and the area of the current block Sb is equal to 8*8. Thus, the memory bandwidth is For the 8-tap DCT IF used in ITU-T H.265, VVC and EVC,

[0284] 1.2 Block-based affine transform prediction

[0285] To simplify the motion-compensated prediction, a block-based affine transform prediction is applied. For example, to derive the motion vector for each 8x8 luma sub-block, the motion vector of the center sample of each sub-block (as shown in FIG. 8 ) is calculated according to the above equation and the resulting value is rounded to have 1 / 16 fractional precision. Then, a motion-compensated interpolation filter is applied to generate the predicted samples of each sub-block by the derived motion vector. The sub-block size for chroma components is set to 4x4.

[0286] 1.2.1 Block-based affine transform prediction for 8x8 sub-blocks

[0287] Sub-block affine motion compensation with minimum sub-block size 8x8 is more hardware friendly than affine motion compensation with minimum sub-block size 4x4. There are at least three reasons for this.

[0288] 1. Memory bandwidth: Using the minimum sub-block size 8x8 for affine motion compensation does not increase the memory bandwidth compared to ITU-T H.265, because in ITU-T H.265, the 8x8 bi-prediction block corresponds to the highest requirement case in memory bandwidth calculation. In EVC, the 8x8 bi-prediction block also does not change the highest requirement case in memory bandwidth (in EVC 3.0, the 8x4 / 4x8 bi-prediction block corresponds to the highest requirement case, in EVC 4.0, the 4x16 / 16x4 bi-prediction block corresponds to the highest requirement case). Basically, in EVC and ITU-T H.265, the 8x8 bi-prediction block can appear in regular inter prediction, so this sub-block affine motion compensation with the minimum block size does not increase the complexity of motion compensation.

[0289] 2. Multiplication number: The motion compensation of 8x8 sub-blocks requires much fewer multiplications than the motion compensation of 4 4x4 sub-blocks.

[0290] 3. Memory access: In some hardware implementations, no less than 16 samples can be read. From this perspective, in the case of 8-tap DCTIF using (8+8-1)*(8+8-1) reference samples, the 8x8 block utilizes the memory much more efficiently than the 4x4 block.

[0291] However, the performance of sub-block affine motion compensation with the minimum sub-block size 8x8 decreases more significantly than that of sub-block affine motion compensation with the minimum sub-block size 4x4, especially for fast-rotating content. Therefore, for these contents, EIF can be used.

[0292] 1.3 Enhanced bi-linear interpolation filter

[0293] The enhanced bi-linear interpolation filter (EIF) can be used for the prediction block and on a sub-block basis. The filtering process for the luma signal and the chroma signal is the same, which includes the following steps:

[0294] 1. Derive the pixel-based motion vector field from the CPMV according to equation (1-1).

[0295] 2. Obtain the interpolated samples from the derived motion vectors by using bi-linear interpolation for the fractional offsets.

[0296] 3. Use the fixed 3-tap high-pass filter [-1, 10, -1] and the normalization factor is 8, first horizontal filtering, then vertical filtering.

[0297] The first two steps are performed on a (w+2)x(h+2) region, where w and h denote the width and height of the prediction block, respectively; in order to apply the 3-tap filter in the third step, a pixel margin is added at each boundary. The original affine block and the corresponding (w+2)x(h+2) block used in the intermediate step of the EIF are illustrated in mvWx

[0298] An exemplary implementation of the EIF is described below.

[0299] Interpolation process of the enhanced interpolation filter

[0300] The inputs of this process include:

[0301] - the position (xCb, yCb) in full samples;

[0302] - two variables cbWidth and cbHeight denoting the width and height of the current coding block;

[0303] - the horizontal change of the motion vector dX;

[0304] - the vertical change of the motion vector dY;

[0305] - the motion vector mvBaseScaled;

[0306] - the selected reference picture sample array refPicLX;

[0307] - the sample bit depth bitDepth;

[0308] - the width of the picture pic_width in samples;

[0309] - the height of the picture pic_height in samples;

[0310] - the flag clipMV denoting the MV clipping type;

[0311] - the flag isLuma denoting whether the luma or the chroma is processed.

[0312] The outputs of this process include:

[0313] - a (cbWidth / SubWidthC)x(cbHeight / SubHeightC) array of prediction sample values predSamplesLX.

[0314] The interpolation filter coefficients T[p] for each fractional sample position p (equal to xFrac or yFrac) are as in Tables 8-16.

[0315] ​The variables hor max, ver max, hor min and ver min are derived by invoking the process defined in section 8.5.4.5, with as inputs of this process the position (xCb, yCb) in lull samples, two variables cbWidth and cbHeight representing the width and height of the current coding block, the horizontal change of the motion vector dX, the vertical change of the motion vector dY, the motion vector mvBaseScaled, the picture width in samples pic width, the picture height in samples pic height and the flag clipMV, and as outputs of this process hor max, ver max, hor min and ver min.

[0316] If isLuma is equal to FALSE (false), the variables mvBaseScaled, hor min, hor max, ver min, ver max are modified as follows:

[0317] - xCb = xCb / SubWidthC (8-748)

[0318] - yCb = yCb / SubHeightC (8-749)

[0319] - cbWidth = cbWidth / SubWidthC (8-750)

[0320] - cbHeight = cbHeight / SubHeightC (8-751)

[0321] - mvBaseScaled[0] = mvBaseScaled[0] / SubWidthC (8-752)

[0322] - mvBaseScaled[1] = mvBaseScaled[1] / SubHeightC (8-753)

[0323] - hor min = hor min / SubWidthC (8-754)

[0324] - hor max = hor max / SubWidthC (8-755)

[0325] - ver min = ver min / SubHeightC (8-756)

[0326] - ver max = ver max / SubHeightC (8-757)

[0327] The variables shiftO, shiftl, offsetO and offsetl are derived as follows:

[0328] shiftO is set equal to bitDepth - 8, and offsetO is equal to 0;

[0329] shiftl is set equal to 12 - shiftO, and offsetl is equal to 2 shift1-1 .

[0330] When x = -1..cbWidth, y = -1..cbHeight, the following applies:

[0331] - The motion vector mvX is derived as follows:

[0332] mvX[0] = (mvBaseScaled[0] + dX[0] * x + dY[0] * y) (8-758)

[0333] mvX[1] = (mvBaseScaled[1] + dX[1] * x + dY[1] * y) (8-759)

[0334] mvX[0] = Clip3(hor_min, hor_max, mvX[0]) / / Clipped horizontal component of the motion vector (8-760)

[0335] mvX[1] = Clip3(ver_min, ver_max, mvX[1]) / / Clipped vertical component of the motion vector (8-761)

[0336] - The variables xInt, yInt, xFrac and yFrac are derived as follows:

[0337] xInt = xCb + (mvX[0] » 9) + x (8-762)

[0338] yInt = yCb + (mvX[1] » 9) + y (8-763)

[0339] xFrac = (mvX[0] » 4) & 31 (8-764)

[0340] yFrac = (mvX[1] » 4) & 31 (8-765)

[0341] - The variables A and B are derived as follows:

[0342] A = ( refPicLX[ xInt ][ yInt ] * T[ xFrac ][ 0 ] + + refPicLX[ xInt + 1 ][ yInt ] * T[ xFrac ][ 1 ] + offset0 ) » shift0 (8-766)

[0343] B = ( refPicLX[ xInt ][ yInt + 1 ] * T[ xFrac ][ 0 ] + + refPicLX[ xInt + 1 ][ yInt + 1 ] * T[ xFrac ][ 1 ] + offset0 ) » shift0 (8-767)

[0344] - the sample value b corresponding to the position (x, y) x,y is derived as follows:

[0345] b x,y = ( A * T[ yFrac ][ 0 ] + B * T[ yFrac ][ 1 ] + offset1 ) » shift1 (8-768)

[0346] The enhanced interpolation filter coefficients eF[] are denoted as {-1, 10, -1}.

[0347] The variables shift2, shift3, offset2 and offset3 are derived as follows:

[0348] shift2 is set equal to max( bit_depth - 11, 0 ), offset2 is equal to 2 shift2-1 ;

[0349] shift3 is set to ( 6 - max( bit_depth - 11, 0 ) ), offset3 is equal to 2 shift3-1 .

[0350] When x = 0..cbWidth - 1, y = -1..cbHeight, the following applies:

[0351] - h x,y = ( eF[ 0 ] * b x-1,y + eF[ 1 ] * b x,y + eF[ 2 ] * b x+1,y + offset2 ) » shift2 (8-769)

[0352] When x = 0..cbWidth - 1, y = 0..cbHeight - 1, the following applies:

[0353] - predSamplesLX L[x][y] = Clip3(0, (1 « bitDepth) - 1, (eF[0] * h x,y-1 + eF[1] * h x,y + eF[2] * b x,y+1 + offset3) » shift3) (8-770)

[0354] Table 8-16: Specification of the interpolation filter coefficients T[p] for each fractional sample position p

[0355]

[0356]

[0357] 1.4 Adaptive use of EIF and block-based affine motion compensation

[0358] EIF has less number of multiplications than sub-block affine motion compensation with minimum sub-block size of 8x8 and 4x4. However, without the affine motion model restriction, the memory bandwidth of EIF can be huge. In addition, to efficiently implement the hardware of EIF, there can be some additional requirements. For example, from the hardware point of view, there can be the following requirements for EIF.

[0359] A The internal buffer is limited to N lines, where N can be, for example, equal to 3, 4, 5, or a larger value. This means that during the processing of one line / row of the current block (sub-block), at most N lines in the reference picture can be used.

[0360] B The memory access needs to be sequential, which means if the jthline of the reference picture is fetched to get the ithline of the current block, only the (j+1)thline, the (j+2)thline, etc. are fetched to get the (i+1)thline of the current block.

[0361] C At most one additional line can be fetched for all lines of the current block (except the first line).

[0362] To combine the advantages of 8x8 sub-block based affine motion compensation and EIF, the following provides an adaptive use scheme of EIF and sub-block affine motion compensation.

[0363] 1.4.1 Basic algorithm for adaptive use of EIF and block-based affine motion compensation

[0364] Algorithm 1: Adaptive use of EIF and block-based affine motion compensation (basic algorithm)

[0365] The basic algorithm for adaptive use of EIF and sub-block affine motion compensation is described as follows.

[0366] 1. Compute the optimal sub-block size MxN according to the affine motion model parameters.

[0367] 2. If the optimal sub-block width M and the optimal sub-block height N are both greater than or equal to 8, perform sub-block motion compensation of MxN sub-blocks.

[0368] 3. Otherwise, check EIF applicability conditions:

[0369] 3.1. EIF applicability condition 1

[0370] 3.2. EIF applicability condition 2

[0371] 3.3....

[0372] 3.4. EIF applicability condition P

[0373] 3.5. If all EIF applicability conditions are met, perform EIF motion compensation. The EIF motion compensation process includes:

[0374] 3.5.1. Check EIF applicability condition P+1;

[0375] 3.5.2. Check EIF applicability condition P+2;

[0376] 3.5.3....

[0377] 3.5.4. Check EIF applicability condition P+K.

[0378] 3.5.5. If one of the EIF applicability conditions P+1 to P+K is not met, then

[0379] 3.5.5.1. Compute a first range of motion vectors that satisfy the said not met condition;

[0380] 3.5.5.2. Set the second range of motion vectors to the first range of motion vectors.

[0381] 3.5.6. Otherwise,

[0382] 3.5.6.1. Compute a third range of motion vectors;

[0383] 3.5.6.2. Set the second range of motion vectors to the third range of motion vectors.

[0384] 3.5.7. Clip the motion vectors computed according to the affine model to guarantee that these vectors fall within the second range of motion vectors.

[0385] 3.6. Otherwise, set M = max(M, 8), N = max(N, 8), and perform sub-block motion compensation of MxN sub-blocks.

[0386] In some examples, P and K can be equal to 0, indicating that if P is equal to 0, the above algorithm does not include steps 3.1 to 3.4; if K is equal to 0, the above algorithm does not include steps 3.5.1 to 3.5.4.

[0387] Details of some steps of this basic algorithm are described as follows.

[0388] 1.4.1.1 Step 1: Calculate the optimal sub-block size.

[0389] One method for deriving the affine sub-block size is based on the motion vector difference of the affine control points and the width and height of the affine block. The sub-block size MxN can be derived according to equation (2-1), where MvPre represents the motion vector precision (e.g., ¼-pixel precision in the HEVC standard or 1 / 16-pixel precision in the VVC and EVC standards), and the affine motion model parameters dHorX, dHorY, dVerX, and dVerY are calculated according to equations (1-4) to (1-7) for the 6-parameter model and equations (1-8) to (1-11) for the 4-parameter model.

[0390]

[0391] If necessary, M and N derived in equation (2-1) can be adjusted to be smaller to ensure that w and h can be divided by M and N, respectively.

[0392] Another method is to build a three-dimensional lookup table and then obtain the sub-block size directly from the lookup table according to the motion vector difference, the block size, and the motion vector precision. For example, set M as Table_M[x][y][z], where x is equal to max(abs(v 1x -v 0x ), abs(v 1y -v 0y )), y is equal to the affine block width, and z is equal to the motion vector precision; and set N as Table_N[x][y][z], where x is equal to max(abs(v 2x -v 0x ), abs(v 2y -v 0y )), y is equal to the affine block height, and z is equal to the motion vector precision.

[0393] In some example implementations, step 1 is implemented as follows:

[0394] 1.4.1.1.1 Derivation process of the affine sub-block size

[0395] The inputs of this process include:

[0396] - two variables cbWidth and cbHeight representing the width and height of the luma coding block;

[0397] - the number of control point motion vectors numCpMv;

[0398] - the control point motion vectors cpMvLX[ cpIdx ], with cpIdx = 0.. numCpMv - 1, and X being either 0 or 1 ;

[0399] - the prediction list usage flag predFlagLX, with X being either 0 or 1.

[0400] The output of the process comprises:

[0401] - the size of the luma coding sub-block in horizontal direction sizeSbX and in vertical direction sizeSbY.

[0402] with sizeSbX set equal to cbWidth and sizeSbY set equal to cbHeight.

[0403] When predFlagLX is equal to 1, the following applies, with X being either 0 or 1 :

[0404] - the horizontal change of motion vector dX, the vertical change of motion vector dY and the base motion vector mvBaseScaled are derived by invoking the process defined in subclause 1.4.1.1.2, with the width of the luma coding block cbWidth, the height of the luma coding block cbHeight, the number of control point motion vectors numCpMv and the control point motion vectors cpMvLX[ cpIdx ] with cpIdx = 0.. numCpMv - 1 as inputs to the process.

[0405] - the variables mvWx and mvWy are derived as follows:

[0406] - mvWx = max( abs( dX[ 0 ] ), abs( dX[ 1 ] ) )

[0407] - mvWy = max( abs( dY[ 0 ] ), abs( dY[ 1 ] ) )

[0408] - the variable sizeSbXTemp is derived according to the value of mvWx as specified in Table 8-5.

[0409] - the variable sizeSbYTemp is derived according to the value of mvWy as specified in Table 8-5.

[0410] - the variable sizeSbX is modified as follows:

[0411] - sizeSbX = min( sizeSbX, sizeSbXTemp )

[0412] - The variable sizeSbY is modified as follows:

[0413] - sizeSbY = min( sizeSbY, sizeSbYTemp )

[0414] Table 8-5: sizeSbXTemp specification according to various input values of mvWx

[0415] sizeSbX 0 1 2 3 4 >4 cbWidth mvWx 32 16 8 8 4

[0416] Table 8-5: sizeSbYTemp specification according to various input values of mvWy

[0417] sizeSbY 0 1 2 3 4 >4 cbHeight FIG. 9 32 16 8 8 4

[0418] Where section 1.4.1.1.2 is as follows.

[0419] 1.4.1.1.2 Process to derive affine motion model parameters from control point motion vectors

[0420] The input of this process includes:

[0421] - two variables cbWidth and cbHeight indicating the width and height of the luma coding block;

[0422] - the number of control point motion vectors numCpMv;

[0423] - the control point motion vectors cpMvLX[ cpIdx ], where cpIdx = 0..numCpMv - 1 and X is either 0 or 1.

[0424] The output of this process includes:

[0425] - the horizontal change of motion vector dX;

[0426] - the vertical change of motion vector dY;

[0427] - the motion vector mvBaseScaled corresponding to the top-left corner of the luma coding block.

[0428] The variables log2CbW and log2CbH are derived as follows:

[0429] log2CbW = Log2( cbWidth ) (8-688)

[0430] log2CbH = Log2( cbHeight ) (8-689)

[0431] The horizontal change dX of the motion vector is derived as follows:

[0432] dX[0] = (cpMvLX[1][0] - cpMvLX[0][0]) « (7 - log2CbW) (8-690)

[0433] dX[1] = (cpMvLX[1][1] - cpMvLX[0][1]) « (7 - log2CbW) (8-691)

[0434] The vertical change dY of the motion vector is derived as follows:

[0435] - If numCpMv is equal to 3, dY is derived as follows:

[0436] dY[0] = (cpMvLX[2][0] - cpMvLX[0][0]) « (7 - log2CbH) (8-692)

[0437] dY[1] = (cpMvLX[2][1] - cpMvLX[0][1]) « (7 - log2CbH) (8-693)

[0438] - Else if numCpMv is equal to 2, dY is derived as follows:

[0439] dY[0] = -dX[1] (8-694)

[0440] dY[1] = dX[0] (8-695) The motion vector mvBaseScaled corresponding to the top-left corner of the luma coding block is derived as follows:

[0441] mvBaseScaled[0] = cpMvLX[0][0] « 7 (8-696)

[0442] mvBaseScaled[1] = cpMvLX[0][1] « 7 (8-697)

[0443] 1.4.1.2 Step 3: EIF applicability condition

[0444] For example, the EIF applicability condition can be as follows:

[0445] 1. Memory bandwidth limitation condition: This limitation condition guarantees that the size of the region in the reference picture corresponding to the current affine block (EIF block) does not exceed a pre-defined threshold T. An exemplary definition of the region in the reference picture corresponding to the current affine block is illustrated in FIG. 1 1. FIG. 9

[0446] ​2. The internal buffer restriction is R lines, where R is a pre-defined value, e.g., can be 3, 4, 5, or larger. This means that during the processing of a line / row of the current block (subblock), at most R lines in the reference picture can be used.

[0447] 3. The memory access needs to be sequential, which means if the j-th line of the reference picture is fetched to get the i-th line of the current block, then only the (j+1)-th line, (j+2)-th line, etc. are fetched to get the (i+1)-th line of the current block.

[0448] 4. For all lines of the current block (except the first line), at most one additional line can be fetched.

[0449] 5. For example, the inequalities for the affine motion model parameters are as follows:

[0450] a. where a, b, c, d, e, f, g, and h are pre-defined values or positive / negative infinity.

[0451] b. where a and b are pre-defined values.

[0452] In one example with the specific EIF applicability condition, the basic algorithm is as follows:

[0453] Algorithm 2: Adaptive use of EIF and block-based affine motion compensation with specific EIF applicability condition

[0454] 1. Calculate the optimal subblock size MxN according to the affine motion model parameters.

[0455] 2. If either the optimal subblock width M or the optimal subblock height N is greater than or equal to 8, perform subblock motion compensation for MxN subblocks.

[0456] 3. Otherwise, check the EIF applicability condition:

[0457] 3.1. EIF applicability condition 1: The internal buffer restriction is R lines, where R is a pre-defined value, e.g., can be 3, 4, 5, or larger. This means that during the processing of a line / row of the current block (subblock), at most R lines in the reference picture can be used.

[0458] 3.2. EIF applicability condition 2: The memory access needs to be sequential, which means if the j-th line of the reference picture is fetched to get the i-th line of the current block, then only the (j+1)-th line, (j+2)-th line, etc. are fetched to get the (i+1)-th line of the current block.

[0459] 3.3. EIF applicability condition 3: At most one additional row can be fetched for all rows of the current block (except the first row).

[0460] 3.4. …

[0461] 3.5. EIF applicability condition P

[0462] 3.6. If all EIF applicability conditions are satisfied, perform EIF motion compensation. The EIF motion compensation process includes:

[0463] 3.6.1. Check EIF applicability condition P+1 : Check memory bandwidth limitation condition. This limitation condition guarantees that the size of the region in the reference picture corresponding to the current affine block (EIF block) does not exceed a pre-defined threshold T. An exemplary definition of the region in the reference picture corresponding to the current affine block is shown in FIG. 3. FIG. 8

[0464] 3.6.2. …

[0465] 3.6.3. Check EIF applicability condition P+K.

[0466] 3.6.4. If one of the EIF applicability conditions P+1 to P+K is not satisfied, then

[0467] 3.6.4.1. Calculate the first range of motion vectors that satisfy the unsatisfied condition;

[0468] 3.6.4.2. Set the second range of motion vectors to be the first range of motion vectors.

[0469] 3.6.5. Otherwise,

[0470] 3.6.5.1. Calculate the third range of motion vectors;

[0471] 3.6.5.2. Set the second range of motion vectors to be the third range of motion vectors.

[0472] 3.6.6. Clip the motion vectors calculated according to the affine model to guarantee that these vectors fall within the second range of motion vectors.

[0473] 3.7. Otherwise, set M = max(M, 8), N = max(N, 8), and perform sub-block motion compensation for M x N sub-blocks.

[0474] In one exemplary implementation, steps 3.1 to 3.5 are implemented as follows:

[0475] When predFlagLX is equal to 1, the following applies, where X is 0 or 1:

[0476] ​The horizontal change of the motion vector dX, the vertical change of the motion vector dY and the base motion vector mvBaseScaled are derived by invoking the process defined in section 1.4.1.1.2 with the width of the luma coding block cbWidth, the height of the luma coding block cbHeight, the number of control point motion vectors numCpMv and the control point motion vectors cpMvLX[cpIdx] with cpIdx = 0..numCpMv - 1 as inputs of the process.

[0477] If dY[1] is less than ((-1) « 9), the variable eifCanBeAppliedX is equal to FALSE (false).

[0478] - Otherwise,

[0479] - If (max(0, dY[1]) + Abs(dX[1])) * (1 + eifSubblockSize) is greater than (1 « 9), the variable eifCanBeAppliedX is equal to FALSE (false).

[0480] The variable eifCanBeAppliedX is equal to TRUE (true), which means here that all the EIF applicability conditions 1 to P (steps 3.1 to 3.5 of the algorithm 2) are fulfilled.

[0481] The detailed content of step 3.6.1 of the algorithm 2 is described as follows.

[0482] 1.4.1.2.1 Calculation of the memory access cost of an affine block when EIF is used (step 3.6.1 of the algorithm 2)

[0483] The following steps are performed to calculate the memory access cost:

[0484] 1. Derive the position of each corner sample of the WxH sub-block.

[0485] 2. Derive the position of each corner sample of the sub-block used in step 3 of EIF, which is denoted as the EIF intermediate sub-block.

[0486] 3. Derive the motion vector of each corner sample of the EIF intermediate sub-block.

[0487] 4. Derive the position of the transformed sub-block in the reference picture.

[0488] 5. Derive the size of the bounding box of the transformed sub-block.

[0489] 6. Get the memory access cost from the size of the bounding box of the transformed sub-block size and the filter length. (EIF uses bilinear interpolation, so the filter length is equal to 2).

[0490] Detailed implementation of these steps is described as follows.

[0491] Step 1: Derive the position of each corner sample of the affine sub-block.

[0492] Let (x0, y0) denote the coordinates of the top-left sample of the affine block. In this embodiment, when calculating the memory access cost, it is assumed that the coordinates of the top-left sample of the affine block is equal to (1, 1). When calculating the memory access cost, it is not necessary to use the position (x0, y0), and the following equations will be simpler if (x0, y0) = (1, 1).

[0493] Then, the position of the affine block can be described by the coordinates of the corner samples (top-left, top-right, bottom-left, bottom-right) of the affine block:

[0494]

[0495] Step 2: Derive the position of each corner sample of the EIF intermediate sub-block.

[0496] Since EIF uses a 3-tap filter in Step 3, bilinear interpolation in Step 2 of EIF is performed on a (W+2) x (H+2) sub-block (one pixel margin is added at each boundary). This (W+2) x (H+2) sub-block is denoted as the intermediate EIF sub-block. The coordinates (top-left, top-right, bottom-left, bottom-right) of the corner samples of the intermediate EIF block are:

[0497]

[0498] The coordinates of the corners of the affine sub-block and the intermediate EIF sub-block are shown as FIG. 9

[0499] Step 3: Derive the motion vector of each corner sample of the EIF intermediate sub-block.

[0500] When calculating the memory access cost, it is not necessary to use the initial motion vector (mv 0x , mv 0y ), and the following equations will be simpler if (mv 0x , mv 0y ) = (dHorX + dVerX, dHorY + dVerY).

[0501] Derive the motion vector according to equation (1-1).

[0502]

[0503] Step 4: Derive the position of the transformed block in the reference picture.

[0504] ​The position of the transform block in the reference picture can be described by the coordinates of the corner samples of the transform block (top-left, top-right, bottom-left, bottom-right):

[0505]

[0506] Step 5: Derive the size of the boundary box of the transform sub-block.

[0507] The size of the boundary box of the transform sub-block in the reference picture can be calculated according to the following equation, where the max function returns the maximum of the arguments and the min function returns the minimum of the arguments:

[0508]

[0509] The position and the corresponding boundary box of the transform sub-block in the reference picture are shown as mvWx

[0510] In one example, W' = Ceil(W') and H' = Ceil(H') are performed after equation (3-5).

[0511] In another example, W' = Floor(W') and H' = Floor(H') are performed after equation (3-5).

[0512] Step 6: Get the memory access consumption.

[0513] The memory access consumption of an affine sub-block in one reference picture can be determined by the size of the boundary box of the transform sub-block and the length of the MC interpolation filter used for the affine motion block T', e.g., 2, 4, 6, 8, etc.:

[0514] Mem = (W' + T' - 1) * (H' + T' - 1) (3-6)

[0515] For EIF, the bilinear interpolation is used, thus the filter length is 2, and the memory access consumption is equal to:

[0516] Mem = (W' + 1) * (H' + 1) (3-7)

[0517] 1.4.1.2.2 Affine motion model restriction condition for EIF (Step 3.6.1 of Algorithm 2)

[0518] The target memory bandwidth under the most demanding case is denoted as where W and H denote the width and height of the current sub-block, respectively, and S wc denotes the maximum allowed memory access consumption of the current sub-block according to the target memory bandwidth under the most demanding case. To guarantee that the EIF memory bandwidth is not larger than the target memory bandwidth under the worst case, the memory access consumption of the EIF sub-block should be limited to the following condition:​

[0519]

[0520] or

[0521] (W' + 1) * (H' + 1) ≤ T * W * H (3-8)

[0522] or

[0523] (W' + 1) * (H' + 1) ≤ S wc

[0524] The value T can be predefined at the encoder and decoder side, or specified in the parameter set of the codec video sequence, such as specified in the parameter set at the sequence level, image level, slice level, etc.

[0525] In one example, if the maximum allowed memory access consumption of each sample is defined as the memory access consumption of a 4x4 block, T can be derived as follows, where T represents the length of the interpolation filter:

[0526]

[0527] When T' is equal to 6, the restriction condition is as follows:

[0528] (W' + 1) * (H' + 1) ≤ (4 + 6 - 1) * (4 + 6 - 1) (3-10)

[0529] In another example, if the maximum allowed memory access consumption of each sample is defined as the memory access consumption of an 8x8 block, T can be derived as follows, where T represents the length of the interpolation filter:

[0530]

[0531] In another example, the maximum allowed memory access consumption of each sample can be different according to the different prediction directions of the current block, that is, when the current block is unidirectionally predicted, the threshold T UNI , and when the current block is bidirectionally predicted, the threshold T BI .

[0532] For example, T UNI is defined as the memory access consumption of a 4x4 block, T BI is defined as the memory access consumption of an 8x4 block, then:

[0533]

[0534]

[0535] In another example, T UNITBI is defined as the memory access cost of a 4x4 block, and TBI is defined as the memory access cost of an 8x8 block, then:

[0536]

[0537]

[0538] In the above example, T' is the length of the motion compensation (MC) interpolation filter used for translating the motion block, for example 2, 4, 6, 8, etc.

[0539] T, T UNI and T BI The value of TBI can depend on the width and height of the current block.

[0540] For bi-predictive affine blocks, the above constraints apply to listO and listl, respectively.

[0541] In another example, the memory access cost of listO and listl are calculated as MemO and Meml, and the sum of these elements is limited. For example, if TBI BI TBI is defined as the memory access cost of an 8x8 block, then the limit condition is as follows:

[0542] MemO + Meml ≤ 2 * (8 + T' - 1) * (8 + T' - 1) (3-16)

[0543] If condition (3-8) is not satisfied, the range of motion vectors required to guarantee the bounding box is derived, and the horizontal part of the motion vector and the vertical part of the motion vector are clipped according to the derived range in the process of calculating the MV in EIF according to equation (1-1). The derivation of the range of motion vectors is described in section 1.4.1.2.3.

[0544] Step 3.6.1 of algorithm 2 can be implemented as follows:

[0545] The variable clipMVX is derived as follows, where X is 0 or 1:

[0546] - clipMVX is set equal to FALSE.

[0547] The variable cifSubblockSize is set equal to 4.

[0548] When predFlagLX is equal to 1, the following applies, where X is 0 or 1:

[0549] - The horizontal change of motion vector dX, the vertical change of motion vector dY and the base motion vector mvBaseScaled are derived by invoking the process defined in subclause 1.4.1.1.2, with the width of the luma coding block cbWidth, the height of the luma coding block cbHeight, the number of control point motion vectors numCpMv and the control point motion vectors cpMvLX[ cpIdx ] (where cpIdx = 0..numCpMv - 1) as inputs to the process.

[0550] - The variables mvWx and mvWy are derived as follows:

[0551] - mvWx = max( abs(dX[0]), abs(dX[1]) )

[0552] - mvWy = max( abs(dY[0]), abs(dY[1]) )

[0553] - The variable sizeSbXTemp is derived from the value of mvWx as shown in Table 8-5.

[0554] - The variable sizeSbYTemp is derived from the value of mvWy as shown in Table 8-5.

[0555] - The variable sizeSbX is modified as follows:

[0556] - sizeSbX = min( sizeSbX, sizeSbXTemp )

[0557] - The variable sizeSbY is modified as follows:

[0558] - sizeSbY = min( sizeSbY, sizeSbYTemp )

[0559] Table 8-5: Specification of sizeSbXTemp from various input values of mvWx

[0560] sizeSbX 0 1 2 3 4 >4 cbWidth mvWx 32 16 8 8 4

[0561] Table 8-5: Specification of sizeSbYTemp from various input values of mvWy

[0562] sizeSbY 0 1 2 3 4 >4 cbHeight FIGS. 10-14 32 16 8 8 4

[0563] - The variable clipMVX is modified as follows:

[0564] - The arrays X[i] and Y[i] are derived as follows:

[0565] - X[0] = 0;

[0566] - X[1] = (eifSubblockSize + 1) * (dX[0] + (1 « 9));

[0567] - X[2] = (eifSubblockSize + 1) * dY[0];

[0568] - X[3] = X[1] + X[2];

[0569] - Y[0] = 0;

[0570] - Y[1] = (eifSubblockSize + 1) * dX[1];

[0571] - Y[2] = (eifSubblockSize + 1) * (dY[1] + (1 « 9));

[0572] - Y[3] = Y[1] + Y[2];

[0573] - The variable Xmax is set equal to the maximum of X[i] for i equal to 0..3;

[0574] - The variable Xmin is set equal to the minimum of X[i] for i equal to 0..3;

[0575] - The variable Ymax is set equal to the maximum of Y[i] for i equal to 0..3;

[0576] - The variable Ymin is set equal to the minimum of Y[i] for i equal to 0..3;

[0577] - The variable W is set equal to (Xmax - Xmin + (1 « 9) - 1) » 9;

[0578] - The variable H is set equal to (Ymax - Ymin + (1 « 9) - 1) » 9;

[0579] - If (W + 2) * (H + 2) is greater than 81, the variable clipMVX is set equal to TRUE.

[0580] The variables eifCanBeApplied and clipMV are derived as follows:

[0581] - clipMV = clipMV0 | clipMV1

[0582] The variable clipMV being TRUE here indicates that the memory bandwidth EIF applicability condition is not met and further MV clipping (steps 3.6.4 to 3.6.6 of algorithm 2) is needed.

[0583] The following section describes existing methods for updating the range of motion vectors near the image boundary. An example of applying the existing methods is FIG. 10 The left part (under the heading "Design in the prior art") is described.

[0584] 1.4.1.2.3 Derivation of clipping parameters for affine motion vectors (steps 3.6.4 to 3.6.5 of algorithm 2)

[0585] The inputs of the process comprise:

[0586] - the position (xCb, yCb) in full samples;

[0587] - two variables cbWidth and cbHeight representing the width and height of the current coding block;

[0588] - the horizontal variation of the motion vector dX;

[0589] - the vertical variation of the motion vector dY;

[0590] - the motion vector mvBaseScaled;

[0591] - the width of the picture pic_width in samples;

[0592] - the height of the picture pic_height in samples;

[0593] - the flag clipMV indicating whether or not MV clipping is to be performed (this flag indicates whether or not the EIF memory access condition is met).

[0594] The outputs of the process comprise:

[0595] - hor_max, ver_max, hor_min and ver_min representing the maximum and minimum allowed motion vector horizontal and vertical components.

[0596] The central motion vector mv_center is derived as follows:

[0597] mv_center[0] = (mvBaseScaled[0] + dX[0] * (cbWidth » 1) + dY[0] * (cbHeight » 1)) (8-743)

[0598] mv_center[1] = (mvBaseScaled[1] + dX[1] * (cbWidth » 1) + dY[1] * (cbHeight » 1)) (8-743)

[0599] The integer process of motion vector defined in section 8.5.3.10 is invoked with mv_center, rightShift set equal to 5 and leftShift set equal to 0 as inputs, and the integer motion vector is returned as mv_center.

[0600] The motion vector mv_center is clipped as follows:

[0601] mv_center[0] = Clip3(-2 17 , 2 17 -1, mv_center[0]) (8-686)

[0602] mv_center[1] = Clip3(-2 17 , 2 17 -1, mv_center[1]) (8-686)

[0603] The variables hor_max_pic, ver_max_pic, hor_min_pic and ver_min_pic are derived as follows:

[0604] hor_max_pic = (pic_width + 128 - xCb - cbWidth) « 4 (8-743)

[0605] ver_max_pic = (pic_height + 128 - yCb - cbHeight) « 4 (8-743)

[0606] hor_min_pic = (-128 - xCb) « 4 (8-743)

[0607] ver_min_pic = (-128 - yCb) « 4 (8-743)

[0608] If clipMV is equal to FALSE (false), the output variables hor_max, ver_max, hor_min and ver_min to represent the maximum and minimum allowed motion vector horizontal components and vertical components are derived as follows (this corresponds to step 3.6.5 of algorithm 2):

[0609] hor_max = hor_max_pic « 5 (8-743)

[0610] ver_max = ver_max_pic « 5 (8-743)

[0611] hor_min = hor_min_pic « 5 (8-743)

[0612] ver_min = ver_min_pic « 5 (8-743)

[0613] Otherwise, if clipMV is equal to TRUE (true), the following steps are applied (this corresponds to step 3.6.4 of algorithm 2):

[0614] - The variables mv hor min, mv ver min, mv hor max and mv ver max are derived as follows:

[0615] mv hor min = mv center [0] - deviationMV [log2CbWidth - 3] (8-743)

[0616] mv ver min = mv center [1] - deviationMV [log2CbHeight - 3] (8-743)

[0617] mv hor max = mv center [0] + deviationMV [log2CbWidth - 3] (8-743)

[0618] mv ver max = mv center [1] + deviationMV [log2CbHeight - 3] (8-743)

[0619] The array deviationMV is denoted as deviationMV [k] = {64, 128, 272, 560, 1136} when k = 0..4.

[0620] - The variables hor max, ver max, hor min and ver min are derived as follows:

[0621] mv hor max = max (mv hor max, hor min) (8-743)

[0622] mv ver max = max (mv ver max, ver min) (8-743)

[0623] mv hor min = min (mv hor min, hor max) (8-743)

[0624] mv ver min = min (mv ver min, ver max) (8-743)

[0625] hor_max = min(hor_max_pic, mv_hor_max) « 5 (8-743)

[0626] ver_max = min(ver_max_pic, mv_ver_max) « 5 (8-743)

[0627] hor_min = max(hor_min_pic, mv_hor_min) « 5 (8-743)

[0628] ver_min = max(ver_min_pic, mv_ver_min) « 5 (8-743)

[0629] 2. Problems in the design in the prior art

[0630] The derivation of the MV clipping parameters in the design in the prior art has the following problems:

[0631] 1. If the center motion vector is close to the picture boundary, the motion vector range is significantly reduced.

[0632] For example, if the center motion vector points outside the picture boundary, the motion vector range is significantly reduced, as shown in FIG. 11 , FIG. 14 In some special cases, the motion vector range can even be reduced to a point, as shown in FIG. 12 The reduction of the motion vector range leads to the degradation of the motion model. Therefore, the prediction signal is generated according to the motion field that is damaged by being clipped to a smaller motion vector range, instead of being generated according to the true motion field ("true motion") (e.g., affine motion such as rotation or scaling). This in turn leads to the prediction signal quality degradation, resulting in either the code rate increase (due to more bits required for residual signal coding) or the reconstructed signal quality degradation.

[0633] 3. Detailed description of the present application

[0634] 3.1 Derivation of the motion vector range near the picture boundary

[0635] In the design in the prior art for the MV clipping range derivation for EIF, if the center MV calculated in section 1.4.1.2.3 is close to the picture boundary or outside the picture boundary, the MV range is significantly reduced. For example, in the horizontal direction, if the MV dispersion is greater than the distance from (x0+mv_center[0]) to the left boundary of the picture, the motion vector range will be reduced, as shown in FIG. 13If (x0+mv_center[0]) refers to a position outside the picture boundaries (e.g., x0+mv_center[0]<0 or x0+mv_center[0]<-MAX_CU_SIZE in some implementations), the MV range will be reduced to only one point, as FIG. 12 The same principle applies for the image right boundary and the image left boundary for this picture.

[0636] To avoid reducing the MV range in the particular case described, the following solution is proposed. If the MV range computed according to the center motion vector (mv_center) includes MVs pointing outside the picture boundaries (outside the picture boundaries with a margin of MAX_CU_SIZE in some implementations), the MV range is computed according to the picture boundaries, not according to the center motion vector. FIG. 13 and mvWx An example of using the new design is described.

[0637] An example implementation of computing the MV range (section 1.4.1.2.3) is described below.

[0638] Derivation of the clipping parameters for affine motion vectors (implementation 1)

[0639] The inputs of this process include:

[0640] - a position (xCb, yCb) in full samples;

[0641] - two variables cbWidth and cbHeight representing the width and height of the current coding block;

[0642] - a horizontal change of motion vector dX;

[0643] - a vertical change of motion vector dY;

[0644] - a motion vector mvBaseScaled;

[0645] - a width of the picture pic_width in samples;

[0646] - a height of the picture pic_height in samples;

[0647] - a flag clipMV representing the MV clipping type.

[0648] The outputs of this process include:

[0649] - hor_max, ver_max, hor_min and ver_min representing the maximum and minimum allowed motion vector horizontal and vertical components.

[0650] The variables log2CbW and log2CbH are derived as follows:

[0651] log2CbWidth = Log2(cbWidth) (8-774)

[0652] log2CbHeight = Log2(cbHeight) (8-775)

[0653] The variables hor_max_pic, ver_max_pic, hor_min_pic, and ver_min_pic are derived as follows:

[0654] hor_max_pic = (pic_width + 128 - xCb - cbWidth - 1) « 5 (8-776)

[0655] ver_max_pic = (pic_height + 128 - yCb - cbHeight - 1) « 5 (8-777)

[0656] hor_min_pic = (-128 - xCb) « 5 (8-778)

[0657] ver_min_pic = (-128 - yCb) « 5 (8-779)

[0658] The variables hor_max_pic, ver_max_pic, hor_min_pic, and ver_min_pic characterize a second motion vector range for checking whether a motion vector is pointing inside the “first region of the reference picture”.

[0659] The center motion vector mv_center is derived as follows:

[0660] mv_center[0] = (mvBaseScaled[0] + dX[0] * (cbWidth » 1) + dY[0] * (cbHeight » 1)) (8-780)

[0661] mv_center[1] = (mvBaseScaled[1] + dX[1] * (cbWidth » 1) + dY[1] * (cbHeight » 1)) (8-781)

[0662] The step “obtaining a center motion vector mv_center of the coding block” in the present application corresponds to equations (8-780) and (8-781).

[0663] The integer process of motion vector defined in section 8.5.3.10 is invoked with mv_center, rightShift set equal to 4 and leftShift set equal to 0 as inputs, and the integer motion vector is returned as mv_center.

[0664] If clipMV is equal to FALSE (false), the output variables hor_max, ver_max, hor_min and ver_min for representing the maximum and minimum allowed motion vector horizontal and vertical components are set equal to the variables hor_max_pic, ver_max_pic, hor_min_pic and ver_min_pic, respectively.

[0665] Otherwise, the following steps are applied:

[0666] - The array deviationMV is set equal to {128, 256, 544, 1120, 2272}.

[0667] - The variables mv_hor_min, mv_ver_min, mv_hor_max and mv_ver_max (hereinafter denoted as hor_min,

[0668] ver_min, hor_max and ver_max) are derived as follows:

[0669] hor_min = mv_center[0] - deviationMV[log2CbWidth - 3] (8-788)

[0670] ver_min = mv_center[1] - deviationMV[log2CbHeight - 3] (8-789)

[0671] hor_max = mv_center[0] + deviationMV[log2CbWidth - 3] (8-790)

[0672] ver_max = mv_center[1] + deviationMV[log2CbHeight - 3] (8-791)

[0673] The step "deriving a first motion vector range of the coding block depending on the center motion vector and a motion vector dispersion, wherein the motion vector dispersion depends on a size of the coding block" in the present application corresponds to equations (8-788) to (8-791). The motion vector dispersion is represented by a horizontal motion vector dispersion deviationMV[log2CbWidth-3] and a vertical motion vector dispersion deviationMV[log2CbHeight-3].

[0674] - If hor_min is smaller than hor_min_pic, the variables hor_min and hor_max will be updated as follows:

[0675] hor_min = hor_min_pic (8-792)

[0676] hor_max = min(hor_max_pic, hor_min_pic + 2*deviationMV[log2CbWidth-3]) (8-793)

[0677] - Else if hor_max is greater than hor_max_pic, the variables hor_min and hor_max will be updated as follows:

[0678] hor_min = max(hor_min_pic, hor_max_pic - 2*deviationMV[log2CbWidth-3]) (8-794)

[0679] hor_max = hor_max_pic (8-795)

[0680] - If ver_min is smaller than ver_min_pic, the variables ver_min and ver_max will be updated as follows:

[0681] ver_min = ver_min_pic (8-796)

[0682] ver_max = min(ver_max_pic, ver_min_pic + 2*deviationMV[log2CbHeight-3]) (8-797)

[0683] - Else if ver_max is greater than ver_max_pic, the variables ver_min and ver_max will be updated as follows:

[0684] ver_min = max(ver_min_pic, ver_max_pic - 2*deviationMV[log2CbHeight-3]) (8-798)

[0685] ver_max = ver_max_pic (8-799)

[0686] The step "if the first motion vector range is at least partially pointing outside the first region comprising the reference picture, update the first motion vector range to point inside the first region" in the present application corresponds to equations (8-792) to (8-799) and the corresponding conditions described above.

[0687] The output variables hor_max, ver_max, hor_min and ver_min are clipped as follows:

[0688] hor_max = Clip3(-2 17 , 2 17 -1, hor_max) (8-800)

[0689] ver_max = Clip3(-2 17 , 2 17 -1, ver_max) (8-801)

[0690] hor_min = Clip3(-2 17 , 2 17 -1, hor_min) (8-802)

[0691] ver_min = Clip3(-2 17 , 2 17 -1, ver_min) (8-803)

[0692] The step "perform a clipping operation on the updated first motion vector range so that it is within the range [-2 17 , 2 17 -1]" in the present application corresponds to equations (8-800) to (8-803).

[0693] It is to be noted that the second motion vector range is denoted by hor_max_pic, ver_max_pic, hor_min_pic and ver_min_pic in equations (8-776) to (8-779).

[0694] It is noted that the first motion vector range is denoted by hor_min, ver_min, hor_max and ver_max in equations (8-788) to (8-791).

[0695] It is noted that the updated first motion vector range is denoted by hor_min, ver_min, hor_max and ver_max in equations (8-792) to (8-799).

[0696] It is noted that the variables hor_min, ver_min, hor_max and ver_max on the right side of equations (8-800) to (8-803) denote the first minimum MV horizontal component value, the first minimum MV vertical component value, the first maximum MV horizontal component value and the first maximum MV vertical component value of the updated first motion vector range, respectively.

[0697] It is noted that the variables hor_min, ver_min, hor_max and ver_max on the left side of equations (8-800) to (8-803) denote the first minimum MV horizontal component value, the first minimum MV vertical component value, the first maximum MV horizontal component value and the first maximum MV vertical component value of the updated and clipped first motion vector range, respectively.

[0698] Derivation of clipping parameters for affine motion vectors (implementation 2)

[0699] The inputs of the process comprise:

[0700] - a position (xCb, yCb) in whole samples;

[0701] - two variables cbWidth and cbHeight denoting the width and height of the current coding block;

[0702] - a horizontal change of motion vector dX;

[0703] - a vertical change of motion vector dY;

[0704] - a motion vector mvBaseScaled;

[0705] - a width of the picture pic_width in samples;

[0706] - a height of the picture pic_height in samples;

[0707] - a flag clipMV denoting the MV clipping type.

[0708] The outputs of the process comprise:

[0709] -hor_max, ver_max, hor_min, and ver_min represent the maximum and minimum allowed horizontal and vertical components of the motion vector.

[0710] The derivation of variables log2CbW and log2CbH is as follows:

[0711] log2CbWidth=Log2(cbWidth) (8-774)

[0712] log2CbHeight=Log2(cbHeight) (8-775)

[0713] The derivation of variables hor_max_pic, ver_max_pic, hor_min_pic, and ver_min_pic is as follows:

[0714]

[0715]

[0716]

[0717]

[0718] The derivation of the center motion vector mv_centerr is as follows:

[0719] mv_center[0]=(mvBaseScaled[0]+dX[0]*(cbWidth>>1)+dY[0]*(cbHeight>>1)) (8-780)

[0720] mv_center[1]=(mvBaseScaled[1]+dX[1]*(cbWidth>>1)+dY[1]*(cbHeight>>1)) (8-781)

[0721] The process of rounding the motion vector defined in Section 8.5.3.10 is invoked, where mv_center, rightShift set to 4, and leftShift set to 0 are taken as inputs, and the rounded motion vector is returned as mv_center.

[0722] If clipMV is equal to FALSE, then the output variables hor_max, ver_max, hor_min and ver_min used to represent the maximum and minimum allowed motion vector horizontal and vertical components are set equal to the variables hor_max_pic, ver_max_pic, hor_min_pic and ver_min_pic, respectively.

[0723] Otherwise, the following steps are applied:

[0724] - The array deviationMV is set equal to {128, 256, 544, 1120, 2272}.

[0725] - The variables mv_hor_min, mv_ver_min, mv_hor_max and mv_ver_max (denoted hereafter as hor_min,

[0726] ver_min, hor_max and ver_max) are derived as follows:

[0727] hor_min = mv_center[0] - deviationMV[log2CbWidth-3] (8-788)

[0728] ver_min = mv_center[1] - deviationMV[log2CbHeight-3] (8-789)

[0729] hor_max = mv_center[0] + deviationMV[log2CbWidth-3] (8-790)

[0730] ver_max = mv_center[1] + deviationMV[log2CbHeight-3] (8-791)

[0731] - If hor_min is less than hor_min_pic, then the variables hor_min and hor_max will be updated as follows:

[0732] hor_min = hor_min_pic (8-788)

[0733] hor_max = min(hor_max_pic, hor_min_pic + 2*deviationMV[log2CbWidth-3]) (8-790)

[0734] - Otherwise, if hor_max is greater than hor_max_pic, the variables hor_min and hor_max are updated as follows:

[0735] hor_min = max(hor_min_pic, hor_max_pic - 2*deviationMV[log2CbWidth-3]) (8-788)

[0736] hor_max = hor_max_pic (8-790)

[0737] - If ver_min is less than ver_min_pic, the variables ver_min and ver_max are updated as follows:

[0738] ver_min = ver_min_pic (8-788)

[0739] ver_max = min(ver_max_pic, ver_min_pic + 2*deviationMV[log2CbHeight-3]) (8-790)

[0740] - Otherwise, if ver_max is greater than ver_max_pic, the variables ver_min and ver_max are updated as follows:

[0741] ver_min = max(ver_min_pic, ver_max_pic - 2*deviationMV[log2CbHeight-3]) (8-788)

[0742] ver_max = ver_max_pic (8-790)

[0743] The output variables hor_max, ver_max, hor_min, and ver_min are clipped as follows:

[0744] hor_max = Clip3(-2 17 , 2 17 -1, hor_max) (8-784)

[0745] ver_max = Clip3(-2 17 , 2 17 -1, ver_max) (8-785)

[0746] hor_min = Clip3(-2 17 , 2 17- 1, hor_min) (8-786)

[0747] ver_min = Clip3(-2 17 , 2 17 - 1, ver_min) (8-787)

[0748] The difference between implementation 1 and implementation 2 is the derivation of the picture boundary. In implementation 1, it is assumed that there is a 128-pixel margin (MAX_CU_SIZE) around the picture. The margin size can depend on the size of the coding tree unit.

[0749] 3.2 Clipping of the boundary motion vector

[0750] In EVC, it is assumed that each MV component can be stored by 18 bits. To meet this general requirement, the present invention adds an operation of clipping to the range [-2 17 , 2 17 - 1] as the last step of the derivation of the boundary motion vector. To perform this clipping operation without reducing the magnitude of the motion vector, the operations defined in section 1.4.1.2.3 can be performed with 5-bit precision instead of 9-bit precision. With these changes, the following changes (additions marked with and deletions marked with ) are needed to the general EIF algorithm (as described in section 1.3):

[0751] - The motion vector mvX is derived as follows:

[0752]

[0753]

[0754] mvX[0] = Clip3(hor_min, hor_max, mvX[0]) (8-760)

[0755] mvX[1] = Clip3(ver_min, ver_max, mvX[1]) (8-761)

[0756] The step “perform a clipping operation on the motion vector of the pixels of the coding block, within a range, to obtain a clipped motion vector, wherein the range depends on the updated first motion vector range” in the present invention corresponds to equations (8-760) and (8-761).

[0757] - The variables xInt, yInt, xFrac and yFrac are derived as follows:

[0758]

[0759]

[0760] xFrac = (mvX[0] » 4) & 31 (8-764)

[0761] yFrac = (mvX[1] » 4) & 31 (8-765)

[0762] 3.3 Alignment of memory bandwidth limitation conditions

[0763] In the implementation of step 6 described in section 1.4.1.2.2 for 4x4 EIF sub-blocks, the specification text example described above uses a threshold value of 81. The reason for using this threshold value will be explained in the next section.

[0764] According to the motion vector clipping dispersion defined by the array deviationMV (as described in section 1.4.1.2.3), the memory bandwidth threshold value for an 8x8 uni-prediction affine block (8x8 corresponds to the highest requirement case in terms of memory bandwidth according to the array deviationMV) is equal to This value corresponds to the value 9 for bi-prediction blocks.

[0765] The threshold value 81 used in the implementation of step 6 described in section 1.4.1.2.2 for 4x4 EIF sub-blocks corresponds to a memory bandwidth for uni-prediction blocks of This value corresponds to the value 10 for bi-prediction blocks.

[0766] In order to align the threshold values used in section 1.4.1.2.2 and section 1.4.1.2.3, the present invention changes the threshold value used in section 1.4.1.2.2 from 81 to 72, so that the memory bandwidth threshold value for uni-prediction blocks is equal to 4.5 and the memory bandwidth threshold value for bi-prediction blocks is equal to 9. With such a change, the memory bandwidth limitation conditions in section 1.4.1.2.2 and section 1.4.1.2.3 are aligned.

[0767] 4 Example of EVC specification text in the present invention

[0768] 8.5.3.8 Derivation process of affine sub-block size

[0769] The inputs of this process include:

[0770] - two variables cbWidth and cbHeight representing the width and height of the luma coding block;

[0771] - the number of control point motion vectors numCpMv;

[0772] - control point motion vector cpMvLX[ cpIdx ], with cpIdx = 0.. numCpMv - 1, and X being equal to 0 or 1 ;

[0773] - a prediction list usage flag predFlagLX, with X being equal to 0 or 1.

[0774] The output of the process includes:

[0775] - the size sizeSbX of the luma coding sub-block in the horizontal direction and the size sizeSbY in the vertical direction;

[0776] - the number numSbX of luma coding sub-blocks in the horizontal direction and the number numSbY in the vertical direction;

[0777] - a flag clipMV indicating the type of motion vector clipping applied to the block by the EIF.

[0778] where sizeSbX is set equal to cbWidth and sizeSbY is set equal to cbHeight.

[0779] The variables cifCanBeAppliedX and clipMVX are derived as follows, with X being equal to 0 or 1:

[0780] - cifCanBeAppliedX is set equal to TRUE;

[0781] - clipMVX is set equal to FALSE.

[0782] The variable cifSubblockSize is set equal to 4.

[0783] When predFlagLX is equal to 1, the following applies, with X being equal to 0 or 1:

[0784] - the horizontal change of motion vector dX, the vertical change of motion vector dY and the base motion vector mvBaseScaled are derived by invoking the process defined in section 8.5.3.9, with the width cbWidth of the luma coding block, the height cbHeight of the luma coding block, the number numCpMv of control point motion vectors and the control point motion vectors cpMvLX[ cpIdx ] with cpIdx = 0.. numCpMv - 1 as inputs to the process.

[0785] - the variables mvWx and mvWy are derived as follows:

[0786] - mvWx = max( abs( dX[ 0 ] ), abs( dX[ 1 ] ) )

[0787] - mvWy = max ( abs ( dY [0] ), abs ( dY [1] ) )

[0788] - The variable sizeSbXTemp is derived according to the value of mvWx as shown in Table 8-10.

[0789] - The variable sizeSbYTemp is derived according to the value of mvWy as shown in Table 8-11.

[0790] - The variable sizeSbX is modified as follows:

[0791] - sizeSbX = min ( sizeSbX, sizeSbXTemp )

[0792] - The variable sizeSbY is modified as follows:

[0793] - sizeSbY = min ( sizeSbY, sizeSbYTemp )

[0794] Table 8-10: Specification of sizeSbXTemp for various input values of mvWx

[0795] sizeSbX 0 1 2 3 4 >4 cbWidth mvWx 32 16 8 8 4

[0796] Table 8-11: Specification of sizeSbYTemp for various input values of mvWy

[0797] sizeSbY 0 1 2 3 4 >4 cbHeight FIG. 17 32 16 8 8 4

[0798] - The variables eifCanBeAppliedX and clipMVX are modified as follows:

[0799] - The arrays X[i] and Y[i] are derived as follows:

[0800] - X[0] = 0;

[0801] - X[1] = (eifSubblockSize + 1) * (dX[0] + (1 « 9));

[0802] - X[2] = (eifSubblockSize + 1) * dY[0];

[0803] - X[3] = X[1] + X[2];

[0804] - Y[0] = 0;

[0805] - Y[1] = (eifSubblockSize + 1) * dX[1];

[0806] - Y[2] = (eifSubblockSize + 1) * (dY[1] + (1 « 9));

[0807] - Y[3] = Y[1] + Y[2];

[0808] - The variable Xmax is set equal to the maximum value of X[i] for i equal to 0..3;

[0809] - The variable Xmin is set equal to the minimum value of X[i] for i equal to 0..3;

[0810] - The variable Ymax is set equal to the maximum value of Y[i] for i equal to 0..3;

[0811] - The variable Ymin is set equal to the minimum value of Y[i] for i equal to 0..3;

[0812] - The variable W is set equal to (Xmax - Xmin + (1 « 9) - 1) » 9;

[0813] - The variable H is set equal to (Ymax - Ymin + (1 « 9) - 1) » 9;

[0814] - If (W + 2) * (H + 2) is greater than 72, the variable clipMVX is set equal to TRUE;

[0815] - If dY[1] is less than ((-1) « 9), the variable eifCanBeAppliedX is equal to FALSE;

[0816] - Otherwise,

[0817] - If (max(0, dY[1]) + Abs(dX[1])) * (1 + eifSubblockSize) is greater than (1 « 9), the variable eifCanBeAppliedX is equal to FALSE.

[0818] The variables eifCanBeApplied and clipMV are derived as follows:

[0819] - eifCanBeApplied = eifCanBeApplied0 & eifCanBeApplied1

[0820] - clipMV = clipMV0 | clipMV1

[0821] If eifCanBeApplied is equal to FALSE, the variables sizeSbX and sizeSbY are modified as follows:

[0822] - sizeSbX = max( 8, sizeSbX )

[0823] - sizeSbY = max( 8, sizeSbY )

[0824] The number of luma coding sub-blocks in the horizontal direction numSbX and in the vertical direction numSbY are derived as follows:

[0825] - numSbX = cbWidth / sizeSbX

[0826] - numSbY = cbHeight / sizeSbY

[0827] [...]

[0828] 8.5.4.3 Interpolation process for the enhanced interpolation filter

[0829] The input of the process consists of:

[0830] - a position ( xCb, yCb ) in luma sample units;

[0831] - two variables cbWidth and cbHeight representing the width and height of the current coding block;

[0832] - a horizontal change of motion vector dX;

[0833] - a vertical change of motion vector dY;

[0834] - a motion vector mvBaseScaled;

[0835] - a selected reference picture sample array refPicLX;

[0836] - a sample bit depth bitDepth;

[0837] - a width of the picture pic_width in samples;

[0838] - a height of the picture pic_height in samples;

[0839] - a flag clipMV indicating the type of MV clipping;

[0840] - a flag isLuma indicating whether luma or chroma is processed.

[0841] The output of the process consists of:

[0842] - an array of ( cbWidth / SubWidthC ) x ( cbHeight / SubHeightC ) of predicted sample values predSamplesLX.

[0843] The interpolation filter coefficients T[p] for each fractional sample position p (equal to xFrac or yFrac) are as in Table 8-16.

[0844] The variables hor_max, ver_max, hor_min and ver_min are derived by invoking the process defined in section 8.5.4.5 with as input the position (xCb, yCb) in luma samples, two variables cbWidth and cbHeight representing the width and height of the current coding block, the horizontal change of the motion vector dX, the vertical change of the motion vector dY, the motion vector mvBaseScaled, the picture width in samples pic_width, the picture height in samples pic_height and the flag clipMV and as output hor_max, ver_max, hor_min and ver_min.

[0845] If isLuma is equal to FALSE (false), the variables mvBaseScaled, hor_min, hor_max, ver_min, ver_max are modified as follows:

[0846] - xCb = xCb / SubWidthC (8-748)

[0847] - yCb = yCb / SubHeightC (8-749)

[0848] - cbWidth = cbWidth / SubWidthC (8-750)

[0849] - cbHeight = cbHeight / SubHeightC (8-751)

[0850] - mvBaseScaled[ 0 ] = mvBaseScaled[ 0 ] / SubWidthC (8-752)

[0851] - mvBaseScaled[ 1 ] = mvBaseScaled[ 1 ] / SubHeightC (8-753)

[0852] - hor_min = hor_min / SubWidthC (8-754)

[0853] - hor_max = hor_max / SubWidthC (8-755)

[0854] - ver_min = ver_min / SubHeightC (8-756)

[0855] - ver_max = ver_max / SubHeightC (8-757)

[0856] The variables hor_min, hor_max, ver_min and ver_max before applying equations (8-754) to (8-757) represent the updated first motion vector range. The variables hor_min, hor_max, ver_min and ver_max after applying equations (8-754) to (8-757) represent the range involved in the step "performing a clipping operation on the motion vector of a pixel of the coded block, such that it is within a range, to obtain a clipped motion vector, wherein the range depends on the updated first motion vector range".

[0857] The variables shift0, shift1, offset0 and offset1 are derived as follows:

[0858] shift0 is set equal to bitDepth - 8 and offset0 is equal to 0;

[0859] shift1 is set equal to 12 - shift0 and offset1 is equal to 2 shift1-1 .

[0860] When x = -1..cbWidth, y = -1..cbHeight, the following applies:

[0861] - The motion vector mvX is derived as follows:

[0862] mvX[0] = (mvBaseScaled[0] + dX[0] * x + dY[0] * y) » 4 (8-758)

[0863] mvX[1] = (mvBaseScaled[1] + dX[1] * x + dY[1] * y) » 4 (8-759)

[0864] mvX[0] = Clip3(hor_min, hor_max, mvX[0]) (8-760)

[0865] mvX[1] = Clip3(ver_min, ver_max, mvX[1]) (8-761)

[0866] The step of "performing a clipping operation on the motion vector of the pixels of the coding block, so that it is within a range, to obtain a clipped motion vector, wherein the range depends on the updated first motion vector range" in the present application corresponds to equations (8-760) and (8-761).

[0867] The following equation corresponds to the step of "performing pixel-based motion compensation according to the clipped motion vector".

[0868] - The variables xInt, yInt, xFrac and yFrac are derived as follows:

[0869] xInt = xCb + (mvX[0] » 5) + x (8-762)

[0870] yInt = yCb + (mvX[1] » 5) + y (8-763)

[0871] xFrac = mvX[0] & 31 (8-764)

[0872] yFrac = mvX[1] & 31 (8-765)

[0873] - The variables A and B are derived as follows:

[0874] A = (refPicLX[xInt][yInt] * T[xFrac][0] + + refPicLX[xInt+1][yInt] * T[xFrac][1] + offset0) » shift0 (8-766)

[0875] B = (refPicLX[xInt][yInt+1] * T[xFrac][0] + + refPicLX[xInt+1][yInt+1] * T[xFrac][1] + offset0) » shift0 (8-767)

[0876] - The sample value b corresponding to the position (x, y) x,y is derived as follows:

[0877] b x,y = (A * T[yFrac][0] + B * T[yFrac][1] + offset1) » shift1 (8-768)

[0878] The enhanced interpolation filter coefficients eF[] are represented as {-1, 10, -1}.

[0879] The variables shift2, shift3, offset2 and offset3 are derived as follows:

[0880] shift2 is set equal to max( bit_depth - 11, 0 ), offset2 is equal to 2 shift2-1 ;

[0881] shift3 is set to ( 6 - max( bit_depth - 11, 0 ) ), offset3 is equal to 2 shift3-1 .

[0882] When x = 0..cbWidth - 1, y = -1..cbHeight, the following applies:

[0883] - h x,y = ( eF[0] * b x-1,y + eF[1] * b x,y + eF[2] * b x+1,y +offset2 ) >> shift2 (8-769)

[0884] When x = 0..cbWidth - 1, y = 0..cbHeight - 1, the following applies:

[0885] - predSamplesLX L [x][y] = Clip3( 0, (1<<bitDepth)-1, ( eF[0] * h x,y-1 + eF[1] * h x,y + eF[2] * b x,y+1 +offset3 ) >> shift3 ) (8-770)

[0886] Table 8-16: Specification of the interpolation filter coefficients T[p] for each fractional sample position p

[0887]

[0888]

[0889] 8.5.4.5 Derivation of clipping parameters for affine motion vectors (variant 1)

[0890] The inputs of the process include:

[0891] - a position ( xCb, yCb ) in units of full samples;

[0892] - two variables cbWidth and cbHeight representing the width and height of the current coding block;

[0893] - a horizontal change of the motion vector dX;

[0894] - a vertical change of the motion vector dY;

[0895] - motion vector mvBaseScaled;

[0896] - width of the picture in samples, pic_width;

[0897] - height of the picture in samples, pic_height;

[0898] - flag clipMV indicating the type of MV clipping.

[0899] The output of the process comprises:

[0900] - hor_max, ver_max, hor_min and ver_min indicating the maximum and minimum allowed horizontal and vertical components of a motion vector.

[0901] The variables log2CbW and log2CbH are derived as follows:

[0902] log2CbWidth = Log2(cbWidth) (8-774)

[0903] log2CbHeight = Log2(cbHeight) (8-775)

[0904] The variables hor_max_pic, ver_max_pic, hor_min_pic and ver_min_pic are derived as follows:

[0905] hor_max_pic = (pic_width + 128 - xCb - cbWidth - 1) « 5 (8-776)

[0906] ver_max_pic = (pic_height + 128 - yCb - cbHeight - 1) « 5 (8-777)

[0907] hor_min_pic = (-128 - xCb) « 5 (8-778)

[0908] ver_min_pic = (-128 - yCb) « 5 (8-779)

[0909] The variables hor_max_pic, ver_max_pic, hor_min_pic and ver_min_pic characterize a second range of motion vectors used to check whether a motion vector is pointing inside the first region of the reference picture.

[0910] The center motion vector mv_center is derived as follows:

[0911] mv_center[0] = ( mvBaseScaled[0] + dX[0]*(cbWidth » 1) + dY[0]*(cbHeight » 1) ) (8-780)

[0912] mv_center[1] = ( mvBaseScaled[1] + dX[1]*(cbWidth » 1) + dY[1]*(cbHeight » 1) ) (8-781)

[0913] The step of "obtaining the center motion vector mv_center of the coding block" in the present application corresponds to equations (8-780) and (8-781).

[0914] The process of rounding a motion vector defined in section 8.5.3.10 is invoked with mv_center, rightShift set equal to 4 and leftShift set equal to 0 as inputs, and the rounded motion vector is returned as mv_center.

[0915] If clipMV is equal to FALSE (false), the output variables hor_max, ver_max, hor_min and ver_min used to represent the maximum and minimum allowed motion vector horizontal and vertical components are set equal to the variables hor_max_pic, ver_max_pic, hor_min_pic and ver_min_pic, respectively.

[0916] Otherwise, the following steps are applied:

[0917] - the array deviationMV is set equal to { 128, 256, 544, 1120, 2272}.

[0918] - the variables mv_hor_min, mv_ver_min, mv_hor_max and mv_ver_max (denoted below as hor_min,

[0919] ver_min, hor_max and ver_max) are derived as follows:

[0920] hor_min = mv_center[0] - deviationMV[log2CbWidth - 3] (8-788)

[0921] ver_min = mv_center[1] - deviationMV[log2CbHeight - 3] (8-789)

[0922] hor_max = mv_center[0] + deviationMV[log2CbWidth - 3] (8-790)

[0923] ver_max = mv_center[1] + deviationMV[log2CbHeight - 3] (8-791)

[0924] The step "deriving a first motion vector range of the coding block depending on the center motion vector and a motion vector dispersion, wherein the motion vector dispersion depends on a size of the coding block" in the present application corresponds to equations (8-788) to (8-791). The motion vector dispersion is represented by a horizontal motion vector dispersion deviationMV[log2CbHeight - 3] and a vertical motion vector dispersion [log2CbHeight - 3].

[0925] - If hor_min is less than hor_min_pic, the variables hor_min and hor_max will be updated as follows:

[0926] hor_min = hor_min_pic (8-792)

[0927] hor_max = min(hor_max_pic, hor_min_pic + 2*deviationMV[log2CbWidth - 3]) (8-793)

[0928] - Otherwise, if hor_max is greater than hor_max_pic, the variables hor_min and hor_max will be updated as follows:

[0929] hor_min = max(hor_min_pic, hor_max_pic - 2*deviationMV[log2CbWidth - 3]) (8-794)

[0930] hor_max = hor_max_pic (8-795)

[0931] - If ver_min is less than ver_min_pic, the variables ver_min and ver_max will be updated as follows:

[0932] ver_min = ver_min_pic (8-796)

[0933] ver_max = min( ver_max_pic, ver_min_pic + 2*deviationMV[ log2CbHeight - 3 ] ) (8-797)

[0934] - Otherwise, if ver_max is greater than ver_max_pic, the variables ver_min and ver_max are updated as follows:

[0935] ver_min = max( ver_min_pic, ver_max_pic - 2*deviationMV[ log2CbHeight - 3 ] ) (8-798)

[0936] ver_max = ver_max_pic (8-799)

[0937] The step "update the first motion vector range to point inside the first region if the first motion vector range points at least partially outside the first region comprising the reference picture" in the present invention corresponds to equations (8-792) to (8-799) and the corresponding conditions described above.

[0938] The output variables hor_max, ver_max, hor_min and ver_min are clipped as follows:

[0939] hor_max = Clip3( -2 17 , 2 17 -1, hor_max ) (8-800)

[0940] ver_max = Clip3( -2 17 , 2 17 -1, ver_max ) (8-801)

[0941] hor_min = Clip3( -2 17 , 2 17 -1, hor_min ) (8-802)

[0942] ver_min = Clip3( -2 17 , 2 17 -1, ver_min ) (8-803)

[0943] The step "perform a clipping operation on the updated first motion vector range to be within the range [-2 17 , 2 17 -1]" in the present invention corresponds to equations (8-800) to (8-803).

[0944] 8.5.4.5 Derivation of clipping parameters for affine motion vectors (Variation 2)

[0945] The inputs of the process comprise:

[0946] - a position (xCb, yCb) in units of full samples;

[0947] - two variables cbWidth and cbHeight representing the width and height of the current coding block;

[0948] - a horizontal change of motion vector dX;

[0949] - a vertical change of motion vector dY;

[0950] - a motion vector mvBaseScaled;

[0951] - a width of the picture pic_width in units of samples;

[0952] - a height of the picture pic_height in units of samples;

[0953] - a flag clipMV indicating the MV clipping type.

[0954] The outputs of the process comprise:

[0955] - hor_max, ver_max, hor_min and ver_min representing the maximum and minimum allowed motion vector horizontal and vertical components.

[0956] The variables log2CbW and log2CbH are derived as follows:

[0957] log2CbWidth = Log2(cbWidth) (8-774)

[0958] log2CbHeight = Log2(cbHeight) (8-775)

[0959] The variables hor_max_pic, ver_max_pic, hor_min_pic and ver_min_pic are derived as follows:

[0960] hor_max_pic = (pic_width - xCb - cbWidth - 1) « 5 (8-776)

[0961] ver_max_pic = (pic_height - yCb - cbHeight - 1) « 5 (8-777)

[0962] hor_min_pic = (- xCb) « 5 (8-778)

[0963] ver_min_pic = (- yCb) « 5 (8-779)

[0964] The center motion vector mv_center is derived as follows:

[0965] mv_center[0] = (mvBaseScaled[0] + dX[0] * (cbWidth » 1) + dY[0] * (cbHeight » 1)) (8-780)

[0966] mv_center[1] = (mvBaseScaled[1] + dX[1] * (cbWidth » 1) + dY[1] * (cbHeight » 1)) (8-781)

[0967] The rounding process of motion vector defined in section 8.5.3.10 is invoked with mv_center, rightShift set equal to 4 and leftShift set equal to 0 as inputs and the rounded motion vector is returned as mv_center.

[0968] If clipMV is equal to FALSE (false), the output variables hor_max, ver_max, hor_min and ver_min used to represent the maximum and minimum allowed motion vector horizontal component and vertical component are set equal to the variables hor_max_pic, ver_max_pic, hor_min_pic and ver_min_pic, respectively.

[0969] Otherwise, the following steps are applied:

[0970] - the array deviationMV is set equal to {128, 256, 544, 1120, 2272}.

[0971] - the variables mv_hor_min, mv_ver_min, mv_hor_max and mv_ver_max are derived as follows:

[0972] hor_min = mv_center[0] - deviationMV[log2CbWidth - 3] (8-788)

[0973] ver_min = mv_center[1] - deviationMV[log2CbHeight - 3] (8-789)

[0974] hor_max = mv_center[0] + deviationMV[log2CbWidth - 3] (8-790)

[0975] ver_max = mv_center[1] + deviationMV[log2CbHeight - 3] (8-791)

[0976] - If hor_min is less than hor_min_pic, the variables hor_min and hor_max are updated as follows:

[0977] hor_min = hor_min_pic (8-788)

[0978] hor_max = min(hor_max_pic, hor_min_pic + 2*deviationMV[log2CbWidth - 3]) (8-790)

[0979] - Else if hor_max is greater than hor_max_pic, the variables hor_min and hor_max are updated as follows:

[0980] hor_min = max(hor_min_pic, hor_max_pic - 2*deviationMV[log2CbWidth - 3]) (8-788)

[0981] hor_max = hor_max_pic (8-790)

[0982] - If ver_min is less than ver_min_pic, the variables ver_min and ver_max are updated as follows:

[0983] ver_min = ver_min_pic (8-788)

[0984] ver_max = min(ver_max_pic, ver_min_pic + 2*deviationMV[log2CbHeight - 3]) (8-790)

[0985] - Else if ver_max is greater than ver_max_pic, the variables ver_min and ver_max are updated as follows:

[0986] ver_min = max(ver_min_pic, ver_max_pic - 2*deviationMV[log2CbHeight-3]) (8-788)

[0987] ver_max = ver_max_pic (8-790)

[0988] The output variables hor_max, ver_max, hor_min, and ver_min are clipped as follows:

[0989] hor_max = Clip3(-2 17 , 2 17 -1, hor_max) (8-784)

[0990] ver_max = Clip3(-2 17 , 2 17 -1, ver_max) (8-785)

[0991] hor_min = Clip3(-2 17 , 2 17 -1, hor_min) (8-786)

[0992] ver_min = Clip3(-2 17 , 2 17 -1, ver_min) (8-787)

[0993] FIG. 12 A block diagram of a method provided by the present disclosure; a method for coding video data using an affine motion model includes the following steps:

[0994] 1701: Obtain a center motion vector of a coding block.

[0995] 1702: Derive a first motion vector range of the coding block according to the center motion vector and a motion vector dispersion, wherein the motion vector dispersion depends on a size of the coding block.

[0996] 1703: If the first motion vector range at least partially points outside a first region comprising a reference picture, update the first motion vector range to point inside the first region, such that a minimum value and / or a maximum value of the updated first motion vector range points to a boundary of the first region, wherein a difference between the maximum value and the minimum value of the updated first motion vector range is equal to a smaller value of a double value of the motion vector dispersion and a size of the first region.

[0997] 1704: performing pixel-based motion compensation according to the updated first motion vector range.

[0998] FIG. 18 and 13 An example of a condition where the motion vector range at least partially lies outside the boundaries of the reference picture is shown in Fig. 17, where the motion vector range (extending to the left and right of the center motion vector) partially lies outside the left boundary of the picture. Similarly, the example also applies to the right boundary, the lower boundary and the upper boundary. This approach avoids a significant reduction of the motion vector range near the picture boundaries. The motion vector range is here computed according to the motion vector dispersion and the reference picture boundaries, instead of the motion vector dispersion and the center motion vector as in the prior art.

[0999] The present application provides the following further embodiments:

[1000] 1. A method for enhanced interpolation filtering of a picture boundary, comprising:

[1001] obtaining a motion vector, wherein the motion vector indicates a target block being a block adjacent to the picture boundary;

[1002] determining a motion vector range according to the motion vector, wherein the motion vector range is determined according to a size of the target block, the motion vector range is within the picture and starts from a boundary of the picture;

[1003] performing enhanced interpolation filtering according to the motion vector range.

[1004] 2. The method according to embodiment 1, wherein the picture is a reference picture of a current block, and the motion vector is a motion vector of the current block.

[1005] 3. The method according to embodiment 1, wherein the picture comprises a margin, the margin referring to an extended area around the original picture.

[1006] 4. The method according to embodiment 3, wherein the size of the margin depends on a maximum CTU size.

[1007] 5. The method according to embodiment 1, wherein the motion vector range is computed according to integers predefined for each preset block width and each preset block height.

[1008] 6. A codec comprising processing circuitry for performing the method according to any one of the above embodiments 1 to 5.

[1009] 7. The codec according to embodiment 6, wherein the codec comprises an encoder or a decoder.

[1010] 8. An apparatus for enhanced interpolation filtering of image boundaries, comprising:

[1011] a memory, a processor coupled to the memory, wherein the processor is configured to:

[1012] obtain a motion vector, wherein the motion vector indicates a target block that is adjacent to the image boundary;

[1013] determine a motion vector range according to the motion vector, wherein the motion vector range is determined according to a size of the target block, the motion vector range is within the image, and starts from a boundary of the image;

[1014] perform enhanced interpolation filtering according to the motion vector range.

[1015] 9. The apparatus of embodiment 8, wherein the image is a reference image of a current block, and the motion vector is a motion vector of the current block.

[1016] 10. The apparatus of embodiment 8, wherein the image comprises a margin, wherein the margin refers to an extended area around an original image.

[1017] 11. The apparatus of embodiment 10, wherein a size of the margin depends on a maximum CTU size.

[1018] 12. The apparatus of embodiment 8, wherein the motion vector range is calculated according to integers predefined for each preset block width and each preset block height.

[1019] Embodiment 13: A coding method implemented by a decoding / encoding device for coding video data, comprising:

[1020] deriving a first motion vector of a center point of an affine block;

[1021] deriving a motion vector dispersion according to a size of the affine block (e.g., a size WxH of the affine block);

[1022] deriving a first motion vector range according to the first motion vector and the motion vector dispersion;

[1023] deriving a second motion vector range to ensure that the MV points to a first predefined area;

[1024] checking whether the first motion vector range is contained in the second motion vector range;

[1025] if the first motion vector range is not contained in the second motion vector range,

[1026] deriving a third range of motion vectors from the second range of motion vectors and the motion vector dispersion;

[1027] otherwise, setting the third range of motion vectors equal to the first range of motion vectors;

[1028] in the course of computing the motion vector of the affine block, clipping the motion vector according to the third range.

[1029] Embodiment 14: the first predefined area is an area within the reference picture.

[1030] Embodiment 15: the first predefined area refers to an area within the reference picture with a margin, wherein the margin refers to an extended area around the reference picture.

[1031] Embodiment 16: the margin depends on a maximum CTU size.

[1032] Embodiment 17: the motion vector dispersion is computed from integers predefined for each possible block width and each possible block height.

[1033] According to another aspect, the application relates to a method for inter-prediction of a current picture block in a current picture of a video, the method being used by a decoding / encoding device, the method comprising:

[1034] computing a sub-block size MxN from affine motion model parameters or from information from which affine motion model parameters can be derived;

[1035] if the sub-block width M or the sub-block height N is smaller than or equal to a predefined value, performing an enhanced bi-linear interpolation filter (EIF) motion compensation process, wherein performing the EIF motion compensation process comprises:

[1036] deriving a motion vector of a respective sub-block of the picture block (e.g. affine picture block) from the affine motion model parameters based on P x Q (e.g. 1 x 1) sub-blocks;

[1037] clipping the motion vector of the sub-block so that the clipped motion vector belongs to a range of motion vectors (e.g. second range of motion vectors). The range of motion vectors of the coded block is derived from a center motion vector of the coded block and a size of the coded block. If the range of motion vectors is at least partially located outside the boundaries of a reference picture, the method further comprises updating the range of motion vectors so that it is located within the picture, so that a start value or an end value of the updated range of motion vectors is located at the boundaries of the reference picture.

[1038] FIG. 19 A block diagram of a decoder 1500 according to an embodiment of the present application is provided. The decoder 1500 includes one or more processors 1510; a non-transitory computer-readable storage medium 1520 coupled to the one or more processors 1510 and storing instructions to be executed by the one or more processors 1510, wherein the instructions, when executed by the one or more processors 1510, cause the decoder 1500 to perform the method according to an embodiment of the present application.

[1039] FIG. 15 A block diagram of an encoder 1600 according to an embodiment of the present application is provided. The encoder 1600 includes one or more processors 1610; a non-transitory computer-readable storage medium 1620 coupled to the one or more processors 1610 and storing instructions to be executed by the one or more processors 1610, wherein the instructions, when executed by the one or more processors 1610, cause the encoder 1600 to perform the method according to an embodiment of the present application.

[1040] The application of the encoding method and the decoding method shown in the above-described embodiments and the system using these methods are described below.

[1041] FIG. 16 A block diagram of a content providing system 3100 for implementing a content distribution service is provided. The content providing system 3100 includes a capturing device 3102, a terminal device 3106, and optionally a display 3126. The capturing device 3102 communicates with the terminal device 3106 through a communication link 3104. The communication link can include the communication channel 13 described above. The communication link 3104 includes, but is not limited to, Wi-Fi, Ethernet, cable, wireless (3G / 4G / 5G), USB, or any kind of combination thereof, etc.

[1042] The capture device 3102 generates data and can encode the data by the encoding method shown in the above embodiments. Alternatively, the capture device 3102 can distribute the data to a streaming server (not shown in the figure), which encodes the data and sends the encoded data to the terminal device 3106. The capture device 3102 includes, but is not limited to, a video camera, a smartphone or a tablet, a computer or a notebook, a video conference system, a PDA, an in-vehicle device, or any combination thereof, etc. For example, the capture device 3102 can include the source device 12 as described above. When the data includes video, the video encoder 20 included in the capture device 3102 can actually perform the video encoding process. When the data includes audio (i.e., sound), the audio encoder included in the capture device 3102 can actually perform the audio encoding process. For some practical scenarios, the capture device 3102 distributes the encoded video data and the encoded audio data together by multiplexing the encoded video data and the encoded audio data. For other practical scenarios, such as in a video conference system, the encoded audio data and the encoded video data are not multiplexed. The capture device 3102 distributes the encoded audio data and the encoded video data to the terminal device 3106 separately.

[1043] In the content providing system 3100, the terminal device 3106 receives and reproduces the encoded data. The terminal device 3106 can be a device with data receiving and restoring capability, such as a smartphone or a tablet 3108, a computer or a notebook 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a television (TV) 3114, a set top box (STB) 3116, a video conference system 3118, a video surveillance system 3120, a personal digital assistant (PDA) 3122, an in-vehicle device 3124, or any combination thereof, or a device capable of decoding the above encoded data. For example, the terminal device 3106 can include the destination device 14 as described above. When the encoded data includes video, the video decoder 30 included in the terminal device performs the video decoding process preferentially. When the encoded data includes audio, the audio decoder included in the terminal device performs the audio decoding process preferentially.

[1044] For terminal devices with displays, such as a smartphone or tablet 3108, a computer or laptop 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a television (TV) 3114, a personal digital assistant (PDA) 3122, or a car device 3124, the terminal device can feed the decoded data to the display of the terminal device. For terminal devices without displays, such as a STB 3116, a video conference system 3118, or a video surveillance system 3120, an external display 3126 should be connected to receive and display the decoded data.

[1045] When each device in the system performs encoding or decoding, an image encoding device or an image decoding device as shown in the above embodiments can be used.

[1046] FIG. 16 A schematic diagram of an exemplary structure of a terminal device 3106. After receiving a stream from a capture device 3102, a protocol proceeding unit 3202 analyzes the transport protocol of the stream. The protocol includes, but is not limited to, Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live streaming protocol (HLS), MPEG-DASH, Real-time Transport protocol (RTP), Real Time Messaging Protocol (RTMP), or any kind of combination thereof, etc.

[1047] After the protocol proceeding unit 3202 processes the stream, a stream file is generated. The file is output to a demultiplexing unit 3204. The demultiplexing unit 3204 can separate the multiplexed data into encoded audio data and encoded video data. As mentioned above, for some practical scenarios, such as in a video conference system, the encoded audio data and the encoded video data are not multiplexed. In this case, without using the demultiplexing unit 3204, the encoded data can be sent to a video decoder 3206 and an audio decoder 3208.

[1048] By demultiplexing, video elementary streams (video ESs), audio ESs, and optional subtitles are generated. Video decoder 3206 comprises video decoder 30 described in the above embodiments, which decodes the video ESs by the decoding method shown in the above embodiments to generate video frames and feeds this data to synchronization unit 3212. Audio decoder 3208 decodes the audio ESs to generate audio frames and feeds this data to synchronization unit 3212. Optionally, the video frames can be stored in a buffer (not shown in FIG. 16) before being fed to synchronization unit 3212. Similarly, the audio frames can be stored in a buffer before being fed to synchronization unit 3212. ​

[1049] Synchronization unit 3212 synchronizes the video frames and the audio frames and provides video / audio to video / audio display 3214. For example, synchronization unit 3212 synchronizes the presentation of the video information and the audio information. The information can be coded into the syntax according to timestamps related to the presentation of the coded audio and visual data and timestamps related to the distribution of the data streams themselves.

[1050] If subtitles are included in the stream, subtitle decoder 3210 decodes the subtitles, synchronizes the subtitles with the video frames and the audio frames, and provides video / audio / subtitles to video / audio / subtitles display 3216.

[1051] The present application is not limited to the above-described system, and the image encoding apparatus or the image decoding apparatus in the above embodiments can be incorporated into other systems, such as a car system.

[1052] Mathematical operators

[1053] The mathematical operators used in this application are similar to those used in the C programming language. However, the results of integer division and arithmetic shift operations are defined precisely in this application, and other operations, such as power and real division operations, are also defined. The numbering and counting convention generally starts from 0, e.g., "first" corresponds to 0th, "second" corresponds to 1st, and so on.

[1054] Arithmetic operators

[1055] The arithmetic operators are defined as follows:

[1056] + addition operation

[1057] - subtraction operation (as a binary operator) or negation operation (as a unary prefix operator)

[1058] * multiplication operation, including matrix multiplication operation

[1059] ​

[1060] ÷ used to denote division in a mathematical equation, but does not truncate or round the result.

[1061] ÷ used to denote division in a mathematical equation, but does not truncate or round the result.

[1062] Used to calculate the sum of f(i) for all integer values of i between x and y, inclusive.

[1063] x % y Modulo operation, representing the remainder of x divided by y, where x and y are integer values, and x >= 0 and y > 0.

[1064] Logical operators

[1065] Logical operators are defined as follows:

[1066] x && y Boolean logical AND operation of x and y

[1067] x || y Boolean logical OR operation of x and y

[1068] ! Boolean logical NOT operation

[1069] x? y : z Returns the value of y if x is TRUE or not equal to 0, otherwise returns the value of z.

[1070] Relational operators

[1071] Relational operators are defined as follows:

[1072] > Greater than

[1073] >= Greater than or equal to

[1074] < Less than

[1075] <= Less than or equal to

[1076] == Equal to

[1077] != Not equal to

[1078] When a relational operator is applied to a syntax element or variable that has been assigned the value "na" (not applicable), the value "na" is considered to be a distinct value from the syntax element or variable. The value "na" is considered not equal to any other value.

[1079] Bitwise operators

[1080] Bitwise operators are defined as follows:

[1081] & Bitwise AND. When operating on integer arguments, they are treated as binary values. When operating on binary arguments, the shorter argument is extended by adding high-order zeros to match the bit-length of the longer argument.

[1082] | Bitwise OR. When operating on integer arguments, they are treated as binary values. When operating on binary arguments, the shorter argument is extended by adding low-order zeros to match the bit-length of the longer argument.

[1083] ^ Bitwise XOR. When operating on integer arguments, they are treated as binary values. When operating on binary arguments, the shorter argument is extended by adding low-order zeros to match the bit-length of the longer argument.

[1084] x>>y Arithmetic shift right of the two's complement integer representation of x by y binary digits. The function is defined only for non-negative integer values of y. The result of the right shift is the bit shifted into the most significant bit (MSB) equal to the MSB of x before the shift operation.

[1085] x<<y Arithmetic shift left of the two's complement integer representation of x by y binary digits. The function is defined only for non-negative integer values of y. The result of the left shift is the bit shifted into the least significant bit (LSB) equal to zero.

[1086] Assignment operator

[1087] The assignment operator is defined as follows:

[1088] = Assignment operator

[1089] ++ Increment, i.e., x++ is equivalent to x = x + 1; used in array indexing, the value of the variable is taken before the increment operation.

[1090] -- Decrement, i.e., x-- is equivalent to x = x - 1; used in array indexing, the value of the variable is taken before the decrement operation.

[1091] + = Increment by a specified value, e.g., x + = 3 is equivalent to x = x + 3, x + = (-3) is equivalent to x = x + (-3).

[1092] - = Decrement by a specified value, e.g., x - = 3 is equivalent to x = x - 3, x - = (-3) is equivalent to x = x - (-3).

[1093] Range notation

[1094] The following notation is used to specify ranges of values:

[1095] x = y..z x takes on the integer values from y, through z, inclusive, where x, y, and z are all integers, and z is greater than y.

[1096] Mathematical functions

[1097] Mathematical functions are defined as follows:

[1098]

[1099] Asin(x) inverse sine function, operating on the argument x, in the range -1.0 to 1.0, inclusive, and returning a value in the range -π÷2 to π÷2, inclusive, in radians.

[1100] Atan(x) inverse tangent function, operating on the argument x, and returning a value in the range -π÷2 to π÷2, inclusive, in radians.

[1101]

[1102] Ceil(x) the smallest integer greater than or equal to x.

[1103] Clip1 Y (x) = Clip3(0, (1 << BitDepth Y )- 1, x)

[1104] Clip1 C (x) = Clip3(0, (1 << BitDepth C )- 1, x)

[1105]

[1106] Cos(x) cosine function, operating on the argument x, in radians.

[1107] Floor(x) the largest integer less than or equal to x.

[1108]

[1109] Ln(x) returns the natural logarithm (logarithm to the base e, where e is the natural logarithm base constant 2.718281828… ) of x.

[1110] Log2(x) returns the logarithm to the base 2 of x.

[1111] Log10(x) returns the logarithm of x to the base 10.

[1112]

[1113]

[1114] Round(x) = Sign(x) * Floor(Abs(x) + 0.5)

[1115]

[1116] Sin(x) trigonometric sine function, operating on the argument x in radians.

[1117]

[1118] Swap(x, y) = (y, x)

[1119] Tan(x) trigonometric tangent function, operating on the argument x in radians.

[1120] Order of operations

[1121] When no parentheses are used to explicitly indicate the order of precedence in an expression, the following rules apply:

[1122] - High-priority operations are performed before any lower-priority operations.

[1123] - Operations of the same priority are performed from left to right.

[1124] The following table shows the precedence of operations from highest to lowest, with higher positions in the table indicating higher precedence.

[1125] If these operators are also used in the C programming language, the order of precedence used here is the same as that used in the C programming language.

[1126] Table: Order of operation precedence from highest (top of table) to lowest (bottom of table)

[1127]

[1128] Textual description of logical operations

[1129] In text, the following logical operation statements are described in mathematical form as follows:

[1130]

[1131]

[1132] Can be described by:

[1133] …as follows / …as below:

[1134] - If condition 0, then statement 0

[1135] - Else if condition 1, then statement 1

[1136] - …

[1137] - Else (a hint about the remaining conditions), then statement n

[1138] Each "if…else if…else…" statement in text is introduced by a "…as follows" or "…as below" statement followed by "if…". The last condition "if…else if…else…" always follows an "else…". The "if…else if…else…" statements in between can be identified by matching "…as follows" or "…as below" with the ending content "else…".

[1139] In text, the following logical operation statements are described in mathematical form:

[1140]

[1141] Can be described by:

[1142] …as follows / …as below:

[1143] - If all of the following conditions are true, then statement 0:

[1144] - Condition 0a

[1145] - Condition 0b

[1146] - Else if one or more of the following conditions are true, then statement 1:

[1147] - Condition 1a

[1148] - Condition 1b

[1149] - …

[1150] - Else, statement n

[1151] In text, the following logical operation statements are described in mathematical form:

[1152] if(condition 0)

[1153] statement 0

[1154] if(condition 1)

[1155] statement 1

[1156] This can be described by:

[1157] When condition 0, statement 0

[1158] When condition 1, statement 1

[1159] Although the embodiments of the present application are primarily described in terms of video coding, it is noted that the embodiments of the coding system 10, the encoder 20 and the decoder 30 (respectively, the system 10) and other embodiments described herein can also be used for still image processing or coding, i.e. processing or coding of a single image independent of any previous or consecutive image in video coding. Generally, if the image processing coding is limited to a single image 17, only the inter prediction units 244 (in the encoder) and 344 (in the decoder) can not be available. All other functionalities (also referred to as tools or techniques) of the video encoder 20 and the video decoder 30 can likewise be used for still image processing, e.g. residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, partitioning 262 / 362, intra prediction 254 / 354 and / or loop filter 220 / 320, entropy encoding 270 and entropy decoding 304.

[1160] The embodiments of the encoder 20 and the decoder 30 and the functionalities described herein with respect to the encoder 20 and the decoder 30 and the like can be implemented by hardware, software, firmware, or any combination thereof. If implemented by software, the functions can be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer- readable media generally can correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as signals or carrier waves. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. A computer program product can include a computer-readable medium.

[1161] By way of example, and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other storage medium that can be used to store desired program code means in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or a wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[1162] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, as used herein the term "processor" can refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the various functions described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[1163] The techniques of this disclosure can be implemented in a variety of devices or apparatuses including a wireless phone, an integrated circuit (IC), or a set of ICs (e.g., a chip set). Various components, modules, or units are described herein to emphasize functionality of the device that performs the disclosed techniques, but those need not necessarily be implemented as separate hardware components. In fact, various components, modules, or units can be implemented together in a codec hardware unit, or provided by a collection of inter-operational hardware components, including one or more processors as described above, and / or by appropriate combinations thereof.

Claims

1. A method for decoding video data, characterized in that, include: Obtain the center motion vector mv_center of the encoded block; The first motion vector range of the coded block is derived based on the central motion vector and the motion vector dispersion, wherein the motion vector dispersion depends on the size of the coded block; If the first motion vector range at least partially points outside a first region including the reference image, the first motion vector range is updated to point inside the first region, such that the minimum and / or maximum value of the updated first motion vector range points to the boundary of the first region, wherein the difference between the maximum and minimum values ​​of the updated first motion vector range is equal to the smaller of double the motion vector dispersion and the size of the first region; pixel-based motion compensation is performed based on the updated first motion vector range.

2. The method according to claim 1, characterized in that, The motion vector dispersion is represented by horizontal motion vector dispersion and / or vertical motion vector dispersion, wherein the horizontal motion vector dispersion is derived based on the width of the coded block, and the vertical motion vector dispersion is derived based on the height of the coded block.

3. The method according to claim 2, characterized in that, The horizontal motion vector dispersion is denoted as deviationMV[log2CbWidth-3], and the vertical motion vector dispersion is denoted as deviationMV[log2CbHeight-3], where cbWidth and cbHeight represent the width and height of the coded block.

4. The method according to claim 2 or 3, characterized in that, The first motion vector range is represented by the first MV horizontal component range and / or the first MV vertical component range, wherein, The first MV level component range includes the first minimum MV level component value hor_min and the first maximum MV level component value hor_max. The first MV vertical component range includes the first minimum MV vertical component value ver_min and the first maximum MV vertical component value ver_max.

5. The method according to claim 4, characterized in that, Also includes: The range of the second motion vector is derived based on the size of the image including the coded block, wherein... The range of the second motion vector is represented by the range of the second MV horizontal component and / or the range of the second MV vertical component, wherein, The second MV level component range includes the second minimum MV level component value hor_min_pic and the second maximum MV level component value hor_max_pic. The second MV vertical component range includes the second minimum MV vertical component value ver_min_pic and the second maximum MV vertical component value ver_max_pic.

6. The method according to claim 5, characterized in that, The step of updating the first motion vector range to point within the first region if the first motion vector range at least partially points outside the first region including the reference image includes: If the first minimum MV level component value hor_min is less than the second minimum MV level component value hor_min_pic Set the updated value of the first minimum MV level component value hor_min to be equal to the second minimum MV level component value hor_min_pic. The updated value of the first maximum MV horizontal component value hor_max is derived by summing the second minimum MV horizontal component value hor_min_pic with the double value of the horizontal motion vector dispersion.

7. The method according to claim 5, characterized in that, The step of updating the first motion vector range to point within the first region if the first motion vector range at least partially points outside the first region including the reference image includes: If the first maximum MV horizontal component value hor_max is greater than the second maximum MV horizontal component value hor_max_pic Set the updated value of the first maximum MV horizontal component value hor_max to be equal to the second maximum MV horizontal component value hor_max_pic. The updated value of the first minimum MV horizontal component value hor_min is derived based on the subtraction between the second maximum MV horizontal component value hor_max_pic and the double value of the horizontal motion vector dispersion.

8. The method according to claim 5, characterized in that, The step of updating the first motion vector range to point within the first region if the first motion vector range at least partially points outside the first region including the reference image includes: If the first minimum MV vertical component value ver_min is less than the second minimum MV vertical component value ver_min_pic Set the updated value of the first minimum MV vertical component value ver_min to be equal to the second minimum MV vertical component value ver_min_pic. The updated value of the first maximum MV vertical component value ver_max is derived by summing the second minimum MV vertical component value ver_min_pic with the double value of the vertical motion vector dispersion.

9. The method according to claim 5, characterized in that, The step of updating the first motion vector range to point within the first region if the first motion vector range at least partially points outside the first region including the reference image includes: If the first maximum MV vertical component value ver_max is greater than the second maximum MV vertical component value ver_max_pic Set the updated value of the first maximum MV vertical component value ver_max to be equal to the second maximum MV vertical component value ver_max_pic. The updated value of the first minimum MV vertical component value ver_min is derived based on the subtraction between the second maximum MV vertical component value ver_max_pic and the double value of the vertical motion vector dispersion.

10. The method according to claim 5 or 6, characterized in that, If the first motion vector range points at least partially outside the left boundary of the first region, the variables hor_min and hor_max, representing the range of the first MV horizontal component of the first motion vector range, are updated as follows: hor_min = hor_min_pic hor_max = min(hor_max_pic, hor_min_pic + 2 * horizontal motion vector dispersion), where, hor_min represents the first minimum MV level component after the update, and hor_max represents the first maximum MV level component after the update. hor_min_pic and hor_max_pic represent the range of the second MV horizontal component of the second motion vector range. hor_min_pic represents the second minimum MV level component within the second MV level component range, and hor_max_pic represents the second maximum MV level component within the second MV level component range; The range of the second motion vector depends on the size of the image including the coded block.

11. The method according to claim 5 or 7, characterized in that, If the range of the first motion vector at least partially points outside the right boundary of the first region, Update the variables hor_min and hor_max, which represent the range of the first MV horizontal component of the first motion vector range, to: hor_min = max(hor_min_pic, hor_max_pic - 2 * horizontal motion vector dispersion), hor_max = hor_max_pic, where, hor_min represents the first minimum MV level component after the update, and hor_max represents the first maximum MV level component after the update. hor_min_pic and hor_max_pic represent the range of the second MV horizontal component of the second motion vector range. hor_min_pic represents the second minimum MV level component within the second MV level component range, and hor_max_pic represents the second maximum MV level component within the second MV level component range; The range of the second motion vector depends on the size of the image including the coded block.

12. The method according to claim 5 or 8, characterized in that, If the range of the first motion vector at least partially points outside the upper boundary of the first region, Update the variables ver_min and ver_max, which represent the range of the first MV vertical component, to: ver_min = ver_min_pic ver_max = min(ver_max_pic, ver_min_pic + 2 * vertical motion vector dispersion), where, ver_min represents the first minimum vertical MV component after the update, and ver_max represents the first maximum vertical MV component after the update; ver_min_pic and ver_max_pic represent the range of the second MV vertical component of the second motion vector range. ver_min_pic represents the second minimum MV vertical component within the second MV vertical component range, and ver_max_pic represents the second maximum MV vertical component within the second MV vertical component range; The range of the second motion vector depends on the size of the image including the coded block.

13. The method according to claim 5 or 9, characterized in that, If the range of the first motion vector is at least partially located outside the lower boundary of the first region, Update the variables ver_min and ver_max, which represent the range of the first MV vertical component, to: ver_min = max(ver_min_pic, ver_max_pic - 2 * vertical motion vector dispersion), ver_max = ver_max_pic, where... ver_min represents the first minimum MV vertical component after the update of the first MV vertical component range, and ver_max represents the first maximum MV vertical component after the update of the first MV vertical component range. ver_min_pic and ver_max_pic represent the range of the second MV vertical component of the second motion vector range. ver_min_pic represents the second minimum MV vertical component within the second MV vertical component range, and ver_max_pic represents the second maximum MV vertical component within the second MV vertical component range; The range of the second motion vector depends on the size of the image including the coded block.

14. The method according to claim 1, characterized in that, The minimum or maximum value of the horizontal component of the updated first motion vector range points to the left or right boundary of the first region, and / or the minimum or maximum value of the vertical component of the updated first motion vector range points to the upper or lower boundary of the first region, respectively.

15. The method according to any one of claims 1 to 3, 5 to 9, and 14, characterized in that, The first region includes the reference image and the extended region surrounding the reference image.

16. The method according to claim 15, characterized in that, The size of the extended region depends on the size of the coding tree unit (CTU).

17. The method according to claim 15, characterized in that, The size of the extended area is 128 pixels.

18. The method according to any one of claims 1 to 3, 5 to 9, 14, and 16 to 17, characterized in that, Also includes: A limiting operation is performed on the updated first motion vector range, making it within the range [-2]. 17 ,2 17 -1] inside; The step of performing pixel-based motion compensation based on the updated first motion vector range includes: performing pixel-based motion compensation based on the updated and limited first motion vector range.

19. The method according to any one of claims 1 to 3, 5 to 9, 14, and 16 to 17, characterized in that, The step of performing pixel-based motion compensation based on the updated first motion vector range includes: A limiting operation is performed on the motion vectors of the pixels of the coded block to make them within a certain range, so as to obtain the limited motion vectors, wherein the range depends on the updated first motion vector range; Pixel-based motion compensation is performed based on the limited motion vector.

20. The method according to claim 19, characterized in that, Pixel-based motion compensation is performed using an enhanced interpolation filter (EIF).

21. A decoder, characterized in that, Includes processing circuitry for performing the method according to any one of claims 1 to 20.

22. The decoder according to claim 21, characterized in that, The decoder includes an encoder or a decoder.

23. A computer program product, characterized in that, Includes a program that, when executed by a computer, causes the computer to perform the method according to any one of claims 1 to 20.

24. A decoder (30, 1800), characterized in that, include: One or more processors (1810); A non-transitory computer-readable storage medium (1820) coupled to and storing a program executed by the one or more processors (1810), wherein, when the program is executed by the one or more processors (1810), the decoder (30, 1800) performs the method according to any one of claims 1 to 20.

25. An encoder (20, 1900), characterized in that, include: One or more processors (1910); A non-transitory computer-readable storage medium (1920) coupled to and storing a program executed by the one or more processors (1910), wherein, when the program is executed by the processor (1910), the encoder (20, 1900) performs the method according to any one of claims 1 to 20.

Images