Method and apparatus for processing video blocks of a video stream

Abstract

The present disclosure relates generally to video coding, and in particular to methods and systems for selecting an interpolation filter for generating an inter-frame prediction block based on a motion vector magnitude and / or a motion vector pixel resolution. For example, a method for processing a video block of a video stream is disclosed. The method may include: receiving a flag in the video stream; determining based on the received flag that the video block is to be inter-frame predicted based on a motion vector (MV) through a reference frame; deriving the MV based on a reference MV and an MVD (MV difference) of the video block; deriving a signal indicating an interpolation filter, wherein the signal is based on at least one of the magnitude and resolution of the MVD; selecting an interpolation filter based on the signal; and performing inter-frame prediction for the video block through the reference frame based on the MV and the selected interpolation filter.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority of U.S. non-provisional patent application No. 17 / 956,559 filed on September 29, 2022, which is based on and claims the benefit of priority of U.S. provisional patent application No. 63 / 286,327 filed on December 6, 2021. The titles of the above-mentioned U.S. non-provisional patent application and U.S. provisional patent application are both "Interpolation Filters for Adaptive Motion Vector DifferenceResolution", and these prior patent applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to video coding, and more particularly to methods and apparatus, computer devices for processing video blocks of a video stream. Background Art

[0004] This background description is provided herein for the purpose of generally presenting the context of the present disclosure. To the extent that the work is described in this background section, the work of the presently named inventors and aspects that may not otherwise be qualified as prior art at the time of filing this application are neither explicitly nor implicitly admitted to be prior art against the present disclosure.

[0005] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video may include a series of pictures, each of which has a spatial dimension of, for example, 1920×1080 luminance samples and all associated or subsampled chrominance samples. The series of pictures may have a fixed or variable picture rate (alternatively referred to as a frame rate), such as 60 pictures per second or 60 frames per second. Uncompressed video has specific bit rate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920×1080, a frame rate of 60 frames per second, and a chrominance subsampling of 4:2:0 at 8 bits per pixel per color channel requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 gigabytes of storage space.

[0006] One purpose of video encoding and decoding can be to reduce redundancy in an uncompressed input video signal by compression. Compression can help reduce the bandwidth and / or storage space requirements mentioned above, in some cases by two orders of magnitude or more. Both lossless compression and lossy compression and their combinations can be used. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal via a decoding process. Lossy compression refers to an encoding / decoding process in which the original video information is not fully retained during encoding and is not fully restored during decoding. When lossy compression is used, the reconstructed signal may be different from the original signal, but the distortion between the original signal and the reconstructed signal is made small enough so that the reconstructed signal is useful for the intended application even if there is some information loss. In the case of video, lossy compression is widely used in many applications. The amount of tolerable distortion depends on the application. For example, users of certain consumer video streaming applications may tolerate higher distortion than users of movie or television broadcast applications. The compression ratio that can be achieved by a specific encoding algorithm can be selected or adjusted to reflect various distortion tolerances: higher tolerable distortion generally allows encoding algorithms that produce higher losses and higher compression ratios.

[0007] Video encoders and decoders may utilize techniques from several broad categories and steps including, for example, motion compensation, Fourier transforms, quantization, and entropy coding.

[0008] Video codec techniques may include techniques known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into sample blocks. When all sample blocks are encoded in intra-mode, the picture may be referred to as an intra-picture. Intra-pictures and their derivatives such as independent decoder refresh pictures may be used to reset decoder states, and may therefore be used as the first picture in an encoded video bitstream and video session or as a still image. The samples of the intra-predicted blocks may then be transformed into the frequency domain, and the transform coefficients thus generated may be quantized before entropy coding. Intra-prediction represents a technique for minimizing sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficient, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0009] Conventional intra-frame coding, such as that known from, for example, MPEG (Moving Picture Experts Group, MPEG)-2 generation coding techniques, does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to encode / decode blocks based on surrounding sample data and / or metadata obtained during encoding and / or decoding of data blocks that are spatially adjacent and precede the data block being intra-coded or decoded in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses reference data only from the current picture being reconstructed, and does not use reference data from other reference pictures.

[0010] There can be many different forms of intra-frame prediction. When more than one such technique can be used in a given video coding technique, the techniques used can be referred to as intra-frame prediction modes. One or more intra-frame prediction modes can be provided in a particular codec. In some cases, a mode can have sub-modes and / or can be associated with various parameters, and the mode / sub-mode information and intra-frame coding parameters for a video block can be encoded separately or included together in a mode codeword. Which codeword is used for a given mode, sub-mode and / or parameter combination may have an impact on the coding efficiency gain through intra-frame prediction, and therefore the entropy coding technique used to convert the codeword into a bitstream. Summary of the invention

[0011] The present disclosure relates generally to video coding, and more particularly to methods and systems for selecting an interpolation filter for generating an inter-prediction block based on motion vector magnitude and / or motion vector pixel resolution.

[0012] In an example implementation, a method for processing a video block of a video stream is disclosed. The method may include: receiving a flag in the video stream; determining, based on the received flag, to perform inter-frame prediction on the video block according to a motion vector (MV) through a reference frame; deriving the MV according to a reference MV and an MVD (MV difference) of the video block; deriving a signal indicating an interpolation filter, wherein the signal is based on at least one of a magnitude and a resolution of the MVD; selecting an interpolation filter according to the signal; and performing inter-frame prediction on the video block through the reference frame according to the MV and the selected interpolation filter.

[0013] In the above implementation, the export signal may include: determining the amplitude of the MVD based on the MV level index associated with the video block, the MV level index being extracted from the video stream; and when the MV level index is not lower than an MV level index threshold, exporting a signal for defaulting to use a predefined interpolation filter.

[0014] In any of the above implementations, the predefined interpolation filter is among a plurality of interpolation filters forming a pool of candidate interpolation filters for video blocks having MV level indices below the MV level index threshold.

[0015] In any of the above implementations, the predefined interpolation filter comprises a REGULAR interpolation filter, a SMOOTH interpolation filter, a SHARP interpolation filter, or a BILINEAR interpolation filter.

[0016] In any of the above implementations, the derived signal may include: determining that an adaptive pixel resolution is applied to the MVD; determining a pixel resolution of the MVD among a plurality of adaptive MVD pixel resolutions; and deriving a signal for defaulting to use a predefined interpolation filter in response to the pixel resolution of the MVD being at one or more predefined levels.

[0017] In any of the above implementations, the one or more predefined levels include an MVD resolution level that is not higher than a predefined MVD pixel resolution threshold.

[0018] In any of the above implementations, the pixel resolution of the MVD is determined based on the MV level index of the video block extracted from the video stream.

[0019] In any of the above implementations, the predefined MVD pixel resolution threshold corresponds to the MV level index threshold.

[0020] In any of the above implementations, the predefined interpolation filter is among a plurality of interpolation filters forming a pool of candidate interpolation filters for video blocks having an adaptive MVD pixel resolution different from the one or more predefined levels.

[0021] In any of the above implementations, the predefined interpolation filter comprises one of a REGULAR interpolation filter, a SMOOTH interpolation filter, a SHARP interpolation filter, or a BILINEAR interpolation filter.

[0022] In any of the above implementations, the derived signal may include: determining an MV level index corresponding to the amplitude of the MVD from a video stream; when the MV level index of the MV is not lower than a predefined MV level index threshold, using a predefined interpolation filter subset in the interpolation filter set by default; and extracting a selection indicator from the video stream as a signal for selecting an interpolation filter from the predefined interpolation filter subset.

[0023] In any of the above implementations, the derived signal may include: determining that an adaptive pixel resolution is applied to the MVD; determining a pixel resolution of the MVD among multiple adaptive MVD pixel resolutions; in response to the pixel resolution of the MVD being not greater than a predefined MVD pixel resolution threshold, using a predefined subset of interpolation filters in an interpolation filter set by default; and extracting a selection indicator from the video stream as a signal for selecting an interpolation filter from the predefined subset of interpolation filters.

[0024] In any of the above implementations, the derived signal may include: determining an MVD to which an adaptive pixel resolution is applied to the video block; determining a pixel resolution of the MVD among multiple adaptive MVD pixel resolutions, or determining an MV level index of the MV among multiple MV level indexes; identifying a context set for encoding an interpolation filter selection indicator based on the pixel resolution of the MVD or the MV level index; and decoding the video stream according to the context set to obtain the interpolation filter selection indicator as the signal.

[0025] In any of the above implementations, the derived signal may include: determining that adaptive pixel resolution is applied to the MVD; determining the MV level index of the video block among multiple MV level indexes; when the MV level index is not less than an MV level index threshold, decoding the video stream using a first context set to obtain an interpolation filter selection indicator as the signal; and when the MV level index is less than the MV level index threshold, decoding the video stream using a second context set different from the first context set to obtain the interpolation filter selection indicator as the signal.

[0026] In any of the above implementations, deriving the signal may include, in response to determining that the MVD pixel resolution of the MVD is non-fractional, defaulting to using an interpolation filter.

[0027] In any of the above implementations, the derived signal may include: in response to determining that the MVD pixel resolution of the MVD is non-fractional, using a predefined subset of interpolation filters in the interpolation filter set by default; and extracting a selection indicator from the video stream as a signal for selecting an interpolation filter from the predefined subset of interpolation filters.

[0028] Aspects of the present disclosure also provide a video encoding or decoding device or apparatus, which includes a circuit system configured to perform any one of the above method implementations.

[0029] According to various aspects of the present disclosure, a device for processing a video block of a video stream is disclosed. The device includes: a receiving unit configured to receive a flag in the video stream and store the video stream in the buffer memory; a determining unit configured to determine, based on the received flag, whether to perform inter-frame prediction on the video block according to a motion vector (MV) through a reference frame; a first deriving unit configured to derive the MV according to a reference MV and an MVD (MV difference) of the video block; a second deriving unit configured to derive a signal indicating an interpolation filter, wherein the signal is based on at least one of an amplitude and a resolution of the MVD; a selecting unit configured to select the interpolation filter according to the signal; and an inter-frame prediction unit configured to perform inter-frame prediction on the video block through the reference frame according to the MV and the selected interpolation filter.

[0030] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions which, when executed by a computer for video decoding and / or encoding, cause the computer to perform a method for video decoding and / or encoding.

[0031] According to various aspects of the present disclosure, a computer device is provided, comprising a processor and a memory. The memory is used to store program code and transmit the program code to the processor; the processor is used to execute according to the instructions in the program code: receiving a flag in the video stream; determining to perform inter-frame prediction on the video block through a reference frame according to a motion vector (MV) based on the received flag; deriving the MV according to a reference MV and an MVD (MV difference) of the video block; deriving a signal indicating an interpolation filter, wherein the signal is based on at least one of an amplitude and a resolution of the MVD; selecting an interpolation filter according to the signal; and performing inter-frame prediction on the video block through the reference frame according to the MV and the selected interpolation filter.

[0032] According to various aspects of the present disclosure, a flag in the video stream is received, a determination is made based on the received flag that the video block is to be inter-predicted based on a motion vector (MV) through a reference frame, the MV is derived based on a reference MV and an MVD (MV difference) of the video block, a signal indicating an interpolation filter is derived, wherein the signal is based on at least one of an amplitude and a resolution of the MVD, an interpolation filter is selected based on the signal, and inter-prediction is performed for the video block through the reference frame based on the MV and the selected interpolation filter. In this way, when the MVD and the reference MV are used to derive an actual MV of a coding block, depending on the amplitude and / or pixel resolution of the MVD, certain types of interpolation filters can help provide better coding gain than other types of interpolation filters when used to generate a prediction block. Therefore, for certain MVD amplitudes or pixel resolution values ​​or value ranges, a reduced set of interpolation filters can be predefined as candidate interpolation filters to improve signaling efficiency while maintaining coding gain. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Additional features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0034] Figure 1A A schematic diagram showing an exemplary subset of intra prediction direction modes;

[0035] Figure 1B A diagram showing exemplary intra prediction directions;

[0036] Figure 2 A schematic diagram of a current block and its surrounding spatial merging candidates for motion vector prediction in an example is shown;

[0037] Figure 3 A schematic illustration showing a simplified block diagram of a communication system according to an example embodiment;

[0038] Figure 4 A schematic illustration showing a simplified block diagram of a communication system according to an example embodiment;

[0039] Figure 5 A schematic illustration showing a simplified block diagram of a video decoder according to an example embodiment;

[0040] Figure 6 A schematic illustration showing a simplified block diagram of a video encoder according to an example embodiment;

[0041] Figure 7 shows a block diagram of a video encoder according to another example embodiment;

[0042] Figure 8shows a block diagram of a video decoder according to another example embodiment;

[0043] Fig. 9 A scheme for coding block segmentation according to an example implementation of the present disclosure is shown;

[0044] Fig.10 Another scheme of coding block segmentation according to an example implementation of the present disclosure is shown;

[0045] Fig.11 Another scheme of coding block segmentation according to an example implementation of the present disclosure is shown;

[0046] Fig.12 An example of partitioning a basic block into coding blocks according to an example partitioning scheme is shown;

[0047] Fig.13 An example three-way segmentation scheme is shown;

[0048] Fig.14 An example quadtree binary tree coding block partitioning scheme is shown;

[0049] Fig.15 A scheme for splitting a coding block into a plurality of transform blocks and a coding order of the transform blocks according to an example implementation of the present disclosure is shown;

[0050] Fig.16 Another scheme for splitting a coding block into multiple transform blocks and the encoding order of the transform blocks according to an example implementation of the present disclosure is shown;

[0051] Fig.17 Another scheme for splitting a coding block into a plurality of transform blocks according to an example implementation of the present disclosure is shown;

[0052] Fig.18 A flowchart showing a method according to an example implementation of the present disclosure;

[0053] Fig.19 A schematic diagram of a computer system according to an example implementation of the present disclosure is shown. DETAILED DESCRIPTION

[0054] Throughout the specification and claims, terms may have nuanced meanings that are suggested or implied in the context beyond the explicitly stated meaning. The phrases "in one embodiment" or "in some embodiments" as used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" as used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" as used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" as used herein do not necessarily refer to different implementations. For example, it is intended that the claimed subject matter includes combinations of all or part of the exemplary embodiments / implementations.

[0055] Typically, terms can be understood at least in part based on usage in context. For example, as used herein, terms such as "and", "or" or "and / or" can include a variety of meanings depending at least in part on the context in which such terms are used. Typically, if "or" is used for an association list such as A, B or C, "or" is intended to represent A, B and C, which are used here in an inclusive sense, and A, B or C, which are used here in an exclusive sense. In addition, at least in part depending on the context, as used herein, the terms "one or more" or "at least one" can be used to describe any feature, structure or characteristic in a singular sense, or can be used to describe a combination of features, structures or characteristics in a plural sense. Similarly, terms such as "one", "an" or "the" can also be understood to convey singular usage or to convey plural usage, which depends at least in part on the context. In addition, the term "based on" or "determined by..." can be understood to not necessarily be intended to convey an exclusive set of factors, and can alternatively allow the presence of additional factors that are not necessarily explicitly described, which also depends at least in part on the context.

[0056] Certain modes of intra prediction were introduced with H.264, refined in H.265, and further refined in newer coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark set (BMS). Typically, for intra prediction, the predictor block can be formed using neighboring sample values ​​that have become available. For example, the available values ​​of a particular set of neighboring samples along a particular direction and / or line can be copied into the predictor block. A reference to the direction in use can be encoded in the bitstream, or the reference to the direction in use can be predicted by itself.

[0057] Reference Figure 1A, depicted at the bottom right is a subset of nine predictor directions specified in the 33 possible intra-frame predictor directions of H.265 (corresponding to the 33 angular modes of the 35 intra-frame modes specified in H.265). The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which the sample at 101 is predicted using neighboring samples. For example, arrow (102) indicates that sample (101) is predicted based on one or more neighboring samples at a 45 degree angle to the upper right and horizontal direction. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more neighboring samples at a 22.5 degree angle to the horizontal direction to the lower left of sample (101).

[0058] Still refer to Figure 1A , depicted at the top left is a square block (104) of 4×4 samples (indicated by bold dashed lines). The square block (104) includes 16 samples, each of which is labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the size of the block is 4×4 samples, S44 is at the bottom right. Also shown are example reference samples, which follow a similar numbering scheme. The reference samples are labeled with R, their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, prediction samples that are adjacent to the block being reconstructed are used.

[0059] The intra picture prediction of block 104 may start by copying reference sample values ​​from neighboring samples according to a signaled prediction direction. For example, assume that the encoded video bitstream includes signaling indicating, for this block 104, the prediction direction of arrow (102) - i.e., the samples are predicted according to one or more prediction samples to the upper right at a 45 degree angle to the horizontal direction. In such a case, samples S41, S32, S23, and S14 are predicted according to the same reference sample R05. Then, sample S44 is predicted according to reference sample R08.

[0060] In some cases, the values ​​of multiple reference samples may be combined, such as by interpolation, in order to calculate the reference sample; in particular, if the directions do not divide evenly by 45 degrees.

[0061] As video coding technology continues to develop, the number of possible directions is also increasing. For example, in H.264 (2003), there are nine different directions that can be used for intra-frame prediction. In H.265 (2013), this increased to 33, and at the time of this disclosure, JEM / VVC / BMS can support up to 65 directions. Experimental studies have been conducted to help identify the most appropriate intra-frame prediction directions, and certain techniques in entropy coding can be used to encode those most appropriate directions with a small number of bits, thereby accepting a specific bit penalty for the direction. In addition, the direction itself can sometimes be predicted based on the neighboring directions used in the intra-frame prediction of decoded neighboring blocks.

[0062] Figure 1B A schematic diagram (180) is shown depicting 65 intra prediction directions according to JEM to illustrate that the number of prediction directions increases in various encoding techniques evolving over time.

[0063] The manner in which bits representing intra-prediction directions are mapped to prediction directions in the coded video bitstream may vary from one video coding technique to another; and may range, for example, from simple direct mappings of prediction directions to intra-prediction modes, to codewords, to complex adaptive schemes involving most probable modes, and the like. In all cases, however, there may be some directions used for leading prediction that are statistically less likely to appear in the video content than some other directions. Since the goal of video compression is to reduce redundancy, in a well-designed video coding technique, those less likely directions may be represented by a larger number of bits than more likely directions.

[0064] Inter-picture prediction or inter-prediction may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or a portion thereof (reference picture) may be used to predict a newly reconstructed picture or picture portion (e.g., block) after being spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV). In some cases, the reference picture may be the same as the picture in the current reconstruction. The MV may have two dimensions, X and Y, or three dimensions, where the third dimension is an indication of the reference picture used (similar to the time dimension).

[0065] In some video compression techniques, the current MV applicable to a particular region of sample data can be predicted based on other MVs, for example, based on other MVs related to other regions of sample data that are spatially adjacent to the region being reconstructed and that precede the current MV in the decoding order. Doing so can significantly reduce the total amount of data required to encode the MV by eliminating redundancy in the related MVs, thereby improving compression efficiency. MV prediction can work effectively, for example, because when encoding an input video signal obtained from a camera (called natural video), there is a statistical possibility that a larger region than the region to which a single MV applies moves in a similar direction in the video sequence, and therefore in some cases similar motion vectors derived from MVs of neighboring regions can be used to predict the larger region. This causes the actual MV for a given region to be similar or identical to the MV predicted based on the surrounding MVs. After entropy coding, such an MV can then be represented with fewer bits than would be used if the MV was encoded directly rather than predicted based on neighboring MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself may be lossy, for example due to round-off errors when computing the predictor from several surrounding MVs.

[0066] Various MV prediction mechanisms are described in H.265 / HEVC (High Efficiency Video Coding, HEVC) (ITU-T (International Telecommunication Union-Telecommunication Standardization Sector, ITU-T) H.265 Recommendation, "High Efficiency Video Coding", December 2016). Among the multiple MV prediction mechanisms specified by H.265, the following is a technique referred to as "spatial merging" hereinafter.

[0067] Specifically, refer to Figure 2 , the current block (201) includes samples discovered by the encoder during the motion search process that can be predicted from a previous block of the same size that has been spatially shifted. Instead of encoding the MV directly, the MV associated with any of the five surrounding samples represented by A0, A1 and B0, B1, B2 (corresponding to 202 to 206, respectively) can be derived from metadata associated with one or more reference pictures, for example, the MV is derived from the most recent (in decoding order) reference picture. In H.265, MV prediction can use a predictor from the same reference picture used by a neighboring block.

[0068] Figure 3 A simplified block diagram of a communication system (300) according to an embodiment of the present disclosure is shown. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). Figure 3 In the example, a first pair of terminal devices (310) and (320) can perform unidirectional transmission of data. For example, the terminal device (310) can encode video data (e.g., video data of a video picture stream captured by the terminal device (310)) for transmission to another terminal device (320) via a network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to recover the video picture, and display the video picture based on the recovered video data. Unidirectional data transmission may be implemented in a media service application or the like.

[0069] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which bidirectional transmission may be implemented, for example, during a video conferencing application. For the bidirectional transmission of data, in the example, each of the terminal devices (330) and (340) can encode video data (e.g., video data of a video picture stream captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) can also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), and can decode the encoded video data to restore the video picture, and can display the video picture at an accessible display device based on the restored video data.

[0070] exist Figure 3In the example, terminal devices (310), (320), (330) and (340) can be implemented as servers, personal computers and smart phones, but the applicability of the basic principles of the present disclosure may not be so limited. Implementations of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, etc. Network (350) represents any number or type of network that transmits encoded video data between terminal devices (310), (320), (330) and (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) can exchange data in circuit switching channels, packet switching channels and / or other types of channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purpose of this discussion, unless explicitly stated herein, the architecture and topology of the network (350) may be irrelevant to the operation of the present disclosure.

[0071] As examples of applications for the disclosed subject matter, Figure 4 The placement of the video encoder and the video decoder in a video streaming environment is shown. The disclosed subject matter can be equally applicable to other video applications, including, for example, video conferencing, digital TV, broadcasting, gaming, virtual reality, storing compressed video on digital media including CD (Compact Disc, CD), DVD (Digital Versatile Disc, DVD), memory sticks, etc.

[0072] The video streaming system may include a video capture subsystem (413) that may include a video source (401), such as a digital camera, for creating an uncompressed video picture or image stream (402). In an example, the video picture stream (402) includes samples recorded by the digital camera of the video source 401. The video picture stream (402) is depicted as a thick line to emphasize the high amount of data when compared to the encoded video data (404) (or encoded video bitstream), which may be processed by an electronic device (420) coupled to the video source (401) including a video encoder (403). The video encoder (403) may include hardware, software, or a combination thereof to implement or embody aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is depicted as a thin line to emphasize the lower data volume when compared to the uncompressed video picture stream (402), which can be stored on the streaming server (405) for future use or directly to a downstream video device (not shown). One or more streaming client subsystems such as Figure 4The client subsystems (406) and (408) in the streaming server (405) can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and creates an output video picture stream (411) that is uncompressed and can be presented on a display (412) (e.g., a display screen) or another rendering device (not depicted). The video decoder 410 can be configured to perform some or all of the various functions described in the present disclosure. In some streaming systems, the encoded video data (404), (407) and (409) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T Recommendation H.265. In the example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed subject matter can be used in the context of VVC as well as other video coding standards.

[0073] Note that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0074] Figure 5 A block diagram of a video decoder (510) according to any of the following embodiments of the present disclosure is shown. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used to replace Figure 4 A video decoder (410) is shown in an example.

[0075] The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510). In the same embodiment or another embodiment, one encoded video sequence can be decoded at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Each video sequence can be associated with multiple video frames or images. The encoded video sequence can be received from a channel (501), which can be a hardware / software link to a storage device storing the encoded video data or to a streaming source transmitting the encoded video data. The receiver (531) can receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective processing circuit systems (not depicted). The receiver (531) can separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) can be set between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) can be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be external to the video decoder (510) and separate from the video decoder (510) (not depicted). In still other applications, there may be a buffer memory (not depicted) external to the video decoder (510) for purposes such as preventing network jitter, and another additional buffer memory (515) may be internal to the video decoder (510) to, for example, handle playback timing. When the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may not be required, or the buffer memory (515) may be smaller. For use on an optimal packet network such as the Internet, a buffer memory (515) of sufficient size may be required, and its size may be relatively large. Such a buffer memory may be implemented with an adaptive size and may be implemented at least in part in an operating system or similar element (not depicted) external to the video decoder (510).

[0076] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (510) and potential information for controlling a presentation device such as a display (512) (e.g., a display screen), which may or may not be an integral part of the electronic device (530), but may be coupled to the electronic device (530), such as Figure 5As shown. The control information for the rendering device may be in the form of a supplementary enhancement information (SEI (Supplemental Enhancement Information, SEI) message) or a video usability information (VUI (Video Usability Information, VUI)) parameter set fragment (not depicted). The parser (520) may parse / entropy decode the encoded video sequence received by the parser (520). The entropy encoding of the encoded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the subgroup. The subgroup may include a group of pictures (Group of Pictures, GOP), a picture, a tile, a slice, a macroblock, a coding unit (Coding Unit, CU), a block, a transform unit (Transform Unit, TU), a prediction unit (Prediction Unit, PU), etc. The parser (520) may also extract information such as transform coefficients (eg, Fourier transform coefficients), quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0077] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0078] Depending on the type of coded video picture or part of coded video picture (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of symbol (521) may involve multiple different processing or functional units. The units involved and the way of involvement can be controlled by subgroup control information parsed by parser (520) from the coded video sequence. For simplicity, such subgroup control information flow between parser (520) and the following multiple processing or functional units is not depicted.

[0079] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a plurality of functional units as described below. In a practical implementation operating under commercial constraints, many of these functional units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed subject matter, the conceptual subdivision into the following functional units is adopted in the following disclosure.

[0080] The first unit may include a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) may receive quantized transform coefficients as symbols (521) and control information from the parser (520), including information indicating which type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrix, etc. The sealer / inverse transform unit (551) may output a block including sample values, which may be input into an aggregator (555).

[0081] In some cases, the output samples of the scaler / inverse transform (551) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may use surrounding block information that has already been reconstructed and stored in a current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator (555) may add the prediction information that has been generated by the intra-prediction unit (552) to the output sample information as provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0082] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-frame coded block and potentially a motion compensated block. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to obtain samples for inter-frame picture prediction. After motion compensation of the obtained samples according to the symbol (521) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (551) by the aggregator (555) (the output of unit 551 may be referred to as residual samples or residual signals) to generate output sample information. The address in the reference picture memory (557) at which the motion compensated prediction unit (553) obtains the predicted samples may be controlled by a motion vector, the MV being available to the motion compensated prediction unit (553) in the form of a symbol (521), the symbol (521) may have, for example, an X component, a Y component (shift) and a reference picture component (time). Motion compensation may also include interpolation of sample values ​​such as those obtained from the reference picture memory (557) when using sub-sample accurate motion vectors, and may also be associated with motion vector prediction mechanisms, etc.

[0083] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and available to the loop filter unit (556) as symbols (521) from the parser (520), but the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of the coded picture or coded video sequence, as well as to previously reconstructed and loop filtered sample values. Several types of loop filters may be included as part of the loop filter unit 556 in various orders, as will be described in further detail below.

[0084] The output of the loop filter unit (556) may be a sample stream that may be output to a rendering device (512) and stored in a reference picture memory (557) for use in future inter-picture prediction.

[0085] Once fully reconstructed, certain coded pictures can be used as reference pictures for future inter-picture prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557) and a new current picture buffer can be reallocated before starting to reconstruct a subsequent coded picture.

[0086] The video decoder (510) may perform decoding operations according to a predetermined video compression technique adopted in a standard such as ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence follows both the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as tools that can only be used under the profile. In order to conform to the standard, the complexity of the encoded video sequence may be within a range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further defined by the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the encoded video sequence.

[0087] In some example embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The video decoder (510) may use the additional data to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0088] Figure 6 A block diagram of a video encoder (603) according to an example implementation of the present disclosure is shown. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may also include a transmitter (640) (e.g., a transmission circuit system). The video encoder (603) may be used to replace Figure 4 The video encoder (403) in the example.

[0089] The video encoder (603) can be obtained from a video source (601) (not Figure 6 In another example, the video source (601) may be implemented as a part of the electronic device (620).

[0090] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by a video encoder (603), the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCb, RGB, XYZ, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) may be a storage device capable of storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures or images that are given motion when viewed sequentially. The picture itself may be organized as a spatial pixel array, wherein each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by a person skilled in the art. The following description focuses on samples.

[0091] According to some example embodiments, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed constitutes a function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units as described below. For simplicity, the coupling is not depicted. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate-distortion optimization techniques, ...), picture size, picture group (Group of Picture, GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other suitable functions belonging to the video encoder (603) optimized for a specific system design.

[0092] In some example embodiments, the video encoder (603) may be configured to operate in an encoding loop. As a simplified description, in an example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder would also create sample data, even if the embedded decoder 633 processes the encoded video stream of the source encoder 630 without entropy coding (because in the video compression techniques contemplated by the disclosed subject matter, any compression between the symbols and the encoded video bitstream in entropy coding can be lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces a bit-accurate result that is independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. The basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, for example due to channel errors) is used to improve encoding quality.

[0093] The operation of the "local" decoder (633) can be combined with the "remote" decoder such as has been described above. Figure 5 The operation of the video decoder (510) described in detail is the same. However, reference is also briefly made to Figure 5, since the symbols are available and the encoding of the symbols into a coded video sequence by the entropy encoder (645) and the decoding of the symbols by the parser (520) can be lossless, the entropy decoding portion of the video decoder (510) including the buffer memory (515) and the parser (520) may not be fully implemented in the local decoder (633) in the encoder.

[0094] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding that may only exist in the decoder may also necessarily need to exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter may sometimes focus on the decoder operation related to the decoding part of the encoder. Therefore, the description of the encoder technology can be simplified because the encoder technology is opposite to the decoder technology described comprehensively. A more detailed description of the encoder is provided only in some areas or aspects below.

[0095] In some example implementations, during operation, the source encoder (630) may perform motion compensated predictive coding that predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as "reference pictures". In this manner, the encoding engine (632) encodes the differences (or residuals) in color channels between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture. The term "residual" and its adjective form "residual" may be used interchangeably.

[0096] The local video decoder (633) can decode the encoded video data of the picture that can be designated as the reference picture based on the symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the encoded video data can be decoded at the video decoder ( Figure 6 When the video encoder (603) is decoded at a local video decoder (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the far-end (remote) video decoder.

[0097] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may be used as appropriate prediction references for the new picture. The predictor (635) may operate on a pixel block by pixel block basis to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (634).

[0098] The controller (650) can manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0099] The outputs of all the above mentioned functional units may be subjected to entropy coding in an entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0100] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via the communication channel (560), which can be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0101] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain encoding picture type to each encoded picture, which may affect the encoding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned one of the following picture types:

[0102] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those of ordinary skill in the art are aware of those variations of I pictures and their corresponding applications and features.

[0103] A predictive picture (P picture) may be a picture that can be encoded and decoded using inter prediction or intra prediction to predict sample values ​​of each block using at most one motion vector and a reference index.

[0104] Bi-directional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using inter-prediction or intra-prediction that uses up to two motion vectors and reference indices to predict sample values ​​for each block. Similarly, multi-predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0105] The source picture may typically be spatially subdivided into a plurality of coding blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples, respectively), and encoded block by block. These blocks may be predictively encoded with reference to other (already encoded) blocks, which are determined by the coding allocation applied to the corresponding picture of the block. For example, a block of an I picture may be non-predictively encoded, or a block of an I picture may be predictively encoded (spatial prediction or intra-frame prediction) with reference to an already encoded block of the same picture. A pixel block of a P picture may be predictively encoded via spatial prediction or via temporal prediction with reference to a previously encoded reference picture. A block of a B picture may be predictively encoded via spatial prediction or via temporal prediction with reference to one or two previously encoded reference pictures. A source picture or an intermediately processed picture may be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same approach, as described in further detail below.

[0106] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In the operation of the video encoder (603), the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0107] In some example embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0108] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often referred to simply as intra-picture prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits temporal or other correlations between pictures. For example, a particular picture in encoding / decoding, which is called the current picture, can be divided into blocks. The blocks in the current picture can be encoded by a vector called a motion vector if they are similar to reference blocks in a reference picture that has been previously encoded and still buffered in the video. The motion vector points to a reference block in a reference picture, and in the case of using multiple reference pictures, the motion vector can have a third dimension that identifies the reference picture.

[0109] In some example embodiments, bidirectional prediction techniques may be used for inter-picture prediction. According to such bidirectional prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture that are both subsequent to the current picture in the video in decoding order (but may be in the past or future in display order, respectively). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block may be jointly predicted by a combination of the first reference block and the second reference block.

[0110] In addition, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0111] According to some example embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, a picture in a video picture sequence is divided into coding tree units (Coding Tree Unit, CTU) for compression, and the CTUs in the picture may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU may include three parallel coding tree blocks (Coding Tree Block, CTB): a luminance CTB and two chrominance CTBs. Each CTU may be recursively split into one or more coding units (Coding Unit, CU) in a quadtree. For example, a 64×64 pixel CTU may be split into a 64×64 pixel CU, or four 32×32 pixel CUs. Each of one or more of the 32×32 blocks may be further split into four 16×16 pixel CUs. In some example embodiments, each CU may be analyzed during encoding to determine a prediction type for the CU among various prediction types, such as an inter prediction type or an intra prediction type. According to temporal and / or spatial predictability, a CU can be split into one or more prediction units (PUs). Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In an embodiment, the prediction operation in decoding (encoding / decoding) is performed in units of prediction blocks. Splitting a CU into PUs (or PBs of different color channels) can be performed in various spatial modes. For example, a luminance or chrominance PB may include a matrix of values ​​(e.g., luminance values) for samples such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, etc.

[0112] Figure 7 A diagram of a video encoder (703) according to another example implementation of the present disclosure is shown. The video encoder (703) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures, and encode the processed block into an encoded picture that is part of an encoded video sequence. The example video encoder (703) may be used instead of Figure 4 The video encoder (403) in the example.

[0113] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) then uses, for example, Rate-Distortion Optimization (RDO) to determine whether to use intra mode, inter mode, or bidirectional prediction mode to best encode the processing block. In the case where it is determined that the processing block is to be encoded in intra mode, the video encoder (703) can use intra prediction techniques to encode the processing block into an encoded picture; and in the case where it is determined that the processing block is to be encoded in inter mode or bidirectional prediction mode, the video encoder (703) can use inter prediction or bidirectional prediction techniques to encode the processing block into an encoded picture, respectively. In some example embodiments, merge mode can be used as a sub-mode of inter-picture prediction, in which a motion vector is derived from one or more motion vector predictors without the aid of an encoded motion vector component external to the predictor. In some other example embodiments, there may be a motion vector component applicable to the subject block. Therefore, the video encoder (703) may include Figure 7 Components not explicitly shown, such as a mode decision module for determining a prediction mode for a processing block.

[0114] exist Figure 7 In this example, the video encoder (703) includes Figure 7 An inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together are shown in the example arrangement.

[0115] The inter-frame encoder (730) is configured to: receive samples of a current block (e.g., a processing block); compare the block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture in a display sequence); generate inter-frame prediction information (e.g., description of redundant information, motion vectors, merge mode information according to an inter-frame coding technique); and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some examples, the reference picture is embedded in Figure 6 The decoding unit 633 (shown as Figure 7 The residual decoder 728, as described in further detail below, decodes the decoded reference picture based on the encoded video information.

[0116] The intra encoder (722) is configured to: receive samples of a current block (e.g., a processing block); compare the block with an already encoded block in the same picture; and generate quantization coefficients after transformation; and in some cases also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). The intra encoder (722) can calculate an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same picture.

[0117] The general controller (721) can be configured to determine general control data and control other components of the video encoder (703) based on the general control data. In an example, the general controller (721) determines a prediction mode for a block and provides a control signal to a switch (726) based on the prediction mode. For example, when the prediction mode is intra mode, the general controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream; and when the prediction mode for the block is inter mode, the general controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0118] The residual calculator (723) may be configured to calculate the difference (residual data) between the received block and the prediction result for the block selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) may be configured to encode the residual data to generate a transform coefficient. For example, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain to generate a transform coefficient. Then, the transform coefficient is subjected to a quantization process to obtain a quantized transform coefficient. In various example embodiments, the video encoder (703) further includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) may generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) may generate a decoded block based on the decoded residual data and the intra prediction information. The decoded blocks are appropriately processed to generate decoded pictures, and these decoded pictures may be buffered in a memory circuit (not shown) and used as reference pictures.

[0119] The entropy encoder (725) may be configured to format the bitstream to include the encoded blocks and perform entropy encoding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. When the block is encoded in an inter-frame mode or a merge sub-mode of a bi-prediction mode, there may be no residual information.

[0120] Figure 8 A diagram of an example video decoder (810) according to another embodiment of the present disclosure is shown. The video decoder (810) is configured to receive an encoded picture as part of an encoded video sequence and decode the encoded picture to generate a reconstructed picture. In an example, the video decoder (810) can be used to replace Figure 4 A video decoder (410) is shown in an example.

[0121] exist Figure 8 In this example, the video decoder (810) includes Figure 8 An entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together are shown in the example arrangement.

[0122] The entropy decoder (871) can be configured to reconstruct certain symbols from the encoded picture, which represent syntax elements constituting the encoded picture. Such symbols may include, for example, a mode for encoding a block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, merge sub-mode, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can identify certain samples or metadata used by the intra-frame decoder (872) or the inter-frame decoder (880) for prediction, respectively, residual information in the form of, for example, quantized transform coefficients, etc. In the example, when the prediction mode is inter-frame mode or bidirectional prediction mode, the inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is an intra-frame prediction type, the intra-frame prediction information is provided to the intra-frame decoder (872). The residual information can be subjected to inverse quantization and provided to the residual decoder (873).

[0123] The inter-frame decoder (880) may be configured to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0124] The intra decoder (872) may be configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0125] The residual decoder (873) can be configured to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) can also utilize certain control information (to include quantizer parameters (Quantizer Parameter, QP)), which can be provided by the entropy decoder (871) (since this may only be a small amount of control information, the data path is not depicted).

[0126] The reconstruction module (874) can be configured to combine the residual output by the residual decoder (873) with the prediction result (in this case, it can be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which forms a part of the reconstructed picture as part of the reconstructed video. Note that other suitable operations such as deblocking operations can also be performed to improve visual quality.

[0127] Note that the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using any suitable technology. In some example embodiments, the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using one or more processors that execute software instructions.

[0128] Turning to block segmentation for encoding and decoding, the general segmentation can start from the basic block and can follow a predefined set of rules, a specific pattern, a segmentation tree, or any segmentation structure or scheme. The segmentation can be hierarchical and recursive. After following any of the example segmentation processes described below or other processes or combinations thereof to divide or segment the basic block, a final set of partitions or coding blocks can be obtained. Each of these partitions can be at one of the various segmentation levels in the segmentation hierarchy and can have various shapes. Each partition in the partition can be referred to as a coding block (Coding Block, CB). For various example segmentation implementations further described below, each generated CB can have any allowed size and segmentation level. Such partitions are called coding blocks because they can form the following units, for which some basic encoding / decoding decisions can be made and encoding / decoding parameters can be optimized, determined, and signaled in the coded video bitstream. The highest or deepest level in the final partition represents the depth of the coding block segmentation tree structure. The coding block can be a luminance coding block or a chrominance coding block. The CB tree structure for each color can be referred to as a coding block tree (Coding Block Tree, CBT).

[0129] The coding blocks of all color channels may be collectively referred to as coding units (CUs). The hierarchical structure of all color channels may be collectively referred to as coding tree units (CTUs). The partitioning modes or structures for various color channels in a CTU may be the same or different.

[0130] In some implementations, the segmentation tree schemes or structures for the luma channel and the chroma channels may not need to be the same. In other words, the luma channel and the chroma channel may have separate coding tree structures or modes. In addition, whether the luma channel and the chroma channel use the same or different coding segmentation tree structures and the actual coding segmentation tree structures to be used may depend on whether the slice being encoded is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma channel and the luma channel may have separate coding segmentation tree structures or coding segmentation tree structure modes, while for a P slice or a B slice, the luma channel and the chroma channel may share the same coding segmentation tree scheme. When a separate coding segmentation tree structure or mode is applied, the luma channel may be segmented into CBs by one coding segmentation tree structure, and the chroma channel may be segmented into chroma CBs by another coding segmentation tree structure.

[0131] In some example implementations, a predetermined partitioning pattern may be applied to a basic block. Fig. 9As shown, the example 4-way partitioning tree can start from a first predefined level (e.g., 64×64 block level or other size as basic block size), and the basic block can be hierarchically partitioned down to a predefined lowest level (e.g., 4×4 level). For example, the basic block can be subjected to four predefined partitioning options or modes indicated by 902, 904, 906 and 908, where the partition designated as R is allowed for recursive partitioning because it can be repeated at lower levels as shown in FIG. Fig. 9 In some implementations, the same partitioning options indicated can be used for Fig. 9 Additional restrictions apply to the partitioning scheme. Fig. 9 In the implementation of , rectangular partitioning (e.g., 1:2 / 2:1 rectangular partitioning) may be allowed, but rectangular partitioning may not be allowed to be recursive, while square partitioning may be allowed to be recursive. If necessary, follow Fig. 9 The recursive case of the split generates a final set of coding blocks. The coding tree depth can be further defined to indicate the split depth from the root node or root block. For example, the coding tree depth for the root node or root block, such as a 64×64 block, can be set to 0, and the root block follows Fig. 9 After being further split once, the coding tree depth increases by 1. For the above scheme, the maximum or deepest level of the minimum split from 64×64 basic blocks to 4×4 will be 4 (starting from level 0). Such a splitting scheme can be applied to one or more color channels. It can be followed Fig. 9 The scheme of splitting each color channel independently (for example, the splitting mode or option in the predefined mode can be independently determined for each of the color channels at each hierarchical level). Alternatively, two or more color channels in the color channel can share Fig. 9 The same hierarchical mode tree (e.g., the same segmentation mode or option in the predefined mode can be selected for two or more color channels at each hierarchical level).

[0132] Fig.10 Another example predefined partitioning pattern that allows recursive partitioning to form a partitioning tree is shown. Fig.10 As shown, an example 10-way partitioning structure or pattern may be predefined. A root block may start at a predefined level (eg, from a basic block at a 128x128 level or a 64x64 level). Fig.10 Example partitioning structures include various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitionings. Fig.10 The partition type with 3 sub-partitions indicated as 1002, 1004, 1006 and 1008 in the second row of can be referred to as a "T-type" partition. The "T-type" partitions 1002, 1004, 1006 and 1008 can be referred to as a left T-type, a top T-type, a right T-type and a bottom T-type. In some example implementations, Fig.10 The rectangular partition is not allowed to be further subdivided. The coding tree depth can be further defined to indicate the split depth from the root node or root block. For example, the coding tree depth for the root node or root block, such as a 128×128 block, can be set to 0, and the root block follows Fig.10 After being further split once, the coding tree depth increases by 1. In some implementations, only full square partitions in 1010 may be allowed to follow Fig.10 The pattern recursively splits to the next level of the partition tree. In other words, for the square partition with T-patterns 1002, 1004, 1006 and 1008, recursive partitioning may not be allowed. If necessary, follow Fig.10 The recursive case of segmentation generates the final set of coding blocks. Such a scheme can be applied to one or more color channels. In some implementations, more flexibility can be added to the use of segmentation below the 8×8 level. For example, 2×2 chroma inter-frame prediction can be used in some cases.

[0133] In some other example implementations for coding block segmentation, a quadtree structure can be used to split a basic block or an intermediate block into quadtree partitions. Such a quadtree split can be applied hierarchically and recursively to any square-shaped partition. Whether a basic block or an intermediate block or a partition is further quadtree split can be adapted to various local characteristics of the basic block or intermediate block / partition. The quadtree splitting at the image boundary can be further adjusted. For example, an implicit quadtree splitting can be performed at a picture boundary so that the block will remain quadtree split until the size fits the picture boundary.

[0134] In some other example implementations, hierarchical binary segmentation from basic blocks can be used. For such a scheme, a basic block or an intermediate level block can be segmented into two partitions. The binary segmentation can be horizontal or vertical. For example, horizontal binary segmentation can segment a basic block or an intermediate block into equal right and left partitions. Similarly, vertical binary segmentation can split a basic block or an intermediate block into equal upper and lower partitions. Such binary segmentation can be hierarchical and recursive. The following decision can be made at each of the basic blocks or intermediate blocks: whether the binary segmentation scheme should continue, and if the scheme does continue further, whether horizontal binary segmentation or vertical binary segmentation should be used. In some implementations, further segmentation can end at a predefined minimum segmentation size (in one or two dimensions). Alternatively, further segmentation can end once a predefined segmentation level or depth from the basic block is reached. In some implementations, the aspect ratio of the partition can be limited. For example, the aspect ratio of the partition can be no less than 1:4 (or greater than 4:1). Thus, a vertical stripe partition having a vertical to horizontal aspect ratio of 4:1 can only be further vertically binary-partitioned into upper and lower partitions each having a vertical to horizontal aspect ratio of 2:1.

[0135] In yet other examples, the ternary partitioning scheme may be used to partition a basic block or any intermediate block, such as Fig.13 The ternary model can be implemented vertically, as Fig.13 1302, or implemented horizontally, as Fig.13 1304. Although Fig.13 The example split ratios in are shown as 1:2:1 vertically or horizontally, but other ratios may also be predefined. In some implementations, two or more different ratios may be predefined. Such a ternary segmentation scheme may be used to supplement a quadtree segmentation structure or a binary segmentation structure, because such a ternary tree segmentation is able to capture objects located at the center of a block in one continuous partition, while quadtrees and binary trees always split along the center of a block and thus split objects into separate partitions. In some implementations, the width and height of the partitions of the example ternary tree are always powers of 2 to avoid additional transformations.

[0136] The above partitioning schemes can be combined in any way at different partitioning levels. As an example, the above quadtree partitioning scheme and binary partitioning scheme can be combined to partition the basic block into a quadtree-binary-tree (QTBT) structure. In such a scheme, a basic block or an intermediate block / partition can be quadtree-split or binary-split according to a set of predefined conditions (if specified). Fig.14 A specific example is shown in FIG. Fig.14In the example of , the basic block is first split into four partitions by a quadtree, as shown in 1402, 1404, 1406 and 1408. Thereafter, each of the generated partitions is split into four additional partitions (e.g., 1408) by a quadtree at the next level, or is binary split into two additional partitions (e.g., horizontally or vertically, such as 1402 or 1406, both of which are symmetrical), or is not split (e.g., 1404). For square-shaped partitions, binary splits or quadtree splits can be recursively allowed, as shown in the entire example partitioning pattern of 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree splits and dashed lines represent binary splits. A flag can be used for each binary split node (non-leaf binary partition) to indicate whether the binary split is horizontal or vertical. For example, as shown in 1420, consistent with the partitioning structure of 1410, the flag "0" can represent a horizontal binary split and the flag "1" can represent a vertical binary split. For quadtree split partitions, there is no need to indicate the split type, because quadtree splits always split a block or partition horizontally and vertically to produce 4 sub-blocks / partitions of equal size. In some implementations, a flag "1" may indicate a horizontal binary split and a flag "0" may indicate a vertical binary split.

[0137] In some example implementations of QTBT, the quadtree and binary splitting rule set may be represented by the following predefined parameters and corresponding functions associated therewith:

[0138] –CTU size: the root node size of the quadtree (the size of the basic block)

[0139] –MinQTSize: Minimum allowed quadtree leaf node size

[0140] –MaxBTSize: Maximum allowed binary tree root node size

[0141] –MaxBTDepth: Maximum allowed binary tree depth

[0142] –MinBTSize: The minimum allowed binary tree leaf node size

[0143] In some example implementations of the QTBT partitioning structure, the CTU size can be set to 128×128 luma samples with two corresponding 64×64 blocks of chroma samples (when example chroma subsampling is considered and used), MinQTSize can be set to 16×16, MaxBTSize can be set to 64×64, MinBTSize (for both width and height) can be set to 4×4, and MaxBTDepth can be set to 4. Quadtree partitioning can be first applied to the CTU to generate quadtree leaf nodes. The quadtree leaf node can have a size from its minimum allowed size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., the CTU size). If the node is 128×128, the node will not be split first through the binary tree because the size exceeds the MaxBTSize (i.e., 64×64). Otherwise, nodes that do not exceed MaxBTSize can be split through the binary tree. Fig.14 In the example of , the basic block is 128×128. According to a predefined set of rules, the basic block can only be quadtree split. The basic block has a split depth of 0. Each of the four partitions generated is 64×64, does not exceed MaxBTSize, and can be further quadtree or binary split at level 1. Continue this process. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further splitting can be considered. When the binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal splitting can be considered. Similarly, when the binary tree node has a height equal to MinBTSize, no further vertical splitting is considered.

[0144] In some example implementations, the above-mentioned QTBT scheme can be configured to support the flexibility of having the same QTBT structure or separate QTBT structures for luma and chroma. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU can share the same QTBT structure. However, for I slices, the luma CTB can be partitioned into CBs by the QTBT structure, and the chroma CTB can be partitioned into chroma CBs by another QTBT structure. This means that a CU can be used to refer to different color channels in an I slice, for example, an I slice can consist of coding blocks of a luma component or coding blocks of two chroma components, and a CU in a P or B slice can consist of coding blocks of all three color components.

[0145] In some other implementations, the QTBT solution can be supplemented with the above ternary solution. Such an implementation can be called a multi-type tree (MTT) structure. For example, in addition to the binary split of the node, Fig.13In some implementations, only square nodes can be subjected to a ternary split. An additional flag may be used to indicate whether the ternary split is horizontal or vertical.

[0146] The design of two-level trees or multi-level trees, such as the QTBT implementation and the QTBT implementation supplemented by ternary splitting, can be mainly for reducing complexity. In theory, the complexity of traversing the tree is T D , where T represents the number of split types and D is the depth of the tree. A trade-off can be made by using multiple types (T) while reducing the depth (D).

[0147] In some implementations, the CB may be further segmented. For example, for the purpose of intra-frame or inter-frame prediction during the encoding and decoding process, the CB may also be segmented into a plurality of prediction blocks (Prediction Block, PB). In other words, the CB may also be divided into different sub-partitions, in which separate prediction decisions / configurations may be made. In parallel, for the purpose of depicting the level of transformation or inverse transformation of the video data, the CB may also be segmented into a plurality of transform blocks (Transform Block, TB). The schemes for segmenting the CB into PBs and TBs may be the same or different. For example, each segmentation scheme may be performed using its own process based on, for example, various features of the video data. In some example implementations, the PB segmentation scheme and the TB segmentation scheme may be independent. In some other example implementations, the PB segmentation scheme and the TB segmentation scheme may be related to the boundary. In some implementations, for example, the TB may be segmented after the PB segmentation, and in particular, after each PB is determined after segmenting the coding block, the PB may then be further segmented into one or more TBs. For example, in some implementations, the PB may be split into one, two, four, or other number of TBs.

[0148] In some implementations, in order to split the basic block into coding blocks and further split into prediction blocks and / or transform blocks, the luma channel and the chroma channel may be processed differently. For example, in some implementations, splitting the coding block into prediction blocks and / or transform blocks may be allowed for the luma channel, while such splitting of the coding block into prediction blocks and / or transform blocks may not be allowed for the chroma channel. In such an implementation, the transformation and / or prediction of the luma block can therefore be performed only at the coding block level. For another example, the minimum transform block size for the luma channel and the chroma channel may be different, for example, the coding block for the luma channel may be allowed to be split into smaller transform blocks and / or prediction blocks than the chroma channel. For another example, the maximum depth of splitting the coding block into transform blocks and / or prediction blocks may be different between the luma channel and the chroma channel, for example, the coding block for the luma channel may be allowed to be split into deeper transform blocks and / or prediction blocks than the chroma channel. For a specific example, the luma coding block may be partitioned into transform blocks of multiple sizes that may be represented by recursive partitioning down to up to 2 levels, and transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4 and transform block sizes from 4×4 to 64×64 may be allowed. However, for chroma blocks, only the largest possible transform block specified for the luma block may be allowed.

[0149] In some example implementations for partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partitioning may depend on whether the PB is intra-coded or inter-coded.

[0150] Splitting a coding block (or prediction block) into transform blocks can be implemented in various example schemes, including but not limited to quadtree splitting and predefined pattern splitting in a recursive or non-recursive manner, and with additional consideration of transform blocks at the boundaries of coding blocks or prediction blocks. In general, the resulting transform blocks may be at different split levels, may not be of the same size, and may not need to be square in shape (e.g., they may be rectangular with some allowed size and aspect ratio). Fig.15 , Fig.16 and Fig.17 Additional examples are described in further detail.

[0151] However, in some other implementations, the CB obtained via any partitioning scheme can be used as a basic or minimum coding block for prediction and / or transformation. In other words, no further splitting is performed for the purpose of performing inter-frame prediction / intra-frame prediction and / or for the purpose of transformation. For example, the CB obtained according to the above-mentioned QTBT scheme can be directly used as a unit for performing prediction. Specifically, such a QTBT structure removes the concept of multiple partitioning types, that is, such a QTBT structure removes the separation of CU, PU and TU, and supports more flexibility for the CU / CB partition shape as described above. In such a QTBT block structure, the CU / CB can have a square or rectangular shape. The leaf nodes of such a QTBT are used as units for prediction and transformation processing without any further partitioning. This means that in such an example QTBT coding block structure, CU, PU and TU have the same block size.

[0152] The various CB segmentation schemes described above and further segmenting the CB into PB and / or TB (including not performing PB / TB segmentation) may be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0153] A specific example implementation of coding block and transform block segmentation is described below. In such an example implementation, the recursive quadtree splitting described above or a predefined splitting pattern (e.g. Fig. 9 and Fig.10 The basic block is split into coding blocks according to those splitting modes in . At each level, whether the further quadtree splitting of a specific partition should continue can be determined by local video data features. The generated CB can be at various quadtree splitting levels and have various sizes. A decision can be made at the CB level (or CU level, for all three color channels) about whether to use inter-frame picture (time) prediction or intra-frame picture (spatial) prediction to encode the picture area. Each CB can be further split into one, two, four or other number of PBs according to a predefined PB splitting type. Inside a PB, the same prediction process can be applied, and relevant information can be sent to the decoder based on the PB. After obtaining the residual block by applying the prediction process based on the PB splitting type, the CB can be divided into TBs according to another quadtree structure similar to the coding tree for the CB. In this specific implementation, the CB or TB can be, but not necessarily limited to, a square shape. In addition, in this specific example, the PB for inter-frame prediction can be a square or rectangular shape, while the PB for intra-frame prediction can be only a square. The coding block can be split into, for example, four square-shaped TBs. Each TB can be further recursively split (using quadtree splitting) into smaller TBs, which are called Residual Quad-Tree (RQT).

[0154] Another example implementation for partitioning a basic block into CBs, PBs, and / or TBs is further described below. Fig. 9 or Fig.10 Instead of using multiple segmentation unit types such as those shown in the figure, a quadtree with nested multi-type trees using binary and ternary split partition structures (e.g., QTBT or QTBT with ternary split as described above) can be used. The separation of CB, PB and TB (i.e., CB is split into PB and / or TB, and PB is split into TB) can be abandoned, except in the following case: a CB with a size that is too large for the maximum transform length is required, where such a CB may need to be further split. The example segmentation scheme can be designed to support more flexibility for the CB partition shape, so that both prediction and transformation can be performed at the CB level without further division. In such a coding tree structure, the CB can have a square or rectangular shape. Specifically, the coding tree block (CTB) can be first split by a quadtree structure. Then, the quadtree leaf nodes can be further split by nesting the multi-type tree structure. Fig.11 An example of a nested multi-type tree structure using binary or ternary splitting is shown in FIG. Fig.11 The example multi-type tree structure includes four split types, referred to as vertical binary split (SPLIT_BT_VER) (1102), horizontal binary split (SPLIT_BT_HOR) (1104), vertical ternary split (SPLIT_TT_VER) (1106), and horizontal ternary split (SPLIT_TT_HOR) (1108). The CB then corresponds to the leaves of the multi-type tree. In this example implementation, unless the CB is too large for the maximum transform length, the partition is used for both prediction and transform processing without any further partitioning. This means that in most cases, CB, PB, and TB have the same block size in a quadtree with a nested multi-type tree coding block structure. The exception occurs when the maximum supported transform length is less than the width or height of the color component of the CB. In some implementations, in addition to the binary or ternary split, Fig.11 The nested pattern can also include quadtree splits.

[0155] Fig.12 A specific example of a quadtree with a nested multi-type tree coding block structure (including quadtree, binary and ternary split options) for block partitioning of a basic block is shown in FIG. Fig.12 The basic block 1200 is shown to be split into four square partitions 1202, 1204, 1206 and 1208 by a quadtree. Further use is made for each of the quadtree split partitions Fig.11 The multi-type tree structure and quadtree are used to make further splitting decisions. Fig.12 In the example of , partition 1204 is not further split. Partitions 1202 and 1208 are each split using another quadtree. For partition 1202, the upper left, upper right, lower left, and lower right partitions of the second level quadtree split are split using the third level quadtree, Fig.11 Horizontal binary split 1104, non-split and Fig.11 The horizontal ternary split 1108 of the partition 1208 is adopted by another quadtree split, and the upper left, upper right, lower left and lower right partitions of the second level quadtree split are respectively adopted by the third level Fig.11 The vertical ternary split 1106 split, non-split, non-split and Fig.11 The horizontal binary split 1104. The two sub-partitions in the sub-partition of the third level upper left partition 1208 are respectively based on Fig.11 The horizontal binary split 1104 and the horizontal ternary split 1108 are further split. The partition 1206 adopts the following Fig.11 The second level splitting pattern of the vertical binary split 1102 forms two partitions, and the two partitions are divided into two partitions according to the third level. Fig.11 The horizontal ternary split 1108 and vertical binary split 1102 are further split. Fig.11 A fourth level split is further applied to one of the partitions by horizontal binary split 1104 .

[0156] For the specific example above, the maximum luma transform size may be 64x64, and the maximum supported chroma transform size may be different from luma, such as 32x32. Fig.12 The above example CB is usually not further split into smaller PBs and / or TBs, but when the width or height of the luma coding block or the chroma coding block is larger than the maximum transform width or height, the luma coding block or the chroma coding block can be automatically split in the horizontal and / or vertical directions to meet the transform size limit in that direction.

[0157] In a specific example for partitioning a basic block into the above-mentioned CBs, and as described above, the coding tree scheme can support the ability for luma and chroma to have separate block tree structures. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU can share the same coding tree structure. For example, for I slices, luma and chroma can have separate coding block tree structures. When a separate block tree structure is applied, the luma CTB can be partitioned into luma CBs by one coding tree structure, and the chroma CTB can be partitioned into chroma CBs by another coding tree structure. This means that a CU in an I slice can include coding blocks for a luma component or coding blocks for two chroma components, while a CU in a P slice or a B slice always includes coding blocks for all three color components unless the video is monochrome.

[0158] When a coding block is further partitioned into multiple transform blocks, the transform blocks in the coding block can be ordered in the bitstream in various orders or scanning modes. The following is further described in detail example implementations for partitioning a coding block or prediction block into transform blocks and the encoding order of the transform blocks. In some example implementations, as described above, transform partitioning can support transform blocks of multiple shapes such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, where the transform block size ranges from, for example, 4×4 to 64×64. In some implementations, if the coding block is less than or equal to 64×64, the transform block partitioning can be applied only to the luminance component, so that for the chrominance block, the transform block size is the same as the coding block size. Otherwise, if the coding block width or height is greater than 64, both the luminance coding block and the chrominance coding block can be implicitly split into multiple min(W,64)×min(H,64) transform blocks and min(W,32)×min(H,32) transform blocks, respectively.

[0159] In some example implementations of transform block partitioning, for both intra-coded blocks and inter-coded blocks, the coding block may be further partitioned into multiple transform blocks with a partitioning depth up to a predefined number of levels (e.g., 2 levels). The transform block partitioning depth and size may be related. For some example implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown below in Table 1.

[0160] Table 1: Transform segment size settings

[0161]

[0162] Based on the example mapping of Table 1, for a 1:1 square block, the next level transform split can create four 1:1 square sub-transform blocks. The transform partitioning can stop at 4×4, for example. Therefore, the transform size of 4×4 at the current depth corresponds to the same size of 4×4 at the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform split can create two 1:1 square sub-transform blocks, and for a 1:4 / 4:1 non-square block, the next level transform split can create two 1:2 / 2:1 sub-transform blocks.

[0163] In some example implementations, for the luma component of an intra-coded block, additional restrictions may be applied with respect to transform block partitioning. For example, for each level of transform partitioning, all sub-transform blocks may be restricted to be of equal size. For example, for a 32×16 coding block, a level 1 transform split creates two 16×16 sub-transform blocks and a level 2 transform split creates eight 8×8 sub-transform blocks. In other words, the second level split must be applied to all first level sub-blocks to keep the transform units of equal size. Fig.15An example of transform block partitioning for intra-coded square blocks following Table 1 and the coding order shown by arrows is shown in 1502. Specifically, a square coding block is shown at 1502. A first level splitting into 4 equal-sized transform blocks according to Table 1 is shown at 1504, where the coding order is indicated by arrows. A second level splitting of all first level equal-sized blocks into 16 equal-sized transform blocks according to Table 1 is shown at 1506, where the coding order is indicated by arrows.

[0164] In some example implementations, the above restrictions on intra coding may not apply to the luma component of an inter-coded block. For example, after the first level of transform splitting, any of the sub-transform blocks may be further independently split into a higher level. Thus, the resulting transform blocks may or may not have the same size. Fig.16 An example of splitting an inter-frame coding block into transform blocks and the coding order of the transform blocks is shown in FIG. Fig.16 In the example of FIG. 1602, according to Table 1, the inter-coded block 1602 is split into two levels of transform blocks. At the first level, the inter-coded block is split into four transform blocks of equal size. Then, only one of the four transform blocks (rather than all transform blocks) is further split into four sub-transform blocks, resulting in a total of 7 transform blocks with two different sizes, as shown in 1604. The example coding order of these 7 transform blocks is as follows: Fig.16 As shown by the arrow in 1604.

[0165] In some example implementations, for chroma components, some additional restrictions on transform blocks may be applied. For example, for chroma components, the transform block size may be as large as the coding block size, but not smaller than a predefined size, such as 8×8.

[0166] In some other example implementations, for coding blocks having a width (W) or a height (H) greater than 64, both luma coding blocks and chroma coding blocks may be implicitly split into multiple min(W, 64)×min(H, 64) transform units and min(W, 32)×min(H, 32) transform units, respectively. Here, in the present disclosure, "min(a, b)" may produce a smaller value between a and b.

[0167] Fig.17 Another alternative example scheme for partitioning a coding block or prediction block into transform blocks is also shown. Fig.17 As shown, instead of using recursive transform partitioning, a predefined set of partition types can be applied to the coding block according to the transform type of the coding block. Fig.17In the specific example shown, one of six example partitioning types may be applied to split the coding block into various numbers of transform blocks. Such a generated transform block partitioning scheme may be applied to either a coding block or a prediction block.

[0168] In more detail, Fig.17 The partitioning scheme of provides up to 6 example partitioning types for any given transform type (transform type refers to, for example, the type of the main transform, such as ADST and others). In this scheme, a transform partitioning type can be assigned to each coding block or prediction block based on, for example, rate-distortion cost. In an example, the transform partitioning type assigned to a coding block or prediction block can be determined based on the transform type of the coding block or prediction block. A specific transform partitioning type can correspond to a transform block split size and mode (or partitioning type), such as Fig.17 The six transform partition types shown are shown. The correspondence between various transform types and various transform partition types can be predefined. An example is shown below, where the capitalized mark indicates the transform partition type that can be assigned to the coding block or prediction block based on the rate-distortion cost:

[0169] PARTITION_NONE: Allocate a transform size equal to the block size

[0170] PARTITION_SPLIT: Allocate a transform size of 1 / 2 the block size in width and 1\2 the block size in height.

[0171] PARTITION_HORZ: Allocates a transform size with the same width as the block size and a height of 1\2 of the block size.

[0172] PARTITION_VERT: Allocates a transform size with a width of 1\2 the block size and the same height as the block size.

[0173] PARTITION_HORZ4: Allocates a transform size with the same width as the block size and a height of 1\4 of the block size.

[0174] PARTITION_VERT4: Allocates a transform size with a width of 1\4 of the block size and the same height as the block size.

[0175] In the above example, if Fig.17 The transform partition type shown contains uniform transform sizes for all the partitioned transform blocks. This is only an example and not a limitation. In some other implementations, a mixed transform block size may be used for the partitioned transform blocks under a specific partition type (or mode).

[0176] The PB (or CB, also referred to as a PB when not further partitioned into prediction blocks) obtained according to any of the above partitioning schemes can then become a separate block for encoding via intra-frame prediction or inter-frame prediction. For inter-frame prediction for the current PB, a residual between the current block and the prediction block can be generated, encoded, and included in the encoded bitstream.

[0177] Inter prediction can be implemented, for example, in a single reference mode or a composite reference mode. In some implementations, a skip flag may first be included in the bitstream for the current block (or at a higher level) to indicate whether the current block is inter-coded and the current block is not skipped. If the current block is inter-coded, another flag may be further included in the bitstream as a signal for indicating whether a single reference mode or a composite reference mode is used to predict the current block. For a single reference mode, one reference block may be used to generate a prediction block for the current block. For a composite reference mode, two or more reference blocks may be used to generate a prediction block, for example, by weighted averaging. A composite reference mode may be referred to as more than one reference mode, two reference modes, or multiple reference modes. One or more reference blocks may be identified using a reference frame index or multiple reference frame indexes and additionally using a corresponding motion vector or multiple motion vectors indicating a displacement between a reference block and a current block in position, for example, in horizontal pixels and vertical pixels. For example, in single reference mode, an inter-frame prediction block for a current block may be generated based on a single reference block identified by a motion vector in a reference frame, while for composite reference mode, a prediction block may be generated by a weighted average of two reference blocks in two reference frames indicated by two reference frame indices and two corresponding motion vectors. The motion vector may be encoded and included in the bitstream in various ways.

[0178] In some implementations, the encoding or decoding system may maintain a decoded picture buffer (Decoded PictureBuffer, DPB). Some images / pictures may be kept in the DPB to wait for display (in the decoding system), and some images / pictures in the DPB may be used as reference frames to enable inter-frame prediction (in the decoding system or the encoding system). In some implementations, the reference frames in the DPB may be marked as short-term references or long-term references for the current image being encoded or decoded. For example, a short-term reference frame may include a frame used to inter-frame predict a block in the current frame or a block in a predefined number (e.g., 2) of subsequent video frames closest to the current frame in decoding order. A long-term reference frame may include a frame in the DPB that may be used to predict image blocks in frames that are more than a predefined number of frames away from the current frame in decoding order. Information about such tags for short-term reference frames and long-term reference frames may be referred to as a reference picture set (Reference Picture Set, RPS) and may be added to the header of each frame in the encoded bitstream. Each frame in the encoded video stream can be identified by a Picture Order Counter (POC), which is numbered in an absolute manner according to the playback order or relative to a group of pictures, for example starting from an I frame.

[0179] In some example implementations, one or more reference picture lists containing identifications of short-term reference frames and long-term reference frames for inter-frame prediction may be formed based on information in the RPS. For example, a single picture reference list may be formed for unidirectional inter-frame prediction, denoted as L0 reference (or reference list 0), while two picture reference lists may be formed for bidirectional inter-frame prediction, denoted as L0 (or reference list 0) and L1 (or reference list 1) for corresponding directions in two prediction directions. The reference frames included in the L0 list and the L1 list may be ordered in various predetermined ways. The lengths of the L0 list and the L1 list may be signaled in the video bitstream. Unidirectional inter-frame prediction may be performed in a single reference mode, or in a composite reference mode when multiple references used to generate a prediction block by weighted averaging in a composite prediction mode are on the same side of the block to be predicted. Bidirectional inter-frame prediction may be performed only in composite mode because bidirectional inter-frame prediction involves at least two reference blocks.

[0180] In some implementations, a merge mode (MM) for inter-frame prediction may be implemented. Typically, for merge mode, one or more of the motion vectors in the motion vector in the single reference prediction or the motion vector in the composite reference prediction for the current PB may be derived from other motion vectors, rather than being calculated and signaled separately. For example, in an encoding system, the current motion vector for the current PB may be represented by the difference between the current motion vector and one or more other already encoded motion vectors (referred to as reference motion vectors). Such a motion vector difference, rather than the entirety of the current motion vector, may be encoded and included in the bitstream and may be associated with the reference motion vector. Accordingly, in a decoding system, a motion vector corresponding to the current PB may be derived based on the decoded motion vector difference and the decoded reference motion vector associated with the decoded motion vector difference. As a specific form of general merge mode (MM) inter-frame prediction, this inter-frame prediction based on motion vector difference may be referred to as a merge mode (MMVD) using motion vector difference. Therefore, MM (in general) or MMVD (in particular) can be implemented to exploit the correlation between motion vectors associated with different PBs to improve coding efficiency. For example, adjacent PBs may have similar motion vectors, and thus the MVD may be small and can be efficiently encoded. For another example, for blocks that are similarly positioned / located in space, the motion vectors may be correlated in time (between frames).

[0181] In some example implementations, an MM flag may be included in the bitstream during the encoding process to indicate whether the current PB is in merge mode. Additionally or alternatively, an MMVD flag may be included in the bitstream during the encoding process and the MMVD flag may be signaled to indicate whether the current PB is in MMVD mode. The MM flag and / or the MMVD flag or indicator may be set at the PB level, CB level, CU level, CTB level, CTU level, slice level, picture level, etc. For a specific example, both the MM flag and the MMVD flag may be included for the current CU, and the MMVD flag may be signaled immediately after the skip flag and the MM flag to specify whether the MMVD mode is used for the current CU.

[0182] In some example implementations of MMVD, a list of reference motion vector (RMV) candidates or MV predictor candidates for motion vector prediction may be formed for a block being predicted. The RMV candidate list may contain a predetermined number (e.g., 2) of MV predictor candidate blocks, whose motion vectors may be used to predict the current motion vector. The RMV candidate blocks may include blocks selected from neighboring blocks in the same frame and / or temporal blocks (e.g., blocks of the same position in the previous or subsequent frame of the current frame). These options represent blocks that may have similar or identical motion vectors as the current block at a spatial position or temporal position relative to the current block. The size of the MV predictor candidate list may be predetermined. For example, the list may contain two or more candidates. In order to become a member of the RMV candidate list, the candidate block may, for example, need to: have the same reference frame or frames as the current block; must exist (e.g., when the current block is close to the edge of the frame, a boundary check needs to be performed); and must have been encoded during the encoding process and / or must have been decoded during the decoding process. In some implementations, if spatially adjacent blocks are available and meet the above conditions, the merge candidate list can be first populated with the spatially adjacent blocks (scanned in a specific predefined order); and then if there is still available space in the list, the merge candidate list is populated with temporal blocks. For example, the adjacent RMV candidate blocks can be selected from the left block and the top block of the current block. The RMV predictor candidate list can be dynamically formed at various levels (sequence, picture, frame, slice, super block, etc.) as a dynamic reference list (DRL). The DRL can be signaled in the bitstream.

[0183] In some implementations, the actual MV predictor candidate used as a reference motion vector for predicting the motion vector of the current block can be signaled. In the case where the RMV candidate list contains two candidates, a one-bit flag (referred to as the merge candidate flag) can be used to indicate the selection of the reference merge candidate. For the current block being predicted in composite mode, each of the multiple motion vectors predicted using the MV predictor can be associated with a reference motion vector from the merge candidate list. The encoder can determine which of the RMV candidates more closely predicts the MV of the current coding block and signal the selection as an index into the DRL.

[0184] In some example implementations of MMVD, after an RMV candidate is selected and used as a basic motion vector predictor for a motion vector to be predicted, a motion vector difference (MVD or delta MV) representing the difference between the motion vector to be predicted and a reference candidate motion vector may be calculated in the encoding system. Such an MVD may include information representing the magnitude of the MV difference and the direction of the MV difference, both of which may be signaled in the bitstream. The magnitude of the motion difference and the direction of the motion difference may be signaled in various ways.

[0185] In some example implementations of the MMVD, a distance index may be used to specify magnitude information of a motion vector difference and to indicate one of a set of predefined offsets representing a predefined motion vector difference from a starting point (reference motion vector). The MV offset according to the signaled index may then be added to the horizontal or vertical component of the starting (reference) motion vector. Whether the horizontal or vertical component of the reference motion vector should be offset may be determined by the directional information of the MVD. An example predefined relationship between a distance index and a predefined offset is specified in Table 2.

[0186] Table 2 - Example relationship between distance index and predefined MV offset

[0187]

[0188] In some example implementations of MMVD, a direction index may also be signaled, and the direction index is used to indicate the direction of the MVD relative to a reference motion vector. In some implementations, the direction may be limited to either the horizontal direction or the vertical direction. An example 2-bit direction index is shown in Table 3. In the example of Table 3, the interpretation of the MVD may vary depending on the information of the start / reference MV. For example, when the start / reference MV corresponds to a unidirectional prediction block or to a bidirectional prediction block in which two reference frame lists point to the same side of the current picture (i.e., the POCs of both reference pictures are greater than the POC of the current picture, or are less than the POC of the current picture), the symbol in Table 3 may specify the symbol (direction) of the MV offset added to the start / reference MV. When the start / reference MV corresponds to a bidirectional prediction block where two reference pictures are on different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture, and the POC of the other reference picture is less than the POC of the current picture) and the difference between the reference POC in picture reference list 0 and the current frame is greater than the difference between the reference POC in picture reference list 1 and the current frame, the symbols in Table 3 may specify the sign of the MV offset added to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture in picture reference list 1 may have an opposite value (opposite sign of the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is greater than the difference between the reference POC in picture reference list 0 and the current frame, the symbols in Table 3 may specify the sign of the MV offset added to the reference MV associated with picture reference list 1, and the sign of the offset of the reference MV associated with picture reference list 0 has an opposite value.

[0189] Table 3 - Example implementation of the sign of the MV offset specified by the direction index

[0190] Direction Index 00 01 10 11 x-axis (horizontal) + – not applicable not applicable y-axis (vertical) not applicable not applicable + –

[0191] In some example implementations, the MVD may be scaled based on the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling is required. Otherwise, if the difference in POC in reference list 0 is greater than the difference in POC in reference list 1, the MVD of reference list 1 is scaled. If the difference in POC of reference list 1 is greater than list 0, the MVD of list 0 may be scaled in the same manner. If the starting MV is unidirectionally predicted, the MVD is added to the available MVs or reference MVs.

[0192] In some example implementations of MVD encoding and signaling for bidirectional composite prediction, in addition to or as an alternative to separately encoding and signaling two MVDs, symmetric MVD encoding may be implemented such that only one MVD needs to be signaled and the other MVD may be derived from the signaled MVD. In such an implementation, motion information including reference picture indices for both list 0 and list 1 is signaled. However, only the MVD associated with, for example, reference list 0 is signaled, and the MVD associated with reference list 1 is not signaled but derived. Specifically, at the slice level, a flag called "mvd_l1_zero_flag" may be included in the bitstream to indicate whether reference list 1 is not signaled in the bitstream. If the flag is 1, indicating that reference list 1 is equal to zero (and therefore not signaled), the bidirectional prediction flag (called "BiDirPredFlag") may be set to 0, meaning that there is no bidirectional prediction. Otherwise, if mvd_l1_zero_flag is 0, BiDirPredFlag may be set to 1 in the case where the nearest reference picture in list 0 and the nearest reference picture in list 1 form a pair of forward and backward reference pictures or a pair of backward and forward reference pictures, and both the list 0 reference picture and the list 1 reference picture are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. BiDirPredFlag of 1 may indicate that a symmetric mode flag is additionally signaled in the bitstream. When BiDirPredFlag is 1, the decoder may extract the symmetric mode flag from the bitstream. The symmetric mode flag may be signaled, for example, at the CU level (if necessary), and the symmetric mode flag may indicate whether a symmetric MVD coding mode is being used for the corresponding CU. When the symmetric mode flag is 1, it indicates that the symmetric MVD coding mode is used, and only the reference picture index (referred to as "mvp_l0_flag" and "mvp_l1_flag") of both list 0 and list 1 and the MVD associated with list 0 (referred to as "MVD0") are signaled, while another motion vector difference "MVD1" will be derived instead of being signaled. For example, MVD1 can be derived as -MVD0. Therefore, in the example symmetric MVD mode, only one MVD is signaled. In some other example implementations of MV prediction, for both MV prediction of single reference mode and composite reference mode, a coordination scheme can be used to implement general merge mode, MMVD, and some other types of MV prediction. Various syntax elements can be used to signal the way to predict the MV of the current block.

[0193] For example, for single reference mode, the following MV prediction modes may be signaled:

[0194] NEARMV - directly uses one of the motion vector predictors (MVP) in the list indicated by the DRL (Dynamic Reference List) index without using any MVD.

[0195] NEWMV - uses one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference and applies a delta to the MVP (eg, using MVD).

[0196] GLOBALMV - Use motion vectors based on frame-level global motion parameters.

[0197] Likewise, for the composite reference inter prediction mode using two reference frames corresponding to the two MVs to be predicted, the following MV prediction modes may be signaled:

[0198] NEAR_NEARMV-For each of the two MVs in the MV to be predicted, use one motion vector predictor (MVP) in the list signaled by the DRL index instead of using MVD.

[0199] NEAR_NEWMV - For predicting the first of the two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as the reference MV without using the MVD; for predicting the second of the two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as the reference MV in combination with the incremental MV (MVD) additionally signaled.

[0200] NEW_NEARMV - For predicting the second of the two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as the reference MV without using the MVD; for predicting the first of the two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as the reference MV in combination with an additionally signaled delta MV (MVD).

[0201] NEW_NEWMV - For each of the two MVs, one of the motion vector predictors (MVPs) in the list signaled by the DRL index is used as the reference MV and combined with the additionally signaled delta MV for prediction.

[0202] GLOBAL_GLOBALMV - Use the MV from each reference based on its frame level global motion parameters.

[0203] Thus, the term "NEAR" above refers to MV prediction using a reference MV instead of an MVD as in the general merge mode, while the term "NEW" refers to MV prediction involving the use of a reference MV and offsetting the reference MV using a signaled MVD as in the MMVD mode. For composite inter prediction, the reference base motion vector and motion vector increment above may typically be different or independent between the two references, even though the two references may be correlated, and this correlation may be exploited to reduce the amount of information required to signal the two motion vector increments. In this case, joint signaling of the two MVDs may be implemented and indicated in the bitstream.

[0204] The above dynamic reference list (DRL) can be used to store a set of indexed motion vectors that are dynamically maintained and considered as candidate motion vector predictors.

[0205] In some example implementations, the pixel resolution of the MVD may be allowed to be predefined. For example, a motion vector precision (or accuracy) of 1 / 8 pixel may be allowed. The MVD described above may be constructed and signaled in various ways in various MV prediction modes. In some implementations, various syntax elements may be used to signal the motion vector differences in reference frame list 0 or list 1 above.

[0206] For example, a syntax element called "mv_joint" may specify which component of the motion vector difference associated with it is non-zero. For MVD, mv_joint is signaled jointly for all non-zero components. For example:

[0207] A value of 0 for mv_joint may indicate that there is no non-zero MVD in the horizontal or vertical direction;

[0208] A value of 1 for mv_joint may indicate that there is a non-zero MVD only along the horizontal direction;

[0209] A value of 2 for mv_joint may indicate that there is a non-zero MVD only along the vertical direction;

[0210] A value of 3 for mv_joint may indicate that there is a non-zero MVD in both the horizontal and vertical directions.

[0211] When the "mv_joint" syntax element of the MVD indicates that there are no non-zero MVD components, no additional MVD information is signaled. However, if the "mv_joint" syntax indicates that there are one or two non-zero components, additional syntax elements may be further signaled for each of the non-zero MVD components, as described below.

[0212] For example, a syntax element called "mv_sign" may be used to additionally specify whether the corresponding motion vector difference amount is positive or negative.

[0213] For another example, a syntax element called "mv_class" can be used to specify the class of the motion vector difference of the corresponding non-zero MVD component in a predefined set of classes. The predefined motion vector difference classes can be used, for example, to divide the continuous motion vector difference amplitude space into non-overlapping ranges, where each range corresponds to an MVD class. Therefore, the MVD class notified by the signal indicates the amplitude range of the corresponding MVD component. In the example implementation shown in Table 4 below, a higher class corresponds to a motion vector difference with a larger amplitude range. In Table 4, the symbol (n,m] is used to represent a motion vector difference range greater than n pixels and less than or equal to m pixels.

[0214] Table 4: Motion vector difference magnitude levels

[0215] MV Level MVD Amplitude MV_CLASS_0 (0,2] MV_CLASS_1 (2,4] MV_CLASS_2 (4,8] MV_CLASS_3 (8,16] MV_CLASS_4 (16,32] MV_CLASS_5 (32,64] MV_CLASS_6 (64,128] MV_CLASS_7 (128,256] MV_CLASS_8 (256,512] MV_CLASS_9 (512,1024] MV_CLASS_10 (1024,2048]

[0216] In some other examples, a syntax element referred to as "mv_bit" may be further used to specify the integer portion of the offset between a non-zero motion vector difference component and the starting amplitude of the correspondingly signaled MV level amplitude range. Thus, the mv_bit may indicate the amplitude or magnitude of the MVD. The number of bits required to signal the full range of each MVD level in the "mv_bit" may vary depending on the MV level. For this example, MV_CLASS_0 and MV_CLASS_1 in the implementation of Table 4 may require only a single bit to indicate an integer pixel offset of 1 or 2 from a starting MVD of 0; each higher MV_CLASS in the example implementation of Table 4 may require an incrementally larger "mv_bit" of one bit compared to the previous MV_CLASS.

[0217] In some other examples, a syntax element called "mv_fr" can be further used to specify the first 2 fractional bits of the motion vector difference of the corresponding non-zero MVD component, and a syntax element called "mv_hp" can be used to specify the third fractional bit (high resolution bit) of the motion vector difference of the corresponding non-zero MVD. The two bits of "mv_fr" basically provide an MVD resolution of 1 / 4 pixel, and the "mv_hp" bit can further provide a resolution of 1 / 8 pixel. In some other implementations, more than one "mv_hp" bit can be used to provide an MVD pixel resolution finer than 1 / 8 pixel. In some example implementations, an additional flag can be signaled at one or more of the various levels to indicate whether an MVD resolution of 1 / 8 pixel or higher is supported. If the MVD resolution is not applied to a particular coding unit, the above-mentioned syntax elements for the corresponding unsupported MVD resolution may not be signaled.

[0218] In some of the example implementations above, fractional resolution may be independent of different levels of MVD. In other words, a similar option for motion vector resolution may be provided using a predefined number of "mv_fr" bits and "mv_hp" bits to signal the fractional MVD of non-zero MVD components, regardless of the magnitude of the motion vector difference.

[0219] However, in some other example implementations, the resolution of motion vector differences in various MVD amplitude levels may be different. Specifically, high-resolution MVD for large MVD amplitudes of higher MVD levels may not provide statistically significant improvements in compression efficiency. Therefore, for a larger MVD amplitude range corresponding to a higher MVD amplitude level, the MVD can be encoded with a reduced resolution (integer pixel resolution or fractional pixel resolution). Similarly, typically for larger MVD values, the MVD can be encoded with a reduced resolution (integer pixel resolution or fractional pixel resolution). This MVD level-related or MVD amplitude-related MVD resolution can generally be referred to as an adaptive MVD resolution, an amplitude-related adaptive MVD resolution, or an amplitude-related MVD resolution. The term "resolution" can also be referred to as "pixel resolution". Adaptive MVD resolution can be implemented in various ways as described below by example implementations to achieve overall better compression efficiency. In particular, due to the statistical observation that processing the MVD resolution of a large magnitude or high level MVD at a similar level to the MVD resolution of a low magnitude or low level MVD in a non-adaptive manner may not significantly increase the coding efficiency of the inter-frame prediction residual of blocks with large magnitude or high level MVD, the reduction in the number of signaling bits by targeting a less precise MVD may be greater than the additional bits required to encode the inter-frame prediction residual due to such less precise MVD. In other words, using a higher MVD resolution for a large magnitude or high level MVD may not produce a significant coding gain relative to using a lower MVD resolution.

[0220] In some general example implementations, the pixel resolution or precision of the MVD may decrease or may not increase as the MVD level increases. A decreased pixel resolution of the MVD corresponds to a coarser MVD (or a larger step size from one MVD level to the next MVD level). In some implementations, the correspondence between MVD pixel resolution and MVD level may be specified, predefined, or preconfigured, and thus the correspondence may not need to be signaled in the coded bitstream.

[0221] In some example implementations, the MV levels of Table 3 may each be associated with a different MVD pixel resolution.

[0222] In some example implementations, each MVD level may be associated with a single allowed resolution. In some other implementations, one or more MVD levels may be associated with two or more optional MVD pixel resolutions. Thus, the signal in the bitstream for the current MVD component having such an MVD level may be followed by additional signaling indicating the optional pixel resolutions selected for the current MVD component.

[0223] In some example implementations, the adaptively allowed MVD pixel resolutions may include, but are not limited to, 1 / 64 pixel (pel), 1 / 32 pixel, 1 / 16 pixel, 1 / 8 pixel, 1 / 4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels, 4 pixels ... (in descending order of resolution). Therefore, each of the ascending MVD levels may be associated with one of these MVD pixel resolutions in a non-ascending manner. In some implementations, an MVD level may be associated with two or more of the above resolutions, and the higher resolution may be lower than or equal to the lower resolution of the previous MVD level. For example, if MV_CLASS_3 of Table 4 is associated with an optional 1 pixel resolution and a 2 pixel resolution, the highest resolution that may be associated with MV_CLASS_4 of Table 4 may be 2 pixels. In some other implementations, the highest allowable resolution of an MV level may be higher than the lowest allowable resolution of the previous (lower) MV level. However, the general case of the allowable resolutions of the ascending MV levels may only be non-ascending.

[0224] In some implementations, when fractional pixel resolutions higher than 1 / 8 pixel are allowed, the "mv_fr" and "mv_hp" signaling may be extended to more than a total of 3 fractional bits accordingly.

[0225] In some implementations, fractional pixel resolutions are allowed only for MVD levels that are lower than or equal to a MVD level threshold. For example, fractional pixel resolutions may be allowed only for MVD_CLASS_0 of Table 4 and not for all other MV levels. Similarly, fractional pixel resolutions may be allowed only for MVD levels that are lower than or equal to any of the other MV levels of Table 4. For other MVD levels that are higher than the MVD level threshold, only integer pixel resolutions are allowed for the MVD. In this way, for MVDs that are signaled to have MVD levels that are higher than or equal to the MVD level threshold, it may not be necessary to signal one or more of the fractional resolution signaling, such as the "mv_fr" bit and / or the "mv_hp" bit. For MVD levels with resolutions lower than 1 pixel, the number of bits in the "mv-bit" signaling may be further reduced. For example, for MV_CLASS_5 in Table 4, the range of MVD pixel offsets is (32, 64], so 5 bits are required to signal the entire range in the case of 1 pixel resolution. However, if MV_CLASS_5 is associated with 2-pixel MVD resolution (lower than 1 pixel resolution), "mv-bit" may require 4 bits instead of 5 bits, and after "mv_class" is signaled as MV_CLASS_5, "mv-fr" and "mv-hp" do not need to be signaled.

[0226] In some example implementations, fractional pixel resolution may be allowed only for MVDs with integer values ​​below an integer pixel value threshold. For example, fractional pixel resolution may be allowed only for MVDs less than 5 pixels. Corresponding to this example, fractional resolution may be allowed for MV_CLASS_0 and MV_CLASS_1 of Table 4 but not for all other MV levels. For another example, fractional pixel resolution may be allowed only for MVDs less than 7 pixels. Corresponding to this example, fractional resolution may be allowed for MV_CLASS_0 and MV_CLASS_1 of Table 4 (with a range below 5 pixels) but not for MV_CLASS_3 and higher levels (with a range above 5 pixels). For MVDs belonging to MV_CLASS_2 whose pixel range includes 5 pixels, fractional pixel resolution of the MVD may or may not be allowed, depending on the "mv-bit" value. If the "mv-bit" value is signaled as 1 or 2 (so that the integer part of the signaled MVD is 5 or 6 - calculated as the start of the pixel range of MV_CLASS_2 and with an offset of 1 or 2 as indicated by "mv-bit"), fractional pixel resolution may be allowed. Otherwise, if the "mv-bit" value is signaled as 3 or 4 (so that the integer part of the signaled MVD is 7 or 8), fractional pixel resolution may not be allowed.

[0227] In some other implementations, only a single MVD value may be allowed for MV levels that are equal to or higher than an MV level threshold. For example, such an MV level threshold may be MV_CLASS_2. Therefore, only MV_CLASS_2 and above may be allowed to have a single MVD value and not have fractional pixel resolution. The single allowed MVD values ​​for these MV levels may be predefined. In some examples, the allowed single value may be a higher end value of the corresponding ranges of these MV levels in Table 4. For example, MV_CLASS_2 to MV_CLASS_10 may be higher than or equal to the level threshold of MV_CLASS_2, and the single allowed MVD values ​​for these levels may be predefined as 8, 16, 32, 64, 128, 256, 512, 1024, and 2048, respectively. In some other examples, the allowed single value may be an intermediate value of the corresponding ranges of these MV levels in Table 4. For example, MV_CLASS_2 to MV_CLASS_10 may be above the class threshold, and the single allowed MVD values ​​for these classes may be predefined as 3, 6, 12, 24, 48, 96, 192, 384, 768, and 1536, respectively. Any other value within the range may also be defined as the single allowed resolution for the corresponding MVD class.

[0228] In the above implementation, when the signaled "mv_class" is equal to or above the predefined MVD level threshold, only the "mv_class" signaling is sufficient to determine the MVD value. Then, "mv_class" and "mv_sign" will be used to determine the magnitude and direction of the MVD.

[0229] Therefore, when the MVD is signaled for only one reference frame (from reference frame list 0 or reference frame list 1, but not from both) or when the MVD is signaled for two reference frames jointly, the accuracy (or resolution) of the MVD can depend on the associated motion vector difference level and / or the magnitude of the MVD in Table 3.

[0230] In some other implementations, the pixel resolution or precision of the MVD may decrease or may not increase as the MVD amplitude increases. For example, the pixel resolution may depend on the integer portion of the MVD amplitude. In some implementations, fractional pixel resolution may only be allowed for MVD amplitudes less than or equal to an amplitude threshold. For a decoder, the integer portion of the MVD amplitude may first be extracted from the bitstream. Then, the pixel resolution may be determined, and then a decision may be made regarding whether there are any fractional MVDs in the bitstream and whether the fractional MVDs need to be parsed (e.g., if the fractional pixel resolution is not allowed for a particular extracted MVD integer amplitude, then the fractional MVD bits may not be included in the bitstream that needs to be extracted). The example implementations above with respect to MVD level-dependent adaptive MVD pixel resolution are applicable to MVD amplitude-dependent adaptive MVD pixel resolution. For a specific example, an MVD level above or including an amplitude threshold may be allowed to have only one predefined value.

[0231] For the current coding block, in single inter prediction mode or composite inter prediction mode and regardless of whether one or more motion vectors used for inter prediction are based on MVD, once the motion vector of the current coding block relative to the reference frame is determined, a predictor block can be generated (in the encoder or decoder) based on the reference frame. In some implementations, the predictor block in the corresponding prediction frame can be generated or derived by interpolating samples in the prediction frame based on the corresponding motion vector. For example, in some cases, the motion vector corresponding to the reference frame for the current coding block can be fractional. Specifically, the motion vector indicated as one of the multiple reference motion vectors in the motion vector reference list can be fractional in one or both of the horizontal dimension or the vertical dimension. For another example, the MVD can be fractional in one or both of the horizontal dimension or the vertical dimension. In this case, the predictor block as indicated by the motion vector in the corresponding reference frame may not directly coincide with a particular sample block in the reference frame, but needs to be interpolated between sample blocks.

[0232] For example, such interpolation can be implemented by applying a filter called an interpolation filter. The interpolation filter applied to a specific current coding block can be selected from a set of predetermined or preconfigured interpolation filters. Interpolation filtering can be implemented separately on each of the two dimensions (horizontal dimension and vertical dimension). Therefore, the interpolation filter can be implemented as a one-dimensional filter that can be applied in the horizontal dimension or the vertical dimension. In some implementations, some of these interpolation filters may include a predefined finite impulse response (Finite Impulse Response, FIR) filter. For example, a predetermined or preconfigured interpolation filter may include, but is not limited to, a 6-tap conventional FIR filter (REGULAR), a 6-tap FIR filter (SMOOTH), an 8-tap FIR filter (SHARP) and a 2-tap BILINEAR filter, etc.

[0233] In some implementations, when dual filter mode is enabled, interpolation filters in the horizontal dimension and the vertical dimension can be implemented independently. Whether dual filter mode is enabled can be signaled at various levels (e.g., in a sequence header, a picture header, a frame header, a slice header, a coding tree level, a macroblock level, a block level). For dual filter mode, the interpolation filters used in each of the two dimensions can be separately signaled at various levels above. Otherwise, when dual filter mode is not enabled, only one interpolation filter can be selected for use / sharing by the coding block in both the horizontal dimension and the vertical dimension. The interpolation filters used can be separately signaled at various levels above. Two groups of interpolation filters that can be selected in dual filter mode and in single filter mode can be separately specified. The two groups of interpolation filters can be filters of the same group or filters of different groups.

[0234] Statistical studies have shown that when a motion vector difference (MVD) and a reference motion vector (MV) are used to derive the actual MV of a coding block, depending on the magnitude of the MVD and / or the pixel resolution, certain types of interpolation filters can help provide better coding gain than other types of interpolation filters when used to generate a prediction block. Therefore, for certain MVD magnitude or pixel resolution values ​​or value ranges, a reduced set of interpolation filters can be predefined as candidate interpolation filters to improve signaling efficiency while maintaining negligible coding gain degradation.

[0235] Likewise, statistical studies have also shown that, depending on the magnitude and / or pixel resolution of the actual MV of the block, certain types of interpolation filters can help provide better coding gain than other types of interpolation filters when used to generate a prediction block. Therefore, for certain MV magnitudes or pixel resolution values ​​or value ranges, a reduced set of interpolation filters can be predefined as candidate interpolation filters to improve signaling efficiency while maintaining negligible coding gain reduction. The actual MV can be the MV directly signaled in the video stream, or can be signaled as a reference motion vector in the DRL described above, or can be a combination of the reference motion vector in the DRL and the MVD.

[0236] In some implementations, the MV amplitude or MVD amplitude can be used as the basis for selecting an interpolation filter. For example, one or more values ​​or value ranges of the MV amplitude or MVD amplitude can be predefined. Each of the values ​​or value ranges of the MV amplitude or MVD amplitude can be pre-associated with a predetermined interpolation filter subset in the entire candidate interpolation filter set. Therefore, the mapping between the values ​​or value ranges of the MV amplitude or MVD amplitude and the interpolation filter subset can be predefined and is therefore known to the encoder and decoder. Then, the interpolation filter subset corresponding to the value or value range of a specific MV amplitude or MVD amplitude can be determined by the encoder during the encoding process, and can also be derived by the decoder based on the value or value range of the corresponding MV amplitude or MVD amplitude, without relying on any additional signaling in the bitstream.

[0237] Similarly, MV pixel resolution or MVD pixel resolution can be used to select interpolation filters. For example, one or more values ​​or value ranges of MV pixel resolution or MVD pixel resolution can be predefined. Each of the values ​​or value ranges of MV pixel resolution or MVD pixel resolution can be associated with a predetermined subset of interpolation filters in the entire candidate interpolation filter set. Therefore, the mapping between the values ​​or value ranges of MV pixel resolution or MVD pixel resolution and the subset of interpolation filters can be predefined and therefore known to the encoder and decoder. Then, the subset of interpolation filters corresponding to the value or value range of a specific MV pixel resolution or MVD pixel resolution can be determined by the encoder during the encoding process, and can also be derived by the decoder based on the value or value range of the corresponding MV pixel resolution or MVD pixel resolution, without relying on any additional signaling in the bitstream.

[0238] The number of interpolation filters in each of the above interpolation filter subsets for different MV amplitudes or pixel resolutions or MVD amplitudes or pixel resolutions may be different. In some implementations, the above specific interpolation filter subset may include two or more candidate interpolation filters. In this case, for the corresponding MV amplitude or pixel resolution or MVD amplitude or pixel resolution value or value range, the encoder may also select a specific filter from the interpolation filter subset at various levels (e.g., block level, slice level, frame level, picture level, sequence level, etc.) (e.g., by selecting a filter that provides a better coding gain), and the corresponding signaling (referred to as an interpolation filter selection indicator) may be included in the bitstream to indicate that a specific filter is selected from the interpolation filter subset predefined by the value or value range of the MV amplitude or pixel resolution or MVD amplitude or pixel resolution. Because the interpolation filter subset includes a smaller possible number of filters compared to the entire candidate interpolation filter set, the signaling for selecting a specific filter within the subset requires fewer bits, thereby resulting in improved signaling efficiency.

[0239] In some other implementations, the above specific subset of interpolation filters may include only a single default interpolation filter. In this case, the value or value range of the corresponding MVD amplitude is directly mapped to the default predefined interpolation filter. Therefore, there is no need to include additional signaling in the bitstream for selecting a specific predefined interpolation filter. The predefined default interpolation filter can be identified directly based on the value or value range of the MVD amplitude.

[0240] In some example implementations, threshold levels for MV magnitude or pixel resolution or MVD magnitude or pixel resolution may be predefined to delineate the value ranges for the above various magnitudes or pixel resolutions. In some specific implementations, a single threshold level may be predefined to delineate the MV magnitude or pixel resolution or MVD magnitude or pixel resolution into two ranges, one range being above the threshold level and the other not being above the threshold level, the two ranges each corresponding to a predefined subset of interpolation filters.

[0241] For example, a threshold level of MV amplitude or MVD amplitude may be predefined. For blocks having MV amplitude or MVD amplitude above the threshold level, interpolation filters of a first predefined subset may be used as candidate interpolation filters, while for blocks having MV amplitude or MVD amplitude not above the threshold level, interpolation filters of a second predefined subset may be used as candidate interpolation filters. Selection of a specific interpolation filter within a subset of filters may be additionally signaled in the bitstream. For a specific example, a first subset of interpolation filters corresponding to MV amplitude or MVD amplitude above the threshold level may include a single default interpolation filter, while a second subset of interpolation filters corresponding to MV amplitude or MVD amplitude not above the threshold level may include an entire (or smaller) set of candidate interpolation filters. In such an implementation, if the MV amplitude or MVD amplitude is above the threshold level, additional signaling for interpolation filter selection is not required.

[0242] For another example, a threshold level for MV pixel resolution or MVD pixel resolution may be predefined. For blocks having MV pixel resolution or MVD pixel resolution below the threshold level, interpolation filters of a first predefined subset may be used as candidate interpolation filters, and for blocks having MV pixel resolution or MVD pixel resolution not below the threshold level, interpolation filters of a second predefined subset may be used as candidate interpolation filters. Selection of a specific interpolation filter within a subset of filters may be additionally signaled in the bitstream. For a specific example, a first subset of interpolation filters corresponding to a MV pixel resolution or MVD pixel resolution below the threshold level may include a single default interpolation filter, and a second subset of interpolation filters corresponding to a MV pixel resolution or MVD pixel resolution not below the threshold level may include an entire (or smaller) set of candidate interpolation filters. In such an implementation, if the MV pixel resolution or MVD pixel resolution is below the threshold level, no additional signaling for interpolation filter selection is required.

[0243] In some example implementations, the MVD amplitude level may be represented by the MV class index described above with respect to Table 4. Thus, the individual MV classes exemplarily shown in Table 4 may be depicted as two additional class ranges using a set of MV threshold classes. Each MV class range may correspond to a default predefined interpolation filter subset. For a specific example, a single MV class threshold may be predefined to provide two MV class ranges. Specifically, the MV class threshold may be set to MV_CLASS_0. Alternatively, the MV class threshold may be set to MV_CLASS_1 or other MV classes. Taking the MV class threshold MV_CLASS_1 as an example, a class range of MV classes (including MV_CLASS_2 to MV_CLASS_10) above MV_CLASS_1 may be associated with a first predefined default interpolation filter subset, while MV classes MV_CLASS_0 and MV_CLASS_1 may be associated with a second predefined default interpolation filter subset. For example, a first interpolation filter subset corresponding to MV_CLASS_2 to MV_CLASS_10 may include a single default interpolation filter. For example, the second subset of interpolation filters corresponding to MV_CLASS_0 and MV_CLASS_1 may include the entire (or a smaller) set of candidate interpolation filters.

[0244] In some example implementations, the MVD pixel resolution level may be divided into various predefined ranges, each of which corresponds to a predefined subset of interpolation filters. In the adaptive MVD implementation described above, the MVD pixel resolution of a particular block may be one of the candidate pixel resolutions in the entire set of candidate pixel resolutions, including but not limited to 1 / 64, 1 / 32, 1 / 16, 1 / 8, 1 / 4, 1 / 2, 1, 2, 4, ... Although any type of division of the pixel resolution range may be predefined, in some specific implementations, the available MVD pixel resolutions may be divided into two ranges, the first range being MVD pixel resolutions of 1 pixel and below (lower pixel resolutions correspond to higher pixel values, so the first range includes 1, 2, 4, ... pixel resolutions), and the second range being MVD pixel resolutions above 1 pixel (including all fractional pixel resolutions above). The first range may be associated with a first default subset of interpolation filters, and the second range may be associated with a second default subset of interpolation filters. The first subset may, for example, include a single default interpolation filter, and the second subset may, for example, include the entire (or smaller) set of candidate interpolation filters.

[0245] In some specific example implementations, when fractional MVD is not allowed for coding blocks, the interpolation filter subset may not be the entire candidate interpolation filter set. In other words, in such an implementation, only a subset of interpolation filters is allowed instead of the entire candidate interpolation filter set. Specifically, when fractional MVD is not allowed, there may be only a single subset of interpolation filters for selection. A single subset of interpolation filters may be part of the entire set of allowable candidate interpolation filters. For example, the interpolation filter subset may include a REGULAR filter and a SMOOTH filter. In some other example implementations, a single subset of interpolation filters may include a single default interpolation filter. In other words, for an MVD implementation in which fractional MVD is not allowed, a single default interpolation filter may be predefined. In such an implementation, when fractional MVD is not allowed in an adaptive MVD implementation, signaling for interpolation filter selection may not be required in the bitstream. A single default interpolation filter may be, for example, one of a REGULAR filter, a SMOOTH filter, a SHARP filter, or a BILINEAR filter.

[0246] In some other implementations, as described above, the correspondence or mapping between the MVD amplitude (represented by the MV level index of Table 4) and the adaptive MVD pixel resolution may be predefined. In this case, the predefined division of the MV level into different ranges for the purpose of mapping to different interpolation filter subsets may be equivalent to or coordinated with the division of the MVD pixel resolution into different pixel resolution ranges.

[0247] The above example embodiments recognize that statistical studies have shown that at higher MVD amplitudes or lower MVD pixel resolutions, using the entire set of available candidate interpolation filters for selection does not provide very high coding gain while requiring higher signaling overhead compared to using a smaller set of candidate interpolation filters or even a single default interpolation filter.

[0248] In the above various example implementations, the entire candidate interpolation filter set may include, but is not limited to, one or more of a 6-tap conventional FIR filter (REGULAR), a 6-tap FIR filter (SMOOTH), an 8-tap FIR filter (SHARP), and a 2-tap BILINEAR filter, etc. The interpolation filter subset may be any combination of filters within the entire candidate interpolation filter set. Specific examples of the interpolation filter subset may include a REGULAR filter and a SMOOTH filter. When the filter subset includes a single default interpolation filter, such a default interpolation filter may be any filter within the entire candidate interpolation filter set. In particular, the single default filter may be predefined as one of a REGULAR filter and a SMOOTH filter.

[0249] In some of the above implementations, the signaling for selecting an interpolation filter (e.g., in a subset of interpolation filters or in the entire interpolation filter set) can be entropy coded in the context of an MV amplitude or pixel resolution or an MVD amplitude or pixel resolution. For a specific example, the entropy coding context can be determined based on whether an adaptive MVD resolution is applied and / or based on an MVD pixel resolution or an MV level of the MVD. For example, when an adaptive MVD resolution is applied, the MVD pixel resolution or the MV level can be divided into a plurality of ranges associated with a plurality of interpolation filter subsets. The signaling for selecting the interpolation filter in each subset can be encoded using a specific context set. Therefore, each MVD pixel resolution range or MV level range or interpolation filter subset will correspond to a coding context set. These coding context sets can be determined separately to maximize coding efficiency.

[0250] In some example implementations, these MVD pixel resolution ranges or MV level ranges may be delineated by multiple thresholds. In one example implementation, two ranges delineated by a single threshold may be predefined. The threshold (MVD pixel resolution threshold or MV level index threshold) is used to determine the two ranges and the corresponding predefined interpolation filter subsets. Signaling within a first interpolation filter subset may be encoded under a first context set, while signaling for a second interpolation filter subset may be encoded under a second context set. Such an implementation may provide further coding gain because the number of filters within each subset is different and the filter selection statistics within each subset may also be different relative to many coding blocks.

[0251] Fig.18A flowchart 1800 of an example method for selecting and signaling an interpolation filter that follows the basic principles of the above implementation is shown. The example method flow starts at 1801. In S1810, a flag in a video stream is received. In S1820, based on the received flag, it is determined that a video block is to be inter-predicted based on a motion vector (MV) through a reference frame. In S1830, the MV is derived based on a reference MV and an MVD (MV difference) of the video block. In S1840, a signal indicating an interpolation filter is derived, wherein the signal is based on at least one of an amplitude and a resolution of the MVD. In S1850, an interpolation filter is selected based on the signal. In S1860, inter-prediction is performed for the video block through the reference frame based on the MV and the selected interpolation filter. The example method ends at S1899.

[0252] In some examples, deriving the signal includes:

[0253] determining a magnitude of the MVD based on an MV level index associated with the video block, the MV level index being extracted from the video stream; and

[0254] When the MV level index is not lower than a MV level index threshold, a signal for using a predefined interpolation filter by default is derived.

[0255] In some examples, the predefined interpolation filter is included in a plurality of interpolation filters that form a pool of candidate interpolation filters for video blocks having MV level indices below the MV level index threshold.

[0256] In some examples, the predefined interpolation filter includes a REGULAR interpolation filter, a SMOOTH interpolation filter, a SHARP interpolation filter, or a BILINEAR interpolation filter.

[0257] In some examples, deriving the signal includes:

[0258] Determining to apply an adaptive pixel resolution to the MVD;

[0259] determining a pixel resolution of the MVD among a plurality of adaptive MVD pixel resolutions; and

[0260] In response to the pixel resolution of the MVD being at one or more predefined levels, a signal for defaulting to use a predefined interpolation filter is derived.

[0261] In some examples, the one or more predefined levels include an MVD resolution level that is no higher than a predefined MVD pixel resolution threshold.

[0262] In some examples, the pixel resolution of the MVD is determined based on an MV level index of the video block extracted from the video stream.

[0263] In some examples, the predefined MVD pixel resolution threshold corresponds to a MV level index threshold.

[0264] In some examples, the predefined interpolation filter is included in a plurality of interpolation filters that form a pool of candidate interpolation filters for video blocks having an adaptive MVD pixel resolution different than the one or more predefined levels.

[0265] In some examples, the predefined interpolation filter includes one of a REGULAR interpolation filter, a SMOOTH interpolation filter, a SHARP interpolation filter, or a BILINEAR interpolation filter.

[0266] In some examples, deriving the signal includes:

[0267] Determining, from the video stream, an MV level index corresponding to the magnitude of the MVD;

[0268] When the MV level index of the MV is not lower than a predefined MV level index threshold, using a predefined interpolation filter subset in the interpolation filter set by default; and

[0269] A selection indicator is extracted from the video stream as a signal for selecting the interpolation filter from the subset of predefined interpolation filters.

[0270] In some examples, deriving the signal includes:

[0271] Determining to apply adaptive pixel resolution to the MVD;

[0272] determining a pixel resolution of the MVD among a plurality of adaptive MVD pixel resolutions;

[0273] In response to the pixel resolution of the MVD being not greater than a predefined MVD pixel resolution threshold, using a predefined interpolation filter subset in the interpolation filter set by default; and

[0274] A selection indicator is extracted from the video stream as a signal for selecting the interpolation filter from the subset of predefined interpolation filters.

[0275] In some examples, deriving the signal includes:

[0276] determining the MVD that applies adaptive pixel resolution to the video block;

[0277] Determining a pixel resolution of the MVD among a plurality of adaptive MVD pixel resolutions, or determining an MV level index of the MV among a plurality of MV level indexes;

[0278] identifying a context set for encoding an interpolation filter selection indicator based on a pixel resolution of the MVD or the MV level index; and

[0279] The video stream is decoded according to the context set to obtain the interpolation filter selection indicator as the signal.

[0280] In some examples, deriving the signal includes:

[0281] Determining to apply an adaptive pixel resolution to the MVD;

[0282] Determining an MV level index of the video block among a plurality of MV level indexes;

[0283] When the MV level index is not less than an MV level index threshold, decoding the video stream using a first context set to obtain an interpolation filter selection indicator as the signal; and

[0284] When the MV level index is less than the MV level index threshold, the video stream is decoded using a second context set different from the first context set to obtain the interpolation filter selection indicator as the signal.

[0285] In some examples, deriving the signal includes deriving the signal for defaulting to use the interpolation filter in response to determining that an MVD pixel resolution of the MVD is non-fractional.

[0286] In some examples, deriving the signal includes:

[0287] In response to determining that the MVD pixel resolution of the MVD is non-fractional, using by default a predefined subset of interpolation filters in a set of interpolation filters; and

[0288] A selection indicator is extracted from the video stream as the signal for selecting the interpolation filter from the subset of predefined interpolation filters.

[0289] In some examples, the predefined subset of interpolation filters includes REGULAR interpolation filters and SMOOTH interpolation filters.

[0290] According to an embodiment of the present disclosure, a device for processing a video block of a video stream is also provided. The device includes: a receiving unit configured to receive a flag in the video stream and store the video stream in the buffer memory; a determining unit configured to determine, based on the received flag, whether to perform inter-frame prediction on the video block according to a motion vector (MV) through a reference frame; a first deriving unit configured to derive the MV according to a reference MV and an MVD (MV difference) of the video block; a second deriving unit configured to derive a signal indicating an interpolation filter, wherein the signal is based on at least one of an amplitude and a resolution of the MVD; a selecting unit configured to select the interpolation filter according to the signal; and an inter-frame prediction unit configured to perform inter-frame prediction on the video block through the reference frame according to the MV and the selected interpolation filter.

[0291] According to an embodiment of the present disclosure, a computer device is also provided, comprising a processor and a memory. The memory is used to store program code and transmit the program code to the processor; the processor is used to execute according to the instructions in the program code: receiving a flag in the video stream; determining to perform inter-frame prediction on the video block through a reference frame according to a motion vector (MV) based on the received flag; deriving the MV according to a reference MV and MVD (MV difference) of the video block; deriving a signal indicating an interpolation filter, wherein the signal is based on at least one of the magnitude and resolution of the MVD; selecting an interpolation filter according to the signal; and performing inter-frame prediction on the video block through the reference frame according to the MV and the selected interpolation filter.

[0292] In the embodiments and implementations of the present disclosure, any steps and / or operations can be combined or arranged in any amount or order as needed. Two or more steps and / or operations in the steps and / or operations can be performed in parallel. The embodiments and implementations in the present disclosure can be used alone or in combination in any order. In addition, each of the method (or embodiment), encoder and decoder can be implemented by a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transient computer-readable medium. The embodiments in the present disclosure can be applied to luminance blocks or chrominance blocks. The term block can be interpreted as a prediction block, a coding block or a coding unit, i.e., a CU. The term block can also be used here to refer to a transform block. In the following items, when the size of a block is specified, it can refer to the width or height of the block, or the maximum value of the width and height of the block, or the minimum value of the width and height of the block, or the area size of the block (width*height), or the aspect ratio of the block (width: height or height: width).

[0293] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.19 A computer system (1900) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0294] Computer software may be encoded using any suitable machine code or computer language, which may be subjected to mechanisms such as assembly, compilation, and linking to create code comprising instructions that may be executed directly by one or more computer central processing units (CPU), graphics processing units (GPU), etc., or by interpretation, microcode execution, etc.

[0295] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0296] Fig.19 The components for the computer system (1900) shown in the example are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components shown in the exemplary embodiment of the computer system (1900).

[0297] The computer system (1900) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), olfactory input (not depicted). The human-machine interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0298] Input human interface devices may include one or more of the following (only one of each is depicted): keyboard (1901), mouse (1902), trackpad (1903), touch screen (1910), data gloves (not shown), joystick (1905), microphone (1906), scanner (1907), and camera (1908).

[0299] The computer system (1900) may also include certain human interface output devices. Such human interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include: tactile output devices (e.g., tactile feedback via a touch screen (1910), a data glove (not shown), or a joystick (1905), but there may also be tactile feedback devices that do not function as input devices); audio output devices (e.g., speakers (1909), headphones (not depicted)); visual output devices (e.g., screens (1910), including CRT (Cathode Ray Tube, CRT) screens, LCD (Liquid Crystal Display, LCD) screens, plasma screens, OLED (Organic Light Emitting Diode, OLED) screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which may be able to output two-dimensional visual output or more than three-dimensional output by means such as stereoscopic image output; virtual reality glasses (not depicted); holographic displays and cigarette cans (not depicted)); and printers (not depicted).

[0300] The computer system (1900) may also include storage devices and their associated media accessible to humans, such as optical media including CD / DVD ROM (Read Only Memory, ROM) / RW (1920) with CD / DVD and other media (1921), thumb drives (1922), removable hard drives or solid-state drives (1923), legacy magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC (Application Specific Integrated Circuit, ASIC) / PLD (Programmable Logic Device, PLD) based devices such as security dongles (not depicted), etc.

[0301] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0302] The computer system (1900) may also include an interface (1954) to one or more communication networks (1955). The network may be, for example, wireless, wired, optical. The network may also be local, wide, metropolitan, vehicle and industrial, real-time, delay-tolerant, etc. Examples of networks include: local area networks such as Ethernet; wireless LAN (Local Area Network, LAN); cellular networks including GSM (Global System for Mobile Communications, GSM), 3G (the Third Generation, 3G), 4G (the Fourth Generation, 4G), 5G (the Fifth Generation, 5G), LTE (Long Term Evolution, LTE), etc.; TV wired or wireless wide area digital networks including cable TV, satellite TV and terrestrial broadcast TV; vehicle and industrial networks including CAN bus, etc. Some networks typically require an external network interface adapter attached to some general data port or peripheral bus (1949) (such as, for example, a USB port of the computer system (1900)); other networks are typically integrated into the core of the computer system (1900) by attaching to a system bus as described below (e.g., to an Ethernet interface in a PC computer system or to a cellular network interface in a smartphone computer system). Using any of these networks, the computer system (1900) can communicate with other entities. Such communications can be one-way receive only (e.g., broadcast TV), one-way send only (e.g., a CAN bus to some CAN bus devices), or two-way, for example, to other computer systems using a local area digital network or a wide area digital network. Certain protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

[0303] The above-mentioned human-machine interface devices, human-accessible storage devices, and network interfaces may be attached to the core ( 1940 ) of the computer system ( 1900 ).

[0304] The core (1940) may include one or more central processing units (CPUs) (1941), graphics processing units (GPUs) (1942), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1943), hardware accelerators for certain tasks (1944), graphics adapters (1950), etc. These devices, along with read-only memory (ROM) (1945), random access memory (1946), internal mass storage devices (1947) such as internal non-user accessible hard drives, SSDs (Solid State Drives, SSDs), etc., may be connected via a system bus (1948). In some computer systems, the system bus (1948) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached to the core's system bus (1948) directly or via a peripheral bus (1949). In an example, a screen (1910) may be connected to a graphics adapter (1950). The architecture of the peripheral bus includes PCI (Peripheral Component Interconnect / Interface, PCI), USB (Universal Serial Bus, USB), etc.

[0305] The CPU (1941), GPU (1942), FPGA (1943) and accelerator (1944) can execute certain instructions, which can be combined to form the above-mentioned computer code. The computer code can be stored in ROM (1945) or RAM (Random Access Memory, RAM) (1946). Transient data can also be stored in RAM (1946), while permanent data can be stored in, for example, an internal large-capacity storage device (1947). Fast storage and retrieval of any storage device in the storage device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (1941), GPUs (1942), large-capacity storage devices (1947), ROMs (1945), RAMs (1946), etc.

[0306] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or the medium and computer code may be of a type well known and available to those skilled in the art of computer software.

[0307] As a non-limiting example, a computer system (1900) having an architecture and in particular a core (1940) can provide functionality due to a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software contained in one or more tangible computer-readable media. Such a computer-readable medium can be a medium associated with a user-accessible mass storage device as introduced above, as well as certain storage devices of the core (1940) having non-transitory properties, for example, a core internal mass storage device (1947) or a ROM (1945). Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core (1940). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the core (1940) and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in the RAM (1946) and modifying such a data structure according to a process defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of hard-wired logic or otherwise embodied in circuitry (e.g., accelerator (1944)) that may replace or operate in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and references to logic may also include software. Where appropriate, references to computer-readable media may include circuitry (e.g., an integrated circuit (IC)) storing software for execution, circuitry implementing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0308] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various alternative equivalents that fall within the scope of the present disclosure. It will therefore be appreciated that those skilled in the art will be able to conceive of many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

[0309] Appendix A: Acronyms

[0310] JEM: Joint Development Model

[0311] VVC: Versatile Video Coding

[0312] BMS: Benchmark Set

[0313] MV: Motion Vector

[0314] HEVC: High Efficiency Video Coding

[0315] SEI: Supplementary Enhancement Information

[0316] VUI: Video Availability Information

[0317] GOP: Group of Pictures

[0318] TU: Transform Unit

[0319] PU: prediction unit

[0320] CTU: Coding Tree Unit

[0321] CTB: Coding Tree Block

[0322] PB: prediction block

[0323] HRD: Hypothesized Reference Decoder

[0324] SNR: Signal to Noise Ratio

[0325] CPU: Central Processing Unit

[0326] GPU: Graphics Processing Unit

[0327] CRT: cathode ray tube

[0328] LCD: Liquid Crystal Display

[0329] OLED: Organic Light Emitting Diode

[0330] CD: Compact Disc

[0331] DVD: Digital Video Disc

[0332] ROM: Read Only Memory

[0333] RAM: Random Access Memory

[0334] ASIC: Application-Specific Integrated Circuit

[0335] PLD: Programmable Logic Device

[0336] LAN: Local Area Network

[0337] GSM: Global System for Mobile Communications

[0338] LTE: Long Term Evolution

[0339] CANBus: Controller Area Network Bus

[0340] USB: Universal Serial Bus

[0341] PCI: Peripheral Component Interconnect

[0342] FPGA: Field Programmable Gate Array

[0343] SSD: Solid State Drive

[0344] IC: Integrated Circuit

[0345] HDR: High Dynamic Range

[0346] SDR: Standard Dynamic Range

[0347] JVET: Joint Video Exploration Team

[0348] MPM: Most Probable Mode

[0349] WAIP: Wide Angle Intra Prediction

[0350] CU: Coding Unit

[0351] PU: prediction unit

[0352] TU: Transform Unit

[0353] CTU: Coding Tree Unit

[0354] PDPC: Position Dependent Prediction Combination

[0355] ISP: Intra-frame sub-partitioning

[0356] SPS: Sequence parameter settings

[0357] PPS: Picture Parameter Set

[0358] APS: Adaptive Parameter Set

[0359] VPS: Video Parameter Set

[0360] DPS: Decoding Parameter Set

[0361] ALF: Adaptive Loop Filter

[0362] SAO: Sample Adaptive Offset

[0363] CC-ALF: Cross-Component Adaptive Loop Filter

[0364] CDEF: Constrained Directional Enhancement Filter

[0365] CCSO: Cross Component Sample Offset

[0366] LSO: Local Sample Offset

[0367] LR: Loop Restoration Filter

[0368] AV1: Alliance for Open Media Video 1

[0369] AV2: Alliance for Open Media Video 2

[0370] MVD: Motion Vector Difference

[0371] CfL: Predicting Chroma from Luminance

[0372] SDT: Semi-Decoupled Tree

[0373] SDP: Semi-Decoupled Partitioning

[0374] SST: Semi-Separating Tree

[0375] SB: Super Block

[0376] IBC (or IntraBC): Intra Block Copy

[0377] CDF: Cumulative density function

[0378] SCC: Screen Content Coding

[0379] GBI: Generalized Bidirectional Prediction

[0380] BCW: Bidirectional Prediction with CU Level Weights CIIP: Combined Intra-Inter Prediction

[0381] POC: Image Sequential Counting

[0382] RPS: Reference Picture Set

[0383] DPB: Decoded Picture Buffer

[0384] MMVD: Merge mode with motion vector difference

Claims

1. A method for processing a video block of a video stream, characterized in that: include: Receiving a logo in the video stream; Determining, based on the received flag, to inter-predict the video block using a reference frame according to a motion vector (MV); deriving the MV according to a reference MV and MVD (MV difference) of the video block; deriving a signal indicative of an interpolation filter, wherein the signal is based on at least one of a magnitude and a resolution of the MVD; selecting the interpolation filter according to the signal; and performing inter-frame prediction on the video block through the reference frame according to the MV and the selected interpolation filter; Deriving the signal comprises: determining a magnitude of the MVD based on an MV level index associated with the video block, the MV level index being extracted from the video stream; and When the MV level index is below an MV level index threshold, deriving a signal for using a first subset of interpolation filters, the first subset of interpolation filters comprising a REGULAR interpolation filter, a SMOOTH interpolation filter, a SHARP interpolation filter, or a BILINEAR interpolation filter; When the MV level index is not lower than the MV level index threshold, a signal for using a second interpolation filter subset is derived, the number of interpolation filters in the second interpolation filter subset is less than the number of interpolation filters in the first interpolation filter subset, and the second interpolation filter subset includes REGULAR interpolation filters and SMOOTH interpolation filters.

2. The method according to claim 1, characterized in that Deriving the signal comprises: determining the MVD that applies adaptive pixel resolution to the video block; Determining an MV level index of the MV among a plurality of MV level indexes; identifying a context set for encoding an interpolation filter selection indicator based on the MV level index; and The video stream is decoded according to the context set to obtain the interpolation filter selection indicator as the signal.

3. The method according to any one of claims 1 to 2, characterized in that Deriving the signal comprises: Determining to apply adaptive pixel resolution to the MVD; Determining an MV level index of the video block among a plurality of MV level indexes; When the MV level index is not less than an MV level index threshold, decoding the video stream using a first context set to obtain an interpolation filter selection indicator as the signal; and When the MV level index is less than the MV level index threshold, the video stream is decoded using a second context set different from the first context set to obtain the interpolation filter selection indicator as the signal.

4. An apparatus for processing a video block of a video stream, comprising: a receiving unit configured to receive a marker in the video stream and store the video stream in a buffer memory; A determining unit configured to determine, based on the received flag, that inter-frame prediction is to be performed on the video block through a reference frame according to a motion vector (MV); A first deriving unit configured to derive the MV according to a reference MV and an MVD (MV difference) of the video block; a second deriving unit configured to derive a signal indicative of an interpolation filter, wherein the signal is based on at least one of a magnitude and a resolution of the MVD; a selection unit configured to select the interpolation filter according to the signal; and an inter-frame prediction unit configured to perform inter-frame prediction for the video block through the reference frame according to the MV and the selected interpolation filter; deriving the signal comprises: determining a magnitude of the MVD based on an MV level index associated with the video block, the MV level index being extracted from the video stream; and When the MV level index is below an MV level index threshold, deriving a signal for using a first subset of interpolation filters, the first subset of interpolation filters comprising a REGULAR interpolation filter, a SMOOTH interpolation filter, a SHARP interpolation filter, or a BILINEAR interpolation filter; When the MV level index is not lower than the MV level index threshold, a signal for using a second interpolation filter subset is derived, the number of interpolation filters in the second interpolation filter subset is less than the number of interpolation filters in the first interpolation filter subset, and the second interpolation filter subset includes REGULAR interpolation filters and SMOOTH interpolation filters.

5. A computer device comprising a processor and a memory, characterized in that: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the method according to any one of claims 1 to 3 according to instructions in the program code.

Images