Method for decoding, encoding, storing or transmitting video bit stream, video decoding and encoding device

Abstract

The invention provides a method for decoding, encoding and processing a video stream, and video decoding and encoding equipment. The method comprises the following steps: receiving the video stream; determining that the video block is inter-coded based on the prediction block and a motion vector (MV), where the MV derives a reference motion vector (RMV) and a motion vector difference (MVD) based on the video block; determining an RMV index range according to whether the MVD of the video block is encoded with a size dependent adaptive MVD pixel resolution, parsing the video stream based on the RMV index range to extract an RMV index of the video block, where the RMV index of the video block is to be mapped into a dynamic reference list (DRL), the DRL being to identify a plurality of ordered candidate RMVs; extracting the MVD from the video stream; deriving an MV based on the extracted RMV index and the MVD; and reconstructing the video block based on the MV and the prediction block.

Description

[0001] This application is a divisional application of an actively modified version of a patent application with an application date of June 1, 2022, Chinese patent application number 202280014441.X, and invention name “Method for decoding and encoding video blocks of a video stream, video decoding device, and video encoding device”. Technical Field

[0002] The present application relates generally to video encoding and decoding, and more particularly to methods and systems for providing adaptive resolution of motion vector differences in inter-frame prediction of video blocks. Background Art

[0003] The background technology description provided herein is for the purpose of generally describing the contents of the embodiments of the present application. Certain works of the inventors (i.e., works described in this background technology section) and the contents in the specification concerning certain prior art that has not yet become the prior art before the filing date are not considered to be the prior art with respect to the embodiments of the present application, whether explicitly or implicitly.

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each picture having a spatial size of, for example, 1920×1080 luminance samples and associated fully sampled or subsampled chrominance samples. The series of pictures may have a fixed or variable picture rate (alternatively, referred to as a frame rate) of, for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bit rate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920×1080, a frame rate of 60 frames per second, and a chrominance subsampling of 4:2:0 with 8 bits per pixel per color channel requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 GB of storage space.

[0005] One purpose of video encoding and decoding can be to reduce redundancy in an uncompressed input video signal by compression. Compression can help reduce the above-mentioned bandwidth and / or storage space requirements, in some cases by two orders of magnitude or more. Lossless compression and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal through a decoding process. Lossy compression refers to a coding / decoding process in which the original video information is not fully retained during encoding and the original video information is not fully restored during decoding. When lossy compression is used, the reconstructed signal may be different from the original signal, although some information is lost, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal usable for the intended application. In the case of video, lossy compression is widely used in many applications. The amount of distortion that can be tolerated depends on the application. For example, users of some consumer video streaming applications can tolerate higher distortion than users of movie or television broadcast applications. The compression rate achievable by a particular encoding algorithm can be selected or adjusted to reflect various distortion tolerances: higher tolerable distortions generally allow the encoding algorithm to produce higher losses and higher compression rates.

[0006] Video encoders and decoders may utilize techniques from multiple categories and steps including, for example, motion compensation, Fourier transforms, quantization, and entropy coding.

[0007] Video codec techniques may include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into blocks of samples. When all blocks of samples are encoded in intra-mode, the picture may be referred to as an intra-picture. Intra-pictures and their derivatives (e.g., independent decoder refresh pictures) may be used to reset the decoder state and may therefore be used as the first picture in an encoded video stream and video session, or as a still image. The samples of the block after intra-prediction may then be transformed in the frequency domain, and the transform coefficients so generated may be quantized prior to entropy coding. Intra-prediction represents a technique for minimizing sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficient, the fewer bits are needed to represent the entropy coded block at a given quantization step size.

[0008] Traditional intra-frame coding, such as known from coding techniques such as MPEG-2, does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to encode / decode blocks based on surrounding sample data and / or metadata obtained, for example, during spatially adjacent encoding and / or decoding, preceding the data block encoded or decoded within the frame in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. It should be noted that, at least in some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, and does not use reference data from other reference pictures.

[0009] Intra-frame prediction can take many different forms. When more than one such technique can be used in a given video coding technique, the technique in use may be referred to as an intra-frame prediction mode. One or more intra-frame prediction modes may be provided in a particular codec. In some cases, a mode may have sub-modes, and / or may be associated with various parameters, and the mode / sub-mode information and intra-frame coding parameters for a video block may be encoded separately or included together in a mode codeword. Which codeword is used for a given mode, sub-mode, and / or parameter combination may have an impact on the coding efficiency gain through intra-frame prediction, and the entropy coding technique used to convert the codeword into a bitstream may also have an impact on it.

[0010] H.264 introduced a certain intra-frame prediction mode, which was improved in H.265 and further improved in new coding techniques such as the Joint Exploration Model (JEM), the next generation video coding (Versatile Video Coding, VVC), and the Benchmark Set (BMS). Typically, for intra-frame prediction, the values ​​of neighboring samples that have become available can be used to form a prediction block. For example, the available values ​​of a particular set of neighboring samples can be copied to the prediction block along some directions and / or lines. A reference to the used direction can be encoded in the bitstream, or it can be predicted by itself. A reference to the used direction can be encoded in the bitstream, or it can be predicted by itself.

[0011] refer to Figure 1A, a subset of 9 prediction directions specified in the 33 possible intra-frame prediction directions of H.265 (corresponding to the 33 angular modes of the 35 intra-frame modes specified in H.265) is depicted at the bottom right. The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction along which the neighboring samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more neighboring samples at a 45 degree angle to the horizontal direction at the upper right. Similarly, arrow (103) indicates that sample (101) is predicted from one or more neighboring samples at a 22.5 degree angle to the horizontal direction at the lower left of sample (101).

[0012] Still referring to Figure 1, a square block (104) of 4×4 samples is depicted in the upper left (indicated by bold dashed lines). The square block (104) contains 16 samples, each sample is labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (starting from the top) and the first sample in the X dimension (starting from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the Y dimension and the X dimension. Since the size of the block is 4×4 samples, S44 is in the lower right corner. Example reference samples following a similar numbering scheme are also shown. The reference samples are labeled with R and their Y position (e.g., row index) and X position (column index) relative to the block (104). In H.264 and H.265, prediction samples adjacent to the block being reconstructed are used.

[0013] The intra picture prediction of block 104 may start by copying reference sample values ​​from neighboring samples according to a signaled prediction direction. For example, assume that the encoded video bitstream includes signaling indicating the prediction direction of arrow (102) for this block 104, i.e., predicting samples from one or more prediction samples at the top right and at a 45 degree angle to the horizontal direction. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Sample S44 is then predicted from reference sample R08.

[0014] In some cases, the values ​​of multiple reference samples may be combined, such as by interpolation, in order to calculate the reference sample, especially when the direction is not divisible by 45 degrees.

[0015] As video coding technology continues to develop, the number of possible directions increases. For example, in H.264 (2003), nine different directions can be used for intra-frame prediction. In H.265 (2013), this increased to 33 directions, and at the time of the present application embodiment, JEM / VVC / BMS can support up to 65 directions. Experimental studies have been conducted to help identify the most appropriate intra-frame prediction directions, and some techniques in entropy coding can be used to encode those most appropriate directions with a small number of bits, accepting a certain bit cost for the direction. In addition, the direction itself can sometimes be predicted from the adjacent directions used in the intra-frame prediction of the adjacent blocks that have been decoded.

[0016] Figure 1B A schematic diagram (180) is shown depicting 65 intra prediction directions according to JEM to illustrate the increase in the number of prediction directions in various coding techniques developed over time.

[0017] The manner in which the bits representing the intra prediction direction in the coded video bitstream are mapped to the prediction direction may vary between video coding techniques; for example, it may range from a simple direct mapping of the prediction direction to the intra prediction mode, to a codeword, to a complex adaptive scheme involving the most probable mode, and similar techniques. However, in all cases, for intra prediction, there may be certain directions that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, in a well-designed video coding technique, those less likely directions will be represented by more bits than the more likely directions.

[0018] Inter-picture prediction or inter-prediction may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or portion thereof (reference picture) may be used to predict a newly reconstructed picture or picture portion (e.g., block) after being spatially shifted in a direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, with the third dimension indicating the reference picture being used (similar to the time dimension).

[0019] In some video compression techniques, the current MV applicable to a region of sample data may be predicted based on other MVs, for example, other MVs related to other regions of sample data that are spatially adjacent to the region being reconstructed and that precede the current MV in decoding order. This can greatly reduce the total amount of data required to encode the MV by eliminating redundancy in the related MVs, thereby increasing compression efficiency. MV prediction can work effectively, for example, because when encoding an input video signal obtained from a camera (called natural video), there is a statistical probability that a larger area than the area to which a single MV applies moves in a similar direction in a video sequence, so in some cases, similar motion vectors derived from MVs of adjacent areas can be used to predict the larger area. This makes the actual MV for a given region similar or identical to the MV predicted from the surrounding MVs. Further, after entropy coding, the MV can be represented with fewer bits than when the MV is encoded directly (rather than predicted from adjacent MVs). In some cases, MV prediction can be an example of losslessly compressing a signal (i.e., an MV) derived from an original signal (i.e., a sample stream). In other cases, the MV prediction itself may be lossy, for example due to round-off errors when computing the prediction value from multiple surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding", December 2016). Among the multiple MV prediction mechanisms specified by H.265, this article describes a technique referred to below as "spatial merging".

[0021] Specifically, refer to Figure 2 , the current block (201) includes samples that have been discovered by the encoder during the motion search process, and the samples can be predicted based on the previous block of the same size that has generated a spatial offset. In addition, the MV can be derived from metadata associated with one or more reference pictures instead of encoding the MV directly. For example, the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 202 to 206, respectively) is used to derive the MV from the metadata of the nearest reference picture (in decoding order). In H.265, MV prediction can use prediction values ​​from the same reference picture that is being used by neighboring blocks. Summary of the invention

[0022] Embodiments of the present application provide a method for decoding, encoding, storing or sending a video bit stream, and a video decoding and encoding device.

[0023] The technical solution of the embodiment of the present application is implemented as follows:

[0024] The embodiment of the present application provides a method for decoding a video block of a video stream, including:

[0025] receiving the video stream;

[0026] Determining that the video block is inter-coded based on a prediction block and a motion vector MV, wherein the MV is derived based on a reference motion vector RMV and a motion vector difference MVD of the video block;

[0027] Determine an RMV index range according to whether the MVD of the video block is encoded with a size-dependent adaptive MVD pixel resolution, and parse the video stream based on the RMV index range to extract the RMV index of the video block, wherein the RMV index of the video block is used to map into a dynamic reference list DRL, and the DRL is used to identify a plurality of ordered candidate RMVs; and

[0028] extracting the MVD from the video stream;

[0029] deriving the MV based on the extracted RMV index and the MVD; and

[0030] The video block is reconstructed based on the MV and the prediction block.

[0031] The embodiment of the present application provides a method for encoding a video block of a video stream, including:

[0032] Receiving the video stream to be encoded;

[0033] Setting the video block to be inter-frame coded based on a prediction block and a motion vector MV, wherein the MV is derived based on a reference motion vector RMV and a motion vector difference MVD of the video block;

[0034] Determine an RMV index range according to whether the MVD of the video block is encoded with a size-dependent adaptive MVD pixel resolution, and parse the video stream based on the RMV index range to extract the RMV index of the video block, wherein the RMV index of the video block is used to map into a dynamic reference list DRL, and the DRL is used to identify a plurality of ordered candidate RMVs; and

[0035] extracting the MVD from the video stream;

[0036] deriving the MV based on the extracted RMV index and the MVD; and

[0037] An encoding process of the video block is performed based on the MV and the prediction block.

[0038] An embodiment of the present application provides a video decoding device, including a processor and a memory for storing computer instructions. When the processor executes the computer instructions, it is configured to cause the video decoding device to perform the decoding method described in the embodiment of the present application.

[0039] An embodiment of the present application provides a video encoding device, including a processor and a memory for storing computer instructions. When the processor executes the computer instructions, it is configured to enable the video encoding device to perform the encoding method described in the embodiment of the present application.

[0040] An embodiment of the present application provides a method for storing or sending a video bit stream, wherein the video bit stream is decoded based on the decoding method described in the embodiment of the present application, or is generated based on the encoding method described in the embodiment of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which:

[0042] Figure 1A A schematic diagram showing an exemplary subset of intra prediction direction modes.

[0043] Figure 1B A diagram of exemplary intra prediction directions is shown.

[0044] Figure 2 A schematic diagram showing a current block and surrounding spatially merged candidate blocks for motion vector prediction in an example.

[0045] Figure 3 A schematic diagram showing a simplified block diagram of a communication system according to an example embodiment.

[0046] Figure 4 A schematic diagram showing a simplified block diagram of a video streaming system according to an example embodiment.

[0047] Figure 5 A schematic diagram showing a simplified block diagram of a video decoder according to an example embodiment.

[0048] Figure 6 A schematic diagram showing a simplified block diagram of a video encoder according to an example embodiment.

[0049] Figure 7 A block diagram of a video encoder according to another example embodiment is shown.

[0050] Figure 8 A block diagram of a video decoder according to another example embodiment is shown.

[0051] Fig. 9A scheme for dividing coding blocks according to an exemplary embodiment of an embodiment of the present application is shown.

[0052] Fig.10 Another scheme of coding block partitioning according to an exemplary embodiment of an embodiment of the present application is shown.

[0053] Fig.11 Another scheme of coding block partitioning according to an exemplary embodiment of an embodiment of the present application is shown.

[0054] Fig.12 An example of partitioning a basic block into coding blocks according to an example partitioning scheme is shown.

[0055] Fig.13 An exemplary ternary partitioning scheme is shown.

[0056] Fig.14 An exemplary quadtree binary tree coding block partitioning scheme is shown.

[0057] Fig.15 A scheme for dividing a coding block into multiple transform blocks and an encoding order of these transform blocks according to an exemplary embodiment of an embodiment of the present application are shown.

[0058] Fig.16 Another scheme for dividing a coding block into multiple transform blocks and the encoding order of these transform blocks according to an exemplary embodiment of an embodiment of the present application is shown.

[0059] Fig.17 Another scheme for dividing a coding block into multiple transform blocks according to an exemplary embodiment of an embodiment of the present application is shown.

[0060] Fig.18 A flow chart of a method according to an exemplary embodiment of an embodiment of the present application is shown.

[0061] Fig.19 Another flow chart of a method according to an example embodiment of the present application is shown.

[0062] Fig. 20 A schematic diagram of a computer system according to an exemplary embodiment of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0063] Throughout the specification and claims, terms may have nuanced meanings that are suggested or implied by the context beyond the explicitly stated meanings. The phrases "in one embodiment" or "in some embodiments" used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one embodiment" or "in some embodiments" used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" used herein do not necessarily refer to different embodiments. For example, the claimed subject matter includes a combination of all or part of the exemplary embodiments / embodiments.

[0064] In general, terms can be understood at least in part from usage in context. For example, terms such as "and", "or" or "and / or" used herein can include various meanings, which can depend at least in part on the context in which these terms are used. Generally, if "or" is used to associate a list such as A, B or C, it is intended to represent A, B and C (here for inclusion) and A, B or C (here for exclusion). In addition, the terms "one or more" or "at least one" used herein, at least in part depending on the context, can be used to describe any feature, structure or characteristic in a singular sense, or can be used to describe a combination of features, structures or characteristics in a plural sense. Similarly, terms such as "one", "an" or "the" can also be understood to convey singular usage or to convey plural usage, which depends at least in part on the context. In addition, belonging to "based on" or "determined by..." can be understood as not necessarily intended to convey a set of exclusive factors, but may allow the presence of other factors that are not necessarily explicitly described, which also depends at least in part on the context. Figure 3 A simplified block diagram of a communication system (300) according to one embodiment of the present application is shown. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). Figure 3 In the example of , the first terminal device pair (310) and (320) can perform unidirectional data transmission. For example, the terminal device (310) can encode video data (for example, video data of a video picture stream collected by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data is transmitted in the form of one or more encoded video code streams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to restore the video picture, and display the video picture based on the restored video data. The unidirectional data transmission can be implemented in applications such as media services.

[0065] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which can be implemented, for example, during a video conferencing application. For bidirectional data transmission, in one example, each of the terminal devices (330) and (340) can encode video data (e.g., video data of a video picture stream collected by the terminal device) for transmission to the other terminal device (330) and (340) via a network (350). Each of the terminal devices (330) and (340) can also receive the encoded video data transmitted by the other terminal device (330) and (340), and can decode the encoded video data to restore the video picture, and can display the video picture on an accessible display device based on the restored video data.

[0066] exist Figure 3 In the example of, terminal device (310), terminal device (320), terminal device (330) and terminal device (340) can be implemented as a server, a personal computer and a smart phone, but the applicability of the basic principle of the embodiment of the present application may not be limited to this. The embodiment of the embodiment of the present application can be implemented on a desktop computer, a laptop computer, a tablet computer, a media player, a wearable computer, a dedicated video conferencing device, and / or the like. The network (350) represents any number or type of network that transmits encoded video data between terminal device (310), terminal device (320), terminal device (330) and terminal device (340), including, for example, a wired (wired) and / or wireless communication network. The communication network (350) can exchange data in a circuit switching channel, a packet switching channel, and / or other types of channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purpose of this discussion, unless explicitly stated herein, the architecture and topology of the network (350) may be irrelevant to the operation of the embodiment of the present application.

[0067] As examples of applications for the disclosed subject matter, Figure 4 is a schematic diagram of a simplified block diagram of a video streaming system according to an example embodiment, Figure 4 The placement of the video encoder and the video decoder in a video streaming environment is shown. The disclosed subject matter is equally applicable to other video applications, including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, storing compressed video on digital media including CDs, DVDs, memory sticks, etc., and the like.

[0068] The video streaming system 400 may include a video acquisition subsystem (413), which may include a video source (401), such as a digital camera, that creates an uncompressed video picture stream or image (402). In one example, the video picture stream (402) includes samples recorded by the digital camera of the video source 401. The video picture stream (402) is depicted as a thick line to emphasize the high data volume of the video picture stream compared to the encoded video data (404) (or encoded video bitstream), and the video picture stream (302) can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted as thin lines to emphasize the lower data volume compared to the uncompressed video picture stream (402), can be stored on the streaming server (405) for future use, or directly stored in a downstream video device (not shown). One or more streaming client subsystems, such as Figure 4 The client subsystem (406) and the client subsystem (408) in the streaming server (405) can access the streaming server (405) to retrieve the copy (407) and the copy (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and generates an uncompressed output video picture stream (411) that can be presented on a display (412) (e.g., a display screen) or other presentation device (not depicted). The video decoder 410 may be configured to perform some or all of the various functions described in the embodiments of the present application. In some streaming transmission systems, the encoded video data (404), the video data (407), and the video data (409) (e.g., a video bitstream) may be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as the next generation video coding (Versatile Video Coding, VVC). The disclosed subject matter may be used in the context of VVC, and may be used in other video coding standards.

[0069] It should be noted that the electronic device (420) and the electronic device (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0070] In the following, Figure 5A block diagram of a video decoder (510) according to any embodiment of the present application is shown. The video decoder (510) may be arranged in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 4 A video decoder (410) in an example of FIG.

[0071] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510). In the same or another embodiment, one encoded video sequence may be decoded at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. Each video sequence may be associated with multiple video frames or images. The encoded video sequence may be received from a channel (501), which may be a hardware / software link leading to a storage device storing the encoded video data or a streaming source sending the encoded video data. The receiver (531) may receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective processing circuits (not depicted). The receiver (531) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be provided between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be located outside the video decoder (510) and separate from the video decoder (510) (not depicted). In still other applications, a buffer memory (not depicted) is provided outside the video decoder (510) to, for example, prevent network jitter, and another additional buffer memory (515) may be provided inside the video decoder (510) to, for example, handle broadcast timing. When the receiver (531) receives data from a storage / forward device with sufficient bandwidth and controllability or from an isochronous network, it may not be necessary to configure the buffer memory (515), or the buffer memory may be made smaller. In order to be used on a service packet network such as the Internet, a buffer memory (515) of sufficient size may be required, and the size of the buffer memory (515) may be relatively large. Such a buffer memory may be implemented to have an adaptive size and may be implemented at least partially in an operating system or a similar element (not depicted) outside the video decoder (510).

[0072] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (510) and potential information for controlling a rendering device such as a display (512) (e.g., a display screen), which may or may not be an integral part of the electronic device (530), but may be coupled to the electronic device (530), such as Figure 5 As shown. The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI message) or a Video Usability Information (VUI) parameter set fragment (not depicted). The parser (520) may parse / entropy decode the encoded video sequence received by the parser (520). The entropy encoding of the encoded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a subgroup parameter set for at least one subgroup of the subgroups of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the subgroup. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), etc. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients (eg, Fourier transform coefficients), quantizer parameter values, motion vectors, etc.

[0073] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515), thereby creating symbols (521).

[0074] Depending on the type of coded video picture or portion of coded video picture (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of the symbol (521) may involve multiple different processing or functional units. Which units are involved and how they are involved can be controlled by the parser (520) through subgroup control information parsed from the coded video sequence. For simplicity, such subgroup control information flow between the parser (520) and the multiple processing or functional units described below is not depicted.

[0075] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In actual implementations that operate under commercial constraints, many of these functional units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed subject matter, the conceptual subdivision into multiple functional units is adopted below in the embodiments of the present application.

[0076] The first unit may include a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) may receive quantized transform coefficients as symbols (521) and control information from the parser (520), including information indicating which type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrices, etc. The sealer / inverse transform unit (551) may output a block including sample values, which may be input into an aggregator (555).

[0077] In some cases, the output samples of the sealer / inverse transform (551) may belong to an intra-coded block; that is, a block that does not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may use surrounding block information that has been reconstructed and stored in a current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator (555) may add the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the sealer / inverse transform unit (551) on a per-sample basis.

[0078] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to extract samples for inter-frame picture prediction. After the extracted samples are motion compensated according to the symbols (521) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (551) by the aggregator (555) (the output of unit 551 may be referred to as residual samples or residual signals), thereby generating output sample information. The extraction of prediction samples by the motion compensated prediction unit (553) from the address in the reference picture memory (557) may be controlled by a motion vector, and the motion vector may be provided to the motion compensated prediction unit (553) in the form of a symbol (521), which may have, for example, an X component, a Y component (offset) and a reference picture component (time). Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory (557) when using sub-sample accurate motion vectors, and may also be associated with motion vector prediction mechanisms, etc.

[0079] The output samples of the aggregator (555) may be employed by various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques that are controlled by parameters included in an encoded video sequence (also referred to as an encoded video bitstream) and that are available to the loop filter unit (556) as symbols (521) from the parser (520). However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, as well as to previously reconstructed and loop filtered sample values. A variety of types of loop filters may be included as part of the loop filter unit 556 in various orders, as will be described in further detail below.

[0080] The output of the loop filter unit (556) may be a sample stream that may be output to a rendering device (512) and stored in a reference picture memory (557) for future inter-picture prediction.

[0081] Once fully reconstructed, certain coded pictures may be used as reference pictures for future inter-picture prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557), and a new current picture buffer may be reallocated before starting to reconstruct a subsequent coded picture.

[0082] The video decoder (510) may perform decoding operations according to a predetermined video compression technique employed in a standard such as ITU-T H.265 Recommendation. The coded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the coded video sequence follows the syntax of the video compression technique or standard and the profile recorded in the video compression technique or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technique or standard as the only tools available for use under the profile. In order to conform to the standard, the complexity of the coded video sequence may be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further defined by the Hypothetical Reference Decoder (HRD) specification and metadata of the HRD buffer management signaled in the coded video sequence.

[0083] In some exemplary embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0084] Figure 6 6 is a block diagram of a video encoder (603) according to an exemplary embodiment disclosed in the present application. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., a transmission circuit). The video encoder (603) may be used to replace Figure 4 A video encoder (403) in an example.

[0085] The video encoder (603) can be used to obtain the video source (601) (not Figure 6 In an example of an electronic device (620) receiving video samples, a video source (601) can capture video images to be encoded by a video encoder (603). In another example, the video source (601) can be implemented as a part of the electronic device (620).

[0086] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by a video encoder (603), the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCB, RGB, XYZ, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) may be a storage device capable of storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures or images that are given motion when viewed sequentially. The picture itself may be constructed as a spatial pixel array, wherein each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by a person skilled in the art. The following description focuses on the samples.

[0087] According to some exemplary embodiments, the video encoder (603) may encode and compress the pictures of the source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed constitutes a function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to other functional units as described below and control the other functional units. For simplicity, the coupling is not depicted in the figure. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, etc.), picture size, picture group (GOP) layout, maximum motion vector search range, etc. The controller (650) can be used to have other suitable functions that are related to the video encoder (603) optimized for a certain system design.

[0088] In some exemplary embodiments, the video encoder (603) may be configured to operate in an encoding loop. As an oversimplified description, in one example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to the way the (remote) decoder creates sample data, even if the embedded decoder 633 processes the encoded video stream through the source encoder 630 without entropy coding (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols in the entropy coding and the encoded video code stream can be lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces a bit-accurate result that is independent of the decoder location (local or remote), the contents in the reference picture memory (634) are also bit-accurately corresponding between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, e.g. due to channel errors) is used to improve encoding quality.

[0089] The operation of the "local" decoder (633) can be combined with the above Figure 5 The "remote" decoder described in detail for the video decoder (510) is identical. However, additional brief reference is made to Figure 5 , since the symbols are available and the entropy encoder (645) and the parser (520) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510) including the buffer memory (515) and the parser (520) may not be fully implemented in the local decoder (633), in the encoder.

[0090] At this point, it can be observed that any decoder technology, except for the parsing / entropy decoding that may only exist in the decoder, must also exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter sometimes focuses on the decoder operation, which cooperates with the decoding part of the encoder. Therefore, the description of the encoder technology can be simplified because the encoder technology is mutually inverse to the decoder technology described comprehensively. Only some areas or aspects of the encoder are described in more detail below.

[0091] During operation, in some exemplary implementations, the source encoder (630) may perform motion compensated predictive coding that predictively encodes an input picture by referencing one or more previously encoded pictures from a video sequence designated as "reference pictures." In this manner, the encoding engine (632) encodes the differences (or residuals) in color channels between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture. The term "residual" and its adjective form "residual" may be used interchangeably.

[0092] The local video decoder (633) may decode the encoded video data of the picture that may be designated as the reference picture based on the symbol created by the source encoder (630). The operation of the encoding engine (632) may be a lossy process. When the encoded video data may be decoded at the video decoder ( Figure 6 When the video encoder (603) is decoded on a local (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that can be performed by the video decoder on the reference picture and can cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) can store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture that will be obtained by the far-end (remote) video decoder.

[0093] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new picture. The predictor (635) may operate pixel-by-pixel based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it may be determined that the input picture may have prediction references taken from a plurality of reference pictures stored in the reference picture memory (634).

[0094] The controller (650) may manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0095] The outputs of all the above functional units may be entropy encoded in an entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0096] The transmitter (640) may buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) may combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0097] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types:

[0098] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow for different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those of ordinary skill in the art are aware of the variations of I pictures and their corresponding applications and features.

[0099] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0100] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstructing a single block.

[0101] The source picture may typically be spatially subdivided into a plurality of coding blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined by the coding allocation applied to the corresponding picture of the block. For example, a block of an I picture may be non-predictively coded, or the block may be predictively coded (spatial prediction or intra-frame prediction) with reference to an already coded block of the same picture. A pixel block of a P picture may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. A block of a B picture may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction. For other purposes, the source picture or the intermediate processed picture may be subdivided into other types of blocks. The division of coding blocks and other types of blocks may or may not follow the same approach, as described in further detail below.

[0102] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0103] In an exemplary embodiment, the transmitter (640) may transmit additional data when transmitting the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0104] The captured video may be taken as a plurality of source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. For example, a particular picture being encoded / decoded may be divided into blocks, and the particular picture being encoded / decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a reference picture that was previously encoded in the video and is still buffered, the block in the current picture may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case where multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0105] In some exemplary embodiments, a bidirectional prediction technique may be used for inter-picture prediction. According to this bidirectional prediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture that precede the current picture in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block may be jointly predicted by a combination of the first reference block and the second reference block.

[0106] In addition, merge mode technology can be used for inter-picture prediction to improve coding efficiency.

[0107] According to some exemplary embodiments of the embodiments of the present application, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, a picture in a video picture sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the picture may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU may include three parallel coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Each CTU may be recursively divided into one or more coding units (CUs) in a quadtree. For example, a 64×64 pixel CTU may be divided into a 64×64 pixel CU, or four 32×32 pixel CUs. Each of the one or more 32×32 blocks may be further divided into four 16×16 pixel CUs. In some exemplary embodiments, each CU may be analyzed during encoding to determine a prediction type for the CU among various prediction types, such as an inter-prediction type or an intra-prediction type. According to temporal and / or spatial predictability, the CU can be divided into one or more prediction units (PUs). Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In an embodiment, the prediction operation in the encoding (encoding / decoding) is performed in units of prediction blocks. The division of the CU into PUs (or PBs of different color channels) can be performed in various spatial modes. For example, a luminance or chrominance PB may include a matrix of values ​​for samples (e.g., luminance values), and the samples are, for example, 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, etc.

[0108] Figure 7A diagram of another exemplary embodiment of a video encoder (703) according to an embodiment of the present application is shown. The video encoder (703) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures and encode the processed block into an encoded picture that is part of an encoded video sequence. The exemplary video encoder (703) may be used instead of Figure 4 A video encoder (403) in an example.

[0109] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples, etc. The video encoder (703) then uses, for example, rate-distortion optimization (RDO) to determine whether to use intra mode, inter mode, or bidirectional prediction mode to best encode the processing block. When it is determined that the processing block is encoded in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded picture; and when it is determined that the processing block is encoded in inter mode or bidirectional prediction mode, the video encoder (703) may use inter prediction or bidirectional prediction techniques to encode the processing block into an encoded picture, respectively. In some exemplary embodiments, merge mode may be used as an inter-picture prediction submode, in which motion vectors are derived from one or more motion vector predictors without the aid of encoded motion vector components external to the predictor. In some other exemplary embodiments, there may be motion vector components applicable to the subject block. Therefore, the video encoder (703) may include a submode that is not in Figure 7 Components explicitly shown in, such as a mode decision module for determining a prediction mode for a processing block.

[0110] exist Figure 7 In the example of FIG. 7 , the video encoder ( 703 ) includes Figure 7 The exemplary arrangement shown is an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721) and an entropy encoder (725) coupled together.

[0111] The inter-frame encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture in display order), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some examples, the reference picture is based on the encoded video information, using a decoded reference picture decoded by the decoding unit 633, which is embedded in the decoding unit 633. Figure 6 In the exemplary encoder 620 (shown as Figure 7 residual decoder 728, as described in further detail below).

[0112] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block with an encoded block in the same picture, generate quantization coefficients after transformation, and in some cases also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). The intra encoder (722) can calculate an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same picture.

[0113] The general controller (721) can be configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines a prediction mode of a block and provides a control signal to a switch (726) based on the prediction mode. For example, when the prediction mode is an intra-frame mode, the general controller (721) controls the switch (726) to select an intra-frame mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select intra-frame prediction information and include the intra-frame prediction information in the bitstream; and when the prediction mode of the block is an inter-frame mode, the general controller (721) controls the switch (726) to select an inter-frame prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select inter-frame prediction information and include the inter-frame prediction information in the bitstream.

[0114] The residual calculator (723) may be configured to calculate the difference (residual data) between the received block and the block prediction result selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) may be configured to encode the residual data to generate a transform coefficient. For example, the residual encoder (724) may be configured to transform the residual data from the spatial domain to the frequency domain to generate a transform coefficient. The transform coefficient is then quantized to obtain a quantized transform coefficient. In various exemplary embodiments, the video encoder (703) further includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data may be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) may generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) may generate a decoded block based on the decoded residual data and the intra prediction information. The decoded blocks are appropriately processed to generate decoded pictures, and the decoded pictures may be buffered in a memory circuit (not shown) and used as reference pictures.

[0115] The entropy encoder (725) may be configured to format the code stream to include the encoded blocks and perform entropy encoding. The entropy encoder (725) may be configured to include various information in the code stream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the code stream. When the block is encoded in the inter-frame mode or the merge sub-mode of the bidirectional prediction mode, the residual information may not be present.

[0116] Figure 8 FIG. 8 is a diagram of an exemplary video decoder (810) according to another embodiment of an embodiment of the present application. The video decoder (810) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In one example, the video decoder (810) can be used instead of Figure 4 A video decoder (410) in an example of FIG.

[0117] exist Figure 8 In the example of FIG. 8 , the video decoder ( 810 ) includes Figure 8 The exemplary arrangement shown is an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874) and an intra-frame decoder (872) coupled together.

[0118] The entropy decoder (871) can be used to reconstruct certain symbols from the encoded picture, which represent syntax elements that constitute the encoded picture. Such symbols may include, for example, a mode for encoding a block (e.g., intra mode, inter mode, bidirectional prediction mode, merge submode, or another submode), prediction information (e.g., intra prediction information or inter prediction information) that can identify certain samples or metadata used by the intra decoder (872) or the inter decoder (880) for prediction, respectively, residual information in the form of, for example, quantized transform coefficients, etc. In one example, when the prediction mode is inter or bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can be inverse quantized and provided to the residual decoder (873).

[0119] The inter-frame decoder (880) may be configured to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0120] The intra decoder (872) may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0121] The residual decoder (873) may be configured to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also use certain control information (to include quantizer parameters (QP)), which may be provided by the entropy decoder (871) (data path not depicted because this is only low data volume control information).

[0122] The reconstruction module (874) may be configured to combine the residual output by the residual decoder (873) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module, as appropriate) in the spatial domain to form a reconstructed block, which forms part of a reconstructed picture, which is part of the reconstructed video. It should be noted that other suitable operations such as deblocking operations may also be performed to improve visual quality.

[0123] It should be noted that the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using any suitable technology. In some exemplary embodiments, the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoder (403), video encoder (603) and video encoder (603) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using one or more processors that execute software instructions.

[0124] Turning to block partitioning for encoding and decoding, the general partitioning can start from the basic block and can follow a predefined set of rules, a specific pattern, a partitioning tree, or any partitioning structure or scheme. The partitioning can be hierarchical and recursive. After the basic block is split or divided according to any of the example partitioning processes described below or other processes or combinations thereof, a final set of partitions or coding blocks can be obtained. Each of these partitions can be at one of the different partitioning levels in the partitioning hierarchy and can have various shapes. Each partition can be referred to as a coding block (CB). For the various example partitioning implementations further described below, each generated CB can be of any allowed size and partitioning level. Such partitions are called coding blocks because they can form units, some basic encoding / decoding can be performed on these units, and the encoding / decoding parameters can be optimized, determined, and written in the encoded video code stream. The highest or deepest level in the final partition represents the depth of the coding block partitioning structure of the tree. The coding block can be a luminance coding block or a chrominance coding block. The CB tree structure for each color can be referred to as a coding block tree (CBT).

[0125] The coding blocks of all color channels can be collectively referred to as coding units (CUs). The hierarchical structure of all color channels can be collectively referred to as coding tree units (CTUs). The division patterns or structures of different color channels in a CTU may be the same or different.

[0126] In some implementations, the partitioning tree schemes or structures for luma and chroma channels may not need to be the same. In other words, luma and chroma channels may have separate coding tree structures or modes. In addition, whether luma and chroma channels use the same or different coding partitioning tree structures and the actual coding partitioning tree structures to be used may depend on whether the slice being encoded is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma channel and the luma channel may have separate coding partitioning tree structures or coding partitioning tree structure modes, while for a P slice or a B slice, the luma and chroma channels may share the same coding partitioning tree scheme. When a separate coding partitioning tree structure or mode is applied, the luma channel may be divided into CBs by one coding partitioning tree structure, and the chroma channel may be divided into chroma CBs by another coding partitioning tree structure.

[0127] In some exemplary implementations, a predetermined partitioning pattern may be applied to the basic blocks. Fig. 9As shown, the exemplary 4-way partitioning tree may start from a first predefined level (e.g., 64×64 block level or other size as a basic block size), and the basic block may be hierarchically partitioned downward to a predefined lowest level (e.g., 4×4 level). For example, a basic block may have four predefined partitioning options or modes indicated by 902, 904, 906, and 908, where a partition designated as R is allowed for recursive partitioning, such that Fig. 9 The same partitioning options indicated in can be repeated at lower scales until the lowest level (e.g., 4×4 level). Fig. 9 Additional restrictions apply to the partitioning scheme. Fig. 9 In the implementation of , rectangular partitions (for example, 1:2 / 2:1 rectangular partitions) may be allowed, but these rectangular partitions are not allowed to be recursive, while square partitions are allowed to be recursive. If necessary, follow Fig. 9 The recursive partitioning of will generate a final set of coding blocks. The coding tree depth can be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth of the root node or root block (e.g., 64×64 blocks) can be set to 0, and the root block is divided according to Fig. 9 After being further split once, the coding tree depth increases by 1. For the above scheme, the maximum or deepest level from the 64×64 basic block to the 4×4 minimum partition will be 4 (starting from level 0). This partitioning scheme can be applied to one or more color channels. Fig. 9 The scheme of dividing each color channel independently (for example, the division mode or option in the predefined mode can be determined independently for each color channel of each hierarchical level). Optionally, two or more color channels can share Fig. 9 The same hierarchical pattern tree (for example, the same partitioning pattern or option in the predefined pattern can be selected for two or more color channels at each hierarchical level).

[0128] Fig.10 Another exemplary predefined partitioning pattern that allows recursive partitioning to form a partitioning tree is shown. Fig.10 As shown, an exemplary 10-way partitioning structure or pattern may be predefined. A root block may start from a predefined level (eg, from a basic block at a 128×128 level or a 64×64 level). Fig.10 Example partition structures include various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. Fig.10 The partition type of the three sub-partitions 1002, 1004, 1006, and 1008 indicated in the second row of the table may be referred to as a "T-type" partition. The "T-type" partitions 1002, 1004, 1006, and 1008 may be referred to as a left T-type, an upper T-type, a right T-type, and a lower T-type, respectively. In some example implementations, further subdivision is not allowed. Fig.10 The coding tree depth may be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth of the root node or root block (e.g., 128×128 block) may be set to 0, and the coding tree depth of the root block may be set to 0. Fig.10 After being further split once, the coding tree depth increases by 1. In some implementations, only full square partitions in 1010 may be allowed according to Fig.10 In other words, for the square partitions within the T-patterns 1002, 1004, 1006, and 1008, recursive partitioning may not be allowed. Fig.10 The recursive partitioning process of generates a final set of coding blocks. This scheme can be applied to one or more color channels. In some implementations, more flexibility can be added when using partitions below the 8×8 level. For example, 2×2 chroma inter-frame prediction can be used in some cases.

[0129] In some other exemplary implementations for coding block partitioning, a quadtree structure can be used to partition a basic block or an intermediate block into quadtree partitions. This quadtree partitioning can be applied hierarchically and recursively to any square partition. Whether a basic block or an intermediate block or a partition is further quadtree partitioned can be adapted to various local characteristics of the basic block or the intermediate block or the partition. The quadtree partitioning at the picture boundary can be further adjusted. For example, an implicit quadtree partitioning can be performed at the picture boundary so that the block will remain quadtree partitioned until the size fits the picture boundary.

[0130] In some other example implementations, a hierarchical binary partitioning from a basic block may be used. For such a scheme, a basic block or an intermediate block may be divided into two partitions. The binary partitioning may be horizontal or vertical. For example, a horizontal binary partitioning may split a basic block or an intermediate block into equal right and left partitions. Similarly, a vertical binary partitioning may split a basic block or an intermediate block into equal upper and lower partitions. This binary partitioning may be hierarchical and recursive. It may be decided on each basic block or intermediate block whether the binary partitioning scheme should be continued, and if the scheme continues, whether horizontal or vertical binary partitioning should be used. In some implementations, further partitioning may stop at a predefined minimum partition size (in one or two dimensions). Optionally, further partitioning may stop once a predefined partitioning level or depth from a basic block is reached. In some implementations, the aspect ratio of the partition may be limited. For example, the aspect ratio of the partition may be no less than 1:4 (or greater than 4:1). Therefore, a vertical strip partition with an aspect ratio of 4:1 can only be further divided into two vertical partitions, upper and lower, each with an aspect ratio of 2:1.

[0131] In some other examples, such as Fig.13 As shown, the trifurcated partitioning scheme can be used to partition the basic block or any intermediate block. The trifurcated pattern can be Fig.13 1302 is implemented vertically, or as shown in Fig.13 1304 is shown as being implemented horizontally. Fig.13 An exemplary split ratio (vertically or horizontally) in is shown as 1:2:1, but other ratios may also be predefined. In some implementations, two or more different ratios may be predefined. This ternary partitioning scheme may be used to complement a quadtree or binary partitioning structure, because this ternary tree partitioning is able to capture an object located at the center of a block in one continuous partition, whereas quadtrees and binary trees always partition along the center of the block, so the object is divided into separate partitions. In some implementations, the width and height of the example ternary tree partitioning are always powers of 2 to avoid additional transformations.

[0132] The above partitioning schemes can be combined in any way at different partitioning levels. As an example, the above quadtree and binary partitioning schemes can be combined to partition a basic block into a quadtree-binary-tree (QTBT) structure. In this scheme, a basic block or an intermediate block / partition can be a quadtree partition or a binary partition, if specified, depending on a set of predefined conditions. Fig.14 A specific example is shown. Fig.14 In the example of , a basic block is first divided into four partitions by a quadtree, as shown in 1402, 1404, 1406 and 1408. Thereafter, each generated partition is either quadtree-partitioned into four further partitions (e.g., 1408) or binary-partitioned into two further partitions (e.g., horizontally or vertically, such as 1402 or 1406, both of which are symmetrical) at the next level, or is not partitioned (e.g., 1404). For square partitions, binary or quadtree partitioning can be recursively allowed, as shown in the overall partitioning pattern example of 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree partitioning and dashed lines represent binary tree partitioning. A flag can be used for each binary partition node (non-leaf binary partition) to indicate whether the binary partition is horizontal or vertical. For example, as shown in 1420, consistent with the partitioning structure of 1410, the flag "0" can represent a horizontal binary partition and the flag "1" can represent a vertical binary partition. For quadtree partitioning, there is no need to indicate the partition type, because quadtree partitioning always partitions a block or partition horizontally and vertically to produce 4 sub-blocks / partitions of equal size. In some implementations, a flag "1" may indicate a horizontal binary partition, and a flag "0" may indicate a vertical binary partition.

[0133] In some example implementations of QTBT, the quadtree and binary splitting rule set may be represented by the following predefined parameters and corresponding functions associated therewith:

[0134] CTU size: The root node size of the quadtree (the size of the basic block)

[0135] MinQTSize: The minimum allowed size of a quadtree leaf node

[0136] MaxBTSize: Maximum allowed binary tree root node size

[0137] MaxBTDepth: Maximum allowed binary tree depth

[0138] MaxBTSize: The minimum allowed binary tree leaf node size

[0139] In some example implementations of the QTBT partitioning structure, the CTU size can be set to 128×128 luma samples and two corresponding blocks of 64×64 chroma samples (when example chroma subsampling is considered and used), MinQTSize can be set to 16×16, MaxBTSize can be set to 64×64, MinBTSize (for width and height) can be set to 4×4, and MaxBTDepth can be set to 4. Quadtree partitioning can be first applied to the CTU to generate quadtree leaf nodes. The size of the quadtree leaf node can be from its minimum allowed size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., the CTU size). If a node is 128×128, it will not be split by the binary tree first because its size exceeds the MaxBTSize (i.e., 64×64). Otherwise, nodes that do not exceed MaxBTSize can be partitioned by the binary tree. Fig.14 In the example, the basic block is 128×128. According to the predefined set of rules, the basic block can only be quadtree split. The partition depth of the basic block is 0. Each of the four partitions obtained is 64×64, not exceeding MaxBTSize, and can be further quadtree or binary tree split at level 1. This process continues. When the binary tree depth reaches MaxBTDepth (i.e. 4), further splitting can be ignored. When the width of the binary tree node is equal to MinBTSize (i.e. 4), further horizontal splitting can be ignored. Similarly, when the height of the binary tree node is equal to MinBTSize, no further vertical splitting is considered.

[0140] In some example implementations, the above-mentioned QTBT scheme can be configured to support the flexibility of having the same QTBT structure or separate QTBT structure for luma and chroma. For example, for P slices and B slices, the luma and chroma CTBs in one CTU can share the same QTBT structure. However, for I slices, the luma CTB can be divided into CBs by the QTBT structure, and the chroma CTB can be divided into chroma CBs by another QTBT structure. This means that a CU can be used to refer to different color channels in an I slice, for example, an I slice can consist of coding blocks of a luma component or coding blocks of two chroma components, and a CU in a P slice or a B slice can consist of coding blocks of all three color components.

[0141] In some other implementations, the QTBT scheme can be supplemented with the above three-pronged scheme. Such an implementation can be called a multi-type-tree (MTT) structure. For example, in addition to the binary split of the node, you can choose Fig.13 One of the three-pronged splitting modes. In some implementations, only square nodes can be trifurcated. An additional flag can be used to indicate whether the trifurcated split is horizontal or vertical.

[0142] Two-level or multi-level tree designs, such as the QTBT implementation and the QTBT implementation supplemented by a three-way split, may be designed primarily for complexity reduction. In theory, the complexity of traversing the tree is T D , where T is the number of split types and D is the depth of the tree. A trade-off can be made by using more types (T) while reducing the depth (D).

[0143] In some implementations, the CB may be further divided. For example, for the purpose of intra-frame or inter-frame prediction during the encoding and decoding process, the CB may be further divided into a plurality of prediction blocks. In other words, the CB may be further divided into different sub-partitions in which separate prediction decisions / configurations may be made. In parallel, in order to depict the level of transformation or inverse transformation of the video data, the CB may be further divided into a plurality of transform blocks (TB). The schemes for dividing the CB into PBs and TBs may be the same or different. For example, each division scheme may be performed using its own process, for example, based on various features of the video data. In some example implementations, the division schemes of the PB and TB may be independent. In some other example implementations, the division schemes and boundaries of the PB and TB may be interrelated. For example, in some implementations, the TB may be divided after the PB division, in particular, each PB is determined after the coding block division, and then may be further divided into one or more TBs. For example, in some implementations, a PB may be divided into one, two, four, or other number of TBs.

[0144] In some implementations, in order to divide the basic block into coding blocks and further into prediction blocks and / or transform blocks, the luma channel and the chroma channel may be processed differently. For example, in some implementations, for the luma channel, it may be allowed to divide the coding block into prediction blocks and / or transform blocks, while for the chroma channel, it may not be allowed to divide the coding block into prediction blocks and / or transform blocks. Therefore, in such an implementation, the transformation and / or prediction of the luma block may be performed only at the coding block level. For another example, the minimum transform block size of the luma channel and the chroma channel may be different, for example, the coding block of the luma channel may be allowed to be divided into smaller transform and / or prediction blocks than the chroma channel. For yet another example, the maximum depth of dividing the coding block into transform blocks and / or prediction blocks may be different between the luma channel and the chroma channel, for example, it may be allowed to divide the coding block of the luma channel into deeper transform blocks and / or prediction blocks than the chroma channel. For a specific example, the luma coding block may be partitioned into transform blocks of multiple sizes, which may be represented by recursive partitioning with up to 2 levels, and may allow transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, and transform block sizes from 4×4 to 64×64. However, for chroma blocks, only the largest possible transform block specified for the luma block may be allowed.

[0145] In some example implementations of partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partition may depend on whether the PB is intra-coded or inter-coded.

[0146] The partitioning of the coding block (or prediction block) into transform blocks can be implemented in various example schemes, including but not limited to recursively or non-recursively quadtree partitioning and predetermined pattern partitioning, and additionally considering transform blocks at the boundaries of the coding block or prediction block. In general, the resulting transform blocks may be at different partitioning levels, may not have the same size, and may not need to be square in shape (e.g., they may be rectangular with some allowed size and aspect ratio). Fig.15 , Fig.16 and Fig.17 Other examples are described in more detail.

[0147] However, in some other implementations, the CBs obtained via any of the above-mentioned partitioning schemes may be used as basic or minimum coding blocks for prediction and / or transformation. In other words, no further segmentation is performed for the purpose of performing inter-frame prediction / intra-frame prediction and / or transformation. For example, the CBs obtained according to the above-mentioned QTBT scheme can be directly used as units for performing prediction. Specifically, this QTBT structure eliminates the concept of multiple partitioning types, that is, it eliminates the separation of CU, PU and TU, and provides greater flexibility for the CU / CB partition shape as described above. In this QTBT block structure, the CU / CB can have a square or rectangular shape. The leaf nodes of this QTBT are used as units for prediction and transformation processing without any further partitioning. This means that in this exemplary QTBT coding block structure, the CU, PU and TU have the same block size.

[0148] The various CB partitioning schemes described above and further partitioning of CB into PBs and / or TBs (excluding PB / TB partitioning) may be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0149] A specific example implementation of coding block and transform block partitioning is described below. In such an example implementation, recursive quadtree partitioning or the above-mentioned predefined partitioning mode (e.g. Fig. 9 and Fig.10 The basic block is divided into coding blocks according to the pattern in (in). At each level, whether further quadtree segmentation should be continued for a specific partition can be determined by local video data characteristics. The resulting CBs can be at various quadtree segmentation levels and have various sizes. The decision on whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode the picture area can be made at the CB level (or CU level, for all three color channels). Each CB can be further divided into one, two, four or other number of PBs according to a predefined PB segmentation type. Within a PB, the same prediction process can be applied, and relevant information can be sent to the decoder based on the PB. After obtaining the residual block by applying the prediction process based on the PB segmentation type, the CB can be divided into TBs according to another quadtree structure of the coding tree similar to the CB. In this specific implementation, the CB or TB can be, but not limited to, a square. In addition, in this specific example, for inter-frame prediction, the PB can be a square or rectangular, while for intra-frame prediction, the PB can be only a square. The coding block can be divided into, for example, four square TBs. Each TB can be further recursively partitioned (using quadtree partitioning) into smaller TBs, called residual quadtrees (RQTs).

[0150] Another example implementation for partitioning basic blocks into CBs, PBs, and / or TBs is further described below. For example, instead of using a quadtree such as Fig. 9 or Fig.10 The multiple partition unit types shown in . The separation of CB, PB and TB (i.e., dividing CB into PBs and / or TBs, and dividing PBs into TBs) may be abandoned unless the size of CBs is too large for the maximum transform length, and such CBs may need to be further split. The example partitioning scheme can be designed to support greater flexibility in the shape of CB partitions, so that both prediction and transformation can be performed at the CB level without further partitioning. In such a coding tree structure, the CB can have a square or rectangular shape. Specifically, the coding tree block (CTB) can first be divided by a quadtree structure. Then, the quadtree leaf nodes can be further divided by an embedded multi-type tree structure. Fig.11 An example of an embedded multi-type tree structure using binary or ternary partitioning is shown in FIG. Fig.11 The exemplary multi-type tree structure includes four types of splits, referred to as vertical binary split (SPLIT_BT_VER) (1102), horizontal binary split (SPLIT_BT_HOR) (1104), vertical ternary split (SPLIT_TT_VER) (1106), and horizontal ternary split (SPLIT_TT_HOR) (1108). The CBs then correspond to the leaves of the multi-type tree. In this example implementation, unless the CB is too large for the maximum transform length, the split is used for both prediction and transform processing without any further partitioning. This means that in most cases, in a quadtree with an embedded multi-type tree coding block structure, the CB, PB, and TB have the same block size. An exception occurs when the maximum supported transform length is less than the width or height of the color component of the CB. In some implementations, in addition to binary or ternary splits, Fig.11 The embedded mode can also include quadtree partitioning.

[0151] Fig.12 A specific example of a quadtree of an embedded multi-type tree coding block structure for block partitioning (including quadtree, binary tree and ternary partitioning options) for a basic block is shown. In more detail, Fig.12 The basic block 1200 is shown to be partitioned by a quadtree into four square partitions 1202, 1204, 1206, and 1208. For each quadtree partition, it is decided to further use Fig.11 The multi-type tree structure and quadtree are further segmented. Fig.12In the example of FIG. 1204, partition 1204 is not further partitioned. Partition 1202 and partition 1208 each use another quadtree partition. For partition 1202, the upper left, upper right, lower left, and lower right partitions of the second quadtree partition use quadtrees, Fig.11 Horizontal binary split 1104, non-split and Fig.11 The horizontal three-branch partition 1108 is a three-level partition. The partition 1208 uses another quadtree partition. The upper left, upper right, lower left, and lower right partitions of the second quadtree partition are respectively Fig.11 The vertical trident split 1106, no split, no split, Fig.11 The horizontal binary segmentation 1104 is a three-level segmentation. Fig.11 The horizontal binary partition 1104 and the horizontal trifurcated partition 1108 further partition the two sub-partitions of the third level upper left partition 1208. Partition 1206 adopts the following Fig.11 The second level segmentation mode after the vertical binary segmentation 1102 is divided into two partitions, and then according to Fig.11 The third level segmentation is performed by the horizontal trifurcated segmentation 1108 and the vertical binary segmentation 1102. Fig.11 The horizontal binary split 1104 further applies the fourth level split to one of them.

[0152] For the specific example above, the maximum luma transform size may be 64x64, and the maximum supported chroma transform size may be different from luma, e.g. 32x32. Fig.12 The above example CBs are typically not further split into smaller PBs and / or TBs. When the width or height of a luminance coding block or a chrominance coding block is greater than the maximum transform width or height, the luminance coding block or the chrominance coding block may be automatically split in the horizontal and / or vertical directions to meet the transform size constraints in that direction.

[0153] In the above specific example for dividing the basic block into CBs, as described above, the coding tree scheme can support the ability of luma and chroma to have separate block tree structures. For example, for P slices and B slices, the luma and chroma CTBs in a CTU can share the same coding tree structure. For example, for I slices, luma and chroma can have separate coding block tree structures. When a separate block tree structure is used, the chroma CTB can be divided into chroma CBs using one coding tree structure and the chroma CTB can be divided into chroma CBs using another coding tree structure. This means that a CU in an I slice may consist of a coding block of a luma component or a coding block of two chroma components, while a CU in a P slice or a B slice always consists of coding blocks of all three color components unless the video is monochrome.

[0154] When the coding block is further divided into multiple transform blocks, the transform blocks therein can be ordered in the code stream in various orders or scanning modes. The following is a further detailed description of example implementations of dividing a coding block or prediction block into transform blocks and the encoding order of the transform blocks. In some example implementations, as described above, transform partitioning can support transform blocks of multiple shapes, such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transform block sizes ranging from 4×4 to 64×64. In some implementations, if the coding block is less than or equal to 64×64, the transform block partitioning can be applied only to the luminance component, so that for the chrominance block, the transform block size is the same as the coding block size. Otherwise, if the coding block width or height is greater than 64, both the luminance and chrominance coding blocks can be implicitly split into multiples of min(W, 64)×min(H, 64) and min(W, 32)×min(H, 32) transform blocks, respectively.

[0155] In some example implementations of transform block partitioning, for intra-coded blocks and inter-coded blocks, the coding block can be further partitioned into multiple transform blocks, and its partition depth can reach a predetermined number of levels (e.g., 2 levels). The transform block partition depth and size can be associated. For some example implementations, the mapping from the transform block size of the current depth to the transform block size of the next depth is shown in Table 1 below.

[0156] Table 1 Transform block partition size settings

[0157]

[0158] Based on the example mapping of Table 1, for a 1:1 square block, the next level transform partitioning can create four 1:1 square sub-transform blocks. The transform block partitioning can stop at, for example, 4×4. Therefore, the transform block size of 4×4 at the current depth corresponds to the same size 4×4 at the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform partitioning can create two 1:1 square sub-transform blocks, and for a 1:4 / 4:1 non-square block, the next level transform partitioning can create two 1:2 / 2:1 sub-transform blocks.

[0159] In some examples, for the luma component of an intra-coded block, additional restrictions may be imposed relative to the transform block partitioning. For example, for each level of transform partitioning, all sub-transform blocks may be constrained to have equal size. For example, for a 32×16 coding block, a level 1 transform partitioning creates two 16×16 sub-transform blocks and a level 2 transform partitioning creates eight 8×8 sub-transform blocks. In other words, the second level partitioning must be applied to all first level sub-blocks to keep the transform unit sizes equal. Fig.151504 shows an example of dividing an intra-coded square block into transform blocks according to Table 1, and the coding order shown by arrows. Specifically, 1502 shows a square coding block. 1504 shows that the coding block is divided into 4 transform blocks of equal size by the first level division according to Table 1, and the coding order of these 4 transform blocks is indicated by arrows. 1506 shows that all the first-level equal-sized blocks are divided into 16 transform blocks of equal size by the second level division according to Table 1, and the coding order of these 16 transform blocks is indicated by arrows.

[0160] In some example implementations, the above restrictions on intra-frame coding may not apply to the luminance component of the inter-frame coded block. For example, after the first level transform partitioning, any sub-transform block can be further independently partitioned into more than one level. Therefore, the resulting transform blocks can be of the same size or different sizes. Fig.16 An example of dividing an inter-frame coding block into multiple transform blocks with their coding order is shown in FIG. Fig.16 In the example of FIG. 1602, according to Table 1, the inter-coded block 1602 is divided into two levels of transform blocks. At the first level, the inter-coded block is divided into four transform blocks of equal size. Then, only one of the four transform blocks (not all) is further divided into four sub-transform blocks, resulting in a total of 7 transform blocks with two different sizes, as shown in 1604. The example coding order of these 7 transform blocks is as follows: Fig.16 As shown by the arrow in 1604.

[0161] In some example implementations, for one or more chroma components, some additional restrictions on transform blocks may be applied. For example, for one or more chroma components, the transform block size may be as large as the coding block size, but not smaller than a predefined size, such as 8×8.

[0162] In some other example implementations, for coding blocks with a width (W) or height (H) greater than 64, both luminance and chrominance coding blocks can be implicitly split into multiples of min(W, 64)×min(H, 64) and min(W, 32)×min(H, 32) transform units, respectively. Here, in an embodiment of the present application, "min(a, b)" may return the smaller value between a and b.

[0163] Fig.17 Another alternative example scheme for dividing a coding block or prediction block into multiple transform blocks is further shown. Fig.17 As shown, a set of predefined partition types can be applied to the coding block according to the transform type of the coding block, instead of using recursive transform partitioning. Fig.17In the specific example shown, one of six example partition types may be applied to split a coding block into various numbers of transform blocks. This scheme for generating transform block partitions may be applied to coding blocks or prediction blocks.

[0164] In more detail, Fig.17 The partitioning scheme of provides up to 6 example partitioning types for any given transform type (transform type refers to, for example, the type of the main transform, such as ADST, etc.). In this scheme, a transform partitioning type can be assigned to each coding block or prediction block based on, for example, rate-distortion cost. In the example, the transform partitioning type assigned to the coding block or prediction block can be determined based on the transform type of the coding block or prediction block. A specific transform partitioning type can correspond to a transform block partition size and mode, such as Fig.17 As shown in the 6 transform partition types shown in . The correspondence between various transform types and various transform partition types can be predefined. An example is shown below, where the uppercase label indicates the transform partition type that can be assigned to the coding block or prediction block based on the rate-distortion cost:

[0165] PARTITION_NONE (no partitioning): Allocate a transform size equal to the block size.

[0166] PARTITION_SPLIT: Allocates a transform size whose width is half the width of the block size and whose height is half the height of the block size.

[0167] PARTITION_HORZ (partition horizontally): Allocates a transform size whose width is the same as the width of the block size and whose height is half the height of the block size.

[0168] PARTITION_VERT (vertical partition): Allocate a transform size whose width is half the width of the block size and whose height is the same as the height of the block size.

[0169] PARTITION_HORZ4 (Partition Horizontal by 4): Allocates a transform size whose width is the same as the width of the block size and whose height is one quarter of the height of the block size.

[0170] PARTITION_VERT4 (vertical partition by 4): Allocates a transform size whose width is one quarter of the width of the block size and whose height is the same as the height of the block size.

[0171] In the above example, if Fig.17 The transform partition type shown contains uniform transform sizes for the partitioned transform blocks. This is just an example and is not limiting. In some other implementations, mixed transform block sizes can be used for the partitioned transform blocks in a specific partition type (or mode).

[0172] The PBs (or CBs, also referred to as PBs when not further divided into prediction blocks) obtained according to any of the above partitioning schemes can become a single block for encoding via intra-frame or inter-frame prediction. For inter-frame prediction of the current PB, a residual between the current block and the prediction block can be generated, encoded, and included in the encoded bitstream.

[0173] Inter-frame prediction can be implemented, for example, in a single reference mode or a composite reference mode. In some implementations, a skip flag can first be included in the code stream of the current block (or higher level) to indicate whether the current block is inter-frame encoded and whether it is not skipped. If the current block is inter-frame encoded, another flag can be further included in the code stream as a signal to indicate whether a single reference mode or a composite reference mode is used to predict the current block. For a single reference mode, one reference block can be used to generate a prediction block for the current block. For a composite reference mode, two or more reference blocks can be used, for example, to generate a prediction block by weighted averaging. The composite reference mode can be referred to as more than one reference mode, two reference modes, or multiple reference modes. One or more reference blocks can be identified using one or more reference frame indexes and additionally using one or more corresponding motion vectors indicating the position (e.g., horizontal and vertical pixel) offset between the reference block and the current block. For example, the inter-frame prediction block of the current block can be generated from a single reference block identified as a prediction block in a single reference mode by a motion vector in a reference frame, while for a composite reference mode, the prediction block can be generated by weighted averaging two reference blocks in two reference frames indicated by two reference frame indices and two corresponding motion vectors. The motion vector can be encoded and included in the codestream in various ways.

[0174] In some implementations, the encoding or decoding system may have a decoded picture buffer (DPB). Some images / pictures may be stored in the DPB waiting to be displayed (in the decoding system), and some images / pictures in the DPB may be used as reference frames to implement inter-frame prediction (in the decoding system or the encoding system). In some implementations, the reference frames in the DPB may be marked as short-term references or long-term references for the current image being encoded or decoded. For example, a short-term reference frame may include a frame for performing the following operations, for inter-frame prediction of a block in the current frame or for inter-frame prediction of a block in a predetermined number (e.g., 2) of subsequent video frames closest to the current frame in decoding order. Long-term reference frames may include frames in the DPB that can be used to predict image blocks in frames that are more than a predefined number of frames away from the current frame in decoding order. Information about such tags of short-term reference frames and long-term reference frames may be referred to as a reference picture set (RPS), and this information may be added to the header of each frame in the encoded code stream. Each frame in the encoded video stream may be identified by a Picture Order Counter (POC), which is numbered in an absolute manner or in an order relative to a group of pictures starting from, for example, the i-frame according to the playback sequence.

[0175] In some example implementations, one or more reference picture lists containing identifiers of short-term and long-term reference frames for inter-frame prediction may be formed based on the information in the RPS. For example, a single picture reference list may be formed for unidirectional inter-frame prediction, denoted as L0 reference (or reference list 0), and two picture reference lists may be formed for bidirectional inter-frame prediction, with the two prediction directions being denoted as L0 (or reference list 0) and L1 (or reference list 1). The reference frames included in the L0 and L1 lists may be sorted in various predetermined ways. The lengths of the L0 and L1 lists may be written into the video bitstream. When multiple reference frames used to generate a prediction block by weighted averaging in a composite prediction mode are located on the same side of the block to be predicted, the unidirectional inter-frame prediction may be in a single reference mode or in a composite reference mode. Bidirectional inter-frame prediction may be in composite mode only because bidirectional inter-frame prediction involves at least two reference blocks.

[0176] In some implementations, a merge mode (MM) for inter-frame prediction may be implemented. Typically, for merge mode, a motion vector in a single reference prediction of the current PB or one or more motion vectors in a composite reference prediction may be derived from other motion vectors, rather than being calculated and written independently. For example, in a coding system, the current motion vector of the current PB is represented by the difference between the current motion vector and one or more other already encoded motion vectors (referred to as reference motion vectors). This motion vector difference, rather than the entirety of the current motion vector, may be encoded and included in the bitstream, and this difference may be linked to the reference motion vector. Accordingly, in a decoding system, the motion vector corresponding to the current PB may be derived based on the decoded motion vector difference and the decoded reference motion vector linked thereto. As a specific form of general merge mode (MM) inter-frame prediction, this inter-frame prediction based on motion vector difference may be referred to as a merge mode with motion vector difference (MMVD). Therefore, a general MM or a specific MMVD may be implemented to improve coding efficiency by utilizing the correlation between motion vectors associated with different PBs. For example, adjacent PBs may have similar motion vectors, and therefore, the MVD may be small and may be efficiently encoded. For another example, for blocks of similar location / position in space, the motion vectors may be correlated in time (between frames).

[0177] In some example implementations, during the encoding process, an MM flag may be included in the codestream to indicate whether the current PB is in merge mode. Additionally or alternatively, an MMVD flag may be included and written into the codestream during the encoding process to indicate whether the current PB is in MMVD mode. The MM and / or MMVD flags or indicators may be provided at the PB level, CB level, CU level, CTB level, CTU level, slice level, picture level, etc. For a specific example, an MM flag and an MMVD flag may be included for the current CU, and the MMVD flag may be written immediately after the skip flag and the MM flag to specify whether the MMVD mode is for the current CU.

[0178] In some example implementations of MMVD, a reference motion vector (RMV) candidate list or MV prediction candidate list for motion vector prediction may be formed for the predicted block. The RMV candidate list may contain a predetermined number (e.g., 2) of MV prediction candidate blocks, whose motion vectors may be used to predict the current motion vector. The RMV candidate blocks may include blocks selected from adjacent blocks and / or temporary blocks of the same frame (e.g., blocks at the same position in the previous frame or the next frame of the current frame). These options represent blocks at spatial or temporal positions relative to the current block, which may have motion vectors similar to or the same as the current block. The size of the MV predictor candidate list may be predetermined. For example, the list may contain two or more candidates. To be on the RMV candidate list, the candidate block may, for example, need to have the same reference frame (or multiple reference frames) as the current block, must exist (e.g., when the current block is close to the edge of the frame, a boundary check needs to be performed), and must have been encoded during the encoding process, and / or decoded during the decoding process. In some implementations, if the merge candidate list is available and meets the above conditions, it can be filled first with spatially adjacent blocks (scanned in a specific predefined order), and then if there is still space available in the list, it can be filled with temporary blocks. For example, these adjacent RMV candidate blocks can be selected from the left block and the top block of the current block. The RMV prediction candidate list can be dynamically formed at different levels (sequence, picture, frame, slice, super block, etc.) as a dynamic reference list (Dynamic Reference List, DRL). The DRL can be written into the bitstream.

[0179] In some implementations, the actual MV prediction candidate used as a reference motion vector for predicting the motion vector of the current block can be written to the bitstream. In the case where the RMV candidate list contains two candidates, a one-bit flag called the merge candidate flag can be used to indicate the selection of the reference merge candidate. For the current block predicted in composite mode, each of the multiple motion vectors predicted using the MV predictor can be associated with a reference motion vector from the merge candidate list. The encoder can determine which of the RMV candidates better predicts the current coding block and write the selection as an index into the DRL.

[0180] In some example implementations of MMVD, after an RMV candidate is selected and used as a basic motion vector predictor for a motion vector to be predicted, a motion vector difference (MVD or delta MV, representing the difference between a motion vector to be predicted and a reference candidate motion vector) may be calculated in the encoding system. Such an MVD may include information representing the magnitude of the MV difference and the direction of the MV difference, both of which may be written into the bitstream. The magnitude of the motion difference and the direction of the motion difference may be written into the bitstream in various ways.

[0181] In some example implementations of the MMVD, the distance index may be used to specify the magnitude information of the motion vector difference and indicate one of a set of predefined offsets representing a predefined motion vector difference relative to a starting point (reference motion vector). The MV offset according to the index indicated by the signal may then be added to the horizontal component or the vertical component of the starting (reference) motion vector. Whether the horizontal component or the vertical component of the reference motion vector should be offset may be determined by the direction information of the MVD. An example of a predefined relationship between a distance index and a predefined offset is specified in Table 2.

[0182] Table 2 Example relationship between distance index and predefined MV offset

[0183]

[0184] In some exemplary implementations of MMVD, a direction index may be further written into the bitstream and used to indicate the direction of the MVD relative to the reference motion vector. In some implementations, the direction may be limited to either the horizontal direction or the vertical direction. An example of a 2-bit direction index is shown in Table 3. In the example of Table 3, the interpretation of the MVD may vary depending on the information of the start / reference MVs. For example, when the start / reference MV corresponds to a single-prediction block or to a dual-prediction block, where both reference frame lists point to the same side of the current picture (i.e., the POCs of both reference pictures are greater than the POC of the current picture, or are less than the POC of the current picture), the symbol in Table 3 may specify the sign (direction) of the MV offset added to the start / reference MV. When the start / reference MV corresponds to a bi-predicted block of two reference pictures on different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture, and the POC of the other reference picture is less than the POC of the current picture), and the difference between the reference POC in picture reference list 0 and the current frame is greater than the difference between the reference POC in picture reference list 1 and the current frame, the symbols in Table 3 may specify the sign of the MV offset added to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture in picture reference list 1 may have an opposite value (have an opposite sign for the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is greater than the difference between the reference POC in picture reference list 0 and the current frame, the symbols in Table 3 may specify the sign of the MV offset added to the reference MV associated with picture reference list 1, and the sign of the offset of the reference MV associated with picture reference list 0 has an opposite value.

[0185] Table 3 Example implementation of the sign of the MV offset specified by the direction index

[0186] Directions to IDX 00 01 10 11 x-axis (horizontal) + – N / A N / A y-axis (vertical) N / A N / A + –

[0187] In some example implementations, the MVD may be scaled based on the difference in POCs in each direction. If the difference in POCs is the same in both lists, no adjustment is required. Otherwise, if the POC difference in reference list 0 is greater than the POC difference in reference list 1, the MVD of reference list 1 is adjusted. If the POC difference in reference list 1 is greater than the POC difference in list 0, the MVD of list 0 may be adjusted in the same manner. If the starting MV is uni-predicted, the MVD is added to the available MV or reference MV.

[0188] In some example implementations of MVD encoding and writing for bidirectional composite prediction, in addition to encoding and writing two MVDs separately, symmetric MVD encoding can also be implemented so that only one MVD needs to be written, and the other MVD can be derived based on the written MVD. In this implementation, motion information including reference picture indexes of both list-0 and list-1 is written to the bitstream. However, only the MVD associated with, for example, reference list-0 is written, while the MVD associated with reference list-1 is not written but derived. Specifically, at the slice level, a flag can be included in the bitstream, called "mvd_l1_zero_flag", which is used to indicate whether reference list-1 is not written to the bitstream. If the flag is 1, indicating that reference list-1 is equal to zero (and therefore not written), a bidirectional prediction flag called "BiDirPredFlag" can be set to 0, which means that there is no bidirectional prediction. Otherwise, if mvd_l1_zero_flag is zero, BiDirPredFlag may be set to 1 if the nearest reference picture in list-0 and the nearest reference picture in list-1 form a forward and backward reference picture pair or a backward and forward reference picture pair, and the reference pictures of list-0 and list-1 are both short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. BiDirPredFlag of 1 may indicate that a symmetric mode flag is additionally written to the bitstream. When BiDirPredFlag is 1, the decoder may extract the symmetric mode flag from the bitstream. For example, the symmetric mode flag may be written at the CU level (if necessary), and it may indicate whether the symmetric MVD coding mode is being used for the corresponding CU. When the symmetric mode flag is 1, it indicates that the symmetric MVD coding mode is used, and only the reference picture indexes of both list-0 and list-1 (called "mvp_l0_flag" and "mvp_l1_flag") and the MVD associated with list-0 (called "MVD0") are written, and another motion vector difference "MVD1" will be derived instead of being written. For example, MVD1 can be derived as -MVD0. Therefore, in the exemplary symmetric MVD mode, only one MVD is written to the bitstream. In some other exemplary implementations of MV prediction, for single reference mode and composite reference mode MV prediction, a coordination scheme can be used to implement the general merge mode MMVD and some other types of MV prediction. Various syntax elements can be used to indicate the way in which the MV of the current block is predicted.

[0189] For example, for single reference mode, the following MV prediction modes can be written into the bitstream:

[0190] NEARMV - directly uses one of the motion vector predictors (MVP) in the list indicated by the Dynamic Reference List (DRL) index without using any MVD.

[0191] NEWMV – Uses one of the motion vector predictors (MVPs) in the list written by the DRL index as a reference and applies a delta to the MVP (e.g., using MVD).

[0192] GLOBALMV – Use motion vectors based on frame-level global motion parameters.

[0193] Likewise, for the composite reference inter prediction mode using two reference frames corresponding to the two MVs to be predicted, the following MV prediction modes can be written into the bitstream:

[0194] NEAR_NEARMV - For each of the two MVs to be predicted, use one of the motion vector predictors (MVPs) in the list written by the DRL index instead of the MVD.

[0195] NEAR_NEWMV - To predict the first of the two motion vectors, one of the motion vector predictors (MVPs) in the list written by the DRL index is used as a reference MV without using MVD; to predict the second of the two motion vectors, one of the motion vector predictors (MVPs) in the list written by the DRL index is used as a reference MV in combination with an additionally written incremental MV (MVD).

[0196] NEW_NEARMV - To predict the second of the two motion vectors, one of the motion vector predictors (MVPs) in the list written by the DRL index is used as a reference MV without using MVD; to predict the first of the two motion vectors, one of the motion vector predictors (MVPs) in the list written by the DRL index is used as a reference MV in combination with an additionally written incremental MV (MVD).

[0197] NEW_NEWMV - uses one of the motion vector predictors (MVPs) in the list written by the DRL index as the reference MV, and uses it in conjunction with the additionally written delta MV to predict each of the two MVs.

[0198] GLOBAL_GLOBALMV - Use the MV of each reference according to the frame-level global motion parameters.

[0199] Therefore, the term "NEAR" mentioned above refers to MV prediction using a reference MV without using MVD as a general merge mode, while the term "NEW" refers to MV prediction involving using a reference MV and offsetting it with a written MVD as an MMVD mode. For composite inter-frame prediction, the reference basic motion vector and motion vector increment mentioned above can usually be different or independent between the two references, even if they can be correlated, and this correlation can be used to reduce the amount of information required to write the two motion vector increments. In this case, the joint writing of the two MVDs can be achieved and indicated in the bitstream.

[0200] The above dynamic reference list (DRL) can be used to store a set of indexed motion vectors, which are dynamically stored and considered as candidate motion vector predictors.

[0201] In some example implementations, a predefined resolution of the MVD may be allowed. For example, a motion vector precision (or accuracy) of 1 / 8 pixel may be allowed. The MVD described above in various MV prediction modes may be constructed and written to the bitstream in various ways. In some implementations, various syntax elements may be used to represent the above motion vector differences in reference frame list 0 or list 1.

[0202] For example, a syntax element called "mv_joint" may specify which components of the motion vector difference associated with it are non-zero. For MVD, this is written jointly for all non-zero components. For example, mv_joint has the following values.

[0203] 0 may indicate that there is no non-zero MVD in the horizontal or vertical direction;

[0204] 1 may indicate that there is non-zero MVD only in the horizontal direction;

[0205] 2 may indicate that there is a non-zero MVD only in the vertical direction;

[0206] 3 may indicate that there is a non-zero MVD in both the horizontal and vertical directions.

[0207] When the "mv_joint" syntax element of the MVD signal indicates that there are no non-zero MVD components, no further MVD information may be written. If the "mv_joint" syntax indicates that there are one or two non-zero components, additional syntax elements may be written for each non-zero MVD component, as described below.

[0208] For example, a syntax element called "mv_sign" may be used to additionally specify whether the corresponding motion vector difference amount is positive or negative.

[0209] For another example, a syntax element called "mv_class" can be used to specify the class of motion vector differences in a predefined set of classes for corresponding non-zero MVD components. For example, the predefined classes of motion vector differences can be used to divide the continuous size space of motion vector differences into non-overlapping ranges, each range corresponding to an MVD class. Therefore, the written MVD class indicates the size range of the corresponding MVD component. In the example implementation shown in Table 4 below, a higher class corresponds to a motion vector difference with a larger size range. In Table 4, the symbol (n, m] is used to represent a range of motion vector differences greater than n pixels and less than or equal to m pixels.

[0210] Table 4 Size categories of motion vector differences

[0211]

[0212]

[0213] In some other examples, a syntax element called "mv_bit" may be further used to specify the integer portion of the offset between the non-zero motion vector difference component and the starting size of the corresponding written MV class size range. The number of bits required in "my_bit" to write the entire range of each MVD class may vary as a function of the MV class. For example, in the implementation of Table 4, MV_CLASS 0 and MV_CLASS1 may only require a single bit to represent an integer pixel offset of 1 or 2 from MVD 0. In the example implementation of Table 4, each higher MV_CLASS may require one "mv_bit" bit to be gradually increased from the previous MV_CLASS.

[0214] In some other examples, a syntax element called "mv_fr" may be further used to specify the first 2 decimal places of the motion vector difference for the corresponding non-zero MVD component, and a syntax element called "mv_hp" may be used to specify the third decimal place (high resolution place) of the motion vector difference for the corresponding non-zero MVD component. The two "mv_fr" bits actually provide an MVD resolution of 1 / 4 pixel, while the "mv_hp" bits may further provide a resolution of 1 / 8 pixel. In some other implementations, multiple "mv_hp" bits may be used to provide an MVD pixel resolution finer than 1 / 8 pixel. In some example implementations, additional flags may be written to the bitstream at one or more different levels to indicate whether an MVD resolution of 1 / 8 pixel or higher is supported. If the MVD resolution is not applied to a particular coding unit, the syntax element above for the corresponding unsupported MVD resolution may not be written to the bitstream.

[0215] In the example implementation above, fractional resolution may be independent of different categories of MVD. In other words, a fractional MVD for non-zero MVD components may be written using a predefined number of "mv_fr" and "mv_hp" bits regardless of the size of the motion vector difference, thereby providing similar options for motion vector resolution.

[0216] However, in some other example implementations, the resolution of motion vector differences in various MVD size categories can be distinguished. Specifically, for MVDs with larger sizes in higher MVD categories, high-resolution MVDs may not provide statistically significant improvements in compression efficiency. Therefore, for larger MVD size ranges, MVDs can be encoded with decreasing resolutions (integer pixel resolutions or fractional pixel resolutions), and the larger MVD size ranges correspond to higher MVD size categories. Similarly, for generally larger MVD values, MVDs can be encoded with decreasing resolutions (integer pixel resolutions or fractional pixel resolutions). This MVD resolution that depends on MVD categories or depends on MVD size can generally be referred to as adaptive MVD resolution, adaptive MVD resolution that depends on amplitude, or MVD resolution that depends on size. The term "resolution" can be further referred to as "pixel resolution". Adaptive MVD resolution can be implemented in various situations described in the following example implementations to achieve better compression efficiency overall. In particular, since it is statistically observed that treating the MVD resolution of a larger or higher-level MVD in a non-adaptive manner similar to the MVD resolution of a smaller or lower-level MVD may not significantly increase the residual coding efficiency of inter-frame prediction for blocks of the larger or higher-level MVD, the number of bits of signaling reduced by focusing on the lower-precision MVD may be greater than the additional number of bits required for residual coding of inter-frame prediction due to such lower-precision MVD. In other words, for larger or higher-level MVDs, using a higher MVD resolution may not produce much coding gain over using a lower MVD resolution.

[0217] In some general example implementations, the pixel resolution or precision of the MVD may decrease or not increase as the MVD class increases. Decreasing the pixel resolution of the MVD corresponds to a coarser MVD (or a larger step size from one MVD level to the next MVD level). In some implementations, the correspondence between MVD pixel resolution and MVD class may be specified, predefined, or preconfigured, and therefore may not need to be written into the encoded bitstream.

[0218] In some example implementations, the MV categories of Table 3 may each be associated with a different MVD pixel resolution.

[0219] In some example implementations, each MVD category may be associated with a single allowed resolution. In some other implementations, one or more MVD categories may be associated with two or more selectable MVD pixel resolutions. Thus, after writing the current MVD component having such an MVD category into the bitstream, an additional write follows indicating which selectable pixel resolution is selected for the current MVD component.

[0220] In some example implementations, the adaptively allowed MVD pixel resolutions may include, but are not limited to, 1 / 64 pixel, 1 / 32 pixel, 1 / 16 pixel, 1 / 8 pixel, 1-4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels, 4 pixels ... (in descending order of resolution). Therefore, each ascending MVD category may be associated with one of these resolutions in a non-ascending manner. In some implementations, an MVD category may be associated with two or more resolutions, and the higher resolution may be lower than or equal to the lower resolution of the previous MVD category. For example, if MV_CLASS_3 of Table 4 may be associated with optional 1 pixel and 2 pixel resolutions, the highest resolution that MV_CLASS_4 of Table 4 may be associated with will be 2 pixels. In some other implementations, the highest allowed resolution of an MV category may be higher than the lowest allowed resolution of a previous (lower) MV category. However, the allowed average resolution of an ascending MV category may only be non-ascending.

[0221] In some implementations, when fractional pixel resolutions higher than 1 / 8 pixel are allowed, the "mv_fr" and "mv_hp" signaling may be correspondingly extended to a total of more than 3 fractional digits.

[0222] In some example implementations, fractional pixel resolution may be allowed only for MVD classes that are lower than or equal to a threshold MVD class. For example, fractional pixel resolution may be allowed only for MVD-CLASS 0 and not for all other MV classes of Table 4. Similarly, fractional pixel resolution may be allowed only for MVD classes that are lower than or equal to any one of the other MV classes of Table 4. For other MVD classes that are higher than the threshold MVD class, only integer pixel resolutions of the MVD are allowed. In this way, for MVDs written with MVD classes that are higher than or equal to the threshold MVD class, it may not be necessary to write fractional resolution signaling such as one or more of the "mv-fr" and / or "mv-hp" bits. For MVD classes with resolutions lower than 1 pixel, the number of bits in the "mv-bit" signaling may be further reduced. For example, for MV_CLASS_5 in Table 4, the range of MVD pixel offset is (32, 64], so 5 bits are required to write the entire range of 1 pixel resolution. However, if MV_CLASS_5 is associated with 2-pixel MVD resolution (lower than 1 pixel resolution), "mv-bit" may require 4 bits instead of 5 bits, and neither "mv-fr" nor "mv-hp" need to be written after writing "mv_class" as "MV_CLASS_5".

[0223] In some example implementations, MVDs with integer values ​​below a threshold integer pixel value may only allow fractional pixel resolution. For example, for MVDs less than 5 pixels, only fractional pixel resolution may be allowed. Corresponding to this example, fractional resolution may be allowed for MV_CLASS_0 and MV_CLASS_1 of Table 4, while fractional resolution is not allowed for all other MV classes. For another example, for MVDs less than 7 pixels, only fractional pixel resolution may be allowed. Corresponding to this example, fractional resolution may be allowed for MV_CLASS_0 and MV_CLASS_1 in Table 4 (range less than 5 pixels), while fractional resolution is not allowed for MV_CLASS_3 and higher (range greater than 5 pixels). For MVDs belonging to MV_CLASS_2 with a pixel range containing 5 pixels, fractional pixel resolution of the MVD may be allowed or not allowed depending on the "mv-bit" value. If the "m-bit" value is written as 1 or 2 (so that the integer part of the written MVD is 5 or 6, calculated as the start of the pixel range of MV_CLASS_2, offset by 1 or 2, indicated by "m-bit"), then fractional pixel resolution may be allowed. Otherwise, if the "mv-bit" value is written as 3 or 4 (so that the integer part of the written MVD is 7 or 8), then fractional pixel resolution may not be allowed.

[0224] In some other implementations, only a single MVD value may be allowed for MV classes that are equal to or higher than a threshold MV class. For example, such a threshold MV class may be MV_CLASS2. Therefore, MV_CLASS_2 and above may only be allowed to have a single MVD value and no fractional pixel resolution. The single allowed MVD values ​​for these MV classes may be predefined. In some examples, the allowed single values ​​may be the higher end values ​​of the respective ranges of these MV classes in Table 4. For example, MV_CLASS_2 to MV_CLASS_10 may be higher than or equal to the threshold class of MV_CLASS2, and the single allowed MVD values ​​for these classes may be predefined as 8, 16, 32, 64, 128, 256, 512, 1024, and 2048, respectively. In some other examples, the allowed single values ​​may be the middle values ​​of the respective ranges of these MV classes in Table 4. For example, MV_CLASS_2 to MV_CLASS_10 may be above the class threshold, and the single allowed MVD values ​​for these classes may be predefined as 3, 6, 12, 24, 48, 96, 192, 384, 768, and 1536, respectively. Any other value within the range may also be defined as a single allowed resolution for each MVD class.

[0225] In the above implementation, when the written "mv_class" is equal to or above the predefined MVD level threshold, only the "mv_class" signaling is sufficient to determine the MVD value. Then "mv_class" and "mv_sign" are used to determine the size and direction of the MVD.

[0226] Therefore, when the MVD is written for only one reference frame (from reference frame list 0 or list 1, but not both), or when it is written for two reference frames jointly, the accuracy (or resolution) of the MVD can depend on the category of the associated motion vector difference in Table 3 and / or the size of the MVD.

[0227] In some other implementations, the pixel resolution or precision of the MVD may decrease or not increase as the MVD size increases. For example, the pixel resolution may depend on the integer portion of the MVD size. In some implementations, fractional pixel resolution may be allowed only for MVD sizes that are less than or equal to an amplitude threshold. For a decoder, the integer portion of the MVD size may be first extracted from the bitstream. The pixel resolution may then be determined, and then it may be determined whether there are any fractional MVDs in the bitstream and need to be parsed (e.g., if fractional pixel resolution is not allowed for a particular extracted MVD integer size, then the bitstream that needs to be extracted may not contain fractional MVD bits). The above example implementations related to adaptive MVD pixel resolutions that depend on MVD categories are applicable to adaptive MVD pixel resolutions that depend on MVD size. For a specific example, MVD categories that are higher than or contain a size threshold may only be allowed to have a predefined value.

[0228] The various example implementations above are for single reference mode. These implementations also apply to the NEW_NEARMV, NEAR_NEWMV, and / or NEW_NEWMV mode examples in composite prediction under MMVD. These implementations are generally applicable to adaptive resolution of any MVD.

[0229] When an adaptive (more specifically, size-adaptive) pixel resolution of MVD is used, the parameters related to the MV and MVD of the coding block may be interdependent. Parameters considered to be related to MV or MVD broadly refer to those information items that can affect how to select and detect RMV, how to write, calculate or derive RMV and MVD. These MV or MVD related parameters may include, but are not limited to:

[0230] The above-mentioned dynamic reference list (DRL) for identifying an ordered list of candidate reference motion vectors (RMVs) for predicting the MV of the current coding block. For example, the DRL can identify a set of spatial or temporal neighboring block positions that may have similar motion vectors as the current block. These positions correspond to RMVs that can be used as candidates for predicting the current motion vector. The encoder can select the RMV that best matches the RMV of the current coding block from these candidate RMVs, and then use the RMV and derive the corresponding MVD. The selected RMV can be represented or identified, for example, by the corresponding position or index in the DRL.

[0231] • The DRL index corresponding to the candidate RMV selected for the current coding block.

[0232] Indication to use adaptive MVD pixel resolution.

[0233] MVD information, including but not limited to mv_joint, mv_sign, mv_class, mv_bit, mv_fr, mv_hp as described above.

[0234] • An indication to use a motion compensation mode such as overlapped block motion compensation mode.

[0235] • An indication to use an early motion compensation mode such as warp motion mode.

[0236] Because these information items or parameters can be interdependent, particularly when adaptive MVD pixel resolution is used, whether written or derived, the order in which they are written, the amount of syntax used to write, and the derivation of the context used to encode / write these syntax elements can all be taken into account when designing a more efficient encoding-decoding scheme.

[0237] In some example implementations, whether one or more parameters related to MV or MVD are written into the video stream or derived from other written information may depend on whether adaptive MVD resolution is applied. Alternatively or additionally, if such specific parameters are written into the video stream, the manner in which they are written may depend on whether adaptive MVD resolution is applied.

[0238] For example, when adaptive MVD pixel resolution is applied, it can be specified that only the first N RMV candidates in the DRL are allowed to be used in the encoder. In other words, the encoder may be required to select one of the first N entries from the DRL of the current coding block that is inter-coded at adaptive MVD pixel resolution. In such an implementation, if adaptive MVD pixel resolution is not applied, the encoder can freely select the prediction for the current MV from the entire DRL list. The basis of this implementation can be based on the observation that RMV candidates with lower indices (corresponding to closer spatial neighboring blocks) in the DRL are statistically more likely to better predict the motion vector of the current block. In this way, the DRL index written to the codestream for blocks using adaptive MVD pixel resolution can require a smaller number of bits and can, for example, use a separate syntax element for writing.

[0239] Here, N is a positive number. N can be smaller than the entire index space of DRL. In some specific implementations, N can be 1 or 2, which means that when the current block uses adaptive MVD pixel resolution, the predictor MV or RMV is always the first RMV candidate, or the first or second RMV candidate in DRL.

[0240] The value of N may be predefined or may be written into the bitstream. For example, N may be written at various coding levels, including but not limited to sequence level, frame level, slice level, title level, or super block level. The value of N applies to various blocks that employ adaptive MVD pixel resolution within the level written.

[0241] In some implementations, when the N value is specified or written as 1, then for blocks using adaptive MVD pixel resolution, the DRL index does not need to be written to the bitstream. The index of the RMV extracted based on the DRL will be automatically derived as 1 at the decoder, referring to the first RMV candidate in the DRL. For example, when the N value is specified or written as 2, a single bit can be used to write the DRL index of the coded block.

[0242] In some other implementations, the method described above for writing or deriving DRL indices may only be applicable when inter-frame prediction is a single reference mode. In other words, when adaptive MVD pixel resolution is applied, and when inter-frame prediction is a single reference mode rather than a composite reference mode, it can be specified or written that only the first N MVP candidates in the DRL are allowed to be used by the encoder. Under this condition, the derivation or writing of the range of DRL indices can be similar to the derivation or writing described above, for example, N can be limited and can be written or predefined. When the inter-frame prediction mode is a composite reference mode, or when the inter-frame prediction mode is a single reference mode but adaptive MVD pixel resolution is not adopted, the encoder can select the RMV of the current coding block from the candidate RMVs corresponding to the full index range of the DRL.

[0243] Therefore, the DRL index written to the bitstream for each coding block can vary in range (or number of bits). For coding blocks using adaptive MVD pixel resolution, if these coding blocks are written, their DRL index may be from 1 to N, while for coding blocks that do not use adaptive MVD pixel resolution, their DRL index may be from 1 to the full index range of DRL. Therefore, the DRL index can follow different probability models in these different cases, so the entropy coding context of the written DRL index can be adopted / derived in different ways depending on whether adaptive MVD pixel resolution is adopted. In other words, when adaptive MVD pixel resolution is adopted, a set of contexts can be used to write DRL indexes. Otherwise, another set of contexts can be used to write DRL indexes. The derivation of the context for decoding the DRL index at the decoder will depend accordingly on whether adaptive MVD pixel resolution is adopted.

[0244] In some example implementations, the writing of motion compensation modes such as OBMC mode and / or warp motion may depend on whether adaptive MVD pixel resolution is adopted. For example, a flag for whether to use OBMC mode and / or warp motion may be written only when adaptive MVD pixel resolution is adopted for the current coding block. Otherwise, such a flag is not written (e.g., the decoder may assume that OBMC mode and / or warp motion is not adopted).

[0245] In some example implementations, the writing of the above-mentioned OBMC mode and / or warp motion may also be conditional on the inter-frame prediction mode of the current coding block being a single reference mode rather than a composite reference mode. In other words, the OBMC mode and / or warp motion flag may be written only when the current coding block is associated with a single reference frame and an adaptive MVD pixel resolution is adopted.

[0246] In some other example implementations, adaptive MVD pixel resolution may be used only with a single reference inter prediction mode. Thus, when adaptive MVD pixel resolution is used for a current coding block determined by writing or by deriving, this indicates that inter prediction is not a composite reference mode. In this case, there is no need to write a flag in the bitstream to indicate whether the current block uses a single reference mode or a composite reference mode.

[0247] In some other example implementations, whether to use a flag indicating whether the composite inter-intra mode is used may depend on whether adaptive MVD pixel resolution is applied and whether the current block is in a single reference mode. For example, the composite inter-intra mode of mixed inter and intra prediction for the current block may be potentially used only in a single reference mode and only when inter prediction is based on the use of adaptive MVD pixel resolution. Therefore, no flag is needed to write whether the composite inter-intra mode is used, and when the current block is not predicted by a single reference mode or does not rely on adaptive MVD pixel resolution, the composite inter-intra mode is considered not to be used.

[0248] In some other example implementations, the context derivation (contextderivation) for writing other MVD related grammars can depend on whether to apply adaptive MVD pixel resolution. As described above, these MVD related grammars can also include mv_joint, mv_class, mv_bit, mv_fr, mv_hp, etc. For example, mv_joint and / or mv_class can be statistically related to adaptive MVD pixel resolution, and can follow different probability models according to whether adaptive MVD pixel resolution is applied. Specifically, if adaptive MVD resolution is applied, a context can be used / derived to write mv_joint (or mv_class). Otherwise, another one or more different contexts may be used / derived to write mv_joint (or mv_class). The context dependency of mv_joint and / or mv_class to the use of adaptive MVD pixel resolution is only an example. Other MVD related grammatical elements can also be associated with the context derivation depending on whether adaptive MVD pixel resolution is applied.

[0249] In the various example implementations above, it is assumed that whether to use adaptive MVD pixel resolution in the current coding block can be extracted (as written) or derived from the bitstream before determining some other MV or MVD related information items or parameters, such as DRL index, etc. In some other example implementations, on the contrary, such other information items can be written or derived before determining whether adaptive MVD pixel resolution is used for the current coding block. Due to the correlation or correlation between these other information items and whether adaptive MVD pixel resolution should be used, the writing or deriving of whether to use adaptive MVD pixel resolution can depend on these other information items that have been extracted or derived from the bitstream.

[0250] For example, in some specific implementations, whether to apply adaptive MVD resolution may depend on the value of the DRL index. In such an implementation, the DRL index for determining the RMV of the current coding block may first be written into the codestream (or derived from the codestream in some other way). When the DRL index for the current coding block is written as a value in the range of 1 to N, it indicates that the RMV candidate in the DRL can be selected within a limited DRL index range. In this case, the encoder may further include a flag in the codestream to write whether the current coding block adopts adaptive MVD pixel resolution. However, if the written DRL index is outside the range 1-N, this may indicate that the current coding block does not adopt adaptive MVD pixel resolution, and therefore there is no need to include a flag indicating whether adaptive MVD pixel resolution is adopted in the codestream. In this case, the decoder will simply determine that adaptive MVD pixel resolution is not used by derivation. Here, N is a positive number less than the full index range of DRL. For example, N can be 1 or 2. In some implementations, N can be predefined. In some other example implementations, N can be written. For example, N may be written at various write levels, such as one of a sequence level, a frame level, a slice level, a header level, or a superblock level.

[0251] Fig.18 A flowchart 1800 of an example method is shown, which follows the principles of the implementation for adaptive MVD resolution and writing thereof described above. The example decoding method flow starts from S1801. In S1810, a video stream is received. In S1820, it is determined that the video block is inter-coded based on a prediction block and a motion vector (MV), wherein the MV is derived based on a reference motion vector (RMV) and a motion vector difference (MVD) of the video block. In S1830, data items associated with at least one of the RMV or the MVD are extracted or derived from the video stream depending on whether the MVD is encoded with a size-dependent adaptive MVD pixel resolution. In S1840, the MVD is extracted from the video stream; the MV is derived based on the extracted RMV and the MVD; and the video block is reconstructed based on at least the MV and the prediction block. The example method flow ends at S1899.

[0252] Fig.19A flowchart 1900 of another example method is shown, which follows the principles of the implementation of adaptive MVD resolution and writing thereof described above. The example decoding method flow starts from S1901. In S1910, a video stream is received. In S1920, it is determined that the video block is inter-coded based on a prediction block and a motion vector (MV), wherein the MV is derived based on a reference motion vector (RMV) and a motion vector difference (MVD) of the video block. In S1930, an RMV index of the video block mapped to a dynamic reference list (DRL) is extracted, and the DRL is used to identify multiple ordered candidate RMVs. In S1940, it is determined whether the MVD is encoded with a size-dependent adaptive MVD pixel resolution based on the value of the RMV index. The example method flow ends at S1999.

[0253] In the embodiments and embodiments of the embodiments of the present application, any steps and / or operations can be combined or arranged in any number or order as needed. Two or more of the steps and / or operations can be performed in parallel. The embodiments and implementation methods in the embodiments of the present application can be used alone or in combination in any order. In addition, each method (or embodiment), encoder and decoder can be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-temporary computer-readable medium. The embodiments in the embodiments of the present application can be applied to luminance blocks or chrominance blocks. The term block can be interpreted as a prediction block, a coding block, or a coding unit, i.e., a CU. The term block here can also be used to refer to a transform block. In the following items, when the block size is said, it can refer to the width or height of the block, or the maximum value of the width and height, or the minimum value of the width and height, or the area size (width*height), or the aspect ratio of the block (width: height, or height: width).

[0254] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig. 20 A computer system (2000) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0255] Computer software may be encoded using any suitable machine code or computer language that may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through interpreted code, microcode, etc.

[0256] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, IoT devices, etc.

[0257] Fig. 20 The components shown for the computer system (2000) are exemplary in nature and are not intended to impose any limitations on the scope of use or functionality of the computer software implementing the embodiments of the present application. The configuration of the components should not be interpreted as having any dependency or requirement on any one or combination of components shown in the exemplary embodiment of the computer system (2000).

[0258] The computer system (2000) may include certain human-computer interface input devices. Such human-computer interface input devices may be responsive to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not depicted). Human-computer interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), video (e.g., two-dimensional video, three-dimensional video including stereoscopic video), etc.

[0259] The input human-machine interface device may include one or more of the following (only one of each is shown): keyboard (2001), mouse (2002), touch pad (2003), touch screen (2010), data gloves (not shown), joystick (2005), microphone (2006), scanner (2007), camera (2008).

[0260] The computer system (2000) may also include certain human-computer interface output devices. Such human-computer interface output devices may, for example, stimulate one or more human user senses through tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen (2010), a data glove (not shown), or a joystick (2005), but may also be a tactile feedback device that is not an input device), audio output devices (e.g., speakers (2009), headphones (not shown)), visual output devices (e.g., screens (2010) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which are capable of outputting two-dimensional visual outputs or more than three-dimensional outputs through devices such as stereo image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).

[0261] The computer system (2000) may also include human-accessible storage devices and their associated media: for example, optical media including CD / DVD ROM / RW (2020) with CD / DVD and other media (2021), thumb drives (2022), removable hard drives or solid-state drives (2023), traditional magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security software dogs (not shown), etc.

[0262] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0263] The computer system (2000) may also include an interface (2054) to one or more communication networks (2055). The network may be, for example, a wireless network, a wired network, an optical network. The network may further be a local network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a delay-tolerant network, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial television including CAN Bus, etc. Some networks typically require an external network interface adapter connected to some general data port or peripheral bus (2049) (e.g., a USB port of the computer system (2000)); as described below, other network interfaces are typically integrated into the kernel of the computer system (2000) by connecting to the system bus (e.g., connecting to an Ethernet interface in a PC computer system or connecting to a cellular network interface in a smartphone computer system). The computer system (2000) can use any of these networks to communicate with other entities. Such communications may be one-way receive only (e.g., broadcast television), one-way send only (e.g., a CANbus connected to certain CANbus devices), or bidirectional, for example, connecting to other computer systems using a LAN or WAN digital network. As described above, certain protocols and protocol stacks may be used on each of those networks and network interfaces.

[0264] The above-mentioned human-machine interface device, human-machine accessible storage device and network interface may be attached to the kernel (2040) of the computer system (2000).

[0265] The kernel (2040) may include one or more central processing units (CPUs) (2041), graphics processing units (GPUs) (2042), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (2043), hardware accelerators for certain tasks (2044), graphics adapters (2050), etc. These devices, as well as read-only memory (ROM) (2045), random access memory (2046), internal mass storage (2047) such as internal non-user accessible hard drives, SSDs, etc., may be connected via a system bus (2048). In some computer systems, the system bus (2048) may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the kernel's system bus (2048) or to the kernel's system bus (1848) via a peripheral bus (2049). In one example, a screen (2010) may be connected to a graphics adapter (2050). The architecture of the peripheral bus includes PCI, USB, etc.

[0266] The CPU (2041), GPU (2042), FPGA (2043) and accelerator (2044) can execute certain instructions, which can be combined to form the above-mentioned computer code. The computer code can be stored in ROM (2045) or RAM (2046). Transition data can also be stored in RAM (2046), while permanent data can be stored in, for example, internal mass storage (2047). Fast storage and retrieval of any storage device can be performed by using a high-speed cache, which can be closely associated with the following: one or more CPUs (2041), GPUs (2042), mass storage (2047), ROM (2045), RAM (2046), etc.

[0267] The computer readable medium may have thereon computer code for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of the embodiments of the present application, or the medium and computer code may be of a type known and available to those skilled in the art of computer software.

[0268] As a non-limiting example, a computer system having an architecture (2000), particularly a kernel (2040), can provide functionality due to one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as some non-temporary kernel (2040) memories, such as kernel internal mass storage (2047) or ROM (2045). Software for implementing various embodiments of the present application can be stored in such devices and executed by the kernel (2040). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can enable the kernel (2040), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform specific processes or specific parts of specific processes described herein, including defining data structures (2046) stored in RAM and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality due to hardwiring or otherwise embodied in logic in a circuit (e.g., accelerator (2044)) that may replace software or run in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, a reference to a portion of software may include logic and vice versa. Where appropriate, a reference to a portion of a computer-readable medium may include a circuit (e.g., an integrated circuit (IC)) storing software for execution, a circuit embodying logic for execution, or both. Embodiments of the present application include any suitable combination of hardware and software.

[0269] Although the embodiments of the present application have described some exemplary embodiments, there are changes, substitutions and various alternative equivalents that fall within the scope of the embodiments of the present application. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the embodiments of the present application and therefore fall within the spirit and scope of the embodiments of the present application.

[0270] Appendix A: Abbreviations

[0271] JEM: Joint Exploration Model

[0272] VVC: Next Generation Video Coding

[0273] BMS: Benchmark Set

[0274] MV: Motion Vector

[0275] HEVC: High Efficiency Video Coding

[0276] SEI: Supplemental Enhancement Information

[0277] VUI: Video Availability Information

[0278] GOPs: Group of Pictures

[0279] TUs: Transformation Units

[0280] PUs: Prediction Units

[0281] CTUs: Coding Tree Units

[0282] CTBs: Coding Tree Blocks

[0283] PBs: prediction blocks

[0284] HRD: Hypothesized Reference Decoder

[0285] SNR: Signal to Noise Ratio

[0286] CPUs: Central Processing Units

[0287] GPUs: Graphics Processing Units

[0288] CRT: cathode ray tube

[0289] LCD: Liquid Crystal Display

[0290] OLED: Organic Light Emitting Diode

[0291] CD: compact disc

[0292] DVD: Digital Video Disc

[0293] ROM: Read Only Memory

[0294] RAM: Random Access Memory

[0295] ASIC: Application-Specific Integrated Circuit

[0296] PLD: Programmable Logic Device

[0297] LAN: Local Area Network

[0298] GSM: Global System for Mobile Communications

[0299] LTE: Long Term Evolution

[0300] CANBus: Controller Area Network Bus

[0301] USB: Universal Serial Bus

[0302] PCI: Peripheral Component Interconnect

[0303] FPGA: Field Programmable Gate Array

[0304] SSD: Solid State Drive

[0305] IC: Integrated Circuit

[0306] HDR: High Dynamic Range

[0307] SDR: Standard Dynamic Range

[0308] JVET: Joint Video Exploration Team

[0309] MPM: Most Probable Mode

[0310] WAIP: Wide Angle Intra Prediction

[0311] CU: Coding Unit

[0312] PU: prediction unit

[0313] TU: Transform Unit

[0314] CTU: Coding Tree Unit

[0315] PDPC: Position Dependent Prediction Combination

[0316] ISP: Intra-frame sub-block partitioning

[0317] SPS: Sequence parameter settings

[0318] PPS: Picture Parameter Set

[0319] APS: Adaptive Parameter Set

[0320] VPS: Video Parameter Set

[0321] DPS: Decoding Parameter Set

[0322] ALF: Adaptive Loop Filter

[0323] SAO: Sample Adaptive Offset CC-ALF: Cross-Component Adaptive Loop Filter CDEF: Constrained Directional Enhancement Filter

[0324] CCSO: Cross-component sample offset

[0325] LSO: Local Sampling Offset

[0326] LR: Loop Restoration Filter

[0327] AV1: Alliance for Open Media Video 1

[0328] AV2: Alliance for Open Media Video 2

[0329] MVD: Motion Vector Difference

[0330] CfL: Predicting Chroma from Luma

[0331] SDT: Semi-Decoupled Tree

[0332] SDP: Semi-Decoupled Partitioning

[0333] SST: Semi-Separating Tree

[0334] SB:Super Block

[0335] IBC (or IntraBC): Intra-Block Replication

[0336] CDF: Cumulative density function

[0337] SCC: Screen Content Coding

[0338] GBI: Generalized Bi-Index

[0339] BCW: CU-level weighted bidirectional prediction

[0340] CIIP: Combined Intra-Inter Prediction

[0341] POC: Image sequence number

[0342] RPS: Reference Picture Set

[0343] DPB: Decoded Picture Buffer

[0344] MMVD: Motion Vector Difference Merging Mode

Claims

1. A method for decoding a video block of a video stream, characterized in that: include: receiving the video stream; Determining that the video block is inter-coded based on a prediction block and a motion vector MV, wherein the MV is derived based on a reference motion vector RMV and a motion vector difference MVD of the video block; Determine an RMV index range according to whether the MVD of the video block is encoded with a size-dependent adaptive MVD pixel resolution, and parse the video stream based on the RMV index range to extract the RMV index of the video block, wherein the RMV index of the video block is used to map into a dynamic reference list DRL, and the DRL is used to identify a plurality of ordered candidate RMVs; and extracting the MVD from the video stream; deriving the MV based on the extracted RMV index and the MVD; and The video block is reconstructed based on the MV and the prediction block.

2. The method according to claim 1, characterized in that The RMV indices 1 to N are mapped to a predetermined set of locations in the DRL.

3. The method according to claim 2, characterized in that The RMV indices 1 to N map to first N candidate RMVs among the plurality of ordered candidate RMVs identified by the DRL.

4. The method according to claim 3, characterized in that N is 1 or 2.

5. The method according to claim 3, characterized in that: N is written into the video stream as a syntax element at a sequence level, a frame level, a slice level, a header level or a super block level, and the method further includes extracting N from the video stream.

6. The method according to claim 3, characterized in that N = 1; and The RMV index is not present in the video stream and is derived in response to determining that N=1.

7. The method according to claim 1, characterized in that The method further comprises: In addition to whether the MVD is encoded with a size-dependent adaptive MVD pixel resolution, it also depends on whether the video block is predicted in a single reference mode.

8. The method according to claim 1, characterized in that Extracting the RMV index from the video stream; and The context in which the RMV index is written to the video stream depends on whether the MVD is encoded with a size-dependent adaptive MVD pixel resolution.

9. The method according to claim 8, characterized in that The RMV is written into the video stream using a first context when the MVD is encoded with a size-dependent adaptive MVD pixel resolution, and the RMV is written into the video stream using a second context different from the first context when the MVD is not encoded with a size-dependent adaptive MVD pixel resolution.

10. The method according to claim 1, characterized in that The method further comprises, in response to the video block being encoded with a size-dependent adaptive MVD pixel resolution and when the video block is predicted in a single reference mode, extracting from the video stream an information item indicating whether overlapped block motion compensation (OBMC) or warp motion is employed.

11. The method according to claim 1, characterized in that: The method further includes, in response to the video block being encoded with a size-dependent adaptive MVD pixel resolution, and when the video block is predicted in a single reference mode, a composite inter-intra mode of hybrid inter and intra prediction for the video block is used.

12. The method according to claim 1, characterized in that A context derivation for writing at least one syntax element related to the MVD depends on whether the video block is encoded with a size-dependent adaptive MVD pixel resolution.

13. The method according to claim 12, characterized in that The at least one syntax element associated with the MVD comprises at least one of the following: a first MVD syntax element, for indicating which components of the MVD are non-zero; A second MVD syntax element, used to specify a sign of the MVD; A third MVD syntax element is used to specify a size range of the MVD; a fourth MVD syntax element for specifying an integer size offset within a size range of the MVD; or The fifth MVD syntax element is used to specify the pixel resolution of the MVD.

14. The method according to claim 13, characterized in that When the video block is encoded with a size-dependent adaptive MVD pixel resolution, a first context is derived to decode the at least one syntax element related to the MVD, and when the video block is not encoded with a size-dependent adaptive MVD pixel resolution, a second context different from the first context is derived to decode the at least one syntax element related to the MVD.

15. A method for encoding a video block of a video stream, characterized in that: include: Receiving the video stream to be encoded; Setting the video block to be inter-frame coded based on a prediction block and a motion vector MV, wherein the MV is derived based on a reference motion vector RMV and a motion vector difference MVD of the video block; Determine an RMV index range according to whether the MVD of the video block is encoded with a size-dependent adaptive MVD pixel resolution, and parse the video stream based on the RMV index range to extract the RMV index of the video block, wherein the RMV index of the video block is used to map into a dynamic reference list DRL, and the DRL is used to identify a plurality of ordered candidate RMVs; and extracting the MVD from the video stream; deriving the MV based on the extracted RMV index and the MVD; and An encoding process of the video block is performed based on the MV and the prediction block.

16. A video decoding device, characterized in that: The device comprises a processor and a memory for storing computer instructions, wherein when the processor executes the computer instructions, the processor is configured to cause the video decoding device to perform the decoding method according to any one of claims 1 to 14.

17. A video encoding device, characterized in that: The video encoding device comprises a processor and a memory for storing computer instructions, and when the processor executes the computer instructions, it is configured to cause the video encoding device to perform the encoding method according to claim 15.

18. A method for storing or sending a video bitstream, characterized in that: The video bit stream is decoded based on the decoding method according to any one of claims 1 to 14, or is generated based on the encoding method according to claim 15.

Images