Method and apparatus for video coding

Abstract

To provide a method, an apparatus, and a storage medium for video encoding / decoding with block-level super-resolution coding.SOLUTION: A decoder decodes a video bitstream to obtain a reduced-resolution residual block for a current block, determines a block-level flag that is set to a predefined value indicating that the current block is coded with reduced-resolution coding, generates a reduced-resolution prediction block for the current block by down-sampling a full-resolution reference block for the current block on the basis of the block-level flag, generates a reduced-resolution reconstructed block for the current block on the basis of the reduced-resolution prediction block and the reduced-resolution residual block, and generates a full-resolution reconstructed block for the current block by up-sampling the reduced-resolution reconstructed block.SELECTED DRAWING: Figure 23

Description

[Technical field]

[0001]

[0001] Incorporation by Reference This application claims priority to U.S. Patent Application No. 17 / 488,027, entitled "Method and Apparatus for Video Coding," filed September 28, 2021, which claims priority to U.S. Provisional Application No. 63 / 137,350, entitled "Mixed Resolution Prediction for CU-Based Super-Resolution Coding," filed January 14, 2021. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002]

[0002] Technical Field This disclosure describes embodiments generally relating to video coding. [Background technology]

[0003] The background statement provided in this application is intended to generally present the context of the present disclosure. Work under the names of the current inventors is not admitted, expressly or impliedly, as prior art to the present disclosure to the extent that such work is described in this background section or in a descriptive manner that would not otherwise qualify as prior art at the time of filing.

[0004]

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a sequence of pictures, each having spatial dimensions of, for example, 1920x1080 luminance samples and associated chrominance samples. The sequence of pictures can have a fixed or variable picture rate (informally known as frame rate) of, for example, 60 pictures / second, i.e., 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luminance sample resolution at a frame rate of 60 Hz) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0005]

[0005] One of the goals of video encoding and decoding can be said to be the reduction of redundancy in the input video signal by compression. Compression can help reduce the aforementioned bandwidth or storage space requirements, in some cases by more than two orders of magnitude. Both lossless and non-lossless compression, as well as combinations thereof, can be used. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal. When using non-lossless compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough that the reconstructed signal is useful for the intended application. For video, non-lossless compression is widely used. The amount of distortion that is tolerated depends on the application, for example, a user of a particular consumer streaming application may be able to tolerate higher distortion than a user of a television distribution application. The achievable compression ratio can reflect that a higher tolerable / tolerable distortion may result in a higher compression ratio.

[0006]

[0006] Video encoders and decoders may utilize techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding.

[0007]

[0007] Video codec techniques may include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference picture. In some video codecs, a picture is spatially divided into blocks of samples. If all blocks of samples are coded in intra mode, the picture may be an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, may be used to reset the decoder state and therefore may be used as the first picture in a coded video bitstream and video session or as still images. A transform may be applied to the samples of an intra block, and the transform coefficients may be quantized before entropy coding. Intra prediction may be a technique that minimizes the sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and the smaller the AC coefficients after the transform, the fewer bits are needed for a given quantization step size to represent the block after entropy coding.

[0008]

[0008] Traditional intra-coding, as known for example from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to predict sample values, for example from surrounding sample data and / or metadata obtained during the encoding and / or decoding of spatially adjacent and preceding ones of a data block in decoding order. Such techniques are hereafter referred to as "intra-prediction" techniques. It should be noted that at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from reference pictures.

[0009]

[0009] There may be many different forms of intra prediction. In a given video coding technique, if more than one such technique may be used, the technique used may be coded in an intra prediction mode. In some cases, the mode may have sub-modes and / or parameters that may be coded separately or may be included in a mode codeword. The codeword used for a given mode, sub-mode, and / or parameter combination may affect coding efficiency gains through intra prediction, as may the entropy coding technique used to convert the codeword into a bitstream.

[0010]

[0010] Certain intra prediction modes were introduced in H.264, improved in H.265, and further improved in newer coding techniques such as the Joint Exploration Model (JEM), Generic Video Coding (VVC), and Benchmark Set (BMS). A predictor block can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of the neighboring samples are copied to the predictor block according to a certain direction. The reference to the direction in use may be coded in the bitstream or may be predicted itself.

[0011]

[0011] Referring to FIG. 1A, shown at the bottom right is a subset of 9 predictor directions known from the 33 possible predictor directions of H.265 (corresponding to the 33 angle modes of the 35 intra modes). The points where the arrows meet (101) represent the samples to be predicted. The arrows indicate the directions from which the samples are predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples toward the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples toward the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] 1A, at the top left is shown a square block (104) of 4x4 samples (indicated by a dashed bold line). The square block (104) contains 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample of the block (104) in both the Y and X dimensions. Since the size of the block is 4x4 samples, S44 is at the bottom right. Also shown are reference samples that follow a similar numbering scheme. The reference samples are labeled R and their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, the prediction samples are adjacent to the block being reconstructed; therefore, negative values ​​do not need to be used.

[0013]

[0013] Intra-picture prediction can be handled by copying reference sample values ​​from adjacent samples as necessary depending on the signaled prediction direction. For example, assume that the coded video bitstream contains signaling for this block indicating a prediction direction consistent with the arrow (102), i.e., the sample is predicted from one or more prediction samples toward the top right at an angle of 45 degrees from the horizontal. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. And sample S44 is predicted from reference sample R08.

[0014] In some cases, particularly when the orientation is not evenly divisible by 45 degrees, the values ​​of multiple reference samples can be combined, for example by interpolation, to calculate a reference sample.

[0015]

[0015] The number of possible directions has increased as video coding techniques have developed. In H.264 (2003), nine different directions could be represented. This was increased to 33 in H.265 (2013), and JEM / VVC / BMS at the time of this disclosure can support up to 65 directions. Experiments are performed to identify the most likely directions, and certain techniques are used in entropy coding to represent the more likely directions with a small number of bits, accepting some penalty for the less likely directions. Furthermore, the direction itself can often be predicted from the neighboring directions used in neighboring, already decoded blocks.

[0016] FIG. 1B shows a schematic (105) illustrating 65 intra prediction directions according to JEM, showing an increasingly large number of prediction directions.

[0017]

[0017] The mapping of intra-prediction direction bits in the coded video bitstream to represent directions may vary from one video coding technique to another; for example, it may range from a simple direct mapping of prediction directions to intra-prediction modes to codewords, to complex adaptive schemes involving the most likely modes, and similar techniques. However, in all cases, there may be certain directions that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, in a well-performing video coding technique, less likely directions are represented with more bits than more likely directions.

[0018]

[0018] Motion compensation may be a non-lossless compression technique, in which blocks of sample data from a previously reconstructed picture or part thereof (reference picture) are used to predict a newly reconstructed picture or part of a picture, after being spatially shifted in a direction indicated by a motion vector (hereafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or three dimensions, the third of which points to the reference picture in use (the latter may indirectly be the temporal dimension).

[0019]

[0019] In some video compression techniques, the MV applicable to a particular area of ​​sample data can be predicted from other MVs, e.g., from those associated with other areas of sample data that are spatially adjacent to the area being reconstructed and that precede that MV in decoding order. In this way, the amount of data required to code the MV can be significantly reduced, thereby removing redundancy and increasing compression. For example, when coding an input video signal derived from a camera (known as natural video), MV prediction can work effectively because there is a statistical likelihood that areas larger than the area to which a single MV is applicable move in similar directions and therefore can in some cases be predicted using similar motion vectors derived from MVs of neighboring areas. This results in an MV that, for a given area, is found to be similar or identical to the MV predicted from the surrounding MVs, which, after entropy coding, can be represented with fewer bits than would be used to code the MV directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., a sample stream). In other cases, the MV prediction itself may be non-lossless due to rounding errors, for example when computing a predictor from several surrounding MVs.

[0020]

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms provided by H.265, the one described in this application is a technique hereafter called "spatial merging".

[0021]

[0021] Referring to Fig. 1, a current block (111) contains samples that have been found by the encoder during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of coding its MV directly, the MV can be derived from metadata associated with one or more reference pictures, for example from the most recent reference picture (in decoding order) using MVs associated with any of the five surrounding samples (112 to 116, respectively) denoted A0, A1, and B0, B1, B2. In H.265, MV prediction can use predictors from the same reference picture as the neighboring blocks use. Summary of the Invention

[0022]

[0022] An aspect of the present disclosure provides an apparatus for video encoding / decoding. The apparatus includes a processing circuit that decodes a video bitstream to obtain a reduced-resolution residual block for a current block. The processing circuit determines that a block-level flag is set to a predefined value. The predefined value indicates that the current block is coded with reduced-resolution coding. The processing circuit generates a reduced-resolution prediction block for the current block by downsampling a full-resolution reference block for the current block based on the block-level flag. The processing circuit generates a reduced-resolution reconstructed block for the current block based on the reduced-resolution prediction block and the reduced-resolution residual block. The processing circuit generates a full-resolution reconstructed block for the current block by upsampling the reduced-resolution reconstructed block.

[0023]

[0023] In an embodiment, the processing circuit determines a size of the reduced resolution prediction block based on the size of the full resolution reference block and the down-sampling rate of the current block.

[0024]

[0024] In an embodiment, the processing circuit decodes a block level flag for a current block from the video bitstream, the block level flag indicating that the current block is coded with reduced resolution coding.

[0025] In an embodiment, the processing circuit decodes from the video bitstream filter coefficients or one of indices of said filter coefficients, the filter coefficients being used in upsampling the reduced resolution reconstructed block.

[0026]

[0026] In an embodiment, the processing circuit scales a motion vector of a first neighboring block of the current block based on a scaling factor that is a ratio of down-sampling rates of the current block and the first neighboring block. The processing circuit constructs a first motion vector candidate list for the current block. The first motion vector candidate list includes the scaled motion vector of the first neighboring block.

[0027] In an embodiment, the processing circuit determines the scaled motion vector based on the shift operation depending on whether the scaling factor is a power of 2. N In another example, if the scaling factor is 2 N If, first, the motion vector is rounded up by a rounding factor (e.g., 2 N-1 ), and then the least significant N bits of the horizontal component of the motion vector and the least significant N bits of the vertical component of the motion vector are discarded to obtain a scaled motion vector.

[0028]

[0028] In an embodiment, the processing circuit determines the scaled motion vector based on a look-up table in response to the scaling factor not being a power of two.

[0029]

[0029] In an embodiment, the processing circuit determines a priority of the scaled motion vector in the first motion vector candidate list based on the down-sampling ratios of the current block and the first neighboring block.

[0030]

[0030] In an embodiment, the processing circuitry constructs a second motion vector candidate list for the current block based on one or more second neighboring blocks of the current block, each of the one or more second neighboring blocks having the same down-sampling rate as the current block. The processing circuitry constructs a third motion vector candidate list for the current block based on one or more third neighboring blocks of the current block, each of the one or more third neighboring blocks having a different down-sampling rate than the current block.

[0031] In an embodiment, the processing circuit scans the third motion vector candidate list based on the number of motion vector candidates in the second motion vector candidate list that is less than a specified number.

[0032]

[0032] In an embodiment, the processing circuit determines a fourth motion vector candidate list for the current block by merging the second motion vector candidate list and the third motion vector candidate list in an interleaved manner.

[0033]

[0033] In an embodiment, the processing circuit determines the affine parameters of the current block based on a down-sampling rate of the current block.

[0034]

[0034] An aspect of the present disclosure discloses a method for video encoding / decoding. In the method, a video bitstream is decoded to obtain a reduced-resolution residual block for a current block. A block level flag is determined, the block level flag being set to a predefined value. The predefined value indicates that the current block is coded with reduced-resolution coding. A reduced-resolution prediction block for the current block is generated by down-sampling a full-resolution reference block for the current block based on the block level flag. A reduced-resolution reconstructed block is generated for the current block based on the reduced-resolution prediction block and the reduced-resolution residual block. A full-resolution reconstructed block is generated for the current block by up-sampling the reduced-resolution reconstructed block.

[0035]

[0035] Aspects of the present disclosure also provide a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform any one or combination of methods for video encoding / decoding. [Brief description of the drawings]

[0036] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1A]

[0037] FIG. 1A is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B]

[0038] FIG. 1B is an illustration of an example intra prediction direction. [Figure 1C]

[0039] FIG. 1C is a schematic diagram of a current block and its surrounding spatial merging candidates in one example. [Diagram 2]

[0040] FIG. 2 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment. [Diagram 3]

[0041] FIG. 3 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment. [Figure 4]

[0042] FIG. 4 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment. [Diagram 5]

[0043] FIG. 5 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment. [Figure 6]

[0044] FIG. 6 shows a block diagram of an encoder according to another embodiment. [Figure 7]

[0045] FIG. 7 shows a block diagram of a decoder according to another embodiment. [Figure 8]

[0046] FIG. 8 illustrates an example block partition according to some embodiments of the present disclosure. [Figure 9A]

[0047] FIG. 9A illustrates an example block partitioning using a quad tree plus binary tree (QTBT) and corresponding tree structure according to an embodiment of the present disclosure. [Figure 9B] FIG. 9B illustrates an example block partitioning using a quad tree plus binary tree (QTBT) and corresponding tree structure according to an embodiment of the present disclosure. [Figure 10]

[0048] FIG. 10 illustrates an example nominal angle according to an embodiment of the present disclosure. [Figure 11]

[0049] FIG. 11 illustrates the locations of the top, left, and left bottom samples for one pixel in a current block according to an embodiment of the present disclosure. [Figure 12]

[0050] FIG. 12 illustrates an exemplary recursive filter intra mode according to an embodiment of the present disclosure. [Figure 13]

[0051] FIG. 13 illustrates an exemplary multi-layer reference frame structure according to an embodiment of the present disclosure. [Figure 14]

[0052] FIG. 14 illustrates an exemplary candidate motion vector list construction process according to an embodiment of the present disclosure. [Figure 15]

[0053] FIG. 15 illustrates an exemplary motion field estimation process according to an embodiment of the present disclosure. [Figure 16A]

[0054] 16A and 16B show example overlapped regions (shaded regions) predicted using the above and left neighboring blocks, respectively. [Figure 16B] 16A and 16B show example overlapped regions (shaded regions) predicted using the upper and left neighboring blocks, respectively. [Figure 17]

[0055] Figure 17 shows an exemplary two-step warping process in which a horizontal shear is followed by a vertical shear. [Figure 18]

[0056] Figure 18 shows the overall loop filtering pipeline including frame-level super-resolution in AV1. [Figure 19]

[0057] FIG. 19 illustrates an example implementation using block level flags according to an embodiment of the present disclosure. [Figure 20]

[0058] FIG. 20 illustrates an example spatially neighboring motion vector referencing according to an embodiment of this disclosure. [Figure 21]

[0059] FIG. 21 illustrates an exemplary temporally adjacent motion vector referencing according to an embodiment of this disclosure. [Figure 22]

[0060] FIG. 22 illustrates an example spatially neighboring motion vector referencing for affine motion prediction according to an embodiment of this disclosure. [Diagram 23]

[0061] FIG. 23 illustrates an exemplary flow chart according to an embodiment. [Figure 24]

[0062] FIG. 24 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037]

[0063] I. Video Decoder and Encoder Systems

[0064] FIG. 2 illustrates a simplified block diagram of a communication system (200) according to an embodiment of the present disclosure. The communication system (200) includes a plurality of terminal devices capable of communicating with each other, for example, via a network (250). For example, the communication system (200) includes a first pair of terminal devices (210) and (220) interconnected via the network (250). In the example of FIG. 2, the first pair of terminal devices (210) and (220) perform a unidirectional transmission of data. For example, the terminal device (210) may code video data (e.g., a stream of video pictures captured by the terminal device (210)) for transmission to the other terminal device (220) via the network (250). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (220) can receive the coded video data from the network (250), decode the coded video data to reconstruct the video picture, and display the video picture according to the reconstructed video data. The unidirectional data transmission may be common in media serving applications, etc.

[0038]

[0065] In another example, the communication system (200) includes a second pair of terminal devices (230) and (240) for bidirectional transmission of coded video data, which may occur, for example, during a video conference. With respect to the bidirectional transmission of data, for example, each of the terminal devices (230) and (240) may code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (230) and (240) over the network (250). Each of the terminal devices (230) and (240) may also receive coded video data transmitted by the other of the terminal devices (230) and (240), may decode the coded video data to reconstruct the video pictures, and may display the video pictures on an accessible display device according to the reconstructed video data.

[0039]

[0066] In the example of FIG. 2, terminal devices (210), (220), (230), and (240) are shown as servers, personal computers, and smartphones, although the principles of the present disclosure are not so limited. Embodiments of the present disclosure find application in laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (250) represents any number of networks that carry coded video data between terminal devices (210), (220), (230), and (240), including, for example, wired (wired) and / or wireless communication networks. Communications network (250) may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of the present disclosure, the architecture and topology of network (250) may not be important to the operation of the present disclosure, unless otherwise described below.

[0040]

[0067] Figure 3 illustrates the placement of a video encoder and video decoder in a streaming environment as an example application of the disclosed subject matter, which is equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.), etc.

[0041]

[0068] The streaming system may include a video source (301), e.g., a digital camera, and may include a capture subsystem (313) capable of generating a stream of, e.g., uncompressed video pictures (302). In one example, the stream of video pictures (302) includes samples captured by a digital camera. The stream of video pictures (302), depicted as a thick line to emphasize the larger amount of data when compared to the encoded video data (304) (or coded video bitstream), may be processed by an electronic device (320) that includes a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in more detail below. The encoded video data (304) (or encoded video bitstream (304)), depicted as thin lines to emphasize the smaller amount of data when compared to the stream of video pictures (302), can be stored in the streaming server (305) for future use. One or more streaming client subsystems, such as the client subsystems (306) and (308) of FIG. 3, can access the streaming server (305) to retrieve copies (307) and (309) of the encoded video data (304). The client subsystem (306) can include a video decoder (310), for example within the electronic device (330). The video decoder (310) decodes the incoming copy of the encoded video data (307) and generates an output stream of video pictures (311) that can be rendered on a display (312) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (304), (307), and (309) (e.g., a video bitstream) may be encoded according to a particular video coding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0042]

[0069] It should be noted that the electronic devices (320) and (330) can include other components (not shown). For example, the electronic device (320) can include a video decoder (not shown) and the electronic device (330) can include a video encoder (not shown).

[0043]

[0070] 4 shows a block diagram of a video decoder (410) according to an embodiment of the present disclosure. The video decoder (410) may be included in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., receiving circuitry). The video decoder (410) may be used in place of the video decoder (310) in the example of FIG. 3.

[0044]

[0071] The receiver (431) may receive one or more coded video sequences to be decoded by the video decoder (410); in the same or another embodiment, it may receive one coded video sequence at a time, where the decoding of each coded video sequence is independent of the other coded video sequences. The coded video sequences may be received from a channel (401), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (431) may receive the coded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be transferred using respective entities (not shown). The receiver (431) may separate the coded video sequences from the other data. To address network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420), hereafter referred to as the "parser (420)". In certain applications, the buffer memory (415) is part of the video decoder (410). In other cases, it may be outside the video decoder (410) (not shown). In yet another example, there may be a buffer memory (not shown) outside the video decoder (410), for example to deal with network jitter, and even another buffer memory (415) inside the video decoder (410), for example to handle playback timing. If the receiver (431) is receiving data from a store-and-forward device with sufficient bandwidth and controllability, or from a synchronous network, the buffer memory (415) may not be needed or may be small.For use in a best effort packet network such as the Internet, a buffer memory (415) may be required, which may be relatively large and may advantageously be adaptively sized and may be implemented at least in part in an operating system or similar element (not shown) outside the video decoder (410).

[0045]

[0072] The video decoder (410) may include a parser (420) to reconstruct symbols (421) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (410) and potentially information for controlling a rendering device such as a rendering device (412) (e.g., a display screen) that is not an integral part of the electronic device (430) but may be coupled to the electronic device (430) as shown in FIG. 4. The control information for the rendering device may be in the form of supplemental enhancement information (SEI message) or video usability information (VUI) parameter set fragments (not shown). The parser (420) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context effects, etc. The parser (420) can extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on the at least one parameter corresponding to the group. The subgroups can include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (420) can also extract from the coded video sequence information such as transform coefficients, quantization parameter values, motion vectors, etc.

[0046]

[0073] The parser (420) is capable of performing an entropy decoding / parsing process on the video sequence received from the buffer memory (415) to generate symbols (421).

[0047]

[0074] The reconstruction of the symbols (421) may include different units depending on the type of coded video picture or part thereof (e.g.: inter and intra pictures, inter and intra blocks) and other factors. Which units are included and how can be controlled by subgroup control information parsed by the parser (420) from the coded video sequence. The flow of such subgroup control information between the parser (420) and subsequent units is not depicted for clarity.

[0048]

[0075] Beyond the functional blocks already mentioned, the video decoder (410) may be conceptually subdivided into a number of functional units as described below. In a practical implementation operating within commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate.

[0049]

[0076] The first unit is a scalar / inverse transform unit (451), which receives the quantized transform coefficients as well as control information (including the transform to use, block size, quantization factor, quantization scaling matrix, etc.) from the parser (420) as symbols (421). The scalar / inverse transform unit (451) can output blocks containing sample values ​​that can be input to an aggregator (455).

[0050]

[0077] In some cases, the output samples of the scalar / inverse transform (451) may relate to intra-coded blocks: i.e., blocks that do not use prediction information from a previously reconstructed picture, but are able to use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by an intra picture prediction unit (452). In some cases, the intra picture prediction unit (452) generates blocks of the same size and shape of the block being reconstructed using already reconstructed surrounding information retrieved from a current picture buffer (458). The current picture buffer (458) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (455) adds, possibly on a sample-by-sample basis, the prediction information generated by the intra prediction unit (452) to the output sample information as provided by the scalar / inverse transform unit (451).

[0051]

[0078] In other cases, the output samples of the scalar / inverse transform unit (451) may relate to a block that may be inter-coded and motion compensated. In such a case, the motion compensated prediction unit (453) may access the reference picture memory (457) to retrieve samples used for prediction. After motion compensating the retrieved samples according to the symbols (421) related to the block, these samples are added by the aggregator (455) to the output of the scalar / inverse transform unit (451) (in this case called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (457) from which the motion compensated prediction unit (453) retrieves the prediction samples may be controlled by the motion vectors available to the motion compensated prediction unit (453), for example in the form of symbols (421), which may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​taken from a reference picture memory (457), motion vector prediction mechanisms, etc., where sub-sample accurate motion vectors are used.

[0052]

[0079] The output samples of the aggregator (455) may be subjected to various loop filtering techniques in a loop filter unit (456). Video compression techniques are controlled by parameters contained in the coded video sequence (also called coded video bitstream) and made available to the loop filter unit (456) as symbols (421) from the parser (420), but may include in-loop filter techniques that may be responsive to meta-information obtained during decoding of previous parts of the coded picture or coded video sequence (in decoding order) as well as to previously reconstructed loop filtered sample values.

[0080] The output of the loop filter unit (456) may be a sample stream that can be output to a rendering device (412) or stored in a reference picture memory (457) for use in future inter-picture prediction.

[0053]

[0081] Once a given coded picture has been fully reconstructed, it can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (420)), the current picture buffer (458) can become part of the reference picture memory (457) and a new current picture buffer can be reallocated before starting the reconstruction of a future coded picture.

[0082] The video decoder (410) may perform decoding operations according to a given video compression technique in a standard such as ITU-T Rec. H.265. A coded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile as documented in the video compression technique or standard. Specifically, a profile may select a particular tool from all tools available in the video compression technique or standard as the only tool that may be used under that profile. Also required for compliance is that the complexity of the coded video sequence falls within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited by a hypothetical reference decoder (HRD) specification and metadata for HRD buffer management that may be signaled in the coded video sequence.

[0054]

[0083] In an embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (410) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0055]

[0084] 5 shows a block diagram of a video encoder (503) according to an embodiment of the present disclosure. The video encoder (503) is included in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmission circuit). The video encoder (503) can be used in place of the video encoder (303) in the example of FIG. 3.

[0056]

[0085] The video encoder (503) may receive video samples from a video source (501) (which in the example of FIG. 5 is not part of the electronic device (520)) capable of capturing video images to be coded by the video encoder (503). In another example, the video source (501) is part of the electronic device (520).

[0057]

[0086] The video source (501) may provide a source video sequence to be coded by the video encoder (503) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (501) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (501) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures that convey motion when viewed in sequence. The picture itself may be organized as a spatial array of pixels, each of which may contain one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art will readily appreciate the relationship between pixels and samples. The following discussion focuses on examples.

[0058]

[0087] According to an embodiment, the video encoder (503) is capable of coding and compressing pictures of a source video sequence into a coded video sequence (543) in real-time or under any other time constraint required by an application. Enforcing an appropriate coding rate is one function of the controller (550). In some embodiments, the controller (550) controls and is operatively coupled to other functional units as described below, the couplings of which are not depicted for clarity. Parameters set by the controller (550) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (550) may be configured with other suitable functions associated with the video encoder (503) optimized for a particular system design.

[0059]

[0088] In some embodiments, the video encoder (503) is configured to operate in a coding loop. As a simplified explanation, in one example, the coding loop can include a source coder (530) (e.g., responsible for generating symbols such as a symbol stream based on an input picture to be coded and reference pictures) and a (local) decoder (533) embedded in the video encoder (503). The decoder (533) reconstructs the symbols to generate sample data in a similar manner to that generated by the (remote) decoder (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques contemplated by the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (534). Since the decoding of the symbol stream produces bit-exact results independent of the location of the decoder (local or remote), the contents in the reference picture memory (534) are also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder would "see" if it used prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift if synchronicity cannot be maintained, e.g. due to channel errors) is used in several related techniques as well.

[0060]

[0089] The operation of the "local" decoder (533) may be the same as that of a "remote" decoder, such as the video decoder (410) already described in detail above in connection with Figure 4. However, with brief reference to Figure 4 as well, since symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (545) and parser (420) may be lossless, the entropy decoding portion of the video decoder (410), including the buffer memory (415) and the parser (420), may not be fully implemented in the local decoder (533).

[0061]

[0090] An observation that can be made at this point is that any decoder technique other than analysis / entropy decoding that is present in the decoder must necessarily be present in the corresponding encoder, in substantially the same functional form. For this reason, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder techniques can be omitted, since they are the inverse of the decoder techniques that have been described generically. Only in certain areas is a more detailed description required, which is given below.

[0062]

[0091] During operation, the source coder (530), in some examples, may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (532) codes differences between pixel blocks of the input picture and pixel blocks of the reference pictures that may be selected as predictive references for the input picture.

[0063]

[0092] The local video decoder (533) can decode the coded video data of the pictures that can be designated as reference pictures based on the symbols generated by the source coder (530). The operation of the coding engine (532) can advantageously be a non-lossless process. If the coded video data can be decoded by a video decoder (not shown in FIG. 5), the reconstructed video sequence can be a replica of the source video sequence, typically with some errors. The local video decoder (533) can repeat the decoding process that can be performed by the video decoder on the reference pictures, causing the reconstructed reference pictures to be stored in the reference picture cache (534). In this way, the video encoder (503) can locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures obtained by the far-end video decoder (assuming there are no transmission errors).

[0064]

[0093] The predictor (535) may perform a prediction search for the coding engine (532). That is, for a new picture to be coded, the predictor (535) may search the reference picture memory (534) for sample data (as candidate reference pixel blocks) or predefined metadata (reference picture motion vectors, block shapes, etc.), which may serve as suitable prediction references for the new picture. The predictor (535) may operate on a sample block-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (535), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (534).

[0065]

[0094] The controller (550) may manage the coding operations of the source coder (530), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0066]

[0095] All outputs of the aforementioned functional units can be subjected to entropy coding in an entropy coder (545), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0067]

[0096] The transmitter (540) can buffer the coded video sequence as produced by the entropy coder (545) and prepare it for transmission over a communication channel (560), which may be a hardware / software link to a storage device that stores the coded video data. The transmitter (540) can merge the coded video data from the video coder (503) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0068]

[0097] The controller (550) can manage the operation of the video encoder (503). During coding, the controller (550) can assign a particular coded picture type to each of the coded pictures, which can affect the coding technique that can be applied to each picture. For example, a picture may be frequently assigned as one of the following picture types:

[0069]

[0098] An intra picture (I-picture) is one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures, as well as their respective uses and characteristics.

[0070]

[0099] A predictive picture (P-picture) is one that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0071]

[0100] Bidirectionally predicted pictures (B-pictures) are those that can be coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a block.

[0072]

[0101] A source picture is usually spatially subdivided into several sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples) and can be coded block by block. Blocks can be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective picture. For example, blocks of I pictures can be non-predictively coded or they can be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P pictures can be predictively coded with spatial or temporal prediction with reference to one previously coded reference picture. Blocks of B pictures can be predictively coded with spatial or temporal prediction with reference to one or two previously coded reference pictures.

[0073]

[0102] The video encoder (503) may perform coding operations according to a given video coding technique or standard, such as ITU-T Rec. H.265, in which the video encoder (503) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. The coded video data may thus conform to a syntax specified by the video coding technique or standard being used.

[0074]

[0103] In an embodiment, the transmitter (540) may transmit additional data along with the encoded video. The source coder (530) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data (redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.).

[0075]

[0104] A video can be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra prediction) exploits spatial correlation in a given picture, and inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture under encoding / decoding, called the current picture, is partitioned into blocks. If a block in the current picture is similar to a reference block in a reference picture that was previously coded in the video and is still buffered, the block in the current picture can be coded by a vector called MV. MV refers to a reference block in the reference picture and can have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0076]

[0105] In some embodiments, bi-prediction techniques can be used for inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both preceding in decoding order (but potentially past and future, respectively, in display order) a current picture in a video. A block in the current picture can be coded with a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted by a combination of the first and second reference blocks.

[0077]

[0106] Furthermore, to improve coding efficiency, it is possible to use merge mode techniques for inter-picture prediction.

[0078]

[0107] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be partitioned into one CU of 64×64 pixels, four CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more Prediction Units (PUs) depending on temporal and / or spatial predictability. In general, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luma values) for pixels, such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0079]

[0108] 6 shows a diagram of a video encoder (603) according to another embodiment of the present disclosure. The video encoder (603) is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video picture in a sequence of video pictures and to encode the processed block into a coded picture that is part of a coded video sequence. In one example, the video encoder (603) is used in place of the video encoder (303) of the example of FIG. 3.

[0080]

[0109] In an HEVC example, the video encoder (603) receives a matrix of sample values ​​for a processing block, such as a predicted block of 8x8 samples. The video encoder (603) determines whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode, e.g., using rate-distortion optimization. If the processing block is to be coded in intra mode, the video encoder (603) may use intra prediction techniques to code the processing block into a coded picture; if the processing block is to be coded in inter mode or bi-predictive mode, the video encoder (603) may use inter prediction techniques or bi-predictive techniques, respectively, to code the processing block into a coded picture. In certain video coding techniques, the merge mode may be an inter prediction picture sub-mode, in which case the motion vector is derived from one or more motion vector predictors without benefit of a coded motion vector component outside the predictor. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (603) includes other components, such as a mode decision module (not shown) to determine the mode of the processing block.

[0081]

[0110] In the example of FIG. 6, the video encoder (603) includes an inter-encoder (630), an intra-encoder (622), a residual calculator (623), a switch (626), a residual encoder (624), a generalized controller (621), and an entropy encoder (625) coupled together as shown in FIG. 6.

[0082]

[0111] The inter encoder (630) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks of a previous picture and blocks in a subsequent picture), generate inter prediction information (e.g., a description of redundant information due to the encoding technique, motion vectors, merge mode information), and calculate an inter prediction result (e.g., a prediction block) based on the inter prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that has been decoded based on the encoded video information.

[0083]

[0112] The intra encoder (622) is configured to receive samples of a current block (e.g., a processing block), possibly compare the block to previously coded blocks in the same picture, generate transformed and quantized coefficients, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). In one example, the intra encoder (622) also calculates an intra prediction result (e.g., a prediction block) based on the intra prediction information and reference blocks in the same picture.

[0084]

[0113] The general controller (621) is configured to determine general control data and control other components of the video encoder (603) based on the general control data. In one example, the general controller (621) determines the mode of the block and provides a control signal to the switch (626) based on the mode. For example, if the mode is intra mode, the general controller (621) controls the switch (626) to select intra mode results for use by the residual calculator (623) and controls the entropy encoder (625) to select intra prediction information and include the intra prediction information in the bitstream; if the mode is inter mode, the general controller (621) controls the switch (626) to select inter prediction results for use by the residual calculator (623) and controls the entropy encoder (625) to select inter prediction information and include the inter prediction information in the bitstream.

[0085]

[0114] The residual calculator (623) is configured to calculate a difference (residual data) between the received block and a prediction result selected from the intra-encoder (622) or the inter-encoder (630). The residual encoder (624) is configured to operate on the residual data to encode the residual data and generate transform coefficients. In one example, the residual encoder (624) is configured to transform the residual data from a spatial domain to a frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intra-encoder (622) and the inter-encoder (630) as appropriate. For example, the inter encoder (630) may generate decoded blocks based on the decoded residual data and the inter prediction information, and the intra encoder (622) may generate decoded blocks based on the decoded residual data and the intra prediction information. The decoded blocks are suitably processed to generate decoded pictures, which may be buffered in a memory circuit (not shown) and, in some examples, used as reference pictures.

[0086]

[0115] The entropy encoder (625) is configured to format a bitstream to include the encoded blocks. The entropy encoder (625) is configured to include various information in accordance with an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (625) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. It should be noted that when coding blocks in a merged sub-mode of either the inter mode or the bi-prediction mode in accordance with the disclosed subject matter, the residual information is not present.

[0087]

[0116] 7 shows a diagram of a video decoder (710) according to another embodiment of the present disclosure. The video decoder (710) is configured to receive coded pictures that are part of a coded video sequence and to decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (710) is used in place of the video decoder (310) in the example of FIG. 3.

[0088]

[0117] In the example of FIG. 7, the video decoder (710) includes an entropy decoder (771), an inter decoder (780), a residual decoder (773), a reconstruction module (774), and an intra decoder (772), coupled together as shown in FIG. 7.

[0089]

[0118] The entropy decoder (771) can be configured to reconstruct from the coded picture certain symbols that represent syntax elements that make up the coded picture. Such symbols can include, for example, prediction information (e.g., intra- or inter-prediction information), which can identify the mode in which the block is coded (e.g., intra- or inter-prediction mode, merged submode, or the latter two in another submode), certain samples or metadata used for prediction by the intra-decoder (772) or inter-decoder (780), respectively, residual information (e.g., in the form of quantized transform coefficients), etc. In one example, if the prediction mode is an inter- or bi-prediction mode, the inter-prediction information is provided to the inter-decoder (780); if the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (772). The residual information can be inverse quantized and provided to the residual decoder (773).

[0090]

[0119] The inter decoder (780) is configured to receive the inter prediction information and to generate inter prediction results based on the inter prediction information.

[0091]

[0120] The intra decoder (772) is configured to receive intra prediction information and to generate a prediction result based on the intra prediction information.

[0092]

[0121] The residual decoder (773) is configured to perform inverse quantization to extract unquantized transform coefficients and to process the unquantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require certain control information (including a quantization parameter (QP)), which may be provided by the entropy decoder (771) (this may only be a small amount of control information, so a data path is not depicted).

[0093]

[0122] The reconstruction module (774) is configured to combine, in the spatial domain, the residual as output by the residual decoder (773) and the prediction result (possibly output by an inter- or intra-prediction module) to form a reconstructed block, the reconstructed block being part of a reconstructed picture, which may be part of a reconstructed video. It should be noted that other suitable processes, such as deblocking processes, may be performed to improve visual quality.

[0123] It should be noted that the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using any suitable technology. In an embodiment, the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using one or more processors executing software instructions.

[0094]

[0124] II. Block Partition

[0125] FIG. 8 illustrates an example block partition according to some embodiments of the present disclosure.

[0095]

[0126] In some related cases, such as VP9 proposed by AOMedia (Alliance for Open Media), a four-way partition tree can be used, starting from the 64x64 level down to the 4x4 level, with some additional restrictions for blocks smaller than 8x8, as shown in Figure 8. A partition designated by R can be called a recursive partition, i.e. the same partition tree is repeated at lower scales until the lowest 4x4 level is reached.

[0096]

[0127] In some related cases, such as AV1 proposed by AOMedia and based on VP9, ​​the partition tree can be expanded to a 10-way structure as shown in Figure 8, and the maximum coding block size (called superblock in VP9 / AV1 terminology) is increased to start at 128x128. Note that 4:1 / 1:4 rectangular partitions are included in AV1 but not in VP9. Rectangular partitions cannot be further subdivided. Furthermore, AV1 can support more flexibility in the use of partitions below the 8x8 level, since in some cases it is possible to perform inter prediction for 2x2 chroma blocks.

[0097]

[0128] In some related cases such as HEVC, the CTUs can be partitioned into CUs by using a quadtree structure, denoted as a coding tree, to accommodate various local features. The decision of whether to code a picture area using inter-picture (temporal) or intra-picture (spatial) prediction can be made at the CU level. Each CU can be further partitioned into one, two, or four PUs depending on the PU partition type. Within one PU, the same prediction process can be applied, and related information can be transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partition type, the CUs can be partitioned into TUs according to another quadtree structure, such as a coding tree for CUs. One of the important features of the HEVC structure is that it has multiple partition concepts, including CUs, PUs, and TUs. In HEVC, a CU or TU can only be square in shape, while a PU can be square or rectangular in shape in case of an inter-predicted block. In HEVC, a coding block may be further divided into four square sub-blocks, and a transform process may be performed on each sub-block, or TU. Each TU may be further divided recursively (e.g., using quadtree partitioning) into smaller TUs. The quadtree partitioning may be called a residual quadtree (RQT).

[0098]

[0129] At picture boundaries, HEVC does an implicit quadtree split, so that a block can continue to undergo quadtree splitting until its size fits into a picture boundary.

[0099]

[0130] In some relevant cases, such as VVC, a quad-tree plus binary tree (QTBT) partitioning structure can be applied, which eliminates the notion of multiple partition types (i.e., it eliminates the distinction between the concepts of CU, PU, ​​and TU) and supports a lot of flexibility for CU partition shapes.

[0100]

[0131] 9A and 9B show an example block partitioning according to an embodiment of the present disclosure using QTBT and corresponding tree structure. Solid lines indicate QT partitioning and dotted lines indicate BT partitioning. At each partition (i.e., non-leaf) node of BT, one flag is signaled to indicate which partition type (i.e., horizontal or vertical) is used. In FIG. 9B, 0 indicates horizontal partitioning and 1 indicates vertical partitioning. In the case of QT partitioning, there is no need to specify the partition type because QT partitioning always partitions a block both horizontally and vertically, generating four sub-blocks of equal size.

[0101]

[0132] In the QTBT structure, a CU can have either a square or rectangular shape. As shown in Figures 9A and 9B, a CTU is first partitioned by the QT structure. The QT leaf node can be further partitioned by the BT structure. There are two partition types in the BT partition: symmetric horizontal partition and symmetric vertical partition. The BT leaf node is a CU, and the segmentation into two CUs is used for prediction and transform processing without any additional partitioning. Therefore, a CU, a PU, and a TU can have the same block size in the QTBT structure.

[0102]

[0133] A CU may contain CBs of different color components as in JEM. For example, in case of P and B slices with 4:2:0 chroma format, one CU may contain one luma CB and two chroma CBs. In other examples, a CU may contain a single component CB, for example, in case of an I slice, one CU may contain only one luma CB or only two chroma CBs.

[0103]

[0134] For the QTBT partitioning scheme the following parameters are defined: CTU size (QT root node size, e.g. as in HEVC), MinQTSize (minimum allowed QT leaf node size), MaxBTSize (maximum allowed BT root node size), MaxBTDepth (maximum allowed BT depth), MinBTSize (minimum allowed BT leaf node size).

[0104]

[0135] In one example of a QTBT partitioning structure, the CTU size is set as 128x128 luma samples, with two corresponding 64x64 blocks of chroma samples, MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize is set to 4x4 (for both width and height), and MaxBTDepth is set to 4. QT partitioning is first applied to the CTU to generate QT leaf nodes. QT leaf nodes can have sizes from 16x16 (i.e. MinQTSize) to 128x128 (i.e. CTU size). If the leaf QT node is 128x128, it will not be further split by BT since its size exceeds MaxBTSize (i.e. 64x64). Otherwise, the leaf QT node can be further split by the BT tree. Thus, the QT leaf node is also the root node of the BT and has a BT depth of 0. If the BT depth reaches MaxBTDepth (i.e. 4), no further splits are considered. If a BT node has a width equal to MinBTSize (i.e. 4), no further horizontal splits are considered. Similarly, if a BT node has a height equal to MinBTSize, no further vertical splits are considered. BT leaf nodes are further processed by the prediction and transform process without any additional partitions. For example, the maximum CTU size is 256x256 luma samples as in JEM.

[0105]

[0136] III. Predictions in AV1

[0137] In some related cases, such as VP9, ​​eight directional modes are supported, corresponding to angles from 45 degrees to 207 degrees. In some related cases, such as AV1, the directional intra modes are extended to a finer granularity set of angles to exploit the spatial redundancy of more diverse directional textures. The original eight angles are slightly modified and referred to as nominal angles, and these eight angles are named V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED.

[0106]

[0138] FIG. 10 shows an example nominal angle according to an embodiment of the present disclosure. Each nominal angle can be associated with seven finer angles, so that in some related cases such as AV1, there may be a total of 56 directional angles. The prediction angle can be expressed by the nominal intra angle plus the angle delta. The angle delta is equal to the coefficient multiplied by a step size of 3 degrees. The coefficient can be in the range of -3 to 3. In AV1, eight nominal modes are first signaled along with five non-angular smooth modes. Then, if the current mode is an angular mode, an index is further signaled to indicate the angle delta relative to the corresponding nominal angle. To implement the directional prediction modes in AV1 in a general way, all 56 directional intra prediction angles in AV1 can be realized using a unified directional predictor, which projects each pixel to a reference sub-pixel position and interpolates the reference sub-pixel by a two-tap bilinear filter.

[0107]

[0139] In some related cases such as AV1, there are five omnidirectional smooth intra prediction modes: DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H. For DC prediction, the average of the top-left neighboring samples is used as the predictor of the block to be predicted. For PAETH prediction, the top, left, and top-left reference samples are first taken, and then the closest value to (top + left - top-left) is set as the predictor of the pixel to be predicted.

[0108]

[0140] 11 shows the locations of the top, left, and top-left samples for one pixel in the current block according to an embodiment of the present disclosure. For SMOOTH, SMOOTH_V, and SMOOTH_H modes, the block is predicted using quadratic interpolation in the vertical or horizontal direction, or an average in both directions.

[0109]

[0141] In some relevant cases, such as AV1, intra-prediction modes based on recursive filtering can be used.

[0110]

[0142] FIG. 12 illustrates an exemplary recursive filter intra mode according to an embodiment of the present disclosure.

[0111]

[0143] To capture the decaying spatial correlation with the reference at the edge, FILTER INTRA mode is designed for luma blocks. In AV1, five filter intra modes are defined, and each mode is represented by a set of eight 7-tap filters that reflect the correlation between pixels in a 4x2 patch and its seven neighbors. For example, the weighting coefficients of the 7-tap filters are position-dependent. As shown in Figure 12, an 8x8 block is divided into eight 4x2 patches, which are denoted by B0, B1, B2, B3, B4, B5, B6, and B7. For each patch, the seven neighbors, denoted by R0-R7, are used to predict the pixels in the respective patch. For patch B0, all the neighbors have already been reconstructed. However, for other patches, not all the neighbors have been reconstructed, and the predicted values ​​of the previous neighbors are used as the reference values. For example, not all the neighbors of patch B7 have been reconstructed, so the predicted samples of the neighbors of patch B7 (i.e., B5 and B6) are used instead.

[0112]

[0144] In some related cases, such as VP9, ​​three reference frames can be used for inter prediction, including LAST (nearest past), GOLDEN (far past), and ALTREF (temporally filtered future) frames.

[0113]

[0145] In some related cases, such as AV1, extended reference frames can be used. For example, in addition to the three reference frames used in VP9, ​​AV1 can use four more types of reference frames. The four more types of reference frames include LAST2, LAST3, BWDREF, and ALTREF2 frames. LAST2 and LAST3 frames are two near past frames, and BWDREF and ALTREF2 frames are two future frames. Furthermore, BWDREF frames are look-ahead frames coded without temporal filtering, and are more useful as backward references at relatively short distances. ALTREF2 frames are intermediate filtered future reference frames between GOLDEN and ALTREF frames.

[0114]

[0146] FIG. 13 illustrates an exemplary multi-layer reference frame structure according to an embodiment of the present disclosure. In FIG. 13, an adaptive number of frames share the same GOLDEN and ALTREF frames. The BWDREF frames are look-ahead frames that are directly coded without applying temporal filtering, and are therefore more applicable as backward references at relatively short distances. The ALTREF2 frames serve as intermediate filtered future references between the GOLDEN and ALTREF frames. All new references can be selected by a single prediction mode, or can be combined into pairs to form a composite mode. AV1 provides a rich set of reference frame pairs, resulting in both bidirectional and unidirectional composite prediction, and thus enabling a variety of videos with dynamic temporal correlation properties to be coded in a more adaptive and optimal manner.

[0115]

[0147] FIG. 14 illustrates an exemplary candidate motion vector list construction process according to an embodiment of the present disclosure. Spatial and temporal reference motion vectors can be classified into two categories based on where they appear (closest spatial neighbors, and others). In some relevant cases, motion vectors from immediately above, to the left, and to the right of the current block can have a higher correlation with the current block than the other blocks, and thus are considered to have a higher priority. Within each category, motion vectors are ranked in the spatial and temporal search range in descending order of their occurrence counts. Motion vector candidates with higher occurrence counts can be considered to be "popular" in the local region, i.e., have a higher prior probability. The two categories are concatenated to form a ranked list.

[0116]

[0148] FIG. 15 illustrates an exemplary motion field estimation process according to an embodiment of the present disclosure.

[0117]

[0149] In some related cases, such as AV1, dynamic spatial and temporal motion vector references can be used. For example, a motion vector reference selection scheme can be incorporated to efficiently code motion vectors. In the motion vector reference selection scheme, the spatial neighborhood can be wider than that used in VP9. Furthermore, a motion field estimation process can be used to find temporal motion vector reference candidates. The motion field estimation process can operate in three stages: motion vector buffering, motion trajectory creation, and motion vector projection. First, for each coded frame, the reference frame index and associated motion vector of the respective coded frame can be stored. The stored information can be referenced by the next coding frame to generate the motion field of the next coding frame. The motion field estimation can be performed, for example, by translating a block in one reference frame Ref2 to another reference frame Ref0 in FIG. 15. ref2 A motion trajectory aimed at (e.g., MV Ref2 ) can be examined. The motion field estimation process then searches through the collocated 128x128 area with 8x8 block resolution for all motion trajectories passing through each 64x64 processing unit. Then, at the coding block level, once the reference frame is determined, the motion field estimation process can be performed by linearly projecting the passing motion trajectory into the desired reference frame, e.g., MV in Fig. 15. ref2 To MV0 or MV1, motion vector candidates can be derived.

[0118]

[0150] Once all the candidate motion vectors have been found, they can be sorted, merged and ranked up to four final candidates. The index of the reference motion vector selected from the list can then be signaled and, optionally, the motion vector difference can be coded.

[0119]

[0151] In some related cases, such as AV1, to reduce prediction error around block boundaries by combining predictions obtained from neighboring motion vectors, the block-based prediction can be combined with secondary predictors from the top and left edges by applying 1-D filters in the vertical and horizontal directions, respectively. This method can be referred to as overlapped block motion compensation (OBMC).

[0120]

[0152] 16A and 16B show an example overlapped region (shaded region) predicted using the top neighboring block (2) and the left neighboring block (4), respectively. The shaded region of the prediction block (0) can be predicted by recursively generating blended prediction samples with a 1-D raised cosine filter.

[0121]

[0153] In some related cases such as AV1, two affine prediction models can be used, called global warped motion compensation and a local warped motion compensation. The former signals a frame-level affine model between a frame and its reference, while the latter handles varying local motion implicitly with minimal overhead. Local motion parameters can be derived at the block level by using 2D motion vectors from causal neighborhoods. This affine model is realized by successive horizontal and vertical shearing operations based on 8-tap interpolation filters at 1 / 64 pixel accuracy.

[0122]

[0154] Figure 17 shows an exemplary two-stage warping process in which a horizontal shear is followed by a vertical shear. In Figure 17, an affine model is realized with local warping motion compensation that first performs a horizontal shear and then a vertical shear.

[0123]

[0155] IV. Frame-based super-resolution in AV1

[0156] Figure 18 shows the overall loop filtering pipeline including frame-level super-resolution in AV1. At the encoder side, the source frame can first be downscaled in a non-normative way and encoded at a lower resolution. At the decoder side, a deblocking filter and a constrained directional enhancement filter (CDEF) can be applied to remove coding artifacts while preserving edges at a lower resolution. A linear upsampling filter can then be applied only along the horizontal direction to obtain a full-resolution reconstruction. Optionally, a loop restoration filter can be applied afterwards at full resolution to recover high-frequency details lost during downsampling and quantization.

[0124]

[0157] In some related cases, such as AV1, super-resolution is a special mode signaled at the frame level. Each coded frame can use the horizontal-only super-resolution mode at any resolution within the ratio constraints. Whether to apply linear upsampling after decoding and the scaling ratio to be used can be signaled. The upsampling ratio can have 9 possible values, given as d / 8 for d=8, 9,... and 16. The corresponding downsampling ratio before encoding can be 8 / d.

[0125]

[0158] Given an output frame dimension WxH and an upsampling ratio d, both the encoder and decoder are able to calculate the low-resolution coded frame dimension as wxH, where the reduced width is w=(8W+d / 2) / d. The linear upscaling process takes a reduced-resolution frame wxH and outputs a frame of dimensions WxH as specified in the frame header. The canonical horizontal linear upscaler in AV1 uses a 1 / 16-phase linear 8-tap filter for the interpolation of each row.

[0126]

[0159] V. CU-based super-resolution coding

[0160] In some related cases, such as AV1, super-resolution is performed at the frame level, i.e., it is applied to all areas of the picture with a constant scaling ratio. However, the signal statistics in different areas in a picture can be very different. Therefore, applying down-sampling and / or up-sampling to all areas may not necessarily be a good rate-distortion trade-off.

[0127]

[0161] Adaptively applying downsampling and / or upsampling to picture regions can be performed using pre-processing and / or post-processing approaches, such as using masks and / or segment information to select picture regions for downsampling and / or upsampling. However, this process cannot guarantee that the rate-distortion performance improvement will be greater than other coding methods that do not use super-resolution.

[0128]

[0162] The present disclosure includes a method for block-level super-resolution coding.

[0129]

[0163] According to aspects of the present disclosure, mixed-resolution prediction can be used to emulate frame-level super-resolution adapted for use at the block level.

[0130]

[0164] In one embodiment, a block level flag may be used to specify whether reduced resolution coding is used for a coding block. For example, if the block level flag is set to a first predefined value (e.g., 1), reduced resolution coding is enabled for that block. Prediction sample generation, such as the prediction procedure in AV1 as described above, may be performed by using reference samples or pictures in reduced resolution to generate a low resolution prediction block for the coding block. The reduced resolution prediction block may then be used to generate a reduced resolution reconstructed block for the coding block. Finally, the reduced resolution reconstructed block may be upsampled to a full resolution reconstructed block for the coding block.

[0131]

[0165] In one embodiment, if the block level flag is set to a second predefined value (e.g., 0), prediction sample generation is performed at full resolution (or original resolution), followed by full resolution reconstruction.

[0132]

[0166] FIG. 19 illustrates an example implementation of using a block level flag according to an embodiment of the present disclosure. When the block level flag is on, e.g., the block level flag is equal to a first predefined value, a source block (1901) at the encoder side can be downsampled by a downsampler module (1920) to generate a downsampled source block (1902). The downsampled source block (1902) can be combined with a low resolution prediction block (1903) to generate a downsampled residual block (1904). The downsampled residual block (1904) can then be coded into a coded video bitstream by a module (or modules not shown in FIG. 19) including transform, quantization, and entropy coding processes. To decode a source block (1901), the received coded video bitstream at the decoder side may be processed by entropy decoding, inverse quantization, and inverse transform processes to generate a downsampled residual block (1911). The downsampled residual block (1911) may be combined with a reduced resolution prediction block (1912) to generate a downsampled reconstructed block (1913). The downsampled reconstructed block (1913) may be upsampled by an upsampling module (1930) to generate a full resolution reconstructed block (1914) for the source block (1901). It should be noted that the reduced resolution prediction block (1912) may be generated by downsampling the full resolution reconstructed block with respect to the reference block of the source block (1901) by a downsampler module (1940).

[0133]

[0167] When the block level flag is off, e.g., when the block level flag is equal to a second predefined value, the downsampler modules (1920), (1940) and the upsampler module (1930) of Figure 19 are not applied. The downsampled or reduced resolution blocks (1902)-(1904) and (1911)-(1913) can be their full resolution counterparts.

[0134]

[0168] In one embodiment, for a coding block having a size of MxN, a reference block having a size of MxN is down-sampling factor D along the horizontal and vertical directions. x and D. y are downsampled by (M / D x )×(N / D y ) of the reduced resolution prediction block. Exemplary values ​​of M and N may include, but are not limited to, 256, 128, 64, 32, 16, and 8. The downsampling rate D x and D. y is an integer including, but not limited to, 2, 4, and 8.

[0169] In one embodiment, block level flags can be adaptively signaled or estimated on a per CU, superblock, prediction block, transform block, tile, coded segment, frame, or sequence basis.

[0135]

[0170] In one embodiment, for an upsampler module (1930) used to upsample the reduced resolution reconstructed block, the upsampling filter coefficients can be directly signaled or an index of a set of filter coefficients among multiple sets of predefined coefficients can be signaled.

[0136]

[0171] According to aspects of the present disclosure, the motion vectors of blocks that are spatially and / or temporally neighboring the current block may be scaled at the same or a different resolution as the current block in order to be referenced by the motion vector of the current block.

[0137]

[0172] For example, if the current block has a sampling ratio (or downsampling ratio) D x and D. y respectively along the horizontal and vertical directions, the sampling rate D ref,x and D. ref,y The motion vector of a spatially and / or temporally adjacent block has the horizontal and vertical components

[0138]

number

[0139]

[0173] FIG. 20 illustrates an example spatially neighboring motion vector referencing according to an embodiment of this disclosure.

[0140]

[0174] In an embodiment, based on a motion vector reference method such as the spatially adjacent motion vector reference method in AV1, the reference motion vector list construction process can search adjacent regions in units of WxH luma samples in the order of (1) to (8) shown in FIG. ij , the left WxH area L ij , the motion vectors for the top-left WxH region TL and the top-right WxH region TR are used as reference motion vectors in the candidate list.

[0141]

number

[0142]

[0175] FIG. 21 illustrates an exemplary temporally adjacent motion vector referencing according to an embodiment of this disclosure.

[0143]

[0176] In one embodiment, based on a motion vector reference method such as the temporally adjacent motion vector reference method in AV1, motion vectors mf_mv_1 and mf_mv_2 for a current block located at (blk_row, blk_col) in the current frame can be obtained as follows:

[0144]

[0177] As shown in Figure 21, the motion vector ref_mv for a WxH region located at (ref_blk_row, ref_blk_col) in a specified search area of ​​a reference frame (reference_frame1) is used to find a motion trajectory towards the previous frame (prior_frame). If the trajectory intersects with the current block located at (blk_row, blk_col), the motion vectors mf_mv_1 and mf_mv_2 for reference_frame1 and reference_frame2 are given as follows:

[0145]

number

[0146]

[0178] The derived motion vectors mf_mv_1 and mf_mv_2 are each subjected to the following steps before being used as reference motion vectors in the candidate list:

[0147]

number

[0148]

[0179] According to an aspect of the present disclosure, the motion vector candidates of a current block can be classified into two (or multiple) categories. As shown in FIG. 14, the motion vectors of spatially neighboring blocks located in the immediately upper row, the immediately left column, and the upper right corner of the current block can be classified as a first category (e.g., category 1) of motion vector candidates of the current block, while all other candidates are classified as a second category (e.g., category 2). Within each category, the motion vector candidates can be sorted in descending order of the count number in which each candidate appears. That is, a first motion vector candidate that appears more frequently than a second motion vector candidate in the candidate list is placed before the second motion vector candidate in the candidate list. Furthermore, the candidate list of the first category (e.g., category 1) can be concatenated with the candidate list of the second category (e.g., category 2) to form a single candidate list.

[0149]

[0180] In some embodiments, the sampling ratio used for the motion vector candidates, in addition to the categories and count numbers within each category, can be incorporated in constructing the candidate list.

[0150]

[0181] In one embodiment, motion vector candidates from neighboring blocks that have the same sampling rate as the current block may have a higher priority for being selected in the candidate list of each category under the same occurrence count.

[0151]

[0182] In one embodiment, two separate candidate lists can be constructed: a first candidate list can contain only candidate motion vectors from neighboring blocks with the same sampling ratio; a second candidate list can contain candidate motion vectors from neighboring blocks with different sampling ratios. Which candidate list is used can be signaled or inferred in addition to signaling the index of the selected reference motion vector in the candidate list.

[0152]

[0183] In one embodiment, the second candidate list may be scanned only if the number of candidates in the first candidate list is less than the specified number of indexed and signaled reference motion vectors.

[0153]

[0184] In one embodiment, based on an interleaving scheme, two separate candidate lists can be merged to form a single list. For example, if the total number of candidates from both lists is equal to or greater than the specified number of reference motion vectors, one from the first candidate list and another from the second candidate list can be selected in ascending order of entry position in the list until the specified number of reference motion vectors in the combined list is reached.

[0154]

[0185] FIG. 22 illustrates an example spatially neighboring motion vector referencing for affine motion prediction according to an embodiment of this disclosure.

[0155]

[0186] In one embodiment, based on a motion vector reference method such as the affine motion estimation method in AV1, an affine model for projecting a sample at (x, y) in a current block to a predicted sample at (x', y') in a reference block in a reference frame is given as follows:

[0156]

number

[0187] The affine parameter {h ij: i = 1, 2 and j = 1, 2} can be obtained as follows: The sample position in the current frame is (a k , b k ) = (x k , y k ) - (x0, y0), where k is the index of the neighboring block that has the same reference frame as the current block (k = 0 corresponds to the current block). In the example of Figure 22, k = 2, 3, 5, 6.

[0157]

[0188] The corresponding sample position in the reference frame is then given by:

[0158]

number

[0159]

[0189] A least-squares solution can be obtained as follows:

[0160]

number

[0161]

number

[0190] In the above explanation, the affine parameters (h 13 , h 23 ) is assumed to correspond to a translational motion vector at full resolution. If reduced-resolution prediction with sampling ratio D is applied to a block, then (h 13 , h23 ) accordingly

[0162]

number

[0163]

[0191] According to aspects of the present disclosure, when scaling adjacent motion vectors for coding a block using a reduced resolution coding mode, the horizontal and vertical components of the scaled motion vector may be derived using one of the following approaches:

[0164]

[0192] In the first approach, the scaling factor is a power of two (e.g., 2 N ), then the least significant N bits of the horizontal component of the motion vector and the least significant N bits of the vertical component of the motion vector are discarded to obtain a scaled motion vector.

[0165]

[0193] In the second approach, the scaling factor is a power of two (e.g., 2 N ), the motion vector is first rounded up by a rounding factor (e.g., 2 N-1 ), and then the least significant N bits of the horizontal component of the motion vector and the least significant N bits of the vertical component of the motion vector are discarded to obtain a scaled motion vector.

[0166]

[0194] In a third approach, when the scaling factor is not a power of two, a look-up table can be used to derive the values ​​of the horizontal and vertical components of the scaled motion vector.

[0167]

[0195] VI. Flowchart

[0196] FIG. 23 shows a flow chart outlining an exemplary process (2300) according to an embodiment of the present disclosure. In various embodiments, the process (2300) is performed by a processing circuit such as a processing circuit of a terminal device (210), (220), (230), (240), a processing circuit performing the function of a video encoder (303), a processing circuit performing the function of a video decoder (310), a processing circuit performing the function of a video decoder (410), a processing circuit performing the function of an intra prediction module (452), a processing circuit performing the function of a video encoder (503), a processing circuit performing the function of a predictor (535), a processing circuit performing the function of an intra encoder (622), or a processing circuit performing the function of an intra decoder (772). In some embodiments, the process (2300) is implemented with software instructions, and the processing circuit performs the process (2300) when the processing circuit executes the software instructions.

[0168]

[0197] The process (2300) may generally begin at step (S2310), where the process (2300) decodes a video bitstream to obtain a low resolution residual block for a current block. The process (2300) then proceeds to step (S2320).

[0169]

[0198] In step S2330, the process 2300 determines that a block level flag is set to a predefined value, the predefined value indicating that the current block is coded with reduced resolution coding, and then the process 2300 proceeds to step S2330.

[0170]

[0199] In step S2330, the process S2300 generates a reduced resolution prediction block for the current block by downsampling the full resolution reference block of the current block. The process S2300 then proceeds to step S2340.

[0171]

[0200] In step (S2340), the process (2300) generates a reduced-resolution reconstructed block for the current block based on the reduced-resolution prediction block and the reduced-resolution residual block. The process (2300) then proceeds to step (S2350).

[0172]

[0201] In step 2350, the process 2300 generates a full-resolution reconstructed block for the current block by upsampling the reduced-resolution reconstructed block. The process 2300 then ends.

[0173]

[0202] In an embodiment, the process (2300) determines the size of the reduced resolution prediction block based on the size of the full resolution reference block and the down-sampling ratio of the current block.

[0174]

[0203] In an embodiment, the process (2300) decodes a block level flag for the current block from the video bitstream, the block level flag indicating that the current block is coded with reduced resolution coding.

[0175]

[0204] In an embodiment, the process (2300) decodes one of filter coefficients or indices of filter coefficients from a video bitstream, the filter coefficients being used in upsampling the reduced resolution reconstructed block.

[0176]

[0205] In an embodiment, the process (2300) scales a motion vector of a first neighboring block of a current block based on a scaling factor that is a ratio of down-sampling rates of the current block and the first neighboring block. The process (2300) builds a first motion vector candidate list for the current block. The first motion vector candidate list includes the scaled motion vector of the first neighboring block.

[0177]

[0206] In an embodiment, the process (2300) determines a scaled motion vector based on a shift operation depending on whether the scaling factor is a power of 2. N In another example, if the scaling factor is 2 N If, first, the motion vector is rounded up by a rounding factor (e.g., 2 N-1 ), and then the least significant N bits of the horizontal component of the motion vector and the least significant N bits of the vertical component of the motion vector are discarded to obtain a scaled motion vector.

[0178]

[0207] In an embodiment, the process (2300) determines a scaled motion vector based on a look-up table in response to the scaling factor not being a power of two.

[0179]

[0208] In an embodiment, the process (2300) determines the priority of the scaled motion vector in the first motion vector candidate list based on the down-sampling ratios of the current block and the first neighboring block.

[0180]

[0209] In an embodiment, the process (2300) builds a second motion vector candidate list for the current block based on one or more second neighboring blocks of the current block, each of which has the same down-sampling rate as the current block. The process (2300) builds a third motion vector candidate list for the current block based on one or more third neighboring blocks of the current block, each of which has a different down-sampling rate than the current block.

[0181]

[0210] In an embodiment, the process (2300) scans the third motion vector candidate list based on the number of motion vector candidates in the second motion vector candidate list that is less than a specified number.

[0182]

[0211] In an embodiment, the process (2300) determines a fourth motion vector candidate list for the current block by merging the second motion vector candidate list and the third motion vector candidate list in an interleaved manner.

[0183]

[0212] In an embodiment, the process (2300) determines affine parameters for the current block based on the down-sampling ratio of the current block.

[0184]

[0213] VII. Computer Systems

[0214] The techniques described above may be implemented as computer software using computer readable instructions and may be physically stored on one or more computer readable media. For example, Figure 24 illustrates a computer system (2400) suitable for implementing certain embodiments of the disclosed subject matter.

[0185]

[0215] Computer software may be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to produce code that contains instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or instructions that may be executed via interpretation, microcode execution, etc.

[0186]

[0216] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0187]

[0217] 24 for the computer system (2400) are exemplary in nature and are not intended to suggest any limitations on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of the computer system (2400).

[0188]

[0218] The computer system (2400) may include certain human interface input devices that may be responsive to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), auditory input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic pictures).

[0189]

[0219] The input human interface devices may include one or more of (although only one of each is depicted) a keyboard (2401), a mouse (2402), a trackpad (2403), a touch screen (2910), a data glove (not shown), a joystick (2405), a microphone (2406), a scanner (2407), and a camera (2408).

[0190]

[0220] The computer system (2400) may also include certain human interface output devices that may stimulate one or more of the senses of a human user, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (2410), data gloves (not shown), joystick (2405), although there may be haptic feedback devices that do not serve as input devices), auditory output devices (e.g., speakers (2409), headphones (not shown)), visual output devices (e.g., screens (2410), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, three- or more-dimensional output by means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown). These visual output devices (such as the screen (2410)) may be connected to the system bus (2448) via a graphics adapter (2450).

[0191]

[0221] The computer system (2400) may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2420) using media such as CD / DVD (2421), thumb drives (2422), removable hard drives or solid state drives (2423), legacy magnetic media (not shown) such as tapes and floppy disks (not shown), and specialized ROM / ASIC / PLD based devices such as security dongles (not shown).

[0192]

[0222] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other transitional signals.

[0193]

[0223] The computer system (2400) may also include an interface to one or more communications networks (2455). The one or more networks (2455) may be, for example, wireless, wired, or optical. The one or more networks (2455) may further be local, wide area, metropolitan, vehicle industrial, real-time, delay tolerant, etc. Examples of the one or more networks (2455) include Ethernet, wireless LAN, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), TV wired or wireless wide area digital networks (including cable TV, satellite TV, and terrestrial TV), vehicle industrial including CANBus, etc. Certain networks typically require an external network interface adapter that is attached to a particular general-purpose data port or peripheral bus (2449) (e.g., a USB port on the computer system (2400)); others are typically integrated into the core of the computer system (2400) by attaching to a system bus as described below (e.g., an Ethernet interface is integrated in a PC computer system, a cellular network interface is integrated in a smartphone computer system). Using any of these networks, the computer system (2400) can communicate with other entities. Such communications can be one-way receive only (e.g., broadcast television), one-way transmit only (e.g., CANbus to certain CANbus devices), or bi-directional, such as to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks can be used with each of these networks and network interfaces, as described above.

[0194]

[0224] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (2440) of the computer system (2400).

[0195]

[0225] The cores (2440) may include one or more central processing units (CPUs) (2441), graphics processing units (GPUs) (2442), specialized programmable processing devices in the form of field programmable gate areas (FPGAs) (2443), task specific hardware accelerators (2444), etc. These devices may be connected via a system bus (2448) along with read only memory (ROM) (2445), random access memory (2446), and internal mass storage devices (e.g., internal non-user accessible hard drives, SSDs, etc.) (2447). In some computer systems, the system bus (2448) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (2448) or via a peripheral bus (2449). In one example, a screen 2410 may be connected to a graphics adapter 2450. Peripheral bus architectures include PCI, USB, and the like.

[0196]

[0226] The CPU (2441), GPU (2442), FPGA (2443), and accelerator (2444) may combine to execute certain instructions that may constitute the aforementioned computer code. The computer code may be stored in a ROM (2445) or a RAM (2446). Temporary data may be stored in the RAM (2446), while persistent data may be stored, for example, in an internal mass storage (2447). Fast storage and retrieval from any memory device may be made possible by utilizing a cache memory, which may be closely associated with one or more of the CPU (2441), GPU (2442), mass storage (2447), ROM (2445), RAM (2446), etc.

[0197]

[0227] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code can be specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those having skill in the computer software arts.

[0198]

[0228] By way of example, and not by way of limitation, a computer system having the architecture (2400), and in particular the core (2440), as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.), can provide the functionality of executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as the core's internal mass storage (2447) or specific storage of the core (2440) of a non-transitory nature, such as ROM (2445). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2440). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (2440) and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (2446) and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embedded in circuitry (e.g., accelerator (2444)), which may execute in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, as appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0199]

[0229] While this disclosure has described several exemplary embodiments, there are modifications, permutations, and various alternative equivalents that fall within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

[0200]

[0230] Additional Notes (Appendix 1) 1. A video decoding method in a decoder, comprising: decoding the video bitstream to obtain a reduced resolution residual block for a current block; determining that a block level flag is set to a predefined value, the predefined value indicating that the current block is coded with reduced resolution coding; generating a reduced-resolution prediction block for the current block by down-sampling a full-resolution reference block of the current block based on the block-level flag; generating a reduced-resolution reconstructed block for the current block based on the reduced-resolution prediction block and the reduced-resolution residual block; and generating a full resolution reconstructed block for the current block by upsampling the reduced resolution reconstructed block; The method includes:

[0201] (Appendix 2) 2. The method according to claim 1, wherein the step of generating the reduced resolution prediction block comprises: determining a size of the reduced-resolution prediction block based on a size of the full-resolution reference block and a down-sampling ratio of the current block; A method comprising:

[0202] (Appendix 3) 2. The method of claim 1, wherein the determining step comprises: 23. The method of claim 22, further comprising: decoding from the video bitstream the block level flag for the current block, the block level flag indicating that the current block is coded with the reduced resolution coding.

[0203] (Appendix 4) 2. The method of claim 1, wherein the determining step comprises: The method of claim 1, further comprising the step of decoding one of filter coefficients or indices of the filter coefficients from the video bitstream, the filter coefficients being used in upsampling the reduced resolution reconstructed block.

[0204] (Appendix 5) In the method according to Appendix 1: Scaling the motion vector of the first neighboring block of the current block based on a scaling factor, which is a ratio of down-sampling rates of the current block and the first neighboring block; and constructing a first motion vector candidate list for the current block, the first motion vector candidate list including a scaled motion vector of the first neighboring block; The method further comprises:

[0205] (Appendix 6) 6. The method of claim 5, wherein the scaling step comprises: determining the scaled motion vector based on a shift operation in response to the scaling factor being a power of two; and determining the scaled motion vector based on a look-up table in response to the scaling factor being not a power of two; A method comprising:

[0206] (Appendix 7) 6. The method of claim 5, further comprising: determining a priority of the scaled motion vector in the first motion vector candidate list based on down-sampling ratios of the current block and the first neighboring block; The method further comprises:

[0207] (Appendix 8) In the method according to Appendix 1: constructing a second motion vector candidate list for the current block based on one or more second neighboring blocks of the current block, each of the one or more second neighboring blocks having the same down-sampling rate as the current block; and constructing a third motion vector candidate list for the current block based on one or more third neighboring blocks of the current block, each of the one or more third neighboring blocks having a different down-sampling rate than the current block; The method further comprises:

[0208] (Appendix 9) In the method according to claim 8, scanning the third motion vector candidate list based on a number of motion vector candidates in the second motion vector candidate list being less than a specified number; The method further comprises:

[0209] (Appendix 10) In the method according to claim 8, determining a fourth motion vector candidate list for the current block by merging the second motion vector candidate list and the third motion vector candidate list in an interleaved manner; The method further comprises:

[0210] (Appendix 11) In the method according to Appendix 1: determining affine parameters for the current block based on a down-sampling ratio of the current block; The method further comprises:

[0211] (Appendix 12) 1. An apparatus including a processing circuit, the processing circuit comprising: decoding the video bitstream to obtain a reduced resolution residual block for a current block; determining that a block level flag is set to a predefined value, the predefined value indicating that the current block is coded with reduced resolution coding; generating a reduced-resolution prediction block for the current block by down-sampling a full-resolution reference block of the current block based on the block-level flag; generating a reduced-resolution reconstructed block for the current block based on the reduced-resolution prediction block and the reduced-resolution residual block; and generating a full resolution reconstructed block for the current block by upsampling the reduced resolution reconstructed block; The apparatus is configured to:

[0212] (Appendix 13) 13. The apparatus of claim 12, wherein the processing circuitry: determining a size of the reduced-resolution prediction block based on a size of the full-resolution reference block and a down-sampling ratio of the current block; The apparatus, further configured to:

[0213] (Appendix 14) 13. The apparatus of claim 12, wherein the processing circuitry: the block level flag for the current block from the video bitstream, the block level flag indicating that the current block is coded with the reduced resolution coding.

[0214] (Appendix 15) 13. The apparatus of claim 12, wherein the processing circuitry: The apparatus is further configured to decode one of filter coefficients or indices of the filter coefficients from the video bitstream, the filter coefficients being used in up-sampling the reduced resolution reconstructed block.

[0215] (Appendix 16) 13. The apparatus of claim 12, wherein the processing circuitry: Scaling the motion vector of the first neighboring block of the current block based on a scaling factor, which is a ratio of down-sampling rates of the current block and the first neighboring block; and constructing a first motion vector candidate list for the current block, the first motion vector candidate list including a scaled motion vector of the first neighboring block; The apparatus, further configured to:

[0216] (Appendix 17) 17. The apparatus of claim 16, wherein the processing circuitry comprises: determining the scaled motion vector based on a shift operation in response to the scaling factor being a power of two; and determining the scaled motion vector based on a look-up table in response to the scaling factor being not a power of two; The apparatus, further configured to:

[0217] (Appendix 18) 17. The apparatus of claim 16, wherein the processing circuitry comprises: The apparatus is further configured to determine a priority of the scaled motion vector in the first motion vector candidate list based on down-sampling ratios of the current block and the first neighboring block.

[0218] (Appendix 19) 13. The apparatus of claim 12, wherein the processing circuitry: constructing a second motion vector candidate list for the current block based on one or more second neighboring blocks of the current block, each of the one or more second neighboring blocks having the same down-sampling rate as the current block; and constructing a third motion vector candidate list for the current block based on one or more third neighboring blocks of the current block, each of the one or more third neighboring blocks having a different down-sampling rate than the current block; The apparatus, further configured to:

[0219] (Appendix 20) A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to: decoding the video bitstream to obtain a reduced resolution residual block for a current block; determining that a block level flag is set to a predefined value, the predefined value indicating that the current block is coded with reduced resolution coding; generating a reduced-resolution prediction block for the current block by down-sampling a full-resolution reference block of the current block based on the block-level flag; generating a reduced-resolution reconstructed block for the current block based on the reduced-resolution prediction block and the reduced-resolution residual block; and generating a full resolution reconstructed block for the current block by upsampling the reduced resolution reconstructed block; A storage medium that executes the above. [Explanation of symbols]

[0220]

[0231] Appendix A: Acronyms ALF: Adaptive Loop Filter (adaptive multiple transformation) AMVP: Advanced Motion Vector Prediction APS: Adaptation Parameter Set ASIC: Application-Specific Integrated Circuit ATMVP: Alternative / Advanced Temporal Motion Vector Prediction AV1: AOMedia Video 1 AV2: AOMedia Video 2 BMS: Benchmark Set BV: Block Vector CANBus: Controller Area Network Bus CB: Coding Block CC-ALF: Cross-Component Adaptive Loop Filter CD: Compact Disc CDEF: Constrained Directional Enhancement Filter CPR: Current Picture Referencing CPU: Central Processing Unit CRT: Cathode Ray Tube CTB: Coding Tree Block CTU: Coding Tree Unit CU: Coding Unit DPB: Decoder Picture Buffer DPCM: Differential Pulse-Code Modulation DPS: Decoding Parameter Set DVD: Digital Video Disc FPGA: Field Programmable Gate Area JCCR: Joint CbCr Residual Coding JVET: Joint Video Exploration Team GOP: Groups of Pictures GPU: Graphics Processing Unit GSM: Global System for Mobile communications HDR: High Dynamic Range HEVC: High Efficiency Video Coding HRD: Hypothetical Reference Decoder IBC: Intra Block Copy IC: Integrated Circuit ISP: Intra Sub-Partitions JEM: Joint Exploration Model LAN: Local Area Network LCD: Liquid-Crystal Display LR: Loop Restoration Filter LRU: Loop Restoration Unit LTE: Long-Term Evolution MPM: Most Probable Mode MV: Motion Vector OLED: Organic Light-Emitting Diode PBs: Prediction Blocks PCI: Peripheral Component Interconnect PDPC: Position Dependent Prediction Combination PLD: Programmable Logic Device PPS: Picture Parameter Set PU: Prediction Unit RAM: Random Access Memory ROM: Read-Only Memory SAO: Sample Adaptive Offset SCC: Screen Content Coding SDR: Standard Dynamic Range SEI: Supplementary Enhancement Information SNR: Signal Noise Ratio SPS: Sequence Parameter Set SSD: Solid-state drive TU: Transform Unit USB: Universal Serial Bus VPS: Video Parameter Set VUI: Video Usability Information VVC: Versatile Video Coding WAIP: Wide-Angle Intra Prediction

Claims

1. 1. A video decoding method in a decoder, comprising: decoding the video bitstream to obtain a reduced-resolution residual block for a current block; generating a reduced resolution prediction block for the current block by down-sampling a full resolution reference block of the current block; generating a reduced-resolution reconstructed block for the current block based on the reduced-resolution prediction block and the reduced-resolution residual block; and generating a full resolution reconstructed block for the current block by up-sampling the reduced resolution reconstructed block; The method comprises: scaling motion vectors of neighboring blocks of the current block based on a scaling factor that is a ratio of down-sampling rates of the current block and the neighboring blocks, and constructing a motion vector candidate list including the scaled motion vectors of the neighboring blocks; wherein a priority of the scaled motion vector in the motion vector candidate list is determined such that, when a motion vector is selected from the motion vector candidate list for reconstruction of the current block, a motion vector candidate from the neighboring block that has the same down-sampling ratio as the current block has a higher priority.

2. 2. The method of claim 1, wherein the motion vector candidates are classified into a plurality of categories, and motion vector candidates from immediately adjacent blocks above, to the left, or to the upper right of the current block have a higher priority than motion vector candidates from other blocks.

3. 2. The method of claim 1, wherein generating the reduced resolution prediction block comprises: determining a size of the reduced resolution prediction block based on a size of the full resolution reference block and a down-sampling ratio of the current block; A method comprising:

4. The method of claim 1, 23. The method of claim 22, further comprising: decoding a block level flag for the current block from the video bitstream, the block level flag indicating that the current block is coded at the reduced resolution.

5. The method of claim 1, The method of claim 1, further comprising the step of decoding one of filter coefficients or indices of the filter coefficients from the video bitstream, the filter coefficients being used in upsampling the reduced resolution reconstructed block.

6. 2. The method of claim 1, wherein the scaling step comprises: determining the scaled motion vector based on a shift operation in response to the scaling factor being a power of two; and determining the scaled motion vector based on a look-up table in response to the scaling factor not being a power of two; A method comprising:

7. 13. The method of claim 1 : constructing a second motion vector candidate list for the current block based on one or more second neighboring blocks of the current block, each of the one or more second neighboring blocks having the same down-sampling rate as the current block; and constructing a third motion vector candidate list for the current block based on one or more third neighboring blocks of the current block, each of the one or more third neighboring blocks having a different down-sampling rate than the current block; The method further comprises:

8. 8. The method of claim 7, scanning the third motion vector candidate list based on a number of motion vector candidates in the second motion vector candidate list being less than a specified number; The method further comprises:

9. 8. The method of claim 7, determining a fourth motion vector candidate list for the current block by merging the second motion vector candidate list and the third motion vector candidate list in an interleaved manner; The method further comprises:

10. 13. The method of claim 1 : determining affine parameters for the current block based on a down-sampling rate of the current block; The method further comprises:

11. 11. Apparatus comprising processing circuitry configured to carry out the method of any one of claims 1 to 10.

12. A computer program product configured to cause a computer processor to carry out a method according to any one of claims 1 to 10.

Images