METHOD AND APPARATUS FOR VIDEO CODING - Patent application

Abstract

To provide a method and an apparatus for video coding.SOLUTION: Aspects of the present disclosure include a method, an apparatus, and a non-transitory computer-readable storage media for video encoding / decoding. The apparatus includes a processing circuit that decodes prediction information of a current block within a current picture that is a part of a coded video sequence. Prediction information indicates one of multiple intra prediction directions for the current block. The processing circuit determines a subset of the plurality of reference lines on the basis of one of the plurality of intra prediction directions indicated in the prediction information. The processing circuit performs intra prediction of the current block on the basis of the determined subset of the plurality of reference lines. The processing circuit reconstructs the current block on the basis of the intra prediction of the current block.SELECTED DRAWING: Figure 20

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Patent Application No. 17 / 360,803, entitled "METHOD AND APPARATUS FOR VIDEO CODING," filed June 28, 2021, which in turn claims the benefit of priority to U.S. Provisional Application No. 63 / 082,806, entitled "INTERPOLATION-FREE DIRECTIONAL INTRA PREDICTION," filed September 24, 2020. The entire disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] This disclosure describes embodiments that relate generally to video coding. [Background technology]

[0003] The discussion of the background art provided herein is intended to generally present the context for the present disclosure. To the extent that it is provided in this Background section, the inventors' work described herein, as well as aspects of the description that may not be admitted as prior art at the time of filing, are not admitted expressly or implicitly as prior art to the present disclosure.

[0004] Video coding and decoding may be performed using inter-picture prediction with motion compensation. Uncompressed digital video may include a series of pictures, each with spatial dimensions of, for example, 1920 x 1080 luma samples and associated chrominance samples. The series of pictures may have a fixed or variable picture rate (informally known as a frame rate), for example, 60 pictures per second, or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video (1920 x 1080 luma sample resolution at a 60 Hz frame rate) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 GBytes of storage.

[0005] One goal of video coding and decoding is to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage requirements by two or more orders of magnitude, in some cases. Both lossless and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to techniques that can reconstruct an exact copy of the original signal from a compressed version. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal can be used for its intended purpose. For video, lossy compression is widely adopted. The amount of acceptable distortion depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect that the higher the acceptable / tolerable distortion, the higher the compression ratio.

[0006] Video encoders and decoders may utilize techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding.

[0007] Video codec technology may include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to other data from samples or previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into blocks of samples. If all blocks of samples are coded in intra mode, the picture may be an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, may be used to reset the decoder state and therefore may be used as the first picture in a coded video bitstream and video session or as a still image. Samples of intra-blocks may undergo a transform, and the transform coefficients may be quantized before entropy coding. Intra-prediction may be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, smaller DC values ​​and smaller AC coefficients after the transform require fewer bits at a given quantization step size to represent the block after entropy coding.

[0008] Conventional intra-coding, such as that known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to predict intra-prediction from surrounding sample data and / or metadata obtained during encoding and / or decoding of spatially neighboring and preceding data blocks in decoding order. Such techniques are hereinafter referred to as "intra-prediction" techniques. Note that in at least some cases, intra-prediction uses only reference data from the current picture being reconstructed, not from reference pictures.

[0009] Intra-prediction can take many different forms. If two or more of such techniques can be used in a given video coding technique, the techniques used can be coded as intra-prediction modes. In some cases, modes can have sub-modes and / or parameters, which can be coded separately or included in the mode's codeword. The codeword used for a given mode, sub-mode, and / or parameter combination can affect coding efficiency via intra-prediction, as can the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain modes of intra prediction were introduced in H.264, improved in H.265, and further refined with new coding techniques such as Joint Search Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Predictor blocks can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of the neighboring samples are copied into the predictor block according to a direction. The reference to the direction used can be coded into the bitstream or can itself be predicted.

[0011] Referring to FIG. 1A, shown at the bottom right is a subset of nine known predictor directions from the 33 possible predictor directions in H.265 (corresponding to the 33 angle modes out of the 35 intra modes). The point where the arrows converge (101) represents the sample being predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right at a 45-degree angle from horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left sample (101) at a 22.5-degree angle from horizontal.

[0012] 1A, a square block (104) of 4x4 samples (shown in bold dashed lines) is shown in the upper left. The square block (104) contains 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. Because the block size is 4x4 samples, S44 is located in the lower right. Reference samples are also shown, following a similar numbering scheme. The reference samples are labeled R and their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, predicted samples are neighbors of the block being reconstructed, and therefore, negative values ​​need not be used.

[0013] Intra-picture prediction can work by copying reference sample values ​​from neighboring samples as appropriate according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling for this block indicating a prediction direction consistent with arrow (102), i.e., the sample is predicted from one or more prediction samples to the upper right, at a 45-degree angle from horizontal. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, particularly when the orientation is not evenly divisible by 45 degrees, multiple reference sample values ​​can be combined, for example by interpolation, to calculate the reference sample.

[0015] The number of possible directions has increased as video coding technology has evolved. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions as of the time of this disclosure. Experiments have been conducted to determine the most likely directions, using certain techniques in entropy coding to represent possible directions with a small number of bits while accepting a certain penalty for less likely directions. Furthermore, the direction itself may be predictable from neighboring directions used in nearby already decoded blocks.

[0016] Figure 1B shows a schematic diagram (105) showing 65 intra prediction directions with JEM to illustrate the increasing number of prediction directions over time.

[0017] The mapping of intra-prediction direction bits in a coded video bitstream to represent directions may vary from one video coding technique to another and may range, for example, from a simple direct mapping of prediction directions to complex adaptive schemes including intra-prediction modes, codewords, most-probable modes, and similar techniques. However, in all cases, there may be certain directions that are statistically less likely to occur in the video content than certain other directions. Because the goal of video compression is to reduce redundancy, in a well-performing video coding technique, these less likely directions are represented by more bits than more likely directions.

[0018] Motion compensation may be a lossy compression technique, and may refer to a technique in which blocks of sample data from a previously reconstructed picture or portion thereof (reference picture) are spatially shifted in a direction indicated by a motion vector (hereinafter, MV) and then used to predict a newly reconstructed picture or picture portion. In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, with the third dimension being an indication of the reference picture used (the latter may indirectly be a temporal dimension).

[0019] In some video compression techniques, the MV applicable to a given region of sample data can be predicted from other MVs, e.g., from an MV associated with another region of sample data that is spatially adjacent to the region being reconstructed and precedes that MV in decoding order. Doing so can significantly reduce the amount of data required to code the MV, thereby eliminating redundancy and increasing compression. For example, when coding an input video signal derived from a camera (known as natural video), MV prediction can work effectively because there is a statistical likelihood that regions larger than the region to which a single MV is applicable will move in a similar direction, and therefore, in some cases, can be predicted using similar MVs derived from MVs in nearby regions. This ensures that the MV found for a given region will be similar or identical to the MV predicted from surrounding MVs, and as a result, after entropy coding, it can be represented using fewer bits than would be required to code the MV directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., a sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating a predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms provided by H.265, the one described herein is a technique hereinafter referred to as "spatial merging".

[0021] Referring to Figure 1C, the current block (111) may contain samples that the encoder found during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of directly coding its MV, the MV can be derived from metadata associated with one or more reference pictures, e.g., from the most recent reference picture (in decoding order), using the MV associated with any one of five surrounding samples, denoted A0, A1, and B0, B1, B2 (112-116, respectively). In H.265, MV prediction can use predictors from the same reference picture as neighboring blocks. Summary of the Invention [Means for solving the problem]

[0022] An aspect of the present disclosure provides an apparatus for video encoding / decoding. The apparatus includes a processing circuit that decodes prediction information for a current block in a current picture that is part of a coded video sequence. The prediction information indicates one of a plurality of intra-prediction directions for the current block. The processing circuit determines a subset of a plurality of reference lines based on the one of the plurality of intra-prediction directions indicated in the prediction information. The processing circuit performs intra-prediction of the current block based on the determined subset of the plurality of reference lines. The processing circuit reconstructs the current block based on the intra-prediction of the current block.

[0023] In one embodiment, the number of reference lines in the determined subset of the plurality of reference lines is greater than one.

[0024] In one embodiment, the intra-prediction direction associated with a first reference line of the plurality of reference lines is different from the intra-prediction direction associated with a second reference line of the plurality of reference lines.

[0025] In one embodiment, the multiple intra-prediction directions are associated with a first reference line of the multiple reference lines, and a subset of the multiple intra-prediction directions is associated with a second reference line of the multiple reference lines.

[0026] In one embodiment, the processing circuit determines, for each sample of the current block, one of a subset of a plurality of reference lines based on one of a plurality of intra-prediction directions and the position of the respective sample.

[0027] In one embodiment, the prediction information includes a syntax element that indicates whether intra prediction should be performed for the current block based on multiple reference lines.

[0028] In one embodiment, the current block is not located adjacent to the top boundary of the coding tree unit that contains the current block.

[0029] In one embodiment, one of the tangent and cotangent values ​​of the prediction angle associated with one of the multiple intra-prediction directions is an integer.

[0030] In one embodiment, the processing circuit determines, for each row of samples of the current block, a reference line index of one of a subset of multiple reference lines based on a tangent value of a prediction angle associated with one of the intra-prediction directions and a row number of the respective row of samples of the current block.

[0031] An aspect of the present disclosure provides a method for video encoding / decoding. In the method, prediction information of a current block in a current picture that is part of a coded video sequence is decoded. The prediction information indicates one of a plurality of intra-prediction directions for the current block. A subset of a plurality of reference lines is determined based on the one of the plurality of intra-prediction directions indicated in the prediction information. Intra-prediction of the current block is performed based on the determined subset of the plurality of reference lines. The current block is reconstructed based on the intra-prediction of the current block.

[0032] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform any one or combination of methods for video decoding.

[0033] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0034] [Figure 1A] FIG. 2 is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B] FIG. 1 is a diagram of an exemplary intra-prediction direction. [Figure 1C] FIG. 1 is a schematic diagram of an example of a current block and its surrounding spatial merge candidates. [Figure 2] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Figure 3] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Figure 4] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 5] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 6]FIG. 10 is a block diagram of an encoder according to another embodiment. [Figure 7] FIG. 10 is a block diagram of a decoder according to another embodiment. [Figure 8] FIG. 2 illustrates an example block partition according to some embodiments of the present disclosure. [Figure 9] FIG. 2 illustrates an example block partition according to some embodiments of the present disclosure. [Figure 10] FIG. 2 illustrates an example block partition according to some embodiments of the present disclosure. [Figure 11] FIG. 1 illustrates an exemplary quadtree with nested multi-type tree coding block structure, according to one embodiment of the present disclosure. [Figure 12] FIG. 10 illustrates an exemplary nominal angle according to one embodiment of the present disclosure. [Figure 13] FIG. 2 illustrates the locations of the top, left, and top-left samples of one pixel in a current block according to one embodiment of the present disclosure. [Figure 14] FIG. 2 illustrates an exemplary bilinear interpolation for deriving prediction samples at fractional positions according to one embodiment of the present disclosure. [Figure 15] FIG. 10 illustrates an exemplary multi-line intra prediction using four reference lines adjacent to a coding block unit, according to one embodiment of the present disclosure. [Figure 16] FIG. 10 is a diagram illustrating exemplary angles of intra-prediction directions according to one embodiment of this disclosure. [Figure 17] FIG. 10 illustrates an example predicted angle according to some embodiments of the present disclosure. [Figure 18] FIG. 10 illustrates an exemplary intra prediction using two reference lines, according to one embodiment of the present disclosure. [Figure 19] FIG. 10 illustrates an exemplary intra prediction using three reference lines, according to one embodiment of the present disclosure. [Figure 20] FIG. 1 illustrates an exemplary flowchart according to one embodiment of the present disclosure. [Figure 21]FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0035] I. Video Decoder and Encoder Systems FIG. 2 shows a simplified block diagram of a communication system (200) according to one embodiment of the present disclosure. The communication system (200) includes multiple terminal devices capable of communicating with each other, for example, via a network (250). For example, the communication system (200) includes a first pair of terminal devices (210) and (220) interconnected via the network (250). In the example of FIG. 2, the first pair of terminal devices (210) and (220) perform unidirectional transmission of data. For example, the terminal device (210) may code video data (e.g., a stream of video pictures captured by the terminal device (210)) for transmission to another terminal device (220) via the network (250). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (220) may receive the coded video data from the network (250), decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. One-way data transmission may be common, such as in media serving applications.

[0036] In another example, the communication system (200) includes a second pair of terminal devices (230) and (240) that perform bidirectional transmission of coded video data, such as may occur during a video conference. For the bidirectional transmission of data, in one example, each of the terminal devices (230) and (240) can code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (230) and (240) over the network (250). Each of the terminal devices (230) and (240) can also receive coded video data transmitted by the other of the terminal devices (230) and (240), decode the coded video data to recover the video pictures, and display the video pictures on an accessible display device in accordance with the recovered video data.

[0037] In the example of FIG. 2 , the terminal devices 210, 220, 230, and 240 may be depicted as a server, a personal computer, and a smartphone, although the principles of the present disclosure need not be so limited. Embodiments of the present disclosure apply to laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. The network 250 represents any number of networks that convey coded video data between the terminal devices 210, 220, 230, and 240, including, for example, wired (cabled) and / or wireless communication networks. The communication network 250 can exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of the network 250 may not be important to the operation of the present disclosure, unless otherwise described herein below.

[0038] 3 shows the arrangement of a video encoder and a video decoder in a streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0039] The streaming system may include a capture subsystem (313), which may include a video source (301), such as a digital camera, that generates a stream of uncompressed video pictures (302). In one example, the stream of video pictures (302) includes samples captured by the digital camera. The stream of video pictures (302), shown as a thick line to emphasize its high data volume compared to the encoded video data (304) (or coded video bitstream), may be processed by an electronic device (320) that includes a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (304), shown as a thin line to emphasize its low data volume compared to the stream of video pictures (302), may be stored on a streaming server (305) for future use. One or more streaming client subsystems, such as the client subsystems (306) and (308) of Figure 3, can access the streaming server (305) to obtain copies (307) and (309) of the encoded video data (304). The client subsystem (306) can include a video decoder (310), for example, within an electronic device (330). The video decoder (310) decodes the input copy (307) of the encoded video data and creates an output stream (311) of video pictures that can be rendered on a display (312) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (304), (307), and (309) (e.g., a video bitstream) can be encoded according to certain video coding / compression standards. An example of such a standard is ITU-T Recommendation H.265.In one example, a video standard under development is informally known as Versatile Video Coding (VVC), and the disclosed subject matter may be used in the context of VVC.

[0040] It should be noted that the electronic devices (320) and (330) may include other components (not shown). For example, the electronic device (320) may include a video decoder (not shown), and the electronic device (330) may also include a video encoder (not shown).

[0041] 4 shows a block diagram of a video decoder (410) according to one embodiment of the present disclosure. The video decoder (410) may be included in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., a receiving circuit). The video decoder (410) may be used in place of the video decoder (310) in the example of FIG. 3.

[0042] The receiver (431) may receive one or more coded video sequences to be decoded by the video decoder (410), or in the same or other embodiments, one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel (401), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (431) may receive the encoded video data with other data, such as coded audio data and / or auxiliary data streams, which may be transferred to each other using entities (not shown). The receiver (431) may separate the coded video sequences from other data. To combat network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420) (hereinafter, "parser (420)"). In certain applications, the buffer memory (415) is part of the video decoder (410). In other cases, it may be external to the video decoder (410) (not shown). In still other cases, there may be a buffer memory (not shown) external to the video decoder (410), for example, to combat network jitter, and another buffer memory (415) internal to the video decoder (410), for example, to handle playout timing. When the receiver (431) is receiving data from a store / forward device with sufficient bandwidth and control possibilities or from an isosynchronous network, the buffer memory (415) may not be needed or may be small. For use with best-effort packet networks such as the Internet, a buffer memory (415) may be needed, may be relatively large, advantageously adaptively sized, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (410).

[0043] The video decoder (410) may include a parser (420) to reconstruct symbols (421) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (410) and, potentially, information for controlling a drawing device, such as a drawing device (412) (e.g., a display screen) that is not an integral part of the electronic device (430) but may be coupled to the electronic device (430), as shown in FIG. 4. The control information for the drawing device(s) may be in the form of a supplemental enhancement information (SEI) message or a parameter set fragment (not shown) of video usability information (VUI). The parser (420) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context-dependent coding, etc. The parser (420) may extract a set of subgroup parameters for at least one of the pixel subgroups in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroups may include Groups of Pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (420) may also extract from the coded video sequence information such as transform coefficients, quantization parameter values, MVs, etc.

[0044] The parser (420) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (415) to create symbols (421).

[0045] The reconstruction of the symbols (421) can involve several different units, depending on the type of coded video picture or portion thereof (e.g., inter-picture and intra-picture, inter-block and intra-block, etc.), as well as other factors. Which units participate and how may be controlled by subgroup control information parsed from the coded video sequence by the parser (420). The flow of such subgroup control information between the parser (420) and the following units is not shown for clarity.

[0046] Beyond the functional blocks already described, the video decoder (410) can be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0047] The first unit is a scalar / inverse transform unit (451), which receives quantized transform coefficients as well as control information from the parser (420) as symbol(s) (421), including the transform used, block size, quantization coefficients, quantization scaling matrix, etc. The scalar / inverse transform unit (451) can output blocks containing sample values ​​that can be input to an aggregator (455).

[0048] In some cases, the output samples of the scaler / inverse transform (451) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) generates blocks of the same size and shape as the block being reconstructed using already reconstructed surrounding information retrieved from a current picture buffer (458). The current picture buffer (458), for example, buffers partially reconstructed and / or fully reconstructed current pictures. The aggregator (455) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451).

[0049] In other cases, the output samples of the scalar / inverse transform unit (451) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion-compensated prediction unit (453) can access a reference picture memory (457) to retrieve samples used for prediction. After motion-compensating the retrieved samples according to the symbols (421) associated with the block, these samples may be added to the output of the scalar / inverse transform unit (451) by the aggregator (455) to generate output sample information (in this case, referred to as residual samples or residual signals). The addresses in the reference picture memory (457) from which the motion-compensated prediction unit (453) retrieves prediction samples may be controlled by the MV, for example, so that they are available to the motion-compensated prediction unit (453) in the form of symbols (421) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory (457) when sub-sample accurate MVs are used, MV prediction mechanisms, etc.

[0050] The output samples of the aggregator (455) may be subjected to various loop filtering techniques in the loop filter unit (456). Video compression techniques may include in-loop filter techniques that are controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (456) as symbols (421) from the parser (420), but may also be responsive to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and may also be responsive to previously reconstructed and loop-filtered sample values.

[0051] The output of the loop filter unit (456) can be a sample stream that can be output to a drawing device (412) and stored in a reference picture memory (457) for use in future inter-picture prediction.

[0052] Certain coded pictures, once fully reconstructed, may be used as reference pictures for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (420)), the current picture's buffer (458) may become part of the reference picture memory (457), and the new current picture's buffer may be relocated before beginning reconstruction of the subsequent coded picture.

[0053] The video decoder (410) may perform decoding operations according to a predetermined video compression technique in a standard, such as ITU-T Rec. H.265. The coded video sequence may comply with the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, the profile may select certain tools from all tools available in the video compression technique or standard as the only tools usable under that profile. Compliance may also require that the complexity of the coded video sequence be within the boundaries defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further constrained by a hypothetical reference decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0054] In one embodiment, the receiver (431) may receive additional (redundant) data with the encoded video. The additional data may be included as part of the coded video sequence(s). The additional data may be used by the video decoder (410) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0055] 5 shows a block diagram of a video encoder (503) according to one embodiment of the present disclosure. The video encoder (503) is included in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmitting circuit). The video encoder (503) can be used in place of the video encoder (303) in the example of FIG. 3.

[0056] The video encoder (503) can receive video samples from a video source (501) (which in the example of FIG. 5 is not part of the electronic device (520)) that can capture the video image(s) to be coded by the video encoder (503). In another example, the video source (501) is part of the electronic device (520).

[0057] The video source (501) may provide a source video sequence to be coded by the video encoder (503) in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (501) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (501) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that, when viewed sequentially, impart motion. The picture itself may be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description will focus on samples.

[0058] According to one embodiment, the video encoder (503) may code and compress pictures of a source video sequence into a coded video sequence (543) in real time or under any other time constraint required by the application. Applying the appropriate coding rate is one function of the controller (550). In some embodiments, the controller (550) controls and is operatively coupled to other functional units as described below. For clarity, coupling is not depicted. Parameters set by the controller (550) may include rate control-related parameters (e.g., picture skip, quantizer, lambda value for rate-distortion optimization techniques), picture size, group of pictures (GOP) layout, maximum MV allowed reference region, etc. The controller (550) may be configured with other appropriate functions for the video encoder (503) optimized for a given system design.

[0059] In some embodiments, the video encoder (503) is configured to operate in a coding loop. As an overly simplified explanation, in one example, the coding loop can include a source coder (530) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be coded and reference picture(s)) and a (local) decoder (533) embedded in the video encoder (503). The decoder (533) reconstructs the symbols and creates sample data in a manner similar to that created by a (remote) decoder (because in the video compression techniques contemplated by the disclosed subject matter, any compression between the symbols and the coded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (534). Because decoding of the symbol stream produces bit-accurate results regardless of the decoder location (local or remote), the contents of the reference picture memory (534) are also bit-accurate between the local encoder and the remote encoder. In other words, the predicted portion that the encoder "sees" as the reference picture samples exactly the same sample values ​​that the decoder "sees" when using the prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift if synchrony cannot be maintained, e.g., due to channel error) is also used in several related technologies.

[0060] The operation of the "local" decoder (533) may be the same as the operation of a "remote" decoder, such as the video decoder (410), which is described in detail above in connection with Figure 4. However, briefly referring also to Figure 4, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (545) and parser (420) may be lossless, the entropy decoding portion of the video decoder (410), including the buffer memory (415) and parser (420), may not be fully implemented in the local decoder (533).

[0061] An observation that can be made at this point is that decoder techniques other than analysis / entropy decoding that are present in a decoder must necessarily be present in the corresponding encoder in substantially identical functional form. For this reason, the disclosed subject matter focuses on decoder operation. Descriptions of encoder techniques may be omitted, as they are the inverse of the decoder techniques that have been comprehensively described. Only in certain areas are more detailed descriptions required, and these are provided below.

[0062] In operation, in some examples, the source coder (530) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (532) codes differences between pixel blocks of the input picture and pixel blocks of reference picture(s) that may be selected as predictive reference(s) for the input picture.

[0063] The local video decoder (533) may decode coded video data of pictures that may be designated as reference pictures based on symbols generated by the source coder (530). The operation of the coding engine (532) may advantageously be a lossy process. If the coded video data can be decoded by a video decoder (not shown in FIG. 5), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (533) may replicate the decoding process that may be performed on the reference pictures by the video decoder and store the reconstructed reference pictures in a reference picture cache (534). In this way, the video encoder (503) may locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures that will be retrieved by the far-end video decoder (in the absence of transmission errors).

[0064] The predictor (535) may perform a predictive search for the coding engine (532). That is, for a new picture to be coded, the predictor (535) may search the reference picture memory (534) for sample data (as candidate reference pixel blocks) or certain metadata, such as the MVs and block shapes of reference pictures, that can serve as suitable prediction references for the new picture. The predictor (535) may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, as determined by search results obtained by the predictor (535), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (534).

[0065] The controller (550) may manage the coding operations of the source coder (530), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0066] The output of all the aforementioned functional units can undergo entropy coding in an entropy coder (545), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0067] The transmitter (540) can buffer the coded video sequence(s) produced by the entropy coder (545) and prepare them for transmission over a communication channel (560), which can be a hardware / software link to a storage device that stores the encoded video data. The transmitter (540) can merge the coded video data from the video coder (503) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0068] The controller (550) may manage the operation of the video encoder (503). During coding, the controller (550) may assign a certain coded picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures are often assigned as one of the following picture types:

[0069] An intra-picture (I-picture) may be one that can be coded and decoded without using other pictures in a sequence as a prediction source. Some video codecs allow different types of intra-pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0070] A predicted picture (P picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which uses at most one MV and reference index to predict the sample values ​​of each block.

[0071] A bidirectionally predicted picture (B picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which uses up to two MVs and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0072] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the block's respective picture. For example, blocks of an I-picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0073] The video encoder (503) may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In doing so, the video encoder (503) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy within the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0074] In one embodiment, the transmitter (540) may transmit additional data along with the encoded video. The source coder (530) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0075] Video may be captured in time sequence as multiple source pictures (video pictures). Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In one example, a certain picture being encoded / decoded, called the current picture, is partitioned into blocks. If a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture may be coded by a vector called a MV. The MV refers to a reference block in a reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0076] In some embodiments, bi-prediction techniques can be used for inter-picture prediction. According to bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which are before the decoding order of a current picture in a video (although their display orders may be past and future, respectively). A block in a current picture may be coded by a first MV that points to a first reference block in the first reference picture and a second MV that points to a second reference block in the second reference picture. A block may be predicted by a combination of the first reference block and the second reference block.

[0077] Furthermore, merge mode techniques can be used for inter-picture prediction to improve coding efficiency.

[0078] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed block-by-block. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree-decomposed into one or more coding units (CUs). For example, a 64x64 pixel CTU may be divided into one CU of 64x64 pixels, four CUs of 32x32 pixels, or 16 CUs of 16x16 pixels. In one example, each CU is analyzed to determine the CU's prediction type, such as inter prediction or intra prediction. The CU is then divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values) of 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0079] 6 shows a diagram of a video encoder (603) according to another embodiment of this disclosure. The video encoder (603) is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video picture in a sequence of video pictures and encode the processed block into a coded picture that is part of a coded video sequence. In one example, the video encoder (603) is used in place of the video encoder (303) of the example of FIG. 3.

[0080] In an HEVC example, the video encoder (603) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8x8 samples. The video encoder (603) determines whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode, e.g., using rate-distortion optimization. If the processing block is coded in intra mode, the video encoder (603) may use intra prediction techniques to encode the processing block into a coded picture, and if the processing block is coded in inter mode or bi-predictive mode, the video encoder (603) may use inter prediction techniques or bi-predictive techniques, respectively, to encode the processing block into a coded picture. In certain video coding techniques, the merge mode may be an inter-picture prediction sub-mode, in which MVs are derived from one or more MV predictors without the benefit of coded MV components outside the predictors. In certain other video coding techniques, there may be MV components applicable to the current block. In one example, the video encoder (603) includes other components, such as a mode decision module (not shown) for determining the mode of the processing block.

[0081] In the example of Figure 6, the video encoder (603) includes an inter-encoder (630), an intra-encoder (622), a residual calculator (623), a switch (626), a residual encoder (624), a general controller (621), and an entropy encoder (625), which are coupled together as shown in Figure 6.

[0082] The inter-encoder (630) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in reference pictures (e.g., blocks in previous and subsequent pictures), generate inter-prediction information (e.g., description of redundant information by inter-encoding techniques, MVs, merge mode information), and calculate inter-prediction results (e.g., predicted blocks) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that is decoded based on the encoded video information.

[0083] The intra encoder (622) is configured to receive samples of a current block (e.g., a processing block), possibly compare the block with blocks already coded in the same picture, generate quantized coefficients after transformation, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). In one example, the intra encoder (622) calculates an intra prediction result (e.g., a prediction block) based on the intra prediction information and a reference block in the same picture.

[0084] The general-purpose controller (621) is configured to determine general-purpose control data and control other components of the video encoder (603) based on the general-purpose control data. In one example, the general-purpose controller (621) determines the mode of the block and provides a control signal to the switch (626) based on the mode. For example, if the mode is intra-mode, the general-purpose controller (621) controls the switch (626) to select an intra-mode result to be used by the residual calculator (623) and controls the entropy encoder (625) to select intra-prediction information and include the intra-prediction information in the bitstream; if the mode is inter-mode, the general-purpose controller (621) controls the switch (626) to select an inter-prediction result to be used by the residual calculator (623) and controls the entropy encoder (625) to select inter-prediction information and include the inter-prediction information in the bitstream.

[0085] The residual calculator (623) is configured to calculate the difference (residual data) between the received block and a prediction result selected from the intra-encoder (622) or inter-encoder (630). The residual encoder (624) is configured to operate on the residual data to encode the residual data and generate transform coefficients. In one example, the residual encoder (624) is configured to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is configured to perform an inverse transform and generate decoded residual data. The intra-encoder (622) and inter-encoder (630) can suitably use the decoded residual data. For example, the inter-encoder (630) can generate decoded blocks based on the decoded residual data and inter-prediction information, and the intra-encoder (622) can generate decoded blocks based on the decoded residual data and intra-prediction information. In some examples, the decoded blocks are appropriately processed to generate decoded pictures, which may be buffered in a memory circuit (not shown) and used as reference pictures.

[0086] The entropy encoder (625) is configured to format a bitstream to include the encoded blocks. The entropy encoder (625) is configured to include various information in accordance with an appropriate standard, such as HEVC. In one example, the entropy encoder (625) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. Note that, according to the disclosed subject matter, when coding a block in a merged sub-mode of either an inter-mode or a bi-prediction mode, residual information is not present.

[0087] 7 shows a diagram of a video decoder (710) according to another embodiment of the present disclosure. The video decoder (710) is configured to receive coded pictures that are part of a coded video sequence and to decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (710) is used in place of the video decoder (310) of the example of FIG. 3.

[0088] In the example of Figure 7, the video decoder (710) includes an entropy decoder (771), an inter-decoder (780), a residual decoder (773), a reconstruction module (774), and an intra-decoder (772), which are coupled together as shown in Figure 7.

[0089] The entropy decoder (771) may be configured to reconstruct, from a coded picture, certain symbols that represent the syntax elements of which the coded picture is composed. Such symbols may include, for example, the mode in which the block is coded (e.g., intra mode, inter mode, bi-prediction mode, the latter two being merged or separate submodes), prediction information (e.g., intra-prediction information or inter-prediction information) that may identify certain samples or metadata used for prediction by the intra decoder (772) or inter decoder (780), respectively, residual information in the form of quantized transform coefficients, etc. In one example, if the prediction mode is an inter-prediction mode or a bi-prediction mode, the inter-prediction information is provided to the inter decoder (780), and if the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra decoder (772). The residual information may undergo inverse quantization and be provided to the residual decoder (773).

[0090] The inter decoder (780) is configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0091] The intra decoder (772) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0092] The residual decoder (773) is configured to perform inverse quantization to extract dequantized transform coefficients and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require certain control information (to include quantizer parameters (QP)), which may be provided by the entropy decoder (771) (data path not shown as this may only be low volume control information).

[0093] The reconstruction module (774) is configured to combine, in the spatial domain, the residual as output by the residual decoder (773) and the prediction result (possibly as output by an inter- or intra-prediction module) to form a reconstructed block that may be part of a reconstructed picture, which may be part of a reconstructed video. It should be noted that other appropriate operations, such as a deblocking operation, may be performed to improve visual quality.

[0094] It should be noted that the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using any suitable technology. In one embodiment, the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using one or more processors executing software instructions.

[0095] II. Block Partition FIG. 8 illustrates an example block partition according to some embodiments of the present disclosure. In one embodiment, the example block partition of FIG. 8 may be used in VP9, ​​which is proposed by the Alliance for Open Media (AOMedia). As shown in FIG. 8, a four-way partition tree may be used, starting from a 64x64 level and going up to a 4x4 level with some additional restrictions for 8x8 blocks. Note that a partition designated as R may be referred to as a recursive partition; that is, the same partition tree is repeated at lower scales until the lowest 4x4 level is reached.

[0096] FIG. 9 illustrates an example block partition according to some embodiments of the present disclosure. In one embodiment, the example block partition in FIG. 9 may be used in AV1, proposed by AOMedia. As shown in FIG. 9, the partition tree can be expanded to a 10-way structure, and the largest coding block size (called a superblock in VP9 / AV1 terminology) is increased to start at 128x128. Note that the 4:1 / 1:4 rectangular partition in the first row of FIG. 9 does not exist in VP9. The partition type with three subpartitions in the second row of FIG. 9 is called a T-type partition. None of the rectangular partitions can be further subdivided. In addition to the coding block size, a coding tree depth is defined to indicate the division depth from the root node. In one embodiment, the coding tree depth of the root node, e.g., 128x128, may be set to 0. After the coding block is further divided, the coding tree depth increases by 1.

[0097] Instead of being forced to use a fixed transform unit size as in VP9, ​​luma coding blocks in AV1 are allowed to be partitioned into transform units of multiple sizes that can be represented by recursive partitioning down up to two levels. To incorporate the extended coding block partitioning in AV1, square, 2:1 / 1:2, and 4:1 / 1:4 transform sizes are supported from 4x4 to 64x64. For chroma coding blocks, only the largest possible transform unit is allowed.

[0098] In some related examples, such as HEVC, CTUs may be divided into CUs by using a quadtree structure, referred to as a coding tree, to accommodate various local characteristics. The decision of whether to code a picture region using inter-picture (temporal) or intra-picture (spatial) prediction may be made at the CU level. Each CU may be further divided into one, two, or four PUs depending on the PU partition type. The same prediction process may be applied within one PU, and related information may be transmitted to the decoder on a PU-by-PU basis. After obtaining residual blocks by applying a prediction process based on the PU partition type, the CU may be partitioned into TUs according to another quadtree structure, such as a CU coding tree. One important feature of the HEVC structure is the concept of multiple partitions, including CUs, PUs, and TUs. In HEVC, CUs or TUs can only be square-shaped, while PUs can be square or rectangular for inter-predicted blocks. In HEVC, one coding block may be further divided into four square sub-blocks, and a transform process may be performed on each sub-block, i.e., TU. Each TU can be further recursively divided into smaller TUs (e.g., using quadtree division), which can be called a residual quadtree (RQT).

[0099] At picture boundaries, HEVC employs implicit quadtree partitioning, allowing a block to continue undergoing quadtree partitioning until its size fits into the picture boundary.

[0100] In some related examples, such as VVC, a quadtree with nested multitype trees using binary and ternary segmentation structures can replace the concept of multiple partition unit types. That is, the separation of CU, PU, ​​and TU concepts is eliminated except when required for CUs with sizes too large for the maximum transform length. Therefore, these examples can support greater flexibility in CU partition shapes. In the VVC coding tree structure, CUs can have either square or rectangular shapes. CTUs can be first partitioned using a quadtree (or quadtree) structure. The quadtree leaf nodes can then be further partitioned using a multitype tree structure.

[0101] FIG. 10 illustrates an exemplary block partition for a multi-type tree partitioning mode according to some embodiments of the present disclosure. In one embodiment, the exemplary block partition of FIG. 10 can be used in VVC. As shown in FIG. 10, the multi-type tree structure has four partition types: vertical binary partition (SPLIT_BT_VER), horizontal binary partition (SPLIT_BT_HOR), vertical ternary partition (SPLIT_TT_VER), and horizontal ternary partition (SPLIT_TT_HOR). The leaf nodes of the multi-type tree are called CUs. As long as the CU is not too large relative to the maximum transform length, the multi-type tree structure is used for the prediction and transform processes without further partitioning. This means that in most cases, CUs, PUs, and TUs can have the same block size in a quadtree with a nested multi-type tree coding block structure. One exception occurs when the maximum supported transform length is smaller than the width or height of the color components of the CU.

[0102] FIG. 11 illustrates an exemplary quadtree with a nested multi-type tree coding block structure according to one embodiment of the present disclosure.

[0103] In some relevant examples, such as VVC, the maximum supported luma transform size is 64 x 64 and the maximum supported chroma transform size is 32 x 32. If the width or height of the CB is larger than the maximum transform width or height, the CB can be automatically split along the horizontal and / or vertical direction to satisfy the transform size limitation in that direction.

[0104] In some related examples, such as VTM7, the coding tree scheme can support the luma CTB and chroma CTB within one CTU to have separate block tree structures. For example, for P slices and B slices, the luma CTB and chroma CTB within one CTU share the same coding tree structure. However, for I slices, the luma CTB and chroma CTB within one CTU can have separate block tree structures. When the separate block tree mode is applied, the luma CTB is partitioned into CUs by one coding tree structure, and the chroma CTB is partitioned into chroma CUs by another coding tree structure. This means that a CU within an I slice may contain a coding block for the luma component or coding blocks for two chroma components, while a CU within a P or B slice always contains coding blocks for all three color components unless the video is monochrome.

[0105] III. Intra Prediction In some related examples, such as VP9, ​​eight directional modes are supported, corresponding to angles from 45 degrees to 207 degrees. To take advantage of more diverse spatial redundancy in directional textures, in some related examples, such as AV1, the directional intra mode is extended to angles set with finer granularity. The original eight angles are slightly modified and referred to as nominal angles, and these eight nominal angles are referred to as V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED.

[0106] FIG. 12 illustrates exemplary nominal angles according to one embodiment of the present disclosure. Each nominal angle can be associated with seven finer angles, resulting in a total of 56 possible directional angles, such as AV1. The prediction angle is expressed as the nominal intra angle plus an angle delta, which ranges from -3 to 3 times with a step size of 3 degrees. To implement the AV1 directional prediction mode in a general way, all 56 AV1 directional intra prediction angles can be implemented with a unified directional predictor that projects each pixel to a reference subpixel position and interpolates the reference subpixel with a two-tap bilinear filter.

[0107] In some related examples such as AV1, there are five omnidirectional smooth intra prediction modes: DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H. For DC prediction, the average of the left and upper neighboring samples is used as the predictor of the block to be predicted. For PAETH prediction, the upper, left, and upper left reference samples are first fetched, and then the closest value (upper + left - upper left) is set as the predictor of the pixel to be predicted.

[0108] 13 shows the locations of the top, left, and top-left samples of one pixel in the current block according to one embodiment of the present disclosure. For SMOOTH, SMOOTH_V, and SMOOTH_H modes, the block is predicted using quadratic interpolation in the vertical or horizontal direction, or an average in both directions.

[0109] For the chroma components, in addition to the 56 directional and 5 omnidirectional modes, a chroma-only intra prediction mode, which can be called chroma from luma (CfL) mode, models chroma pixels as a linear function of the corresponding reconstructed luma pixels. CfL prediction can be expressed as: CfL(α)=α×L AC +DC formula (1) where L AC represents the AC contribution of the luma component, α represents a parameter of the linear model, and DC represents the DC contribution of the chroma component. In one example, the reconstructed luma pixels are subsampled to the chroma resolution and then the mean value is subtracted to form the AC contribution. To approximate the chroma AC components from the AC contribution, instead of requiring the decoder to calculate scaling parameters, as in some related examples, the AC1 CfL mode determines the parameter α based on the original chroma pixels and signals them in the bitstream. This reduces decoder complexity and results in more accurate predictions. Regarding the DC contribution of the chroma components, it is calculated using an intra-DC mode, which is sufficient for most chroma content and has a mature, fast implementation.

[0110] For signaling chroma intra prediction modes, eight nominal directional modes, five omnidirectional modes, and a CfL mode are first signaled. The context for signaling these modes depends on the corresponding luma mode of the top-left position of the current block. Next, if the current chroma mode is a directional mode, one additional flag is signaled to indicate the delta angle relative to the nominal angle.

[0111] Screen content video coding has become increasingly important in various applications, such as desktop sharing, video conferencing, and distance learning. Compared with natural camera-captured content, screen content generally has different characteristics, such as sharp edges. Conventional directional intra-prediction modes, such as the directional intra-prediction mode described above, require interpolation operations (e.g., 2-tap bilinear interpolation, 4-tap cubic interpolation) to generate predicted sample values ​​at fractional sample positions. The interpolation operations inevitably smooth the sharp edges and generate high frequencies in the residual blocks, which are expensive to encode.

[0112] To preserve sharp edges in intra prediction, a nearest neighbor (NN) interpolation method can be applied instead of applying an interpolation operation. Two options are described below. In the first alternative, which is an implicit pixel-based method, both the encoder and the decoder can decide whether to perform NN interpolation based on the predicted pixels. In the second alternative, the encoder performs a rate-distortion search at the block level and explicitly signals a flag to the decoder to indicate when to use NN interpolation.

[0113] FIG. 14 illustrates an exemplary bilinear interpolation for deriving a predicted sample at a fractional position according to one embodiment of the present disclosure. NN interpolation essentially selects one of the reference samples along the prediction direction instead of using a weighted sum of multiple reference samples. For example, in FIG. 14, a bilinear interpolation filter is used to derive a predicted value of sample C using two reference samples A and B. In bilinear interpolation, the predicted sample value is calculated as (A*b+B*a) / (a+b). With NN interpolation, the predicted sample value is derived as (a>b)?B:A.

[0114] Figure 15 illustrates an example of multi-line intra prediction using four reference lines adjacent to a coding block unit according to one embodiment of the present disclosure. For multi-line intra prediction, the encoder determines and signals which reference lines are used to generate the intra predictor. The reference line index is signaled before the intra prediction mode, and if a non-zero reference line index is signaled, only the most probable mode is allowed. Figure 15 illustrates an example of four reference lines, each consisting of six segments, i.e., segments A to F, and a top-left reference sample. Furthermore, segments A and F are padded with the nearest samples from segments B and E, respectively.

[0115] IV. Directional Intra Prediction Without Interpolation In some related examples, such as AV1, there are multiple delta angles (e.g., 7) for each directional nominal mode, and all delta angles are signaled and analyzed without regard to the direction of adjacent nominal modes, which is not optimal.

[0116] This disclosure includes a method for interpolation-free directional intra-prediction.

[0117] In this disclosure, if one directional intra-prediction mode is close to another directional intra-prediction mode, this means that the absolute difference in prediction angle between these two modes is within a given threshold T. In one example, T is set to 1 or 2.

[0118] 16 shows an example angle of intra-prediction direction according to one embodiment of the present disclosure, where α is the prediction angle, the solid arrow indicates the prediction direction, and the tangent value of the prediction angle is tan(α)=y / x.

[0119] According to an aspect of the present disclosure, for each sample of a current block to be predicted, given one of a plurality of intra-prediction directions, one sample from one of a plurality of reference lines is selected as a prediction sample, and the selected prediction sample is located at an integer sample position in one of the plurality of reference lines.

[0120] In one embodiment, the number of reference lines is less than a threshold. For example, up to N reference lines are used for intra prediction of the current block. Example values ​​of N include, but are not limited to, 2, 3, 4, 5, 6, 7, and 8.

[0121] In one embodiment, the tangent values ​​of the prediction angles associated with the multiple intra-prediction directions include ±N or ±1 / N, where N is an integer and example values ​​of N are 1, 2, 3, 4, 5, 6, 7, and 8.

[0122] According to some embodiments of the present disclosure, for one of a plurality of reference lines having a reference line index m (m may be 0, 1, 2, ..., and N-1, as shown in Figure 15), the reference samples in the reference line having index m may be used only with a subset of the plurality of intra-prediction directions. In some embodiments, one or more reference lines may be used only with a subset of the plurality of intra-prediction directions. In different embodiments, the subsets of the plurality of intra-prediction directions for one or more reference lines may be different, overlapping, or the same. The tangent values ​​of the prediction angles associated with the subsets of the plurality of intra-prediction directions are ±(m+1) and ±1 / (m+1).

[0123] FIG. 17 illustrates an example predicted angle according to some embodiments of the present disclosure.

[0124] In some embodiments, different reference lines may be associated with different intra-prediction directions. For example, a solid line may indicate an intra-prediction direction used with reference line 0 of FIG. 15 to perform intra-prediction. The solid line includes three diagonal directions (tangent values ​​of ±1), a horizontal direction (tangent value of 0), and a vertical direction (tangent value of ∞). A dashed line may indicate an intra-prediction direction used with reference line 1 of FIG. 15 to perform intra-prediction. The dashed line includes four prediction directions (tangent values ​​of ±½ and ±2). A dotted line may indicate an intra-prediction direction used with reference line 2 of FIG. 15 to perform intra-prediction. The dotted line includes four prediction directions (tangent values ​​of ±⅓ and ±3). A dashed and dotted line may indicate an intra-prediction direction used with reference line 3 of FIG. 15 to perform intra-prediction. The dashed and dotted lines include four prediction directions (tangent values ​​of ±¼ and ±4).

[0125] In some embodiments, different reference lines may be associated with different subsets of intra-prediction directions. Different subsets of intra-prediction directions associated with a given reference line may overlap, such as by sharing the same intra-prediction direction. For example, a solid line may indicate an intra-prediction direction used with reference line 0 to perform intra-prediction. The solid line includes three diagonal directions (tangent values ​​±1), a horizontal direction (tangent value 0), and a vertical direction (tangent value ∞). A dashed line may indicate an intra-prediction direction used with reference lines 0 and / or 1 to perform intra-prediction. The dashed line includes four prediction directions (tangent values ​​±1 / 2 and ±2). A dotted line may indicate an intra-prediction direction used with reference lines 0, 1, and / or 2 to perform intra-prediction. The dotted line includes four prediction directions (tangent values ​​±1 / 3 and ±3). The dashed and dotted lines may indicate intra prediction directions used with reference lines 0, 1, 2, and / or 3 to perform intra prediction. The dashed and dotted lines include four prediction directions (tangent values ​​±1 / 4 and ±4).

[0126] In one embodiment, for one of multiple reference lines having a reference line index m (m can be 0, 1, 2, ..., and N-1, as shown in Figure 15), the reference sample in the reference line with index m may be used only in a subset of multiple intra-prediction directions. The tangent values ​​of the prediction angle associated with the subset of multiple intra-prediction directions are ±(m+1) and ±1 / (m+1). When one of the prediction angles points to a fractional sample position within a given reference line index, the sample at the nearest integer position can be used as the reference sample.

[0127] According to aspects of the present disclosure, when performing intra prediction at a given intra prediction angle, predicted samples for different rows of pixels of the current block may be from different reference lines of the current block. For example, which reference line is used for intra prediction may differ for one or more rows of pixels of the current block.

[0128] In some embodiments, when performing intra prediction, for a prediction angle with a tangent value of ±m or ±1 / m, the predicted sample for the pixel in the nth row may be from a reference line with a line index of (m-1)-(n%m), where % is a modulo operation.

[0129] Figure 18 illustrates an exemplary intra prediction using two reference lines according to one embodiment of the present disclosure. In Figure 18, black circles indicate reference (or prediction) samples, dashed circles indicate predicted samples, and solid lines indicate prediction directions. The prediction sample in the nth row of predicted samples may be from a reference line whose line index is (m-1)-(n%m). In this example, m=2. Therefore, the prediction samples in the even rows (rows 0, 2, 4, ...) are from reference line 1, and the prediction samples in the odd rows (rows 1, 3, 5, ...) are from reference line 0.

[0130] Figure 19 illustrates an exemplary intra prediction using three reference lines according to one embodiment of the present disclosure. In Figure 19, black circles indicate reference (or prediction) samples, dashed circles indicate predicted samples, and solid lines indicate prediction directions. The prediction sample in the nth row of predicted samples may be from a reference line whose line index is (m-1)-(n%m). In this example, m=3. Thus, the prediction samples in the first plurality of rows (rows 0, 3, 6, ...) are from reference line 2, the prediction samples in the second plurality of rows (rows 1, 4, 7, ...) are from reference line 1, and the prediction samples in the third plurality of rows (rows 2, 5, 8, ...) are from reference line 0.

[0131] According to aspects of the present disclosure, the above-described intra prediction mode may be referred to as an interpolation-free intra prediction mode and may be signaled as an alternative to a conventional intra prediction mode for performing intra prediction. It may be determined whether to use the interpolation-free intra prediction mode or the conventional intra prediction mode. For example, a flag may be signaled for a block to indicate whether a conventional intra prediction mode (e.g., mode set #0, with interpolation) or the above-described intra prediction mode (e.g., mode set #1, directional mode without interpolation) is applied.

[0132] In one embodiment, different intra-prediction mode schemes can be applied to mode set #0 (with interpolation) and mode set #1 (without interpolation).

[0133] In one embodiment, the predicted angles for mode set #1 are a subset of the predicted angles for mode set #0.

[0134] In one embodiment, one or more of vertical, horizontal, and 45 degree angles are not included in Mode Set #1.

[0135] In one embodiment, to signal directional prediction modes in mode set #1 (no interpolation), one flag is first signaled to indicate whether Most Probable Mode (MPM) is applied. If MPM is not applied, one of the remaining intra-prediction modes can be coded using a fixed-length code.

[0136] In one embodiment, for mode set #1, in addition to the non-interpolated directional modes described above, other omnidirectional modes may also be signaled, including, but not limited to, DC mode, Planar mode, SMOOTH mode, SMOOTH_H mode, SMOOTH_V mode, Paeth mode, recursive filtering mode, and / or matrix-based intra-prediction mode (MIP).

[0137] In one embodiment, when mode set #1 is selected, the reference line index is not signaled or parsed in the bitstream.

[0138] In one embodiment, when mode set #1 is selected, reference line indices are not signaled or parsed in the bitstream, but reference lines with non-zero indices may still be used for intra prediction.

[0139] In one embodiment, the above-mentioned interpolation-free intra prediction modes may only be applied to certain block locations, such as when the block is not located at the top boundary of the CTU that contains the block.

[0140] V. Flowchart 20 shows a flowchart outlining an exemplary process (2000) according to one embodiment of the present disclosure. In various embodiments, the process (2000) is performed by a processing circuit, such as a processing circuit of a terminal device (210), (220), (230), or (240), a processing circuit that performs the functions of a video encoder (303), a processing circuit that performs the functions of a video decoder (310), a processing circuit that performs the functions of a video decoder (410), a processing circuit that performs the functions of an intra-prediction module (452), a processing circuit that performs the functions of a video encoder (503), a processing circuit that performs the functions of a predictor (535), a processing circuit that performs the functions of an intra-encoder (622), or a processing circuit that performs the functions of an intra-decoder (772). In some embodiments, the process (2000) is implemented in software instructions, and thus, the processing circuit performs the process (2000) when the processing circuit executes the software instructions.

[0141] The process (2000) may generally begin at step (S2010), where the process (2000) decodes prediction information for a current block in a current picture that is part of a coded video sequence. The prediction information indicates one of multiple intra-prediction directions for the current block. The process (2000) then proceeds to step (S2020).

[0142] In step (S2020), the process (2000) determines a subset of a plurality of reference lines based on one of a plurality of intra-prediction directions indicated in the prediction information. Then, the process (2000) proceeds to step (S2030).

[0143] In step S2030, the process 2000 performs intra prediction of the current block based on the determined subset of the plurality of reference lines. The process 2000 then proceeds to step S2040.

[0144] In step S2040, the process 2000 reconstructs the current block based on the intra prediction of the current block, and then the process 2000 ends.

[0145] In one embodiment, the number of reference lines in the determined subset of the plurality of reference lines is greater than one.

[0146] In one embodiment, the intra-prediction direction associated with a first reference line of the plurality of reference lines is different from the intra-prediction direction associated with a second reference line of the plurality of reference lines.

[0147] In one embodiment, the multiple intra-prediction directions are associated with a first reference line of the multiple reference lines, and a subset of the multiple intra-prediction directions is associated with a second reference line of the multiple reference lines. For example, the first reference line may be associated with the set of intra-prediction directions, and the remaining reference lines are associated with the subset of intra-prediction directions.

[0148] In one embodiment, the process (2000) determines, for each sample of the current block, one of a subset of a plurality of reference lines based on one of a plurality of intra-prediction directions and the position of the respective sample.

[0149] In one embodiment, the prediction information includes a syntax element that indicates whether intra prediction should be performed for the current block based on multiple reference lines.

[0150] In one embodiment, the current block is not located adjacent to the top boundary of the coding tree unit that contains the current block.

[0151] In one embodiment, one of the tangent and cotangent values ​​of the prediction angle associated with one of the multiple intra-prediction directions is an integer.

[0152] In one embodiment, the process (2000) determines, for each row of samples of the current block, a reference line index of one of a subset of multiple reference lines based on a tangent value of a prediction angle associated with one of the intra-prediction directions and a row number of the respective row of samples of the current block.

[0153] VI. Computer Systems The above techniques may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 21 illustrates a computer system (2100) suitable for implementing certain embodiments of the disclosed subject matter.

[0154] Computer software may be encoded using any suitable machine code or computer language that can be subjected to mechanisms such as assembly, compilation, linking, etc. to produce code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or that can be executed via interpretation, microcode execution, etc.

[0155] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0156] 21 for computer system (2100) are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system (2100).

[0157] The computer system (2100) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0158] The input human interface devices may include one or more (only one of each shown) of a keyboard (2101), a mouse (2102), a trackpad (2103), a touchscreen (2110), a data glove (not shown), a joystick (2105), a microphone (2106), a scanner (2107), and a camera (2108).

[0159] The computer system (2100) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (2110), data gloves (not shown), or joystick (2105), although some haptic feedback devices may not function as input devices), audio output devices (such as speakers (2109), headphones (not shown)), visual output devices (such as screens (2110), including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capability, each with or without haptic feedback capability, some of which may be capable of two-dimensional visual output or three-dimensional or higher output through means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown). These visual output devices (such as a screen 2110 ) may be connected to the system bus 2148 via a graphics adapter 2150 .

[0160] The computer system (2100) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2120) with media (2121) such as CDs / DVDs, thumb drives (2122), removable hard drives or solid state drives (2123), legacy magnetic media such as tape and floppy disks (not shown), and dedicated ROM / ASIC / PLD-based devices (not shown) such as security dongles.

[0161] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0162] The computer system (2100) may also include a network interface (2154) to one or more communication networks (2155). The one or more communication networks (2155) may be, for example, wireless, wired, or optical. Furthermore, the one or more communication networks (2155) may be local, wide area, metropolitan, vehicular and industrial, real-time, delay-tolerant, etc. Examples of the one or more communication networks (2155) include local area networks such as Ethernet, wireless LAN, etc., cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable TV, satellite TV, terrestrial TV, etc., vehicular or industrial networks including CANBus, etc. Certain networks typically require an external network interface adapter (e.g., a USB port on the computer system 2100) attached to a general-purpose data port or peripheral bus 2149, while others are typically integrated into the core of the computer system 2100 by attachment to a system bus (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system) as described below. Using any of these networks, the computer system 2100 can communicate with other entities. Such communications may be unidirectional and receive-only (e.g., broadcast TV), unidirectional and transmit-only (e.g., CANbus to certain CANbus devices), or bidirectional, for example, to other computer systems using local or wide-area digital networks. Certain protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0163] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (2140) of the computer system (2100).

[0164] The cores (2140) may include one or more central processing units (CPUs) (2141), graphics processing units (GPUs) (2142), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (2143), hardware accelerators for certain tasks (2144), graphics adapters (2150), etc. These devices may be connected via a system bus (2148), along with read-only memory (ROM) (2145), random access memory (2146), and internal mass storage (2147), such as an internal, non-user-accessible hard drive or SSD. In some computer systems, the system bus (2148) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (2148) or via a peripheral bus (2149). In one example, a screen (2110) may be connected to the graphics adapter (2150). Peripheral bus architectures include PCI, USB, etc.

[0165] The CPU (2141), GPU (2142), FPGA (2143), and accelerator (2144) can execute certain instructions that, in combination, can constitute the aforementioned computer code. That computer code can be stored in ROM (2145) or RAM (2146). Transient data can also be stored in RAM (2146), while persistent data can be stored, for example, in internal mass storage (2147). Rapid storage and retrieval from any of the memory devices can be enabled through the use of cache memory, which can be closely associated with one or more of the CPU (2141), GPU (2142), mass storage (2147), ROM (2145), RAM (2146), etc.

[0166] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0167] As an example, but not by way of limitation, a computer system having the architecture (2100), and in particular the core (2140), can provide functionality as a result of processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage, as described above, as well as media associated with certain storage of the core (2140) that is non-transitory in nature, such as the core's internal mass storage (2147) or ROM (2145). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2140). The computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause the core (2140), and in particular the processor (including the CPU, GPU, FPGA, etc.) therein, to perform particular processes or portions of particular processes described herein, including defining data structures stored in RAM (2146) and modifying such data structures in accordance with the software-defined processes. Additionally, or alternatively, a computer system may provide functionality as a result of logic embodied in hardwired or otherwise circuitry (e.g., accelerator (2144)), which may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass, where appropriate, circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0168] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art may devise numerous systems and methods not explicitly shown or described herein, but which embody the principles of the present disclosure and are therefore within its spirit and scope.

[0169] Appendix A: Abbreviations ALF: Adaptive Loop Filter AMVP: Advanced Motion Vector Prediction APS: Adaptation Parameter Set ASIC: Application-Specific Integrated Circuit ATMVP: Alternative / Advanced Temporal Motion Vector Prediction AV1:AOMedia Video 1 AOMedia Video 1 AV2:AOMedia Video 2 AOMedia Video 2 BMS:Benchmark Set BV: Block Vector CANBus: Controller Area Network Bus CB: Coding Block CC-ALF: Cross-Component Adaptive Loop Filter CD:Compact Disc CDEF: Constrained Directional Enhancement Filter CPR: Current Picture Referencing CPU: Central Processing Units CRT: Cathode Ray Tube CTB: Coding Tree Blocks CTU: Coding Tree Units CU: Coding Unit DPB: Decoder Picture Buffer DPS: Decoding Parameter Set DVD: Digital Video Disc FPGA: Field Programmable Gate Areas JCCR:Joint CbCr Residual Coding Joint CbCr Residual Coding JVET: Joint Video Exploration Team GOP: Groups of Pictures GPU: Graphics Processing Units GSM: Global System for Mobile communications HDR: High Dynamic Range HEVC: High Efficiency Video Coding HRD: Hypothetical Reference Decoder IBC: Intra Block Copy IC: Integrated Circuit ISP: Intra Sub-Partitions JEM: Joint Exploration Model LAN: Local Area Network LCD: Liquid-Crystal Display LR: Loop Restoration Filter LTE: Long-Term Evolution MPM: Most Probable Mode MV: Motion Vector OLED: Organic Light-Emitting Diode PBs: Prediction Blocks PCI: Peripheral Component Interconnect PDPC: Position Dependent Prediction Combination PLD: Programmable Logic Device PPS: Picture Parameter Set PU: Prediction Units RAM: Random Access Memory ROM: Read-Only Memory SAO:Sample Adaptive Offset Sample Adaptive Offset SCC: Screen Content Coding SDR: Standard Dynamic Range SEI: Supplementary Enhancement Information SNR: Signal Noise Ratio SPS: Sequence Parameter Set SSD: Solid-state Drive TU: Transform Units USB: Universal Serial Bus VPS:Video Parameter Set VUI:Video Usability Information VVC:Versatile Video Coding WAIP: Wide-Angle Intra Prediction [Explanation of symbols]

[0170] 101 Samples 102 Arrow 103 Arrow 104 blocks 111 current block 112 samples 113 samples 114 samples 115 samples 116 samples 200 Communication Systems 210 Terminal Devices 220 Terminal Devices 230 Terminal Devices 240 terminal devices 250 Network 301 Video Sources 302 Video Picture Stream 303 Video Encoder 304 encoded video data 305 Streaming Server 306 Client Subsystem 307 Copy of video data 304 308 Client Subsystem 309 Copy of video data 304 310 Video Decoder 311 Output Stream 312 Display 313 Capture Subsystem 320 Electronic Devices 330 Electronic Devices 401 Channel 410 Video Decoder 412 Drawing Device 415 Buffer Memory 420 Parser 421 Symbol 430 Electronic Devices 431 Receiver 451 Scaler / Descaler Unit 452 intra prediction units 453 Motion Compensation Prediction Unit 455 Aggregator 456 Loop Filter Unit 457 Reference Picture Memory 458 Current Picture Buffer 501 Video Sources 503 Video Encoder 520 Electronic Devices 530 Source Coder 532 Coding Engine 533 decoder 534 Reference Picture Memory 535 Predictor 540 Transmitter 543 coded video sequence 545 Entropy Coder 550 Controller 560 Communication Channels 603 Video Encoder 621 General-purpose controller 622 Intra Encoder 623 Residual Calculator 624 Residual Encoder 625 Entropy Encoder 626 Switch 628 Residual Decoder 630 Interencoder 710 Video Decoder 771 Entropy Decoder 772 Intra Decoder 773 Residual Decoder 774 Reconstruction Module 780 Interdecoder 2000 processes 2100 Computer Systems 2101 Keyboard 2102 Mouse 2103 Trackpad 2105 Joystick 2106 Microphone 2107 Scanner 2108 Camera 2109 Speaker 2110 Screen 2120 CD / DVD ROM / RW 2121 CD / DVD and other media 2122 thumb drive 2123 Removable Hard Drive or Solid State Drive 2140 cores 2141 Central Processing Unit (CPU) 2142 Graphics Processing Unit (GPU) 2143 Field Programmable Gate Area (FPGA) 2144 Accelerator 2145 Read-Only Memory (ROM) 2146 Random Access Memory (RAM) 2147 Internal Mass Storage 2148 System Bus 2149 Peripheral bus 2150 Graphics Adapter 2154 Network Interface 2155 Communication Network

Claims

1. 1. A method of video decoding in a decoder, comprising: decoding prediction information for a current block in a current picture that is part of a coded video sequence, the prediction information indicating one of a plurality of intra-prediction directions for the current block; determining which of a plurality of reference lines is included in the subset of the plurality of reference lines based on a predetermined relationship between (i) a reference line index of each of the subset of the plurality of reference lines and (ii) a ratio of a length of an opposite side to a length of an adjacent side of a prediction angle associated with one of the plurality of intra-prediction directions indicated in the prediction information; performing intra prediction of the current block based on the subset of the plurality of reference lines; reconstructing the current block based on the intra prediction of the current block.

2. The method of claim 1 , wherein the number of reference lines in the subset of the plurality of reference lines is greater than one.

3. 2. The method of claim 1, wherein the intra-prediction direction associated with a first reference line of the plurality of reference lines is different from the intra-prediction direction associated with a second reference line of the plurality of reference lines.

4. 2. The method of claim 1, wherein the plurality of intra-prediction directions are associated with a first reference line of the plurality of reference lines, and a subset of the plurality of intra-prediction directions is associated with a second reference line of the plurality of reference lines.

5. 2. The method of claim 1, wherein the determining step comprises determining, for each sample of the current block, one of the subset of the plurality of reference lines based on the one of the intra-prediction directions and a position of the respective sample.

6. The method of claim 1 , wherein the prediction information includes a syntax element indicating whether the intra prediction should be performed for the current block based on the plurality of reference lines.

7. The method of claim 1 , wherein the current block is not located adjacent to an upper boundary of a coding tree unit that contains the current block.

8. The method of claim 1 , wherein the ratio of lengths is an integer.

9. 2. The method of claim 1, wherein the determining step comprises determining, for each row of samples of the current block, a reference line index of one of the subset of the plurality of reference lines based on the length ratio and a row number of each of the rows of samples of the current block.

10. 1. A method of video encoding in an encoder, comprising: determining prediction information for a current block in a current picture that is part of a video sequence, the prediction information indicating one of a plurality of intra-prediction directions for the current block; determining which of a plurality of reference lines is included in the subset of the plurality of reference lines based on a predetermined relationship between (i) a reference line index of each of the subset of the plurality of reference lines and (ii) a ratio of a length of an opposite side to a length of an adjacent side of a prediction angle associated with one of the plurality of intra-prediction directions indicated in the prediction information; performing intra prediction of the current block based on the subset of the plurality of reference lines; encoding the current block into a coded video sequence based on the intra prediction of the current block.

11. The method of claim 10 , wherein the number of reference lines in the subset of the plurality of reference lines is greater than one.

12. 11. The method of claim 10, wherein the intra-prediction direction associated with a first reference line of the plurality of reference lines is different from the intra-prediction direction associated with a second reference line of the plurality of reference lines.

13. 11. The method of claim 10, wherein the plurality of intra-prediction directions are associated with a first reference line of the plurality of reference lines, and a subset of the plurality of intra-prediction directions is associated with a second reference line of the plurality of reference lines.

14. 11. The method of claim 10, wherein the determining step comprises determining, for each sample of the current block, one of the subset of the plurality of reference lines based on one of the intra-prediction directions and a position of the respective sample.

15. The method of claim 10 , wherein the coded video sequence includes a syntax element that indicates whether the intra prediction should be performed for the current block based on the plurality of reference lines.

16. The method of claim 10 , wherein the current block is not located adjacent to an upper boundary of a coding tree unit that contains the current block.

17. The method of claim 10, wherein the ratio of the lengths is an integer.

18. 11. The method of claim 10, wherein the determining step comprises determining, for each row of samples of the current block, a reference line index of one of the subset of the plurality of reference lines based on the length ratio and a row number of each of the rows of samples of the current block.

19. 1. A method for processing visual media data, the method comprising: converting between a visual media file and a bitstream of visual media data, said converting comprising: determining which of a plurality of reference lines is included in the subset of the plurality of reference lines based on a predetermined relationship between (i) a reference line index of each of the subset of the plurality of reference lines and (ii) a ratio of a length of an opposite side to a length of an adjacent side of a prediction angle associated with one of a plurality of intra-prediction directions indicated by a syntax element; Intra prediction of the current block is performed based on the subset of the plurality of reference lines; the current block is reconstructed based on the intra prediction of the current block; the bitstream includes a syntax element indicating one of a plurality of intra-prediction directions for a current block in a current picture. method.

20. The method of claim 19 , wherein the number of reference lines in the subset of the plurality of reference lines is greater than one.

Images