Method, apparatus, and computer program for entropy coding for intra-prediction modes

Abstract

To provide methods, apparatus, and computer readable storage medium for entropy coding for intra prediction modes in video encoding and / or decoding.SOLUTION: A method includes the steps of: receiving a coded video bitstream for a block; deriving at least one context for a set index and a mode index for the block based on at least one of information of the block and mode information of at least one neighboring block of the block; extracting, from the coded video bitstream, the set index and the mode index according to the at least one context; and determining an intra prediction mode for the block based on the set index and the mode index.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] This application is based on and claims the benefit of priority to U.S. Provisional Application No. 63 / 219,230, filed July 7, 2021, and U.S. Application No. 17 / 573,845, filed January 12, 2022, both of which are incorporated herein by reference in their entirety.

[0002] FIELD This disclosure relates to video encoding and / or decoding techniques, and more particularly to improved entropy coding design and signaling for intra-prediction modes. [Background technology]

[0003] The background discussion provided here is intended to provide a general overview of the context for the disclosure. To the extent described in this background section, the work of the named inventors, and aspects of the disclosure that may not otherwise qualify as prior art at the time of the filing of this application, are not admitted, explicitly or implicitly, as prior art to the present disclosure.

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video includes a series of pictures, each with spatial dimensions of, for example, 1920 x 1080 luminance samples and associated fully sampled or subsampled chrominance samples. The series of pictures can have a fixed or variable picture rate (also called frame rate), such as 60 pictures per second, i.e., 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, video with 1920 x 1080 pixel resolution, 60 frames per second, and 4:2:0 chroma subsampling, with 8 bits per pixel per color channel, requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 GBytes of storage space.

[0005] One goal of video encoding and decoding may be to reduce redundancy in an uncompressed input video signal through compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, sometimes by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations of these, can be used. Lossless compression refers to techniques in which an exact replica of the original signal can be reconstructed from the compressed original signal through a decoding process. Lossy compression refers to an encoding / decoding process in which the original video information is not fully preserved during coding and is not fully recoverable upon decoding. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signal is small enough to make the reconstructed signal useful for its intended application despite some information loss. For video, lossy compression is widely used in many applications. The amount of acceptable distortion depends on the application. For example, users of a particular consumer video streaming application may tolerate higher distortion than users of a movie or television broadcast application. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect different distortion tolerances, with higher tolerable distortion generally allowing for coding algorithms that result in higher losses and higher compression ratios.

[0006] Video encoders and decoders can utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transforms, quantization, and entropy coding.

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference picture. In some video codecs, a picture is spatially subdivided into blocks of samples. If all the samples of a block are coded in intra mode, the picture can be called an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and video session, or as a still image. The samples of the block after intra-prediction can then be transformed to the frequency domain, and the resulting transform coefficients can be quantized before entropy coding. Intra-prediction refers to a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and AC coefficients after the transform, the fewer bits required for a given quantization step size to represent the block after entropy coding.

[0008] Traditional intra-coding, such as that known from the MPEG-2 generation of coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to encode / decode a block based on surrounding sample data and / or metadata obtained during the encoding and / or decoding of, for example, spatially adjacent to and preceding in decoding order the block of data being intra-coded or intra-decoded. Such techniques are hereinafter referred to as "intra-prediction" techniques. Note that, at least in some cases, intra-prediction uses reference data only from the current picture being reconstructed, and not from other reference pictures.

[0009] Many different forms of intra-prediction may exist. When more than one such technique is available for a given video coding technique, the technique used may be referred to as an intra-prediction mode. One or more intra-prediction modes may be provided for a particular codec. In certain cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and intra-coding parameters for a block of video may be coded individually or collectively in a mode codeword. The codeword used for a given mode, sub-mode, and / or parameter combination may affect the coding efficiency gain through intra-prediction, as may the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain modes of intra prediction were introduced in H.264, improved in H.265, and further refined in newer coding techniques, such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). In general, in intra prediction, a predictor block may be formed using available neighboring sample values. For example, values ​​of a particular set of available neighboring samples along a particular direction and / or line may be copied into the predictor block. The reference for the direction to use may be coded into the bitstream or may itself be predicted.

[0011] Referring to FIG. 1A, a subset of nine predictor directions defined in the 33 possible predictor directions of H.265 (corresponding to the 33 angle modes of the 35 intra modes defined in H.265) is depicted in the lower right. The converging arrows (101) represent the sample being predicted. The arrow indicates that neighboring samples from that direction are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more neighboring samples to the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more neighboring samples to the lower left at an angle of 22.5 degrees from the horizontal.

[0012] 1A , a square block (104) of 4×4 samples (indicated by a bold dashed line) is depicted in the upper left. The square block (104) contains 16 samples, each labeled with “S” and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions within the block (104). Because this block is 4×4 samples in size, S44 is located in the lower right. Also shown are example reference samples following a similar numbering scheme. The reference samples are labeled R and their Y position (e.g., row index) and X position (column index) relative to the block (104). Both H.264 and H.265 use predicted samples that neighbor the block being reconstructed.

[0013] Intra-picture prediction of block 104 may begin by appropriately copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling for this block 104 indicating the prediction direction of arrow (102), i.e., that the sample is predicted from one or more predicted samples or samples 45 degrees above and to the right of horizontal. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. And sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the direction is not evenly divisible by 45 degrees, the values ​​of several reference samples may be combined, for example by interpolation, to calculate the reference sample.

[0015] As video coding technology continues to evolve, the number of possible directions is increasing. In H.264 (2003), for example, nine different directions are available for intra prediction. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions as of this disclosure. Empirical studies have been conducted to help identify the most preferred intra prediction directions, and specific techniques in entropy coding can be used to encode these most preferred directions into a small number of bits, accepting a specific bit penalty for the direction. Furthermore, these directions may themselves be predicted from neighboring directions used in intra prediction of previously decoded neighboring blocks.

[0016] FIG. 1B shows a schematic diagram (180) depicting the 65 intra-prediction directions according to JEM to illustrate the increasing number of prediction directions in the various coding techniques that have been developed.

[0017] The manner in which bits representing intra-prediction directions in a coded video bitstream are mapped to intra-prediction directions may vary from one video coding technique to another, ranging, for example, from a simple direct mapping of prediction directions to intra-prediction modes to complex adaptive schemes including codewords, most-probable modes, and similar techniques. However, in any case, there may be certain directions in the video content that are statistically less likely to occur for intra-prediction than certain other directions. Because the goal of video compression is to reduce redundancy, in a well-designed video coding technique, these less likely directions may be represented with more bits than more likely directions.

[0018] Inter-picture prediction, or inter-prediction, may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or part thereof (reference picture) may be used to predict a newly reconstructed picture or picture part (e.g., block) after being spatially shifted in a direction indicated by a motion vector (hereinafter, MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or may have three dimensions, with the third dimension (similar to the temporal dimension) being an indication of the reference picture to be used.

[0019] In some video compression techniques, a current MV applicable to a particular region of sample data can be predicted from another MV, such as from another MV related to sample data of another region that is spatially adjacent to the region being reconstructed and precedes the current MV in decoding order. Doing so relies on removing redundancy in correlated MVs, which can significantly reduce the overall amount of data required to code the MV and thereby increase compression efficiency. MV prediction can work effectively because, for example, when coding an input video signal derived from a camera (known as natural video), there is a statistical likelihood that regions larger than the region to which a single MV is applicable move in a similar direction within the video sequence and therefore can potentially be predicted using similar motion vectors derived from MVs of neighboring regions. This results in the actual MV for a given region being similar or the same as the MV predicted from surrounding MVs. Such an MV can then be represented, after entropy coding, with fewer bits than would be used if the MV were coded directly rather than predicted from one or more neighboring MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from an original signal (i.e., sample stream). In other cases, MV prediction itself may be lossy, for example, due to rounding errors in computing a predictor from some surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, “High Efficiency Video Coding”, December 2016). Among the many MV prediction mechanisms specified by H.265, a technique called “spatial merging” will be described below.

[0021] Specifically, referring to Figure 2, a current block (201) has samples that the encoder found during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of coding the MV directly, the MV can be derived from metadata associated with one or more reference pictures, such as from the immediately preceding reference picture (in decoding order), using the MV associated with any one of five surrounding samples, denoted A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use predictors from the same reference picture as neighboring blocks. Summary of the Invention

[0022] This disclosure describes various embodiments of methods, apparatus, and computer-readable storage media for encoding and / or decoding video.

[0023] According to one aspect, an embodiment of the present disclosure provides a method for entropy coding for intra-prediction modes in video decoding. The method includes receiving, by an apparatus, an encoded video bitstream for a block. The apparatus includes a memory for storing instructions and a processor in communication with the memory. The method also includes deriving, by the apparatus, at least one context for a set index and a mode index for the block based on at least one of information about the block and mode information of at least one neighboring block of the block, where the set index indicates an intra-mode set from a plurality of intra-mode sets and the mode index indicates an intra-prediction mode from the intra-mode set; extracting, by the apparatus, the set index and the mode index from the encoded video bitstream according to the at least one context; and determining, by the apparatus, the intra-prediction mode for the block based on the set index and the mode index.

[0024] According to another aspect, an embodiment of the present disclosure provides a method for entropy coding for intra-prediction modes in video coding. The method includes receiving, by an apparatus, a block to be encoded into a coded video bitstream. The apparatus includes a memory for storing instructions and a processor in communication with the memory. The method also includes deriving, by the apparatus, at least one context for a set index and a mode index for the block based on at least one of information about the block and mode information of at least one neighboring block of the block, where the set index indicates an intra-mode set from a plurality of intra-mode sets and the mode index indicates an intra-prediction mode from the intra-mode set; and encoding, by the apparatus, the set index and the mode index into the coded video bitstream according to the at least one context.

[0025] According to another aspect, an embodiment of the present disclosure provides an apparatus for encoding and / or decoding video, the apparatus including a memory storing instructions and a processor in communication with the memory, the processor configured, when executing the instructions, to cause the apparatus to perform the above-described method for encoding and / or decoding video.

[0026] In another aspect, an embodiment of the present disclosure provides a non-transitory computer-readable medium having stored thereon instructions that, when executed by a computer for decoding and / or encoding video, cause the computer to perform the above-described method for decoding and / or encoding video.

[0027] These and other aspects and their implementations are described in more detail in the drawings, specification, and claims. [Brief explanation of the drawings]

[0028] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1A] 1 shows a schematic diagram of an exemplary subset of intra-prediction direction modes. [Figure 1B] 1 shows a diagram of exemplary intra-prediction directions. [Figure 2] 1 shows a schematic diagram of spatial merging candidates for motion vector prediction in an example of a current block and its surroundings. [Figure 3] 1 shows a simplified block diagram schematic of a communication system (300) according to an example embodiment. [Figure 4] 4 shows a simplified block diagram of a communication system (400) according to an example embodiment. [Figure 5] 1 shows a schematic diagram of a simplified block diagram of a video decoder according to an example embodiment; [Figure 6] 1 shows a schematic diagram of a simplified block diagram of a video encoder according to an example embodiment; [Figure 7] 10 shows a block diagram of a video encoder according to another example embodiment. [Figure 8] 10 shows a block diagram of a video decoder according to another example embodiment. [Figure 9] 1 illustrates directional intra-prediction modes according to an example embodiment of this disclosure. [Figure 10] 1 illustrates a non-directional intra-prediction mode according to an example embodiment of this disclosure. [Figure 11] 1 illustrates a recursive intra-prediction mode according to an example embodiment of this disclosure. [Figure 12] 1 illustrates a flowchart of a method according to an example embodiment of the present disclosure. [Figure 13] 10 illustrates a flowchart of another method according to an example embodiment of the present disclosure. [Figure 14] 1 shows a schematic diagram of a computer system according to an example embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0029] The present invention will now be described in detail hereinafter with reference to the accompanying drawings, which form a part hereof and which show, by way of illustration, specific embodiments. It should be noted, however, that the present invention may be embodied in many different forms and, therefore, the subject matter as directed or claimed should not be construed as being limited to any of the embodiments set forth below. It should also be noted that the present invention may be embodied as a method, apparatus, component, or system. Thus, embodiments of the present invention may take the form of, for example, hardware, software, firmware, or any combination thereof.

[0030] Throughout the specification and claims, terms may have a meaning suggested or implied by context beyond that expressly stated. The phrases "in one embodiment" or "in some embodiments" used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" used herein do not necessarily refer to different implementations. It is intended, for example, that claimed subject matter include combinations of example embodiments / implementations, in whole or in part.

[0031] Generally, terms can be understood, at least in part, from their usage in context. For example, terms such as "and," "or," or "and / or" as used herein can include a variety of meanings that may depend, at least in part, on the context in which such terms are used. Typically, when "or" is used to relate a list, e.g., A, B, or C, it is intended to mean A, B, and C, which are used herein in an inclusive sense, and A, B, or C, which are used herein in an exclusive sense. Also, as used herein, the terms "one or more" or "at least one" can be used to describe any feature, structure, or characteristic in a singular sense or to describe a combination of features, structures, or characteristics in a plural sense, depending, at least in part, on the context. Similarly, terms such as "a," "an," or "the" can be understood to convey a singular sense or a plural sense, depending, at least in part, on the context. Additionally, the terms "based on" or "determined by" are not necessarily intended to convey an exclusive set of factors, but instead may be understood to allow for the existence of additional factors not necessarily explicitly stated, again depending at least in part on the context.

[0032] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to one embodiment of the present disclosure. The communication system (300) includes multiple terminal devices capable of communicating with each other, e.g., via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) may perform unidirectional data transmission. For example, the terminal device (310) may encode video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) may receive the coded video data from the network (350), decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. One-way data transmission can be implemented in media service provision applications and the like.

[0033] In another example, the communication system 300 includes a second pair of terminal devices 330 and 340 for bidirectional transmission of encoded video data, such as may be implemented in a videoconferencing application. In one example, each of the terminal devices 330 and 340 may encode video data (e.g., a stream of video pictures captured by that terminal device) for transmission to the other of the terminal devices 330 and 340 over the network 350. Each of the terminal devices 330 and 340 may also receive encoded video data transmitted by the other of the terminal devices 330 and 340, decode the encoded video data to recover the video pictures, and display the video pictures on an accessible display device in accordance with the recovered video data.

[0034] In the example of FIG. 3 , terminal devices 310, 320, 330, and 340 may be implemented as servers, personal computers, and smartphones, although applicability of the underlying principles of the present disclosure may not be so limited. Embodiments of the present disclosure may also be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated videoconferencing equipment, and / or the like. Network 350 represents any number or type of network that conveys encoded video data between terminal devices 310, 320, 330, and 340, including, for example, wired and / or wireless communication networks. Communication network 350 may exchange data over circuit-switched channels, packet-switched channels, and / or other compatible channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network 350 may not be important to the operation of the present disclosure unless explicitly described herein.

[0035] 4 illustrates the arrangement of a video encoder and a video decoder in a video streaming environment as an example of an application of the disclosed subject matter, which may be equally applicable to other video applications, including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, and storage of compressed video on digital media, including CDs, DVDs, memory sticks, and the like.

[0036] A video streaming system may include a video capture subsystem (413), which may include a video source (401), such as a digital camera, that produces a stream of uncompressed video pictures or images (402). In one example, the stream of video pictures (402) comprises samples recorded by the digital camera of the video source (401). The stream of video pictures (402), depicted as a bold line to emphasize its high data volume compared to the encoded video data (404) (or encoded video bitstream), may be processed by an electronics device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data 404 (or encoded video bitstream 404), depicted as a thin line to emphasize its low data volume compared to the uncompressed video picture stream 402, can be stored on the streaming server 405 for later use or provided directly to a downstream video device (not shown). One or more streaming client subsystems, such as the client subsystems 406 and 408 of FIG. 4, can access the streaming server 405 to retrieve copies 407 and 409 of the encoded video data 404. The client subsystem 406 can include a video decoder 410, for example, within the electronics device 430. The video decoder 410 can decode the incoming copy of the encoded video data 407 and produce an outgoing video picture stream 411, which is uncompressed and can be rendered on a display 412 (e.g., a display screen) or other rendering device (not shown). The video decoder (410) may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data 404, 407, and 409 (e.g., video bitstreams) may be encoded according to a particular video encoding / compression standard. Examples of such standards include ITU-T Recommendation H.265. In one example, one video encoding standard under development is informally known as Versatile Video Coding (VVC). The subject matter disclosed herein may be used in the context of VVC and other video coding standards.

[0037] It should be noted that electronics devices 420 and 430 may include other components (not shown). For example, electronics device 420 may include a video decoder (not shown), and electronics device 430 may also include a video encoder (not shown).

[0038] 5 shows a block diagram of a video decoder (510) according to any of the following embodiments of the present disclosure. The video decoder (510) may be included in an electronics device (530). The electronics device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) in the example of FIG. 4.

[0039] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In the same or another embodiment, one coded video sequence may be decoded at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. Each video sequence may relate to multiple video frames or images. The coded video sequences may be received from a channel (501), which may be a storage device that stores the coded video data or a hardware / software link to a streaming source that transmits the coded video data. The receiver (531) may receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective processing circuits (not shown). The receiver (531) may separate the coded video sequences from other data. To combat network jitter, a buffer memory (515) may be disposed between the receiver (531) and the entropy decoder / parser 520 (hereinafter, "parser (520)"). In certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, it may be external and separate from the video decoder (510) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (510), for example, to combat network jitter, and there may be an additional buffer memory (515) internal to the video decoder (510), for example, to handle playback timing. When the receiver (531) is receiving data from a store-and-forward device with sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory (515) may not be required or may be small. For use over a best-effort packet network, such as the Internet, a sufficiently sized buffer memory (515) may be required, and its size may be relatively large.Such buffer memory may be implemented with an adaptable size and may be implemented, at least in part, in an operating system or similar element (not shown) external to the video decoder (510).

[0040] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (510) and possibly information for controlling a rendering device, such as a display (512) (e.g., a display screen), which may or may not be an integral part of the electronics device (530) but may be coupled to the electronics device (530), as shown in FIG. 5. The control information for the rendering device(s) may be in the form of a Supplementary Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the coded video sequence received by the parser (520). Entropy coding of the coded video sequence may be according to a video coding technique or standard and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the subgroup. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (520) may also extract information from the coded video sequence information, such as transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, motion vectors, etc.

[0041] The parser (520) may perform an entropy decoding / parsing process on the video sequence received from the buffer memory (515) to produce symbols (521).

[0042] The reconstruction of the symbols (521) may involve several different processing or functional units, depending on the type of coded video picture or portion thereof and other factors (e.g., inter-picture vs. intra-picture, inter-block vs. intra-block, etc.). The units involved, and how they participate, may be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following processing or functional units is not shown for simplicity.

[0043] Beyond the functional blocks already described, the video decoder (510) may be conceptually subdivided into a number of functional units, as described below. In practical implementations operating within commercial constraints, many of these functional units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of clearly describing the various functions of the disclosed subject matter, the following disclosure will adopt a conceptual division into functional units.

[0044] The first unit may include a scalar / inverse transform unit (551), which may receive quantized transform coefficients as symbol(s) (521) from the parser (520), along with control information including information indicating what type of inverse transform to use, block size, quantization coefficients / parameters, quantization scaling matrices, and the like. The scalar / inverse transform unit (551) may output blocks of sample values, which may be input to an aggregator (555).

[0045] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may generate blocks of the same size and shape as the block being reconstructed using information from surrounding blocks that have already been reconstructed and stored in the current picture buffer (558). The current picture buffer (558), for example, buffers partially reconstructed and / or fully reconstructed current pictures. In some implementations, the aggregator (555) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0046] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to a block that may be inter-coded and motion-compensated. In such cases, the motion-compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for inter-picture prediction. After motion-compensating the fetched samples according to the symbols (521) related to the block, these samples may be added by the aggregator (555) to the output of the scalar / inverse transform unit (551) (the output of the unit (551) may be referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches prediction samples may be controlled by a motion vector and are available to the motion-compensated prediction unit (553) in the form of symbols (521), which may have, for example, X and Y components (shift), and a reference picture component (time). Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, and may be associated with motion vector prediction mechanisms, etc.

[0047] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but may also be responsive to meta-information obtained during decoding of previous portions of the coded picture or coded video sequence (in decoding order), as well as to previously reconstructed and loop-filtered sample values. As described in more detail below, several types of loop filters may be included as part of the loop filter unit (556) in various orders.

[0048] The output of the loop filter unit (556) can be a sample stream that can be output to a rendering device (512), which can also be stored in a reference picture memory (557) for use in future inter-picture prediction.

[0049] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future inter-picture prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before beginning reconstruction of the next coded picture.

[0050] The video decoder (510) may perform decoding according to a given video compression technique adopted in a standard, such as ITU-T Recommendation H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense of adhering to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, a profile may select specific tools from all tools available in the video compression technique or standard, such that those tools are the only tools available for use under that profile. To be standard-compliant, the complexity of the coded video sequence may be constrained to a range specified by the level of the video compression technique or standard. In some cases, the level constrains the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further constrained, in some cases, through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0051] In some example embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of one or more encoded video sequences. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0052] 6 shows a block diagram of a video encoder (603) according to an example embodiment of the present disclosure. The video encoder (603) may be included in an electronics device (620). The electronics device (620) may further include a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) may be used in place of the video encoder (403) in the example of FIG. 4.

[0053] The video encoder (603) may receive video samples from a video source (601) (not part of the electronics device (620) in the example of FIG. 6) that may capture the video image(s) to be encoded by the encoder (603). In another example, the video source (601) may be implemented as part of the electronics device (620).

[0054] The video source (601) may provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCB, RGB, XYZ, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service provision system, the video source (601) may be a storage device capable of storing pre-prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures or images that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may have one or more samples, depending on the sampling structure, color space, and the like used. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

[0055] According to some example embodiments, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under other time constraints required by the application. Enforcing an appropriate encoding rate constitutes one function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units, as described below. This coupling is not shown for simplicity. Parameters set by the controller (650) may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group-of-picture (GOP) layout, maximum motion vector search range, and the like. The controller (650) can be configured with other suitable functions associated with the video encoder (603) that are optimized for a particular system design.

[0056] In some example embodiments, the video encoder (603) may be configured to operate in a coding loop. As an overly simplified explanation, in one example, the coding loop may include a source coder (630) (e.g., responsible for creating symbols, e.g., a symbol stream, based on an input picture to be coded and one or more reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The embedded decoder (633) processes the video stream coded by the source coder (630) without entropy coding (when the compression between symbols and coded video bitstreams in entropy coding is lossless in the video compression techniques considered in the disclosed subject matter), but the decoder (633) reconstructs the symbols to generate sample data in the same manner as a (remote) decoder would. The reconstructed sample stream (sample data) is input to a reference picture memory (634). Because decoding the symbol stream yields bit-accurate results that are independent of the decoder location (local or remote), the contents in the reference picture memory (634) are also bit-accurate between the local and remote encoders. In other words, the prediction portion of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder "sees" when using prediction during decoding. This fundamental principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, for example, due to channel errors) is used to improve coding quality.

[0057] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510), which has already been described in detail above in connection with Figure 5. However, briefly referring also to Figure 5, because symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), may not be fully implemented in the local decoder (633) within the encoder.

[0058] It can be noted at this point that decoder technology, with the exception of parsing / entropy decoding, that is present only in the decoder, may necessarily also be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosure will sometimes focus on decoder operations, as they apply to the decoding portion of the encoder. A description of the encoder technology can therefore be omitted, as it is the reverse of the decoder technology, which is described in detail below. Only in certain areas or aspects will a more detailed description of the encoder be provided below.

[0059] In operation, in some implementations, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture relative to one or more previously coded pictures from a video sequence designated as “reference pictures.” Thus, the coding engine (632) codes differences (or residuals) in color channels between pixel blocks of the input picture and pixel blocks of one or more reference pictures that can be selected as prediction reference(s) for the input picture.

[0060] The local video decoder (633) may decode coded video data for pictures that may be designated as reference pictures based on symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the coded video data is decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a replica of the source video sequence, with some error. The local video decoder (633) may replicate the decoding process that may be performed by a video decoder on the reference pictures and cause the reconstructed reference pictures to be stored in a reference picture cache (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures that have content in common with reconstructed reference pictures that will be obtained by a far-end (remote) video decoder.

[0061] The predictor (635) may perform a predictive search for the coding engine (632). That is, for a new picture to be encoded, the predictor (636) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors or block shapes, that can serve as suitable prediction references for the new picture. The predictor (635) may operate on a pixel block-by-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0062] The controller (650) may manage the coding process of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0063] The outputs of all the aforementioned functional units may be subjected to entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by lossless compression of the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0064] A transmitter (640) may buffer the coded video sequence(s) produced by the entropy coder (645) and prepare them for transmission over a communication channel (660), which may be a hardware or software link to a storage device that stores the coded video data. The transmitter (640) may merge the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0065] The controller (650) may manage the operation of the video encoder (603). In coding, the controller (650) may assign each coded picture a particular coded picture type, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned one of the following picture types:

[0066] An intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow several different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of these variants of I-pictures and their respective uses and characteristics.

[0067] A predicted picture (P picture) may be encoded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0068] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0069] A source picture is generally spatially subdivided into multiple sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to their respective pictures. For example, blocks of an I-picture may be coded non-predictively, or they may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P-picture may be coded non-predictively or via spatial or temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be coded non-predictively or via spatial or temporal prediction with reference to one or two previously coded reference pictures. Source pictures or intermediate processed pictures may also be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same method, as will be described in more detail below.

[0070] The video encoder (603) may perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Recommendation H.265. In its operation, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. The encoded video data may therefore conform to a syntax specified by the video coding technique or standard being used.

[0071] In some example embodiments, the transmitter (640) may transmit additional data along with the coded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0072] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits temporal or other correlation between pictures. For example, a particular picture being encoded / decoded, called the current picture, may be divided into multiple blocks. When a block in the current picture resembles a reference block in a previously coded and still buffered reference picture in the video, it may be coded by a vector called a motion vector. A motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0073] In some example embodiments, bi-prediction techniques can be used for inter-picture prediction. Such bi-prediction techniques use two reference pictures, such as a first reference picture and a second reference picture, both of which precede the current picture in decoding order (but may be past or future, respectively, in display order) in a video. A block in the current picture can be coded with a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block can be jointly predicted by a combination of the first and second reference blocks.

[0074] Furthermore, merge mode techniques may be used to improve coding efficiency in inter-picture prediction.

[0075] According to some example embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed on a block-by-block basis. For example, a picture in a sequence of video pictures is divided into multiple coding tree units (CTUs) for compression, and the CTUs within a picture may have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. In general, a CTU may include three parallel coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU may be partitioned into one CU of 64x64 pixels or four CUs of 32x32 pixels. One or more of the 32x32 blocks may each be further partitioned into four CUs of 16x16 pixels. In some example embodiments, during encoding, each CU may be analyzed to determine its prediction type from among various prediction types, such as inter prediction or intra prediction. A CU may be divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations during coding (encoding / decoding) are performed in units of prediction blocks. The division of a CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. For example, a luma or chroma PB may include a matrix of sample values ​​(e.g., luma values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 samples, and the like.

[0076] 7 shows a diagram of a video encoder (703) according to another example embodiment of this disclosure. The video encoder (703) is configured to receive sample values ​​of a processing block (e.g., a predictive block) in a current video picture in a sequence of video pictures and to encode the processing block into a coded picture that is part of a coded video sequence. This example video encoder (703) may be used in place of the video encoder (403) in the example of FIG. 4.

[0077] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as 8x8 samples of a predictive block. The video encoder (703) then determines, for example, using rate-distortion optimization (RDO), whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode. If it is determined that the processing block is coded in intra mode, the video encoder (703) can encode the processing block into a coded picture using intra prediction techniques; if it is determined that the processing block is coded in inter mode or bi-predictive mode, the video encoder (703) can encode the processing block into a coded picture using inter prediction techniques or bi-predictive techniques, respectively. In some example embodiments, merge mode can be used as a sub-mode of inter-picture prediction in which a motion vector is derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In other example embodiments, there can be motion vector components applicable to the current block. Thus, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module for determining the prediction mode of a processing block.

[0078] In the example of Figure 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), coupled together as shown in the example configuration of Figure 7.

[0079] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in preceding and following pictures in display order), generate inter-prediction information (e.g., a description of redundant information according to an inter-coding technique, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a reference picture decoded based on coded video information using a decoding unit (633) embedded in the example encoder (620) of FIG. 6 (shown as residual decoder 72 in FIG. 7, described in more detail below).

[0080] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block to previously coded blocks in the same picture, generate transformed quantized coefficients, and in some cases also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). The intra encoder (722) may calculate an intra prediction result (e.g., a prediction block) based on the intra prediction information and reference blocks in the same picture.

[0081] The general controller (721) may be configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines a prediction mode for a block and provides a control signal to the switch (726) based on the prediction mode. For example, when the prediction mode is intra-mode, the general controller (721) controls the switch (726) to select an intra-mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-prediction information and include it in the bitstream. When the prediction mode for the block is inter-mode, the general controller (721) controls the switch (726) to select an inter-prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-prediction information and include it in the bitstream.

[0082] The residual calculator (723) may be configured to calculate the difference (residual data) between a received block and a prediction result of a block selected from the intra-encoder (722) or inter-encoder (730). The residual encoder (724) may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder (724) may be configured to transform the residual data from the spatial domain to the frequency domain to generate the transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various example embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be suitably used by the intra-encoder (722) and the inter-encoder (730). For example, the inter-encoder (730) can generate decoded blocks based on the decoded residual data and inter-prediction information, and the intra-encoder (722) can generate decoded blocks based on the decoded residual data and intra-prediction information. The decoded blocks are suitably processed to generate decoded pictures, which can be buffered in a memory circuit (not shown) and used as reference pictures.

[0083] The entropy encoder (725) may be configured to format a bitstream to include the encoded blocks and to perform entropy coding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. When coding blocks in a merged sub-mode of either an inter mode or a bi-prediction mode, residual information may not be present.

[0084] 8 shows a diagram of an example video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) may be used in place of the video decoder (410) in the example of FIG. 4.

[0085] In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter-decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-decoder (872), coupled together as shown in the example configuration of Figure 8.

[0086] The entropy decoder (871) may be configured to reconstruct, from the coded picture, certain symbols that represent the syntax elements that make up the coded picture. Such symbols may include, for example, prediction information (e.g., intra-prediction information or inter-prediction information) that may identify the mode in which a block is coded (e.g., intra-mode, inter-mode, bi-prediction mode, merged sub-mode, or other sub-mode), certain samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), residual information in the form of, for example, quantized transform coefficients, and the like. In one example, when the prediction mode is an inter-mode or bi-prediction mode, the inter-prediction information is provided to the inter-decoder (880), and when the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information may be subjected to inverse quantization and provided to the residual decoder (873).

[0087] The inter decoder (880) may be configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0088] The intra decoder (872) may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0089] The residual decoder (873) may be configured to perform inverse quantization to extract dequantized transform coefficients, and then process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize certain control information (including quantizer parameters (QP)) that may be provided by the entropy decoder (871) (data path not shown as this may be only low data volume control information).

[0090] The reconstruction module (874) may be configured to combine the residual output by the residual decoder (873) and the prediction results (output by the inter- or intra-prediction module, as the case may be) in the spatial domain to form reconstructed blocks that form part of a reconstructed picture as part of the reconstructed video. Note that other suitable processes, such as deblocking and the like, may also be performed to improve visual quality.

[0091] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In some example embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0092] Returning to the intra-prediction process, a predictive block is generated by predicting samples within a block (e.g., a luma or chroma prediction block, or a coding block if not further divided into prediction blocks) with neighboring samples, next-neighboring samples, or samples from one or more other lines, or a combination thereof. The residual between the actual block being coded and the predictive block may then be processed by transform and subsequent quantization. Various intra-prediction modes may be made available, and parameters related to intra-mode selection and other parameters may be signaled in the bitstream. The various intra-prediction modes may relate, for example, to one or more line positions for predicting samples, the direction along which predictive samples are selected from one or more prediction lines, and other special intra-prediction modes.

[0093] For example, the set of intra-prediction modes (interchangeably referred to as "intra modes") may include a predetermined number of directional intra-prediction modes. As described above in connection with the implementation of FIG. 1, these intra-prediction modes correspond to a predetermined number of directions along which out-of-block samples may be selected as predictions for predicted samples within a particular block. In another particular implementation, eight main directional modes may be supported and predefined, corresponding to angles from 45 degrees to 207 degrees relative to the horizontal axis.

[0094] In some other implementations of intra prediction, directional intra modes may be further expanded to a finer set of angles to further exploit the greater spatial redundancy of directional textures. For example, the above eight-angle implementation may be configured to provide eight nominal angles designated V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, as shown in FIG. 9, and a predetermined number (e.g., seven) of finer angles may be added to each nominal angle. Such expansion may make a larger total number (e.g., 56 in this example) of directional angles available for intra prediction, corresponding to the same number of predefined directional intra modes. A prediction angle may be represented by a nominal intra angle plus an angle delta. In the above specific example, there are seven finer angle directions for each nominal angle, and the angle delta may be a step size of 3 degrees multiplied by -3 to 3.

[0095] The above-directional intra-prediction is sometimes referred to as unidirectional intra-prediction and is distinct from bidirectional intra-prediction (also called intra-bi-prediction) which is described later in this disclosure.

[0096] In some implementations, instead of or in addition to the above directional intra mode, a predetermined number of non-directional intra prediction modes may also be predefined and made available. For example, five non-directional intra modes called smooth intra prediction modes may be defined. These non-directional intra prediction modes may be specifically referred to as DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H intra modes. Prediction of samples of a particular block under these example non-directional modes is shown in FIG. 10. As an example, FIG. 10 illustrates that a 4×4 block 1002 is predicted by samples from an upper adjacent line and / or a left adjacent line. A particular sample 1010 in block 1002 may correspond to a sample 1004 directly above the sample 1010 in the upper adjacent line of block 1002, a sample 1006 above and to the left of sample 1010 as the intersection of the upper and left adjacent lines, and a sample 1008 directly to the left of sample 1010 in the left adjacent line of block 1002. In an example of a DC intra prediction mode, the average of the left and above neighboring samples 1008 and 1004 may be used as a predictor for sample 1010. In an example of a PAETH intra prediction mode, the above, left, and above-left reference samples 1004, 1008, and 1006 may be fetched, and the value of these three reference samples that is closest to (above + left - above-left) may be set as the predictor for sample 1010. In an example of a SMOOTH_V intra prediction mode, sample 1010 may be predicted by quadratic interpolation in the vertical direction of the above-left neighboring sample 1006 and the left neighboring sample 1008. In an example of a SMOOTH_H intra prediction mode, sample 1010 may be predicted by quadratic interpolation in the horizontal direction of the above-left neighboring sample 1006 and the above neighboring sample 1004. In an example of a SMOOTH intra prediction mode, sample 1010 may be predicted by an average of quadratic interpolation in the vertical direction and quadratic interpolation in the horizontal direction. The above non-directional intra-mode implementation is shown merely as a non-limiting example: other adjacent line and other non-directional sample selections and ways of combining prediction samples to predict particular samples within a prediction block are also contemplated.

[0097] The encoder's selection of a particular intra-prediction mode from the above directional or non-directional modes at various coding levels (picture, slice, block, unit, etc.) may be signaled in the bitstream. In some implementations, eight exemplary nominal directional modes may be signaled first, along with five non-angle smooth modes (for a total of 13 options). Then, if the signaled mode is one of the eight nominal angle intra-modes, an index is further signaled to indicate the selected angle delta relative to the corresponding signaled nominal angle. In some other implementations, all intra-prediction modes may be indexed together for signaling (e.g., 56 directional modes plus 5 non-directional modes to generate 61 intra-prediction modes).

[0098] In some implementations, the 56 or other number of directional intra-prediction modes in this example may be implemented with a unified directional predictor that projects each sample of a block to the location of a reference subsample and interpolates the reference sample using a two-tap bilinear filter.

[0099] In some implementations, additional filter modes called FILTER INTRA modes may be designed to capture the decay of spatial correlation with edge references. In these modes, for some patches within a block, predicted samples within the block may be used as intra-prediction reference samples in addition to out-of-block samples. These modes may be predefined and made available, for example, for intra-prediction at least for luma blocks (or only luma blocks). A predetermined number (e.g., five) of filter intra modes may be predesigned, each represented by a set of n-tap filters (e.g., 7-tap filters) that reflect the correlation between a sample within a 4x2 patch and its n neighboring samples. In other words, the weighting coefficients of the n-tap filters may be position-dependent. Taking an 8x8 block, a 4x2 patch, and 7-tap filtering as examples, an 8x8 block 1102 may be divided into eight 4x2 patches as shown in Figure 11. These patches are denoted by B0, B1, B2, B3, B4, B5, B6, and B7 in Figure 11. For each patch, its seven neighbors (denoted R0-R7 in FIG. 11) may be used to predict samples in the current patch. For patch B0, all of the neighbors may already be reconstructed. However, for other patches, some of the neighbors may be within the current block and therefore not reconstructed, and the predictions of the immediate neighbors are used as a reference. For example, not all of the neighbors of patch B7 shown in FIG. 11 are reconstructed, and therefore neighbors that are part of, for example, B4, B5, and / or B6, are used instead.

[0100] In some implementations of intra prediction, one color component may be predicted using one or more other color components. The color components may be any of the components in the YCrCb, RGB, XYZ color spaces, etc. For example, prediction of a chroma component (e.g., a chroma block) from a luma component (e.g., a luma reference sample), referred to as chroma-from-luma (i.e., CfL), may be implemented. In some implementations, cross-color prediction may only be allowed from luma to chroma. For example, chroma samples within a chroma block may be modeled as linear functions of the corresponding reconstructed luma samples. CfL prediction is CfL(α)=α×L AC +DC (1) It can be implemented as:

[0101] where L AC where α represents the AC contribution of the luma component, α represents a parameter of a linear model, and DC represents the DC contribution of the chroma component. For example, the AC components are obtained for each sample of the block, while the DC component is obtained for the entire block. Specifically, the reconstructed luma samples may be subsampled to the chroma resolution, and then the average luma value (luma DC) may be subtracted from each luma value to form the luma AC contribution. The luma AC contribution is then used in the linear mode of Equation (1) to predict the AC values ​​of the chroma components. Instead of requiring the decoder to calculate scaling parameters to approximate or predict the chroma AC components from the luma AC contributions, an example CfL implementation may determine the parameter α based on the original chroma samples and signal them in the bitstream. This reduces decoder complexity and produces more accurate predictions. Regarding the DC contribution of the chroma components, it may be calculated using an intra-DC mode within the chroma components in some implementations.

[0102] In some embodiments, for a particular coding block, coding unit, predictive block, or prediction unit that is intra-coded, its intra mode needs to be signaled by one or more syntax elements in the bitstream. As mentioned above, the number of possible intra-prediction modes can be large, with 62 intra-prediction modes available: 56 directional intra-prediction modes, 5 non-directional modes, and one chroma-from-luma mode (e.g., for the chroma component only). To signal these intra-prediction modes, a first syntax element may be signaled to indicate which nominal angle mode or non-directional mode is equivalent to the nominal mode of the current block. Then, if the mode of the current block is a directional mode, a second syntax element may be signaled to indicate which delta angle is equivalent to that of the current block. In some situations during video encoding or decoding, there may be a strong correlation between the intra-prediction modes of the current block and its neighboring blocks.

[0103] In various embodiments, this correlation may be exploited to design a more efficient syntax for intra-mode coding. In some implementations, the intra-prediction modes available for a current block may be divided into multiple intra-prediction mode sets according to the intra-prediction modes of its neighboring blocks. To obtain the intra-prediction mode of the current block, first, a mode set index indicating the mode set index of the intra-prediction mode of the current block may be signaled, and then a mode index indicating the index of the intra-prediction mode within the mode set may be signaled.

[0104] Here, in various embodiments of the present disclosure, "XYZ is signaled" can refer to XYZ being coded into a coded bitstream during an encoding process, and / or after the coded bitstream is transmitted from one device to another, "XYZ is signaled" can refer to XYZ being decoded / extracted from the coded bitstream during a decoding process.

[0105] For example, in some of the implementations described above, the number of available intra-prediction modes can include 62 distinct modes, such as 56 directional intra-prediction modes (e.g., 8 nominal directions with 7 finer angles per nominal direction), 5 non-directional modes, and one chroma-from-luma mode (e.g., for the chroma component only). When an intra-mode is selected for a particular coding block, coding unit, prediction block, or prediction coding process, signaling corresponding to the selected intra-mode must be included in the bitstream. Signaling syntax(es) must be able to distinguish all 62 modes in some way. For example, these 62 modes can be signaled using a single syntax for 62 indices, each corresponding to a mode. In some other implementations, one syntax may be signaled to indicate which nominal angle mode or non-directional mode is used as the nominal mode in the current block, and another syntax may be additionally signaled to indicate which delta angle is selected for the current block if the nominal mode of the current block is a directional mode.

[0106] Because various syntaxes related to intra-coding typically occupy a large portion of the bitstream and because intra-mode selections must be signaled frequently, e.g., at various coding levels, reducing the number of bits used for intra-mode signaling is important for improving video coding efficiency. In practice, the use of various intra-prediction modes may follow certain statistical patterns, and designing intra-mode indexing and signaling syntax using such usage patterns may improve signaling efficiency. Also, in general, there may be some correlation between blocks in intra-mode selections. Such correlations can be obtained offline on a statistical basis and taken into account in the design of one or more syntaxes for signaling intra-mode selections. The goal is to generally reduce the number of bits for signaling syntax elements in the coded bitstream. For example, some general statistics may indicate that there may be a strong correlation between the optimal intra-prediction modes of a current block and its neighboring blocks. Such correlations may be exploited when designing syntax for intra-mode coding.

[0107] In various embodiments, the intra-prediction modes or mode options available for a current block being coded may be divided into multiple intra-prediction mode sets. A mode set index may be assigned to each set. Each set may include a certain number of intra-prediction modes. The manner in which the available intra-prediction modes are divided and ordered, and the manner in which the intra-prediction modes are ordered within each mode set, may be determined based on correlations between intra-prediction modes between blocks and at least in part according to the intra-prediction modes used by their neighboring blocks. The intra-prediction modes used by the neighboring blocks may be referred to as "reference intra-prediction modes" or "reference modes." An intra-prediction mode may be determined and selected for a particular unit. The selection of an intra-prediction mode may be signaled. First, a mode set index may be signaled to indicate the mode set index of the intra-prediction mode set that includes the selected intra-prediction mode. Next, a mode index (alternatively referred to as a mode position index within a set) may be signaled to indicate the index of the selected intra-prediction mode within that mode set.

[0108] The general implementation of intra-prediction mode partitioning and ordering above and the specific examples below utilize statistical effects and neighbor correlations to dynamically index these modes, allowing the design of syntax for signaling their selection within the coded video bitstream to be optimized to improve coding efficiency. For example, these implementations can help reduce the number of syntaxes for signaling and can help with more efficient context generation for entropy coding.

[0109] The various embodiments and / or implementations described in this disclosure may be used separately or combined in any order. Also, some, all, or partial or complete combinations of these embodiments and / or implementations may be embodied as part of an encoder and / or decoder and may be implemented in hardware and / or software. For example, they may be hard-coded into dedicated processing circuitry (e.g., one or more integrated circuits). In another example, they may be implemented by one or more processors executing a program stored on a non-transitory computer-readable medium.

[0110] There may be some challenges / issues associated with intra-mode coding. For example, when dividing intra-prediction modes into multiple different mode sets, the probability of selecting an intra-prediction mode from different mode sets may be correlated with the intra-prediction modes of neighboring blocks because there is a strong correlation between samples in a current block and its neighboring blocks. This strong correlation and the associated probability distribution are not efficiently utilized, leading to inefficiency and poor performance of some intra-mode coding designs.

[0111] The present disclosure describes various embodiments for intra-prediction mode coding in video encoding and / or decoding, addressing at least one of the above-mentioned challenges / problems, achieving efficient intra-mode coding, and improving performance of intra-mode coding.

[0112] In various embodiments, to exploit the correlations and associated probability distributions between samples in a current block and its neighboring blocks, one or more contexts may be designed / derived for entropy coding of set indexes and / or mode indexes in a bitstream to indicate an intra-prediction mode for the current block, which may be used in an encoding process to encode the set indexes and / or mode indexes into the coded bitstream and / or may be used in a decoding process to decode the set indexes and / or mode indexes from the coded bitstream.

[0113] A context for coding the set index and / or mode index of the current coding block may be selected from a set of contexts. The selection of the context may be indicated by an index. The selection from the set of contexts may depend on various factors. For example, the selection of a coding context for the set index and / or mode index of the current block may depend on information of the current block and mode information of at least one of the neighboring blocks of the current block. For example, the at least one neighboring block may include a neighboring block above (top) and / or to the left of the current block. In another example, the at least one neighboring block may also include a neighboring block to the upper left of the current block.

[0114] In various embodiments, referring to method 1200 for intra-prediction modes in video decoding of FIG. 12, method 1200 may include some or all of the following steps: step 1210, receiving an encoded video bitstream for a block by an apparatus having a memory for storing instructions and a processor in communication with the memory; step 1220, deriving at least one context for a set index and a mode index for the block based on at least one of information about the block and mode information of at least one neighboring block of the block, wherein the set index indicates an intra-mode set from a plurality of intra-mode sets and the mode index indicates an intra-prediction mode from the intra-mode set; step 1230, extracting the set index and the mode index from the encoded video bitstream according to the at least one context; and / or step 1240, determining the intra-prediction mode for the block based on the set index and the mode index. In various embodiments, an intra mode set may refer to a set of intra modes having N intra modes, where N may be a positive integer, and may have a set index indicating a set of intra modes among a group of intra mode sets, and / or may have a mode index indicating a particular intra mode among the N intra modes in the intra mode set. In some implementations, step 1220 may include deriving, by the device, at least one context related to a set index and a mode index for the block based on at least one of information of the block or mode information of at least one neighboring block of the block, where the set index indicates an intra mode set from a plurality of intra mode sets, and the mode index indicates an intra prediction mode from the intra mode set.

[0115] In various embodiments, referring to the method 1300 for intra-prediction modes in video coding of FIG. 13, the method 1300 may include some or all of the following steps: step 1310, receiving a block to be coded into a coded video bitstream by an apparatus having a memory for storing instructions and a processor in communication with the memory; step 1320, deriving at least one context for a set index and a mode index of the block based on at least one of information of the block and mode information of at least one neighboring block of the block, wherein the set index indicates an intra-mode set from a plurality of intra-mode sets and the mode index indicates an intra-prediction mode from the intra-mode set; and / or step 1330, encoding the set index and the mode index into the coded video bitstream according to the at least one context. In some implementations, step 1320 may include the device deriving at least one context for a set index and a mode index of the block based on at least one of information about the block or mode information of at least one neighboring block of the block, wherein the set index indicates an intra mode set from a plurality of intra mode sets and the mode index indicates an intra prediction mode from the intra mode set.

[0116] In various embodiments of the present disclosure, the size of a block (for example, but not limited to, a coding block, a prediction block, or a transform block) may refer to the width or height of the block. The width or height of the block may be an integer number in pixels. In various embodiments of the present disclosure, the size of a block may refer to the area size of the block. The area size of the block may be an integer number calculated by multiplying the width of the block in pixels by the height of the block in pixels. In some various embodiments of the present disclosure, the size of a block may refer to the maximum width or height of the block, the minimum width or height of the block, or the aspect ratio of the block. The aspect ratio of the block may be calculated by dividing the width of the block by the height or the height of the block by the width.

[0117] In some implementations, the mode type of an intra mode may include at least one of a directional mode, a non-directional mode, a smooth mode (e.g., smooth, smooth_v, smooth_h), a DC mode, a PAETH mode, and / or a mode that generates prediction samples according to a given prediction direction. In some other implementations, in a loose classification, the directional mode may be broad and include any mode that is not a smooth mode (e.g., smooth, smooth_v, smooth_h), a DC mode, or a PAETH mode, and any mode that generates prediction samples according to a given prediction direction. In some other implementations, the non-directional mode may include a smooth mode (e.g., smooth, smooth_v, smooth_h), a DC mode, a PAETH mode, and a luma-for-chroma mode. In some other implementations, in a loose classification, the non-directional mode may be broad and include any mode that is not a directional mode.

[0118] In some implementations, the available intra-prediction modes for a current block may be divided / split into multiple intra-prediction mode sets. To obtain the intra-prediction mode of the current block, first, a mode set index indicating the mode set index of the intra-prediction mode of the current block may be signaled, and then a mode index indicating the index of the intra-prediction mode within the mode set may be signaled.

[0119] Here, in various embodiments of this disclosure, a "first" mode set not only refers to "one" mode set but also refers to the "first" mode set with the smallest mode set index, a "second" mode set not only refers to "another" mode set but also refers to the "second" mode set with the second smallest mode set index, and so on. For example, the number of intra-prediction mode sets may be denoted by M, and the mode set index may range, for example, from 1 to M or from 0 to M-1. When the mode set index ranges from 1 to M, the "first" mode set is the "first" mode set with a mode set index of 1, the "second" mode set is the "second" mode set with a mode set index of 2, and so on. When the mode set index ranges from 0 to M-1, the "first" mode set is the "first" mode set with a mode set index of 0, the "second" mode set is the "second" mode set with a mode set index of 1, and so on.

[0120] Here, in various embodiments of the present disclosure, "XYZ is signaled" can refer to XYZ being coded into a coded bitstream during an encoding process, and / or after the coded bitstream is transmitted from one device to another, "XYZ is signaled" can refer to XYZ being decoded / extracted from the coded bitstream during a decoding process.

[0121] Here, in various embodiments of this disclosure, a "block" may refer to a prediction block, a coding block, a transform block, or a coding unit (CU).

[0122] Referring to step 1210, the device may be the electronics device (530) of Figure 5 or the video decoder (810) of Figure 8. In some implementations, the device may be the decoder (633) in the encoder (620) of Figure 6. In other implementations, the device may be part of the electronics device (530) of Figure 5, part of the video decoder (810) of Figure 8, or part of the decoder (633) in the encoder (620) of Figure 6. The coded video bitstream may be the coded video sequence of Figure 8, or the intermediate coded data in Figure 6 or Figure 7. The block may refer to a coding block or an encoding block.

[0123] Referring to step 1220, the device may derive at least one context for a set index and a mode index for the block based on at least one of information about the block and mode information of at least one neighboring block of the block, where the set index indicates an intra mode set from a plurality of intra mode sets and the mode index indicates an intra prediction mode from the intra mode set. In some implementations, the information about the block includes a size of the block, and / or the mode information of the at least one neighboring block includes at least one of a number of a plurality of direction modes of the at least one neighboring block and / or a mode set index of the at least one neighboring block. In one example, the at least one neighboring block may include a neighboring block above (top) and / or to the left of the current block. In another example, the at least one neighboring block may also include a neighboring block to the upper left of the current block.

[0124] In some implementations, the context for signaling the intra mode set index and the mode index within the intra mode set is referred to as a cumulative density function (CDF) and depends on mode information of the current block and / or its neighboring blocks, including but not limited to the block size of the current block, the number of directional modes of at least one of the neighboring blocks, and / or the mode set index of at least one of the neighboring blocks.

[0125] In some other implementations, the device may further construct a list of intra modes for the block and / or divide the intra modes of the list into multiple intra mode sets for the block. For example, the intra mode list for the current block is constructed based on a set of predefined rules. After constructing the intra mode list, the intra prediction modes are divided into multiple intra prediction mode sets according to their corresponding indexes in the intra mode list. To signal the intra prediction mode of the current block, first, a mode set index indicating the mode set index of the intra prediction mode of the current block may be signaled, and then a mode index indicating the index of the intra prediction mode within the mode set may be signaled. The predefined rules may be different for different block sizes.

[0126] In various embodiments, step 1220 may include deriving a context for a set index and a mode index based on the number of the at least one neighboring block coded in a directional intra-prediction mode. For example, the context for signaling an intra-mode set index and / or a mode index within an intra-mode set depends on the number of neighboring blocks coded using a directional intra-prediction mode.

[0127] In some embodiments, deriving a context for a set index and a mode index based on the number of the at least one neighboring block coded in a directional intra-prediction mode may include: deriving a first context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being zero; deriving a second context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being one; and / or deriving a third context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being greater than one.

[0128] In some implementations, the context for the current block may be selected from a total of three, four, or five different contexts. In a specific implementation example, if none of the neighboring blocks above or to the left is coded in a directional intra-prediction mode, the first context (e.g., having a context index value of 0) is selected as the context for the set index and mode index of the current block. If only one of the neighboring blocks above and to the left is coded in a directional intra-prediction mode, the second context (e.g., having a context index value of 1) is selected as the context for the set index and mode index of the current block. Otherwise, the third context (e.g., having a context index value of 2) is selected as the context for the set index and mode index of the current block. Such an implementation is based on a statistical observation of a strong correlation between the number of neighboring blocks coded in a directional intra-prediction mode and the intra-prediction mode indicated by the set index and mode index of the current block.

[0129] In some embodiments, deriving a context for a set index and a mode index based on the number of the at least one neighboring block coded in a directional intra-prediction mode includes deriving a first context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being zero, deriving a second context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being one, and deriving a second context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being one. deriving a third context as a context related to a set index and a mode index in response to the number of neighboring blocks being two and the at least one neighboring block being coded in the same directional intra-prediction mode, and / or deriving a fourth context as a context related to a set index and a mode index in response to the number of neighboring blocks being two and the at least one neighboring block being coded in a different directional intra-prediction mode, or in response to the number of neighboring blocks being coded in a directional intra-prediction mode being more than two.

[0130] In some implementations, if there are no neighboring blocks coded in directional intra-prediction mode, a first context (e.g., with a context index value of 0) is selected as the context for the set index and mode index of the current block. If there is only one neighboring block coded in directional intra-prediction mode, a second context (e.g., with a context index value of 1) is selected as the context for the set index and mode index of the current block. If there are two neighboring blocks coded in directional intra-prediction mode and the two directional intra-prediction modes are the same, a third context (e.g., with a context index value of 2) is selected as the context for the set index and mode index of the current block. Otherwise, a fourth context (e.g., with a context index value of 3) is selected as the context for the set index and mode index of the current block.

[0131] In some embodiments, deriving a context for a set index and a mode index based on the number of the at least one neighboring block coded in a directional intra-prediction mode may include: deriving a first context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being zero; deriving a second context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being one; deriving a third context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being two; and / or deriving a fourth context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being more than two.

[0132] In some implementations, if there are no neighboring blocks coded in directional intra-prediction mode, a first context (e.g., with a context index value of 0) is selected as the context for the set index and mode index of the current block. If there is only one neighboring block coded in directional intra-prediction mode, a second context (e.g., with a context index value of 1) is selected as the context for the set index and mode index of the current block. If there are two neighboring blocks coded in directional intra-prediction mode, a third context (e.g., with a context index value of 2) is selected as the context for the set index and mode index of the current block. If there are more than two neighboring blocks coded in directional intra-prediction mode, a fourth context (e.g., with a context index value of 3) is selected as the context for the set index and mode index of the current block.

[0133] In some implementations, the directional intra-prediction mode includes at least one of an inter-coded mode or an intra-block copy mode. In some other implementations, the directional intra-prediction mode includes an intra-inter mode having a directional intra-mode component. In one example, if a neighboring block is coded in an inter-coded mode or an intra-block copy mode, it is considered a non-directional mode. In another example, if a neighboring block is in an intra-inter mode, the intra-prediction mode used in the intra-inter mode is used to derive a context or determine the selection of a context from a set of contexts.

[0134] In various embodiments, step 1220 may include deriving a context for a set index and a mode index based on the size of the block. For example, the context for signaling an intra-mode set index and / or a mode index within an intra-mode set depends on the block size of the current block. The block size may refer to the block width of the current block, the block height of the current block, the block area size (or the width x height of the current block), the sum of the width and height of the current block, the maximum value between the width and height of the current block, or the minimum value between the width and height of the current block.

[0135] In some embodiments, deriving a context for a set index and a mode index based on a size of the block comprises deriving a first context as the context for a set index and a mode index in response to the size of the block being less than or equal to a first threshold, and / or deriving a second context as the context for a set index and a mode index in response to the size of the block being greater than a first threshold, for example, the first threshold being 8x8.

[0136] In some implementations, if the block size of the current block is 8x8 or less, the first context (e.g., with a context index value of 0) is selected as the context for the set index and mode index of the current block. Otherwise, the second context (e.g., with a context index value of 1) is selected as the context for the set index and mode index of the current block.

[0137] In some embodiments, deriving a context for a set index and a mode index based on a size of the block includes deriving a first context as the context for a set index and a mode index in response to the size of the block being smaller than a first threshold, deriving a second context as the context for a set index and a mode index in response to the size of the block being equal to or greater than the first threshold and being smaller than a second threshold, deriving a third context as the context for a set index and a mode index in response to the size of the block being equal to or greater than the second threshold and being smaller than a third threshold, and / or deriving a fourth context as the context for a set index and a mode index in response to the size of the block being equal to or greater than a third threshold. For example, the first threshold is 64 (or 8×8) in area, the second threshold is 256 (or 16×16) in area, and / or the third threshold is 1024 (or 32×32) in area.

[0138] In some implementations, if the block area size of the current block is smaller than a first threshold (T1), a first context (e.g., having a context index value of 0) is selected as the context for the set index and mode index of the current block. If the block area size of the current block is equal to or greater than T1 but smaller than a second threshold (T2), a second context (e.g., having a context index value of 1) is selected as the context for the set index and mode index of the current block. If the block area size of the current block is equal to or greater than T2 but smaller than a third threshold (T3), a third context (e.g., having a context index value of 2) is selected as the context for the set index and mode index of the current block. Otherwise, a fourth context (e.g., having a context index value of 3) is selected as the context for the set index and mode index of the current block. For example, T1 is set to 64, T2 is set to 256, and / or T3 is set to 1024. In another example, T1 may be set to 16, T2 may be set to 64, and T3 may be set to 256.

[0139] In various embodiments, method 1200 may further optionally include the device determining, based on at least one of mode information of the at least one neighboring block of the block or a size of the block, a set of allowable intra-prediction modes for the block.

[0140] In some implementations, the number of intra-prediction modes allowed for a block may depend on the intra-prediction modes of its neighboring blocks and / or the block size of the current block. In one example, a mode set is selected from all available mode sets depending on the intra-prediction modes of its neighboring blocks and / or the block size of the current block, and only the intra-prediction modes in the selected mode set can be used to code the current block.

[0141] In various embodiments, determining a set of allowable intra-prediction modes for the block includes: determining a set of allowable intra-prediction modes for the block including the non-directional prediction mode and a nominal directional mode in response to the number of non-directional prediction modes for the at least one neighboring block being two or more; determining a set of allowable intra-prediction modes for the block including the non-directional prediction mode and a nominal directional mode in response to the number of non-directional prediction modes for the at least one neighboring block being two or more and in response to a size of the block being equal to or greater than a first threshold; and / or determining a set of allowable intra-prediction modes for the block including all intra-prediction modes in response to the number of directional prediction modes for the at least one neighboring block being one or more. For example, the first threshold is 8x8, 16x16, or 32x32.

[0142] In some implementations, if at least two of the intra-prediction modes of its neighboring blocks are non-directional modes, only those non-directional modes and the nominal directional mode are allowed for the current block.

[0143] In some other implementations, if at least two of the intra-prediction modes of its neighboring blocks are non-directional modes and / or the block size of the current block is equal to or greater than a threshold (TH1), only the non-directional modes and the nominal directional mode are allowed for the current block. In one example, TH1 is set to 32x32. In another example, TH1 may be set to 8x8, 16x16, or 64x64.

[0144] In some other implementations, all available intra prediction modes for the current block are allowed if at least one (or two) of its neighboring blocks' intra prediction modes is a directional intra prediction mode.

[0145] In various embodiments, method 1200 may further optionally include determining, by the device, two or more sets of allowable intra-prediction modes for the block based on at least one of mode information of at least one neighboring block of the block or a size of the block. In some implementations, two or more mode sets are selected from all available mode sets depending on the intra-prediction modes of the neighboring blocks and / or the block size of the current block, and only intra-prediction modes in the selected two or more mode sets may be used to code the current block.

[0146] In some implementations, the neighboring blocks of the current block may include one or more of the top (above or above) block of the current block, the block to the left of the current block, and the top-left (upper-left or above-left) block of the current block.

[0147] Referring to step 1240, the device may determine the intra-prediction mode of the block based on a set index and a mode index. In some implementations, the device may construct a list of intra-modes for the block according to a predefined rule based on the size of the block, and / or the device may divide the intra-modes in the list into multiple intra-mode sets for the block. The predefined rule may be one of a set of predefined rules, and may be selected from the set of predefined rules based on the size of the block. In some other implementations, an intra-mode list is constructed for the current block based on the predefined rule. After constructing the intra-mode list, the intra-prediction modes are divided into multiple intra-prediction mode sets according to their corresponding indexes in the intra-mode list. To signal the intra-prediction mode of the current block, a mode set index indicating the mode set index of the intra-prediction mode of the current block may be signaled first, and then a mode index indicating the index of the intra-prediction mode within the mode set may be signaled. The predefined rules may be different for different block sizes, which may be indicated by the number of samples in the block.

[0148] The embodiments of the present disclosure may be used separately or in combination in any order. Each of these methods (or embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to a luma block or a chroma block, and within a chroma block, the embodiments may be applied to two or more color components separately or to two or more color components together.

[0149] In the present disclosure, the steps or actions in the various embodiments may be combined in any quantity or order as desired. In the present disclosure, two or more steps or actions in the various embodiments may be performed in parallel.

[0150] The techniques described above can be implemented as computer software with computer-readable instructions physically stored on one or more computer-readable media. For example, Figure 14 illustrates a computer system (2600) suitable for implementing certain embodiments of the disclosed subject matter.

[0151] Computer software may be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similarly subjected to mechanisms to produce code having instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), and the like, either directly or via interpretation, microcode execution, and the like.

[0152] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0153] 14 for computer system (2600) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement related to any one or combination of components illustrated in this exemplary embodiment of computer system (2600).

[0154] The computer system 2600 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, moving a data glove, etc.), audio input (e.g., voice, clapping, etc.), visual input (e.g., gestures, etc.), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds, etc.), images (e.g., scanned images, photographic images obtained from a still camera, etc.), or video (e.g., two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0155] The input human interface devices may include one or more of a keyboard (2601), a mouse (2602), a trackpad (2603), a touchscreen (2610), a data glove (not shown), a joystick (2605), a microphone (2606), a scanner (2607), and a camera (2608) (only one of each shown).

[0156] The computer system (2600) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (2610), data gloves (not shown), or joystick (2605), although some haptic feedback devices may not function as input devices), audio output devices (e.g., speakers (2609), headphones (not shown), etc.), visual output devices (e.g., screens (2610) including CRT screens, LCD screens, plasma screens, and OLED screens (each with or without touchscreen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or four or more dimensional output through means such as stereoscopic output), virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), etc.), and printers (not shown).

[0157] The computer system (2600) may also include human-accessible storage devices and their associated media, such as optical media including, for example, a CD / DVD ROM / RW (2620) with a CD / DVD or similar media (2621), a thumb drive (2622), a removable hard drive or solid state drive (2623), legacy magnetic media such as tape and floppy disks (registered trademark, not shown), specialized ROM / ASIC / PLD-based devices (not shown) such as security dongles, and the like.

[0158] Those skilled in the art will also appreciate that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transitory signals.

[0159] The computer system 2600 may also include interfaces 2654 to one or more communications networks 2655. Networks may be, for example, wireless, wired, or optical. Networks may further be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, and vehicular and industrial networks including CANbus. Certain networks generally require an external network interface adapter that attaches to a particular general-purpose data port or peripheral bus 2649 (e.g., a USB port on the computer system 2600), while others are generally integrated into the core of the computer system 2600 by attachment to a system bus (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 2600 can communicate with other entities. Such communication may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to a particular CANbus device), or two-way, for example, to other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks may be used on each network and network interface, as described above.

[0160] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (2640) of the computer system (2600).

[0161] The core (2640) may include one or more central processing units (CPUs) (2641), graphics processing units (GPUs) (2642), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (2643), task-specific hardware accelerators (2644), graphics adapters (2650), etc. These devices may be connected via a system bus (2648), along with read-only memory (ROM) (2645), random access memory (2646), internal mass storage (2647), such as an internal non-user-accessible hard drive, SSD, and the like. In some computer systems, the system bus (2648) may be made accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached either directly to the core's system bus (2648) or via a peripheral bus (2649). In one example, the screen 2610 can be connected to a graphics adapter 2650. Peripheral bus architectures include PCI, USB, and the like.

[0162] The CPU (2641), GPU (2642), FPGA (2643), and accelerator (2644) may execute specific instructions that, in combination, may constitute the aforementioned computer code. The computer code may be stored in ROM (2645) or RAM (2646). Transient data may also be stored in RAM (2646), while permanent data may be stored, for example, in internal mass storage (2647). Rapid storage and retrieval from any of the memory devices may be enabled through the use of cache memory, which may be associated with one or more of the CPU (2641), GPU (2642), mass storage (2647), ROM (2645), RAM (2646), and the like.

[0163] The computer-readable media may have computer code thereon for performing various computer-implemented processes. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0164] As a non-limiting example, a computer system having the architecture (2600), and in particular the core (2640), can provide functionality as a result of the execution by one or more processors (including CPUs, GPUs, FPGAs, accelerators, and the like) of software embodied in one or more tangible computer-readable media. Such computer-readable media can be specific storage of the core (2640) that is non-transitory in nature, such as the core's internal mass storage (2647) or ROM (2645), and media associated with user-accessible mass storage as introduced above. Software implementing various embodiments of the present disclosure can be stored on such devices and executed by the core (2640). The computer-readable media can include one or more memory devices or chips, depending on specific needs. Software may cause the core (2640) and particularly the processors therein (including CPUs, GPUs, FPGAs, and the like) to perform particular processes or portions of particular processes described herein, including by defining data structures stored in RAM (2646) and modifying such data structures according to processes defined by the software. Additionally, or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (2644)) that can operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both, where appropriate. The present disclosure includes any suitable combination of hardware and software.

[0165] While the particular invention has been described with reference to exemplary embodiments, this description is not meant to be limiting. Various modifications of these exemplary embodiments, as well as further embodiments of the invention, will become apparent to those skilled in the art from this description. Those skilled in the art will readily recognize that these and various other modifications can be made to the exemplary embodiments shown and described herein without departing from the spirit and scope of the invention. Accordingly, it is intended that the appended claims cover such modifications and alternative embodiments. In the illustrations, certain proportions may be exaggerated, and other proportions may be minimized. Accordingly, the present disclosure and the figures are to be regarded as illustrative and not restrictive.

Claims

1. 1. A method for entropy coding for intra-prediction modes in video coding, the method being performed by an apparatus having a memory storing instructions and a processor in communication with the memory, the method comprising: deriving, for a block in a video sequence, at least one context for a set index and a mode index of the block based on at least one of information of the block and mode information of at least one neighboring block of the block; the set index indicates an intra-mode set from a plurality of intra-mode sets; deriving the mode index, wherein the mode index indicates an intra-prediction mode from the intra mode set; encoding the set index and the mode index according to the at least one context; encoding the block in an intra prediction mode determined based on the set index and the mode index; and The step of deriving at least one context related to the set index and the mode index comprises: deriving a context for the set index and the mode index based on a number of the at least one neighboring block coded in a directional intra-prediction mode; having method.

2. the information for the block comprises a size of the block; the mode information of the at least one neighboring block includes at least one of a number of multiple direction modes of the at least one neighboring block, or a mode set index of the at least one neighboring block; The method of claim 1.

3. The deriving of the context for the set index and the mode index based on the number of the at least one neighboring block coded in the directional intra-prediction mode comprises: deriving a first context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being zero; deriving a second context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being 1; and deriving a third context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being greater than one; 2. The method of claim 1, comprising:

4. The deriving of the context for the set index and the mode index based on the number of the at least one neighboring block coded in the directional intra-prediction mode comprises: deriving a first context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being zero; deriving a second context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being 1; deriving a third context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being two and the at least one neighboring block being coded in the same directional intra-prediction mode; and deriving a fourth context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being two and the at least one neighboring block being coded in a different directional intra-prediction mode, or in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being more than two; 2. The method of claim 1, comprising:

5. The deriving of the context for the set index and the mode index based on the number of the at least one neighboring block coded in the directional intra-prediction mode comprises: deriving a first context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being zero; deriving a second context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being 1; deriving a third context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being two; and deriving a fourth context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being greater than two; 2. The method of claim 1, comprising:

6. The method of claim 1 , wherein the directional intra-prediction modes include at least one of an inter-coded mode or an intra-block copy mode.

7. The method of claim 1 , wherein the directional intra-prediction modes include intra-inter modes having a directional intra-mode component.

8. The step of deriving the context for the set index and the mode index further comprises: deriving the context for the set index and the mode index based on a size of the block; 2. The method of claim 1, comprising:

9. deriving the context for the set index and the mode index based on the size of the block, deriving a first context as the context for the set index and the mode index in response to the size of the block being less than or equal to a first threshold; and deriving a second context as the context for the set index and the mode index in response to the size of the block being greater than the first threshold; 9. The method of claim 8, comprising:

10. The method of claim 9 , wherein the first threshold is 8×8.

11. deriving the context for the set index and the mode index based on the size of the block, deriving a first context as the context for the set index and the mode index in response to the size of the block being less than a first threshold; deriving a second context as the context for the set index and the mode index in response to the size of the block being greater than or equal to the first threshold and the size of the block being less than a second threshold; deriving a third context as the context for the set index and the mode index in response to the size of the block being greater than or equal to the second threshold and the size of the block being less than a third threshold; and deriving a fourth context as the context related to the set index and the mode index in response to the size of the block being equal to or greater than the third threshold; 9. The method of claim 8, comprising:

12. the first threshold is 64 in area; the second threshold is 256 in area; and the third threshold is 1024 in area; The method of claim 11.

13. determining a set of allowable intra-prediction modes for the block based on at least one of the mode information of the at least one neighboring block of the block or a size of the block; The method of claim 1 further comprising:

14. Determining the set of allowable intra-prediction modes for the block comprises: determining the set of allowable intra-prediction modes for the block comprising the non-directional prediction mode and a nominal directional mode in response to a number of non-directional prediction modes for the at least one neighboring block being greater than or equal to two; determining the set of allowable intra-prediction modes for the block comprising the non-directional prediction mode and a nominal directional mode in response to a number of non-directional prediction modes for the at least one neighboring block being two or more and in response to the size of the block being greater than or equal to a first threshold; and determining the set of allowable intra-prediction modes for the block, the set comprising all intra-prediction modes, in response to a number of directional prediction modes for the at least one neighboring block being greater than or equal to one; 14. The method of claim 13, comprising:

15. The method of claim 14 , wherein the first threshold is 32×32.

16. determining two or more sets of allowable intra-prediction modes for the block based on at least one of the mode information of at least one neighboring block of the block or a size of the block; The method of claim 1 further comprising:

17. 1. A method for video encoding, the method being performed by an apparatus having a memory storing instructions and a processor in communication with the memory, deriving, for a block in a video sequence, at least one context for a set index and a mode index of the block based on at least one of information of the block and mode information of at least one neighboring block of the block; the set index indicates an intra-mode set from a plurality of intra-mode sets; deriving the mode index, wherein the mode index indicates an intra-prediction mode from the intra mode set; encoding the set index and the mode index into a coded video bitstream according to the at least one context; encoding the block into the coded video bitstream in an intra-prediction mode determined based on the set index and the mode index; transmitting the encoded video bitstream; and The step of deriving at least one context related to the set index and the mode index comprises: deriving a context for the set index and the mode index based on a number of the at least one neighboring block coded in a directional intra-prediction mode; having method.

18. 1. A method for video encoding, the method being performed by an apparatus having a memory storing instructions and a processor in communication with the memory, deriving, for a block in a video sequence, at least one context for a set index and a mode index of the block based on at least one of information of the block and mode information of at least one neighboring block of the block; the set index indicates an intra-mode set from a plurality of intra-mode sets; deriving the mode index, wherein the mode index indicates an intra-prediction mode from the intra mode set; encoding the set index and the mode index into a coded video bitstream according to the at least one context; encoding the block into the coded video bitstream in an intra-prediction mode determined based on the set index and the mode index; storing the encoded video bitstream on a computer-readable storage medium; and The step of deriving at least one context related to the set index and the mode index comprises: deriving a context for the set index and the mode index based on a number of the at least one neighboring block coded in a directional intra-prediction mode; having method.

19. 1. An apparatus for entropy coding for intra prediction modes in video coding, comprising: a memory for storing instructions; a processor in communication with the memory, the processor being configured, when executing the instructions, to cause the apparatus to perform the method of any of claims 1 to 18; Device.

20. A computer program causing a computer to carry out the method according to any one of claims 1 to 18.

Images