Method, apparatus, and computer program for entropy coding for intra prediction mode

Abstract

To provide methods, apparatus, and computer readable storage medium for entropy coding for intra prediction modes in video encoding and / or decoding.SOLUTION: A method includes the steps of: receiving a coded video bitstream for a block; deriving at least one context for a set index and a mode index for the block based on at least one of information of the block and mode information of at least one neighboring block of the block; extracting, from the coded video bitstream, the set index and the mode index according to the at least one context; and determining an intra prediction mode for the block based on the set index and the mode index.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] This application is based on and claims the benefit of priority to U.S. Provisional Application No. 63 / 219,230, filed July 7, 2021, and U.S. Application No. 17 / 573,845, filed January 12, 2022, both of which are incorporated herein by reference in their entirety.

[0002] FIELD This disclosure relates to video encoding and / or decoding techniques, and more particularly to improved entropy coding design and signaling for intra-prediction modes. [Background technology]

[0003] This background discussion provided herein is intended to provide a general overview of the context of the disclosure. To the extent that it is described in this background section, the work of the named inventors, as well as aspects of the disclosure that may not otherwise qualify as prior art at the time of the filing of this application, are not admitted, explicitly or implicitly, as prior art to the present disclosure.

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video includes a sequence of pictures, each with spatial dimensions of, for example, 1920x1080 luminance samples and associated full-sampled or sub-sampled chrominance samples. The sequence of pictures can have a fixed or variable picture rate (also called frame rate), for example, 60 pictures per second, i.e., 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920x1080, a frame rate of 60 frames per second, and 4:2:0 chroma subsampling, with 8 bits per pixel per color channel, requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space.

[0005] One objective of video encoding and decoding may be to reduce redundancy in an uncompressed input video signal through compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than one order of magnitude. Both lossless and lossy compression, as well as combinations thereof, may be used. Lossless compression refers to techniques where the decoding process can reconstruct an exact replica of the original signal from the compressed original signal. Lossy compression refers to an encoding / decoding process where the original video information is not fully preserved during coding and is not fully recoverable upon decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals will be small enough to make the reconstructed signal useful for the intended application despite some information loss. For video, lossy compression is widely used in many applications. The amount of acceptable distortion depends on the application. For example, a user of a particular consumer video streaming application may tolerate higher distortion than a user of a movie or television broadcast application. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect different distortion tolerances, with higher tolerable distortion generally allowing for coding algorithms that result in higher losses and higher compression ratios.

[0006] Video encoders and decoders can utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0007] Video codec techniques can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. If all the samples of a block are coded in intra mode, the picture can be called an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and video session or as a still image. The samples of the block after intra prediction can then be subjected to a transform to the frequency domain, and the transform coefficients thus generated can be quantized before entropy coding. Intra-prediction refers to a technique that minimizes the sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficients, the fewer bits are needed for a given quantization step size to represent the block after entropy coding.

[0008] Traditional intra-coding, such as that known from the MPEG-2 generation of coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to code / decode a block based on surrounding sample data and / or metadata obtained during the coding and / or decoding of, for example, spatially adjacent and preceding in decoding order the block of data being intra-coded or intra-decoded. Such techniques are hereinafter referred to as "intra-prediction" techniques. Note that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed and not from other reference pictures.

[0009] There may be many different forms of intra prediction. If more than one such technique is available for a given video coding technique, the technique used may be referred to as an intra prediction mode. One or more intra prediction modes may be provided in a particular codec. In certain cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and intra coding parameters of a block of video may be coded individually or collectively included in a mode codeword. Which codeword is used for a given mode, sub-mode, and / or parameter combination may affect the coding efficiency gain through intra prediction, as may the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain modes of intra prediction were introduced in H.264, improved in H.265, and further improved in newer coding techniques, such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). In general, in intra prediction, a predictor block may be formed using available neighboring sample values. For example, values ​​of a particular set of available neighboring samples along a particular direction and / or line may be copied into the predictor block. The reference of the direction to use may be coded into the bitstream or may itself be predicted.

[0011] Referring to FIG. 1A, a subset of 9 predictor directions defined in the 33 possible predictor directions of H.265 (corresponding to the 33 angle modes of the 35 intra modes defined in H.265) is depicted at the bottom right. The converging point (101) of the arrows represents the sample being predicted. The arrow indicates that neighboring samples from that direction are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more neighboring samples to the top right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more neighboring samples to the bottom left at an angle of 22.5 degrees from the horizontal.

[0012] Still referring to FIG. 1A, a square block (104) of 4×4 samples (shown in bold dashed lines) is depicted at the top left. The square block (104) contains 16 samples, with each sample labeled with “S” and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions in the block (104). Since the block is 4×4 samples in size, S44 is at the bottom right. Also shown are example reference samples following a similar numbering scheme. The reference samples are labeled with R and their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, predicted samples neighboring the block being reconstructed are used.

[0013] Intra-picture prediction of block 104 may begin by appropriately copying reference sample values ​​from adjacent samples according to the signaled prediction direction. For example, assume that for this block 104, the coded video bitstream includes signaling indicating the prediction direction of arrow (102), i.e., that the sample is predicted from one or more prediction samples or samples 45 degrees to the right and above the horizontal. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. And sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the orientation is not evenly divisible by 45 degrees, the values ​​of several reference samples may be combined, for example by interpolation, to calculate the reference sample.

[0015] As video coding technology continues to develop, the number of possible directions is increasing. In H.264 (2003), for example, nine different directions are available for intra prediction. This increases to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions as of this disclosure. Empirical studies have been conducted to help identify the most preferred intra prediction directions, and specific techniques in entropy coding may be used to encode those most preferred directions into a small number of bits, accepting a specific bit penalty for the direction. Furthermore, the directions themselves may be predicted from neighboring directions used in intra prediction of previously decoded neighboring blocks.

[0016] FIG. 1B shows a schematic diagram (180) depicting 65 intra prediction directions according to JEM to illustrate the increasing number of prediction directions in various coding techniques that have been developed.

[0017] The manner in which bits representing intra-prediction directions are mapped to intra-prediction directions in the encoded video bitstream may vary from one video coding technique to another, and may range, for example, from a simple direct mapping of prediction directions to intra-prediction modes to complex adaptation schemes involving codewords, most-probable modes, and similar techniques. In any case, however, there may be certain directions in the video content that are statistically less likely to occur for intra-prediction than certain other directions. Because a goal of video compression is to reduce redundancy, in a well-designed video coding technique, these less likely directions may be represented with a larger number of bits than the more likely directions.

[0018] Inter-picture prediction, i.e., inter prediction, may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or part thereof (reference picture) may be used for prediction of a newly reconstructed picture or picture part (e.g., block) after being spatially shifted in a direction indicated by a motion vector (hereinafter, MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or may have three dimensions, with the third dimension being an indication of the reference picture to be used (similar to the temporal dimension).

[0019] In some video compression techniques, a current MV applicable to a particular region of sample data can be predicted from another MV, e.g., from another MV related to sample data of another region that is spatially adjacent to the region being reconstructed and precedes the current MV in decoding order. Doing so can significantly reduce the overall amount of data required to code the MV, thereby increasing compression efficiency, relying on removing redundancy in correlated MVs. MV prediction can work effectively because, for example, when coding an input video signal (known as natural video) derived from a camera, there is a statistical likelihood that regions larger than the region to which a single MV is applicable move in a similar direction in the video sequence and can therefore be predicted, possibly using similar motion vectors derived from the MVs of neighboring regions. This results in the actual MV for a given region being similar or the same as the MV predicted from the surrounding MVs. Such an MV can then be represented, after entropy coding, with fewer bits than would be used if the MV were directly coded instead of predicted from the neighboring MV(s). In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from an original signal (i.e., a sample stream). In other cases, MV prediction itself may be lossy, for example due to rounding errors in computing a predictor from some surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, “High Efficiency Video Coding”, December 2016). Among the many MV prediction mechanisms specified by H.265, a technique called “spatial merging” will be described below.

[0021] Specifically, referring to Figure 2, a current block (201) has samples that are found by the encoder during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of coding the MV directly, the MV can be derived from metadata associated with one or more reference pictures, such as from the immediately preceding reference picture (in decoding order), using the MV associated with any one of five surrounding samples, denoted as A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use predictors from the same reference picture as the neighboring blocks use. Summary of the Invention

[0022] This disclosure describes various embodiments of methods, apparatus, and computer-readable storage media for video encoding and / or decoding.

[0023] According to one aspect, an embodiment of the present disclosure provides a method for entropy coding for intra-prediction modes in video decoding. The method includes receiving, by an apparatus, an encoded video bitstream for a block. The apparatus includes a memory for storing instructions and a processor in communication with the memory. The method also includes deriving, by the apparatus, at least one context for a set index and a mode index for the block based on at least one of information of the block and mode information of at least one neighboring block of the block, the set index indicating an intra mode set from a plurality of intra mode sets, and the mode index indicating an intra-prediction mode from the intra mode set; extracting, by the apparatus, the set index and the mode index from the encoded video bitstream according to the at least one context; and determining, by the apparatus, the intra-prediction mode for the block based on the set index and the mode index.

[0024] According to another aspect, an embodiment of the present disclosure provides a method for entropy coding for intra-prediction modes in video coding. The method includes receiving, by an apparatus, a block to be coded into a coded video bitstream. The apparatus includes a memory storing instructions and a processor in communication with the memory. The method also includes deriving, by the apparatus, at least one context for a set index and a mode index for the block based on at least one of information of the block and mode information of at least one neighboring block of the block, the set index indicating an intra mode set from a plurality of intra mode sets, and the mode index indicating an intra-prediction mode from the intra mode set, and encoding, by the apparatus, the set index and the mode index into the coded video bitstream according to the at least one context.

[0025] According to another aspect, an embodiment of the present disclosure provides an apparatus for encoding and / or decoding video, the apparatus including a memory storing instructions and a processor in communication with the memory, the processor configured to, when executing the instructions, cause the apparatus to perform the above-described method for encoding and / or decoding video.

[0026] In another aspect, an embodiment of the present disclosure provides a non-transitory computer-readable medium having stored thereon instructions that, when executed by a computer for decoding and / or encoding video, cause the computer to perform the above-described method for decoding and / or encoding video.

[0027] These and other aspects and their implementations are described in more detail in the drawings, specification, and claims. [Brief description of the drawings]

[0028] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1A] 1 shows a schematic diagram of an exemplary subset of intra-prediction direction modes. [Figure 1B] 1 shows a diagram of an example intra-prediction direction. [Diagram 2] 1 shows a schematic diagram of spatial merging candidates for motion vector prediction in an example of a current block and its surroundings. [Diagram 3] 1 shows a simplified block diagram schematic of a communication system (300) according to an example embodiment. [Figure 4] 4 shows a simplified block diagram of a communication system (400) according to an example embodiment. [Diagram 5] 1 shows a schematic diagram of a simplified block diagram of a video decoder according to an example embodiment. [Figure 6] 1 shows a schematic diagram of a simplified block diagram of a video encoder according to an example embodiment; [Figure 7] 4 shows a block diagram of a video encoder according to another example embodiment. [Figure 8] 4 shows a block diagram of a video decoder according to another example embodiment. [Figure 9] 1 illustrates directional intra-prediction modes according to an example embodiment of this disclosure. [Figure 10] 1 illustrates a non-directional intra-prediction mode according to an example embodiment of this disclosure. [Figure 11] 1 illustrates a recursive intra-prediction mode in accordance with an example embodiment of this disclosure. [Figure 12] 1 illustrates a flow chart of a method according to an example embodiment of the present disclosure. [Figure 13] 13 illustrates a flowchart of another method according to an example embodiment of the present disclosure. [Figure 14] 1 shows a schematic diagram of a computer system according to an example embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] The present invention will now be described in detail with reference to the accompanying drawings, which form a part hereof and which show, by way of example, specific embodiments. It should be noted, however, that the present invention may be embodied in many different forms, and thus, the subject matter as subject or claimed should not be construed as being limited to any of the embodiments set forth below. It should also be noted that the present invention may be embodied as a method, an apparatus, a component, or a system. Thus, embodiments of the present invention may take the form of, for example, hardware, software, firmware, or any combination thereof.

[0030] Throughout the specification and claims, terms may have a meaning suggested or implied in the context beyond the meaning explicitly stated. The phrases "in one embodiment" or "in some embodiments" used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" used herein do not necessarily refer to different implementations. It is intended, for example, that the claimed subject matter includes combinations of example embodiments / implementations in whole or in part.

[0031] Generally, terms may be understood, at least in part, from their usage in context. For example, terms such as "and," "or," or "and / or" as used herein may include a variety of meanings that may depend, at least in part, on the context in which such terms are used. Typically, "or," when used to relate a list, such as A, B, or C, is intended to mean A, B, and C, which are used herein in an inclusive sense, and A, B, or C, which are used herein in an exclusive sense. Also, the terms "one or more" or "at least one" as used herein may be used to describe any feature, structure, or characteristic in a singular sense or to describe a combination of features, structures, or characteristics in a plural sense, at least in part, depending on the context. Similarly, terms such as "a," "an," or "the" may be understood to convey a singular sense or to convey a plural sense, at least in part, depending on the context. Additionally, the terms "based on" or "determined by" are not necessarily intended to convey an exclusive set of factors, but instead may be understood as allowing for the existence of additional factors not necessarily explicitly recited, again depending at least in part on the context.

[0032] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to one embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) may perform a one-way transmission of data. For example, the terminal device (310) may encode video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. One-way data transmission may be implemented in media service providing applications and the like.

[0033] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) for bidirectional transmission of encoded video data, which may be implemented, for example, in a videoconferencing application. In a bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., a stream of video pictures captured by that terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) may also receive encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to recover the video pictures, and display the video pictures on an accessible display device in accordance with the recovered video data.

[0034] In the example of FIG. 3, terminal devices 310, 320, 330, and 340 may be implemented as servers, personal computers, and smartphones, although applicability of the underlying principles of the present disclosure may not be so limited. Embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated videoconferencing equipment, and / or the like. Network 350 represents any number or type of network that conveys encoded video data between terminal devices 310, 320, 330, and 340, including, for example, wired (cable) and / or wireless communication networks. Communication network 350 may exchange data over circuit-switched channels, packet-switched channels, and / or other compatible channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network 350 may not be important to the operation of the present disclosure unless explicitly described herein.

[0035] 4 illustrates an arrangement of video encoders and video decoders in a video streaming environment as an example of an application of the disclosed subject matter, which may be equally applicable to other video applications including, for example, videoconferencing, digital TV broadcasting, gaming, virtual reality, and storage of compressed video on digital media including CDs, DVDs, memory sticks, and the like.

[0036] The video streaming system may include a video capture subsystem (413), which may include a video source (401), such as a digital camera, that produces a stream of uncompressed video pictures or images (402). In one example, the stream of video pictures (402) includes samples recorded by the digital camera of the video source (401). The stream of video pictures (402), depicted as a bold line to emphasize its high data volume compared to the encoded video data (404) (or encoded video bitstream), may be processed by an electronics device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted as thin lines to emphasize its low data volume compared to the uncompressed video picture stream (402), can be stored on the streaming server (405) for later use or provided directly to a downstream video device (not shown). One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example in the electronics device (430). The video decoder (410) can decode the incoming copy of the encoded video data (407) and produce an outgoing video picture stream (411), which is uncompressed and can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). The video decoder (410) may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data 404, 407, and 409 (e.g., video bitstreams) may be encoded according to a particular video encoding / compression standard. Examples of such standards include ITU-T Recommendation H.265. In one example, one video encoding standard under development is informally known as Versatile Video Coding (VVC). The subject matter disclosed herein may be used in the context of VVC and other video coding standards.

[0037] It should be noted that electronics devices 420 and 430 may include other components (not shown). For example, electronics device 420 may include a video decoder (not shown), and electronics device 430 may also include a video encoder (not shown).

[0038] 5 shows a block diagram of a video decoder (510) according to any of the following embodiments of the present disclosure. The video decoder (510) may be included in an electronics device (530). The electronics device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) in the example of FIG. 4.

[0039] A receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510). In the same or another embodiment, one encoded video sequence may be decoded at a time, and the decoding of each encoded video sequence is independent of the other encoded video sequences. Each video sequence may be associated with multiple video frames or images. The encoded video sequences may be received from a channel (501), which may be a storage device that stores the encoded video data or a hardware / software link to a streaming source that transmits the encoded video data. The receiver (531) may receive the encoded video data along with other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective processing circuits (not shown). The receiver (531) may separate the encoded video sequences from the other data. To combat network jitter, a buffer memory (515) may be disposed between the receiver (531) and the entropy decoder / parser 520 (hereinafter, "parser (520)"). In certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, it may be external and separate from the video decoder (510) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (510), e.g., to combat network jitter, and there may be an additional buffer memory (515) internal to the video decoder (510), e.g., to handle playback timing. When the receiver (531) is receiving data from a store / forward device with sufficient bandwidth and controllability or from an equally synchronous network, the buffer memory (515) may not be needed or may be small. For use over a best-effort packet network, such as the Internet, a sufficient size buffer memory (515) may be needed, and the size may be relatively large.Such a buffer memory may be implemented to have an adaptable size and may be implemented, at least in part, in an operating system or similar element (not shown) external to the video decoder (510).

[0040] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the coded video sequence. The categories of symbols may include information used to manage the operation of the video decoder (510) and possibly information for controlling a rendering device, such as a display (512) (e.g., a display screen) that may or may not be an integral part of the electronics device (530) but may be coupled to the electronics device (530) as shown in FIG. 5. The control information for the rendering device(s) may be in the form of a Supplementary Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the coded video sequence received by the parser (520). The entropy coding of the coded video sequence may be according to a video coding technique or standard and may follow a variety of principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the subgroup. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (520) may also extract information from the coded video sequence information, such as transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, motion vectors, etc.

[0041] The parser (520) may perform an entropy decoding / parsing process on the video sequence received from the buffer memory (515) to produce symbols (521).

[0042] The reconstruction of the symbols (521) may involve a number of different processing or functional units, depending on the type of coded video picture or portion thereof and other factors (e.g., inter-picture and intra-picture, inter-block and intra-block, etc.). The units involved, and how they are involved, may be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following processing or functional units is not shown for simplicity.

[0043] Beyond the functional blocks already described, the video decoder (510) may be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating under commercial constraints, many of these functional units may closely interact with each other and may be at least partially integrated with each other. However, for purposes of clearly describing the various functions of the disclosed subject matter, the following disclosure adopts a conceptual subdivision into functional units.

[0044] The first unit may include a scalar / inverse transform unit (551), which may receive quantized transform coefficients as symbol(s) (521) from the parser (520), along with control information including information indicating what type of inverse transform to use, block size, quantization coefficients / parameters, quantization scaling matrices, and the like. The scalar / inverse transform unit (551) may output blocks having sample values, which may be input to an aggregator (555).

[0045] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may generate a block of the same size and shape as the block being reconstructed using information of surrounding blocks already reconstructed and stored in the current picture buffer (558). The current picture buffer (558) may, for example, buffer the partially reconstructed and / or fully reconstructed current picture. In some implementations, the aggregator (555) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0046] In other cases, the output samples of the scaler / inverse transform unit (551) may relate to a block that may be inter-coded and motion-compensated. In such a case, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for inter-picture prediction. After motion compensating the fetched samples according to the symbols (521) related to the block, these samples may be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (the output of the unit (551) may be referred to as a residual sample or a residual signal) to generate output sample information. The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by a motion vector and are available to the motion compensated prediction unit (553) in the form of symbols (521) that may have, for example, X, Y components (shift), and a reference picture component (time). Motion compensation may also include interpolation of sample values ​​fetched from a reference picture memory (557) when sub-sample accurate motion vectors are used, and may be associated with a motion vector prediction mechanism, etc.

[0047] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also referred to as coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but may also be responsive to meta-information obtained during decoding of a previous portion of the coded picture or coded video sequence (in decoding order), as well as to previously reconstructed and loop filtered sample values. As will be described in more detail below, several types of loop filters may be included as part of the loop filter unit (556) in various orders.

[0048] The output of the loop filter unit (556) can be a sample stream that can be output to a rendering device (512), which can also be stored in a reference picture memory (557) for use in future inter-picture prediction.

[0049] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future inter-picture prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and that coded picture is identified (e.g., by the parser (520)) as a reference picture, the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before beginning reconstruction of the next coded picture.

[0050] The video decoder (510) may perform the decoding process according to a given video compression technique adopted in a standard, such as ITU-T Recommendation H.265. The encoded video sequence may conform to the syntax defined by the video compression technique or standard used, in the sense of adhering to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. In particular, a profile may select certain tools from all tools available in the video compression technique or standard such that they are the only tools available for use under that profile. To be standard-compliant, the complexity of the encoded video sequence may be brought within a range defined by a level of the video compression technique or standard. In some cases, the level constrains the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further constrained in some cases through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the encoded video sequence.

[0051] In some example embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0052] 6 shows a block diagram of a video encoder (603) according to an example embodiment of the present disclosure. The video encoder (603) may be included in an electronics device (620). The electronics device (620) may further include a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) may be used in place of the video encoder (403) in the example of FIG. 4.

[0053] The video encoder (603) may receive video samples from a video source (601) (which in the example of FIG. 6 is not part of the electronics device (620)) that may capture the video image(s) to be encoded by the encoder (603). In another example, the video source (601) may be implemented as part of the electronics device (620).

[0054] The video source (601) may provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCB, RGB, XYZ ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service provision system, the video source (601) may be a storage device capable of storing pre-prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures or images that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may have one or more samples depending on the sampling structure, color space, and the like used. Those skilled in the art will readily appreciate the relationship between pixels and samples. The following description focuses on samples.

[0055] According to some example embodiments, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real-time or under other time constraints required by an application. Enforcing an appropriate encoding rate constitutes one function of the controller (650). In some embodiments, the controller (650) may be operatively coupled to and control other functional units, as described below, which couplings are not shown for simplicity. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, and the like. The controller (650) may be configured with other suitable functions associated with the video encoder (603) that are optimized for a particular system design.

[0056] In some example embodiments, the video encoder (603) may be configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop may include a source coder (630) (e.g., responsible for creating symbols, e.g., a symbol stream, based on an input picture to be coded and reference picture(s)) and a (local) decoder (633) embedded in the video encoder (603). The embedded decoder (633) processes the coded video stream from the source coder (630) without entropy coding (when the compression between the symbols in the entropy coding and the coded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter), but the decoder (633) reconstructs the symbols to generate sample data in the same way as the (remote) decoder does. The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents in the reference picture memory (634) are also bit-accurate between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder "sees" when using the prediction during decoding. This fundamental principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is used to improve coding quality.

[0057] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510), which has already been described in detail above in relation to Figure 5. However, briefly referring also to Figure 5, because symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), may not need to be fully implemented in the local decoder (633) within the encoder.

[0058] At this point, it can be noted that decoder techniques, except for parsing / entropy decoding, that are present only in the decoder, must necessarily be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosure may occasionally focus on the decoder operation, which applies to the decoding portion of the encoder. The description of the encoder techniques may therefore be omitted, since they are the opposite of the decoder techniques that are described in detail. Only in certain areas or aspects will a more detailed description of the encoder be provided below.

[0059] In operation, in some implementations, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture relative to one or more previously coded pictures from a video sequence designated as “reference pictures.” Thus, the coding engine (632) codes differences (or residuals) in color channels between pixel blocks of the input picture and pixel blocks of the reference picture(s) that may be selected as prediction reference(s) for the input picture.

[0060] The local video decoder (633) may decode the encoded video data of pictures that may be designated as reference pictures based on symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the encoded video data may be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a replica of the source video sequence with some error. The local video decoder (633) may replicate the decoding process that may be performed by a video decoder on the reference pictures, causing the reconstructed reference pictures to be stored in a reference picture cache (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures that have common content with reconstructed reference pictures that will be obtained by a far-end (remote) video decoder.

[0061] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be encoded, the predictor (636) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors or block shapes, that may serve as suitable prediction references for the new picture. The predictor (635) may operate on a pixel block by pixel block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0062] The controller (650) may manage the coding process of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0063] The outputs of all the aforementioned functional units may be subjected to entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by lossless compression of the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0064] A transmitter (640) may buffer the encoded video sequence(s) generated by the entropy coder (645) and prepare them for transmission over a communication channel (660), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) may merge the encoded video data from the video coder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0065] The controller (650) may manage the operation of the video encoder (603). In coding, the controller (650) may assign each coded picture a particular coded picture type, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned one of the following picture types:

[0066] An Intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow several different types of Intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of these variants of I-pictures, as well as their respective uses and characteristics.

[0067] A predictive picture (P picture) may be encoded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0068] Bidirectionally predicted pictures (B-pictures) may be encoded and decoded using intra- or inter-prediction, using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0069] A source picture may generally be spatially subdivided into multiple sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to their respective pictures. For example, blocks of an I-picture may be coded non-predictively or they may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P-picture may be coded non-predictively or via spatial or temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be coded non-predictively or via spatial or temporal prediction with reference to one or two previously coded reference pictures. Source pictures or intermediate processed pictures may be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same method, as will be described in more detail below.

[0070] The video encoder (603) may perform a coding process according to a given video coding technique or standard, such as ITU-T Recommendation H.265. In operation, the video encoder (603) may perform various compression processes, including predictive coding processes that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore conform to a syntax defined by the video coding technique or standard being used.

[0071] In some example embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0072] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits temporal or other correlation between pictures. For example, a particular picture being coded / decoded, called a current picture, may be divided into multiple blocks. When a block in a current picture resembles a reference block in a previously coded and still buffered reference picture in the video, it may be coded by a vector called a motion vector. A motion vector points to a reference block in a reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0073] In some example embodiments, bi-prediction techniques can be used for inter-picture prediction. According to such bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both of which precede the current picture in decoding order (but may be past or future, respectively, in display order) in a video. A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. The block can be jointly predicted by a combination of the first and second reference blocks.

[0074] Furthermore, merge mode techniques may be used to improve coding efficiency in inter-picture prediction.

[0075] According to some example embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed on a block-by-block basis. For example, a picture in a sequence of video pictures is divided into multiple coding tree units (CTUs) for compression, and the CTUs in a picture may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU may include three parallel coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64×64 pixel CTU may be partitioned into one CU of 64×64 pixels or four CUs of 32×32 pixels. Each of one or more of the 32×32 blocks may be further partitioned into four CUs of 16×16 pixels. In some example embodiments, each CU may be analyzed during encoding to determine its prediction type among various prediction types, such as inter prediction type or intra prediction type. A CU may be divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. In general, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations during coding (encoding / decoding) are performed in units of prediction blocks. The division of a CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. For example, a luma or chroma PB may include a matrix of sample values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, and the like.

[0076] 7 shows a diagram of a video encoder (703) according to another example embodiment of the present disclosure. The video encoder (703) is configured to receive sample values ​​of a processing block (e.g., a predictive block) in a current video picture in a sequence of video pictures and to encode the processing block into a coded picture that is part of a coded video sequence. This example video encoder (703) may be used in place of the video encoder (403) in the example of FIG. 4.

[0077] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as 8×8 samples of a predictive block. The video encoder (703) then determines whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode, for example using rate-distortion optimization (RDO). If the processing block is determined to be coded in intra mode, the video encoder (703) can code the processing block into a coded picture using intra prediction techniques, and if the processing block is determined to be coded in inter mode or bi-predictive mode, the video encoder (703) can code the processing block into a coded picture using inter prediction techniques or bi-predictive techniques, respectively. In some example embodiments, a merge mode can be used as a sub-mode of inter-picture prediction in which a motion vector is derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In some other example embodiments, there may be motion vector components applicable to the current block. Thus, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module for determining a prediction mode for a processing block.

[0078] In the example of Figure 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), coupled together as shown in the example configuration of Figure 7.

[0079] The inter encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in preceding and following pictures in display order), generate inter prediction information (e.g., a description of redundant information according to an inter coding technique, motion vectors, merge mode information), and calculate an inter prediction result (e.g., a prediction block) based on the inter prediction information using any suitable technique. In some examples, the reference picture is a reference picture that has been decoded based on encoded video information using a decoding unit (633) embedded in an example of the encoder (620) of FIG. 6 (shown as residual decoder 72 in FIG. 7, described in more detail below).

[0080] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block to previously encoded blocks in the same picture, generate transformed quantized coefficients, and in some cases also generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). The intra encoder (722) may calculate an intra prediction result (e.g., a prediction block) based on the intra prediction information and reference blocks in the same picture.

[0081] The general controller (721) may be configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines a prediction mode of a block and provides a control signal to the switch (726) based on the prediction mode. For example, when the prediction mode is an intra mode, the general controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream, and when the prediction mode of the block is an inter mode, the general controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0082] The residual calculator (723) may be configured to calculate a difference (residual data) between a received block and a prediction result of the block selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder (724) may be configured to transform the residual data from a spatial domain to a frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various example embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be suitably used by the intra-encoder (722) and the inter-encoder (730). For example, the inter-encoder (730) may generate decoded blocks based on the decoded residual data and the inter-prediction information, and the intra-encoder (722) may generate decoded blocks based on the decoded residual data and the intra-prediction information. The decoded blocks are suitably processed to generate decoded pictures, which may be buffered in a memory circuit (not shown) and used as reference pictures.

[0083] The entropy encoder (725) may be configured to format a bitstream to include the encoded blocks and to perform entropy coding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. When coding a block in a merged sub-mode of either an inter mode or a bi-prediction mode, the residual information may not be present.

[0084] 8 shows a diagram of an example of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) may be used in place of the video decoder (410) in the example of FIG. 4.

[0085] In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872), coupled together as shown in the example configuration of Figure 8.

[0086] The entropy decoder (871) may be configured to reconstruct from the coded picture certain symbols that represent syntax elements that make up the coded picture. Such symbols may include, for example, prediction information (e.g., intra- or inter-prediction information) that may identify the mode in which the block is coded (e.g., intra-, inter-, bi-prediction, merged, or other sub-modes), certain samples or metadata used for prediction by the intra- or inter-decoder (872) or (880), residual information, e.g., in the form of quantized transform coefficients, and the like. In one example, when the prediction mode is an inter- or bi-prediction mode, the inter-prediction information is provided to the inter-decoder (880), and when the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information may be subjected to inverse quantization and provided to the residual decoder (873).

[0087] The inter decoder (880) may be configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0088] The intra decoder (872) may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0089] The residual decoder (873) may be configured to perform inverse quantization to extract dequantized transform coefficients, and then process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize certain control information (including quantizer parameters (QP)) that may be provided by the entropy decoder (871) (data path not shown as this may be only low data volume control information).

[0090] The reconstruction module (874) may be configured to combine the residual output by the residual decoder (873) and the prediction result (output by the inter- or intra-prediction module, as the case may be) in the spatial domain to form a reconstructed block that forms part of a reconstructed picture as part of the reconstructed video, although other suitable processes may also be performed to improve visual quality, such as deblocking processes and the like.

[0091] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In some example embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0092] Returning to the intra prediction process, a prediction block is generated by predicting samples in a block (e.g., a luma or chroma prediction block, or a coding block if not further divided into prediction blocks) by neighboring samples, next neighboring samples, or samples of one or more other lines, or a combination thereof. The residual between the actual block to be coded and the prediction block may then be processed by a transform followed by quantization. Various intra prediction modes may be made available, and parameters related to intra mode selection and other parameters may be signaled in the bitstream. The various intra prediction modes may relate, for example, to one or more line positions for predicting samples, a direction along which a prediction sample is selected from one or more prediction lines, and other special intra prediction modes.

[0093] For example, the set of intra-prediction modes (also interchangeably referred to as "intra modes") may include a predetermined number of directional intra-prediction modes. As described above in connection with the implementation of FIG. 1, these intra-prediction modes correspond to a predetermined number of directions along which out-of-block samples may be selected as predictions of predicted samples within a particular block. In another particular implementation, eight main directional modes may be supported and predefined, corresponding to angles from 45 degrees to 207 degrees relative to the horizontal axis.

[0094] In some other implementations of intra prediction, the directional intra modes may be further expanded to a finer set of angles to further exploit the more diverse spatial redundancy of directional textures. For example, the above 8-angle implementation may be configured to provide eight nominal angles called V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, as shown in FIG. 9, and a predetermined number (e.g., seven) of finer angles may be added to each nominal angle. Such expansion may make a larger total number (e.g., 56 in this example) of directional angles available for intra prediction, corresponding to the same number of predefined directional intra modes. The prediction angle may be represented as the nominal intra angle plus an angle delta. In the above specific example, there are seven finer angle directions for each nominal angle, and the angle delta may be a step size of 3 degrees multiplied by -3 to 3.

[0095] The above directional intra-prediction is sometimes referred to as unidirectional intra-prediction, and is distinct from bidirectional intra-prediction (also called intra-bi-prediction) which is described in a later section of this disclosure.

[0096] In some implementations, instead of or in addition to the above directional intra mode, a certain number of non-directional intra prediction modes may also be predefined and made available. For example, five non-directional intra modes called smooth intra prediction modes may be defined. These non-directional intra prediction modes may be specifically called DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H intra modes. Prediction of samples of a particular block under these non-directional mode examples is shown in FIG. 10. As an example, FIG. 10 shows that a 4×4 block 1002 is predicted by samples from an upper neighboring line and / or a left neighboring line. A particular sample 1010 in the block 1002 may correspond to a sample 1004 directly above the sample 1010 in the above neighboring line of the block 1002, a sample 1006 above and to the left of the sample 1010 as the intersection of the above and left neighboring lines, and a sample 1008 directly to the left of the sample 1010 in the left neighboring line of the block 1002. In an example DC intra prediction mode, the average of the left and top neighboring samples 1008 and 1004 may be used as a predictor for sample 1010. In an example PAETH intra prediction mode, the top, left, and top left reference samples 1004, 1008, and 1006 may be fetched, and the closest of these three reference samples to (top+left-top left) may be set as the predictor for sample 1010. In an example SMOOTH_V intra prediction mode, sample 1010 may be predicted by quadratic vertical interpolation of the top left neighboring sample 1006 and the left neighboring sample 1008. In an example SMOOTH_H intra prediction mode, sample 1010 may be predicted by quadratic horizontal interpolation of the top left neighboring sample 1006 and the top neighboring sample 1004. In an example SMOOTH intra prediction mode, sample 1010 may be predicted by an average of quadratic vertical and quadratic horizontal interpolation. The above non-directional intra mode implementations are shown merely as non-limiting examples: other adjacent line and other non-directional sample selections and ways of combining prediction samples to predict a particular sample within a prediction block are also contemplated.

[0097] The selection of a particular intra-prediction mode by the encoder from the above directional or non-directional modes at various coding levels (picture, slice, block, unit, etc.) may be signaled in the bitstream. In some implementations, the exemplary eight nominal directional modes may be signaled first along with the five non-angle smooth modes (a total of 13 options). Then, if the signaled mode is one of the eight nominal angle intra modes, an index is further signaled to indicate the selected angle delta relative to the corresponding signaled nominal angle. In some other implementations, all intra-prediction modes may be indexed together (e.g., 56 directional modes plus 5 non-directional modes to generate 61 intra-prediction modes) for signaling.

[0098] In some implementations, the 56 or other number of directional intra-prediction modes in this example may be implemented with a unified directional predictor that projects each sample of a block to the location of a reference subsample and interpolates the reference sample using a two-tap bilinear filter.

[0099] In some implementations, additional filter modes, called FILTER INTRA modes, may be designed to capture the decay of spatial correlation with references on edges. In these modes, for some patches in a block, predicted samples in the block may be used as intra prediction reference samples in addition to out-of-block samples. These modes may be predefined and made available, for example, for intra prediction at least for luma blocks (or only luma blocks). A certain number (e.g., 5) of filter intra modes may be predesigned, each represented by a set of n-tap filters (e.g., 7-tap filters) that reflects the correlation between a sample in a 4×2 patch and its n neighboring samples. In other words, the weighting coefficients of the n-tap filters may be position-dependent. Taking an 8×8 block, a 4×2 patch, and 7-tap filtering as examples, an 8×8 block 1102 may be divided into eight 4×2 patches as shown in FIG. 11. These patches are indicated as B0, B1, B2, B3, B4, B5, B6, and B7 in FIG. 11. For each patch, its seven neighbors (denoted R0-R7 in FIG. 11) may be used to predict samples in the current patch. For patch B0, it may be assumed that all of the neighbors are already reconstructed. However, for other patches, some of the neighbors may be in the current block and therefore not reconstructed, and the predictions of the immediate neighbors are used as reference. For example, not all of the neighbors of patch B7 shown in FIG. 11 are reconstructed, and therefore the neighbors that are part of, for example, B4, B5, and / or B6 are used instead.

[0100] In some implementations of intra prediction, one color component may be predicted using one or more other color components. The color components may be any of the components in YCrCb, RGB, XYZ color spaces, etc. For example, prediction of a chroma component (e.g., a chroma block) from a luma component (e.g., a luma reference sample), referred to as chroma-from-luma (i.e., CfL), may be implemented. In some implementations, cross-color prediction may only be allowed from luma to chroma. For example, chroma samples within a chroma block may be modeled as linear functions of the corresponding reconstructed luma samples. CfL prediction is CfL(α)=α×L AC +DC (1) It can be implemented as:

[0101] Here, L AC represents the AC contribution of the luma component, α represents a parameter of a linear model, and DC represents the DC contribution of the chroma component. For example, the AC components are obtained for each sample of the block, while the DC components are obtained for the entire block. Specifically, the reconstructed luma samples may be subsampled to the chroma resolution, and then the average luma value (the luma DC) may be subtracted from each luma value to form the luma AC contribution. The luma AC contribution is then used in the linear mode of equation (1) to predict the AC values ​​of the chroma components. Instead of requiring the decoder to calculate scaling parameters to approximate or predict the chroma AC components from the luma AC contributions, an example CfL implementation may determine the parameter α based on the original chroma samples and signal them in the bitstream. This reduces the decoder complexity and produces a more accurate prediction. Regarding the DC contribution of the chroma components, it may be calculated using an intra DC mode within the chroma components in some implementations.

[0102] In some embodiments, for a particular coding block, coding unit, predictive block, or prediction unit that is intra-coded, its intra mode needs to be signaled by one or more syntax elements in the bitstream. As mentioned above, the number of possible intra prediction modes can be large, and 62 intra prediction modes may be available: 56 directional intra prediction modes, 5 non-directional modes, and one chroma-from-luma mode (e.g., only for the chroma component). To signal these intra prediction modes, a first syntax may be signaled to indicate which nominal angle mode or non-directional mode is equal to the nominal mode of the current block. And, if the mode of the current block is a directional mode, a second syntax may be signaled to indicate which delta angle is equal to that of the current block. In some situations during video encoding or decoding, there may be a strong correlation between the intra prediction modes of the current block and its neighboring blocks.

[0103] In various embodiments, this correlation may be exploited to design a more efficient syntax for intra-mode coding. In some implementations, the intra-prediction modes available for a current block may be divided into multiple intra-prediction mode sets according to the intra-prediction modes of its neighboring blocks. To obtain the intra-prediction mode of the current block, first, a mode set index indicating the mode set index of the intra-prediction mode of the current block may be signaled, and then a mode index indicating the index of the intra-prediction mode in the mode set may be signaled.

[0104] Here, in various embodiments of the present disclosure, "XYZ is signaled" can refer to XYZ being encoded into a coded bitstream during an encoding process, and / or after the coded bitstream is transmitted from one device to another, "XYZ is signaled" can refer to XYZ being decoded / extracted from the coded bitstream during a decoding process.

[0105] For example, in some of the implementations described above, the number of available intra-prediction modes may include 62 distinct modes, such as 56 directional intra-prediction modes (e.g., 8 nominal directions with 7 finer angles per nominal direction), 5 non-directional modes, and one chroma-from-luma mode (e.g., only for the chroma component). When an intra-mode is selected for a particular coding block, coding unit, prediction block, or prediction coding process, signaling corresponding to the selected intra-mode must be included in the bitstream. Signaling syntax(es) must be able to distinguish all 62 modes in some way. For example, the 62 modes may be signaled using a single syntax for 62 indices, each corresponding to a mode. In some other implementations, one syntax may be signaled to indicate which nominal angle mode or non-directional mode is used as the nominal mode in the current block, and another syntax may be additionally signaled to indicate which delta angle is selected for the current block if the nominal mode of the current block is a directional mode.

[0106] Since various syntaxes related to intra-coding typically occupy a large portion of the bitstream and intra-mode selections must be signaled frequently, e.g., at various coding levels, reducing the number of bits used for intra-mode signaling is important for improving video coding efficiency. In practice, the use of various intra-prediction modes may follow certain statistical patterns, and such usage patterns may be used to design the intra-mode indexing and signaling syntax to improve signaling efficiency. In addition, in general, there may be some correlation between blocks in intra-mode selections. Such correlations may be obtained offline on a statistical basis and taken into account in the design of the syntax(es) for signaling intra-mode selections. The goal is to generally reduce the number of bits for signaling syntax elements in the coded bitstream. For example, some general statistics may indicate that there may be a strong correlation between the optimal intra-prediction modes of a current block and its neighboring blocks. Such correlations may be exploited when designing a syntax for intra-mode coding.

[0107] In various embodiments, the intra-prediction modes or mode options available for a current block being coded may be divided into multiple intra-prediction mode sets. A mode set index may be assigned to each set. Each set may include a number of intra-mode prediction modes. The manner of dividing and ordering the available intra-prediction modes and ordering the intra-prediction modes within each of the mode sets may be determined based on the correlation between intra-prediction modes between blocks and at least in part according to the intra-prediction modes used by its neighboring blocks. The intra-prediction modes used by the neighboring blocks may be referred to as "reference intra-prediction modes" or "reference modes." An intra-prediction mode may be determined and selected for a particular unit. The selection of the intra-prediction mode may be signaled. First, a mode set index may be signaled to indicate the mode set index of the intra-prediction mode set that includes the selected intra-prediction mode. Next, a mode index (alternatively referred to as a mode position index within a set) may be signaled to indicate the index of the selected intra-prediction mode within the mode set.

[0108] The general implementations of intra-prediction mode partitioning and ordering above and the specific examples below allow for dynamic indexing of these modes using statistical effects and neighbor correlations to optimize the design of syntax for signaling their selection within the coded video bitstream to improve coding efficiency. For example, these implementations can help reduce the number of syntaxes for signaling, as well as help more efficient context generation for entropy coding.

[0109] The various embodiments and / or implementations described in this disclosure may be used separately or combined in any order. Also, some, all, or a partial or full combination of these embodiments and / or implementations may be embodied as part of an encoder and / or decoder and may be implemented in hardware and / or software. For example, they may be hard-coded into dedicated processing circuitry (e.g., one or more integrated circuits). In another example, they may be implemented by one or more processors executing a program stored on a non-transitory computer-readable medium.

[0110] There may be some challenges / issues associated with intra-mode coding. For example, when dividing intra-prediction modes into multiple different mode sets, the probability of selecting an intra-prediction mode from a different mode set may be correlated with the intra-prediction modes of the neighboring blocks, since there is a strong correlation between samples in a current block and its neighboring blocks. This strong correlation and the associated probability distribution are not efficiently exploited, leading to inefficiencies and poor performance of some intra-mode coding designs.

[0111] The present disclosure describes various embodiments for intra-prediction mode coding in video encoding and / or decoding, addressing at least one of the above-mentioned challenges / problems, achieving efficient intra-mode coding, and improving performance of intra-mode coding.

[0112] In various embodiments, to exploit the correlation and associated probability distribution between samples in a current block and its neighboring blocks, one or more contexts may be designed / derived for entropy coding of set indexes and / or mode indexes for indicating an intra-prediction mode for a current block in a bitstream, which may be used in an encoding process to code the set indexes and / or mode indexes into the encoded bitstream and / or which may be used in a decoding process to decode the set indexes and / or mode indexes from the encoded bitstream.

[0113] A context for coding the set index and / or mode index of the current coding block may be selected from a set of contexts. The selection of the context may be indicated by an index. The selection from the set of contexts may depend on various factors. For example, the selection of the coding context for the set index and / or mode index of the current block may depend on information of the current block and mode information of at least one of the neighboring blocks of the current block. For example, the at least one neighboring block may include a top and / or left neighboring block of the current block. In another example, the at least one neighboring block may also include a top left neighboring block of the current block.

[0114] In various embodiments, referring to method 1200 for intra prediction modes in video decoding of FIG. 12, method 1200 may include some or all of the following steps: step 1210, receiving an encoded video bitstream for a block by an apparatus having a memory for storing instructions and a processor in communication with the memory; step 1220, deriving at least one context for a set index and a mode index for the block based on at least one of information of the block and mode information of at least one neighboring block of the block, where the set index indicates an intra mode set from a plurality of intra mode sets and the mode index indicates an intra prediction mode from the intra mode set; step 1230, extracting a set index and a mode index from the encoded video bitstream according to the at least one context; and / or step 1240, determining an intra prediction mode for the block based on the set index and the mode index. In various embodiments, an intra mode set may refer to a set of intra modes having N intra modes, where N may be a positive integer, and the intra mode set may have a set index indicating a set of intra modes among a group of intra mode sets, and / or the intra mode set may have a mode index indicating a particular intra mode among the N intra modes in the intra mode set. In some implementations, step 1220 may include deriving, by the device, at least one context for a set index and a mode index for the block based on at least one of information of the block or mode information of at least one neighboring block of the block, where the set index indicates an intra mode set from a plurality of intra mode sets and the mode index indicates an intra prediction mode from the intra mode set.

[0115] In various embodiments, referring to method 1300 for intra-prediction modes in video coding of FIG. 13, method 1300 may include some or all of the following steps: step 1310, receiving a block to be encoded into a coded video bitstream by an apparatus having a memory for storing instructions and a processor in communication with the memory; step 1320, deriving by the apparatus at least one context for a set index and a mode index of the block based on at least one of information of the block and mode information of at least one neighboring block of the block, where the set index indicates an intra mode set from a plurality of intra mode sets and the mode index indicates an intra-prediction mode from the intra mode set; and / or step 1330, encoding by the apparatus the set index and the mode index into the coded video bitstream according to the at least one context. In some implementations, step 1320 may include the device deriving at least one context for a set index and a mode index of the block based on at least one of information of the block or mode information of at least one neighboring block of the block, where the set index indicates an intra mode set from a plurality of intra mode sets and the mode index indicates an intra prediction mode from the intra mode set.

[0116] In various embodiments of the present disclosure, the size of a block (such as, but not limited to, a coding block, a prediction block, or a transform block) may refer to the width or height of the block. The width or height of the block may be an integer number in pixels. In various embodiments of the present disclosure, the size of a block may refer to the area size of the block. The area size of the block may be an integer number calculated by multiplying the width of the block in pixels by the height of the block in pixels. In some various embodiments of the present disclosure, the size of a block may refer to the maximum width or height of the block, the minimum width or height of the block, or the aspect ratio of the block. The aspect ratio of the block may be calculated by dividing the width of the block by the height or the height of the block by the width.

[0117] In some implementations, the mode type of the intra mode may include at least one of a directional mode, a non-directional mode, a smooth mode (e.g., smooth, smooth_v, smooth_h), a DC mode, a PAETH mode, and / or a mode that generates a prediction sample according to a given prediction direction. In some other implementations, in a loose classification, the directional mode may be broad and include any mode that is not a smooth mode (smooth, smooth_v, smooth_h), a DC mode, or a PAETH mode, and any mode that generates a prediction sample according to a given prediction direction. In some other implementations, the non-directional mode may include a smooth mode (e.g., smooth, smooth_v, smooth_h), a DC mode, a PAETH mode, and a luma-for-chroma mode. In some other implementations, in a loose classification, the non-directional mode may be broad and include any mode that is not a directional mode.

[0118] In some implementations, the intra-prediction modes available for the current block may be divided / split into multiple intra-prediction mode sets. To obtain the intra-prediction modes of the current block, first, a mode set index indicating the mode set index of the intra-prediction mode of the current block may be signaled, and then a mode index indicating the index of the intra-prediction mode in the mode set may be signaled.

[0119] Here, in various embodiments of the present disclosure, a "first" mode set does not only refer to "one" mode set, but also refers to the "first" mode set with the smallest mode set index, a "second" mode set does not only refer to "another" mode set, but also refers to the "second" mode set with the second smallest mode set index, and so on. For example, the number of intra-prediction mode sets may be denoted as M, and the mode set index may range, for example, from 1 to M, or from 0 to M-1. If the mode set index ranges from 1 to M, the "first" mode set is the "first" mode set with a mode set index of 1, the "second" mode set is the "second" mode set with a mode set index of 2, and so on. If the mode set index ranges from 0 to M-1, the "first" mode set is the "first" mode set with a mode set index of 0, the "second" mode set is the "second" mode set with a mode set index of 1, and so on.

[0120] Here, in various embodiments of the present disclosure, "XYZ is signaled" can refer to XYZ being encoded into a coded bitstream during an encoding process, and / or after the coded bitstream is transmitted from one device to another, "XYZ is signaled" can refer to XYZ being decoded / extracted from the coded bitstream during a decoding process.

[0121] Here, in various embodiments of this disclosure, a "block" may refer to a prediction block, a coding block, a transform block, or a coding unit (CU).

[0122] Referring to step 1210, the device may be the electronics device (530) of Figure 5 or the video decoder (810) of Figure 8. In some implementations, the device may be the decoder (633) in the encoder (620) of Figure 6. In other implementations, the device may be part of the electronics device (530) of Figure 5, part of the video decoder (810) of Figure 8, or part of the decoder (633) in the encoder (620) of Figure 6. The coded video bitstream may be the coded video sequence of Figure 8, or the intermediate coded data in Figure 6 or Figure 7. The block may refer to a coding block or a coding block.

[0123] Referring to step 1220, the device may derive at least one context for a set index and a mode index of the block based on at least one of information of the block and mode information of at least one neighboring block of the block, where the set index indicates an intra mode set from a plurality of intra mode sets, and the mode index indicates an intra prediction mode from the intra mode set. In some implementations, the information of the block includes a size of the block, and / or the mode information of the at least one neighboring block includes at least one of a number of a plurality of direction modes of the at least one neighboring block and / or a mode set index of the at least one neighboring block. In one example, the at least one neighboring block may include a top and / or left neighboring block of the current block. In another example, the at least one neighboring block may also include a top-left neighboring block of the current block.

[0124] In some implementations, the context for signaling the intra mode set index and the mode index within the intra mode set is referred to as a cumulative density function (CDF) and depends on the mode information of the current block and / or its neighboring blocks, including but not limited to the block size of the current block, the number of directional modes of at least one of the neighboring blocks, and / or the mode set index of at least one of the neighboring blocks.

[0125] In some other implementations, the device may further construct a list of intra modes for the block and / or divide the intra modes of the list into a plurality of intra mode sets for the block. For example, an intra mode list for the current block is constructed based on a set of predefined rules. After constructing the intra mode list, the intra prediction modes are divided into a plurality of intra prediction mode sets according to their corresponding indexes in the intra mode list. To signal the intra prediction mode of the current block, first, a mode set index indicating the mode set index of the intra prediction mode of the current block may be signaled, and then a mode index indicating the index of the intra prediction mode in the mode set may be signaled. The predefined rules may be different for different block sizes.

[0126] In various embodiments, step 1220 may include deriving a context for a set index and a mode index based on the number of the at least one neighboring block coded with a directional intra-prediction mode. For example, the context for signaling an intra-mode set index and / or a mode index within an intra-mode set depends on the number of neighboring blocks coded with a directional intra-prediction mode.

[0127] In some embodiments, deriving a context for a set index and a mode index based on the number of the at least one neighboring block coded in a directional intra-prediction mode may include: deriving a first context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being zero; deriving a second context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being one; and / or deriving a third context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being greater than one.

[0128] In some implementations, the selection of the context for the current block may be made from a total of three, four, or five different contexts. In a specific implementation example, if none of the upper or left neighboring blocks are coded in a directional intra-prediction mode, the first context (e.g., with a context index value of 0) is selected as the context for the set index and mode index of the current block. If only one of the upper and left neighboring blocks is coded in a directional intra-prediction mode, the second context (e.g., with a context index value of 1) is selected as the context for the set index and mode index of the current block. Otherwise, the third context (e.g., with a context index value of 2) is selected as the context for the set index and mode index of the current block. Such an implementation is based on a statistical observation of a strong correlation between the number of neighboring blocks coded in a directional intra-prediction mode and the intra-prediction mode indicated by the set index and mode index of the current block.

[0129] In some embodiments, deriving a context for a set index and a mode index based on the number of the at least one neighboring block coded in a directional intra-prediction mode includes deriving a first context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being zero, deriving a second context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being one, and deriving a second context as a context for a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being one. deriving a third context as a context related to a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being two and the at least one neighboring block being coded in a same directional intra-prediction mode, and / or deriving a fourth context as a context related to a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being two and the at least one neighboring block being coded in a different directional intra-prediction mode or in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being more than two.

[0130] In some implementations, if there is no neighboring block coded in a directional intra-prediction mode, the first context (e.g., with a context index value of 0) is selected as the context for the set index and mode index of the current block. If there is only one neighboring block coded in a directional intra-prediction mode, the second context (e.g., with a context index value of 1) is selected as the context for the set index and mode index of the current block. If there are two neighboring blocks coded in a directional intra-prediction mode and the two directional intra-prediction modes are the same, the third context (e.g., with a context index value of 2) is selected as the context for the set index and mode index of the current block. Otherwise, the fourth context (e.g., with a context index value of 3) is selected as the context for the set index and mode index of the current block.

[0131] In some embodiments, deriving a context related to a set index and a mode index based on the number of the at least one neighboring block coded in a directional intra-prediction mode may include: deriving a first context as a context related to a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being zero; deriving a second context as a context related to a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being one; deriving a third context as a context related to a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being two; and / or deriving a fourth context as a context related to a set index and a mode index in response to the number of the at least one neighboring block coded in a directional intra-prediction mode being greater than two.

[0132] In some implementations, if there are no neighboring blocks coded in directional intra-prediction mode, a first context (e.g., with a context index value of 0) is selected as the context for the set index and mode index of the current block. If there is only one neighboring block coded in directional intra-prediction mode, a second context (e.g., with a context index value of 1) is selected as the context for the set index and mode index of the current block. If there are two neighboring blocks coded in directional intra-prediction mode, a third context (e.g., with a context index value of 2) is selected as the context for the set index and mode index of the current block. If there are more than two neighboring blocks coded in directional intra-prediction mode, a fourth context (e.g., with a context index value of 3) is selected as the context for the set index and mode index of the current block.

[0133] In some implementations, the directional intra-prediction mode includes at least one of an inter-coded mode or an intra-block copy mode. In some other implementations, the directional intra-prediction mode includes an intra-inter mode with a directional intra-mode component. In one example, if a neighboring block is coded in an inter-coded mode or an intra-block copy mode, it is considered a non-directional mode. In another example, if a neighboring block is an intra-inter mode, the intra-prediction mode used in the intra-inter mode is used to derive a context or determine a selection of a context from a set of contexts.

[0134] In various embodiments, step 1220 may include deriving a context for a set index and a mode index based on a size of the block. For example, the context for signaling an intra mode set index and / or a mode index within an intra mode set depends on the block size of the current block. The block size may refer to the block width of the current block, the block height of the current block, the block area size (or the width x height of the current block), the sum of the width and height of the current block, the maximum value between the width and height of the current block, or the minimum value between the width and height of the current block.

[0135] In some embodiments, deriving a context for a set index and a mode index based on a size of the block comprises deriving a first context as the context for a set index and a mode index in response to the size of the block being less than or equal to a first threshold, and / or deriving a second context as the context for a set index and a mode index in response to the size of the block being greater than a first threshold, e.g., the first threshold is 8×8.

[0136] In some implementations, if the block size of the current block is 8x8 or less, the first context (e.g., with a context index value of 0) is selected as the context for the set index and mode index of the current block, otherwise the second context (e.g., with a context index value of 1) is selected as the context for the set index and mode index of the current block.

[0137] In some embodiments, deriving a context for a set index and a mode index based on a size of the block comprises deriving a first context as a context for a set index and a mode index in response to the size of the block being smaller than a first threshold, deriving a second context as a context for a set index and a mode index in response to the size of the block being equal to or greater than the first threshold and the size of the block being smaller than a second threshold, deriving a third context as a context for a set index and a mode index in response to the size of the block being equal to or greater than the second threshold and the size of the block being smaller than a third threshold, and / or deriving a fourth context as a context for a set index and a mode index in response to the size of the block being equal to or greater than a third threshold. For example, the first threshold is 64 (or 8×8) in area, the second threshold is 256 (or 16×16) in area, and / or the third threshold is 1024 (or 32×32) in area.

[0138] In some implementations, if the block area size of the current block is smaller than a first threshold (T1), a first context (e.g., having a context index value of 0) is selected as the context for the set index and mode index of the current block. If the block area size of the current block is equal to or larger than T1 but smaller than a second threshold (T2), a second context (e.g., having a context index value of 1) is selected as the context for the set index and mode index of the current block. If the block area size of the current block is equal to or larger than T2 but smaller than a third threshold (T3), a third context (e.g., having a context index value of 2) is selected as the context for the set index and mode index of the current block. Otherwise, a fourth context (e.g., having a context index value of 3) is selected as the context for the set index and mode index of the current block. For example, T1 is set to 64, T2 is set to 256, and / or T3 is set to 1024. In another example, T1 may be set to 16, T2 may be set to 64, and T3 may be set to 256.

[0139] In various embodiments, method 1200 may further optionally include determining, by the device, a set of permissible intra-prediction modes for the block based on at least one of mode information of the at least one neighboring block of the block or a size of the block.

[0140] In some implementations, the number of intra-prediction modes allowed for a block may depend on the intra-prediction modes of its neighboring blocks and / or the block size of the current block. In one example, a mode set is selected from all available mode sets according to the intra-prediction modes of its neighboring blocks and / or the block size of the current block, and only the intra-prediction modes in the selected mode set can be used to code the current block.

[0141] In various embodiments, determining a set of permissible intra-prediction modes for the block includes: determining a set of permissible intra-prediction modes for the block having the non-directional prediction mode and a nominal directional mode in response to the number of non-directional prediction modes for the at least one neighboring block being 2 or more; determining a set of permissible intra-prediction modes for the block having the non-directional prediction mode and a nominal directional mode in response to the number of non-directional prediction modes for the at least one neighboring block being 2 or more and in response to a size of the block being equal to or greater than a first threshold; and / or determining a set of permissible intra-prediction modes for the block having all intra-prediction modes in response to the number of directional prediction modes for the at least one neighboring block being 1 or more. For example, the first threshold is 8×8, 16×16, or 32×32.

[0142] In some implementations, if at least two of the intra prediction modes of its neighboring blocks are non-directional modes, then only those non-directional modes and the nominal directional mode are allowed for the current block.

[0143] In some other implementations, if at least two of the intra prediction modes of its neighboring blocks are non-directional modes and / or the block size of the current block is equal to or larger than a threshold (TH1), only the non-directional modes and the nominal directional modes are allowed for the current block. In one example, TH1 is set to 32×32. In another example, TH1 may be set to 8×8, 16×16, or 64×64.

[0144] In some other implementations, all intra prediction modes available for the current block are allowed if at least one (or two) of its neighboring blocks' intra prediction modes is a directional intra prediction mode.

[0145] In various embodiments, method 1200 may further optionally include determining, by the device, two or more sets of allowable intra-prediction modes for the block based on at least one of mode information of at least one neighboring block of the block or a size of the block. In some implementations, two or more mode sets may be selected from all available mode sets depending on the intra-prediction modes of the neighboring blocks and / or the block size of the current block, and only intra-prediction modes in the selected two or more mode sets may be used to code the current block.

[0146] In some implementations, the neighboring blocks of the current block may include one or more of the top (upper or upper) block of the current block, the block to the left of the current block, and the top left (upper or upper left) block of the current block.

[0147] Referring to step 1240, the device may determine an intra-prediction mode of the block based on a set index and a mode index. In some implementations, the device may build a list of intra-modes of the block according to a predefined rule based on the size of the block, and / or the device may divide the intra-modes of the list into a plurality of intra-mode sets for the block. The predefined rule may be one of a set of predefined rules, and may be selected from the set of predefined rules based on the size of the block. In some other implementations, an intra-mode list is built for the current block based on the predefined rule. After building the intra-mode list, the intra-prediction modes are divided into a plurality of intra-prediction mode sets according to their corresponding indexes in the intra-mode list. To signal the intra-prediction mode of the current block, first, a mode set index indicating a mode set index of the intra-prediction mode of the current block may be signaled, and then a mode index indicating an index of the intra-prediction mode in the mode set may be signaled. The predefined rules may be different for different block sizes, which may be indicated by the number of samples in the block.

[0148] The embodiments of the present disclosure may be used separately or in combination in any order. Also, each of these methods (or embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to a luma block or a chroma block, and in a chroma block, the embodiments may be applied to two or more color components separately or to two or more color components together.

[0149] In the present disclosure, the steps or actions in the various embodiments may be combined in any quantity or order as desired. In the present disclosure, two or more steps or actions in the various embodiments may be performed in parallel.

[0150] The techniques described above can be implemented as computer software with computer readable instructions physically stored on one or more computer readable media. For example, Figure 14 illustrates a computer system (2600) suitable for implementing certain embodiments of the disclosed subject matter.

[0151] Computer software may be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similarly subjected to code having instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), and the like, either directly or via interpretation, microcode execution, and the like.

[0152] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0153] 14 with respect to computer system (2600) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement related to any one or combination of components illustrated in this exemplary embodiment of computer system (2600).

[0154] The computer system (2600) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, moving a data glove, etc.), audio input (e.g., voice, clapping, etc.), visual input (e.g., gestures, etc.), and olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds, etc.), images (e.g., scanned images, photographic images obtained from a still camera, etc.), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0155] The input human interface devices may include one or more of a keyboard (2601), a mouse (2602), a trackpad (2603), a touch screen (2610), a data glove (not shown), a joystick (2605), a microphone (2606), a scanner (2607), and a camera (2608) (only one of each shown).

[0156] The computer system (2600) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (2610), data gloves (not shown), or joystick (2605), although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (2609), headphones (not shown), etc.), visual output devices (e.g., screens (2610) including CRT screens, LCD screens, plasma screens, OLED screens (each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, or output in four or more dimensions through means such as stereoscopic output), virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), etc.), and printers (not shown).

[0157] The computer system (2600) may also include human-accessible storage devices and their associated media, such as optical media including, for example, a CD / DVD ROM / RW (2620) with CD / DVD or similar media (2621), thumb drives (2622), removable hard drives or solid state drives (2623), legacy magnetic media such as tape and floppy disks (registered trademark, not shown), specialized ROM / ASIC / PLD based devices (not shown) such as security dongles, and the like.

[0158] Those skilled in the art will also appreciate that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transitory signals.

[0159] The computer system (2600) may also include an interface (2654) to one or more communication networks (2655). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks, such as Ethernet, wireless LANs, cellular networks, including GSM, 3G, 4G, 5G, LTE, and the like, TV wired or wireless wide area digital networks, including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial, including CANbus, and the like. Certain networks typically require an external network interface adapter that is attached to a particular general-purpose data port or peripheral bus (2649) (e.g., a USB port of the computer system (2600)), while others are typically integrated into the core of the computer system (2600) by attachment to a system bus, as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2600) can communicate with other entities. Such communications may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to a particular CANbus device), or two-way, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used on each of the networks and network interfaces, as described above.

[0160] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (2640) of the computer system (2600).

[0161] The cores (2640) may include one or more central processing units (CPUs) (2641), graphics processing units (GPUs) (2642), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (2643), hardware accelerators for specific tasks (2644), graphics adapters (2650), etc. These devices may be connected via a system bus (2648) along with read only memory (ROM) (2645), random access memory (2646), internal mass storage (2647), such as an internal non-user accessible hard drive, SSD, and the like (2647). In some computer systems, the system bus (2648) may be made accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached either directly to the core's system bus (2648) or via a peripheral bus (2649). In one example, the screen 2610 can be connected to a graphics adapter 2650. Peripheral bus architectures include PCI, USB, and the like.

[0162] The CPU (2641), GPU (2642), FPGA (2643), and accelerator (2644) may execute certain instructions that may combine to constitute the aforementioned computer code. The computer code may be stored in a ROM (2645) or a RAM (2646). Transient data may also be stored in the RAM (2646), while permanent data may be stored, for example, in an internal mass storage (2647). Rapid storage and retrieval from any of the memory devices may be enabled by the use of cache memories that may be associated with one or more of the CPU (2641), GPU (2642), mass storage (2647), ROM (2645), RAM (2646), and the like.

[0163] The computer-readable media can have computer code thereon for performing various computer-implemented processes. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0164] As a non-limiting example, a computer system having the architecture (2600), and in particular the core (2640), can provide functionality as a result of the processor(s) (including CPUs, GPUs, FPGAs, accelerators, and the like) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with the specific storage of the core (2640) that is non-transitory in nature, such as the core's internal mass storage (2647) or ROM (2645), and the user-accessible mass storage as introduced above. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2640). The computer-readable media can include one or more memory devices or chips according to specific needs. The software may cause the core (2640) and in particular the processors therein (including CPUs, GPUs, FPGAs, and the like) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (2646) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerators (2644)) that may operate in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. Reference to software includes logic, and vice versa, where appropriate. Reference to a computer-readable medium may include circuitry (e.g., an integrated circuit (IC) or the like) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure includes any suitable combination of hardware and software.

[0165] Although a particular invention has been described with reference to exemplary embodiments, this description is not meant to be limiting. Various modifications of these exemplary embodiments and further embodiments of the invention will become apparent to those skilled in the art from this description. Those skilled in the art will readily recognize that these and various other modifications may be made to the exemplary embodiments shown and described herein without departing from the spirit and scope of the invention. Accordingly, the appended claims are intended to cover such modifications and alternative embodiments. Certain proportions may be exaggerated and other proportions may be minimized in the illustrations. Accordingly, the present disclosure and figures should be regarded as illustrative and not restrictive.

Claims

1. 1. A method for entropy coding for intra-prediction modes in video coding, the method being performed by an apparatus having a memory storing instructions and a processor in communication with the memory, the method comprising: deriving, for a block in a video sequence, at least one context for a set index and a mode index of the block based on at least one of information of the block and mode information of at least one neighboring block of the block; the set index indicates an intra-mode set from a plurality of intra-mode sets; deriving the mode index, wherein the mode index indicates an intra-prediction mode from the intra mode set; encoding the set index and the mode index according to the at least one context; encoding the block in an intra prediction mode determined based on the set index and the mode index; and The step of deriving at least one context related to the set index and the mode index comprises: deriving a context for the set index and the mode index based on a number of the at least one neighboring block coded in a directional intra-prediction mode; having method.

2. the information for the block comprises a size of the block; the mode information of the at least one neighboring block includes at least one of a number of multiple direction modes of the at least one neighboring block, or a mode set index of the at least one neighboring block; The method of claim 1.

3. The deriving of the context for the set index and the mode index based on the number of the at least one neighboring block coded in the directional intra-prediction mode comprises: deriving a first context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being zero; deriving a second context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being 1; and deriving a third context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being greater than one; 2. The method of claim 1, comprising:

4. The deriving of the context for the set index and the mode index based on the number of the at least one neighboring block coded in the directional intra-prediction mode comprises: deriving a first context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being zero; deriving a second context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being 1; deriving a third context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being two and the at least one neighboring block being coded in the same directional intra-prediction mode; and deriving a fourth context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being two and the at least one neighboring block being coded in a different directional intra-prediction mode, or in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being more than two; 2. The method of claim 1, comprising:

5. The deriving of the context for the set index and the mode index based on the number of the at least one neighboring block coded in the directional intra-prediction mode comprises: deriving a first context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being zero; deriving a second context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being 1; deriving a third context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being two; and deriving a fourth context as the context related to the set index and the mode index in response to the number of the at least one neighboring block coded in the directional intra-prediction mode being greater than two; 2. The method of claim 1, comprising:

6. The method of claim 1 , wherein the directional intra-prediction modes include at least one of an inter-coded mode or an intra-block copy mode.

7. The method of claim 1 , wherein the directional intra-prediction modes include intra-inter modes having a directional intra-mode component.

8. The step of deriving the context for the set index and the mode index further comprises: deriving the context for the set index and the mode index based on a size of the block; 2. The method of claim 1, comprising:

9. deriving the context for the set index and the mode index based on the size of the block, deriving a first context as the context for the set index and the mode index in response to the size of the block being less than or equal to a first threshold; and deriving a second context as the context for the set index and the mode index in response to the size of the block being greater than the first threshold; 9. The method of claim 8, comprising:

10. The method of claim 9 , wherein the first threshold is 8×8.

11. deriving the context for the set index and the mode index based on the size of the block, deriving a first context as the context for the set index and the mode index in response to the size of the block being less than a first threshold; deriving a second context as the context for the set index and the mode index in response to the size of the block being greater than or equal to the first threshold and the size of the block being less than a second threshold; deriving a third context as the context for the set index and the mode index in response to the size of the block being greater than or equal to the second threshold and the size of the block being less than a third threshold; and deriving a fourth context as the context related to the set index and the mode index in response to the size of the block being equal to or greater than the third threshold; 9. The method of claim 8, comprising:

12. the first threshold is 64 in area; the second threshold is 256 in area; and the third threshold is 1024 in area; The method of claim 11.

13. determining a set of allowable intra-prediction modes for the block based on at least one of the mode information of the at least one neighboring block of the block or a size of the block; The method of claim 1 further comprising:

14. Determining the set of allowable intra-prediction modes for the block comprises: determining the set of allowable intra-prediction modes for the block comprising the non-directional prediction mode and a nominal directional mode in response to a number of non-directional prediction modes for the at least one neighboring block being greater than or equal to two; determining the set of allowable intra-prediction modes for the block comprising the non-directional prediction mode and a nominal directional mode in response to a number of non-directional prediction modes for the at least one neighboring block being two or more and in response to the size of the block being greater than or equal to a first threshold; and determining the set of allowable intra-prediction modes for the block comprising all intra-prediction modes in response to a number of directional prediction modes for the at least one neighboring block being greater than or equal to one; 14. The method of claim 13, comprising:

15. The method of claim 14 , wherein the first threshold is 32×32.

16. determining two or more sets of allowable intra-prediction modes for the block based on at least one of the mode information of at least one neighboring block of the block or a size of the block; The method of claim 1 further comprising:

17. 1. A method for video encoding, the method being performed by an apparatus having a memory storing instructions and a processor in communication with the memory, deriving, for a block in a video sequence, at least one context for a set index and a mode index of the block based on at least one of information of the block and mode information of at least one neighboring block of the block; the set index indicates an intra-mode set from a plurality of intra-mode sets; deriving the mode index, wherein the mode index indicates an intra-prediction mode from the intra mode set; encoding the set index and the mode index into a coded video bitstream according to the at least one context; encoding the block into the coded video bitstream in an intra-prediction mode determined based on the set index and the mode index; transmitting the encoded video bitstream; and The step of deriving at least one context related to the set index and the mode index comprises: deriving a context for the set index and the mode index based on a number of the at least one neighboring block coded in a directional intra-prediction mode; having method.

18. 1. A method for video encoding, the method being performed by an apparatus having a memory storing instructions and a processor in communication with the memory, deriving, for a block in a video sequence, at least one context for a set index and a mode index of the block based on at least one of information of the block and mode information of at least one neighboring block of the block; the set index indicates an intra-mode set from a plurality of intra-mode sets; deriving the mode index, wherein the mode index indicates an intra-prediction mode from the intra mode set; encoding the set index and the mode index into a coded video bitstream according to the at least one context; encoding the block into the coded video bitstream in an intra-prediction mode determined based on the set index and the mode index; storing the encoded video bitstream on a computer-readable storage medium; and The step of deriving at least one context related to the set index and the mode index comprises: deriving a context for the set index and the mode index based on a number of the at least one neighboring block coded in a directional intra-prediction mode; having method.

19. 1. An apparatus for entropy coding for intra prediction modes in video coding, comprising: a memory for storing instructions; a processor in communication with the memory, the processor being configured, when executing the instructions, to cause the apparatus to perform the method of any of claims 1 to 18; Device.

20. A computer program causing a computer to carry out the method according to any one of claims 1 to 18.

Images