Hardware-friendly design for intra mode coding

Abstract

To provide methods, apparatuses, and computer readable storage medium for intra prediction mode coding in video decoding.SOLUTION: A method includes the steps of: receiving a coded video bitstream; constructing a list of intra modes for a block in the coded video bitstream; dividing the list of intra modes into a plurality of intra mode sets for the block; extracting, from the coded video bitstream, a set index indicating an intra mode set from the plurality of intra mode sets; extracting, from the coded video bitstream, a mode index indicating an intra prediction mode from the intra mode set; determining the intra prediction mode for the block based on the set index and the mode index; and decoding the coded video bitstream based on the intra prediction mode.SELECTED DRAWING: Figure 15

Description

Technical Field

[0001] [Related Applications] This application claims the benefit of priority based on U.S. Provisional Application No. 63 / 236,542, filed on Aug. 24, 2021, and U.S. Non-Provisional Application No. 17 / 573,856, filed on Jan. 12, 2022. The entire contents of both applications are incorporated by reference.

[0002] [Technical Field] The present disclosure relates to video encoding and / or decoding techniques, and more particularly to an improved design and signaling for intra prediction mode encoding.

Background Art

[0003] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the inventors named in this application, to the extent that the work is described in this background section, and aspects of this description that may not be eligible as prior art at the time of filing of this application in other respects are not admitted as prior art to the present disclosure, either expressly or implicitly.

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each picture having spatial dimensions, for example, of 1920×1080 luminance samples and associated full or sub-sampled chrominance samples. A series of pictures can have a fixed or variable picture rate (or frame rate), for example, a picture rate of 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, video having a pixel resolution of 1920×1080, a frame rate of 60 frames / second, and 4:2:0 chroma sub-sampling with 8 bits per pixel per color channel requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires a storage space of over 600 GB.

[0005] One purpose of video encoding and decoding can be the reduction of redundancy in an uncompressed input video signal by compression. Compression can help reduce the above bandwidth and / or storage space requirements, possibly by more than an order of magnitude in some cases. Both reversible compression and irreversible compression, as well as combinations thereof, can be used. Reversible compression refers to a technique where, through a decoding process, an exact copy of the original signal can be reconstructed from the compressed original signal. Irreversible compression refers to an encoding / decoding process where the original video information is not fully retained during encoding and cannot be fully recovered during decoding. When using irreversible compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough to usefully render the reconstructed signal for its intended use despite some information loss. In the case of video, irreversible compression is widely used in many applications. The amount of acceptable distortion depends on the application. For example, users of certain consumer video streaming applications may tolerate higher distortion than users of movie or television broadcast applications. The compression ratio achievable by a particular encoding algorithm can be selected or adjusted to reflect various distortion tolerances, and generally, higher acceptable distortion allows encoding algorithms that result in higher loss and higher compression ratios.

[0006] Video encoders and decoders can utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy encoding.

[0007] Video codec technology can include techniques known as intra coding. In intra coding, sample values are represented without reference to samples from previously reconstructed reference pictures or other data. In some video codecs, a picture is spatially divided into blocks of samples. If all blocks of samples are coded in an intra mode, that picture can be called an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as the first picture in an encoded video bitstream and video session or as a still image. Then, the samples of a block after intra prediction can be subjected to a transform to the frequency domain, and the transform coefficients so generated can be quantized prior to entropy coding. Intra prediction represents techniques for minimizing sample values in the pre-transform domain. In some cases, the smaller the DC value and AC coefficients after transformation, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0008] Traditional intra coding, such as that known from MPEG-2 generation coding techniques, does not use intra prediction. However, some newer video compression techniques include techniques that attempt to encode / decoder a block based on surrounding sample data and / or metadata that precede the block in decoding order and that are obtained during encoding and / or decoding of spatially adjacent ones that are intra coded or decoded. Such techniques are hereinafter referred to as "intra prediction" techniques. Note that in at least some cases, intra prediction uses only reference data from the current picture being reconstructed and does not use reference data from other reference pictures.

[0009] There can be various forms of intra prediction. In a given video coding technology, if two or more such technologies are available, the technology used can be called an intra prediction mode. One or more intra prediction modes may be provided in a particular codec. In certain cases, the mode can have sub - modes and / or be associated with various parameters, and the mode / sub - mode information and intra - coding parameters of a video block can be encoded individually or can be included together in a mode codeword. Which codeword to use for a given combination of mode, sub - mode, and / or parameters can affect the coding efficiency gain through intra prediction, and the entropy coding technology used to convert the codeword into the bitstream can similarly have an impact.

[0010] A particular mode of intra prediction was introduced in H.264, refined in H.265, and further refined in newer coding technologies such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). Generally for intra prediction, the predictor block can be formed using the available adjacent sample values. For example, the available values of a particular set of adjacent samples along a particular direction and / or line may be copied into the predictor block. The reference to the direction used can be encoded in the bitstream or can itself be predicted.

[0011] Referring to FIG. 1A, in the lower right, a subset of 9 predictor directions specified in 33 possible intra predictor directions of H.265 (corresponding to 33 angular modes out of 35 intra modes specified in H.265) is depicted. The point (101) where the arrows converge represents the predicted sample. The arrows represent the direction when adjacent samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from the adjacent sample(s) at a 45-degree angle to the upper right from the horizontal direction. Similarly, arrow (103) indicates that sample (101) is predicted from the adjacent sample(s) at a 22.5-degree angle to the lower left of sample (101) from the horizontal direction.

[0012] Continuing to refer to FIG. 1A, in the upper left, a square block (104) of 4×4 samples is depicted (indicated by the thick dashed line). The square block (104) contains 16 samples, and each sample is labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample from the top in the Y dimension and the first sample from the left in the X dimension. Similarly, sample S44 is the fourth sample within block (104) in both the Y and X dimensions. Since the block is of size 4×4 samples, S44 is in the lower right. Further, exemplary reference samples following a similar numbering scheme are shown. The reference samples are labeled with "R" and its Y position (e.g., row index) and X position (column index) with respect to block (104). In both H.264 and H.265, adjacent predicted samples in the vicinity of the block being reconstructed are used.

[0013] The intra-picture prediction of block 104 may begin by copying the reference sample value from adjacent samples according to the predicted direction signaled. For example, assume that the encoded video bitstream includes signaling indicating the prediction direction of arrow (102) for this block 104. That is, the sample is predicted from the upper right prediction sample(s) at an angle of 45 degrees from the horizontal direction. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the direction is not divisible by 45 degrees, the values of multiple reference samples can be combined, for example, by interpolation, to calculate the reference sample.

[0015] As video encoding technology continues to develop, the number of possible directions has increased. In H.264 (2003), for example, nine different directions are available for intra-picture prediction. This increased to 33 in H.265 (2013), and JEM / VVC / BMS at the time of this disclosure can support up to 65 directions. Experiments have been conducted to help identify the most appropriate intra-prediction direction, and specific techniques in entropy coding may be used to encode these most appropriate directions with a small number of bits while accepting a specific bit penalty for the direction. Further, in some cases, the direction itself can be predicted from the adjacent direction used in the intra-prediction of the decoded adjacent block.

[0016] FIG. 1B shows a schematic diagram (180) depicting 65 intra-prediction directions by JEM to show the increasing number of prediction directions in various encoding technologies developed over time.

[0017] The method of mapping intra prediction direction bits to a prediction direction in a symbolized video bitstream may vary for each video encoding technique. For example, it can range from a simple direct mapping of the prediction direction to an intra prediction mode, to a complex adaptation method involving codewords, the most probable mode, and similar techniques. However, in all cases, in video content, there may exist a specific direction of intra prediction that is statistically less likely to occur than certain other directions. Since the goal of video compression is to reduce redundancy, in a well-designed video encoding technique, these less likely methods may be represented by a larger number of bits than the more likely directions.

[0018] Inter-picture prediction or inter prediction may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or a part thereof (reference picture) is spatially shifted in the direction indicated by a motion vector (hereinafter, MV), and then used for the prediction of a newly reconstructed picture or a part thereof (e.g., a block). In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV may have two dimensions of X and Y, or three dimensions, and the third dimension is an indication of the reference picture used (similar to the temporal dimension).

[0019] In some video compression techniques, the current MV applicable to a particular region of sample data can be predicted from other MVs, for example, from other MVs related to other regions of sample data that are spatially adjacent to the region being reconstructed and that precede the current MV in decoding order. By doing so, by relying on reducing redundancy in the related MVs, the overall amount of data required for encoding the MVs can be substantially reduced, thereby increasing the compression efficiency. MV prediction can work effectively, for example, when encoding an input video signal derived from a camera (known as natural video), because regions larger than the regions where a single MV is applicable in the video sequence move in a similar direction and thus, in some cases, there is a statistical likelihood that similar motion vectors derived from the MVs of adjacent regions can be used for prediction. As a result, the actual MV for a given region will be similar or identical to the MV predicted from surrounding MVs. Such MVs may then be represented in fewer bits than would be used if the MVs were directly encoded rather than predicted from adjacent MVs after entropy encoding. In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, the MV prediction itself may be lossy, for example, due to rounding errors when calculating predictors from some surrounding MVs.

[0020] H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016) describes various MV prediction mechanisms. Among the many MV prediction mechanisms specified by H.265, the technique hereinafter referred to as "spatial merge" is described in this specification.

[0021] Specifically, referring to FIG. 2, the current block (201) contains samples found by the encoder during the motion search process that the current block is predictable from the previous block of the same size that has been spatially shifted. Instead of directly encoding the MV, the MV can be derived from metadata associated with one or more reference pictures, for example from the latest reference picture (in decoding order), using an MV associated with any of five surrounding samples denoted as A0, A1, and B0, B1, B2 (202-206 respectively). In H.265, MV prediction can use a predictor from the same reference picture that the neighboring block uses. SUMMARY OF THE INVENTION

[0022] The present disclosure describes various embodiments of methods, apparatuses, and computer-readable storage media for video encoding and / or decoding.

[0023] According to one aspect, an embodiment of the present disclosure provides a method for intra prediction mode encoding in video decoding. The method includes receiving, by a device, an encoded video bitstream. The device includes a memory storing instructions and a processor communicating with the memory. The method also includes constructing, by the device, a list of intra modes of blocks in the encoded video bitstream according to a predetermined rule based on the size of the blocks; splitting, by the device, the list of intra modes into a plurality of intra mode sets of the blocks; extracting, by the device, a set index indicating an intra mode set from the plurality of intra mode sets from the encoded video bitstream; extracting, by the device, a mode index indicating an intra prediction mode from the intra mode set from the encoded video bitstream; determining, by the device, an intra prediction mode of the blocks based on the set index and the mode index; and decoding, by the device, the encoded video bitstream based on the intra prediction mode.

[0024] According to another aspect, an embodiment of the present disclosure provides an apparatus for video encoding and / or decoding. The apparatus includes a memory that stores instructions and a processor that communicates with the memory. When the processor executes the instructions, the processor is configured to cause the apparatus to execute the above method for video decoding and / or encoding.

[0025] In another aspect, an embodiment of the present disclosure provides a non-transitory computer-readable medium that stores instructions which, when executed by a computer for video decoding and / or encoding, cause the computer to execute the above method for video decoding and / or encoding.

[0026] The above and other aspects and their implementations are described in more detail in the drawings, the detailed description, and the claims.

Brief Description of the Drawings

[0027] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14A

Figure 14B

Figure 15

Figure 16

[0028] Hereinafter, the present invention will be described in detail with reference to the accompanying drawings which form a part of the present invention and show specific examples of embodiments as examples. However, note that the present invention may be embodied in various different forms. Therefore, it is intended that the subject matter or the claimed subject matter is not limited to any of the embodiments shown below. Also note that the present invention may be embodied as a method, a device, a component or a system. Therefore, the embodiments of the present invention may take the form of, for example, hardware, software, firmware or any combination thereof.

[0029] Throughout the specification and the claims, terms may have meanings that go beyond their explicitly stated meanings and have nuances suggested or implied in the context. The phrases "in one embodiment" or "in some embodiments" as used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" as used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" as used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" as used herein do not necessarily refer to different implementations. For example, the subject matter of the claims is intended to include, in whole or in part, combinations of exemplary embodiments / implementations.

[0030] Generally, terms may be understood, at least in part, from their use in context. For example, terms such as "and", "or" or "and / or" used herein may include a variety of meanings that may depend, at least in part, on the context in which such terms are used. Ordinarily, "or" as used to associate a list such as A, B or C is intended to mean here A, B and C in an inclusive sense and A, B or C in an exclusive sense here. Further, the terms "one or more" or "at least one" as used herein may, at least in part, depend on context, be used to describe any feature, structure or characteristic in a single sense or, alternatively, be used to describe combinations of features, structures or characteristics in a plurality of senses. Similarly, terms in the singular form (such as "a", "an" or "the", etc.) may, likewise, be understood, at least in part, from context, as conveying either a singular or plural usage. Further, the terms "based on" or "determined by" may not necessarily be intended to convey an exclusive set of factors and, instead, may, in this case as well, at least in part, depend on context and allow for the presence of additional factors not necessarily explicitly described.

[0031] Figure 3 shows a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via a network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) may perform unidirectional transmission of data. For example, the terminal device (310) may encode video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to restore the video pictures, and display the video pictures according to the restored video data. Unidirectional data transmission may be implemented in a media service application or the like.

[0032] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which may be implemented, for example, during a video conferencing application. For bidirectional data transmission, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) may receive the encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to restore the video pictures, and display the video pictures on an accessible display device according to the restored video data.

[0033] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) may be implemented as servers, personal computers, and smartphones, but the applicability of the underlying principles of the present disclosure may not be limited thereto. Embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, and / or dedicated video conferencing facilities. The network (350) represents any number or type of network that transmits encoded video data among the terminal devices (310), (320), (330), and (340), including, for example, wired (wiring) and / or wireless communication networks. The communication network (350) may exchange data in circuit switching, packet switching, and / or another type of channel. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of the discussion herein, the architecture and topology of the network (350) may not be important for the operation of the present disclosure, unless explicitly described below.

[0034] FIG. 4 shows the arrangement of a video encoder and a video decoder in a video streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter may be equally applicable to other video applications, including, for example, video conferencing, digital TV broadcasting, games, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0035] The video streaming system can include a video source (401), such as a digital camera, and may include a video capture subsystem (413) that generates a stream (402) of, for example, uncompressed video pictures or images. In one example, the stream (402) of video pictures includes samples recorded by the digital camera of the video source 401. The stream (402) of video pictures, drawn as a thick line to emphasize the high data volume when compared to the encoded video data (404) (or encoded video bitstream), can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as will be described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), drawn as a thin line to emphasize the lower data volume when compared to the stream (402) of uncompressed video pictures, can be stored in a streaming server (405) for future use, or can be stored directly in a downstream video device (not shown). One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) within an electronic device (430). The video decoder (410) decodes an input copy (407) of the encoded video data and generates an output stream (411) of uncompressed video pictures that can be rendered on a display (412) (such as a display screen) or other rendering device (not shown). The video decoder 410 may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) can be encoded according to a particular video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In one example, a video encoding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC and other video encoding standards.

[0036] Note that the electronic devices (420) and (430) can include other components (not shown). For example, the electronic device (420) can include a video decoder (not shown), and the electronic device (430) can also include a video encoder (not shown).

[0037] FIG. 5 shows a block diagram of a video decoder (510) according to any embodiment of the present disclosure below. The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used in place of the video decoder (310) in the example of FIG. 4.

[0038] Receiver (531) may receive one or more encoded video sequences to be decoded by video decoder (510). In the same or another embodiment, one encoded video sequence may be decoded at a time, and the decoding of each encoded video sequence is independent of other encoded video sequences. Each video sequence may be associated with a plurality of video frames or pictures. The encoded video sequence may be received from channel (501), which may be a hardware / software link to a storage device storing encoded video data or a streaming source transmitting encoded video data. Receiver (531) may receive the encoded video data together with other data such as encoded audio data and / or auxiliary data streams, and these data may be transferred to respective processing circuits (not shown). Receiver (531) can separate the encoded video sequence from other data. As a countermeasure against network jitter, buffer memory (515) may be disposed between receiver (531) and entropy decoder / parser (520) (hereinafter “parser”). In certain applications, buffer memory (515) may be implemented as part of video decoder (510). In other applications, it can exist separately outside video decoder (510) (not shown). In still other applications, for example, to counter network jitter, a buffer memory (not shown) may exist outside video decoder (510), and further, for example, to handle playback timing, another additional buffer memory (515) may exist inside video decoder (510). If receiver (531) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isochronous network, buffer memory (515) may not be needed or may be small. For use in a best-effort packet network such as the Internet, a buffer memory (515) of sufficient size may be required, and its size is relatively large.Such a buffer memory may be implemented in an adaptive size and may be implemented, at least in part, in an operating system or similar element (not shown) external to the video decoder (510).

[0039] Video decoder (510) may include a parser (520) for reconstructing symbols (521) from an encoded video sequence. The categories of these symbols include information used to manage the operation of video decoder (510) and potentially information for controlling a rendering device such as display (512) (e.g., display screen). The display may or may not be an integral part of electronic device (530) and can be coupled to electronic device (530) as shown in FIG. 5. The control information for the rendering device(s) may be in the form of Supplementary Enhancement Information (SEI message) or a Video Usability Information (VUI) parameter set fragment (not shown). Parser (520) can parse / entropy decode the encoded video sequence received by parser (520). The entropy encoding of the encoded video sequence can follow video encoding techniques or standards and can follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser (520) can extract a set of subgroup parameters for at least one of the subgroups of pixels within the video decoder from the encoded video sequence based on at least one parameter corresponding to the subgroup. The subgroups can include Group of Pictures (GOP), picture, tile, slice, macroblock, Coding Unit (CU), block, Transform Unit (TU), Prediction Unit (PU), etc. Parser (520) can also extract information such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0040] The parser (520) can perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (515), thereby generating symbols (521).

[0041] The reconstruction of the symbols (521) can involve multiple different processes or functional units depending on the type of the encoded video picture or a portion thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. The units involved and how they are involved may be controlled by subgroup control information parsed by the parser (520) from the encoded video sequence. Such a flow of subgroup control information between the parser (520) and the multiple processes or functional units described below is not depicted for the sake of brevity.

[0042] In addition to the functional blocks already described, the video decoder (510) can conceptually be divided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these functional units can interact closely with each other and can be at least partially integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed subject matter, a conceptual subdivision into functional units is adopted in the following disclosure.

[0043] The first unit may include a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) may receive quantized transform coefficients and control information as symbols (singular or plural) (521) from the parser (520). The control information includes information indicating which type of inverse transform to use, block size, quantization coefficient / parameter, quantization scaling matrix, etc. The scaler / inverse transform unit (551) can output a block including sample values that can be input to the aggregator (555).

[0044] In some cases, the output samples of the scaler / inverse transform (551) can relate to blocks that are intra-coded, i.e., do not use prediction information from previously reconstructed pictures, but can use prediction information from previously reconstructed parts of the current picture. Such prediction information can be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may use surrounding block information that has already been reconstructed and stored in the current picture buffer (558) to generate blocks of the same size and shape as the block being reconstructed. The current picture buffer (558) buffers, for example, the partially reconstructed current picture and / or the fully reconstructed current picture. Depending on the implementation, the aggregator (555) may add, for each sample, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0045] In other cases, the output samples of the scaler / inverse transform unit (551) can relate to inter-coded and potentially motion-compensated blocks. In such cases, the motion compensation prediction unit (553) can access the reference picture memory (557) to retrieve the samples used for inter-picture prediction. After motion-compensating the retrieved samples according to the symbols (521) regarding the block, these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (the output of unit 551 may also be referred to as residual samples or a residual signal), thereby generating output sample information. The address in the reference picture memory (557) from which the motion compensation unit (553) retrieves the prediction samples can be controlled by the motion vectors available to the motion compensation unit (553) in the form of symbols (521). The symbols can have, for example, X, Y components (shifts), and a reference picture component (time). Motion compensation may include interpolation of the sample values fetched from the reference picture memory (557) when exact motion vectors below the sample level are used, and may also be related to motion vector prediction mechanisms and the like.

[0046] The output samples of the aggregator (555) can be subjected to various loop filtering techniques within the loop filter unit (556). Video compression techniques can include in-loop filter techniques. The in-loop filter techniques are controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but can also respond to meta information obtained during the decoding of the previous part (in decoding order) of the coded picture or coded video sequence, and to previously reconstructed and loop-filtered sample values. As will be explained in more detail below, several types of loop filters may be included as part of the loop filter unit 556 in various orders.

[0047] The output of the loop filter unit (556) can be a sample stream, which can be output to the rendering device (512) and can also be stored in the reference picture memory (557) for use in future inter-picture prediction.

[0048] Once a particular encoded picture is fully reconstructed, it can be used as a reference picture for future inter-picture prediction. For example, when the encoded picture corresponding to the current picture is fully reconstructed and the encoded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a fresh current picture buffer can be reallocated before starting the reconstruction of subsequent encoded pictures.

[0049] The video decoder (510) can perform a decoding operation according to a predetermined video compression technique adopted by a standard such as ITU-T Recommendation H.265. The encoded video sequence can conform to the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, the profile can select specific tools from all the tools available in the video compression technique or standard as the tools that are only available for use under that profile. To comply with the standard, the complexity of the encoded video sequence can be within the range defined by the level of the video compression technique or standard. In some cases, the level restricts the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level can, in some cases, be further restricted through the virtual reference decoder (Hypothetical Reference Decoder, HRD) specifications and metadata signaled in the encoded video sequence for HRD buffer management.

[0050] In some exemplary embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data can be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0051] FIG. 6 shows a block diagram of a video encoder (603) according to an exemplary embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0052] The video encoder (603) can receive video samples from a video source (601) that can capture video images to be encoded by the video encoder (603) (which is not part of the electronic device (620) in the example of FIG. 6). In another example, the video source (601) may be implemented as part of the electronic device (620).

[0053] The video source (601) can provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit,...), any color space (e.g., BT.601 YCrCB, RGB, XYZ,...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) may be a storage device capable of storing pre-prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures or images that impart motion when viewed in sequence. Each picture itself may be organized as a spatial array of pixels, and each pixel can include one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0054] According to some exemplary embodiments, a video encoder (603) can encode and compress pictures of a source video sequence in real time or under any other temporal constraints required by an application to produce an encoded video sequence (643). Enforcing an appropriate encoding speed constitutes one function of a controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units as described below. Such couplings are not depicted for simplicity. Parameters set by the controller (650) can include parameters related to rate control (picture skip, quantizer, lambda value of rate-distortion optimization techniques, …), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other suitable functions regarding the video encoder (603) optimized for a particular system design.

[0055] In some exemplary embodiments, the video encoder (603) may be configured to operate in an encoding loop. As a gross simplification for illustration purposes, in one example, the encoding loop may include a source coder (630) (e.g., responsible for generating symbols such as a symbol stream based on an input picture and reference picture(s) to be encoded) and an (local) decoder (633) embedded within the video encoder (603). Even when the embedded decoder 633 processes the encoded video stream without entropy coding by the source coder 630, the decoder (633) reconstructs the symbols to generate sample data in a manner similar to what a (remote) decoder would also generate (in the video compression techniques contemplated in the disclosed subject matter, any compression between the symbols in the entropy coding and the encoded video bit stream can be lossless). The reconstructed sample stream (sample data) is input into the reference picture memory (634). Since the decoding of the symbol stream yields bit-exact results regardless of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-exact between the local encoder and the remote encoder. In other words, the prediction unit of the encoder "sees" the same sample values as reference picture samples that the decoder "sees" when the decoder uses prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift in the event that synchronization cannot be maintained, e.g., due to channel errors) is used to improve the encoding quality.

[0056] The operation of the "local" decoder (633) may be the same as that of the "remote" decoder, such as the video decoder (410), which has already been described in detail above in connection with FIG. 5. However, referring briefly also to FIG. 5, since symbols are available and the encoding / decoding of symbols into an encoded video sequence by the entropy encoder (645) and the parser (420) can be reversible, the entropy decoding section of the video decoder (410) including the buffer memory (415) and the parser (420) may not be fully implemented in the local decoder (633) of the encoder.

[0057] An observation that can be made at this point is that any decoder technology, except for parse / entropy decoding that may exist only within the decoder, may need to exist in substantially the same functional form within the corresponding encoder. For this reason, the disclosed subject matter may sometimes focus on decoder operation. This is similar to the decoding part of the encoder. Therefore, the description of encoder technology can be omitted since it is the reverse of the decoder technology that is comprehensively described. More detailed explanations are provided below only for specific areas or aspects of the encoder.

[0058] During operation, in some exemplary implementations, the source encoder (630) can perform motion-compensated predictive encoding that predictively encodes an input picture by referring to one or more previously encoded pictures from the video sequence designated as "reference pictures". In this way, the encoding engine (632) encodes the difference (or residue) in the color channels between a pixel block of the input picture and a pixel block (s) of the reference picture (s) that can be selected as a prediction reference for the input picture.

[0059] The local video decoder (633) can decode the encoded video data of a picture that can be specified as a reference picture based on the symbols generated by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the encoded video data can be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence can typically be a reproduction of the source video sequence with some errors. The local video decoder (633) can replicate the decoding process that can be performed on the reference picture by the video decoder and cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture that has (in the absence of transmission errors) common content as the reconstructed reference picture that would be obtained by a remote video decoder.

[0060] The predictor (635) can perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) can search the reference picture memory (634) for sample data (as a candidate reference pixel block) or specific metadata, such as reference picture motion vectors, block shapes, etc., that can function as an appropriate prediction reference for the new picture. The predictor (635) can operate on a sample block-by-pixel block basis to find an appropriate prediction reference. In some cases, depending on what is determined by the search result obtained by the predictor (635), the input picture can have a prediction reference drawn from a plurality of reference pictures stored in the reference picture memory (634).

[0061] The controller (650) can manage the encoding operation of the source encoder (630), for example, including setting the parameters and subgroup parameters used to encode the video data.

[0062] The outputs of all of the above functional units can undergo entropy encoding in an entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into an encoded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0063] The transmitter (640) can put the encoded video sequence generated by the entropy encoder (645) into a buffer and prepare it for transmission via a communication channel (660). The communication channel (660) may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) can merge the encoded video data from the video encoder (630) with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown).

[0064] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) can assign a certain encoded picture type to each encoded picture. The encoded picture type can affect the encoding technique that can be applied to each picture. For example, a picture may often be assigned as one of the following picture types.

[0065] An intra picture (I picture) can be encoded and decoded without using other pictures in the sequence as a prediction source. Some video codecs allow different types of intra pictures, for example, including an Independent Decoder Refresh (IDR) picture. Those skilled in the art will recognize these variations of I pictures, as well as their respective uses and characteristics.

[0066] A predicted picture (P picture) can be encoded and decoded using intra prediction or inter prediction that uses at most one motion vector and a reference index to predict the sample values of each block.

[0067] A bi-directionally predicted picture (B picture) can be encoded and decoded using intra prediction or inter prediction that uses at most two motion vectors and reference indices to predict the sample values of each block. Similarly, a multi-predicted picture can use three or more reference pictures and related metadata for the reconstruction of a single block.

[0068] A source picture is typically divided into a plurality of spatially sample-encoded blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and can be encoded block by block. The blocks can be encoded predictively by referring to other (already encoded) blocks, depending on the encoding assignment applied to each picture of the block. For example, the blocks of an I picture may be encoded non-predictively, or may be encoded predictively by referring to already encoded blocks of the same picture (spatial prediction or intra prediction). The pixel blocks of a P picture may be encoded predictively via spatial prediction or via temporal prediction by referring to one previously encoded reference picture. The blocks of a B picture may be encoded predictively via spatial prediction or via temporal prediction by referring to one or two previously encoded reference pictures. A source picture or an intermediate processed picture may be subdivided into other types of blocks for other purposes. As will be described in more detail below, the division of the encoded blocks and other types of blocks may or may not follow the same scheme.

[0069] The video encoder (603) can perform an encoding operation according to a predetermined video encoding technology or standard such as ITU-T Recommendation H.265. In that operation, the video encoder (603) can perform various compression operations including a predictive encoding operation that utilizes temporal and spatial redundancies in the input video sequence. Thus, the encoded video data can conform to the syntax specified by the video encoding technology or standard used.

[0070] In some exemplary embodiments, the transmitter (640) may transmit additional data together with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and the like.

[0071] Video may be captured as a plurality of source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often abbreviated as intra prediction) utilizes spatial correlations within a given picture, and inter-picture prediction utilizes temporal or other correlations between pictures. For example, a particular picture to be encoded / decoded, called the current picture, may be divided into blocks. If a block within the current picture is similar to a reference block within a reference picture that has been previously encoded and is still in the buffer in the video, it may be encoded by a vector called a motion vector. The motion vector points to the reference block within the reference picture and can have a third dimension that specifies the reference picture when multiple reference pictures are used.

[0072] In some exemplary embodiments, bidirectional prediction techniques can be used in inter-picture prediction. According to such bidirectional prediction techniques, two reference pictures such as a first reference picture and a second reference picture that both precede the current picture in decoding order in the video (however, in display order, they may be in the past or future respectively) are used. A block in the current picture can be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block can be predicted together by a combination of the first reference block and the second reference block.

[0073] Furthermore, in order to improve the coding efficiency, merge mode techniques may be used in inter-picture prediction.

[0074] According to some exemplary embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and those CTUs in a picture may have the same size such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU may include three parallel coding tree blocks (CTBs) which are one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree divided into one or more coding units (CUs). For example, a 64×64 pixel CTU can be divided into one 64×64 pixel CU or four 32×32 pixel CUs. Each of one or more 32×32 blocks may be further divided into four 16×16 pixel CUs. In some exemplary embodiments, each CU may be analyzed to determine a prediction type for that CU among various prediction types such as an inter-prediction type or an intra-prediction type during coding. The CU may be divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. The division of the CU into PUs (or PBs of different color channels) may be performed in various division patterns. For example, a luma or chroma PB may include a matrix of values (e.g., luma values) for pixels such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0075] FIG. 7 shows a diagram of a video encoder (703) according to another exemplary embodiment of the present disclosure. The video encoder (703) receives a processing block (e.g., a prediction block) of sample values within a current video picture in a sequence of video pictures and is configured to encode the processing block into an encoded picture that is part of an encoded video sequence. The exemplary video encoder (703) may be used in place of the video encoder (403) in the example of FIG. 4.

[0076] For example, the video encoder (703) receives a matrix of sample values for a processing block such as a prediction block of 8×8 samples. The video encoder (703) then determines, e.g., using rate - distortion optimization (RDO), which of the intra mode, inter mode, or bi - directional prediction mode the processing block is best encoded using. If it is determined that the processing block is to be encoded in the intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into the encoded picture. If it is determined that the processing block is to be encoded in the inter mode or bi - directional prediction mode, the video encoder (703) may use inter prediction techniques or bi - directional prediction techniques, respectively, to encode the processing block into the encoded picture. In some exemplary embodiments, the merge mode may be used as a sub - mode of inter - picture prediction where the motion vector is derived from one or more motion vector predictors but there is no benefit of the encoded motion vector components outside of the predictors. In some exemplary embodiments, there may be motion vector components applicable to the target block. Thus, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module (not shown) for determining the prediction mode of the processing block.

[0077] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in the exemplary arrangement of FIG. 7.

[0078] The inter-encoder (730) receives samples of a current block (e.g., a processing block), compares the block to one or more reference blocks (e.g., blocks in previous and subsequent pictures in display order) in a reference picture, generates inter-prediction information (e.g., a description of redundant information by an inter-coding technique, a motion vector, merge mode information), and is configured to calculate an inter-prediction result (e.g., a predicted block) using any suitable technique based on the inter-prediction information. In some examples, the reference picture is a decoded reference picture decoded using a decoding unit 633 (shown as the residual decoder 728 of FIG. 7, described in more detail below) embedded in the exemplary encoder 620 of FIG. 6 based on the encoded video information.

[0079] The intra-encoder (722) receives samples of a current block (e.g., a processing block), compares the block to blocks already encoded within the same picture, generates quantized coefficients after transformation, and optionally also generates intra-prediction information (e.g., intra-prediction direction information by one or more intra-coding techniques). The intra-encoder (722) may also calculate an intra-prediction result (e.g., a predicted block) based on the intra-prediction information and reference blocks within the same picture.

[0080] The overall controller (721) may be configured to determine overall control data and control other components of the video encoder (703) based on the overall control data. In one example, the overall controller (721) determines a prediction mode of a block and provides a control signal to the switch (726) based on the prediction mode. For example, when the prediction mode is the intra mode, the overall controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723), selects the intra prediction information, and controls the entropy encoder (725) to include the intra prediction information in the bitstream. When the prediction mode of the block is the inter mode, the overall controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723), selects the inter prediction information, and controls the entropy encoder (725) to include the inter prediction information in the bitstream.

[0081] The residual calculator (723) may be configured to calculate the difference (residual data) between the received block and the prediction result of that block selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain and generate transform coefficients. Next, the transform coefficients are subjected to quantization processing to obtain quantized transform coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform inverse transformation to generate decoded residual data. The decoded residual data can be preferably used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and the intra prediction information. The decoded block is preferably processed to generate a decoded picture, and the decoded picture is buffered in a memory circuit (not shown) and can be used as a reference picture.

[0082] The entropy encoder (725) is configured to format the bitstream to include the encoded block and perform entropy encoding. The entropy encoder (725) is configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other suitable information in the bitstream. When encoding a block in either the merge submode of the inter mode or the bidirectional prediction mode, the residual information may not be present.

[0083] FIG. 8 shows a diagram of an exemplary video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive an encoded picture that is part of an encoded video sequence and decode the encoded picture to generate a reconstructed picture. In one example, the video decoder (810) may be used in place of the video decoder (410) in the example of FIG. 4.

[0084] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) coupled together as shown in the exemplary configuration of FIG. 8.

[0085] The entropy decoder (871) can be configured to reconstruct from the encoded picture specific symbols that represent the syntax elements that the encoded picture is composed of. Such symbols can include, for example, the mode in which a block is encoded (e.g., intra mode, inter mode, bi - directional prediction mode, merge sub - mode or another sub - mode), prediction information (e.g., intra prediction information or inter prediction information) that can identify specific samples or metadata used for prediction by the intra decoder (872) or the inter decoder (880), residual information in the form of, for example, quantized transform coefficients, etc. In one example, when the prediction mode is an inter or bi - directional prediction mode, the inter prediction information is provided to the inter decoder (880). When the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can undergo inverse quantization and is provided to the residual decoder (873).

[0086] The inter decoder (880) may be configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0087] The intra decoder (872) may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0088] The residual decoder (873) may be configured to perform inverse quantization to extract dequantized transform coefficients, process the dequantized transform coefficients, and convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize specific control information (including quantization parameter (QP)), and that information may be provided by the entropy decoder (871) (since this is only low-data-volume control information, the data path is not depicted).

[0089] The reconstruction module (874) may be configured to combine, in the spatial domain, the residual output by the residual decoder (873) and the prediction result (output by the intra or inter prediction module as appropriate) to form a reconstructed block that forms a part of the reconstructed picture as a part of the reconstructed video. Note that other suitable operations such as a deblocking operation may also be performed to improve visual quality.

[0090] Note that the video encoders (403), (603), (703) and the video decoders (410), (510), (810) can be implemented using any suitable technology. In some exemplary embodiments, the video encoders (403), (603), (703) and the video decoders (410), (510), (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), (703) and the video decoders (410), (510), (810) can be implemented using one or more processors that execute software instructions.

[0091] When returning to the intra prediction process, samples within a block (e.g., a luma or chroma prediction block, or an encoding block if not further divided into prediction blocks) are predicted by adjacent, next adjacent, or other line or multiple lines, or combinations thereof, to generate a prediction block. Then, the residual between the actual encoded block and the prediction block may be processed through transformation and subsequent quantization. Various intra prediction modes become available, and parameters related to intra mode selection and other parameters may be signaled in the bitstream. For example, the various intra prediction modes may be related to line positions or multiple positions for predicting samples, the direction from which prediction samples are selected from prediction line or multiple lines, and other special intra prediction modes.

[0092] For example, a set of intra prediction modes (also referred to as "intra modes") may include a predetermined number of directional intra prediction modes. As described above in connection with the exemplary implementation of FIG. 1, these intra prediction modes may correspond to a predetermined number of directions in which samples outside the block are selected as predictions for samples predicted within a particular block. In another particular exemplary implementation, eight main directional modes corresponding to angles from 45° to 207° with respect to the horizontal axis may be supported and predefined.

[0093] In some other implementations of intra prediction, to further exploit more diverse spatial redundancy in the directional texture, the directional intra mode may be further extended to more finely - grained set angles. For example, the above - mentioned implementation with 8 angles may be configured to provide 8 nominal angles called V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED as shown in FIG. 9. For each nominal angle, a predetermined number (e.g., 7) of finer angles may be added. Such an extension may make a larger total number (e.g., 56 in this example) of direction angles available for intra prediction corresponding to the same number of predetermined directional intra modes. The prediction angle may be represented as the nominal intra angle plus an angle delta. In the above - specific example with 7 finer angle directions for each nominal angle, the angle delta may be the multiplication of - 3 to 3 with a step size of 3°.

[0094] The above - mentioned directional intra prediction may also be referred to as single - directional intra prediction, which is different from the bi - directional intra prediction (also called intra - bi - directional prediction) described in the latter part of the present disclosure.

[0095] In some implementations, instead of or in addition to the above directional intra modes, a predetermined number of non-directional intra prediction modes may also be predefined and made available. For example, five non-directional intra modes, called smooth intra prediction modes, may be specified. These non-directional intra mode prediction modes may be particularly called DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H intra modes. The prediction of samples of a particular block in these exemplary non-directional modes is shown in FIG. 10. As an example, FIG. 10 shows that the 4×4 block 1002 is predicted by samples from the upper adjacent line and / or the left adjacent line. A particular sample 1010 within block 1002 may correspond to the directly above sample 1004 of sample 1010 in the upper adjacent line of block 1002, the upper left sample 1006 of sample 1010 as the intersection of the upper and left adjacent lines, and the directly left sample 1008 of sample 1010 in the left adjacent line of block 1002. In an exemplary DC intra prediction mode, the average of the left and upper adjacent samples 1008 and 1004 may be used as the predictor for sample 1010. In an exemplary PAETH intra prediction mode, the upper, left, and upper left reference samples 1004, 1008, and 1006 are taken, and then, among these three reference samples, the value closest to (upper + left - upper left) may be set as the predictor for sample 1010. In an exemplary SMOOTH_V intra prediction mode, sample 1010 may be predicted by quadratic interpolation in the vertical direction of the upper left adjacent sample 1006 and the left adjacent sample 1008. In an exemplary SMOOTH_H intra prediction mode, sample 1010 may be predicted by quadratic interpolation in the horizontal direction of the upper left adjacent sample 1006 and the upper adjacent sample 1004. In an exemplary SMOOTH intra prediction mode, sample 1010 may be predicted by the average of quadratic interpolations in the vertical and horizontal directions. The above implementations of the non-directional intra modes are shown merely as non-limiting examples. Other non-directional selections of adjacent lines and other samples, as well as ways of combining prediction samples to predict particular samples within the prediction block, are also contemplated.

[0096] At various coding levels (picture, slice, block, unit, etc.), the selection of a particular intra prediction mode by the encoder from the above-mentioned directional mode or non-directional mode may be signaled in the bitstream. In some exemplary implementations, eight exemplary nominal directional modes (a total of 13 options) may be signaled first, along with five non-angular smooth modes. Then, if the signaled mode is one of the eight nominal angular intra modes, an index is further signaled to indicate the selected angular delta for the corresponding signaled nominal angle. In some other exemplary implementations, all intra prediction modes may be indexed together for signaling (e.g., adding five non-directional modes to 56 directional modes to produce 61 intra prediction modes).

[0097] In some exemplary implementations, the exemplary 56 directional intra prediction modes or other numbers of directional intra prediction modes may be implemented using an integrated directional predictor that projects each sample of the block to a reference subsample position and interpolates the reference samples by a 2-tap bilinear filter.

[0098] In some implementations, an additional filter mode, called FILTER INTRA mode, may be designed to capture the decaying spatial correlation with references on the edge. In these modes, in addition to out-of-block samples, the predicted samples within the block may be used as intra prediction reference samples for some patches within the block. For example, these modes may be predefined and may be available for intra prediction for at least luma blocks (or luma blocks only). A predetermined number (e.g., 5) of filter intra modes may be pre-designed, each of which may be represented by a set of n-tap filters (e.g., 7-tap filters) that reflect the correlation relationship between samples within a 4×2 patch and n neighboring samples adjacent thereto. In other words, the weight coefficients of the n-tap filter may be position-dependent. Taking an 8×8 block, a 4×2 patch, and 7-tap filtering as an example, as shown in FIG. 11, the 8×8 block 1102 may be divided into eight 4×2 patches. These patches are indicated by B0, B1, B1, B3, B4, B5, B6, and B7 in FIG. 11. For each patch, its seven neighbors indicated by R0 to R7 in FIG. 11 may be used to predict samples within the current patch. In patch B0, all neighbors may already be reconstructed. However, in other patches, some of the neighbors may be within the current block and may not be reconstructed. In that case, the predicted value of the nearest neighbor is used as a reference. For example, as shown in FIG. 11, since all neighbors of patch B7 are not reconstructed, predicted samples of some neighbors of, for example, B4, B5, and / or B6 are used instead.

[0099] In some implementations of intra prediction, one color component may be predicted using one or more other color components. The color component may be any one of the color components in color spaces such as YCrCb, RGB, XYZ, etc. For example, prediction of chroma components (e.g., chroma blocks) from a luma component (e.g., luma reference sample) called CfL (Chroma from Luma) may be implemented. In some exemplary implementations, cross-color (inter-color) prediction may be allowed only from luma to chroma. For example, chroma samples within a chroma block may be modeled as a linear function of the corresponding reconstructed luma samples. CfL prediction may be implemented as follows. CfL(α)=α×L AC +DC (1)

[0100] Here, L AC represents the AC contribution of the luma component, α represents the parameter of the linear model, and DC represents the DC contribution of the chroma component. For example, the AC component is obtained for each sample of the block, while the DC component is obtained for the entire block. Specifically, the reconstructed luma samples may be subsampled to the chroma resolution, and then the average luma value (luma DC) may be subtracted from each luma value to form the AC contribution in luma. Then, the AC contribution of luma is used in the linear mode of Equation (1) to predict the AC value of the chroma component. Instead of requiring the decoder to calculate the scaling parameter to approximate or predict the chroma AC component from the contribution of luma AC, an exemplary CfL implementation may determine the parameter α based on the original chroma samples and signal it in the bitstream. This reduces the complexity of the decoder and results in more accurate prediction. Regarding the DC contribution of the chroma component, in some exemplary implementations, it may be calculated using the intra DC mode within the chroma component.

[0101] Returning to intra prediction, in some exemplary implementations, the prediction of samples within an encoded block or a prediction block may be based on one of a set of reference lines. In other words, rather than always using the closest neighboring line (e.g., the immediately upper neighboring line or the immediately left neighboring line of the prediction block shown in FIG. 1 above), multiple reference lines may be provided as options for the selection of intra prediction. Such an implementation of intra prediction may be referred to as Multiple Reference Line Selection (MRLS). In these implementations, the encoder determines and signals which of the multiple reference lines is to be used to generate an intra predictor. On the decoder side, after parsing the reference line index, the intra prediction of the current intra prediction block can be generated by identifying the reconstructed reference samples by searching for the specified reference line according to the intra prediction mode (directional, non - directional, and other intra prediction modes, etc.). In some implementations, the reference line index may be signaled at the encoded block level, only one of the multiple reference lines may be selected, and it may be used for the intra prediction of one encoded block. In some examples, more than one reference line may be selected simultaneously for intra prediction. For example, more than one reference line may be combined, weighted or unweighted, averaged, interpolated, or otherwise combined to generate a prediction. In some exemplary implementations, MRLS may be applied only to the luma component and not to the chroma component.

[0102] FIG. 12 shows an example of four reference lines MRLS. As shown in the example of FIG. 12, the intra-coded block 1202 may be predicted based on any one of four horizontal reference lines 1204, 1206, 1208, and 1210 and four vertical reference lines 1212, 1214, 1216, and 1218. Among these reference lines, 1210 and 1218 are the nearest adjacent reference lines. The reference lines may be indexed according to the distance from the coded block. For example, the reference lines 1210 and 1218 may be called zero reference lines, and the other reference lines may be called non-zero reference lines. Specifically, the reference lines 1208 and 1216 may be called the first reference lines, the reference lines 1206 and 1214 may be called the second reference lines, and the reference lines 1204 and 1212 may be called the third reference lines.

[0103] In some embodiments, for a particular coded block, coding unit, prediction block, or prediction unit that is intra-coded, its intra mode needs to be signaled by one or more syntax elements in the bitstream. As described above, the number of possible intra prediction modes can be huge, and 62 intra prediction modes of 56 directional intra prediction modes, 5 non-directional modes, and 1 Chroma from Luma mode (e.g., only for chroma components) may be available. To signal these intra prediction modes, a first syntax may be signaled to indicate which nominal angle or non-directional mode is equal to the nominal mode of the current block. Then, if the mode of the current block is a directional mode, a second syntax may be signaled to indicate which delta angle is equal to the delta angle of the current block. In some situations during video encoding and / or decoding, there may be a strong correlation between the intra prediction mode of the current block and its adjacent blocks.

[0104] In various embodiments, this correlation may be utilized to design a more efficient syntax for intra-mode coding. In some implementations, the available intra prediction modes of the current block may be divided into a plurality of intra prediction mode sets according to the intra prediction modes of its adjacent blocks. To obtain the intra prediction mode of the current block, first, a mode set index may be signaled to indicate the mode set index of the intra prediction mode of the current block, and then, a mode index may be signaled to indicate the index of the intra prediction mode within the mode set.

[0105] Here, in various embodiments of the present disclosure, "XYZ is signaled" may indicate that XYZ is encoded into an encoded bitstream during an encoding process, and / or after the encoded bitstream is transmitted from one device to another device, "XYZ is signaled" may indicate that XYZ is decoded / extracted from the encoded bitstream during a decoding process.

[0106] For example, in some of the above implementations, the number of available intra prediction modes may include, for example, 56 directional intra prediction modes (e.g., 8 nominal directions each having 7 fine angles in each nominal direction), 5 non-directional modes, and 1 chroma-from-luma mode (for chroma components only), for a total of 62 different modes. When an intra mode is selected during the encoding process for a particular encoded block, encoding unit, prediction block, or prediction, the signaling corresponding to the selected intra mode needs to be included in the bitstream. The syntax signaling must be able to distinguish all these 62 modes in some way. For example, these 62 modes may be signaled using a single syntax for 62 indices, each corresponding to one mode. In some other exemplary implementations, one syntax may be signaled to indicate which nominal angle or non-directional mode is being used as the nominal mode in the current block, and then, if the nominal mode of the current block is a directional mode, another syntax may be further signaled to indicate which delta angle is selected for the current block.

[0107] Various syntaxes related to intra coding typically occupy a large portion of the bitstream. For example, at various coding levels, since the selection of the intra mode has to be frequently signaled, the reduction of the number of bits used for signaling the intra mode has become important in improving video coding efficiency. In practice, the use of various intra prediction modes may follow certain statistical patterns, and such usage patterns may be utilized to design the indexing of the intra mode and the signaling of the syntax so that the signaling efficiency can be enhanced. Further, there may be some correlation on average between the selection of the intra mode from block to block. Such correlations can be statistically obtained offline and may be considered in the design of the syntax for signaling the selection of the intra mode. The goal is to reduce the average number of bits of the signaling syntax elements in the coded bitstream. For example, some common statistics may indicate that there can be a strong correlation between the optimal intra prediction mode of the current block and its neighboring blocks. Such correlations may be utilized in designing the syntax of the intra mode coding.

[0108] In some embodiments, to improve video coding / decoding performance, after generating the intra prediction samples, an offset-based refinement for intra prediction (ORIP) may be used. When ORIP is applied, the prediction samples are refined by adding offset values.

[0109] As shown in FIG. 13, intra prediction (1330) is performed based on a reference sample. The reference sample may include samples from one or more left reference lines (1312) and / or one or more upper reference lines (1310). An offset-based refinement (ORIP) (1350) for intra prediction may generate an offset value using adjacent reference samples. In some implementations, the adjacent reference samples for ORIP may be the same set as the reference samples for intra prediction. In some other implementations, the adjacent reference samples for ORIP may be a different set from the reference samples for intra prediction.

[0110] In some implementations that refer to FIGS. 14A and 14B, ORIP may be performed at the 4×4 sub-block level. For each 4×4 sub-block (1471, 1472, 1473, and / or 1474), the offset is generated from its adjacent samples. For example, for the first sub-block (1471), the offset is generated from its upper adjacent samples (P1, P2, P3, and P4 at 1420), left adjacent samples (P5, P6, P7, and P8 at 1410), and / or upper left adjacent sample (P0) (1401). In some implementations, the upper adjacent samples may include both the upper adjacent samples (P1, P2, P3, and P4 at 1420) and the upper left adjacent sample (P0) (1401). In other implementations, the left adjacent samples may include both the left adjacent samples (P5, P6, P7, and P8 at 1410) and the upper left adjacent sample (P0) (1401).

[0111] The first sub-block (1471) includes 4×4 pixels, and each pixel of the 4×4 pixels corresponds to a previous adjacent prediction sample predN before refinement. For example, pred0, pred1, pred2,... pred16.

[0112] In various embodiments, the offset value for each pixel of a given sub-block may be calculated based on adjacent samples according to a formula. The formula may be a predetermined formula or a formula indicated by parameters encoded in the encoded bitstream.

[0113] In some implementations with reference to FIG. 14B, the offset value (offset(k)) at the k-th position of a given sub-block may be generated as follows.

Equation

[0114] W kn is a predetermined weight for offset calculation. P n is the value of adjacent samples (e.g., P0, P1, P2,..., P8). pred k is the predicted value of the pixel after application of intra prediction or other prediction (e.g., inter prediction). pred_refined k is the refined value of the pixel after application of ORIP. clip3() is the mathematical function of clip3. n is an integer from 0 to 8 inclusive. k is an integer from 0 to 15 inclusive.

[0115] In some implementations, W kn may be predefined or obtained according to Table 1.

Table 1

[0116] In some other implementations, the sub-block based ORIP may be applied only to a predetermined set of intra prediction modes and / or may vary for luma and chroma depending on the intra prediction mode. Table 2 shows one implementation of the sub-block based ORIP for various intra prediction modes and either the luma channel or the chroma channel. Taking the luma channel as an example, when the prediction mode is either DC or SMOOTH, the ORIP is always ON and no further signaling is required. When the prediction mode is HOR / VER and angle_delta is equal to 0, block-level signaling is required to enable / disable the ORIP, and / or when the intra prediction mode is another mode, the ORIP is always OFF and no further signaling is required.

Table 2

[0117] Referring to the second 4×4 sub-block (1473), due to its relative position with respect to the first 4×4 sub-block (1471), the upper adjacent samples of the second sub-block may be some of the pixels of the first sub-block. P1 of the second sub-block may be pred12 of the first block, P2 of the second sub-block may be pred13 of the first sub-block, P3 of the second sub-block may be pred14 of the first sub-block, and P4 of the second sub-block may be pred15 of the first sub-block. The upper left adjacent sample (P0) of the second sub-block may be the left adjacent sample (P8) of the first sub-block.

[0118] In various embodiments, the available intra prediction modes or mode options for the currently encoded block may be divided into multiple intra prediction mode sets. A mode set index may be assigned to each set. Each set may include multiple intra mode prediction modes. The manner in which the available intra prediction modes are divided and ordered, and the intra prediction modes are ordered within each mode set, may be determined at least in part according to the intra prediction mode used by its neighboring blocks, based on the correlation between the intra prediction modes between blocks. The intra prediction mode used by the neighboring blocks may be referred to as the "reference intra prediction mode" or "reference mode". The intra prediction mode for a particular unit may be determined and selected. The selection of the intra prediction mode may be signaled. First, the mode set index may be signaled to indicate the mode set index of the intra prediction mode set that includes the selected intra prediction mode. Next, the mode index (also referred to as the mode position index within the set) may be signaled to indicate the index of the intra prediction mode selected within the mode set.

[0119] The above general implementation of the division and ordering of the intra prediction modes, and the following specific examples, utilize statistical effects and neighboring correlation relationships to dynamically index these modes, such that the design of the syntax for signaling their selection in the encoded video bitstream can be optimized to improve encoding efficiency. For example, these implementations can help reduce the number of syntaxes for signaling and can contribute to more efficient context generation for entropy encoding.

[0120] The various embodiments and / or implementations described in this disclosure may be used separately or in any order of combination. Further, some, all, or any partial or whole combination of these embodiments and / or implementations may be implemented as part of an encoder and / or decoder, and may be implemented in hardware and / or software. For example, it may be hard-coded in a dedicated processing circuit (e.g., one or more integrated circuits). In another example, it may be implemented by one or more processors executing a program stored on a non-transitory computer-readable medium.

[0121] There may be several problems / issues associated with intra-mode coding. For example, the hardware implementation of intra-mode coding, especially the process of constructing an intra-mode list for small blocks, can be very difficult.

[0122] This disclosure describes various embodiments of intra-prediction mode coding in video encoding and / or decoding, addresses at least one of the above problems / issues, and achieves an efficient software / hardware implementation for improved intra-mode coding. In some embodiments, for an efficient software / hardware implementation, signaling and / or encoding for the intra-prediction mode for small blocks may be implemented according to predetermined rules to simplify the syntax.

[0123] In various embodiments, referring to FIG. 15, a method 1500 for intra prediction mode coding in video decoding may include, by a device including a memory storing instructions and a processor communicating with the memory, receiving an encoded video bitstream at step 1510; constructing, by the device, a list of intra modes of blocks in the encoded video bitstream according to a predetermined rule based on the size of the blocks at step 1520; splitting, by the device, the list of intra modes into a plurality of intra mode sets of the blocks at step 1530; extracting, by the device, a set index indicating an intra mode set from the plurality of intra mode sets from the encoded video bitstream at step 1540; extracting, by the device, a mode index indicating an intra prediction mode from the intra mode set from the encoded video bitstream at step 1550; determining, by the device, an intra prediction mode of the blocks based on the set index and the mode index at step 1560; and decoding, by the device, the encoded video bitstream based on the intra prediction mode at step 1570, and may include some or all of them.

[0124] In various embodiments of the present disclosure, the size of a block (e.g., but not limited to, an encoded block, a prediction block, or a transform block) may indicate the width or height of the block. The width or height of the block may be an integer value in units of pixels. In various embodiments of the present disclosure, the size of the block may indicate the area size of the block. The area size of the block may be an integer value calculated by multiplying the width of the block by the height of the block in units of pixels. In some various embodiments of the present disclosure, the size of the block may indicate the maximum value of the width or height of the block, the minimum value of the width or height of the block, or the aspect ratio of the block. The aspect ratio of the block may be calculated as the width divided by the height of the block, or may be calculated as the height divided by the width of the block.

[0125] In some implementations, the intra-mode type may include at least one of a directional mode, a non-directional mode, a smooth mode (e.g., smooth, smooth_v, smooth_h), a DC mode, a PAETH mode, and / or a mode that generates a predicted sample according to a given prediction direction. In some other implementations, in a loose classification, the directional mode may broadly include any mode that is not a smooth (smooth, smooth_v, smooth_h), DC, or PAETH mode and a mode that generates a predicted sample according to a given direction. In some other implementations, the non-directional mode may include a smooth mode (e.g., smooth, smooth_v, smooth_h), a DC mode, a PAETH mode, and a luma-for-chroma mode. In some other implementations, in a loose classification, the non-directional mode may broadly include any mode that is not a directional mode.

[0126] In some implementations, the available intra prediction modes for the current block may be divided / separated into a plurality of intra prediction mode sets. To obtain the intra prediction mode of the current block, first, a mode set index may be signaled to indicate the mode set index of the intra prediction mode of the current block, and then, a mode index may be signaled to indicate the index of the intra prediction mode within the mode set.

[0127] Here, in various embodiments of the present disclosure, the "first" mode set not only indicates "one" mode set, but also indicates the "first" mode set having the smallest mode set index, and the "second" mode set not only indicates "another" mode set, but also indicates the "second" mode set having the second smallest mode set index, and so on. For example, the number of intra prediction mode sets may be indicated by M, and the mode set index may be in the range of, for example, 1 to M or 0 to M - 1. When the mode set index is in the range of 1 to M, the "first" mode set is the "first" mode set with the mode set index of 1, the "second" mode set is the "second" mode set with the mode set index of 2, and so on. When the mode set index is in the range of 0 to M - 1, the "first" mode set is the "first" mode set with the mode set index of 0, the "second" mode set is the "second" mode set with the mode set index of 1, and so on.

[0128] Here, in various embodiments of the present disclosure, "XYZ is signaled" may indicate that XYZ is encoded into an encoded bitstream during an encoding process, and / or after the encoded bitstream is transmitted from one device to another device, "XYZ is signaled" may indicate that XYZ is decoded / extracted from the encoded bitstream during a decoding process.

[0129] Here, in various embodiments of the present disclosure, a "block" may indicate a prediction block, an encoding block, a transform block, or a coding unit (CU).

[0130] Referring to step 1510, the device may be the electronic device (530) of FIG. 5 or the video decoder (810) of FIG. 8. In some implementations, the device may be the decoder (633) within the encoder (620) of FIG. 6. In other implementations, the device may be a part of the electronic device (530) of FIG. 5, a part of the video decoder (810) of FIG. 8, or a part of the decoder (633) within the encoder (620) of FIG. 6. The encoded video bitstream may be the encoded video sequence of FIG. 8 or the intermediate encoded data of FIGS. 6 or 7. The block may indicate a block during encoding or an encoded block.

[0131] Referring to step 1520, the device may construct a list of intra modes of the block according to a predetermined rule based on the size of the block. Referring to step 1530, the device may divide the list of intra modes into a plurality of intra mode sets of the block. The predetermined rule may be one of a set of predetermined rules or may be selected from the set of predetermined rules based on the size of the block.

[0132] In some implementations, the intra mode list is constructed based on a predetermined rule for the current block. After constructing the intra mode list, the intra prediction mode is divided into a plurality of intra prediction mode sets according to the corresponding index within the intra mode list. To signal the intra prediction mode of the current block, first, a mode set index may be signaled to indicate the mode set index of the intra prediction mode of the current block, and then, a mode index may be signaled to indicate the index of the intra prediction mode within the mode set. The predetermined rule may be different for different block sizes, which may be indicated by the number of samples within the block.

[0133] In various embodiments, in response to the block size being greater than or equal to a first threshold, the intra mode list includes a first sublist at the beginning of the list, and the first sublist includes all non - directional intra prediction modes. For example, if the intra mode list is divided into a plurality of intra mode sets, since the non - directional intra prediction modes are at the beginning of the list, the non - directional intra prediction modes may be divided into the first intra mode set. In some implementations, the non - directional modes may include smooth modes (e.g., smooth, smooth_v, smooth_h), DC mode, PAETH mode, and luma - for - chroma mode. In some other implementations, in a loose classification, the non - directional modes may broadly include any mode that is not a directional mode.

[0134] In some implementations, for blocks whose block size is greater than or equal to a first threshold (TH1), first, all non - directional modes are added to the mode list. Next, to derive the intra prediction mode, an offset is added to the directional intra prediction mode of the adjacent block, and the derived intra prediction mode is added to the intra mode list. Finally, after all the derived intra prediction modes are added, if the intra prediction mode set is not yet full, default modes are used to fill the remaining positions in the intra mode list. For example, the first threshold is 8×8. In another example, the first threshold is 32 in terms of the number of samples.

[0135] In some other implementations, depending on whether the adjacent block uses the directional intra prediction mode, the intra mode list includes a second sub - list that comes after the first sub - list in the list, and the second sub - list includes a plurality of derived intra prediction modes based on the directional intra prediction mode of the adjacent block. For example, when the intra mode list is divided into a plurality of intra mode sets, since the plurality of derived intra prediction modes are immediately after the non - directional intra prediction mode from the beginning of the list, the plurality of derived intra prediction modes based on the directional intra prediction mode of the adjacent block are likely to be divided into either the first intra mode set or the second intra mode set. In some implementations, the adjacent block of the current block may include the upper block of the current block, the left block of the current block, or both the upper block and the left block of the current block.

[0136] In some other implementations, the plurality of derived intra prediction modes include nine directional intra prediction modes by adding the offsets of [0, - 1, + 1, - 2, + 2, - 3, + 3, - 4, + 4] to the directional intra prediction mode of the adjacent block. For example, if the directional intra prediction mode of the adjacent block has a specific direction angle (x°) and the step size of the direction angle is 3°, the plurality of derived intra prediction modes may include nine directional intra prediction modes with direction angles of x, x ± 3, x ± 6, x ± 9, and x ± 12°.

[0137] In another example, the available intra prediction modes for the current block are 61, including 5 non-directional modes and 56 directional modes. If one or more adjacent blocks of the current block are encoded in one or more directional intra prediction modes, first, 5 non-directional modes are added to the mode list, and then, 9 directional intra prediction modes are derived by adding the offsets of [0, -1, +1, -2, +2, -3, +3, -4, +4] to each of the directional modes of the adjacent blocks. If there is no directional mode in the adjacent blocks, the second step may be skipped. For example, if only one adjacent block is encoded in one directional intra prediction mode, 9 directional intra prediction modes may be derived, and as a result, 14 (= 9 + 5) modes are added to the mode list, and then the default mode is added to the mode list. In another example, if two adjacent blocks are encoded in two directional intra prediction modes, 18 (= 9 * 2) directional intra prediction modes may be derived, and as a result, 23 (= 18 + 5) modes are added to the mode list, and then the default mode is added to the mode list. In some situations, when two adjacent blocks are encoded in two directional intra prediction modes, fewer than 18 directional intra prediction modes may be derived after removing any overlap between the two sets of 9 derived directional intra prediction modes. In an extreme situation, when two adjacent blocks are encoded in two identical directional intra prediction modes, only 9 directional intra prediction modes may be derived after removing the overlap between the same two sets of 9 derived directional intra prediction modes.

[0138] In some other implementations, the intra mode list includes a third sublist that is next to the second sublist in the list, and the third sublist includes a default intra prediction mode. In some other implementations, the default intra prediction mode includes at least one nominal directional intra prediction mode with a delta angle of zero. For example, the default intra prediction mode may include all other nominal directional intra prediction modes with a delta angle of zero that are not yet included in the derived intra prediction mode based on the directional intra prediction mode of the adjacent block.

[0139] In various embodiments, in response to the block size being smaller than a first threshold, the intra mode list includes a first sublist at the beginning of the list, and the first sublist includes all non-directional intra prediction modes. In some implementations, in response to the adjacent block using a directional intra prediction mode, the intra mode list includes a second sublist that is next to the first sublist in the list, and the second sublist includes all directional intra prediction modes of the adjacent block. In some other implementations, the intra mode list includes a third sublist that is next to the second sublist in the list, and the third sublist includes a default intra prediction mode. In some other implementations, in response to the number of all directional intra prediction modes of the adjacent block being greater than 1, the directional intra prediction modes of the adjacent block in the intra mode list have an order of angles from smallest to largest.

[0140] In some implementations, for blocks with a block size smaller than a first threshold TH1, first, all non-directional modes are added to the mode list, then the directional intra prediction modes of adjacent blocks are added to the intra mode list, and then default modes are used to fill the remaining positions in the intra mode list. In one example, the first threshold (TH1) is set to a block size of 8×8. In another example, if there are two directional intra prediction modes in an adjacent block, the directional mode with the smaller angle is added to the mode list first. For example, the vertical mode is 90°, the horizontal mode is 180°, and since the angle of the vertical mode is smaller than the angle of the horizontal mode, the vertical mode is added to the mode list first.

[0141] In various embodiments, in response to the block size being smaller than a first threshold, the list of intra modes includes a predetermined default intra prediction mode. In some implementations, the predetermined default intra prediction mode in the list of intra modes is, first, all non-directional intra prediction modes, second, directional intra prediction modes with a delta angle of zero, third, directional intra prediction modes with a delta angle of +2 or 2 times the step size, fourth, directional intra prediction modes with a delta angle of +1 or 1 times the step size, and / or fifth, directional intra prediction modes with a delta angle of +3 or 3 times the step size, in that order. In some implementations, the step size may have a default value of 3°.

[0142] In some implementations, for blocks with a block size smaller than a threshold (TH1), a predetermined default mode is used to fill all positions within the intra mode list. For example, the predetermined default mode may be as described below. First, all non - directional modes are added to the mode list. Second, directional modes with a delta angle equal to 0 are added to the mode list. Third, directional modes with delta angles equal to 2 and - 2 are added to the mode list. Fourth, directional modes with delta angles equal to 1 and - 1 are added to the mode list, and / or fifth, directional modes with delta angles equal to 3 and - 3 are added to the mode list.

[0143] In various embodiments, in response to the block size being smaller than a first threshold, the intra mode set indicated by the set index belongs to N intra mode sets, the N intra mode sets are at the head of a plurality of intra mode sets, and N is a positive integer.

[0144] In some implementations, for blocks smaller than the threshold (TH1), only the first N intra mode sets for the current block are permitted and signaled. N is a positive integer. In one example, N is set to 1, indicating that only the first intra mode set for the current block is permitted and signaled. In another example, N is set to 2, indicating that only the first and second intra mode sets for the current block are permitted and signaled.

[0145] Embodiments of the present disclosure may be used individually or may be combined in any order. Further, each of the methods (or embodiments), encoders, and decoders may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. Embodiments of the present disclosure may be applied to luma blocks or chroma blocks, and in the case of chroma blocks, the embodiments may be applied individually to more than one color component or may be applied together to more than one color component.

[0146] In the present disclosure, any steps or operations in various embodiments may be combined in any amount or in any order as needed. In the present disclosure, two or more steps or operations in various embodiments may be executed in parallel.

[0147] The above-described techniques can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, FIG. 17 shows a computer system (2600) suitable for implementing certain embodiments of the disclosed subject matter.

[0148] Computer software can be coded using any suitable machine code or computer language and be the subject of assembly, compilation, linking, or similar mechanisms to create code that includes instructions executable directly or through interpretation, microcode execution, etc. by one or more computer central processing units (CPUs), graphics processing units (GPUs), and the like.

[0149] The instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0150] The components shown in FIG. 17 for the computer system (2600) are illustrative in nature and are not intended to suggest any limitations as to the use or functionality of the computer software implementing embodiments of the present disclosure. Nor should the configuration of the components be construed as having any dependency or requirement regarding any one or combination of the components shown in the exemplary embodiments of the computer system (2600).

[0151] The computer system (2600) can include a specific human interface input device. Such a human interface input device can respond to input by one or more human users, for example, through tactile input (e.g., keystrokes, swipes, movement of a data glove), voice input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). Also, the human interface device can be used to capture certain media that is not necessarily directly related to conscious human input, such as voice (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., 2D video, 3D video including stereoscopic video).

[0152] The input human interface device may include one or more of a keyboard (2601), a mouse (2602), a trackpad (2603), a touch screen (2610), a data glove (not shown), a joystick (2605), a microphone (2606), a scanner (2607), and a camera (2608) (only one of each is shown).

[0153] The computer system (2600) may also include a specific human interface output device. Such a human interface output device may, for example, stimulate the senses of one or more human users through tactile output, sound, light, and smell / taste. Such a human interface output device may be a tactile output device (e.g., tactile feedback by a touch screen (2610), a data glove (not shown), or a joystick (2605); however, there may also be a tactile feedback device that does not function as an input device), an audio output device (e.g., a speaker (2609), headphones (not shown)), a visual output device (e.g., a screen (2610) including a CRT screen, an LCD screen, a plasma screen, an OLED screen; each may or may not have a touch screen input function, each may or may not have a tactile feedback function, and some of them can output higher than three-dimensional output through means such as two-dimensional visual output or stereoscopic output; virtual reality glasses (not shown), a holographic display, and a smoke tank (not shown)), and a printer (not shown).

[0154] The computer system (2600) may also include an optical medium including a CD / DVD ROM / RW (2620) together with a human-accessible memory device and related media, such as a CD / DVD or similar media (2621), a thumb drive (2622), a removable hard drive or solid state drive (2623), legacy magnetic media such as tapes and floppy disks (not shown), a specialized ROM / ASIC / PLD-based device such as a security dongle (not shown), etc.

[0155] Those skilled in the art should also understand that the term "computer-readable medium" used in connection with the presently disclosed subject matter does not include a transmission medium, a carrier wave, or other transient signals.

[0156] The computer system (2600) can also include an interface (2654) to one or more communication networks (2655). The network can be, for example, wireless, wired, or optical. The network can further be local, wide area, metropolitan area, in-vehicle and industrial, real-time, delay-tolerant, etc. Examples of networks include Ethernet®, wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., cellular networks including cable TV, satellite TV, terrestrial broadcast TV, TV wired or wireless wide area digital networks, in-vehicle and industrial including CAN Bus, etc. A particular network typically requires an external network interface adapter attached to a particular general-purpose data port or peripheral bus (2649) (e.g., a USB port of the computer system (2600), etc.). Others are typically integrated into the core of the computer system (2600) by attachment to a system bus as described later (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2600) can communicate with other entities. Such communication can be unidirectional, receive-only (e.g., broadcast TV), dedicated unidirectional transmission (e.g., CANbus to a specific CANbus device), or bidirectional to other computer systems using, for example, local or wide area digital networks. For each of the networks and network interfaces as described above, a specific protocol and protocol stack can be used.

[0157] The aforementioned human interface device, human-accessible memory device, and network interface can be attached to the core (2640) of the computer system (2600).

[0158] The core (2640) can include one or more central processing units (CPUs) (2641), graphics processing units (GPUs) (2642), specialized programmable processing devices in the form of field programmable gate arrays (FPGAs) (2643), hardware accelerators (2644) for specific tasks, graphics adapters (2650), etc. These devices can be connected through a system bus (2648) together with read-only memory (ROM) (2645), random access memory (2646), internal mass storage devices (2647) such as internal hard drives and solid state drives (SSDs) that are not user-accessible internally. In some computer systems, the system bus (2648) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus (2648) or through a peripheral bus (2649). In one example, a screen (2610) can be connected to the graphics adapter (2650). Architectures for peripheral buses include PCI, USB, etc.

[0159] The CPU (2641), GPU (2642), FPGA (2643), and accelerator (2644) can execute specific instructions that can, in combination, constitute the aforementioned computer code. That computer code can be stored in the ROM (2645) or RAM (2646). Temporary data can also be stored in the RAM (2646), while persistent data can be stored, for example, in the internal mass storage device (2647). By using cache memory that can be closely associated with one or more CPUs (2641), GPUs (2642), mass storage devices (2647), ROM (2645), RAM (2646), etc., fast storage and retrieval to any of the memory devices can be enabled.

[0160] A computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and the computer code may be specially designed and constructed for the purposes of this disclosure, or they may be of the kind well known and available to those having skill in the computer software arts.

[0161] As a non-limiting example, a computer system having an architecture (2600), specifically a core (2640), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied on one or more tangible computer-readable media. Such computer-readable media can be media related to user-accessible mass storage as introduced above and specific storage of the core (2640) of a non-transitory nature such as a mass storage device (2647) inside the core or a ROM (2645). The software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2640). The computer-readable media can include one or more memory devices or chips depending on specific needs. The software includes defining a data structure stored in a RAM (2646) and modifying such a data structure according to a process defined by the software, and causing a specific process or a specific part described herein to be executed by the core (2640) and specifically a processor (including a CPU, GPU, FPGA, etc.) therein. Additionally or alternatively, the computer system can provide functionality as a result of logic wired within a circuit (e.g., an accelerator (2644)) or otherwise embodied, which can operate instead of or in conjunction with software for executing a specific process or a specific part of a specific process described herein. References to software include logic and vice versa as appropriate. References to computer-readable media can include, as appropriate, a circuit (e.g., an integrated circuit (IC)) storing software for execution, a circuit embodying logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0162] Although a particular invention has been described with reference to exemplary embodiments, this description is not intended to be limiting. Various changes to the exemplary embodiments and further embodiments of the present invention will become apparent to those skilled in the art from this description. Those skilled in the art will readily recognize that they can make the above and various other changes to the exemplary embodiments illustrated and described herein without departing from the spirit and scope of the present invention. Accordingly, the appended claims are considered to cover any such changes and alternative embodiments. Specific ratios in the drawings may be exaggerated and other ratios may be minimized. Accordingly, the disclosure and drawings are considered to be illustrative rather than limiting.

Claims

1. A method for intra-prediction mode coding in video decoding, comprising: receiving, by a device including a memory storing instructions and a processor in communication with the memory, an encoded video bitstream; constructing, by the device, a list of intra modes for the blocks in the coded video bitstream from available intra modes according to a first predetermined rule that depends at least on a size of the blocks, and ordering the list of intra modes into an ordered list of intra modes; dividing, by the device, the ordered list of intra-modes into a plurality of intra-mode sets of the blocks according to a second predetermined rule that depends at least on the size of the blocks; extracting, by the device, from the coded video bitstream a set index indicating an intra-mode set from the plurality of intra-mode sets; extracting, by the device, from the encoded video bitstream a mode index indicating a target intra-prediction mode from the intra mode set; determining, by the device, the target intra-prediction mode for the block based on the set index and the mode index; decoding, by the device, the encoded video bitstream based on the target intra-prediction mode; A method comprising:

2. The method of claim 1, wherein when the size of the block is equal to or greater than a first threshold, the first predetermined rule comprises: the ordered list of intra modes includes a first sublist at the beginning of the list; the first sublist includes all non-directional intra-prediction modes; The method of claim 1 , further comprising:

3. The first threshold is 8×8; or The method of claim 2 , wherein the first threshold is 32 samples.

4. A method described in any one of claims 1 to 3, wherein the steps of constructing a list of intra modes for the block and arranging the list of intra modes into an ordered list of intra modes are based on the intra modes used by adjacent blocks of the block when the size of the block satisfies a predetermined condition.

5. The method of claim 1, wherein when a neighboring block uses a directional intra-prediction mode and the size of the block is greater than or equal to a first threshold, the ordered list of intra modes includes a second sublist that is next to a first sublist in the list; The method of claim 4 , wherein the second sub-list includes a plurality of derived intra-prediction modes based on the directional intra-prediction modes of the neighboring blocks.

6. The method described in claim 5, wherein the multiple derived intra prediction modes include nine directional intra prediction modes by adding an offset of [0, -1, +1, -2, +2, -3, +3, -4, +4] to the directional intra prediction mode of the adjacent block.

7. The ordered list of intra modes includes a third sublist that is next to the second sublist in the list; The method of claim 5 or 6, wherein the third sub-list includes a default intra-prediction mode.

8. The method described in claim 7, wherein the default intra-prediction mode includes at least one nominally directional intra-prediction mode having a delta angle of zero.

9. When the size of the block is smaller than a first threshold, the ordered list of intra modes includes a first sublist at the beginning of the list; The method of claim 1 , wherein the first sublist includes all non-directional intra-prediction modes.

10. The ordered list of intra modes includes a second sublist that is next to the first sublist in the list; The method of claim 9 , wherein the second sub-list includes a default intra-prediction mode.

11. A method for intra-prediction mode coding in video encoding, comprising: constructing, by a device including a memory storing instructions and a processor in communication with the memory, a list of intra modes for the block from available intra modes according to a first predetermined rule that depends at least on a size of the block, and ordering the list of intra modes into an ordered list of intra modes; dividing, by the device, the ordered list of intra-modes into a plurality of intra-mode sets of the blocks according to a second predetermined rule that depends at least on the size of the blocks; determining, by the device, a set index indicating an intra-mode set from the plurality of intra-mode sets; encoding, by the device, the set index; determining, by the device, a mode index indicating a target intra-prediction mode from the intra mode set; encoding, by the device, the mode index; determining, by the device, the target intra-prediction mode for the block based on the set index and the mode index; encoding, by the device, the block based on the target intra-prediction mode; A method comprising:

12. The method of claim 1, wherein when the size of the block is equal to or greater than a first threshold, the first predetermined rule comprises: the ordered list of intra modes includes a first sublist at the beginning of the list; the first sublist includes all non-directional intra-prediction modes; The method of claim 11 , wherein the method specifies:

13. The first threshold is 8×8; or The method of claim 12 , wherein the first threshold is 32 samples.

14. A method described in any one of claims 11 to 13, wherein the steps of constructing a list of intra modes for the block and arranging the list of intra modes into an ordered list of intra modes are based on the intra modes used by adjacent blocks of the block when the size of the block satisfies a predetermined condition.

15. The method of claim 15, wherein when a neighboring block uses a directional intra-prediction mode and the size of the block is greater than or equal to a first threshold, the ordered list of intra modes includes a second sublist that is next to a first sublist in the list; The method of claim 14 , wherein the second sub-list includes a plurality of derived intra-prediction modes based on the directional intra-prediction modes of the neighboring blocks.

16. The method described in claim 15, wherein the multiple derived intra prediction modes include nine directional intra prediction modes by adding an offset of [0, -1, +1, -2, +2, -3, +3, -4, +4] to the directional intra prediction mode of the adjacent block.

17. The ordered list of intra modes includes a third sublist that is next to the second sublist in the list; The method of claim 15 or 16, wherein the third sub-list includes a default intra-prediction mode.

18. The method described in claim 17, wherein the default intra-prediction mode includes at least one nominally directional intra-prediction mode having a delta angle of zero.

19. When the size of the block is smaller than a first threshold, the ordered list of intra modes includes a first sublist at the beginning of the list; The method of claim 11 , wherein the first sublist includes all non-directional intra-prediction modes.

20. A method for transmitting a bitstream, comprising: constructing, by a device including a memory storing instructions and a processor in communication with the memory, a list of intra modes for the block from available intra modes according to a first predetermined rule that depends at least on a size of the block, and ordering the list of intra modes into an ordered list of intra modes; dividing, by the device, the ordered list of intra-modes into a plurality of intra-mode sets of the blocks according to a second predetermined rule that depends at least on the size of the blocks; determining, by the device, a set index indicating an intra-mode set from the plurality of intra-mode sets; encoding, by the device, the set index into the bitstream; determining, by the device, a mode index indicating a target intra-prediction mode from the intra mode set; encoding, by the device, the mode index into the bitstream; determining, by the device, the target intra-prediction mode for the block based on the set index and the mode index; encoding, by the device, the block into the bitstream based on the target intra-prediction mode; transmitting the bitstream; A method comprising:

21. An apparatus for intra-prediction mode coding in video decoding, comprising: a memory for storing instructions; a processor in communication with the memory; 11. An apparatus comprising: a processor configured, when the processor executes the instructions, to cause the apparatus to perform the method of any one of claims 1 to 10.

22. An apparatus for intra-prediction mode coding in video encoding, comprising: a memory for storing instructions; a processor in communication with the memory; 20. An apparatus comprising: a processor configured, when the processor executes the instructions, to cause the apparatus to perform the method of any one of claims 11 to 19.

23. An apparatus for transmitting a bitstream, comprising: a memory for storing instructions; a processor in communication with the memory; 21. An apparatus comprising: a processor configured, when the processor executes the instructions, to cause the apparatus to perform the method of claim 20.

24. A computer program causing a processor to carry out a method according to any one of claims 1 to 10.

25. A computer program that causes a processor to carry out a method according to any one of claims 11 to 19.

26. A computer program causing a processor to carry out the method of claim 20.

Images