Intra prediction mode improvement based on available reference samples

Abstract

In one embodiment, a wide angle process may be modified to disable intra prediction modes (IPMs) that will utilize filled reference samples, especially when other IPMs will utilize available reference samples. More generally, depending on whether reference samples are available, certain intra prediction modes may be removed or added, or MPMs may be reordered. The signaling process may then be modified to handle additional modes or removed modes, particularly if this may cause different numbers of IPMs. Storage and propagation of intra modes may also be modified to handle changes made to the IPM.

Description

Technical Field

[0001] This embodiment generally relates to methods and apparatuses for intra prediction in video encoding and decoding. Background Art

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transformation to balance spatial and temporal redundancy in video content. Generally, intra or inter prediction is used to exploit intra or inter picture correlations, and then the difference between the original block and the predicted block (usually denoted as prediction error or prediction residue) is transformed, quantized, and entropy encoded. To reconstruct the video, the compressed data is decoded through inverse processes corresponding to entropy coding, quantization, transformation, and prediction. Summary of the Invention

[0003] According to an embodiment, a video decoding method is provided, including: identifying the availability of one or more reference samples for a block to be decoded in a picture; obtaining, in response to the availability of the one or more reference samples, a set of intra prediction modes for the block; obtaining an intra prediction mode from the set of intra prediction modes; and performing intra prediction on the block to be decoded based on the intra prediction mode for the block to form a predicted block for the block.

[0004] According to another embodiment, a video encoding method is provided, including: identifying the availability of one or more reference samples for a block to be encoded in a picture; obtaining, in response to the availability of the one or more reference samples, a set of intra prediction modes for the block; selecting an intra prediction mode from the set of intra prediction modes; and performing intra prediction on the block to be encoded based on the intra prediction mode for the block to form a predicted block for the block.

[0005] According to another embodiment, an apparatus for video decoding is provided, the apparatus including one or more processors, wherein the one or more processors are configured to: identify the availability of one or more reference samples for a block to be decoded in a picture; obtain, in response to the availability of the one or more reference samples, a set of intra prediction modes for the block; obtain an intra prediction mode from the set of intra prediction modes; and perform intra prediction on the block to be decoded based on the intra prediction mode for the block to form a predicted block for the block.

[0006] According to another embodiment, there is provided an apparatus for video coding, the apparatus including one or more processors, wherein the one or more processors are configured to: identify the availability of one or more reference samples for a block to be coded in a picture; obtain, in response to the availability of the one or more reference samples, a set of intra prediction modes for the block; select an intra prediction mode from the set of intra prediction modes; and perform intra prediction on the block to be coded based on the intra prediction mode for the block to form a prediction block for the block.

[0007] One or more embodiments also provide a computer program including instructions that, when executed by one or more processors, cause the one or more processors to implement an encoding method or a decoding method according to any of the embodiments described herein. One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for video coding or decoding according to the methods described herein.

[0008] One or more embodiments also provide a computer-readable storage medium having stored thereon video data generated according to the above method. One or more embodiments also provide a method and an apparatus for transmitting or receiving video data generated according to the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A block diagram of a system in which aspects of the present embodiment may be implemented is illustrated.

[0010] Figure 2 A block diagram of an embodiment of a video encoder is illustrated.

[0011] Figure 3 A block diagram of an embodiment of a video decoder is illustrated.

[0012] Figure 4 Reference samples for intra prediction are illustrated.

[0013] Figure 5A and Figure 5B Reference sample replacement for intra prediction is illustrated.

[0014] Figure 6 A process of reference sample replacement for intra prediction is illustrated.

[0015] Figure 7A Intra prediction directions in HEVC are illustrated, Figure 7B intra prediction directions in VVC are illustrated, and Figure 7C horizontal and vertical, positive and negative intra prediction modes are illustrated.

[0016] Figure 8A ,Figure 8B and Figure 8C illustrates wide-angle intra prediction.

[0017] Figure 9 illustrates all available intra prediction directions in VVC.

[0018] Figure 10 illustrates the planar mode.

[0019] Figure 11 illustrates the reference samples for each intra prediction mode (IPM) on a PU with an aspect ratio W / H = 4.

[0020] Figure 12A illustrates the CBs within an intra slice, Figure 12B illustrates the unavailable reference samples, and Figure 12C illustrates the removed intra prediction modes.

[0021] Figure 13A illustrates another CB within an intra slice, Figure 13B illustrates the unavailable reference samples, and Figure 13C illustrates the removed intra prediction modes.

[0022] Figure 14 illustrates the workflow for signaling on the encoder side the index of the intra prediction mode selected to predict the current WxH block.

[0023] Figure 15 illustrates the workflow for decoding on the decoder side the index of the intra prediction mode selected to predict the current WxH block.

[0024] Figure 16 illustrates using a search of the already decoded blocks around the current block to identify the unavailable decoded reference samples around the current block.

[0025] Figure 17 illustrates the workflow for signaling on the encoder side the index of the intra prediction mode selected to predict the current WxH block according to another embodiment.

[0026] Figure 18 illustrates the workflow for decoding on the decoder side the index of the intra prediction mode selected to predict the current WxH block according to another embodiment.

[0027] Figure 19 illustrates creating a general list of 22 MPMs for the current luma CB in the ECM.

[0028] Figure 20 illustrates modifying the general list of creating 22 MPMs for the current luma CB in the ECM according to an embodiment.

[0029] Figure 21 Illustrates a general list of creating 22 MPMs for modification of the current brightness CB in the ECM according to another embodiment.

[0030] Figure 22 Illustrates a general list of creating 22 MPMs for modification of the current brightness CB in the ECM according to another embodiment. Detailed Description

[0031] Figure 1 Illustrates a block diagram of an example of a system in which various aspects and embodiments can be implemented. System 100 can be embodied as a device including various components described below and is configured to perform one or more aspects described in the present application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, and servers. The elements of system 100 can be embodied in a single integrated circuit, multiple ICs, and / or discrete components individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more aspects described in the present application.

[0032] System 100 includes at least one processor 110, which is configured to execute instructions loaded therein for implementing various aspects described, for example, in the present application. Processor 110 can include embedded memory, input / output interfaces, and various other circuits known in the art. System 100 includes at least one memory 120 (e.g., volatile memory devices and / or non-volatile memory devices). System 100 includes a storage device 140, which can include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 140 can include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0033] System 100 includes an encoder / decoder module 130, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents the (one or more) modules that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding and a decoding module. Additionally, the encoder / decoder module 130 may be implemented as a separate element of system 100, or may be incorporated within processor 110 as a combination of hardware and software known to those skilled in the art.

[0034] The program code to be loaded onto processor 110 or encoder / decoder 130 to perform the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. According to various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of the various items during the execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operation logic.

[0035] In several embodiments, the memory internal to processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for the processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be processor 110 or encoder / decoder module 130) is used for one or more of these functions. The external memory may be memory 120 and / or storage device 140, e.g., dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, fast external dynamic volatile memory such as RAM is used as the working memory for video encoding and decoding operations (such as for MPEG-2, HEVC, or VVC).

[0036] Input to the elements of system 100 may be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to, (i) an RF section that receives RF signals transmitted over the air, for example, by a broadcaster, (ii) composite input terminals, (iii) USB input terminals, and / or (iv) HDMI input terminals.

[0037] In various embodiments, the input device of block 105 has associated corresponding input processing elements, as is known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting the narrower frequency band again to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as, for example, a frequency selector, a signal selector, a band-limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive an RF signal transmitted through a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF section includes an antenna.

[0038] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting system 100 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of the input processing, such as Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 110 as needed. Similarly, various aspects of the USB or HDMI interface processing may be implemented within a separate interface IC or within processor 110 as needed. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110 and encoder / decoder 130, which operate in combination with memory and storage elements to process the data stream as needed for presentation on an output device.

[0039] The various elements of system 100 may be provided within an integrated housing, within which the various elements may be interconnected and data may be transmitted therebetween using a suitable connection arrangement 115, such as an internal bus known in the art, including an I2C bus, wiring, and a printed circuit board.

[0040] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented, for example, within a wired and / or wireless medium.

[0041] In various embodiments, data is streamed to system 100 using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signals of these embodiments are received via the communication channel 190 and the communication interface 150 suitable for Wi-Fi communication. The communication channel 190 of these embodiments is typically connected to an access point or a router that provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 100, and the set-top box transfers data through the HDMI connection of the input box 105. Still other embodiments use the RF connection of the input box 105 to provide streaming data to system 100.

[0042] System 100 may provide output signals to various output devices, which include a display 165, speakers 175, and other peripheral devices 185. In various examples of the embodiments, the other peripheral devices 185 include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices based on the output-providing function of system 100. In various embodiments, control signals are transmitted between system 100 and the display 165, the speakers 175, or the other peripheral devices 185 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output devices may be communicatively coupled to system 100 via dedicated connections through corresponding interfaces 160, 170, and 180. Alternatively, the output devices may be connected to system 100 using the communication channel 190 via the communication interface 150. The display 165 and the speakers 175 may be integrated with other components of system 100 in a single unit in an electronic device such as a television. In various embodiments, the display interface 160 includes a display driver, such as a timing controller (T Con) chip.

[0043] For example, if the RF portion of the input 105 is part of a standalone set-top box, the display 165 and the speakers 175 may alternatively be separated from one or more other components. In various embodiments where the display 165 and the speakers 175 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0044] Figure 2 FIG. illustrates an example video encoder 200, such as a VVC (Versatile Video Coding) encoder. Figure 2 An encoder that improves the VVC standard or an encoder that employs a technique similar to VVC may also be illustrated.

[0045] In the present application, the terms “reconstruct” and “decode” may be used interchangeably, the terms “encoded” or “coded” may be used interchangeably, and the terms “image”, “picture” and “frame” may be used interchangeably. Generally, but not necessarily, the term “reconstruct” is used on the encoder side, while “decode” is used on the decoder side.

[0046] Before being encoded, a video sequence may undergo a pre-encoding process (201), for example, applying a color transformation to an input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the preprocessing and appended to the bitstream.

[0047] In encoder 200, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (202) and processed in units such as CUs (Coding Units). Each unit is encoded using, for example, an intra mode or an inter mode. When a unit is encoded in the intra mode, it performs intra prediction (260). In the inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which of the intra mode or the inter mode to use to encode the unit and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (210) a prediction block from an original image block.

[0048] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, as well as motion vectors and other syntax elements (such as picture partitioning information), are entropy encoded (245) to output a bitstream. As a non-limiting example, Context-based Adaptive Binary Arithmetic Coding (CABAC) may be used to encode syntax elements into the bitstream.

[0049] The encoder may skip both the transformation and quantization, i.e., the residual is directly encoded without applying the transformation or quantization process.

[0050] The encoder decodes the coded blocks to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse-transformed (250) to decode the prediction residuals. The decoded prediction residuals and the prediction block are combined (255) to reconstruct the image block. The in-loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) / ALF (adaptive in-loop filter) filtering to reduce coding artifacts. The filtered image is stored in the reference picture buffer (280).

[0051] Figure 3 FIG. illustrates a block diagram of an example video decoder 300 (such as a VVC decoder). In decoder 300, the bitstream is decoded by decoder elements as described below. Video decoder 300 generally performs a decoding pass that is the reverse of the encoding pass described in Figure 2 The encoder 200 generally also performs video decoding as part of encoding video data.

[0052] Specifically, the input to the decoder includes a video bitstream that may be generated by video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, prediction modes, motion vectors, and other coding information. The picture partitioning information indicates how the picture is partitioned. Thus, the decoder can partition (335) the picture according to the decoded picture partitioning information. The transform coefficients are dequantized (340) and inverse-transformed (350) to decode the prediction residuals. The decoded prediction residuals and the prediction block are combined (355) to reconstruct the image block. The prediction block can be obtained (370) from intra prediction (360) or motion-compensated prediction (i.e., inter prediction) (375). The in-loop filter (365) is applied to the reconstructed image. The filtered image is stored at the reference picture buffer (380). Note that for a given picture, the content of reference picture buffer 380 on the decoder 300 side is the same as the content of reference picture buffer 280 on the encoder 200 side for the same picture.

[0053] The decoded picture can further undergo post-decoding processing (385), such as inverse color transformation (e.g., from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse of the remapping process performed in the pre-encoding process (201). The post-decoding processing can use metadata derived in the pre-encoding process and signaled in the bitstream.

[0054] Intra Prediction and Reference Sample Replacement

[0055] The intra prediction process in HEVC and VVC consists of the following three steps:

[0056] Reference sample generation;

[0057] Intra prediction; and

[0058] Post - processing of the predicted samples.

[0059] The reference sample generation process is illustrated in Figure 4 . The pixel value at coordinate (x, y) is indicated by P(x, y). The reference sample ref[] is also referred to as L - shaped. For a prediction unit (PU) of size NxN, the top row of (2N + 2*refIdx) decoded samples is formed by the previously reconstructed top and top - right pixels. Similarly, the left column of (2N + 2*refIdx) samples is formed by the reconstructed left and bottom - left pixels. As Figure 4 depicted in, in VVC, the reference rows and columns of samples can be at more than one sample - to - current - block distance (d = refIdx). The index “mrlIdx” is signaled to indicate which value of “d” should be used.

[0060] The corner pixel at the top - left position is also used to fill the gap between the top row and the left - column references. In Figure 5A and Figure 5B , the dashed - line regions correspond to the unavailable regions in the picture (e.g., out - of - bounds or not yet reconstructed), and the missing reference samples are represented by dotted lines. As Figure 6 shown, if some of the top or left samples are unavailable (610) because the corresponding CU is not in the same slice, or the current CU is at the frame boundary (e.g., as Figure 5A shown) or the current CU is at the bottom - right side after quadtree splitting (e.g., as Figure 5B shown), then a method called reference sample replacement is performed, where the missing samples are copied (630) from the available samples in clockwise and counter - clockwise directions. Those copied samples are also called “filled reference samples”, and when the reconstructed samples are used as reference samples, they are also called “non - filled reference samples”

[0061] If the reconstructed top / left reference samples are available, the reconstructed reference samples are copied (620) to the reference sample buffer. After the reference sample replacement process, intra prediction (640) is performed. Then, depending on the current CU size and prediction mode, the reference samples are filtered using a specified filter.

[0062] Intra prediction consists of predicting the pixels of the target CU based on the reference samples. There are different prediction modes. The planar and DC prediction modes are used for both smooth and gradually changing regions, while the angular (angles defined from 45 degrees clockwise to - 135 degrees) prediction mode is used to capture different directional structures. For square blocks, HEVC supports 33 directional prediction modes, which are indexed from 2 to 34. These prediction modes correspond to different prediction directions, asFigure 7A as shown in the figure. The numbers in the figure indicate the prediction mode indices associated with the corresponding directions. Modes 2 to 17 indicate horizontal predictions (H - 26 to H + 32), and modes 18 to 34 indicate vertical predictions (V - 32 to V + 32).

[0063] In VVC, there are 65 angular prediction modes, corresponding to the 33 angular directions defined in HEVC, and there are also 32 directions, each corresponding to the intermediate direction between adjacent pairs as Figure 7B shown in the figure. For square blocks, modes less than 34 indicate horizontal prediction, and modes greater than 34 indicate vertical prediction.

[0064] As mentioned above, the angular directions can be classified as vertical or horizontal. As Figure 7C shown in the figure, the prediction modes in the horizontal direction use only the left reference samples, or some left reference samples and some top reference samples. Similarly, the prediction modes in the vertical direction use only the top reference samples, or some top reference samples and some left reference samples. The positive horizontal direction uses only the left reference samples for prediction. Similarly, the positive vertical direction uses only the top reference samples for prediction. The negative horizontal and negative vertical directions use both the left and top reference samples for prediction.

[0065] In VVC, for non - square blocks, the disallowed regular - direction intra - prediction modes are replaced by wide - angle intra - prediction modes, as Figure 8A , Figure 8B and Figure 8C shown in the figure. Table 1 lists the intra - prediction modes replaced and the added wide - angle modes for different aspect ratios. Note that the block ratio of 32 is included in Table 1, but it cannot actually be used because the partitioning does not allow W / H = 32 or H / W = 32. In Figure 9 , the dashed lines indicate the wide - angle intra - prediction modes (WAIP). Note that in ECM, Figure 9 the indices - 1 to - 14 presented in Figure 9 are remapped to 1 to - 12 so that the angular mode indices are continuous. Modes - 15 (remapped to - 13) and 81 also do not appear in

[0066] Table 1

[0067] Aspect ratio W / H Replaced intra prediction mode index Added angular mode 2 2 to 7 67 to 72 4 2 to 11 67 to 76 8 2 to 13 67 to 78 16 2 to 15 67 to 80 32 2 to 16 67 to 81 1 None None 1 / 2 61 to 66 -4 to 1 1 / 4 57 to 66 -8 to 1 1 / 8 55 to 66 -10 to 1 1 / 16 53 to 66 -12 to 1 1 / 32 52 to 66 -13 to 1

[0068] For a given angular prediction mode, the predictor samples on the reference array are copied along the corresponding direction within the target PU. Some predictor samples may have integer positions, in which case they match the corresponding reference samples; the positions of other predictors will have a fractional part, indicating that their positions will fall between two reference samples. In the latter case, the predictor samples are interpolated (post - processing of the predicted samples) using the closest reference samples. In HEVC, linear interpolation is performed on the two closest reference samples to calculate the predictor sample value. In VVC, to interpolate the predictor samples, a 4 - tap filter fT[] is used, which is selected according to the intra - mode direction.

[0069] In addition to the directional mode, the DC mode fills the prediction with the average value of the samples in the L - shape (except for rectangular CUs that use the average value of the reference samples on the longer side), and the planar mode interpolates the reference samples spatially, as Figure 10 illustrated.

[0070] Intra - prediction mode coding

[0071] Since there are multiple intra - prediction modes available, the decoder needs the mode information to form a prediction for the intra - coded CU. The encoder encodes the mode information using one or more Most Probable Mode (MPM) sets. For example, ECM - 5.0 (Enhanced Compression Model 5.0) uses a first MPM list (with 6 MPMs) and a secondary MPM list (with 16 MPMs). The first MPM list is constructed by sequentially adding the candidate intra - prediction mode indices based on the order of the intra - prediction mode indices used for the immediately adjacent current luminance coding block, where the first MPM index is reserved for the planar mode. The added neighboring indices are the left neighbor, the upper neighbor, the lower - left neighbor, the upper - right neighbor, and the upper - left neighbor.

[0072] The secondary MPM list is constructed in such a way that by first adding the indices of the first DIMD (Decoder - side Intra - mode Derivation) and the second DIMD modes of the current luminance coding block, and then adding the incremented and decremented indices of the first angular MPMs (mpm[1]+1, mpm[1]-1, mpm[1]+2, mpm[1]-2, mpm[1]+3, mpm[1]-3, mpm[1]+4, mpm[1]-4, mpm[2]+1, mpm[2]-1, mpm[2]+2, mpm[2]-2, mpm[2]+3, mpm[2]-3, mpm[2]+4, mpm[2]-4, etc.) so that there are no redundant mode indices in the MPM list, neither in the primary MPM list nor in the secondary list. If the selected intra - prediction mode does not belong to the first MPM list and the secondary MPM list, the remaining intra - prediction mode (IPM) is encoded using a truncated binary coding of 45 symbols.

[0073] In one embodiment, the intra prediction process is modified such that the wide-angle intra prediction mode is selected based not only on the block aspect ratio but also on the availability of reference samples. In this way, the IPMs that typically use padded reference samples will be disabled, while the IPMs that are not normally allowed but will actually use non-padded reference samples will be allowed. Then the signaling is modified to take advantage of the change. When the total number of modes is different from the original 67 indices, this can be achieved by adding the context that uses this change to the CABAC coding bins, or by modifying the code to handle the added modes or the removed modes.

[0074] The propagation of intra modes is also modified to propagate more accurate mode values and to handle the case where neither the neighboring intra mode nor its 180° corresponding mode is available for the current block.

[0075] Select available intra modes based on the availability of neighboring reference samples

[0076] Figure 11 Illustrates which samples are used for availability testing according to an embodiment to understand which IPMs use padded reference samples. The solid lines (1110, 1115) indicate the angular IPMs (modes 12 to 76) used by ECM-6.0 for PUs with a W / H ratio of 4. The dashed lines (1120, 1125) indicate the modes (modes 8 to 72) used by ECM-6.0 for PUs with a W / H ratio of 2. The dashed lines (1130, 1135) indicate the modes (modes 2 to 66) used by ECM-6.0 for PUs with a W / H ratio of 1.

[0077] As Figure 11 illustrated, if a set of modes wishes to use only available reference samples for intra prediction, then it is possible to determine which samples need to be available for this set of modes for a particular block width / height ratio. For example, as Figure 11 shown, if reference sample A is not available (then the samples to the left of A may also not be available), then at least one mode from modes 73 to 76 (i.e., the modes used for W / H = 4 but not for W / H < 4) will use at least one padded sample. If reference sample B is not available (then all samples between A and B are not available), then all modes 73 to 76 will use at least one padded reference sample.

[0078] When the reference samples for an intra prediction mode are determined to be unavailable, that intra prediction mode can be decided to be removed because the prediction would have a lower quality. Additionally, if the reference samples for some other intra prediction modes are all available, it can be checked whether those intra prediction modes can be added back. Note that some of the intra prediction modes proposed to be added back are the modes removed by WAIP. However, since these intra prediction modes use reconstructed reference samples (not filled reference samples), the prediction quality may still be better. For example, as Figure 11 illustrated in

[0079] if X is available (and all reference samples above X are available), then modes 8 to 11 (i.e., the modes for W / H = 2 but not for W / H > 2) will use unfilled reference samples and can be added. If Y is available (and all reference samples above Y are available), then modes 2 to 7 (i.e., the modes for W / H = 1 but not for W / H > 1) will use non-filled reference samples and can be added. Note that intra mode removal can be performed without adding back other intra prediction modes, or some intra prediction modes can be added without removing the intra mode.

[0080] Let ref[0;0] be the position of the top-left sample of the PU to be encoded / decoded, let D be the distance of the reference line given by mrlIdx, and let W and H be the width and height of the current PU.

[0081] - If W > H,

[0082] If the top samples from ref[-1-D;-1-D] up to ref[2*W+W*D / H-1;-1-D] are available (i.e., non-filled samples), then the set of modes defined for the aspect ratio W / H can be used, such as the modes used in ECM-6.0, as described in Table 1

[0083] [E001];

[0084] Otherwise (if at least one of those samples is unavailable), if from ref[-1-

[0085] If all the left samples from ref[-1 - D; -1 - D] up to ref[-1 - D; H + W / 2 + H * D / (W / 2) - 1] are available, then the pattern set defined for the aspect ratio (W / 2) / H can be used.

[0086] Otherwise, (if at least one of those left samples is also not available), then the pattern set defined for the aspect ratio W / H can be used. [E002]

[0087] - Otherwise, if W < H,

[0088] If the left samples from [-1 - D; -1 - D] up to ref[-1 - D; 2 * H + H * D / W - 1] are available (i.e., non - padding samples), then the pattern set defined for the aspect ratio W / H can be used, for example, the patterns used in ECM - 6.0. [E003]

[0089] Otherwise (if at least one of those samples is not available), if from ref[-1 -

[0090] D; -1 - D] up to ref[W + H / 2 + W * D / (H / 2) - 1; -1 - D] all the top samples are available, then the pattern set defined for the aspect ratio W / (H / 2) can be used.

[0091] Otherwise, (if at least one of those top samples is also not available), then the pattern set defined for the aspect ratio W / H can be used [E004].

[0092] The conditions in E001 and E003 check whether all samples are available for the WAIP pattern. If at least one sample is not available, it is possible that, based on subsequent conditions, the regular ECM pattern (i.e., the pattern defined in Table 1) is not used. In some embodiments, this first condition is more conservative and the regular ECM pattern is not used only when all the samples used by the regular ECM pattern are not available. In such embodiments, E001 is written as:

[0093] If at least one top sample from ref[W + W / 2 + (W / 2) * D / H; -1 - D] up to ref[2 * W - 1 + W

[0094] * D / H; -1 - D] is available (i.e., non - padding sample),

[0095] then the pattern set defined for the aspect ratio W / H can be used.

[0096] E003 is written as

[0097] If at least one left sample from ref[-1-D; H+H / 2+(H / 2)*D / W] until ref[-1-D; 2*H-1+H*D / W] is available (i.e., not a padding sample), then the pattern set defined for the aspect ratio W / H can be used.

[0098] Conditions E002 and E004 check whether samples are available for patterns that are not normally used with ECM, and add those patterns only if all samples are available. In some embodiments, this condition is relaxed, and patterns are added if at least one sample is available. In such embodiments, E002 is written as:

[0099] Otherwise, if at least one left sample from ref[-1-D; 2*H+H*D / W] until ref[-1-D; H+W / 2+H*D / (W / 2)-1] is available, then the pattern set defined for the aspect ratio (W / 2) / H can be used. Otherwise, (if none of those samples are available), then the pattern set defined for the aspect ratio W / H can be used.

[0100] E004 is written as:

[0101] Otherwise, if at least one top sample from ref[2*W+W*D / H; -1-D] until ref[W+H / 2-1+W

[0102] *D / (H / 2); -1-D] is available, then the pattern set defined for the aspect ratio W / (H / 2) can be used. Otherwise, (if none of those samples are available), then the pattern set defined for the aspect ratio W / H can be used.

[0103] Some embodiments use combinations of those conditions. Some embodiments use different conditions, depending on but not limited to block size, sequence size, QP, or neighborhood information.

[0104] In some embodiments, more patterns are added. In such embodiments, when the ECM pattern set for W / H is replaced by the pattern set for (W / 2) / H (correspondingly, W / (H / 2)), tests E001 to E004 are performed again as if the width of the block were W / 2 (correspondingly, the height were H / 2). If a new pattern set is selected, this can be performed again until it is no longer preferable to change the pattern set to be used.

[0105] In some embodiments, the existing availability check of neighboring CUs used to build the MPM list can be used to determine whether patterns should be added. For example, as Figure 19As described, the test for the availability of the reference samples is done at positions 1 and 2, and then at positions 3 and 4. In this case, conditions [E001] to [E004] do not depend on whether W is greater than H. In the example, more patterns will be added and no patterns will be removed, which may be particularly useful when signaling is not required, for example, if patterns are added for TIMD or DIMD.

[0106] In one example, if the top samples from ref[-1;-1] until ref[W;-1] are available (i.e., non-padding samples up to position 1), then in addition to the patterns already allowed for that CU, all the patterns described in Table 1 defined for the aspect ratio 2*W / H can be used. If the top sample ref[2*W;-1] is also available (i.e., non-padding sample), then in addition to the patterns already allowed for that CU, all the patterns described in Table 1 defined for the aspect ratio 4*W / H can be used. If the left samples from ref[-1;-1] until ref[-1;H] are available (i.e., non-padding samples up to position 2), then in addition to the patterns already allowed for that CU, all the patterns described in Table 1 defined for the aspect ratio W / (H*2) can be used. If the top sample ref[-1;2*H] is also available (i.e., non-padding sample), then in addition to the patterns already allowed for that CU, all the patterns described in Table 1 defined for the aspect ratio W / (H*4) can be used.

[0107] Signaling of intra mode

[0108] In some embodiments, the number of available IPMs is always kept at 67, with a fixed number of 65 angular IPMs. In such embodiments, no signaling change is required. In other embodiments, the context of the CABAC coding bins for the syntax elements associated with the intra mode signaling, such as the primary MPM flag, the secondary MPM flag, or the first MPM index flag, can be modified to account for the available patterns. For example, one of three context model indices will be selected according to the following conditions:

[0109] 1) The IPMs used are the same as the IPMs currently in ECM-6.0. For example, on a PU where W / H

[0110] = 1, the IPMs will be angular modes 2 to 66 and the planar mode and the DC mode.

[0111] 2) The IPMs used are changed to utilize the IPMs that are typically available for patterns with a larger W / H ratio. For example, on a PU where W / H = 1, this will mean using angular modes from I to I+64 (where I>2) and the planar mode and the DC mode.

[0112] 3) The IPM used is changed to utilize the IPM that is normally available for patterns with a smaller W / H ratio. For example, on a PU with W / H = 1, this would mean using the angular mode from I to I+64 (where I < 2), as well as the planar mode and the DC mode.

[0113] In some embodiments, the flag intra_luma_mpm_flag, which is used to specify whether the intra mode used in the current luminance CB is in the MPM list, causes its syntax to change as follows according to Table 128 in the VTM specification text:

[0114]

[0115] The value of intra_mode_set_diff is derived as follows:

[0116] If the mode set of the current luminance CB for the width and height W and H at positions posX, posY is the set designed for blocks of ratio W / H, as defined in Table 1, then intra_mode_set_diff[posX][posY] is set to 0.

[0117] Otherwise, if the mode set is designed for blocks with a ratio strictly greater than W / H, then intra_mode_set_diff[posX][posY] is set to 1.

[0118] Otherwise, intra_mode_set_diff[posX][posY] is set to 2.

[0119] As defined in Table 70 of the VTM specification text, the context model is initialized as follows:

[0120] Table 70 – Specification of initValue and shiftIdx of ctxIdx for intra_luma_MPM_flag

[0121]

[0122] In some embodiments, three contexts will be defined according to the following:

[0123] 1) The IPM used is the same as the current IPM in ECM-6.0. For example, on a PU with W / H = 1, the IPM will be the angular modes 2 to 66, as well as the planar mode and the DC mode.

[0124] 2) The IPM used is changed to utilize the IPM that is normally available for patterns with a larger abs(log2

[0125] (W / H)) ratio. On a PU with W / H ≥ 1 (correspondingly,

[0126] W / H ≤ 1), which would mean using angular modes from I to I + 64 (where I > J) (correspondingly, I < J) as well as planar mode and DC mode, and J is the smallest angular mode typically available on that PU.

[0127] 3) The IPM used is changed to utilize the IPM that is typically available for modes with a greater abs(log2

[0128] (W / H)) ratio. On PUs where W / H > 1 (correspondingly,

[0129] W / H < 1), which would mean using angular modes from I to I + 64 (where I < J) (correspondingly, I > J) as well as planar mode and DC mode, and J is the smallest angular mode typically available on that PU.

[0130] In some embodiments, the rules mentioned above can be reduced to using only 2 CABAC contexts (i.e., depending on whether the mode changes). In some embodiments, those rules can be combined with other information such as, but not limited to, PU size, sequence size, QP, prediction tools used.

[0131] In some embodiments, the number of available modes varies by PU, and the signaling thus changes to account for the different number of modes. In one embodiment, the first MPM list and the secondary MPM list maintain the original number of flags to be coded, and the remaining IPMs are coded using truncated binary coding of N - 22 symbols, where N is the number of available modes for that PU (where N = 45 means using the same signaling as in ECM-6.0, N > 45 means more modes are available than in ECM-6.0, and N < 45 means fewer modes are available than in ECM-6.0).

[0132] In some embodiments, the limitations on IPM availability are restricted by TIMD (template-based intra mode derivation) and / or DIMD and / or other decoder-side tools to avoid any change in signaling required. In such embodiments, the TIMD search (correspondingly, DIMD search and / or other decoder-side tools) may be restricted to the IPMs considered available.

[0133] In embodiments where TIMD can use additional wide angles, only a subset of the added modes is included to reduce the increase in search complexity. For example, the additional modes can be included in the first part of the search as described below.

[0134] Let W and H be the width and height of the block to be coded, let minOrg and maxOrg be the minimum and maximum angular intra mode values available for the current block, use the TIMD values for 131 modes (e.g., for W / H = 2, according to Table 1, the angular modes range from 8 to 72, which corresponds to minOrg = 13 and maxOrg = 141), and let newMin and newMax be the new minimum and maximum angular intra mode values available for the current block, e.g., determined from E001 to E004 (e.g., if all neighboring samples are available for a block with W / H = 2, then according to Table 1, modes from -4 to 7 and from 73 to 78 are added, which means using the TIMD values for 131 modes, newMin = -9 and newMax = 153). Starting from newMin + 1 with a step size of N (e.g., N = 5) and until newMax, if the mode is not between minOrg and maxOrg, add it to the first part of the TIMD search; otherwise, do not add the mode.

[0135] It is also possible to select to always add certain modes to the search. For example, it can be chosen to always add newMin + 1, orgMin - 1, orgMax + 1, and newMax + 1 to the first part of the search. Those modes can be the only modes added to the search to reduce the design complexity; or they can be added on top of the modes already added to maximize the compression gain. The second part (refinement part) of the TIMD search can be done as in ECM.

[0136] In some embodiments, before decoding the IPM index, if some additional modes are found to be available, decode an additional flag to indicate whether the original 67 modes allowed for the current block size were used, or whether one of the N additional modes was used. If one of the N additional modes was used, use a truncated binary code of N symbols to decode the additional index.

[0137] Propagation of Intra Direction

[0138] In ECM, if the left neighboring block has W / H = 2 and prediction is performed using angular mode 67, the mode index used to construct the MPM list for the current block is 2. However, if the current luma block is square, index 2 corresponds to angular mode 2. Thus, using the mode index of the neighboring mode to construct the MPM list for the current block may result in adding modes to the MPM list that are never actually used and should not therefore be considered "the most likely" modes for decoding the current luma coded block. Additionally, in ECM, each index can correspond to two different angular modes (e.g., index 2 is angular mode 2 or angular mode 67, index 3 is angular mode 3 or angular mode 68, etc.), but the two different modes are not 180° opposite.

[0139] Two modes being opposite by 180° means that they predict the same directional texture (both angle mode 2 and 66 predict a direction of 45°), but from different references (angle mode 2 predicts from bottom left to top right, while angle mode 66 predicts from top right to bottom left).

[0140] In some embodiments, the MPM list is created based on the actual mode used by neighboring blocks rather than the index used. For example, when constructing the MPM list and the neighboring mode is not available, the mode is replaced by the corresponding mode at 180°, i.e., if the mode IPM is less than 34, the mode is replaced by IPM + 64, and otherwise the mode is replaced by IPM - 64. In ECM to ECM-6.0, this is always possible because there is always an 180° angle mode span.

[0141] In an encoder or decoder that removes some modes without adding other modes, such as the encoder or decoder described in some previous embodiments, this may not be the case. If, during the construction of the MPM list, a mode that is not available in the current luminance CB should be added, the mode is replaced by the closest available mode. In some embodiments, the modes are selected to always have a span of 65 angle modes to ensure that any angle is available and it is always possible to replace a mode with its 180° counterpart.

[0142] Ultra-wide angle intra mode

[0143] The selection of available intra prediction modes based on the availability of neighboring decoded reference samples can be extended to rules that no longer involve Table 1. This means that for a given W×H block, depending on the availability of its neighboring decoded reference samples, its set of available intra prediction modes (see Table 1) for its effective ratio W / H will no longer be replaced by a different set of available intra prediction modes associated with the ratio of the width and height close to its effective ratio W / H. Instead, for a given availability of neighboring decoded reference samples of a given W×H block, a given number of intra prediction modes can be removed.

[0144] For example, the rules for potentially suppressing intra prediction modes can be as follows. For a given W×H block, if all W decoded reference samples located on its upper right side are not available, then the last positive vertical intra prediction modes are disabled (e.g., n 0 = 4). The "last" n 0 positive vertical intra prediction modes refer to the n 0 positive vertical intra prediction modes having the largest angle in absolute value with respect to the vertical axis. Following the ECM nomenclature, the "last" n 0 positive vertical intra prediction modes refer to the n having the largest index0 positive vertical intra prediction mode. If all H decoded reference samples located at the lower left side are not available, the previous positive horizontal intra prediction modes are disabled (e.g., n 1 = 4). The "previous" n 1 positive horizontal intra prediction modes refer to the n positive horizontal intra prediction modes having the largest angle in absolute value with respect to the horizontal axis. According to the ECM nomenclature, the "previous" n 1 positive horizontal intra prediction modes refer to the n positive horizontal intra prediction modes having the smallest index. In the case of wide-angle intra prediction, these indices can take negative values. 1 positive horizontal intra prediction modes refer to the n positive horizontal intra prediction modes having the smallest index. 1 In the case of wide-angle intra prediction, these indices can take negative values.

[0145] As an example, in the case where a given W×H luma coding block (CB) belongs to an intra slice in ECM-5.0, this rule can be illustrated in FIG. 12. For this given luma CB (1201) within the intra slice in ECM-5.0, the disabled intra prediction modes are identified according to the partitioning history of (1201). Here, W = 16, and H = 8. The indices 0, 1, 2, and 3 indicate the order of the first four luma CBs encoding / decoding the first 64×64 luma CB (1200) resulting from the QT split of the considered luma CTB.

[0146] In this example, it is important to note that for this given W×H luma CB, at the time of encoding, before any bit of the partitioning of its parent luma coding tree block (CTB) is written to the bitstream, i.e., before any bit associated with the intra prediction within its parent luma CTB is written, the availability of its neighboring decoded reference samples can be fully specified. For this given W×H luma CB, at the time of decoding, immediately after the bits of the partitioning of its parent luma CTB are read from the bitstream, i.e., before any bit associated with the intra prediction within this parent luma CTB is read, the availability of its neighboring decoded reference samples can be fully specified. In the following example, in ECM-5.0, the CTU size is set to 128, as in VVC, to obtain an example more comparable to its version in VVC.

[0147] On the encoder side, a given 128×128 luminance CTB is split into four 64×64 luminance CBs via a quadtree (QT). For example, consider the first 64×64 luminance CB (1200). Let us characterize the split at a given depth by (typeSplit, idxChild), where "typeSplit" refers to the split type in {quadtree (QT), horizontal binary tree (BT_H), vertical binary tree (BT_V), horizontal ternary tree (TT_H), vertical ternary tree (TT_V)}, and "idxChild" indicates the index in the coding order of the considered child CB produced by the split. Then, the partitioning of the considered luminance CB (1201) is fully described by its split tree {(QT, 0), (QT, 0), (BT_V, 1), (TT_H, 2)}, as Figure 12A shown. From this split tree, it becomes evident that W decoded references are not available on the upper-right side of (1201) and H decoded reference samples are not available on the lower-left side of (1201).

[0148] These two parts of unavailable decoded reference samples are summarized as (1202) in Figure 12B . For example, this can be indicated by the flag "is_above_right_full" attached to 0 of (1201) and "is_below_left_full" attached to 0 of (1201), respectively. Finally, the last n 0 positive vertical intra prediction modes are disabled, and (1204) and (1203) represent the directions of the intra prediction modes with the minimum index and the maximum index in the disabled mode set, respectively. The first positive horizontal intra prediction modes are disabled, and (1205) and (1206) represent the directions of the intra prediction modes with the minimum index and the maximum index in the disabled mode set, respectively, as Figure 12C shown.

[0149] Finally, the signaling of the index of the intra prediction mode selected to predict (1201) is adapted to account for the removed intra prediction modes. For example, if the selected intra prediction mode is not a template-based intra prediction (TMP), decoder-side intra mode derivation (DIMD), template-based intra mode derivation (TIMD), or matrix-based intra prediction (MIP) mode, does not use MRL, and is not an MPM, its index is truncated binary coded, where the code length is N - n 0 -n 1 . Note that since ECM-5.0 includes 67 regular intra prediction modes (65 directions, planar, and DC), 6 primary MPMs, and 16 secondary MPMs, the index of a regular intra prediction mode that is not an MPM can have N = 45 possibilities.

[0150] On the decoder side, except for the signaling of the index of the intra prediction mode selected for prediction (1201), the process follows the process on the encoder side. Given the above example, if the selected intra prediction mode is not TMP, DIMD, TIMD or MIP mode, MRL is not used, and it is not MPM, its index is 45 - n 0 -n 1 and decoded with a truncated binary code of the possible symbols.

[0151] Another example of this embodiment can be depicted in FIG. 13 in the case where a given W×H luminance coding block (CB) belongs to an intra slice in ECM - 5.0. For a given W×H luminance CB (1301) within an intra slice in ECM - 5.0, the disabled intra prediction modes are identified based on the partitioning history of (1301). Here, W = 8 and H = 16. Indices 0 to 7 indicate the order of the first eight luminance CBs among the first 64×64 luminance CBs (1300) resulting from the QT splitting of the considered luminance CTB.

[0152] On the encoder side, given that a 128×128 luminance CTB is split into 4 64×64 luminance CBs via QT. For example, consider the first 64×64 luminance CB (1300). The partitioning of the considered luminance CB (1301) is fully specified by its splitting tree {(QT, 0), (QT, 1), (BT_H, 1), (BT_V, 0), (BT_V, 1)}, as Figure 13A shown.

[0153] Based on this splitting tree, it can be directly inferred that all W decoding references on the upper - right side of (1301) are available, while the fact that the H decoding reference samples on the lower - left side of (1301) are not available, as shown by (1302) in Figure 13B . For example, this can be indicated by the flag "is_above_right_full" attached to (1301) at 1 and the flag "is_below_left_full" at 0 respectively. The first positive horizontal intra prediction modes are disabled, and (1303) and (1304) respectively represent the directions of the intra prediction modes with the minimum index and the maximum index in this set of disabled modes, as shown in Figure 13C . Finally, the signaling of the index of the intra prediction mode selected to predict (1301) is adapted to account for the removed intra prediction modes. For example, if the selected intra prediction mode is not TMP, DIMD, TIMD or MIP mode, MRL is not used, and it is not MPM, its index is encoded with a truncated binary code of 45 - n 1 possible symbols.

[0154] On the decoder side, except for the signaling of the index of the intra mode selected for prediction (1301), the process follows the process on the encoder side. Given the above example, if the selected intra prediction mode is not TMP, DIMD, TIMD or MIP mode, MRL is not used, and it is not MPM, its index is decoded with a truncated binary code of 45 - n 1 possible symbols.

[0155] The above example can be applied to any other block in another channel / slice. In addition, the rule for suppressing the intra prediction mode according to the availability of the neighboring decoded reference samples of the current block can be directly modified. For example, this rule can become "for a given W×H block, if the rightmost W / 2 decoded reference samples located on its upper right side are all unavailable, then the last positive vertical intra prediction modes are disabled. If the bottommost H / 2 decoded reference samples located on its lower left side are all unavailable, then the first positive horizontal intra prediction modes are disabled".

[0156] For a given W×H block, the workflow for encoding the index of the selected intra prediction mode on the encoder side following this embodiment can be summarized by Figure 14 . In this embodiment, the rule indicating the conditional relationship between the availability of the neighboring reference samples of a given block and which intra prediction modes are removed for the given block is known on the encoder side. In particular, for the current block, the encoder identifies (1410) the unavailable reference samples based on the partitioning history of the current block. For example, for the partitioning history of {(QT, 0), (QT, 0), (BT_V, 1), (TT_H, 2)} in Figure 12, is_above_right_full = false and is_below_left_full = false. Based on the identified unavailable reference samples and the rule for removing intra prediction modes, the encoder removes (1420) some intra prediction modes (these intra prediction modes are unavailable for the current block). For example, for is_above_right_full = false and is_below_left_full = false, the last n 0 positive vertical intra prediction modes and the first n 1 positive horizontal intra prediction modes are disabled in Figure 12.

[0157] Taking into account the removed modes, the encoder adapts (1430) the signaling of the index of the intra prediction mode selected to predict the current block. Generally, the total number of available intra prediction modes is adjusted by reducing the number of removed intra prediction modes. For example, if the selected intra prediction mode is not TMP, DIMD, TIMD or MIP mode, MRL is not used, and it is not MPM, its index is truncated binary coded in Figure 12, where the code length is 45 - n0 -n 1 Then, the encoded index is written into the (1440)-bit stream.

[0158] For this W×H block, the workflow for decoding the index of the selected intra prediction mode on the decoder side following this embodiment can be outlined in Figure 15 . The steps (1510, 1520, 1530, 1540) on the decoder side correspond to the steps on the encoder side.

[0159] These two figures only illustrate the workflow for a single block. When considering multiple blocks, Figure 14 and Figure 15 the order of the steps in

[0160] In FIG. 12 - Figure 15 , for a given W×H block, unavailable decoded reference samples are identified based on the partitioning history of the block. In other embodiments, a function that searches for already decoded blocks around the current block can be used to identify the unavailability of decoded reference samples.

[0161] For example, as Figure 16 illustrates, for a given W×H CB (1602) in an intra slice in ECM - 5.0, the function "getCURestricted" takes the pixel position "pos" (e.g., "posAR" (1603) or "posBL" (1604)), the coding unit (CU) "curCU" (1602) of the given W×H CB, and the channel type "chType" of (1602) to return a pointer to the decoded CB containing the pixel at "pos". If the pixel at "pos" does not belong to any CB, or it belongs to a CB that has not been decoded, then "getCURestricted" can return the pointer NULL, e.g., "nullptr" in C++. In Figure 16 , the CBs (1600), (1601), and (1602) are generated from the last two splits of BT_V and BT_H in the current state of encoding / decoding. For example, in Figure 16 , since "posAR" belongs to a CB that has not been decoded, "getCURestricted(posAR, curCU, chType)" returns "nullptr".

[0162] For a given W×H block, the workflow for encoding the index of the selected intra prediction mode on the encoder side following this embodiment can be given by Figure 17Summarized. In particular, for the current block, the encoder identifies (1710) unavailable reference samples based on the partitioning history of the current block. For example, if getCURestricted(posAR, curCU, chType) == nullptr, then is_above_right_full = false; if getCURestricted(posBL, curCU, chType) == nullptr, then is_below_left_full = false in Figure 16 . Based on the identified unavailable reference samples and the rules regarding the removal of intra prediction modes, the encoder removes (1720) some intra prediction modes that are unavailable for the current block. For example, for is_above_right_full = false and is_below_left_full = false, the last n 0 positive vertical intra prediction modes and the first n 1 positive horizontal intra prediction modes are disabled.

[0163] Taking into account the removed modes, the encoder adapts (1730) the signaling of the index of the intra prediction mode selected to predict the current block. For example, if the selected intra prediction mode is not a TMP, DIMD, TIMD, or MIP mode, does not use MRL, and is not an MPM, its index is truncated binary coded with a code length of 45 - n 0 -n 1 . Then, the encoded index is written (1740) to the bitstream.

[0164] For the W×H block, the workflow for decoding the index of the selected intra prediction mode on the decoder side following this embodiment can be summarized in Figure 18 . The steps (1810, 1820, 1830, 1840) on the decoder side correspond to the steps on the encoder side.

[0165] In another embodiment, for a given block, instead of removing a given number of intra prediction modes based on the availability of its neighboring decoded reference samples, the MPM list for it can be reordered such that for some intra prediction modes that centrally use unavailable decoded reference samples for prediction and have their indices within that MPM list, their indices are moved towards the end of that MPM list. The intra prediction modes whose indices are moved towards the end of the MPM list of a given block are considered to have a relatively lower likelihood of being selected to predict the given block.

[0166] Figure 19 depicts an embodiment of the current W×H luminance CB in an intra slice in ECM - 5.0. In particular, Figure 19Shows the creation of a general list of 22 MPMs for the currently given W×H luminance CB. The first 6 MPMs in the general list of MPMs correspond to the primary MPM list, while the last 16 MPMs in the general list of MPMs generate the secondary MPM list.

[0167] In Figure 19 it, the planar mode is first added to the general list of MPMs (1900). Then, the indices of the intra prediction modes selected to predict the left, top, bottom - left, top - right, and top - left luminance CBs are added to the general list of MPMs (1901 - 1905). Then, the indices of the two intra prediction modes derived via DIMD for the current luminance CB are added to the general list of MPMs (1906, 1907). Then, if the current second MPM is neither planar nor DC, the indices of its eight neighboring angular intra prediction modes are put into the general list of MPMs (1908). Then, if the current third MPM is neither planar nor DC, the indices of its eight neighboring angular intra prediction modes are put into the general list of MPMs (1909).

[0168] After potentially adding more indices of angular intra prediction modes according to (1910), the index of the default mode is inserted into the general list of MPMs (1911) to reach 22 MPMs. Note that each of the above insertions applies under the condition that there is no redundancy in the general list of MPMs. This means that for the index of the current intra prediction to be inserted into the general list of MPMs, if the index already exists in the list, the insertion is skipped.

[0169] Figure 20 Illustrates the creation of a general list of 22 MPMs for the same current luminance CB according to an embodiment. In Figure 20 it, except that re - ordering may be introduced, the creation of the general list of 22 MPMs for the current W×H luminance CB follows the Figure 19 workflow in it. For example, the function f may take the index of the candidate intra prediction mode to be put into the general list of MPMs as the first argument and the array "res" holding the reserved mode indices as the second argument. Then, if the candidate intra prediction mode is "valid" under the condition depending on the availability of the decoded reference samples of the current W×H luminance CB, f may put the index of the candidate intra prediction mode into the general list of MPMs. Otherwise, f may add the index of the intra prediction mode to "res".

[0170] The planar mode is first added to the general list of MPMs (2000). Then, under the validity condition defined by f, the indices of the intra prediction modes selected to predict the left, top, bottom - left, top - right, and top - left luminance CBs are added to the general list of MPMs (2001 - 2005). Then, all the intra prediction mode indices stored in "res" are added to the general list of MPMs (2006). Then, the indices of the two intra prediction modes derived via DIMD for the current luminance CB are added to the general list of MPMs (2007, 2008). In Figure 19 Figure 19 , the last steps (2009), (2010), (2011), and (2012) follow (1908), (1909), (1910), and (1911) respectively.

[0171] In another embodiment, f can be applied (or not applied) to the index of any candidate intra prediction mode that is potentially to be added to the general list of MPMs for the current luminance CB. Additionally, the step of putting all the intra prediction mode indices stored in "res" into the general list of MPMs for the current luminance CB can occur at any time during the creation of the general list of MPMs.

[0172] For example, in Figure 21 the illustrated embodiment, the addition of all the intra prediction modes stored in "res" (2108) occurs after potentially putting the indices of the two intra prediction modes derived via DIMD for the current luminance CB into the general list of MPMs (2106, 2107).

[0173] For example, in Figure 22 another illustrated embodiment, the addition of all the intra prediction modes stored in "res" (2208) occurs after, under the validity condition defined by f, for the current luminance CB, potentially putting the indices of the two intra prediction modes derived via DIMD (2206, 2207) into the general list of MPMs.

[0174] In one embodiment, the validity condition defined by f for the index of the intra prediction mode passed as the first argument can be that if all W decoded reference samples located on the top - right side of the current W×H block are not available, then the last positive vertical intra prediction modes are invalid (e.g., n 0 = 8). If all H decoded reference samples located on the bottom - left side of the current W×H block are not available, then the first positive horizontal intra prediction modes are invalid (e.g., n 1 = 8).

[0175] In another embodiment, the validity condition defined by f for the index of the intra prediction mode passed as the first independent variable may be that if the rightmost W / 2 decoded reference samples located on the upper right side of the current W×H block are all unavailable, then the last positive vertical intra prediction modes are invalid (e.g., n 0 = 4). If the bottommost H / 2 decoded reference samples located on the lower left side of the current W×H block are all unavailable, then the first positive horizontal intra prediction modes are invalid (e.g., n 1 = 4).

[0176] In yet another embodiment, the validity condition defined by f for the index of the intra prediction mode passed as the first independent variable may be that if the decoded reference samples located above the current W×H block (including upper left, above, and upper right) are all unavailable, then the last vertical intra prediction modes are invalid (e.g., q 0 = 8). If the decoded reference samples located to the left of the current W×H block (including upper left, left, and lower left) are all unavailable, then the first horizontal intra prediction modes are invalid (e.g., q 1 = 8).

[0177] In yet another embodiment, the validity condition defined by f for the index of the intra prediction mode passed as the first independent variable may be a combination of several conditions depending on different states of the availability of the decoded reference samples of the current W×H block. As an example, if the decoded reference samples located above the current block (including upper left, above, and upper right) are all unavailable, then the last vertical intra prediction modes are invalid (e.g., q 0 = 8). Otherwise, check the following condition. If the W decoded reference samples located on the upper right side of the current block are all unavailable, then the last positive vertical intra prediction modes are invalid (e.g., n 0 = 5). As another example, if the decoded reference samples located to the left of the current block (including upper left, left, and lower left) are all unavailable, then the first horizontal intra prediction modes are invalid (e.g., q 1 = 5). Otherwise, check the following condition. If the H decoded reference samples located on the lower left side of the current block are all unavailable, then the first positive horizontal intra prediction modes are invalid (e.g., n 1 = 5).

[0178] This document describes various methods, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of the specific steps and / or actions can be modified or combined. Additionally, terms such as "first", "second", etc. may be used in various embodiments to modify elements, components, steps, operations, etc., such as "first decoding" and "second decoding". Unless specifically required, the use of such terms does not imply an order of the modified operations. Thus, in this example, the first decoding does not need to be performed before the second decoding, and can occur, for example, before the second decoding, during the second decoding, or in a time period overlapping with the second decoding.

[0179] The various methods and other aspects described in this application can be used to modify the modules of the video encoder 200 and decoder 300 as shown in Figure 2 and Figure 3 e.g., the intra prediction modules (260, 360). Additionally, this aspect is not limited to ECM, VVC, or HEVC, but can be applied to, for example, other standards and recommendations, as well as extensions of any such standards and recommendations. Unless otherwise indicated or technically precluded, the aspects described in this application can be used alone or in combination.

[0180] Various numerical values are used in this application. The specific values are for illustrative purposes, and the described aspects are not limited to these specific values.

[0181] Various embodiments relate to decoding. As used in this application, "decoding" can cover, for example, all or part of the process performed on a received encoded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. Based on the context of the specific description, whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to the more general decoding process will be clear and is considered well understood by those skilled in the art.

[0182] Various embodiments relate to encoding. In a manner similar to the discussion above regarding "decoding", as used in this application, "encoding" can cover, for example, all or part of the process performed on an input video sequence to produce an encoded bitstream.

[0183] The various embodiments and aspects described herein can be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of an embodiment (e.g., only as a method), the embodiments of the features discussed can also be implemented in other forms (e.g., an apparatus or a program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. A method can be implemented in, for example, an apparatus such as a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. A processor also includes a communication device such as a computer, a cellular phone, a portable / personal digital assistant (“PDA”), and other devices that facilitate information communication between end users.

[0184] References to “one embodiment” or “an embodiment” or “one implementation” or “an implementation” and other variations thereof mean that the specific features, structures, characteristics, etc. described in connection with the embodiment are included in at least one embodiment. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation” and any other variations that occur throughout this application do not necessarily all refer to the same embodiment.

[0185] Additionally, this application can relate to “determining” various pieces of information. Determining information can include one or more of the following: for example, estimating information, calculating information, predicting information, or retrieving information from a memory.

[0186] Furthermore, this application can relate to “accessing” various pieces of information. Accessing information can include one or more of the following: for example, receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0187] Additionally, this application can relate to “receiving” various pieces of information. Like “accessing”, receiving is intended to be a broad term. Receiving information can include one or more of the following: for example, accessing information or retrieving information (e.g., from a memory). Furthermore, “receiving” is typically involved in one way or another during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0188] It should be appreciated that, for example, in the cases of "A / B", "A and / or B", and "at least one of A and B", the use of any of the following, namely " / ", "and / or", and "at least one", is intended to cover the selection of only the first-listed option (A), or only the second-listed option (B), or the selection of both options (A and B). As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to cover the selection of only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or the selection of only the first and second-listed options (A and B), or the selection of only the first and third-listed options (A and C), or the selection of only the second and third-listed options (B and C), or the selection of all three options (A and B and C). As will be clear to those of ordinary skill in the art and related fields, this can be extended to as many items as are listed.

[0189] Furthermore, as used herein, the word "signal" refers to, among other things, indicating something to a corresponding decoder. For example, in some embodiments, the encoder signals a quantization matrix for dequantization. Thus, in an embodiment, the same parameters are used at both the encoder side and the decoder side. Accordingly, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder such that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters as well as other parameters, signaling without transmission (implicit signaling) can be used to simply allow the decoder to know and select the specific parameters. By avoiding the transmission of any actual functions, bit savings are achieved in various embodiments. It should be appreciated that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. Although the foregoing relates to the verb form of the word "signal", the word "signal" can also be used as a noun herein.

[0190] As will be apparent to those of ordinary skill in the art, embodiments can generate a variety of signals that are formatted to carry information that can be stored or transmitted, for example. The information can include, for example, instructions for performing a method, or data generated by one of the described embodiments. For example, a signal can be formatted to carry the bitstream of a described embodiment. Such a signal can be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, signals can be transmitted over a variety of different wired or wireless links. Signals can be stored on a processor-readable medium.

Claims

1. A video decoding method, comprising: identifying the availability of one or more reference samples for a block to be decoded in a picture; obtaining a set of intra prediction modes for the block in response to the availability of the one or more reference samples; obtaining an intra prediction mode from the set of intra prediction modes; and performing intra prediction on the block to be decoded based on the intra prediction mode for the block to form a predicted block for the block.

2. A video encoding method, comprising: identifying the availability of one or more reference samples for a block to be encoded in a picture; obtaining a set of intra prediction modes for the block in response to the availability of the one or more reference samples; selecting an intra prediction mode from the set of intra prediction modes; and performing intra prediction on the block to be encoded based on the intra prediction mode for the block to form a predicted block for the block.

3. The method according to claim 1 or 2, wherein the availability of the one or more reference samples is identified based on the partitioning history of the block.

4. The method according to any one of claims 1 - 3, wherein, the obtaining of the set of intra prediction modes includes: obtaining a first set of intra prediction modes; and adjusting the first set of intra prediction modes to the set of intra prediction modes in response to the availability of the one or more reference samples.

5. The method according to any one of claims 1 - 4, wherein, the first set of intra prediction modes is obtained based on the aspect ratio of the block.

6. The method according to claim 5, wherein, the set of intra prediction modes is obtained based on an aspect ratio different from the aspect ratio of the block.

7. The method according to claim 6, wherein the different aspect ratio is half or twice the aspect ratio of the block.

8. The method according to any one of claims 1 - 7, wherein the adjustment comprises: removing a plurality of vertical positive intra prediction modes in response to the unavailability of one or more upper - right reference samples.

9. The method according to any one of claims 1 - 8, wherein the adjustment comprises: removing a plurality of horizontal positive intra prediction modes in response to the unavailability of one or more lower - left reference samples.

10. The method according to any one of claims 1 - 9, wherein the adjustment comprises: adding a plurality of vertical positive intra prediction modes in response to the availability of one or more upper - right reference samples.

11. The method according to any one of claims 1 - 10, wherein the adjustment comprises: adding a plurality of horizontal positive intra prediction modes in response to the availability of one or more lower - left reference samples.

12. The method according to any one of claims 1 - 11, wherein in response to the number of intra prediction modes in the set of intra prediction modes, signaling an index corresponding to the intra prediction mode.

13. The method according to claim 12, in response to the intra prediction mode belonging to the remaining mode set, the remaining mode set does not include the most probable mode (MPM), and wherein the index is encoded using truncated binary coding for N - M symbols, where N is the number of intra prediction modes in the intra prediction mode set, and M is the number of MPMs.

14. The method according to any one of claims 1 - 13, wherein the context index for a syntax element depends on the intra prediction mode set.

15. The method according to claim 14, wherein, the syntax element is used to signal the intra prediction mode.

16. An apparatus for video decoding, comprising one or more processors, wherein the one or more processors are configured to: identify the availability of one or more reference samples for a block to be decoded in a picture; in response to the availability of the one or more reference samples, obtain an intra prediction mode set for the block; obtain an intra prediction mode from the intra prediction mode set; and perform intra prediction on the block to be decoded based on the intra prediction mode for the block to form a predicted block for the block.

17. An apparatus for video encoding, comprising one or more processors, wherein the one or more processors are configured to: identify the availability of one or more reference samples for a block to be encoded in a picture; in response to the availability of the one or more reference samples, obtain an intra prediction mode set for the block; select an intra prediction mode from the intra prediction mode set; and perform intra prediction on the block to be encoded based on the intra prediction mode for the block to form a predicted block for the block.

18. The apparatus according to claim 16 or 17, wherein the availability of the one or more reference samples is identified based on the partitioning history for the block.

19. The apparatus according to any one of claims 16 - 18, wherein the one or more processors are configured to obtain the intra prediction mode set by: obtain a first intra prediction mode set; and in response to the availability of the one or more reference samples, adjust the first intra prediction mode set to the intra prediction mode set.

20. The apparatus according to any one of claims 16 - 19, wherein, the first intra prediction mode set is obtained based on the aspect ratio of the block.

21. A signal comprising video data, formed by performing the method according to any one of claims 2 - 15.

22. A computer - readable storage medium having stored thereon instructions for encoding or decoding video according to the method according to any one of claims 1 - 15.

Images