Methods, apparatus, and computer programs for video decoding

Abstract

The present disclosure includes a method and an apparatus including a processing circuit that receives, from a bitstream having a current block in a picture, coding information in the bitstream indicating that the current block is coded in an angular intra prediction mode using intra-interpolation filters. The processing circuit applies each of a predetermined set of intra-interpolation filters to neighboring reconstructed samples in N neighboring lines from a boundary of the current block. The processing circuit selects an intra-interpolation filter from the predetermined set of intra-interpolation filters based on a prediction error associated with each of the predetermined set of intra-interpolation filters, and predicts samples in the current block using the selected intra-interpolation filter using the angular intra prediction mode. The processing circuit reconstructs the current block based on the predicted samples.

Description

Technical Field

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 444,877, filed on February 10, 2023, "Self-guided Intra interpolation filter" and to U.S. Patent Application No. 18 / 384,759, filed on October 27, 2023, "SELF-GUIDED INTRA INTERPOLATION FILTER". The disclosures of these prior applications are hereby incorporated by reference in their entirety.

[0002] This disclosure generally describes aspects related to video coding.

Background Art

[0003] The background description presented herein is for the purpose of generally presenting the situation related to the disclosure. The work of the inventors named herein within the scope described in this background section, as well as aspects of the description that might not otherwise be eligible as prior art at the time of filing, are not admitted as prior art to this disclosure, either explicitly or implicitly.

[0004] Image / video compression can help transmit image / video data across different devices, storage devices, and networks with minimal quality degradation. In some cases, video codec technology can compress video based on spatial and temporal redundancy. In one example, a video codec can use a technique called intra-prediction, which can compress images based on spatial redundancy. For example, intra-prediction can use reference data from the currently reconstructed picture for sample prediction. In another example, a video codec can use a technique called inter-prediction, which can compress images based on temporal redundancy. For example, inter-prediction can use motion compensation to predict samples in the current picture from previously reconstructed pictures. Motion compensation can be represented by motion vectors (MV). [Overview of the Initiative]

[0005] Aspects of this disclosure include methods and apparatus for video coding / decoding. In some examples, the apparatus for video decoding includes a processing circuit. In one example, the processing circuit receives coding information of a bitstream having a current block in a picture, indicating that the current block is coded in angular intra-prediction mode using an intra-interpolation filter. The processing circuit applies each of a predetermined set of intra-interpolation filters to adjacent reconstructed samples in N adjacent lines from the boundary of the current block, where N is a positive integer. Based on the prediction error associated with each of the predetermined set of intra-interpolation filters, the processing circuit selects one intra-interpolation filter from the predetermined set of intra-interpolation filters and uses the selected intra-interpolation filter to predict the samples in the current block using the angular intra-prediction mode.

[0006] In one example, the processing circuit receives the bitstream of the current block in the picture. The coding information in the bitstream indicates that the current block is coded in angular intra-prediction mode using an intra-interpolation filter. Based on the adjacent reconstruction samples of the current block, the processing circuit selects an intra-interpolation filter from a predetermined set of intra-interpolation filters. The adjacent reconstruction samples include reconstruction samples in N lines from one or more boundaries of the current block. The processing circuit applies the selected intra-interpolation filter to reference samples in reference lines in the picture to predict samples in the current block using angular intra-prediction mode, and reconstructs the current block based on the predicted samples.

[0007] In one example, the processing circuit selects either (i) the type of intra interpolation filter or (ii) the number of taps in the intra interpolation filter from a predetermined set of intra interpolation filters based on the adjacent reconstruction samples of the current block.

[0008] In one example, a predetermined set of intra interpolation filters includes different types of intra interpolation filters. The processing circuit selects a type of intra interpolation filter from the different types based on the adjacent reconstruction samples of the current block. The selected intra interpolation filter is one of the following: a bilinear interpolation filter, a cubic interpolation filter, a spline interpolation filter, a DCT-based interpolation filter, or a DST-based interpolation filter.

[0009] In one example, a predetermined set of intra interpolation filters includes different tap counts. The processing circuit selects the number of taps for the intra interpolation filter based on the adjacent reconstructed samples of the current block. The number of taps for the intra interpolation filter is one of 2, 4, 6, or 8 taps.

[0010] In one example, N is greater than 1, and the adjacent reconstruction sample includes adjacent reconstruction samples of multiple lines. For each combination of an intra interpolation filter from a predetermined set of intra interpolation filters and an adjacent reconstruction sample of a line from the multiple lines of adjacent reconstruction samples, the processing circuit uses the respective intra interpolation filter to predict the adjacent reconstruction sample of each line based on one or more remaining lines from the multiple lines of adjacent reconstruction samples, obtains at least one prediction error, and selects the intra interpolation filter corresponding to the smallest prediction error among the obtained prediction errors.

[0011] In one example, whether the adjacent reconstruction sample includes (i) the adjacent reconstruction sample of the left line to the left of the current block, (ii) the adjacent reconstruction sample of the upper line above the current block, or (iii) the adjacent reconstruction sample of the left line to the left of the current block and the adjacent reconstruction sample of the upper line above the current block depends on the intra-prediction direction of the angle intra-prediction mode.

[0012] In one example, the value of N depends on the block size.

[0013] In one example, the processing circuit predicts the adjacent reconstruction sample for each line based on the intra-prediction direction of the angle intra-prediction mode.

[0014] In one example, the processing circuit predicts the adjacent reconstruction sample for each line based on a direction opposite to the intra-prediction direction of the angle intra-prediction mode.

[0015] In one example, the processing circuit selects one intra interpolation filter from a predetermined set of intra interpolation filters and selects an angle intra prediction mode from a plurality of angle intra prediction modes, based on the adjacent reconstruction samples of the current block.

[0016] In one example, the processing circuit derives N1 angle intra-prediction modes from a set of angle intra-prediction modes, each associated with the lowest cost value among N1 of the cost values ​​of each angle intra-prediction mode. Each cost value is determined based on the adjacent reconstruction sample of the current block, each angle intra-prediction mode, and the default intra-interpolation filter from a predetermined set of intra-interpolation filters. For each of the N1 angle intra-prediction modes, the processing circuit selects M1 intra-interpolation filters from the predetermined set of intra-interpolation filters, each associated with the lowest cost value among M1 of the cost values ​​of the predetermined set of intra-interpolation filters. Each cost value is determined based on the adjacent reconstruction sample of the current block, each intra-interpolation filter, and each angle intra-prediction mode from the N1 angle intra-prediction modes. The processing circuit selects one intra-interpolation filter and angle intra-prediction mode from N1 × M1 combinations. Each of the N1 × M1 combinations includes one intra interpolation filter from among the M1 intra interpolation filters and one angle intra prediction mode from among the N1 angle intra prediction modes.

[0017] In one example, the processing circuit selects N2 angle intra-prediction modes from a first set of multiple angle intra-prediction modes based on the adjacent reconstruction sample of the current block and the intra-interpolation filter. The processing circuit selects a first updated intra-interpolation filter based on the adjacent reconstruction sample of the current block and one of the N2 angle intra-prediction modes. The processing circuit selects N3 angle intra-prediction modes from a second set of multiple angle intra-prediction modes based on the adjacent reconstruction sample of the current block and the first updated intra-interpolation filter. The processing circuit selects a second updated intra-interpolation filter based on the adjacent reconstruction sample of the current block and one of the N3 angle intra-prediction modes. The processing circuit selects a second updated intra interpolation filter as the intra interpolation filter to select and one of the N3 angle intra prediction modes as the angle intra prediction mode to select, based on (i) the difference between a first prediction error associated with a first updated intra interpolation filter and one of the N2 directional intra prediction modes and a second prediction error associated with a second updated intra interpolation filter and one of the N3 angle intra prediction modes.

[0018] In one example, for each combination of an angle intra-prediction mode from among several angle intra-prediction modes and an intra-interpolation filter from a predetermined set of intra-interpolation filters, the processing circuit determines a cost value for each combination based on the adjacent reconstruction samples of the current block, selects K combinations based on the determined cost values, and selects an intra-interpolation filter and an angle intra-prediction mode from the K combinations based on index information signaled in the bitstream.

[0019] In one example, the processing circuit calculates feature values ​​based on the adjacent reconstructed samples of the current block and selects an intra interpolation filter based on the calculated feature values.

[0020] In one example, the processing circuit calculates a feature value as one of (i) an absolute gradient value related to adjacent reconstruction samples of the current block, or (ii) a difference between a minimum value and a maximum value among the adjacent reconstruction samples of the current block.

[0021] In one example, the processing circuit obtains an output from a neural network, the input of the neural network includes adjacent reconstruction samples of the current block, and the output indicates an intra interpolation filter to select. The input of the neural network further includes an intra prediction mode index indicating an angular intra prediction mode.

[0022] Aspects of the present disclosure also provide a non - transient computer - readable medium storing instructions that, when executed by a computer, cause the computer to execute a method for video encoding / decoding.

Brief Description of the Drawings

[0023] Further features, properties, and various advantages of the matters related to the disclosure will become even more apparent from the following detailed description and the accompanying drawings. [Figure 1] It is a schematic diagram of an exemplary block diagram of a communication system (100). [Figure 2] It is a schematic diagram of an exemplary block diagram of a decoder. [Figure 3] It is a schematic diagram of an exemplary block diagram of an encoder. [Figure 4] It shows nine predictor directions from possible predictor directions according to one aspect of the present disclosure. [Figure 5] It shows an intra prediction direction according to one aspect of the present disclosure. [Figure 6] It shows eight nominal angles: V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED according to one aspect of the present disclosure. [Figure 7]An example of a non-directional intra predictor according to one aspect of the present disclosure is shown. [Figure 8] An example of a recursive intra filtering mode according to an aspect of the present disclosure is shown. [Figure 9] An example of multi-reference line (MRL) intra prediction is shown. [Figure 10] An exemplary angular intra prediction using reference line 0 adjacent to a block is shown. [Figure 11] An example of a current block coded using an angular intra prediction mode according to one aspect of the present disclosure and an adjacent reconstructed sample of the current block is shown. [Figure 12] A flowchart outlining a decoding process according to some aspects of the present disclosure is shown. [Figure 13] A flowchart outlining an encoding process according to some aspects of the present disclosure is shown. [Figure 14] A flowchart outlining a decoding process according to some aspects of the present disclosure is shown. [Figure 15] A schematic diagram of a computer system according to one aspect is shown.

Best Mode for Carrying Out the Invention

[0024] FIG. 1 shows a block diagram of a video processing system (100) in some examples. The video processing system (100) is an example of an application related to the matters disclosed, which is a video encoder and a video decoder in a streaming environment. The matters disclosed can be equally applied to other uses where video can be used, including, for example, video conferencing, digital TV, streaming services, and storage of compressed video on digital media such as CDs, DVDs, memory sticks, and the like.

[0025] The video processing system (100) may include a capture subsystem (113) which may include a video source (101), such as a digital camera, that produces, for example, a stream (102) of uncompressed video pictures. In one example, the stream (102) of video pictures includes a sample captured by the digital camera. The stream (102) of video pictures is drawn as a thick line to emphasize that it has a higher data volume compared to encoded video data (104) (or encoded video bitstream) and may be processed by electronic equipment (120) which includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement aspects of the matters disclosed in more detail later. The encoded video data (104) (or encoded video bitstream) is drawn as a thin line to emphasize that it has a lower data volume compared to the stream (102) of video pictures and may be stored in a streaming server (105) for later use. For example, one or more streaming client subsystems, such as client subsystems (106) and (108) in Figure 1, can access a streaming server (105) to retrieve copies (107) and (109) of encoded video data (104). The client subsystem (106) may include a video decoder (110), for example, in an electronic device (130). The video decoder (110) can decode the incoming copy of encoded video data (107) to produce an outgoing stream of video pictures (111), which may be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., video bitstreams) may be encoded according to a specific video coding / compression standard.Examples of these standards include ITU-T Recommendation H.265. For example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosures may be used in the context of VVC.

[0026] The electronic devices (120) and (130) may include other components (not shown). For example, the electronic device (120) may include a video decoder (not shown), and the electronic device (130) may also include a video encoder (not shown).

[0027] Figure 2 shows an exemplary block diagram of a video decoder (210). The video decoder (210) may be included in the electronic equipment (230). The electronic equipment (230) may include a receiver (231) (e.g., a receiving circuit). The video decoder (210) can be used in place of the video decoder (110) in the example of Figure 1.

[0028] The receiver (231) can receive one or more coded video sequences, for example, in a bitstream, which will be decoded by the video decoder (210). In one embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device that stores coded video data. The receiver (231) may receive coded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective user entities (not shown). The receiver (231) may isolate the coded video sequences from other data. To combat network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser 520 (hereinafter, “Parser (220)”). In certain applications, the buffer memory (215) is part of the video decoder (210). In other cases, it may be located outside the video decoder (210) (not shown). In yet other cases, for example to counter network jitter, a buffer memory (not shown) may exist outside the video decoder (210), and further, for example to handle playback timing, another buffer memory (215) may exist inside the video decoder (210). When the receiver (231) is receiving data from a storage / transfer device with sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory (215) may not be necessary or may be made small. For example, in use on a best-effort packet network such as the Internet, the buffer memory (215) may be necessary and may be made relatively large and, advantageously, of an adaptable size, and may be implemented at least in part in an operating system or similar element (not shown) outside the video decoder (210).

[0029] The video decoder (210) may include a parser (220) for reconstructing symbols (221) from a coded video sequence. The categories of these symbols may include information used to manage the operation of the video decoder (210), and may also include information for controlling rendering devices, such as a renderer (212) (e.g., a display screen), which are not part of the electronic device (230) but can be coupled to the electronic device (230), as shown in Figure 2. Control information for (one or more) rendering devices may take the form of Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (220) may parse / entropically decode the received coded video sequence. The coding of the coded video sequence may be by video coding techniques or standards and may follow a variety of principles, including variable-length coding, Huffman coding, and context-dependent or non-context-dependent arithmetic coding. The parser(220) can extract from the coded video sequence a set of subgroup parameters relating to at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to a group. Subgroups may include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), and predictive units (PU). The parser(220) can also extract information from the coded video sequence information, such as transform coefficients, quantization parameter values, and motion vectors.

[0030] The parser (220) may perform entropy decoding / parsing on the video sequence received from buffer memory (215) to produce symbols (221).

[0031] The reconstruction of symbol (221) may involve multiple different units, depending on the type of coded video picture or part thereof and other factors (e.g., interpicture and intrapicture, interblock and intrablock, etc.). Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (220). The flow of such subgroup control information between the parser (220) and the following multiple units is not illustrated for clarity.

[0032] Beyond the functional blocks described above, the video decoder (210) can be conceptually subdivided into numerous functional units, as described below. In a practical implementation operating under commercial constraints, many of these units may interact closely with each other and be at least partially integrated. However, for the purpose of illustrating the matters to be disclosed, the following conceptual subdivision into functional units is appropriate.

[0033] The first unit is the scaler / inverse unit (251). The scaler / inverse unit (251) receives quantized transformation coefficients as symbols (221) (one or more) from the parser (220), along with control information including block size, quantization coefficients, and quantization scaling matrix, indicating which transformation should be used. The scaler / inverse unit (251) can output a block containing sample values ​​that can be input to the aggregator (255).

[0034] In some cases, the output samples of the scaler / inverse unit (251) may relate to intracoded blocks. Intracoded blocks are blocks that do not use prediction information from previously reconstructed pictures, but can use prediction information from previously reconstructed portions of the current picture. Such prediction information can be provided by the intrapicture prediction unit (252). In some cases, the intrapicture prediction unit (252) generates a block of the same size and shape as the block being reconstructed, using surrounding already reconstructed information fetched from the current picture buffer (258). The current picture buffer (258) buffers, for example, partially reconstructed current pictures and / or fully reconstructed current pictures. In some cases, the aggregator (255) adds the prediction information generated by the intraprediction unit (252) to the output sample information provided by the scaler / inverse unit (251) on a sample-by-sample basis.

[0035] In other cases, the output samples of the scaler / inverse unit (251) may relate to an intercoded, potentially motion-compensated block. In such cases, a motion-compensated prediction unit (253) can access a reference picture memory (257) to fetch samples to be used for prediction. After the fetched samples are motion-compensated according to symbols (221) related to the block, these samples can be appended by an aggregator (255) to the output of the scaler / inverse unit (251) (in this case, called residual samples or residual signals) to generate output sample information. From there, the addresses in the reference picture memory (257) from which the motion-compensated prediction unit (253) fetches prediction samples can be controlled by a motion vector, which may be available to the motion-compensated prediction unit (253) in the form of symbols (221) having, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from reference picture memory (257) when the precise motion vector of a subsample is used, or motion vector prediction mechanisms.

[0036] The output samples from the aggregator (255) can be subjected to various loop filtering techniques in the loop filter unit (256). The video compression technique may include in-loop filtering, which is controlled by parameters included in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (256) as symbols (221) from the parser (220). The video compression can also respond to metadata obtained during the decoding of preceding portions (in decoding order) of the coded picture or coded video sequence, as well as to previously reconstructed and loop-filtered sample values.

[0037] The output of the loop filter unit (256) can be a sample stream that can be output to the renderer (212), which can also be stored in reference picture memory (257) for use in future interpicture prediction.

[0038] A particular coded picture, once fully reconstructed, can be used as a reference picture for future predictions. For example, when a coded picture corresponding to the current picture is fully reconstructed and that coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) may become part of the reference picture memory (257), and a new current picture buffer may be reallocated before the reconstruction of the next coded picture begins.

[0039] The video decoder (210) may perform decoding according to a specified video compression technique or standard, such as ITU-T Recommendation H.265. The coded video sequence may conform to the syntax defined by the video compression technique or standard used, in the sense that it faithfully adheres to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, a profile may select a particular tool from all the tools available in the video compression technique or standard, such that only that tool is available for use under that profile. Also, for compliance, the complexity of the coded video sequence must be within the range defined by the level of the video compression technique or standard. Depending on the case, the level may constrain the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limitations set by the level may, depending on the case, be further constrained through the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0040] In one embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of one or more coded video sequences. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, a temporal, spatial, or signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, or forward error correction codes.

[0041] Figure 3 shows an exemplary block diagram of a video encoder (303). The video encoder (303) is contained within the electronic device (320). For example, the electronic device (320) contains a transmitter (340) (e.g., a transmitting circuit). The video encoder (303) can be used in place of the video encoder (103) in the example of Figure 1.

[0042] The video encoder (303) may receive video samples from a video source (301) (not part of the electronic device (320) in the example of Figure 6) which can capture (one or more) video images that will be coded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0043] The video source (301) may provide a source video sequence coded by a video encoder (303) in the form of a digital video sample stream, which can have any preferred bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any preferred sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service delivery system, the video source (301) may be a storage device storing pre-prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a series of individual pictures that convey motion when viewed sequentially. These pictures themselves can be organized as a spatial array of pixels, and each pixel may have one or more samples depending on the sampling structure, color space, etc., used. The following description focuses on samples.

[0044] In one embodiment, the video encoder (303) can encode and compress pictures of a source video sequence into a coded video sequence (343) in real time or under other required time constraints. One function of the controller (350) is to enforce an appropriate coding speed. In some embodiments, the controller (350) controls and is functionally coupled to other functional units, such as those described later. The coupling is not illustrated for clarity. Parameters set by the controller (350) may include rate control-related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) can be configured to have other preferred functions related to the video encoder (303) that are optimized for a particular system design.

[0045] In some embodiments, the video encoder (303) is configured to operate in a coding loop. For an oversimplified explanation, in one example, the coding loop may include a source coder (330) (responsible for creating symbols, such as a symbol stream, based on, for example, the input picture to be encoded and one or more reference pictures) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to generate sample data, similar to how a (remote) decoder would create them. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Since decoding the symbol stream yields bit-accurate results independent of the decoder location (local or remote), the contents of the reference picture memory (334) are also bit-accurate between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder "sees" when using predictions during decoding. The fundamental principle of this reference picture synchronization (and the resulting drift when synchronization cannot be maintained, for example, due to channel errors) is also used in some related technologies.

[0046] The operation of the “local” decoder (333) can be the same as that of a “remote” decoder, such as a video decoder (210), which has already been described in detail above in relation to Figure 2. However, also briefly referring to Figure 2, since symbols are available and the encoding / decoding of symbols to a coded video sequence by the entropy coder (345) and parser (220) can be reversible, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), does not need to be fully implemented in the local decoder (333).

[0047] In one embodiment, the decoder technology, excluding parsing / entropy decoding present within the decoder, exists in the corresponding encoder in the same or substantially the same functional form. Therefore, the disclosure focuses on the decoder operation. A description of the encoder technology can be omitted, as it is the inverse of the thoroughly described decoder technology. More detailed explanations are provided below for certain specific areas.

[0048] During operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes the input picture against one or more previously coded pictures from a video sequence designated as “reference pictures”. Thus, the coding engine (332) codes the difference between the pixel blocks of the input picture and the pixel blocks of one or more reference pictures that may be selected as prediction criteria for the input picture.

[0049] The local video decoder (333) can decode the coded video data of a picture that may be designated as a reference picture based on symbols created by the source coder (330). The operation of the coding engine (332) can, advantageously, be a lossy process. When the coded video data can be decoded by a video decoder (not shown in Figure 3), the reconstructed video sequence may typically be a replica of the source video sequence with some error. The local video decoder (333) can replicate the decoding process that may be performed by the video decoder on the reference picture and cause the reconstructed reference picture to be stored in the reference picture memory (334). Thus, the video encoder (303) can locally store a copy of the reconstructed reference picture that has content common to the reconstructed reference picture that will be obtained by the far-end video decoder.

[0050] The predictor (335) may perform a predictive search for the coding engine (332). That is, with respect to a new picture to be coded, the predictor (336) may search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors or block shapes that can serve as appropriate predictive criteria for the new picture. The predictor (335) may operate pixel block by pixel to find appropriate predictive criteria. Depending on the case, the input picture may have predictive criteria drawn from multiple reference pictures stored in the reference picture memory (334), as determined by the search results obtained by the predictor (335).

[0051] The controller (350) may manage the coding process of the source coder (330), including, for example, setting parameters and subgroup parameters used to encode video data.

[0052] The outputs of all the aforementioned functional units can be subjected to entropy coding in the entropy coder (345). The entropy coder (345) converts the symbols generated by the various functional units into coded video sequences by applying lossless compression to the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0053] A transmitter (340) may buffer (one or more) coded video sequences generated by the entropy coder (345) and prepare them for transmission over a communication channel (360). The communication channel (360) may be a hardware / software link to a storage device that stores the coded video data. The transmitter (340) may merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0054] The controller (350) may manage the operation of the video encoder (303). In coding, the controller (350) may assign each coded picture to a specific coded picture type that may influence the coding technique that may be applied to that picture. For example, a picture may often be assigned one of the following picture types:

[0055] An intra-picture (I-picture) can be coded and decoded without using any other picture in the sequence as a source for prediction. Some video codecs allow several different types of intra-pictures, including, for example, Independent Decoder Refresh (IDR) pictures.

[0056] Predictive pictures (P-pictures) can be coded and decoded using intra-prediction or inter-prediction, with motion vectors and reference indices used to predict the sample values ​​of each block.

[0057] A bidirectional predictive picture (B-picture) can be coded and decoded using intra-prediction or inter-prediction, employing two motion vectors and a reference index to predict the sample values ​​for each block. Similarly, a multiple predictive picture can use three or more reference pictures and associated metadata for the reconstruction of a single block.

[0058] A source picture can generally be subdivided spatially into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block can be coded. Blocks can be coded predictively by referring to other (already coded) blocks, determined by the coding assignment applied to each picture in those blocks. For example, blocks of picture I can be coded unpredictably, or they can be coded predictively by referring to already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P can be coded unpredictably, or via spatial or temporal prediction by referring to one previously coded reference picture. Blocks of picture B can be coded unpredictably, or via spatial or temporal prediction by referring to one or two previously coded reference pictures.

[0059] The video encoder (303) may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Recommendation H.265. In its operation, the video encoder (303) may perform various compression operations, including predictive coding operations that take advantage of temporal and spatial redundancy in the input video sequence. The coded video data may therefore conform to the syntax defined by the video coding technique or standard being used.

[0060] In one embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and the like.

[0061] Video can be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes (temporal or other) correlations between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into multiple blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, that block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0062] In some embodiments, a dual prediction technique can be used in interpicture prediction. According to the dual prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture, both of which are earlier in the decoding order than the current picture in the image (but may be past and future in the display order, respectively). A block in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. That block can be predicted by a combination of the first and second reference blocks.

[0063] Furthermore, merge mode techniques can be used to improve coding efficiency in interpicture prediction.

[0064] According to certain aspects of this disclosure, predictions such as interpicture prediction and intrapicture prediction are performed in units of blocks. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into multiple coding tree units (CTUs) for compression, and these CTUs in a picture have the same size, for example, 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU contains three coding tree blocks (CTBs), which are one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree-divided into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, or four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of that CU, for example, inter-prediction type or intra-prediction type. A CU is divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Generally, each PU includes a Luma prediction block (PB) and two Chroma PBs. In one embodiment, the prediction operation during coding (encoding / decoding) is performed in units of prediction blocks. Using a Luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., Luma values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, and similar.

[0065] The video encoders (103) and (303), and the video decoders (110) and (210) may be implemented using any preferred technology. In one embodiment, the video encoders (103) and (303), and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303), and the video decoders (110) and (210) may be implemented using one or more processors that execute software instructions.

[0066] Video codec technology may include intracoding. In intracoding, sample values ​​can be represented without referencing samples or other data from a previously reconstructed reference picture. In some video codecs, a picture may be spatially divided into multiple blocks of samples. If all block samples are coded in intra mode, the picture can be an intrapicture (e.g., an I-picture).

[0067] In some embodiments, predictions may be performed based on surrounding sample data and / or metadata acquired during the encoding and / or decoding of blocks of data. Such techniques are called “intra-prediction” techniques. In one embodiment, intra-prediction (e.g., intra-picture prediction) may use only reference data from the current picture being reconstructed, rather than from a reference picture.

[0068] In intra-prediction, predictor blocks can be formed using adjacent sample values ​​of already available samples. The sample values ​​of adjacent samples can be copied to the predictor block according to a certain direction. A reference to the direction to be used can be coded within the bitstream, or it can be predicted itself.

[0069] For example, various intra-predictive coding tools can be used, such as angular intra-predictive prediction as shown in Figures 4-6 and 10, multiple reference line (MRL) prediction as shown in Figure 9, and / or similar methods. Non-directional intra-predictive modes may also be used.

[0070] Figure 4 shows nine predictor directions from possible predictor directions according to one embodiment (for example, 33 predictor directions corresponding to 33 angular modes out of 35 intra-modes as defined in H.265). Point (401) represents the predicted sample. The arrows indicate the directions from which sample (401) can be predicted. For example, arrow (402) indicates that sample (401) is predicted from one or more samples to the upper right, at an angle of 45° from the horizontal (e.g., the X dimension). Arrow (403) indicates that sample (401) is predicted from one or more samples to the lower left of sample (401), at an angle of 22.5° from the horizontal.

[0071] Figure 4 also shows a block (e.g., a 4x4 sample square block indicated by a thick dashed line) (404). A block (404) can contain 16 samples, each labeled with “S” and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Figure 4 also shows reference samples using a similar numbering scheme. Reference samples are labeled with R and their Y position (e.g., row index) and X position (column index) relative to the block (404). In some embodiments (e.g., H.264 and H.265), predicted samples may be adjacent to the block being reconstructed. In some embodiments, multiple reference lines may be used, as shown in Figure 9.

[0072] Intra-prediction can function by copying a reference sample value from an adjacent sample indicated by the prediction direction (e.g., the signaled prediction direction). For example, if the prediction direction indicated by arrow (402) is signaled in a coded video bitstream, a sample is predicted from the upper right sample at a 45° angle from the horizontal. For example, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Sample S44 can be predicted from reference sample R08.

[0073] In certain cases, as illustrated in Figure 10, for example, the values ​​of multiple reference samples may be combined, for example, through interpolation (e.g., an intra-interpolation filter), in order to calculate a reference sample. For example, when the direction is not divisible by 45°, the values ​​of multiple reference samples are combined through interpolation.

[0074] Any suitable number of possible directions can be used in the intra-prediction. For example, in the example in Figure 4 (in H.264), nine different directions are used. In an example like the one in H.265, 33 different directions are used.

[0075] For example, in cases such as finer-grained angle prediction in VVC, 65 different directions are used. The 65 angle prediction directions can be used for a given block size, and the set of angles can depend on the block size. In the case of a square block, in one example, the 65 angle prediction directions are defined in a clockwise direction from 45° to -135° for a coding block of square shape. Figure 5 shows a schematic diagram (510) showing 65 intra-prediction directions according to one aspect of the present disclosure (e.g., in JEM). In addition to these 65 intra-prediction directions (corresponding to 65 intra-prediction modes 2-66), the intra-prediction modes may include a planar mode ("0") and a DC mode ("1").

[0076] For example, in some cases such as in VVC, wide-angle intra prediction (WAIP) may be used. In WAIP, for non-square blocks, the 14 angles that use prediction from the shorter side of the block can be replaced with more extreme angles that use prediction from the longer side, increasing the total number of angles supported by WAIP to 93, while the number of angle modes that can be signaled for a given block size remains at 65.

[0077] In an example like VP9, ​​eight orientation modes are supported, corresponding to angles from 45° to 207°. Figure 6 shows an example of orientation intra-prediction, as used in Alliance for Open Media (AOMedia) Video 1 (AV1). To take advantage of greater spatial redundancy in directional textures, for example in AV1, the orientation intra-modes can be extended to a finer-grained set of angles. The original eight angles can be slightly modified and become nominal angles. Figure 6 shows the eight nominal angles: V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED. Each nominal angle can have seven finer angles, so that, for example in AV1, 56 orientation angles are used. The predicted angle can be presented by adding an angle delta, which is a step size of 3° multiplied by -3 to 3, to the nominal intra-angle (e.g., V_PRED). In one embodiment, to implement the direction prediction mode in a general manner in AV1, the direction intra-prediction modes in AV1 (e.g., all 56 direction intra-prediction modes) are implemented using a unified direction predictor, as shown in Figure 10, which projects each pixel to a reference subpixel position and interpolates the reference pixel using an intra-interpolation filter (e.g., a 2-tap bilinear filter).

[0078] Figure 7 shows an example of a non-directed intra predictor used in AV1, according to one aspect of this disclosure. Figure 7 shows the positions of the top, left, and top-left samples for a single pixel (701) within the current block (710) to be predicted. In an example like AV1, five non-directed smoothing intra predictor modes are used, including DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H. In the DC predictor, the average of the left and top adjacent samples may be used as the predictor for pixel (701) within the current block (710). In the PAETH predictor, the top, left, and top-left reference samples are first fetched, and then the value closest to (top + left - top-left) may be set as the predictor for pixel (701). The SMOOTH, SMOOTH_V, and SMOOTH_H modes can predict the block (710) using quadratic interpolation in the vertical or horizontal direction, or by averaging in both directions.

[0079] Figure 8 illustrates an intra predictor based on recursive filtering. Figure 8 shows an example of a recursive intra filtering mode according to an aspect of this disclosure. The FILTER INTRA mode may be used for a block (e.g., a Luma block) to capture decaying spatial correlations with references on edges. In AV1, for example, five filter intra modes may be used. Each may be represented by a set of eight 7-tap filters that reflect the correlation between a pixel in a 4x2 patch and seven neighbors adjacent to that 4x2 patch. The weighting coefficients of the 7-tap filters may be position-dependent. Figure 8 shows an example of an 8x8 block (810). The block (810) may be divided into eight 4x2 patches B0, B1, B2, B3, B4, B5, B6, and B7. Seven neighbors of each patch (e.g., indicated by R0-R7) may be used to predict the pixels currently in the patch. In patch B0, all neighbors (e.g., those represented by R0-R7 in patch B0) have already been reconstructed. In other patches, not all neighbors have been reconstructed, in which case the predicted values ​​from the immediately adjacent patch may be used as a reference. For example, the neighbors of patch B7 (e.g., all neighbors) have not been reconstructed, and predicted samples from the adjacent patches (e.g., patches B5 and B6) are used.

[0080] MRL intraprediction can use multiple reference lines for intraprediction. Figure 9 shows an example of MRL intraprediction. Four reference lines 0-3 of the current block (901) are shown in Figure 9. Let i be 0, 1, 2, or 3, then reference line i can include reference samples that are i lines away from the current block (901), for example, i lines away from the boundary of the current block (901) (e.g., i rows away from the top boundary and / or i columns away from the left boundary). For example, reference line i can include reference samples that are i rows above the top boundary of the current block (901) and / or i columns to the left of the left boundary of the current block (901). In one example, reference line 0 can include reference samples adjacent to the current block (901), such as a reconstructed adjacent sample that includes an upper adjacent sample above the current block (901) and a left adjacent sample to the left of the block (901). In one example, reference line 0 can include a reconstructed adjacent sample that is above the upper left.

[0081] Reference lines 0-3 may include multiple segments, such as segments AF. In one example, samples from segments A and F are not fetched from the reconstructed adjacent samples. Samples from segments A and F may be padded (or filled) with the nearest samples from segments B and E, respectively.

[0082] In an example like HEVC, the nearest reference line (i.e., reference line 0) is used for intra-prediction (or intra-picture prediction). In one example, a reference line other than the nearest reference line (e.g., reference line 1) is used for intra-prediction. In one example, multiple reference lines may be used for MRL intra-prediction.

[0083] MRL intra-prediction can be extended to include more reference lines for intra-prediction, for example, in Enhanced Compression Model 5 (ECM5).

[0084] In angle intra prediction, the current sample within the current block can be predicted, for example, using a reference sample (e.g., a predicted sample) within a reference line or an interpolated reference sample.

[0085] Figure 10 shows an example of angular intraprediction using reference line 0 adjacent to block (1000). This explanation can also be suitably applied to using a reference line not adjacent to block (1000) (e.g., reference line 1). A portion of reference line 0 is shown. A sample (1001) within block (1000) can be predicted using one or more samples of reference line 0 in the intrapredictor direction (or angular direction) (1022). Sample (1001) can be projected onto reference line 0 along the angular direction (1022).

[0086] In the example shown in Figure 10, the projected position (1013) of sample (1001) lies between two reference samples (1011)-(1012) on reference line 0 and is referred to as a projected fractional position or fractional sample position. Reference samples around the projected position (1013) may be used to predict sample (1001) by, for example, using an interpolation filter (e.g., an intra-interpolation filter). In one embodiment, the interpolation filter is applied when the projected position (1013) of sample (1001) is a fractional sample position between two adjacent reference samples. The intra-interpolation filter can be any suitable filter, such as a bilinear filter, a cubic filter, a spline interpolation filter, a DCT-based interpolation filter, a DST-based interpolation filter, or similar. In one embodiment, a sample intra-predicted using an angular intra-prediction mode (e.g., sample (1001)) is projected onto a fractional position (e.g., projected position (1013)) between two adjacent reference samples (e.g., (1011)-(1012)), and the filter used to generate the predicted sample value for that sample is referred to as an intra-interpolation filter. In one example, the predicted sample value is based on the interpolation of the sample values ​​of two adjacent reference samples (e.g., (1011)-(1012)), for example, the predicted sample value is a weighted average of the sample values ​​of the two adjacent reference samples using weights corresponding to the filter coefficients of the intra-prediction filter. In one example, these weights are the respective filter coefficients of the intra-prediction filter. In one example, the predicted sample value is based on the interpolation of the sample values ​​of two adjacent reference samples (e.g., (1011)-(1012)) and an additional reference sample.

[0087] Using two reference samples (1011)-(1012) on reference line 0, sample (1001) can be predicted using a two-tap intra interpolation filter (e.g., a bilinear filter). In one example, the filter coefficients in the two-tap intra interpolation filter can each be based on two distances between the projected position (1013) and two adjacent integer positions (indicated by black dots) of the two reference samples (1011)-(1012). For example, if the distance between the projected fractional position (1013) and reference sample (1011) is smaller than the distance between the projected fractional position (1013) and reference sample (1012), then the filter coefficient for reference sample (1011) is larger than the filter coefficient for reference sample (1012).

[0088] In one example, using the four reference samples (1011), (1012), (1014), and (1015) of reference line 0, sample (1001) can be predicted using a 4-tap intra interpolation filter (e.g., a 4-tap linear interpolation filter), such as a DCT-based interpolation filter (DCTIF).

[0089] An intra-interpolation filter can vary depending on the filter type and / or the number of filter coefficients (also called filter taps or taps). Filter types can include different types of intra-interpolation filters, such as bilinear filters, cubic filters, spline filters, DCT-based interpolation filters, DST-based interpolation filters, and similar types. For example, filter types can include intra-interpolation filters associated in different directions, such as horizontal filters (applied to samples in rows, as shown in Figure 10) and vertical filters (applied to samples in columns, for example). Referring to Figure 10, a two-tap intra-interpolation filter applied to two reference samples (1011)-(1012) can be a horizontal filter.

[0090] Intra interpolation filters can differ by the number of filter taps; for example, different intra interpolation filters may include intra interpolation filters with different taps, such as 2-tap, 4-tap, 6-tap, and 8-tap filters.

[0091] The intra-interpolation filter may differ depending on the filter type and filter tap.

[0092] An intra-interpolation filter, known in related technologies, is a fixed interpolation filter for a block (e.g., each block). In one example, an intra-interpolation filter applied to a block is considered fixed when the filter type and number of taps of the intra-interpolation filter are fixed. When the filter type and number of taps of the intra-interpolation filter are fixed, the filter coefficients of the intra-interpolation filter may vary based on the samples to be predicted within the block and the respective intra-prediction modes used to predict those samples.

[0093] The optimal intra interpolation filter may depend on the statistics of samples within an image or video, such as within a picture or frame, and therefore, using a fixed interpolation filter (e.g., a fixed intra interpolation filter) may be suboptimal.

[0094] The current block in the current picture can be coded intra-predictively using the directional intra-predictive mode. The directional intra-predictive mode may also be referred to as the angular intra-predictive mode, angular mode, directional mode, or directional intra-mode. The directional intra-predictive mode or angular intra-predictive mode is an intra-predictive mode that can predict samples in the current block along the prediction direction associated with the angular intra-predictive mode, as described in Figures 4 and 10.

[0095] According to one aspect of this disclosure, an intra-interpolation filter used in an angle intra-prediction mode (e.g., as described in Figure 10) may be determined (e.g., derived or selected) based on adjacent reconstructed samples (also referred to as adjacent reconstructed samples) in the current block. Determining the intra-interpolation filter may include determining one or more properties of the intra-interpolation filter. For example, determining the intra-interpolation filter may include determining the type of intra-interpolation filter and / or the number of taps used in the intra-interpolation filter. The determined intra-interpolation filter may depend on adjacent reconstructed samples in the current block and may therefore be considered self-guided, for example, the determination of the intra-interpolation filter may be guided by adjacent reconstructed samples in the current block. The determined intra-interpolation filter may be referred to as a self-guided intra-interpolation filter and may be applied to predict samples in the current block.

[0096] In one embodiment, an intra-interpolation filter is determined for the current block based on the adjacent reconstruction samples of the current block and the angular intra-prediction mode used to predict the current block. In one example, a first portion of the adjacent reconstruction sample (e.g., reference line 0 in Figure 11) is predicted using an angular intra-prediction mode from a second portion of the adjacent reconstruction sample (e.g., reference line 1 in Figure 11) using a predetermined set of intra-interpolation filters, thereby generating a predicted first portion associated with the predetermined set of intra-interpolation filters. Based on a comparison of the predicted first portion with the first portion, one of the predetermined set of intra-interpolation filters may be selected. For example, a prediction error is generated based on the predicted first portion with the first portion, and one of the predetermined set of intra-interpolation filters may be selected as the intra-interpolation filter associated with the smallest prediction error. In one example, the predetermined set of intra-interpolation filters includes a first filter and a second filter. The predicted first portion includes a predicted first portion I predicted using the first filter and a predicted first portion II predicted using the second filter. The prediction error includes a first prediction error between prediction part I and part 1, and a second prediction error between prediction part II and part 2. If the first prediction error is the smallest prediction error (e.g., the first prediction error is smaller than the second prediction error), then the first filter may be selected as the intra-prediction filter for the current block. The above description may also suitably apply when multiple parts (e.g., reference lines 0-3) are used to determine the intra-prediction filter (instead of just the first part described above).

[0097] In one embodiment, the intra interpolation filter and angular intra prediction mode used to predict the current block are determined for the current block based on the adjacent reconstruction samples of the current block.

[0098] Using self-guided intra-interpolation filters can improve coding efficiency and is therefore more efficient than using fixed interpolation filters in the relevant techniques. In one example, a fixed interpolation filter is predetermined and independent of adjacent reconstructed samples in the current block. In one example, the current block is a luma block. In another example, the current block is a chroma block. The current block can be predicted using MRL intra-prediction, which predicts samples in the current block using a reference line other than reference line 0 (e.g., reference line 1 in Figure 9).

[0099] In this disclosure, the term “block” may be interpreted as a prediction block, coding block, or coding unit (CU), etc. The term “current block” may be interpreted as a prediction block being coded (e.g., being reconfigured), a coding block being coded (e.g., being reconfigured), or a coding unit (CU) being coded (e.g., being reconfigured), etc. The methods described in this disclosure may be applicable to several different video coding standards, including, but are not limited to, AV1, AOMedia Video Model (AVM), AOMedia Video 2 (AV2), Versatile Video Coding (VVC), Enhanced Compression Model (ECM), and / or H.267, etc.

[0100] The intra interpolation filter used in directional intra prediction mode (or angular intra prediction mode) may be derived or selected based on adjacent reconstruction samples of the current block. For example, the current block in a picture is coded in angular intra prediction mode using an intra interpolation filter. The intra interpolation filter used in angular intra prediction mode may be determined (e.g., derived or selected) from a predetermined set of intra interpolation filters based on adjacent samples of the current block. Adjacent samples may include, for example, adjacent reconstruction samples in N lines (e.g., N adjacent lines, N>1) from one or more boundaries of the current block. The determined intra interpolation filter (e.g., selected or derived intra interpolation filter) may be applied to reference samples in reference lines in a picture to predict samples in the current block using angular intra prediction mode.

[0101] In one embodiment, the value of N depends on the block size, where N represents the number of lines and / or columns in a set of reference lines. In one example, the number of lines and / or columns used depends on the block size. Whether adjacent reconstruction samples are used to derive interpolation filters (e.g., intra-interpolation filters) may depend on the block area. For example, when the block area is smaller than a threshold, adjacent pixels (e.g., adjacent reconstruction samples) are used to derive the cost (e.g., prediction error) of each filter (e.g., each intra-interpolation filter in a given set of intra-interpolation filters) in order to reduce (e.g., decrease) signaling overhead.

[0102] Figure 11 shows an example of a current block (1101) coded using an angle-intra prediction mode according to one aspect of the present disclosure, and an adjacent reconstruction sample (1110) of the current block (1101). In this example, the current block (1101) is being reconstructed. The prediction direction of the angle-intra prediction mode may be referred to as the intra-prediction direction. In the example shown in Figure 11, arrows (1121)-(1123) indicate the same direction, for example, the prediction direction of the angle-intra prediction mode. Arrows (1124)-(1126) may indicate the same direction, for example, the opposite direction to the prediction direction of the angle-intra prediction mode. The opposite direction (e.g., downward, indicated by arrows (1124)-(1126)) may be the exact opposite of the prediction direction of the angle-intra prediction mode (e.g., upward, indicated by arrows (1121)-(1123)).

[0103] An adjacent reconstructed sample (1110) of the current block (1101) may be referred to as an adjacent reconstructed sample of the current block (1101). An adjacent reconstructed sample (1110) of the current block (1101) may include samples in multiple reference lines (e.g., two or more reference lines). In the example shown in Figure 11, the adjacent reconstructed sample (1110) of the current block (1101) includes an adjacent reconstructed sample in reference line 0-(N-1) (where N is 4). The method described with reference to Figure 11 may be applied to any suitable N greater than 1. As shown in Figure 11, with i being 0, 1, 2, or 3, reference line i may include reference samples that are i lines away from the current block (1101), for example, i lines away from the boundary of the current block (1101) (e.g., i rows away from the top boundary and / or i columns away from the left boundary). For example, reference line i includes a reference sample i rows above the top boundary of the current block (1101) and / or i columns to the left of the left boundary of the current block (1101). In one example, reference line 0 includes reference samples adjacent to the current block (1101), such as a reconstructed adjacent sample that includes an upper adjacent sample above the current block (1101) and a left adjacent sample to the left of the block (1101).

[0104] In one example, the adjacent reconstruction sample (1110) of the current block (1101) includes an upper reference line (1111), which is a reference line above the upper boundary of the current block (1101). In one example, the adjacent reconstruction sample (1110) of the current block (1101) includes a left reference line (1112), which is a reference line to the left of the left boundary of the current block (1101). In one example, the adjacent reconstruction sample (1110) of the current block (1101) includes both the upper reference line (1111) and the left reference line (1112).

[0105] In one embodiment, N is greater than 1, and the angle intra-prediction mode has already been determined (e.g., associated with the prediction direction indicated by arrows (1121)-(1123)). The adjacent reconstruction sample may include adjacent reconstruction samples of multiple lines (e.g., reference lines 0-3 in Figure 11).

[0106] Each intra interpolation filter from a predetermined set of intra interpolation filters can be applied to the adjacent reconstruction sample of each line in a set of adjacent reconstruction samples of multiple lines, and the adjacent reconstruction sample of each line can be predicted using the angular intra prediction mode, and the prediction error can be obtained. In one example, for each combination of an intra interpolation filter from a predetermined set of intra interpolation filters and an adjacent reconstruction sample of a line in a set of adjacent reconstruction samples of multiple lines, the adjacent reconstruction sample of each line can be predicted based on one or more remaining lines in the set of adjacent reconstruction samples, and at least one prediction error can be obtained.

[0107] In one example, referring to Figure 11, N is 4, the adjacent reconstruction samples of multiple lines include reference lines 0-3, one of a given set of intra interpolation filters is a bilinear filter, and the adjacent reconstruction sample of one of the lines in the adjacent reconstruction samples of multiple lines is reference line 0. The one or more remaining lines in the adjacent reconstruction samples of multiple lines include reference lines 1-3. The adjacent reconstruction sample of the above line (e.g., reference line 0) can be predicted based on reference lines 1-3, and at least one prediction error includes three prediction errors, each associated with reference lines 1-3. For example, the first prediction error is based on the reconstruction sample in reference line 0 and the prediction sample of reference line 0 based on the angle intra prediction mode associated with the prediction direction indicated by the arrow (1121) and reference line 1. Similarly, reference lines 1-3 can be predicted, and the corresponding prediction errors are obtained for the bilinear filter. For example, reference line 1 is predicted based on the angle intra-prediction modes associated with the prediction directions indicated by arrows (1122) and (1124) for each of the reference lines 0, 2, and 3, respectively, and three prediction errors are obtained.

[0108] The above process for one of a predetermined set of intra interpolation filters (e.g., a bilinear filter) can be performed or repeated for other intra interpolation filters in the predetermined set (e.g., a spline filter and / or a cubic filter). An intra interpolation filter corresponding to the smallest prediction error among the acquired prediction errors can be selected. For example, if the smallest prediction error is the prediction error acquired using a particular intra interpolation filter (e.g., a bilinear filter), then that particular intra interpolation filter is selected as the intra interpolation filter to be applied to the current block (1101) for its angular intra prediction mode (e.g., associated with the prediction direction indicated by arrows (1121)-(1123)).

[0109] In one embodiment, given an angle intra-prediction mode (also referred to as a direction intra-prediction mode) and adjacent reconstruction samples of multiple lines (e.g., adjacent reconstruction samples in reference lines 0-3 in Figure 11), a predetermined set of intra-interpolation filters are applied to the reference lines (e.g., reference lines 0-3 in Figure 11) to predict each other along the intra-prediction direction (e.g., upward, indicated by arrows (1121)-(1123)) and / or the opposite direction (e.g., downward, indicated by arrows (1124)-(1126)), and the intra-interpolation filter candidate (also referred to as a candidate interpolation filter) having the smallest prediction error is selected as the intra-interpolation filter for the current block (1101). Referring to Figure 11, reference line 0 is predicted by other reference lines such as reference lines 1, 2, 3, ... and (N-1) (indicated by arrow (1121)). Reference line 1 can be predicted by other reference lines such as reference lines 0, 2, 3, ..., and (N-1) (indicated by arrows (1122) and (1124)). Reference line 2 can be predicted by other reference lines such as reference lines 0, 1, 3, ..., and (N-1) (indicated by arrows (1123) and (1125)). Reference line 3 can be predicted by other reference lines such as reference lines 0, 1, 2, ..., and (N-1) (indicated by arrow (1126)). Each of the candidate interpolation filters is applied to predict the reference lines between them as described above, and the one with the smallest prediction error is used as the prediction filter (e.g., the determined intra interpolation filter).

[0110] In one embodiment, based on the adjacent reconstruction sample of the current block, one of (i) the type of intra interpolation filter or (ii) the number of taps in the intra interpolation filter is selected from a predetermined set of intra interpolation filters.

[0111] In one example, a predetermined set of intra interpolation filters includes different types of intra interpolation filters, such as one of bilinear interpolation filters, cubic interpolation filters, spline interpolation filters, DCT-based interpolation filters, or DST-based interpolation filters. The type of intra interpolation filter can be selected from different types of intra interpolation filters based on the adjacent reconstruction samples of the current block. The selected intra interpolation filter can be one of bilinear interpolation filters, cubic interpolation filters, spline interpolation filters, DCT-based interpolation filters, or DST-based interpolation filters.

[0112] For example, a predetermined set of intra interpolation filters (e.g., candidate intra interpolation filters) may include, but are not limited to, bilinear filters, cubic filters, spline interpolation filters, and DCT / DST-based interpolation filters (e.g., DCT or DST-based interpolation filters).

[0113] In one example, a predetermined set of intra interpolation filters may include one or more filters with different tap counts, such as a 2-tap interpolation filter, a 4-tap interpolation filter, a 6-tap interpolation filter, or an 8-tap interpolation filter. The number of taps in an intra interpolation filter can be selected based on the adjacent reconstruction samples of the current block. The number of taps in an intra interpolation filter can be one of 2, 4, 6, or 8 taps.

[0114] In one example, a given set of intra-interpolation filters (e.g., candidate intra-interpolation filters) may have different numbers of filter taps. For example, candidate intra-interpolation filters may be groups of 2-tap, 4-tap, 6-tap, and 8-tap filters.

[0115] In one example, whether a set of adjacent reconstruction samples of multiple lines (e.g., reference lines 0-(N-1)) includes (i) the left reference line (1112) (e.g., the adjacent reconstruction sample of the leftmost line to the left of the current block (1101)), (ii) the upper reference line (1111) (e.g., the adjacent reconstruction sample of the upper line above the current block (1101)), or (iii) the left reference line (1112) (e.g., the adjacent reconstruction sample of the leftmost line to the left of the current block (1101)) and the upper reference line (1111) (e.g., the adjacent reconstruction sample of the upper line above the current block (1101)) depends on the intra-prediction direction of the angular intra-prediction mode. In the example shown in Figure 11, the intra-prediction direction of the angular intra-prediction mode is upward, indicated by the arrows (1121)-(1123). For example, whether the left reference line (e.g., left reference line (1112)), the upper reference line (also called the upper reference line, such as upper reference line (1111)), or both the left and upper reference lines are used to derive the intra interpolation filter may depend on a given intra prediction direction, as indicated by the arrows (1121)-(1123) in Figure 11. In one example, if the prediction direction is horizontal, the left reference line is used. In another example, if the prediction direction is vertical, the upper reference line is used.

[0116] In one embodiment, only the intra-prediction direction is used to derive the selectable filter. For example, adjacent reconstruction samples of each line of multiple lines (e.g., reference line 0) are predicted based on the intra-prediction direction of the angular intra-prediction mode, such as the prediction direction indicated by arrows (1121)-(1123). Referring to Figure 11, in this case, reference line 0 can be predicted based on the prediction direction indicated by arrows (1121) from reference lines 1-3, reference line 1 can be predicted based on the prediction direction indicated by arrows (1122) from reference lines 2-3, reference line 2 can be predicted based on the prediction direction indicated by arrow (1123) from reference line 3, and no prediction is made for reference line 3.

[0117] In one embodiment, only the opposite direction of the intra-prediction direction is used to derive the selectable filters. For example, adjacent reconstruction samples of each line of multiple lines (e.g., reference line 1) are predicted based on the opposite direction of the intra-prediction direction of the angular intra-prediction mode, such as the opposite direction indicated by arrows (1124)-(1126). Referring to Figure 11, in this case, reference line 1 can be predicted based on the opposite direction indicated by arrow (1124) from reference line 0, reference line 2 can be predicted based on the opposite direction indicated by arrow (1125) from reference lines 0-1, reference line 3 can each be predicted based on the opposite direction indicated by arrow (1126) from reference lines 0-2, and no prediction is made for reference line 0.

[0118] According to one aspect of the present disclosure, the selection of an intra interpolation filter from a predetermined set of intra interpolation filters and the selection of an angular intra prediction mode from a predetermined set of angular intra prediction modes can be performed based on adjacent reconstruction samples in the current block (1101). Both the intra interpolation filters and the angular intra prediction mode can be selected based on adjacent reconstruction samples in the current block (1101).

[0119] In one example, the selection of both the directional intra-prediction mode (or angle intra-prediction mode) and the intra-interpolation filter can be evaluated by a cost value calculated as the prediction error of the prediction reference lines between them, and the best L combinations of intra-modes (e.g., one or more intra-modes from a given set of angle intra-prediction modes) and filters (e.g., one or more filters from a given set of intra-prediction filters) are selected as candidates. From the best L combinations of intra-modes and filters, the actual angle intra-prediction mode (e.g., associated with the intra-prediction angle) and interpolation filter (e.g., the intra-interpolation filter) used in the intra-prediction process can be obtained.

[0120] In one example, the selection of intra-interpolation filters and angular intra-prediction modes includes the following: N1 angular intra-prediction modes can be derived from a plurality of angular intra-prediction modes (e.g., a predetermined set of angular intra-prediction modes), where the N1 angular intra-prediction modes are associated with the N1 lowest cost values ​​among the cost values ​​of each angular intra-prediction mode. Each cost value (e.g., the cost value of each angular intra-prediction mode) may be determined based on the adjacent reconstruction samples of the current block (1101), the respective angular intra-prediction mode, and a default intra-interpolation filter (e.g., a filter from a predetermined set of intra-interpolation filters). For each of the N1 angular intra-prediction modes, M1 intra-interpolation filters may be selected from the predetermined set of intra-interpolation filters, associated with the M1 lowest cost values ​​among the cost values ​​of the predetermined set of intra-interpolation filters. Each cost value (for example, the cost value of each filter in a given set of intra interpolation filters) can be determined based on the adjacent reconstruction samples of the current block, each intra interpolation filter, and each angle intra prediction mode among N1 angle intra prediction modes. Intra interpolation filters and angle intra prediction modes can be selected from N1 × M1 combinations, where each of the N1 × M1 combinations may include an intra interpolation filter from M1 intra interpolation filters and an angle intra prediction mode from N1 angle intra prediction modes.

[0121] For example, a default intra-interpolation filter (which may be predetermined) is applied to derive the best N1 intra-prediction modes (e.g., the N1 angular intra-prediction modes described above) using adjacent reconstruction samples. After the best N1 intra-prediction modes have been selected, the best M1 interpolation filters may be further selected, for example, from a predetermined set of intra-interpolation filters. Examples of M1 and N1 values ​​include, but are not limited to, 1, 2, 3, or 4. M1 and N1 are positive integers.

[0122] In one embodiment, the intra-prediction mode (e.g., angle intra-prediction mode) and the intra-interpolation filter can be determined iteratively by adjacent reconstruction samples. That is, the intra-prediction filter can be selected first using the method described above, for example, as one of M1 intra-interpolation filters. Then, the best L1 intra-prediction modes are further selected by trying each intra-prediction mode within adjacent reconstruction samples and selecting the L1 intra-prediction modes associated with the smallest L1 prediction errors for that intra-prediction filter. Subsequently, the intra-interpolation filter can be further improved using a similar approach. The intra-prediction mode can be further improved using the improved intra-interpolation filter.

[0123] In one example, this iterative search of an intra-prediction mode (e.g., an angle intra-prediction mode) and an intra-interpolation filter may be terminated when the prediction error is no longer reduced, when the prediction error is within a predetermined threshold (e.g., a first predetermined threshold), or when the reduction in the prediction error during the search process is within a predetermined threshold (e.g., a second predetermined threshold) (e.g., the difference in errors between consecutive iterations is within a second predetermined threshold).

[0124] An example of the last two iterations in the iterative process (e.g., the first and second iterations) is described below. In the first iteration, N2 angle intra-prediction modes may be selected from a first set of multiple angle intra-prediction modes based on the adjacent reconstruction sample of the current block and the intra-interpolation filter. As illustrated in Figure 11, a first updated intra-interpolation filter may be selected based on the adjacent reconstruction sample of the current block and one of the N2 directional intra-prediction modes. In the second iteration, N3 angle intra-prediction modes may be selected from a second set of multiple angle intra-prediction modes based on the adjacent reconstruction sample of the current block and the first updated intra-interpolation filter. A second updated intra-interpolation filter may be selected based on the adjacent reconstruction sample of the current block and one of the N3 angle intra-prediction modes. The iteration process can be terminated and an intra interpolation filter and an angular intra prediction mode can be selected based on (i) the first prediction error of the first iteration and the second prediction error of the second iteration, and (ii) the second prediction error of the second iteration. The first prediction error may be associated with the first updated intra interpolation filter and one of the N2 directional intra prediction modes. The second prediction error may be associated with the second updated intra interpolation filter and one of the N3 angular intra prediction modes. In one example, based on the difference between the first and second prediction errors, such as the absolute difference between the first and second prediction errors being within a second predetermined threshold, the second updated intra interpolation filter is selected as the selected intra interpolation filter and one of the N3 angular intra prediction modes is selected as the selected angular intra prediction mode. In one example, based on the second prediction error, such as the second prediction error being within the first predetermined threshold, the second updated intra interpolation filter is selected as the selected intra interpolation filter, and one of the N3 angle intra prediction modes is selected as the selected angle intra prediction mode.

[0125] In one example, the first and second sets of modes are the same as the above-mentioned multiple angle intra-prediction modes. In one example, the first and second sets of modes are subsets of the above-mentioned multiple angle intra-prediction modes. These subsets of the above-mentioned multiple angle intra-prediction modes can be obtained from previous iterations. For example, the second set of modes are the above-mentioned N2 angle intra-prediction modes.

[0126] In one embodiment, the angle intra-prediction mode and intra-interpolation filter may be determined based on adjacent reconstruction samples in another iterative manner, i.e., an angle intra-prediction mode is first selected, then the best intra-prediction filter is obtained, and then, for example, the angle intra-prediction mode is improved using a similar approach based on one of the best intra-prediction filters. The intra-prediction filter may be further improved using the improved angle intra-prediction mode.

[0127] According to one aspect of the present disclosure, for each combination of an angle intra-prediction mode from a plurality of angle intra-prediction modes and an intra-interpolation filter from a predetermined set of intra-interpolation filters, a cost value is determined based on adjacent reconstruction samples of the current block, for example, using the method described in Figure 11. In one example, the cost value is determined based on the reconstruction samples in a reference line (e.g., reference line 0) and the predicted samples of the reference line (e.g., predicted by reference line 1 using the angle intra-prediction mode and intra-interpolation filter in that combination), as shown in Figure 11. In another example, the cost value is determined based on the reconstruction samples in a reference line (e.g., reference line 0) and the predicted samples of the reference line using each of the reference lines (e.g., predicted by reference lines 1 to 3 using the angle intra-prediction mode and intra-interpolation filter in that combination), and the cost value is one of the cost values ​​(e.g., the minimum cost value) as shown in Figure 11. K combinations may be selected based on the determined cost values. Based on index information signaled within the bitstream, an intra-interpolation filter and an intra-angle prediction mode can be selected from K combinations. In one example, the combinations of the intra-angle prediction mode and the intra-interpolation filter are ranked, for example, in ascending order of cost value, and the K combinations are associated with the smallest of the K cost values.

[0128] In one example, the selection of a directional intra-prediction mode and an intra-interpolation filter using adjacent reconstruction samples can be performed on both the encoder and decoder sides. A combination of an intra-prediction mode (e.g., an angular intra-prediction mode) and an intra-interpolation filter may be selected. In one example, the order of the combinations of directional intra-prediction modes and intra-interpolation filters is determined based on the calculated cost value of each combination. Figure 11 shows an example of calculating the cost value of each combination. Syntax elements may be signaled in the bitstream to indicate, for example, which combination should be used for the current block. In one example, combinations of intra-prediction modes (e.g., an angular intra-prediction mode) and intra-interpolation filters may be ranked, for example, based on the calculated cost value of the combination.

[0129] According to one aspect of the present disclosure, feature values ​​can be calculated based on adjacent reconstruction samples of the current block, and based on the calculated feature values, one intra-interpolation filter can be selected from a predetermined set of intra-interpolation filters. Feature values ​​may be calculated as absolute gradient values ​​related to adjacent reconstruction samples of the current block, or as the difference between the minimum and maximum values ​​among the adjacent reconstruction samples of the current block. For example, given adjacent reconstruction samples, feature values ​​such as absolute gradient values ​​and the difference between the minimum and maximum values ​​(e.g., the difference between the minimum and maximum values) are calculated. Then, based on the feature values, an intra-interpolation filter (e.g., an intra-interpolation filter to be used in an angular intra-prediction mode to predict the current block) can be selected from a predetermined group of intra-interpolation filters (e.g., a predetermined set of intra-interpolation filters). In one example, an intra-interpolation filter having the maximum or minimum gradient value is selected. In one example, (one or more) absolute gradient values ​​are calculated based on one or more predetermined directions.

[0130] According to one aspect of this disclosure, an output can be obtained from a neural network (e.g., a convolutional neural network (CNN)). The input to the neural network may include adjacent reconstruction samples of the current block, and the output (e.g., an index) indicates a selected intra-interpolation filter, which is one of a predetermined set of intra-interpolation filters. In one example, the input to the neural network may further include an intra-prediction mode index indicating an angular intra-prediction mode.

[0131] In one example, neighboring reconstructed samples are fed into a convolutional neural network (CNN), and the output includes an index of an intra-interpolation filter selected from a given group of intra-interpolation filters (e.g., a given set of intra-interpolation filters). Network parameters (e.g., CNN parameters) can be pre-trained and evaluated on both the encoder and decoder sides, or they can be trained online using the reconstructed blocks. In one example, the intra-predictive mode index is fed into the CNN, along with the neighboring reconstructed samples, to derive the intra-interpolation filter.

[0132] The description of (i) determining (e.g., selecting) an intra interpolation filter for the current block (1101), or (ii) determining (e.g., selecting) an intra interpolation filter and an angular intra prediction mode, can be based on any suitable neighbor reconstruction sample of the current block (1101). The size of the neighbor reconstruction sample of the current block (1101) can depend on (e.g., increase with) the block size of the current block (1101). Referring to Figure 11, the upper reference line (1111) and the left reference line (1112) can include the region (1113), which is the upper-left neighbor of the current block (1101). In some examples, the upper reference line (1111) and / or the left reference line (1112) may include a portion of the region (1113). In some examples, the upper reference line (1111) and / or the left reference line (1112) may not include the region (1113).

[0133] Figure 12 shows a flowchart outlining a process (1200) according to one aspect of the present disclosure. Process (1200) can be used in a video decoder. In various embodiments, process (1200) is executed by processing circuits, such as a processing circuit that performs the functions of a video decoder (110), a processing circuit that performs the functions of a video decoder (210), and similar circuits. In some embodiments, process (1200) is implemented by software instructions, and therefore the processing circuit executes process (1200) when the processing circuit executes software instructions. Process (1200) starts at (S1201) and proceeds to (S1210).

[0134] In (S1210), a bitstream containing the current block in the picture is received. The coding information of the bitstream may indicate that the current block is coded in angle-intra-predictive mode using an intra-interpolation filter.

[0135] In (S1220), an intra interpolation filter may be determined (e.g., selected) from a predetermined set of intra interpolation filters based on the adjacent reconstruction samples of the current block. The adjacent reconstruction samples may include reconstruction samples within N lines from one or more boundaries of the current block, as shown in Figure 11, for example.

[0136] In one embodiment, based on the adjacent reconstruction sample of the current block, one of (i) the type of intra interpolation filter or (ii) the number of taps in the intra interpolation filter is selected from a predetermined set of intra interpolation filters.

[0137] In one example, a given set of intra interpolation filters includes different types of intra interpolation filters. Based on the adjacent reconstruction samples of the current block, a type of intra interpolation filter may be selected from the different types of intra interpolation filters. The selected intra interpolation filter may be one of the following: a bilinear interpolation filter, a cubic interpolation filter, a spline interpolation filter, a DCT-based interpolation filter, or a DST-based interpolation filter.

[0138] In one example, a given set of intra interpolation filters may include different numbers of taps. The number of taps in an intra interpolation filter may be selected based on the adjacent reconstruction samples of the current block. The number of taps in an intra interpolation filter can be one of 2, 4, 6, or 8 taps.

[0139] In one example, whether the adjacent reconstruction sample includes (i) the adjacent reconstruction sample of the left line to the left of the current block, (ii) the adjacent reconstruction sample of the upper line above the current block, or (iii) the adjacent reconstruction sample of the left line to the left of the current block and the adjacent reconstruction sample of the upper line above the current block depends on the intra-prediction direction of the angle intra-prediction mode.

[0140] In one example, the value of N depends on the block size.

[0141] In one embodiment, N is greater than 1, and the adjacent reconstruction sample includes adjacent reconstruction samples of multiple lines. For each combination of an intra interpolation filter from a predetermined set of intra interpolation filters and an adjacent reconstruction sample of a line from the multiple-line adjacent reconstruction sample, the adjacent reconstruction sample of each line can be predicted using the respective intra interpolation filter based on one or more remaining lines from the multiple-line adjacent reconstruction sample, and at least one prediction error can be obtained. The intra interpolation filter corresponding to the smallest prediction error among the obtained prediction errors can be selected.

[0142] In one example, adjacent reconstructed samples for each line are predicted based on the intra-prediction direction of the angle intra-prediction mode. In another example, adjacent reconstructed samples for each line are predicted based on a direction opposite to the intra-prediction direction of the angle intra-prediction mode.

[0143] In one embodiment, based on adjacent reconstruction samples of the current block, one intra interpolation filter is selected from a predetermined set of intra interpolation filters, and an angle intra prediction mode is selected from a plurality of angle intra prediction modes.

[0144] In one example, N1 angle intraprediction modes are derived from multiple angle intraprediction modes, each associated with the lowest cost value among N1 of the cost values ​​of each angle intraprediction mode. Each cost value may be determined based on the adjacent reconstruction sample of the current block, each angle intraprediction mode, and the default intrainterpolation filter from a given set of intrainterpolation filters. For each of the N1 angle intraprediction modes, M1 intrainterpolation filters may be selected from a given set of intrainterpolation filters, each associated with the lowest cost value among M1 of the cost values ​​of the given set of intrainterpolation filters. Each cost value may be determined based on the adjacent reconstruction sample of the current block, each intrainterpolation filter, and each of the N1 angle intraprediction modes. One intrainterpolation filter and angle intraprediction mode may be selected from N1 × M1 combinations. Each of the N1 × M1 combinations may include an intrainterpolation filter from M1 intrainterpolation filters and an angle intraprediction mode from N1 angle intraprediction modes.

[0145] In one example, the selection of the angle intraprediction mode and intrainterpolation filter is performed iteratively in an iterative process. The iterative process selects N2 angle intraprediction modes from a first set of multiple angle intraprediction modes based on the adjacent reconstruction sample of the current block and the intrainterpolation filter, selects a first updated intrainterpolation filter based on the adjacent reconstruction sample of the current block and one of the N2 directional intraprediction modes, selects N3 angle intraprediction modes from a second set of multiple angle intraprediction modes based on the adjacent reconstruction sample of the current block and the first updated intrainterpolation filter, and selects N3 angle intraprediction modes based on the adjacent reconstruction sample of the current block and the N3 angle Based on one of the intra-prediction modes, a second updated intra-interpolation filter may be selected, and this may include (i) the difference between a first prediction error associated with the first updated intra-interpolation filter and one of the N2 directional intra-prediction modes and a second prediction error associated with the second updated intra-interpolation filter and one of the N3 angular intra-prediction modes, or (ii) based on the second prediction error, the second updated intra-interpolation filter is selected as the intra-interpolation filter to select, and one of the N3 angular intra-prediction modes is selected as the angular intra-prediction mode to select. In one example, the first and second multiple modes are the same as the above multiple angular intra-prediction modes. In one example, the first and second multiple modes are subsets of the above multiple angular intra-prediction modes. These subsets of angular intra-prediction modes can be obtained from previous iterations. For example, the second multiple modes are N2 angular intra-prediction modes.

[0146] In one example, for each combination of an angle intra-prediction mode from among several angle intra-prediction modes and an intra-interpolation filter from a predetermined set of intra-interpolation filters, a cost value is determined based on the adjacent reconstruction samples of the current block. Based on the determined cost values, K combinations can be selected. Based on index information signaled within the bitstream, an intra-interpolation filter and an angle intra-prediction mode can be selected from the K combinations.

[0147] In one embodiment, feature values ​​can be calculated based on adjacent reconstruction samples of the current block, and an intra interpolation filter can be selected based on the calculated feature values.

[0148] In one embodiment, a neural network (e.g., a CNN) may be used to determine an intra-interpolation filter for the current block. The input to the neural network contains adjacent reconstruction samples of the current block, and the output is obtained from the neural network based on the input. The output may indicate the selected intra-interpolation filter. In one example, the input to the neural network further includes an intra-prediction mode index indicating the angular intra-prediction mode.

[0149] In (S1230), for example, as shown in Figure 10, the selected intra interpolation filter can be applied to the reference sample in the reference line within the picture to predict the sample in the current block using the angle intra prediction mode.

[0150] In (S1240), the current block can be reconstructed based on the predicted sample.

[0151] The process then proceeds to (S1299) and terminates.

[0152] Process (1200) can be suitably adapted. One or more steps of process (1200) can be modified and / or omitted. One or more additional steps can be added. Any preferred order of implementation can be used. Figure 14 shows a variation of process (1200).

[0153] Figure 13 shows a flowchart outlining a process (1300) according to one aspect of the present disclosure. Process (1300) can be used in a video encoder. In various embodiments, process (1300) is executed by processing circuits, such as a processing circuit that performs the functions of a video encoder (103), a processing circuit that performs the functions of a video encoder (303), and similar circuits. In some embodiments, process (1300) is implemented by software instructions, and therefore the processing circuit executes process (1300) when the processing circuit executes the software instructions. The process starts at (S1301) and proceeds to (S1310).

[0154] In (S1310), an intra-interpolation filter may be selected from a predetermined set of intra-interpolation filters based on the adjacent reconstruction samples of the current block. The adjacent reconstruction samples may include reconstruction samples within N lines from one or more boundaries of the current block. The current block is coded in angular intra-prediction mode using the selected intra-interpolation filter.

[0155] In (S1320), the selected intra interpolation filter can be applied to the reference sample in the reference line within the picture, and the sample in the current block can be predicted using the angle intra prediction mode.

[0156] In (S1330), the current block is encoded based on the predicted sample.

[0157] The process then proceeds to (S1399) and terminates.

[0158] Process (1300) can be suitably adapted. One or more steps of process (1300) can be modified and / or omitted. One or more additional steps can be added. Any preferred order of implementation can be used.

[0159] Figure 14 shows a flowchart outlining a process (1400) according to one aspect of the present disclosure. Process (1400) can be used in a video decoder. In various embodiments, process (1400) is executed by processing circuits, such as a processing circuit that performs the functions of a video decoder (110), a processing circuit that performs the functions of a video decoder (210), and similar circuits. In some embodiments, process (1400) is implemented by software instructions, and therefore the processing circuit executes process (1400) when the processing circuit executes the software instructions. Process (1400) starts at (S1401) and proceeds to (S1410).

[0160] In (S1410), bitstream coding information is received from the bitstream containing the current block in the picture, indicating that the current block is coded in angle intra-prediction mode using an intra-interpolation filter.

[0161] In (S1420), each of a predetermined set of intra interpolation filters is applied to adjacent reconstruction samples within N adjacent lines from the current block boundary. For example, as explained in Figure 11, N is a positive integer.

[0162] In (S1430), as explained in Figures 11 and 12, for example, one intra interpolation filter is selected from a predetermined set of intra interpolation filters based on the prediction error associated with each of the predetermined set of intra interpolation filters.

[0163] In (S1440), for example, as explained in Figure 12, the samples in the current block are predicted in angle intra-prediction mode using one selected intra-interpolation filter.

[0164] In (S1450), the current block is reconstructed based on the predicted sample.

[0165] The process then proceeds to (S1499) and terminates.

[0166] Process (1400) can be suitably adapted. One or more steps of process (1400) can be modified and / or omitted. One or more additional steps can be added. Any preferred order of implementation can be used.

[0167] The embodiments and examples in this disclosure may be used separately or in combination in any order. Furthermore, each of these methods (or embodiments, examples), encoders, and decoders may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, these one or more processors execute a program stored on a non-temporary computer-readable medium.

[0168] The technologies described above can be implemented as computer software using computer-readable instructions, physically stored on one or more computer-readable media. For example, Figure 15 shows a computer system (1500) suitable for implementing a particular aspect of the matters disclosed.

[0169] Computer software can be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or subjected to similar mechanisms to produce code having instructions that can be executed directly or via interpretation, microcode execution, and similar means by one or more computer central processing units (CPUs), graphics processing units (GPUs), and similar devices.

[0170] Instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, Internet of Things devices, and similar devices.

[0171] The components shown in Figure 15 with respect to the computer system (1500) are essentially illustrative and are not intended to imply any limitation on the scope of use or functionality of computer software implementing aspects of this disclosure. Nor should the configuration of the components be construed as having any dependency or requirement on any one or combination of the components shown in this exemplary aspect of the computer system (1500).

[0172] The computer system (1500) may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users, for example, via tactile input (e.g., keystrokes, swipes, moving a data glove), audio input (e.g., voice, applause), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., conversations, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still camera), or video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0173] The input human interface device may include one or more of the following: keyboard (1501), mouse (1502), trackpad (1503), touchscreen (1510), data glove (not shown), joystick (1505), microphone (1506), scanner (1507), and camera (1508) (only one of each is shown).

[0174] The computer system (1500) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (1510), data glove (not shown), or joystick (1505), although there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speakers (1509), headphones (not shown), etc.), visual output devices (e.g., screens (1510) including CRT screens, LCD screens, plasma screens, and OLED screens (each having or not having touchscreen input functionality; each having or not having tactile feedback functionality; some of these may be able to output two-dimensional visual output, or output of four or more dimensions through means such as stereoscopic output, etc.), virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), etc.), and printers (not shown).

[0175] The computer system (1500) may also include, for example, optical media including CD / DVD ROM / RW (1520) having CD / DVD or similar media (1521), thumb drives (1522), removable hard drives or / or solid-state drives (1523), legacy magnetic media such as tapes and floppy disks (registered trademarks, not shown), specialized ROM / ASIC / PLD-based devices (not shown) such as security dongles, and similar devices, and related media.

[0176] Those skilled in the art will also understand that the term “computer-readable medium” as used in connection with the matters disclosed herein does not include transmission media, carrier waves, or other transient signals.

[0177] The computer system (1500) may also include an interface (1554) to one or more communication networks (1555). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, metropolitan, vehicle and industrial, real-time, or latency-tolerant. Examples of networks include local area networks such as Ethernet®, cellular networks including wireless LANs, GSM, 3G, 4G, 5G, LTE and similar technologies, wired or wireless wide-area digital television networks including cable TV, satellite TV, and terrestrial broadcast TV, and vehicle and industrial networks including CANBus. Certain networks generally require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (1549) (e.g., a USB port on the computer system (1500)), while others are generally integrated into the core of the computer system (1500) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1500) can communicate with other entities. Such communication may be unidirectional reception only (e.g., broadcast television), unidirectional transmission only (e.g., CANbus to a specific CANbus device), or bidirectional to other computer systems, for example, using local or wide-area digital networks. Specific protocols and protocol stacks may be used on each of the networks and network interfaces as described above.

[0178] The aforementioned human interface device, human-accessible storage device, and network interface can be mounted on the core (1540) of the computer system (1500).

[0179] The core (1540) may include one or more central processing units (CPUs) (1541), graphics processing units (GPUs) (1542), specialized programmable processing units in the form of field-programmable gate arrays (FPGAs) (1543), hardware accelerators for specific tasks (1544), graphics adapters (1550), and the like. These devices, along with read-only memory (ROM) (1545), random access memory (1546), and internal mass storage (1547) such as internal non-user-accessible hard drives, SSDs, and similar devices, may be connected via a system bus (1548). In some computer systems, the system bus (1548) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and similar devices. Peripheral devices may be attached either directly to the core's system bus (1548) or via a peripheral bus (1549). In one example, a screen (1510) can be connected to a graphics adapter (1550). Peripheral bus architectures include PCI, USB, and similar technologies.

[0180] The CPU (1541), GPU (1542), FPGA (1543), and accelerator (1544) can execute certain instructions that, in combination, constitute the aforementioned computer code. This computer code may be stored in ROM (1545) or RAM (1546). Transient data may also be stored in RAM (1546), while permanent data may be stored, for example, in internal mass storage (1547). Fast storage and retrieval to any of the memory devices may be made possible by the use of cache memory that may be associated with one or more CPUs (1541), GPUs (1542), mass storage (1547), ROMs (1545), RAM (1546), and similar devices.

[0181] A computer-readable medium may have computer code thereon for performing various computer implementation processes. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software technology.

[0182] As an example, but not limited to, a computer system having architecture (1500), in particular core (1540), can provide functionality as a result of (one or more) processors (including CPUs, GPUs, FPGAs, accelerators, and similar) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be specific storage of the core (1540) that is non-transient in nature, such as internal mass storage (1547) or ROM (1545) within the core, and media related to user-accessible mass storage as described above. Software implementing various aspects of this disclosure can be stored in such devices and executed by the core (1540). The computer-readable media may include one or more memory devices or chips, depending on the specific needs. The software may cause the core (1540) and, in particular, the processor within it (including CPUs, GPUs, FPGAs, and similar devices) to execute the specific processes or specific parts of the specific processes described herein, including defining data structures to be stored in RAM (1546) and modifying such data structures according to processes defined by the software. In addition, or alternatively, a computer system may provide functionality as a result of logic wired or otherwise embodied in circuits (e.g., accelerators (1544)) that can operate in place of or with the software to execute the specific processes or specific parts of the specific processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuits containing software for execution (e.g., integrated circuits (ICs), etc.), circuits embodying logic for execution, or both, where appropriate. This disclosure includes preferred combinations of hardware and software.

[0183] The use of “at least one of” or “one of” in this disclosure is intended to include any one or a combination of the elements described. For example, references to at least one of A, B, or C, at least one of A, B, and C, at least one of A, B, and / or C, and at least one of A through C are intended to include A only, B only, C only, or any combination thereof. References to one of A or B, and one of A and B, are intended to include A or B or (A and B). The use of “one of” does not preclude any combination of the elements described, for example, when those elements are not mutually exclusive.

[0184] While this disclosure describes several exemplary embodiments, there are many variations, substitutions, and equivalent alternatives that fall within the scope of the disclosure. Therefore, it should be understood that those skilled in the art could devise numerous systems and methods, though not explicitly illustrated or described herein, that embody the principles of the disclosure and thus lie within its spirit and scope.

Claims

1. A method for video decoding performed by one or more processors, The steps include receiving coding information of a bitstream having a current block in a picture, indicating that the current block is coded in an angle-intra prediction mode using an intra interpolation filter, Assuming N is a positive integer, the steps include applying each of a predetermined set of intra interpolation filters to adjacent reconstruction samples within N adjacent lines from the boundary of the current block, The steps include selecting one intra interpolation filter from the predetermined set of intra interpolation filters based on the prediction error associated with each of the predetermined set of intra interpolation filters, The steps include: predicting the sample in the current block using the angle intra prediction mode with the selected intra interpolation filter; The steps include: reconstructing the current block based on the predicted sample; A method of having.

2. The step of selecting the aforementioned intra interpolation filter is: Based on the adjacent reconstruction samples of the current block, select from the predetermined set of intra interpolation filters either (i) the type of the intra interpolation filter, or (ii) the number of taps in one of the intra interpolation filters. The method according to claim 1, comprising:

3. The predetermined set of intra interpolation filters includes different types of intra interpolation filters, The step of selecting one intra interpolation filter includes selecting the type of the one intra interpolation filter from the different types of intra interpolation filters based on the adjacent reconstruction samples of the current block, wherein the selected intra interpolation filter is one of a bilinear interpolation filter, a cubic interpolation filter, a spline interpolation filter, a DCT-based interpolation filter, or a DST-based interpolation filter. The method according to claim 2.

4. The predetermined set of intra interpolation filters includes different numbers of taps, The step of selecting one intra interpolation filter includes selecting the number of taps of the one intra interpolation filter based on the adjacent reconstruction samples of the current block, wherein the number of taps of the one intra interpolation filter is one of 2 taps, 4 taps, 6 taps, or 8 taps. The method according to claim 2.

5. N is greater than 1, and the adjacent reconstruction sample includes adjacent reconstruction samples from multiple lines. The step of selecting one intra interpolation filter includes, for each combination of an intra interpolation filter from the predetermined set of intra interpolation filters and an adjacent reconstruction sample of a line from the adjacent reconstruction samples of the plurality of lines, Using each intra interpolation filter, the adjacent reconstruction sample of each line is predicted based on one or more remaining lines from the adjacent reconstruction samples of the multiple lines, thereby obtaining at least one prediction error. Selecting one intra interpolation filter that corresponds to the smallest prediction error among the acquired prediction errors, Having, The method according to claim 1.

6. The method according to claim 1, wherein whether the adjacent reconstruction sample includes (i) an adjacent reconstruction sample of the left line to the left of the current block, (ii) an adjacent reconstruction sample of the upper line above the current block, or (iii) an adjacent reconstruction sample of the left line to the left of the current block and an adjacent reconstruction sample of the upper line above the current block depends on the intra-prediction direction of the angle intra-prediction mode.

7. The method according to claim 1, wherein the value of N depends on the block size.

8. The method according to claim 5, wherein predicting adjacent reconstruction samples for each line comprises predicting adjacent reconstruction samples for each line based on the intra-prediction direction of the angle intra-prediction mode.

9. The method according to claim 5, wherein predicting adjacent reconstruction samples for each line comprises predicting adjacent reconstruction samples for each line based on a direction opposite to the intra-prediction direction of the angle intra-prediction mode.

10. The method according to claim 1, wherein the step of selecting one intra interpolation filter comprises selecting one intra interpolation filter from a predetermined set of intra interpolation filters based on the adjacent reconstruction samples of the current block, and selecting an angle intra prediction mode from a plurality of angle intra prediction modes.

11. The selection of the aforementioned intra interpolation filter and the aforementioned angle intra prediction mode is: The method involves deriving N1 angle-intra-prediction modes from a plurality of angle-intra-prediction modes, where each cost value is determined based on the adjacent reconstruction sample of the current block, the respective angle-intra-prediction mode, and the default intra-interpolation filter from a predetermined set of intra-interpolation filters. For each of the N1 angle intraprediction modes, the selection involves selecting M1 intrainterpolation filters from a predetermined set of intrainterpolation filters, relating to the lowest of the M1 cost values ​​among the predetermined set of intrainterpolation filters, wherein each cost value is determined based on the adjacent reconstruction sample of the current block, the respective intrainterpolation filter, and the respective angle intraprediction mode among the N1 angle intraprediction modes. The process involves selecting one intra interpolation filter and one angle intra prediction mode from N1 × M1 combinations, wherein each of the N1 × M1 combinations includes one intra interpolation filter from the M1 intra interpolation filters and one angle intra prediction mode from the N1 angle intra prediction modes. The method according to claim 10, comprising:

12. The selection of the aforementioned intra interpolation filter and the aforementioned angle intra prediction mode is: Based on the adjacent reconstruction samples and intra interpolation filters of the current block, select N2 angle intra prediction modes from a first set of multiple angle intra prediction modes. Selecting a first updated intra interpolation filter based on the adjacent reconstruction sample of the current block and one of the N2 angle intra prediction modes, Based on the adjacent reconstruction sample of the current block and the first updated intra interpolation filter, select N3 angle intra prediction modes from a second set of modes among the set of angle intra prediction modes, Selecting a second updated intra interpolation filter based on the adjacent reconstruction sample of the current block and one of the N3 angle intra prediction modes, (i) the difference between a first prediction error associated with the first updated intra interpolation filter and one of the N2 angle intra prediction modes and a second prediction error associated with the second updated intra interpolation filter and one of the N3 angle intra prediction modes, or (ii) selecting the second updated intra interpolation filter as the intra interpolation filter to be selected and selecting one of the N3 angle intra prediction modes as the angle intra prediction mode to be selected, The method according to claim 10, comprising:

13. The selection of the aforementioned intra interpolation filter and the aforementioned angle intra prediction mode is: For each combination of an angle intra prediction mode from the plurality of angle intra prediction modes and an intra interpolation filter from the predetermined set of intra interpolation filters, Based on the adjacent reconstruction samples of the current block, determine the cost value for each combination. Selecting K combinations based on the cost values ​​determined above, Based on the index information signaled within the bitstream, one intra interpolation filter and the angle intra prediction mode are selected from the K combinations. The method according to claim 10, comprising:

14. The step of selecting the aforementioned intra interpolation filter is: Calculating feature values ​​based on the adjacent reconstructed samples of the current block, Selecting one intra interpolation filter based on the calculated feature values, The method according to claim 1, comprising:

15. The calculation of the aforementioned feature values ​​is (i) the absolute gradient value related to the adjacent reconstructed sample of the current block, or (ii) the difference between the minimum value and the maximum value of the adjacent reconstructed sample of the current block. The method according to claim 14, having the following characteristics.

16. The step of selecting the aforementioned intra interpolation filter is: The process involves obtaining an output from a neural network, wherein the input to the neural network includes the adjacent reconstruction samples of the current block, and the output indicates one of the selected intra interpolation filters. The method according to claim 1, comprising:

17. The method according to claim 16, wherein the input to the neural network further includes an intra-prediction mode index indicating the angle intra-prediction mode.

18. One or more processors, One or more memory locations containing computer programs, It has, The computer program causes one or more processors to perform the method described in any one of claims 1 to 17. Device.

19. A computer program that causes a computer to perform the method described in any one of claims 1 to 17.

Images