Method and apparatus for video decoding and method for video encoding - Patents.com

Abstract

The processing circuit receives a video bitstream including a current coding block in a picture coded in TMRL mode. The processing circuit determines a combination of a reference line and an intra-prediction mode for the current coding block. A first combination among the combinations includes a first reference line and a first intra-prediction mode. For the first combination, the processing circuit calculates template entry delta values ​​respectively associated with entries of a template of the current coding block based on the first reference line and the first intra-prediction mode, and calculates a first template matching cost value by unequal weighting of the template entry delta values. The processing circuit sorts the combinations according to the template matching cost values ​​respectively associated with the combinations, selects a particular combination from the sorted combinations, and reconstructs the current coding block according to the particular combination.

Description

[Technical field]

[0001] This disclosure describes embodiments generally relating to video coding. [Background technology]

[0002] The background description provided herein is intended to generally present the context of the present disclosure. The work of the presently named inventors is not expressly or impliedly admitted as prior art to the present disclosure to the extent that that work is described in this Background section, and any aspects of the description that may not otherwise qualify as prior art at the time of filing.

[0003] Image / video compression can help transmit image / video files between different devices, storage devices, and networks with minimal quality loss. In some examples, video codec techniques can compress video based on spatial and temporal redundancy. In an example, a video codec can use a technique called intra-prediction, which can compress images based on spatial redundancy. For example, intra-prediction can use reference data from a current picture being reconstructed for sample prediction. In another example, a video codec can use a technique called inter-prediction, which can compress images based on temporal redundancy. For example, inter-prediction can predict samples in a current picture from a previously reconstructed picture by motion compensation. Motion compensation is commonly represented by a motion vector (MV). Summary of the Invention

[0004] Aspects of the present disclosure provide a method and apparatus for video encoding / decoding. In some examples, the apparatus for video decoding includes a receiving circuit and a processing circuit. The processing circuit receives a video bitstream including a current coding block in a picture and determines that the current coding block is coded in a template-based multiple reference line intra prediction (TMRL) mode. The processing circuit determines a combination of a reference line and an intra prediction mode for intra prediction of the current coding block in the picture. A first combination among the combinations includes a first reference line among the reference lines and a first intra prediction mode among the intra prediction modes. The first reference line is a non-adjacent reference line. For the first combination, the processing circuit calculates template entry delta values ​​respectively associated with entries of a template of the current coding block based on the first reference line and the first intra prediction mode, and calculates a first template matching cost value associated with the first combination by unequal weighting of the template entry delta values. The template of the current coding block is in an adjacent reference line. The processing circuitry sorts the combinations of reference lines and intra-prediction modes according to template matching cost values ​​respectively associated with the combinations, the template matching cost values ​​associated with the combinations including a first template matching cost value associated with a first combination, and the processing circuitry selects a particular combination from the sorted combinations and reconstructs the current coding block according to the particular combination.

[0005] In some examples, the template for the current coding block includes neighboring samples to the current coding block as entries, and the processing circuit determines a reference template in a first reference line that corresponds to the template for the current coding block based on the first intra-prediction mode, the reference template including reference entries that respectively correspond to the entries of the template.

[0006] Further, in some examples, the processing circuitry calculates a first template entry delta value as a difference between a first entry among the entries of the template and a first reference entry among the reference entries of the reference template, the first reference entry corresponding to the first entry. The processing circuitry determines a weight for weighting the first template entry delta value based on at least the first entry and the first reference entry.

[0007] In some examples, the processing circuit determines a weight for weighting the first template entry delta value based on the first entry, the first reference entry, and one or more neighboring reference entries of the first reference entry.

[0008] In some examples, the processing circuit determines a weight for weighting the first template entry delta value based on the first template entry delta value and one or more template entry delta values ​​associated with neighboring entries of the template.

[0009] In some examples, the processing circuit determines a weight for weighting the first template entry delta value based on at least a first gradient value at the first entry and / or a second gradient value at the first reference entry.

[0010] In some examples, the processing circuitry calculates a first template matching cost value associated with the first combination based on a portion of the template.

[0011] In an example, the processing circuit calculates a first template matching cost value based on an upper template that includes upper neighboring samples of the current coding block.

[0012] In an example, the processing circuit calculates a first template matching cost value based on a left template that includes left neighboring samples of the current coding block.

[0013] In some examples, the processing circuitry determines a subset of the combinations based on a predefined range of template matching cost values ​​and selects a particular combination from the subset of the combinations.

[0014] In an example, the processing circuitry selects a first subset of the combinations having a first template matching cost value within a predefined range, and in response to a number of first combinations in the first subset of combinations being less than a particular number specifying a number requirement for the number of available combinations, selects one or more second combinations having a second template matching cost value outside a predefined range, the first subset and the one or more second combinations forming the subset of the combinations, and the number of combinations in the subset equals a particular number.

[0015] In another example, the processing circuitry selects a first subset of the combinations having a first template matching cost value within a predefined range, and selects the subset of combinations from the first subset of combinations in response to a first number of combinations in the first subset of combinations being greater than a particular number specifying a number requirement for available combinations, the number of combinations in the subset being equal to a particular number. For example, the processing circuitry selects the subset of the combinations having the lowest template matching cost value or selects the subset of the combinations having the closest reference line to the template.

[0016] In some examples, the template is in a second reference line and the processing circuitry determines that a projection point of an entry of the template is at a fractional position between two samples in the first reference line and generates a reference entry associated with the entry of the template according to an interpolation based on at least the two samples.

[0017] In some examples, to select a particular combination, the processing circuitry decodes an index from the bitstream along with the picture encoded in the bitstream, the index indicating the particular combination from the sorted combinations.

[0018] Aspects of the disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to perform a method of video decoding.

[0019] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a schematic diagram of an example block diagram of a communication system (100). [Diagram 2] FIG. 2 is a schematic diagram of an example block diagram of a decoder. [Diagram 3] FIG. 2 is a schematic diagram of an example block diagram of an encoder; [Figure 4] 1 shows a diagram of intra-prediction modes in some examples. [Diagram 5] 1 shows a diagram of intra-prediction modes in some examples. [Figure 6] 13 shows pseudocode for generating the most probable mode list in some examples. [Figure 7] 1 shows diagrams of multiple reference lines in several examples. [Figure 8] 1 shows a diagram illustrating a template-based multiple reference line intra-prediction mode in some examples. [Figure 9] 13 illustrates a binarization process and a codeword table for template-based multiple reference line intra-prediction mode in some examples. [Figure 10] 13A-13C show diagrams of templates of reference lines in some examples. [Figure 11]13A-13C show other views of the template for reference lines in some examples. [Figure 12] 1 shows a flowchart illustrating a process according to some embodiments of the present disclosure. [Figure 13] 1 shows a flowchart illustrating another process according to some embodiments of the present disclosure. [Figure 14] FIG. 1 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] 1 depicts a video encoder and a video decoder in a streaming environment as an example application of the disclosed subject matter. The disclosed subject matter can be similarly applicable to other video-enabled applications including, for example, video conferencing, digital TV, streaming services, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0022] The video processing system (100) may include a capture subsystem (113) that may include, for example, a video source (101), e.g., a digital camera, that generates a stream of uncompressed video pictures (102). In an example, the stream of video pictures (102) includes samples taken by the digital camera. The stream of video pictures (102) is represented by a bold line to emphasize its high data volume compared to the encoded video data (104) (or coded video bitstream) and may be processed by an electronic device (120) that includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (104) (or coded video bitstream) is represented by a thin line to emphasize its lower data volume compared to the stream of video pictures (102) and may be stored in a streaming server (105) for future use. One or more streaming client subsystems, such as the client subsystems (106) and (108) of FIG. 1, can access the streaming server (105) to retrieve copies (107) and (109) of the encoded video data (104). The client subsystem (106) can include a video decoder (110), for example in an electronic device (130). The video decoder (110) decodes an incoming copy (107) of the encoded video data and generates an outgoing stream (111) of video pictures that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., a video bitstream) can be encoded according to a particular video coding / compression standard. An example of such a standard is ITU-T Recommendation H.265.In an example, the developing video coding standard is commonly known as Versatile Video Coding (VVC), and the disclosed subject matter may be used in conjunction with VVC.

[0023] It should be noted that electronic devices 120 and 130 may include other components (not shown). For example, electronic device 120 may include a video decoder (not shown), and electronic device 130 may similarly include a video encoder (not shown).

[0024] 2 shows an example block diagram of a video decoder (210). The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., a receiving circuit). The video decoder (210) may be used in place of the video decoder (110) of the example of FIG. 1.

[0025] The receiver (231) may receive one or more coded video sequences to be decoded by the video decoder (210). In an embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of the other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (231) may receive the coded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective use entities (not shown). The receiver (231) may separate the coded video sequences from the other data. To combat network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter "parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). In others, it can be external to the video decoder (210) (not shown). In still others, there can be a buffer memory (not shown) external to the video decoder (210), e.g., to combat network jitter, plus another buffer memory (215) within the video decoder (210), e.g., to manipulate playback timing. When the receiver (231) is receiving data from a storage / forwarding device of sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (215) may not be needed or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (215) may be needed and may be relatively large and advantageously of adaptive size, and may be implemented at least partially in an operating system or similar element (not shown) external to the video decoder (210).

[0026] The video decoder (210) may include a parser (220) for reconstructing symbols (221) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (210) and potentially information for controlling a rendering device such as a render device (212) (e.g., a display screen) that is not an essential part of the electronic device (230) but may be coupled to the electronic device (230) as shown in FIG. 2. The control information for the rendering device may take the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, context-dependent or non-context-dependent arithmetic coding, etc. The parser (220) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (220) may also extract information from the coded video sequence information, such as transform coefficients, quantization parameter values, motion vectors, etc.

[0027] The parser (220) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (215) to generate symbols (221).

[0028] The reconstruction of the symbols (221) can have many different units depending on the type of coded video picture or portion thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. Which units are included and how may be controlled by subgroup control information parsed by the parser (220) from the coded video sequence. The flow of such subgroup control information between the parser (220) and the following units is not shown for clarity.

[0029] Beyond the functional blocks already mentioned, the video decoder (210) may be conceptually subdivided into a number of functional units, which are described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate.

[0030] The first unit is a scalar / inverse transform unit (251), which receives the quantized transform coefficients as symbols (221) from the parser (220) along with control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The scalar / inverse transform unit (251) can output blocks containing sample values ​​that can be input to an aggregator (255).

[0031] In some cases, the output samples of the scaler / inverse transformer (251) may relate to intra-coded blocks, which are blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from the current picture buffer (258). The current picture buffer (258) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (255) adds, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (252) to the output sample information provided by the scaler / inverse transformer unit (251).

[0032] In other cases, the output samples of the scaler / inverse transform unit (251) may relate to an inter-coded and potentially motion-compensated block. In such cases, the motion compensated prediction unit (253) may access the reference picture memory (257) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (221) related to the block, the samples may be added by the aggregator (255) to the output of the scaler / inverse transform unit (251) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (257) from which the motion compensated prediction unit (253) fetches prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (253), for example in the form of symbols (221), which may have X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (257) when sub-sample accurate motion vectors are used, as well as motion vector prediction mechanisms.

[0033] The output samples of the aggregator (255) may be subjected to various loop filtering techniques in the loop filter unit (256). Video compression techniques may include in-loop filter techniques that are controlled by parameters that are included in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression may also be responsive to meta-information obtained during the decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and may also be responsive to previously constructed loop filtered sample values.

[0034] The output of the loop filter unit (256) can be a sample stream that can be output to a render device (212) and further stored in a reference picture memory (257) for use in future inter-picture prediction.

[0035] Once a particular coded picture is fully reconstructed, it may be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) may become part of the reference picture memory (257), and any unused current picture buffer may be reallocated before beginning reconstruction of a subsequent coded picture.

[0036] The video decoder (210) may perform decoding operations according to a given video compression technique or standard in a standard such as ITU-T Recommendation H.265. A coded video sequence may conform to a syntax prescribed by the video compression technique or standard in use, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and a profile documented in the video compression technique or standard. In particular, a profile may select a particular tool from all tools available in the video compression technique or standard as the only tool available for use under that profile. Also, for compliance, the complexity of the coded video sequence must be within the bounds defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second), maximum reference picture size, etc. The limits set by the level may in some cases be further limited through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0037] In an embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may also be included as part of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0038] 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit). The video encoder (303) may be used in place of the example video encoder (103) of FIG.

[0039] The video encoder (303) may receive video samples from a video source (301) (which in the example of FIG. 3 is not part of the electronic device (320)) that may capture video images to be coded by the video encoder (303). In other examples, the video source (301) is part of the electronic device (320).

[0040] The video source (301) may provide a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCB, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (301) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that, when viewed in sequence, impart motion. The pictures themselves may be organized as a spatial array of pixels, each of which may have one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The remainder of this specification focuses on samples.

[0041] According to an embodiment, the video encoder (303) may code and compress pictures of a source video sequence into a coded video sequence (343) in real-time or under any other time constraints as required. Imposing an appropriate coding rate is one function of the controller (350). In some embodiments, the controller (350) controls and is operatively coupled to other functional units as described below. Couplings are not shown for clarity. Parameters set by the controller (350) may include parameters related to rate control (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. The controller (350) may be configured with other appropriate functions related to the video encoder (303) optimized for a particular system design.

[0042] In some embodiments, the video encoder (303) is configured to operate in a coding loop. As an oversimplified description, in an example, the coding loop can include a source coder (330) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to generate sample data in a similar manner that a (remote) decoder would also generate. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Since the decoding of the symbol stream produces bit-exact results independent of the location of the decoder (local or remote), the content in the reference picture memory (334) is also bit-perfect between the local and remote encoders. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the decoder would "see" when using the prediction during decoding as reference picture samples. This basic principle of reference picture synchronicity (and the resulting drift when synchronicity cannot be maintained, for example due to channel errors) is also used in several related techniques.

[0043] The operation of the "local" decoder (333) can be the same as a "remote" decoder, such as the video decoder (210), already described in detail above in conjunction with Figure 2. Referring also momentarily to Figure 2, however, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and the parser (220), may not be fully implemented in the local decoder (333), given that symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (345) and parser (220) can be lossless.

[0044] In embodiments, the decoder techniques, with the exception of the parsing / entropy decoding present in the decoder, are present in the corresponding encoder in the same or substantially the same functional form. Thus, the disclosed subject matter focuses on the operation of the decoder. Descriptions of the encoder techniques may be omitted, inasmuch as they are the inverse of the decoder techniques described generically. To the extent specified, more detailed descriptions are provided below.

[0045] In operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from the video sequence designated as "reference pictures." In this manner, the coding engine (332) codes differences between pixel blocks of the input picture and pixel blocks of the reference pictures that may be selected as predictive references for the input picture.

[0046] The local video decoder (333) may decode the coded video data of pictures that may be designated as reference pictures based on the symbols generated by the source coder (330). The operation of the coding engine (332) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 3), the reconstructed video sequence is usually a copy of the source video sequence with some errors. The local video decoder (333) may replicate the decoding process that may be performed by the video decoder on the reference pictures, such that the reconstructed reference pictures are stored in the reference picture cache (334). In this way, the video encoder (303) may locally store copies of reconstructed reference pictures that have common content with the reconstructed reference pictures that would be obtained by the far-end video decoder (without transmission errors).

[0047] The predictor (335) may perform a prediction search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) may look for certain metadata, such as reference picture motion vectors, block shapes, or sample data (as candidate reference pixel blocks) from the reference picture memory (334) that may be suitable prediction references for the new picture. The predictor (335) may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have prediction references derived from multiple reference pictures stored in the reference picture memory (334).

[0048] The controller (350) may manage the coding operations of the source coder (330), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0049] The output of all the above functional units may undergo entropy coding in an entropy coder (345), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0050] The transmitter (340) may buffer the coded video sequence produced by the entropy coder (345) to prepare it for transmission over a communication channel (360), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (340) may merge the coded video data from the video coder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0051] The controller (350) may manage the operation of the video encoder (303). During coding, the controller (350) may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0052] An Intra Picture (I-picture) may be a picture that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow various types of Intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of such variations of I-pictures and their respective applications and characteristics.

[0053] A Predictive Picture (P-picture) may be a picture that can be encoded and decoded by intra- or inter-prediction using at most one motion vector and reference index to predict the sample values ​​of each block.

[0054] A Bi-directionally Predictive Picture (B-picture) may be a picture that can be coded and decoded using intra- or inter-prediction that uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple-predictive picture(s) can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0055] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to each picture of the block. For example, blocks of I pictures may be non-predictively coded or they may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P pictures may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of B pictures may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0056] The video encoder (303) may perform coding operations according to a given video coding technique or standard, such as ITU-T Recommendation H.265. During its operation, the video encoder (303) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax defined by the video coding technique or standard being used.

[0057] In an embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0058] A video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In an example, a particular picture being coded / decoded, called a current picture, is partitioned into blocks. If a block in the current picture is similar to a reference block in a previously coded reference picture in the video that is still buffered, then that block in the current picture may be coded by a vector called a motion vector. A motion vector points to a reference block in a reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0059] In some embodiments, bi-prediction techniques may be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, e.g., a first reference picture and a second reference picture, both preceding in decoding order (but may be past and future, respectively, in display order) the current picture in the video. A block in the current picture may be coded with a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. The block is predictable by a combination of the first and second reference blocks.

[0060] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0061] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be partitioned into one CU of 64×64 pixels, or four CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In an example, each CU is analyzed to determine a prediction type for the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. In general, each PU includes one luma prediction block (PB) and two chroma PBs. In an embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0062] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In some embodiments, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0063] Aspects of the present disclosure provide techniques for template matching for multiple reference line intra prediction.

[0064] FIG. 4 shows a diagram of intra prediction modes in some examples such as HEVC. For example, HEVC uses a total of 35 intra prediction modes (e.g., mode 0 to mode 34). Among the 35 intra prediction modes, some modes are directional modes and some modes are non-directional modes. In some examples, mode 0 and mode 1 are non-directional modes, e.g., mode 0 is a planar mode and mode 1 is a DC mode. Furthermore, mode 2 to mode 34 can be directional modes, e.g., mode 10 is a horizontally-crossed mode, mode 26 is a vertical mode, mode 2, mode 18 and mode 34 are diagonal modes, etc. The value of a sample in a coding block is determined according to the adjacent reference samples in the same picture and the intra prediction mode of the coding block. In an example, in DC mode, an average value is calculated by averaging reference samples in the same picture and can be used for flat surfaces. In another example, in planar mode, the value of each sample in a coding block is calculated assuming an amplitude surface with horizontal and vertical smooth gradients derived from the boundary samples of adjacent blocks. In some examples, the reference samples include neighboring samples in the row immediately above the coding block and / or neighboring samples in the column immediately to the left of the coding block.

[0065] In some examples, the intra prediction mode is signaled based on a list of Most Propable Mode(s) (MPM). For example, for a coding block, an MPM list is determined. In an example, the MPM list includes three modes from 35 intra prediction modes. In that case, when a particular intra prediction mode of a coding block is one of the three modes in the MPM list, an index indicating the one of the three modes is used for signaling. When a particular intra prediction mode of a coding block is not one of the three modes in the MPM list, an index indicating one of the remaining modes (32 modes) is used for signaling. In some examples, the MPM list may include other suitable numbers of modes, such as 6, 10, etc.

[0066] However, any other suitable number of intra-prediction modes may be used.

[0067] 5 shows a diagram of intra-prediction modes in some examples such as VVC. In some examples, VVC can use a total of 95 intra-prediction modes, such as mode -14 to mode 80. Among the 95 intra-prediction modes, mode 0 is the planar mode, mode 1 is the DC mode, mode 18 is the horizontal mode, mode 50 is the vertical mode, and modes 2, 34, and 66 are diagonal modes. Modes -1 to -14 and modes 64 to 80 are referred to as Wide-Angle Intra Prediction (WAIP) modes in some examples.

[0068] In some examples, to code an intra mode (also called an intra prediction mode) of a coding block (e.g., a luma block, a chroma block of a coding unit), a Most Probable Mode (MPM) list of size 3 is constructed based on the intra modes of neighboring blocks of the coding block. The MPM list may be called an MPM list or a primary MPM list. If the intra mode of the coding block is not from the MPM list, a flag is signaled to indicate whether the intra mode belongs to a selected mode in the MPM list.

[0069] FIG. 6 shows pseudo codes for generating an MPM list in some examples. In the example of FIG. 6, leftIntraDir is used to indicate the intra mode of the left block (adjacent to the current coding block), and aboveIntraDir is used to indicate the intra mode of the above block (adjacent to the current coding block). In an example, if the left or above block is not currently available, leftIntraDir or aboveIntraDir may be set to DC_IDX (index of DC mode). In some examples, the variables "offset" and "mod" in FIG. 6 are constant values ​​set to 29 and 32, respectively.

[0070] In some related examples, reference samples for intra prediction of a coding block are located immediately adjacent to the coding block, and the neighboring portion of the coding block formed by the reference samples may be referred to as a reference line.

[0071] In accordance with aspects of the present disclosure, a technique called multi-line intra prediction can use more reference lines for intra prediction. In some examples, the encoder determines and signals which reference lines are used to generate the intra predictor. In examples, the reference line index is signaled before the intra prediction mode, and only the most probable mode is allowed if a non-zero reference line index is signaled.

[0072] FIG. 7 shows a diagram of multiple reference lines in some examples. In FIG. 7, four reference lines of a coding block (710) are shown. A first reference line (720), also called reference line 0, includes a row of reference samples immediately above the coding block (710) and a column of reference samples immediately to the left of the coding block (710). A second reference line (730), also called reference line 1, includes a row of reference samples immediately above the first reference line (720) and a column of reference samples immediately to the left of the first reference line (720). A third reference line (740), also called reference line 2, includes a row of reference samples immediately above the second reference line (730) and a column of reference samples immediately to the left of the second reference line (730). The fourth reference line (750), also referred to as reference line 3, includes a row of reference samples immediately above the third reference line (740) and a column of reference samples immediately to the left of the third reference line (740). Furthermore, each reference line consists of six segments, as indicated by segments A through F, with the top left reference sample indicated by T. In some examples, segments A and F are padded with the nearest samples from segments B and F, respectively.

[0073] In some examples, a technique called template-based multiple reference line intra-prediction (TMRL) mode may be used. The TMRL mode includes four aspects.

[0074] In some examples, for the first aspect of the TMRL mode, two candidate lists are used, one for the reference line and the other for the intra prediction mode. In the examples, an extended reference line candidate list and an intra prediction mode candidate list are used. For example, the extended reference line candidate list includes {1, 3, 5, 7, 12}. The restriction on the top row of the CTU is unchanged. In the examples, the size of the intra prediction mode candidate list is 10. The configuration of the intra prediction mode candidate list is similar to that of MPM. The difference in the intra prediction mode candidates for MPM is that the planar mode is excluded from the intra prediction mode candidate list.

[0075] In some examples, for the second aspect of the TMRL mode, a TMRL combination list is configured. In the examples, combinations of extended reference lines and allowed intra prediction modes may be configured for a coding block. Each combination includes a reference line selected from the extended reference line candidate list and an intra prediction mode selected from the intra prediction mode candidate list. In the examples, five reference lines are in the extended reference line candidate list of a coding block, and ten intra prediction modes are in the intra prediction mode candidate list of a coding block, so that 50 combinations may be configured for the coding block.

[0076] In some examples, the extended reference lines start from reference line 1, and the area covered by reference line 0 is used for template matching.

[0077] FIG. 8 shows a diagram illustrating a template-based multiple reference line intra-prediction mode in some examples. FIG. 8 shows a current coding block (810) with reference lines such as reference line 0 (820), reference line 1 (830), reference line 2 (840), and reference line 3 (850). Based on the area covered by reference line 0 (820), the template area may be defined as a top template, a left template, a template including a top template and a left template, etc. However, in some examples, the template area has already been reconstructed at the time of coding the current coding block (810).

[0078] In some examples, for each of the 50 combinations of the extended reference line and the allowed intra-prediction mode, a projection of the template area may be generated according to the reference line and the intra-prediction mode of the combination. Furthermore, a template matching cost, such as a sum of absolute difference (SAD) cost, is calculated between the projection of the template area and the template area. In the example, the first 20 combinations with the lowest cost are pruned from the combinations in ascending order of cost to form a combination candidate list.

[0079] In some examples, the third aspect of TMRL is related to TMRL signaling. In some examples, such as ECM or VTM, a multiple reference line (MRL) is combined with a non-planar intra-prediction mode from the primary MPM list. If an extended reference line is selected, up to six context-coded bins and three equi-probable (EP) bins may be used to represent the combination of the MRL and MPM index. In some examples, in TMRL modes, truncated Golomb-Rice coding with a divisor of four is used to code the selected combination from the combination list.

[0080] FIG. 9 shows a table of binarization processes and codewords in TMRL mode in some examples.

[0081] In some examples, a fourth aspect of TMRL relates to encoder-side modifications. In examples, encoder-side modifications are performed to further improve coding efficiency. In examples, for intra blocks larger than 8×8, more full-rate distortion optimization (RDO) may be allowed for a selected intra prediction mode after comparing a sum of absolute transformed differences (SATD) cost with a threshold.

[0082] Some aspects of the present disclosure provide techniques for TMRL.

[0083] In the following description, the line index of the nearest reference line of a coding block is 0, the nearest reference line is also called the zero reference line or the adjacent reference line, and the other reference lines are called the non-zero reference lines or the non-adjacent reference lines.

[0084] In the following description, a template may refer to an upper neighboring sample, a left neighboring sample, a right neighboring sample, and a lower neighboring sample in a particular reference line of a current coding block.

[0085] Figure 10 shows a diagram of the templates of the reference lines in some examples. In the example of Figure 10, the template of the reference line 0 is indicated by (1020), also called the template of the current coding block (1010), the template of the reference line 1 is indicated by (1030), and the template of the reference line 2 is indicated by (1040).

[0086] FIG. 11 shows another diagram of a template of a reference line in some examples. In the example of FIG. 11, the top template of reference line 0 is indicated by (1125) and is also called the top template of the current coding block (1110). In the example, the top template (1125) may be projected to a non-zero reference line according to a prediction direction (1135) that may be indicated by an intra-prediction mode. The projected template in the non-zero reference line is also called a reference template in some examples. For example, based on the prediction direction (1135), the top template (1125) is projected to the top template (1145) of reference line 2. The top template (1145) is a projection of the top template (1125) in reference line 2 according to the prediction direction.

[0087] In the example of FIG. 11, the left template of reference line 0 is indicated by (1155) and is also referred to as the left template of the current coding block (1110). In an example, according to a prediction direction (1165), the left template (1155) may be projected to a non-zero reference line. The projected template in the non-zero reference line is also referred to as a reference template in some examples. For example, based on the prediction direction (1165), the left template (1155) is projected to the left template (1175) of reference line 2. The left template (1175) is a projection of the left template (1155) according to the prediction direction.

[0088] According to some aspects of the present disclosure, weighting is applied in addition to the TMRL method. In some examples, the template of reference line 0 is denoted as template0, and the template of reference line i (where i is not zero) (also called the reference template) is denoted as template i and the difference between two templates at entry index x (e.g., sample position x in the template for reference line 0) is template delta (x) and can be calculated according to equation (1):

number

[0089] In accordance with some aspects of the present disclosure, the weighting is used to calculate the cost of the reference template. delta applies to each entry in

[0090] In some examples, the weighting is done by applying the weighting factor to the template before calculating the template matching cost, e.g., according to equation (2). delta applied to each entry in:

number

[0091] In an embodiment, the weight value of entry x (also referred to as sample position x) depends on the sample values ​​in the template of reference line 0 and the reference template of reference line i, and on the sample values ​​of adjacent positions relative to entry x.

[0092] In other embodiments, the weight value of entry x depends on the difference between sample values ​​in the template of reference line 0 and the reference template of reference line i located at position x and adjacent positions, e.g. position x-1 and / or position x+1.

[0093] In another embodiment, the weight value of entry x depends on the gradient value of the sample in the template of reference line 0 and the reference template of reference line i located at position x.

[0094] In some examples, depending on the prediction direction, portions of the template are used to calculate the template matching cost.

[0095] In an embodiment, when the prediction direction is, for example, vertical, the reference sample is derived from the adjacent reconstructed sample on the upper side, for example, the top template may be used for cost calculation. In an example, the index of the vertical direction (vertical intra prediction mode) is 50, and the absolute difference of the index of the prediction direction (intra prediction mode) to the index of the vertical direction is less than a threshold value (for example, 6), in which case the top template may be used for template matching cost calculation.

[0096] In other embodiments, when the prediction direction is, for example, horizontal, i.e., the reference sample comes from the adjacent reconstructed sample on the left side, for example, a left template is used. In the example, the horizontal index (horizontal intra-prediction mode) is 18, and the absolute difference of the index of the prediction direction (intra-prediction mode) to the index of the horizontal direction is less than a threshold value (for example, 6), then the left template can be used to calculate the template matching cost.

[0097] In other embodiments, when the prediction direction is between horizontal and vertical, i.e., the reference samples come from adjacent reconstructed samples on both the upper and left sides, e.g., a top template and a left template are used to calculate the template matching cost.

[0098] In some examples, after the reference samples are reconstructed, a smoothing process may be applied to the reference samples, and in some embodiments, the reconstructed reference samples before the smoothing process are used to calculate the template matching cost.

[0099] In some embodiments, after the template matching costs associated with the candidate reference line and intra-prediction mode combinations are calculated, a template matching cost that is within a predefined range [a, b] is selected, and it is determined that the candidate reference line and intra-prediction mode combination associated with the selected template matching cost within the predefined range [a, b] is available for the current coding block. In an example, both a and b are values ​​greater than 0.

[0100] In some embodiments, if the number of combinations of candidate reference lines and intra-prediction modes whose template matching costs are within [a, b] is less than N, where N indicates the required number of available combinations, combinations of candidate reference lines and intra-prediction modes whose template matching costs are outside the range of [a, b] may be added to the available combinations so that the number of available combinations is N.

[0101] In some embodiments, if the number of combinations of candidate reference lines and intra-prediction modes with template matching costs within [a, b] is greater than N, where N indicates the required number of available combinations, only the first N combinations of candidate reference lines and intra-prediction modes that are closer to reference line 0 (i.e., have lower template matching costs) are selected as available combinations.

[0102] According to an aspect of the present disclosure, to calculate a template matching cost between two reference lines, such as reference line i and reference line j, interpolation may be applied by projecting integer samples of the template in reference line i onto reference line j, or by projecting integer samples of the template in reference line j onto reference line i.

[0103] In some examples, when the projected point is at a fractional position between two integer samples in the reference line, interpolation is applied to generate the fractional sample value.

[0104] In some examples, the interpolation method may be the same as that used in the interpolation process in angular (directional) intra prediction.

[0105] In some examples, the interpolation method may be different compared to the interpolation used in angular (directional) intra prediction, for example, bilinear interpolation, 4-tap cubic interpolation, nearest integer, etc.

[0106] 12 shows a flow chart illustrating a process (1200) according to an embodiment of the present disclosure. The process (1200) may be used in a video encoder. In various embodiments, the process (1200) is performed by a processing circuit, such as a processing circuit performing the functions of the video encoder (203), a processing circuit performing the functions of the video encoder (303), etc. In some embodiments, the process (1200) is implemented in software instructions, such that the processing circuit performs the process (1200) when the processing circuit executes the software instructions. The process (1200) starts at (S1201) and proceeds to (S1210).

[0107] At (S1210), a combination of a reference line and an intra prediction mode for intra prediction of a current coding block in a picture is determined. In an example, it is determined that the current coding block is coded by intra prediction. A first candidate list of reference lines is determined, and a second candidate list of intra prediction modes is determined. Then, a combination of a reference line and an intra prediction mode is determined. Each combination among the combinations includes a reference line among the reference lines and an intra prediction mode among the intra prediction modes. For example, a first combination among the combinations includes a first reference line among the reference lines and a first intra prediction mode among the intra prediction modes.

[0108] At (S1220), for a first combination, template entry delta values ​​respectively associated with entries of the template of the current coding block are calculated based on the first reference line and the first intra prediction mode.

[0109] At (S1230), a first template matching cost value associated with the first combination is calculated by unequal weighting of the template entry delta values.

[0110] At (S1240), the combinations of reference lines and intra-prediction modes are sorted according to template matching cost values ​​respectively associated with the combinations, the template matching cost values ​​associated with the combinations including a first template matching cost value associated with a first combination.

[0111] At (S1250), a particular combination is selected from the sorted combinations, for example, the particular combination is selected based on rate-distortion optimization.

[0112] At (S1260), a signal indicative of the particular combination is encoded in a bitstream carrying coded information for the picture.

[0113] In some embodiments, the template of the current coding block includes neighboring samples to the current coding block as entries (e.g., in reference line 0). A reference template in a first reference line corresponding to the template of the current coding block is determined based on the first intra-prediction mode, and the reference template includes reference entries respectively corresponding to the entries of the template.

[0114] In some embodiments, the first template entry delta value is calculated as a difference between a first entry among the entries of the template and a first reference entry among the reference entries of the reference template, In some examples, a weight for weighting the first template entry delta value is determined based on at least the first entry and the first reference entry.

[0115] In some examples, a weight for weighting the first template entry delta value is determined based on the first entry, the first reference entry, and one or more neighboring reference entries of the first reference entry.

[0116] In some examples, a weight for weighting the first template entry delta value is determined based on the first template entry delta value and one or more template entry delta values ​​associated with neighboring entries of the template.

[0117] In some examples, the weight for weighting the first template entry delta value is determined based on at least a first gradient value at the first entry and / or a second gradient value at the first reference entry.

[0118] In some examples, a portion of the neighboring samples in reference line 0 is used to calculate the template matching cost value. In examples, the template matching cost value is calculated according to a top template (e.g., including the top neighboring samples) of the current coding block. In other examples, the template matching cost value is calculated according to a left template (e.g., including the left neighboring samples) of the current coding block.

[0119] In some embodiments, a subset of the combinations is determined based on a predefined range of template matching cost values, and a particular combination is selected from the subset of the combinations.

[0120] In some examples, a first subset of the combinations having a first template matching cost value within a predefined range is determined. In examples, in response to a number of first combinations in the first subset of the combinations being less than a particular number specifying a number requirement of available combinations, one or more second combinations having a second template matching cost value outside a predefined range are selected, the first subset and the one or more second combinations forming the subset of the combinations, and the number of combinations in the subset is equal to the particular number.

[0121] In another example, in response to a first number of combinations in the first subset of combinations being greater than a particular number specifying a number requirement for available combinations, the subset of combinations is selected from the first subset of combinations, and the number of combinations in the subset is equal to the particular number. For example, the subset of combinations having the lowest template matching cost value is selected from the first subset. In another example, the subset of combinations having the closest reference line to a template is selected from the first subset.

[0122] In some examples, the template is in the second reference line, a projection point of the template entry is at a fractional position between two samples in the first reference line, and a reference entry associated with the template entry is generated according to an interpolation based on at least the two samples.

[0123] The process then proceeds to (S1299) and ends.

[0124] The process 1200 may be adapted as appropriate. Steps of the process 1200 may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0125] FIG. 13 shows a flow chart illustrating a process (1300) according to an embodiment of the present disclosure. The process (1300) may be used in a video decoder. In various embodiments, the process (1300) is performed by a processing circuit, such as a processing circuit performing the functions of the video decoder (110), a processing circuit performing the functions of the video decoder (210), etc. In some embodiments, the process (1300) is implemented with software instructions, such that the processing circuit performs the process (1300) when the processing circuit executes the software instructions. The process (1300) starts at (S1301) and proceeds to (S1310).

[0126] At (S1310), a combination of a reference line and an intra prediction mode for intra prediction of a current coding block in a picture is determined. In an example, a first candidate list of reference lines is determined, and a second candidate list of intra prediction modes is determined. Then, a combination of a reference line and an intra prediction mode is determined. Each combination among the combinations includes a reference line among the reference lines and an intra prediction mode among the intra prediction modes. For example, a first combination among the combinations includes a first reference line among the reference lines and a first intra prediction mode among the intra prediction modes.

[0127] At (S1320), for a first combination, template entry delta values ​​respectively associated with entries of the template of the current coding block are calculated based on the first reference line and the first intra prediction mode.

[0128] At (S1330), a first template matching cost value associated with the first combination is calculated by unequal weighting of the template entry delta values.

[0129] At (S1340), the combinations of the reference lines and the intra-prediction modes are sorted according to template matching cost values ​​respectively associated with the combinations, the template matching cost values ​​associated with the combinations including a first template matching cost value associated with a first combination.

[0130] At (S1350), a particular combination is selected from the sorted combinations. In some examples, a signal indicating an index of a particular combination among the sorted combinations is decoded, and the particular combination is selected based on the decoded signal.

[0131] At (S1360), the current coding block is reconstructed according to the particular combination including the reference line and intra prediction mode.

[0132] In some embodiments, the template of the current coding block includes neighboring samples to the current coding block as entries. A reference template in a first reference line corresponding to the template of the current coding block is determined based on a first intra-prediction mode, and the reference template includes reference entries respectively corresponding to the entries of the template.

[0133] In some embodiments, the first template entry delta value is calculated as a difference between a first entry among the entries of the template and a first reference entry among the reference entries of the reference template, the first reference entry corresponding to the first entry. In some examples, a weight for weighting the first template entry delta value is determined based on at least the first entry and the first reference entry.

[0134] In some examples, a weight for weighting the first template entry delta value is determined based on the first entry, the first reference entry, and one or more neighboring reference entries of the first reference entry.

[0135] In some examples, a weight for weighting the first template entry delta value is determined based on the first template entry delta value and one or more template entry delta values ​​associated with neighboring entries of the template.

[0136] In some examples, the weight for weighting the first template entry delta value is determined based on at least a first gradient value at the first entry and / or a second gradient value at the first reference entry.

[0137] In some examples, a portion of the neighboring samples in reference line 0 is used to calculate the template matching cost value. In examples, the template matching cost value is calculated according to a top template (e.g., including the top neighboring samples) of the current coding block. In other examples, the template matching cost value is calculated according to a left template (e.g., including the left neighboring samples) of the current coding block.

[0138] In some embodiments, a subset of the combinations is determined based on a predefined range of template matching cost values, and a particular combination is selected from the subset of combinations.

[0139] In some examples, a first subset of the combinations having a first template matching cost value within a predefined range is determined. In examples, in response to a number of first combinations in the first subset of the combinations being less than a particular number specifying a number requirement of available combinations, one or more second combinations having a second template matching cost value outside a predefined range are selected, the first subset and the one or more second combinations forming the subset of the combinations, and the number of combinations in the subset being equal to the particular number.

[0140] In another example, in response to a first number of combinations in the first subset of combinations being greater than a particular number specifying a number requirement for available combinations, the subset of combinations is selected from the first subset of combinations, the number of combinations in the subset being equal to the particular number. For example, the subset of combinations having the lowest template matching cost value is selected from the first subset. In another example, the subset of combinations having the closest reference line to a template is selected from the first subset.

[0141] In some examples, the template is in the second reference line, a projection point of the template entry is at a fractional position between two samples in the first reference line, and a reference entry associated with the template entry is generated according to an interpolation based on at least the two samples.

[0142] The process then proceeds to (S1399) and ends.

[0143] The process 1300 may be adapted as appropriate. Steps of the process 1300 may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0144] The above techniques can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 14 illustrates a computer system (1400) suitable for implementing certain embodiments of the disclosed subject matter.

[0145] The computer software may be coded in any suitable machine code or computer language that may be subject to mechanisms such as assembly, compilation, linking, etc. to generate code including instructions that may be executed by one or more central processing units (CPUs), graphics processing units (GPUs), etc., directly or through interpretation, microcode execution, etc.

[0146] The instructions may be executable by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, devices for the Internet of Things, and the like.

[0147] 14 for computer system (1400) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1400).

[0148] The computer system (1400) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more users through, for example, tactile input (e.g., keyboard, swipes, dataglove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0149] The input human interface devices may include one or more of a keyboard (1401), a mouse (1402), a trackpad (1403), a touch screen (1410), a data glove (not shown), a joystick (1405), a microphone (1406), a scanner (1407), and a camera (1408) (only one of each is shown).

[0150] The computer system 1400 may also include certain human interface output devices that may stimulate one or more of the user's senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1410), data gloves (not shown), or joystick (1405), although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (1409), headphones (not shown)), visual output devices (e.g., screens (1410) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which are capable of outputting two-dimensional visual output or output in more than three dimensions by means of stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0151] The computer system (1400) may also include human accessible storage devices and their associated media, such as CD / DVD or similar media (1421), including CD / DVD ROM / RW (1420), thumb drives (1422), removable hard disks or solid state drives (1423), legacy magnetic media, such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices, such as security dongles (not shown), and the like.

[0152] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0153] The computer system (1400) may also include an interface (1454) to one or more communication networks (1455). The networks may be, for example, wireless, wireline, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicle and factory networks including CAN bus, etc. Certain networks generally require an external network interface adapter attached to a particular general purpose digital port or peripheral bus (1449) (e.g., a USB port of the computer system (1400)). Others are generally built into the core of the computer system (1400) by attachment to a system bus as described below (e.g., an Ethernet network to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, computer system (1400) can communicate with other entities. Such communications can be one-way receive-only (e.g., broadcast TV) or one-way transmit-only (e.g., a CAN bus to a specific CAN bus device), or can be bidirectional to other computer systems, for example, using local or wide area digital networks. Specific protocols or protocol stacks can be used with each of the networks and network interfaces as described above.

[0154] The above-mentioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (1440) of the computer system (1400).

[0155] The cores (1440) may include one or more central processing units (CPUs) (1441), graphics processing units (GPUs) (1442), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1443), hardware accelerators for specific tasks (1444), graphics adapters (1450), etc. These devices may be connected through a system bus (1448), along with read only memory (ROM) (1445), random access memory (RAM) (1446), internal mass storage devices such as internal non-user accessible hard drives, SSDs, etc. (1447). In some computer systems, the system bus (1448) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1448) or through a peripheral bus (1449). In an example, a display 1410 may be connected to a graphics adapter 1450. Architectures for peripheral buses include PCI, USB, and the like.

[0156] The CPU (1441), GPU (1442), FPGA (1443), and accelerator (1444) can execute certain instructions that, in combination, may constitute the above computer code. The computer code may be stored in a ROM (1445) or a RAM (1446). Temporary data may also be stored in the RAM (1446), while persistent data may be stored, for example, in an internal mass storage device (1447). Rapid storage and retrieval from any of the memory devices may be made possible by the use of a cache memory. A cache memory may be closely associated with one or more of the CPU (1441), GPU (1442), mass storage device (1447), ROM (1445), RAM (1446), etc.

[0157] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those having skill in the computer software arts.

[0158] By way of example, and not by way of limitation, a computer system having the architecture (1400), and in particular the core (1440), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with the user-accessible mass storage devices introduced above, in addition to specific storage devices of the core (1440) that are non-transitory in nature, such as the core's internal mass storage (1447) or ROM (1445). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1440). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (1440), and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (1446) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system can provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerators (1444)) that can operate in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. References to software can encompass logic, where appropriate, and vice versa. References to computer-readable media can encompass circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both, where appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0159] The use of "at least one of" within this disclosure is intended to include any one or combination of the listed elements. For example, at least one of A, B, or C; at least one of A, B, and C; at least one of A, B, and / or C; and at least one of A through C is intended to include only A, only B, only C, or any combination thereof.

[0160] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that are within the scope of this disclosure. Thus, it will be apparent to those skilled in the art that numerous systems and methods will be contemplated that embody the principles of the present disclosure and are therefore within its spirit and scope, even if not explicitly shown or described herein.

[0161] [Incorporated by reference] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 388,909, filed July 13, 2022, and entitled "Template matching for multiple reference line intra prediction," which claims the benefit of priority to U.S. Provisional Application No. 18 / 207,587, filed June 8, 2023, and entitled "TEMPLATE MATCHING FOR MULTIPLE REFERENCE LINE INTRA PREDICTION," the disclosures of which are incorporated herein by reference in their entireties.

Claims

1. 1. A method of video decoding performed by a video decoder, comprising: receiving a video bitstream including a current coding block in a picture; determining that the current coding block is coded in a template-based multiple reference line intra-prediction (TMRL) mode; determining a combination of a reference line and an intra prediction mode for intra prediction of the current coding block, a first combination among the combinations including a first reference line among the reference lines and a first intra prediction mode among the intra prediction modes, the first reference line being a non-adjacent reference line; For the first combination, calculating template entry delta values ​​respectively associated with entries of a template of the current coding block based on the first reference line and the first intra-prediction mode, the template of the current coding block being in an adjacent reference line; calculating a first template matching cost value associated with the first combination by unequal weighting of the template entry delta values; ordering the combinations of the reference lines and the intra-prediction modes according to template matching cost values ​​respectively associated with the combinations, the template matching cost values ​​associated with the combinations including the first template matching cost value associated with the first combination; selecting a particular combination from the ordered combinations; reconstructing the current coding block according to the particular combination; The method according to claim 1,

2. The template for the current coding block includes as the entries neighboring samples to the current coding block, the method comprising: determining a reference template in the first reference line corresponding to the template of the current coding block based on the first intra-prediction mode, the reference template including reference entries each corresponding to the entries of the template; The method of claim 1.

3. calculating a first template entry delta value as a difference between a first entry among the entries of the template and a first reference entry among the reference entries of the reference template, the first reference entry corresponding to the first entry; determining a weight for weighting the first template entry delta value based on at least the first entry and the first reference entry; The method of claim 2 further comprising:

4. determining the weight for weighting the first template entry delta value based on the first entry, the first reference entry, and one or more neighboring reference entries of the first reference entry. The method according to claim 3.

5. determining the weight for weighting the first template entry delta value based on the first template entry delta value and one or more template entry delta values ​​associated with neighboring entries of the template. The method according to claim 3.

6. determining the weight for weighting the first template entry delta value based on at least a first gradient value at the first entry and / or a second gradient value at the first reference entry. The method according to claim 3.

7. calculating the first template matching cost value associated with the first combination based on a portion of the template. The method of claim 2.

8. calculating the first template matching cost value based on an upper template including upper neighboring samples of the current coding block; The method according to claim 7.

9. calculating the first template matching cost value based on a left template including left neighboring samples of the current coding block; The method according to claim 7.

10. determining a subset of the combinations based on a predefined range of the template matching cost values; selecting the particular combination from the subset of combinations; The method of claim 1 further comprising:

11. selecting a first subset of the combinations having a first template matching cost value within the predefined range; selecting one or more second combinations having a second template matching cost value outside the predefined range in response to a first number of combinations in the first subset of combinations being less than a specific number specifying a number requirement for available combinations, the first subset and the one or more second combinations forming the subset of combinations, the number of combinations in the subset being equal to the specific number; The method of claim 10 further comprising:

12. selecting a first subset of the combinations having a first template matching cost value within the predefined range; selecting the subset of combinations from the first subset of combinations in response to a first number of combinations in the first subset of combinations being greater than a specific number specifying a number requirement for available combinations, the number of combinations in the subset being equal to the specific number; The method of claim 10 further comprising:

13. The step of selecting the subset of the combinations comprises: selecting the subset of the combinations having the lowest template matching cost value. The method of claim 12.

14. The step of selecting the subset of the combinations comprises: selecting the subset of the combinations having closest reference lines to the template. The method of claim 12.

15. The template is within a second reference line, and the method further comprises: determining that a projection point of an entry of the template is at a fractional position between two samples in the first reference line; generating a reference entry associated with said entry of said template according to an interpolation based on at least said two samples; The method of claim 1 further comprising:

16. The step of selecting the specific combination comprises: and decoding an index from the bitstream together with the picture encoded in the bitstream, the index indicating the particular combination from the ordered combinations. The method of claim 1.

17. A method according to any one of claims 1 to 16, comprising: a processing circuit configured to execute software instructions which, when executed by the processing circuit, cause the processing circuit to carry out a method according to any one of claims 1 to 16. Device.

18. 1. A method of video encoding performed by a video encoder, comprising: determining a combination of reference lines and intra prediction modes for intra prediction of a current coding block in a picture, a first combination among the combinations including a first reference line among the reference lines and a first intra prediction mode among the intra prediction modes, the first reference line being a non-adjacent reference line; For the first combination, calculating template entry delta values ​​respectively associated with entries of a template of the current coding block based on the first reference line and the first intra-prediction mode, the template of the current coding block being in an adjacent reference line; calculating a first template matching cost value associated with the first combination by unequal weighting of the template entry delta values; sorting the combinations of the reference lines and the intra-prediction modes according to template matching cost values ​​respectively associated with the combinations, the template matching cost values ​​associated with the combinations including the first template matching cost value associated with the first combination; selecting a particular combination from the sorted combinations; encoding a signal indicative of said particular combination into a bitstream carrying encoded information of said picture; The method according to claim 1,

Images