Video decoding method and apparatus, and video encoding method

Abstract

The processing circuit receives a video bitstream including a current coding block in a picture coded in TMRL mode. The processing circuit determines a combination of a reference line and an intra-prediction mode for the current coding block. A first combination among the combinations includes a first reference line and a first intra-prediction mode. For the first combination, the processing circuit calculates template entry delta values ​​respectively associated with entries of a template of the current coding block based on the first reference line and the first intra-prediction mode, and calculates a first template matching cost value by unequal weighting of the template entry delta values. The processing circuit sorts the combinations according to the template matching cost values ​​respectively associated with the combinations, selects a particular combination from the sorted combinations, and reconstructs the current coding block according to the particular combination.

Description

[Technical Field]

[0001] This disclosure describes embodiments generally related to video coding. [Background technology]

[0002] The background description provided herein is intended to generally present the context for the present disclosure. The work of the presently named inventors, to the extent that that work is described in this background section, and any aspect of the description that may not otherwise qualify as prior art at the time of filing, is not admitted expressly or implicitly as prior art to the present disclosure.

[0003] Image / video compression can help transmit image / video files between different devices, storage devices, and networks with minimal quality loss. In some examples, video codec technology can compress video based on spatial and temporal redundancy. In an example, a video codec can use a technique called intra-prediction, which can compress images based on spatial redundancy. For example, intra-prediction can use reference data from the current picture being reconstructed for sample prediction. In another example, a video codec can use a technique called inter-prediction, which can compress images based on temporal redundancy. For example, inter-prediction can predict samples in a current picture from a previously reconstructed picture through motion compensation. Motion compensation is commonly represented by a motion vector (MV). Summary of the Invention

[0004] Aspects of the present disclosure provide methods and apparatuses for video encoding / decoding. In some examples, the apparatus for video decoding includes a receiving circuit and a processing circuit. The processing circuit receives a video bitstream including a current coding block in a picture and determines that the current coding block is coded in template-based multiple reference line intra prediction (TMRL) mode. The processing circuit determines combinations of reference lines and intra prediction modes for intra prediction of the current coding block in the picture. A first combination among the combinations includes a first reference line among the reference lines and a first intra prediction mode among the intra prediction modes. The first reference line is a non-adjacent reference line. For the first combination, the processing circuit calculates template entry delta values ​​respectively associated with entries of a template for the current coding block based on the first reference line and the first intra prediction mode, and calculates a first template matching cost value associated with the first combination by unequal weighting of the template entry delta values. The template for the current coding block is in the adjacent reference line. The processing circuit sorts the combinations of reference lines and intra-prediction modes according to template matching cost values ​​respectively associated with the combinations, the template matching cost values ​​associated with the combinations including a first template matching cost value associated with the first combination, and the processing circuit selects a particular combination from the sorted combinations and reconstructs the current coding block according to the particular combination.

[0005] In some examples, the template for the current coding block includes neighboring samples to the current coding block as entries, and the processing circuit determines a reference template in a first reference line corresponding to the template for the current coding block based on the first intra-prediction mode, and the reference template includes reference entries that respectively correspond to the entries of the template.

[0006] Further, in some examples, the processing circuitry calculates a first template entry delta value as a difference between a first entry among the entries of the template and a first reference entry among the reference entries of the reference template, the first reference entry corresponding to the first entry. The processing circuitry determines a weight for weighting the first template entry delta value based on at least the first entry and the first reference entry.

[0007] In some examples, the processing circuit determines a weight for weighting the first template entry delta value based on the first entry, the first reference entry, and one or more neighboring reference entries of the first reference entry.

[0008] In some examples, the processing circuit determines a weight for weighting the first template entry delta value based on the first template entry delta value and one or more template entry delta values ​​associated with neighboring entries of the template.

[0009] In some examples, the processing circuit determines a weight for weighting the first template entry delta value based on at least the first gradient value at the first entry and / or the second gradient value at the first reference entry.

[0010] In some examples, the processing circuitry calculates a first template matching cost value associated with the first combination based on a portion of the template.

[0011] In an example, the processing circuit calculates a first template matching cost value based on an upper template that includes upper neighboring samples of the current coding block.

[0012] In an example, the processing circuit calculates a first template matching cost value based on a left template that includes left-proximate samples of the current coding block.

[0013] In some examples, the processing circuitry determines a subset of the combinations based on a predefined range of template matching cost values ​​and selects a particular combination from the subset of the combinations.

[0014] In an example, the processing circuit selects a first subset of the combinations having a first template matching cost value within a predefined range, and in response to a number of first combinations in the first subset of combinations being less than a specific number specifying a number requirement for the number of available combinations, selects one or more second combinations having a second template matching cost value outside the predefined range, the first subset and the one or more second combinations forming the subset of the combinations, and the number of combinations in the subset equals the specific number.

[0015] In another example, the processing circuit selects a first subset of the combinations having a first template matching cost value within a predetermined range, and selects the subset of combinations from the first subset of combinations in response to a first number of combinations in the first subset being greater than a specific number specifying a number requirement for the number of available combinations, where the number of combinations in the subset is equal to the specific number. For example, the processing circuit selects the subset of combinations having the lowest template matching cost value or the subset of combinations having the closest reference line to the template.

[0016] In some examples, the template is within a second reference line, and the processing circuitry determines that a projection point of an entry of the template is at a fractional position between two samples in the first reference line, and generates a reference entry associated with the entry of the template according to interpolation based on at least the two samples.

[0017] In some examples, to select a particular combination, the processing circuitry decodes an index from the bitstream along with the picture encoded in the bitstream, the index indicating the particular combination from the sorted combinations.

[0018] Aspects of the disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to perform a method of video decoding.

[0019] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a schematic diagram of an exemplary block diagram of a communication system (100). [Figure 2] FIG. 2 is a schematic diagram of an exemplary block diagram of a decoder. [Figure 3] FIG. 2 is a schematic diagram of an exemplary block diagram of an encoder. [Figure 4] 1 shows a diagram of intra-prediction modes in some examples. [Figure 5] 1 shows a diagram of intra-prediction modes in some examples. [Figure 6] 10 shows pseudocode for generating the most probable mode list in some examples. [Figure 7] 10A-10C show diagrams of multiple reference lines in some examples. [Figure 8] 1 shows a diagram illustrating a template-based multiple reference line intra-prediction mode in some examples. [Figure 9] 10 shows a table of binarization processes and codewords in template-based multiple reference line intra-prediction mode in some examples. [Figure 10] 10A-10C show diagrams of templates of reference lines in some examples. [Figure 11]10A-10C show other views of templates of reference lines in some examples. [Figure 12] 1 shows a flowchart illustrating a process according to some embodiments of the present disclosure. [Figure 13] 10 shows a flowchart illustrating another process according to some embodiments of the present disclosure. [Figure 14] FIG. 1 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0021] 1 depicts a video encoder and a video decoder in a streaming environment as an example application of the disclosed subject matter. The disclosed subject matter can be similarly applicable to other video-enabled applications including, for example, video conferencing, digital TV, streaming services, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0022] The video processing system 100 may include a capture subsystem 113, which may include, for example, a video source 101, such as a digital camera, that generates a stream of uncompressed video pictures 102. In the example, the stream of video pictures 102 includes samples captured by the digital camera. The stream of video pictures 102 is represented by a bold line to emphasize its high data volume compared to the encoded video data 104 (or coded video bitstream) and may be processed by an electronic device 120 that includes a video encoder 103 coupled to the video source 101. The video encoder 103 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data 104 (or coded video bitstream) is represented by a thin line to emphasize its lower data volume compared to the stream of video pictures 102 and may be stored on a streaming server 105 for future use. One or more streaming client subsystems, such as the client subsystems 106 and 108 of FIG. 1, can access the streaming server 105 to retrieve copies 107 and 109 of the encoded video data 104. The client subsystem 106 may include a video decoder 110, for example, in an electronic device 130. The video decoder 110 decodes the incoming copy 107 of the encoded video data and generates an outgoing stream 111 of video pictures that can be rendered on a display 112 (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data 104, 107, and 109 (e.g., a video bitstream) may be encoded according to a particular video coding / compression standard. An example of such a standard is ITU-T Recommendation H.265.In an example, the video coding standard under development is commonly known as Versatile Video Coding (VVC), and the disclosed subject matter may be used in connection with VVC.

[0023] It should be noted that electronic devices 120 and 130 may include other components (not shown). For example, electronic device 120 may include a video decoder (not shown), and electronic device 130 may similarly include a video encoder (not shown).

[0024] 2 shows an example block diagram of a video decoder (210). The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., a receiving circuit). The video decoder (210) may be used in place of the video decoder (110) in the example of FIG. 1.

[0025] The receiver (231) may receive one or more coded video sequences to be decoded by the video decoder (210). In an embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device storing the coded video data. The receiver (231) may receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective using entities (not shown). The receiver (231) may separate the coded video sequences from other data. To combat network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter "parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). In others, it can be external to the video decoder (210) (not shown). In still other applications, there can be a buffer memory (not shown) external to the video decoder (210), for example, to combat network jitter, plus another buffer memory (215) within the video decoder (210), for example, to manipulate playback timing. When the receiver (231) is receiving data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (215) may not be required or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (215) may be required, but it can be relatively large and advantageously adaptively sized, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (210).

[0026] The video decoder (210) may include a parser (220) for reconstructing symbols (221) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (210) and, potentially, information for controlling a rendering device, such as a render device (212) (e.g., a display screen) that is not an essential part of the electronic device (230) but may be coupled to the electronic device (230) as shown in FIG. 2. Control information for the rendering device may take the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, context-dependent or non-context-dependent arithmetic coding, etc. The parser (220) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (220) may also extract information from the coded video sequence information, such as transform coefficients, quantization parameter values, motion vectors, etc.

[0027] The parser (220) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to generate symbols (221).

[0028] The reconstruction of the symbols (221) can have many different units depending on the type of coded video picture or portion thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. Which units are included and how may be controlled by subgroup control information parsed by the parser (220) from the coded video sequence. The flow of such subgroup control information between the parser (220) and the following units is not shown for clarity.

[0029] Beyond the functional blocks already described, the video decoder (210) may be conceptually subdivided into a number of functional units, which are described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0030] The first unit is a scalar / inverse transform unit (251), which receives quantized transform coefficients as symbols (221) from the parser (220) along with control information including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. The scalar / inverse transform unit (251) can output blocks containing sample values ​​that can be input to an aggregator (255).

[0031] In some cases, the output samples of the scaler / inverse transformer (251) may relate to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the block being reconstructed using surrounding, already reconstructed information fetched from the current picture buffer (258). The current picture buffer (258), for example, buffers a partially reconstructed and / or fully reconstructed current picture. The aggregator (255), in some cases, adds, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (252) to the output sample information provided by the scaler / inverse transformer unit (251).

[0032] In other cases, the output samples of the scalar / inverse transform unit (251) may relate to an inter-coded, and potentially motion-compensated, block. In such cases, the motion-compensated prediction unit (253) may access the reference picture memory (257) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (221) related to the block, the samples may be added by the aggregator (255) to the output of the scalar / inverse transform unit (251) (in this case, referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (257) from which the motion-compensated prediction unit (253) fetches prediction samples may be controlled by motion vectors available to the motion-compensated prediction unit (253), for example, in the form of symbols (221), which may have X, Y, and reference picture components. Motion compensation can also include interpolation of sample values ​​fetched from the reference picture memory (257) when sub-sample accurate motion vectors are used, as well as motion vector prediction mechanisms.

[0033] The output samples of the aggregator (255) can undergo various loop filtering techniques in the loop filter unit (256). Video compression techniques can include in-loop filtering techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression can also respond to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and can also respond to previously constructed loop-filtered sample values.

[0034] The output of the loop filter unit (256) can be a sample stream that can be output to a render device (212) and further stored in a reference picture memory (257) for use in future inter-picture prediction.

[0035] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) can become part of the reference picture memory (257), and any unused current picture buffer can be reallocated before beginning reconstruction of a subsequent coded picture.

[0036] The video decoder (210) may perform decoding operations according to a given video compression technology or standard, such as ITU-T Recommendation H.265. A coded video sequence may conform to the syntax prescribed by the video compression technology or standard in use, in the sense that the coded video sequence conforms to both the syntax of the video compression technology or standard and a profile documented in the video compression technology or standard. Specifically, a profile may select specific tools from all tools available in the video compression technology or standard as the only tools available for use under that profile. Compliance also requires that the complexity of the coded video sequence be within the boundaries defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further constrained through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0037] In embodiments, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may also be included as part of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0038] 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmitting circuit). The video encoder (303) can be used in place of the video encoder (103) in the example of FIG. 1.

[0039] The video encoder (303) may receive video samples from a video source (301) (which is not part of the electronic device (320) in the example of FIG. 3) that may capture video images to be coded by the video encoder (303). In other examples, the video source (301) is part of the electronic device (320).

[0040] The video source (301) may provide a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCB, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (301) may be a storage device storing prepared video. In a video conferencing system, the video source (301) may be a camera capturing local image information as a video sequence. The video data may be provided as multiple individual pictures that, when viewed in sequence, impart motion. The pictures themselves may be organized as a spatial array of pixels, each of which may have one or more samples depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. This specification will focus on samples hereafter.

[0041] According to an embodiment, the video encoder (303) may code and compress pictures of a source video sequence into a coded video sequence (343) in real time or under any other time constraints as needed. Imposing an appropriate coding rate is one function of the controller (350). In some embodiments, the controller (350) controls and is operatively coupled to other functional units, as described below. The coupling is not shown for clarity. Parameters set by the controller (350) may include parameters related to rate control (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. The controller (350) may be configured with other appropriate functions related to the video encoder (303) optimized for a particular system design.

[0042] In some embodiments, the video encoder (303) is configured to operate in a coding loop. As an overly simplified description, in an example, the coding loop can include a source coder (330) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to generate sample data in a manner similar to what a (remote) decoder would also generate. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Because decoding of the symbol stream produces bit-exact results independent of the location (local or remote) of the decoder, the contents of the reference picture memory (334) are also bit-perfect between the local and remote encoders. In other words, the predictive portion of the encoder "sees" exactly the same sample values ​​as the decoder would "see" when using prediction during decoding. This basic principle of reference picture synchronicity (and the resulting drift when synchronicity cannot be maintained, for example due to channel errors) is also used in several related techniques.

[0043] The operation of the "local" decoder (333) can be the same as a "remote" decoder, such as the video decoder (210), already described in detail above in conjunction with Figure 2. Referring also momentarily to Figure 2, however, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333), given the availability of symbols and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (345) and parser (220) can be lossless.

[0044] In embodiments, decoder techniques, with the exception of parsing / entropy decoding, present in a decoder are present in the corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on the operation of the decoder. Descriptions of encoder techniques may be omitted, as they are the inverse of the decoder techniques described generically. To the extent specified, more detailed descriptions are provided below.

[0045] In operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as "reference pictures." In this manner, the coding engine (332) codes differences between pixel blocks of the input picture and pixel blocks of the reference pictures that may be selected as predictive references for the input picture.

[0046] The local video decoder (333) may decode coded video data of pictures that may be designated as reference pictures based on symbols generated by the source coder (330). The operation of the coding engine (332) may advantageously be a lossy process. When the coded video data is decoded by a video decoder (not shown in FIG. 3), the reconstructed video sequence is typically a copy of the source video sequence, with some errors. The local video decoder (333) may replicate the decoding process that may be performed by the video decoder on the reference pictures, causing the reconstructed reference pictures to be stored in a reference picture cache (334). In this way, the video encoder (303) may locally store copies of reconstructed reference pictures that have content in common with reconstructed reference pictures that would be obtained by a far-end video decoder (without transmission errors).

[0047] The predictor (335) may perform a prediction search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) may search the reference picture memory (334) for specific metadata, such as reference picture motion vectors, block shapes, or sample data (as candidate reference pixel blocks) that can serve as suitable prediction references for the new picture. The predictor (335) may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (334).

[0048] The controller (350) may manage the coding operations of the source coder (330), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0049] The output of all of the above functional units may undergo entropy coding in an entropy coder (345), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0050] The transmitter (340) may buffer the coded video sequence produced by the entropy coder (345) to prepare it for transmission over a communication channel (360), which may be a hardware / software link to a storage device that stores the coded video data. The transmitter (340) may merge the coded video data from the video coder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0051] The controller (350) may manage the operation of the video encoder (303). During coding, the controller (350) may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0052] An Intra Picture (I-picture) may be a picture that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow various types of Intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of such variations of I-pictures and their respective applications and characteristics.

[0053] A Predictive Picture (P-picture) may be a picture that can be coded and decoded by intra- or inter-prediction using at most one motion vector and reference index to predict the sample values ​​of each block.

[0054] A Bi-directionally Predictive Picture (B-picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0055] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to each picture of the blocks. For example, blocks of an I-picture may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P-picture may be predictively coded by spatial prediction or temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be predictively coded by spatial prediction or temporal prediction with reference to one or two previously coded reference pictures.

[0056] The video encoder (303) may perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Recommendation H.265. During its operation, the video encoder (303) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax defined by the video coding technique or standard being used.

[0057] In embodiments, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0058] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In an example, a particular picture being encoded / decoded, called the current picture, is partitioned into blocks. If a block in the current picture is similar to a reference block in a previously coded reference picture in the video that is still buffered, the block in the current picture may be coded by a vector called a motion vector. A motion vector points to a reference block within a reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0059] In some embodiments, bi-prediction techniques may be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, e.g., a first reference picture and a second reference picture, both of which precede the current picture in decoding order (but may be past and future, respectively, in display order) in the video. A block in the current picture may be coded with a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. The block is predictable by a combination of the first and second reference blocks.

[0060] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0061] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one CU of 64x64 pixels, four CUs of 32x32 pixels, or 16 CUs of 16x16 pixels. In an example, each CU is analyzed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. A CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In an embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0062] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In some embodiments, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0063] Aspects of the present disclosure provide techniques for template matching for multiple reference line intra prediction.

[0064] FIG. 4 shows a diagram of intra-prediction modes in some examples, such as HEVC. For example, HEVC uses a total of 35 intra-prediction modes (e.g., mode 0 to mode 34). Among the 35 intra-prediction modes, some modes are directional modes and some modes are non-directional modes. In some examples, mode 0 and mode 1 are non-directional modes, e.g., mode 0 is planar mode and mode 1 is DC mode. Furthermore, mode 2 to mode 34 can be directional modes, e.g., mode 10 is horizontal mode, mode 26 is vertical mode, and modes 2, 18, and 34 are diagonal modes. The value of a sample in a coding block is determined according to neighboring reference samples in the same picture and the intra-prediction mode of the coding block. In an example, in DC mode, an average value is calculated by averaging reference samples in the same picture and can be used for flat surfaces. In another example, in planar mode, the value of each sample in a coding block is calculated assuming an amplitude surface with smooth horizontal and vertical gradients derived from the boundary samples of neighboring blocks. In some examples, the reference samples include neighboring samples in the row immediately above the coding block and / or neighboring samples in the column immediately to the left of the coding block.

[0065] In some examples, the intra-prediction mode is signaled based on a list of Most Probable Mode(s) (MPM). For example, an MPM list is determined for a coding block. In an example, the MPM list includes three modes from 35 intra-prediction modes. In that case, when a particular intra-prediction mode of a coding block is one of the three modes in the MPM list, an index indicating that one of the three modes is used for signaling. When a particular intra-prediction mode of a coding block is not one of the three modes in the MPM list, an index indicating one of the remaining modes (32 modes) is used for signaling. In some examples, the MPM list may include other appropriate numbers of modes, such as 6, 10, etc.

[0066] However, any other suitable number of intra-prediction modes may be used.

[0067] 5 shows a diagram of intra-prediction modes in some examples, such as VVC. In some examples, VVC can use a total of 95 intra-prediction modes, such as mode -14 to mode 80. Among the 95 intra-prediction modes, mode 0 is the planar mode, mode 1 is the DC mode, mode 18 is the horizontal mode, mode 50 is the vertical mode, and modes 2, 34, and 66 are diagonal modes. Modes -1 to -14 and modes 64 to 80 are referred to as Wide-Angle Intra Prediction (WAIP) modes in some examples.

[0068] In some examples, to code an intra-mode (also called an intra-prediction mode) of a coding block (e.g., a luma block, a chroma block of a coding unit), a most probable mode (MPM) list of size 3 is constructed based on the intra-modes of neighboring blocks of the coding block. The MPM list may be referred to as an MPM list or a primary MPM list. If the intra-mode of the coding block is not from the MPM list, a flag is signaled to indicate whether the intra-mode belongs to a selected mode in the MPM list.

[0069] 6 shows pseudo codes for generating an MPM list in some examples. In the example of FIG. 6, leftIntraDir is used to indicate the intra mode of the left block (adjacent to the current coding block), and aboveIntraDir is used to indicate the intra mode of the above block (adjacent to the current coding block). In an example, if the left or above block is not currently available, leftIntraDir or aboveIntraDir may be set to DC_IDX (the index of the DC mode). In some examples, the variables "offset" and "mod" in FIG. 6 are constant values ​​set to 29 and 32, respectively.

[0070] In some related examples, reference samples for intra-prediction of a coding block are located immediately adjacent to the coding block, and the adjacent portion of the coding block formed by the reference samples may be referred to as a reference line.

[0071] According to aspects of the present disclosure, a technique called multi-line intra-prediction can use more reference lines for intra-prediction. In some examples, the encoder determines and signals which reference lines are used to generate the intra-predictor. In examples, the reference line index is signaled before the intra-prediction mode, and only the most probable mode is allowed if a non-zero reference line index is signaled.

[0072] Figure 7 shows a diagram of multiple reference lines in some examples. Four reference lines for a coding block (710) are shown in Figure 7. The first reference line (720), also referred to as reference line 0, includes a row of reference samples immediately above the coding block (710) and a column of reference samples immediately to the left of the coding block (710). The second reference line (730), also referred to as reference line 1, includes a row of reference samples immediately above the first reference line (720) and a column of reference samples immediately to the left of the first reference line (720). The third reference line (740), also referred to as reference line 2, includes a row of reference samples immediately above the second reference line (730) and a column of reference samples immediately to the left of the second reference line (730). The fourth reference line (750), also referred to as reference line 3, includes a row of reference samples immediately above the third reference line (740) and a column of reference samples immediately to the left of the third reference line (740). Furthermore, each reference line consists of six segments, as indicated by segments A through F, with the top-left reference sample indicated by T. In some examples, segments A and F are padded with the nearest samples from segments B and F, respectively.

[0073] In some examples, a technique called template-based multiple reference line intra-prediction (TMRL) mode may be used. The TMRL mode includes four aspects.

[0074] In some examples, for the first aspect of the TMRL mode, two candidate lists are used: one for the reference line and one for the intra-prediction mode. In the examples, an extended reference line candidate list and an intra-prediction mode candidate list are used. For example, the extended reference line candidate list includes {1, 3, 5, 7, 12}. The restriction on the top row of the CTU remains unchanged. In the examples, the size of the intra-prediction mode candidate list is 10. The structure of the intra-prediction mode candidate list is similar to that of MPM. The difference between the intra-prediction mode candidates for MPM is that planar mode is excluded from the intra-prediction mode candidate list.

[0075] In some examples, for the second aspect of the TMRL mode, a TMRL combination list is configured. In the example, combinations of extended reference lines and allowed intra prediction modes may be configured for a coding block. Each combination includes a reference line selected from the extended reference line candidate list and an intra prediction mode selected from the intra prediction mode candidate list. In the example, five reference lines are in the extended reference line candidate list for the coding block, and ten intra prediction modes are in the intra prediction mode candidate list for the coding block, so in this case, 50 combinations may be configured for the coding block.

[0076] In some examples, the extended reference lines start from reference line 1, and the area covered by reference line 0 is used for template matching.

[0077] 8 shows a diagram illustrating a template-based multiple reference line intra prediction mode in some examples. FIG. 8 shows a current coding block (810) along with reference lines such as reference line 0 (820), reference line 1 (830), reference line 2 (840), and reference line 3 (850). Based on the area covered by reference line 0 (820), the template area may be defined as a top template, a left template, a template including a top template and a left template, etc. However, in some examples, the template area may already be reconstructed at the time of coding the current coding block (810).

[0078] In some examples, for each of the 50 combinations of the extended reference line and the allowed intra-prediction mode, a projection of the template area may be generated according to the reference line and the intra-prediction mode of the combination. Furthermore, a template matching cost, such as a sum of absolute difference (SAD) cost, is calculated between the projection of the template area and the template area. In the example, the first 20 combinations with the lowest cost are pruned from the combinations in ascending order of cost to form a combination candidate list.

[0079] In some examples, the third aspect of TMRL relates to TMRL signaling. In some examples, such as ECM or VTM, a multiple reference line (MRL) is combined with a non-planar intra-prediction mode from the primary MPM list. When an extended reference line is selected, up to six context-coded bins and three equi-probable (EP) bins can be used to represent the combination of the MRL and MPM index. In some examples, in TMRL modes, truncated Golomb-Rice coding with a divisor of four is used to code the selected combination from the combination list.

[0080] FIG. 9 shows a table of binarization processes and codewords in TMRL mode in some examples.

[0081] In some examples, a fourth aspect of TMRL relates to encoder-side modifications. In examples, encoder-side modifications are performed to further improve coding efficiency. In examples, for intra blocks larger than 8x8, more full-rate distortion optimization (RDO) may be allowed for a selected intra-prediction mode after comparing the Sum of Absolute Transformed Differences (SATD) cost with a threshold.

[0082] Some aspects of the present disclosure provide techniques for TMRL.

[0083] In the following description, the line index of the nearest reference line of a coding block is 0, and the nearest reference line is also called the zero reference line or adjacent reference line, and the other reference lines are called non-zero reference lines or non-adjacent reference lines.

[0084] In the following description, a template may refer to an upper neighboring sample, a left neighboring sample, a right neighboring sample, and a lower neighboring sample within a particular reference line of a current coding block.

[0085] Figure 10 shows diagrams of reference line templates in some examples. In the example of Figure 10, the template of reference line 0 is indicated by (1020), also called the template of the current coding block (1010), the template of reference line 1 is indicated by (1030), and the template of reference line 2 is indicated by (1040).

[0086] FIG. 11 shows another diagram of a template of a reference line in some examples. In the example of FIG. 11, the top template of reference line 0 is indicated by (1125) and is also referred to as the top template of the current coding block (1110). In the example, the top template (1125) may be projected onto a non-zero reference line according to a prediction direction (1135) that may be indicated by an intra-prediction mode. The projected template in the non-zero reference line may also be referred to as a reference template in some examples. For example, based on the prediction direction (1135), the top template (1125) is projected onto the top template (1145) of reference line 2. The top template (1145) is a projection of the top template (1125) in reference line 2 according to the prediction direction.

[0087] In the example of FIG. 11 , the left template of reference line 0 is indicated by (1155) and is also referred to as the left template of the current coding block (1110). In the example, the left template (1155) may be projected onto a non-zero reference line according to the prediction direction (1165). The projected template in the non-zero reference line is also referred to as a reference template in some examples. For example, based on the prediction direction (1165), the left template (1155) is projected onto the left template (1175) of reference line 2. The left template (1175) is a projection of the left template (1155) according to the prediction direction.

[0088] According to some aspects of the present disclosure, weighting is applied in addition to the TMRL method. In some examples, the template for reference line 0 is denoted as template0, and the template for reference line i (where i is not zero) (also referred to as the reference template) is denoted as template1. i and the difference between two templates at entry index x (e.g., sample position x in the template for reference line 0) is template delta It can be denoted as (x) and calculated according to equation (1):

number

[0089] In accordance with some aspects of the present disclosure, weighting is performed to calculate the cost of a reference template. delta applies to each entry in

[0090] In some examples, the weighting is performed by applying the weighting to the template before calculating the template matching cost, e.g., according to equation (2). delta applied to each entry in:

number

[0091] In an embodiment, the weight value of entry x (also referred to as sample position x) depends on the sample values ​​in the template of reference line 0 and the reference template of reference line i, and on the sample values ​​of neighboring positions relative to entry x.

[0092] In other embodiments, the weight value of entry x depends on the difference between sample values ​​in the template of reference line 0 and the reference template of reference line i located at position x and adjacent positions, e.g., position x-1 and / or position x+1.

[0093] In another embodiment, the weight value of entry x depends on the gradient value of the sample in the template of reference line 0 and the reference template of reference line i located at position x.

[0094] In some examples, depending on the prediction direction, portions of the template are used to calculate the template matching cost.

[0095] In an embodiment, if the prediction direction is, for example, vertical, the reference sample is derived from the adjacent reconstructed sample above, and for example, the top template may be used to calculate the cost. In an example, the index of the vertical direction (vertical intra-prediction mode) is 50, and the absolute difference of the index of the prediction direction (intra-prediction mode) relative to the index of the vertical direction is less than a threshold (for example, 6), in which case the top template may be used to calculate the template matching cost.

[0096] In other embodiments, if the prediction direction is, for example, horizontal, i.e., the reference sample comes from the adjacent reconstructed sample on the left, for example, a left template is used. In the example, the horizontal index (horizontal intra-prediction mode) is 18, and the absolute difference of the index of the prediction direction (intra-prediction mode) relative to the horizontal index is less than a threshold value (e.g., 6), then the left template can be used to calculate the template matching cost.

[0097] In other embodiments, when the prediction direction is between horizontal and vertical, i.e., the reference samples come from neighboring reconstructed samples on both the upper and left sides, e.g., the top template and the left template are used to calculate the template matching cost.

[0098] In some examples, a smoothing process may be applied to the reference samples after they are reconstructed, and in some embodiments, the reconstructed reference samples before the smoothing process are used to calculate the template matching cost.

[0099] In some embodiments, after calculating template matching costs associated with combinations of candidate reference lines and intra-prediction modes, a template matching cost within a predefined range [a, b] is selected, and the combination of candidate reference lines and intra-prediction modes associated with the selected template matching cost within the predefined range [a, b] is determined to be available for the current coding block. In the example, both a and b are values ​​greater than 0.

[0100] In some embodiments, if the number of combinations of candidate reference lines and intra-prediction modes whose template matching costs are within [a, b] is less than N, where N indicates the required number of available combinations, combinations of candidate reference lines and intra-prediction modes whose template matching costs are outside the range of [a, b] may be added to the available combinations so that the number of available combinations is N.

[0101] In some embodiments, if the number of combinations of candidate reference lines and intra-prediction modes with template matching costs within [a, b] is greater than N, where N indicates the required number of available combinations, only the first N combinations of candidate reference lines and intra-prediction modes that are closer to reference line 0 (i.e., have lower template matching costs) are selected as available combinations.

[0102] According to aspects of the present disclosure, to calculate a template matching cost between two reference lines, such as reference line i and reference line j, interpolation may be applied by projecting integer samples of the template in reference line i onto reference line j, or by projecting integer samples of the template in reference line j onto reference line i.

[0103] In some instances, if the projection point is at a fractional position between two integer samples in the reference line, interpolation is applied to generate the fractional sample value.

[0104] In some examples, the interpolation method may be the same as that used in the interpolation process in angular (directional) intra prediction.

[0105] In some examples, the interpolation method may be different compared to the interpolation used in angular (directional) intra prediction, e.g., bilinear interpolation, 4-tap cubic interpolation, nearest integer, etc.

[0106] 12 shows a flowchart illustrating a process (1200) according to an embodiment of the present disclosure. The process (1200) may be used in a video encoder. In various embodiments, the process (1200) is performed by a processing circuit, such as a processing circuit performing the functions of the video encoder (203), a processing circuit performing the functions of the video encoder (303), etc. In some embodiments, the process (1200) is implemented by software instructions, such that the processing circuit performs the process (1200) when it executes the software instructions. The process (1200) begins at (S1201) and proceeds to (S1210).

[0107] At (S1210), a combination of a reference line and an intra prediction mode for intra prediction of a current coding block in a picture is determined. In an example, it is determined that the current coding block is coded by intra prediction. A first candidate list of reference lines is determined, and a second candidate list of intra prediction modes is determined. Then, combinations of reference lines and intra prediction modes are determined. Each combination among the combinations includes a reference line among the reference lines and an intra prediction mode among the intra prediction modes. For example, a first combination among the combinations includes a first reference line among the reference lines and a first intra prediction mode among the intra prediction modes.

[0108] At (S1220), for a first combination, template entry delta values ​​respectively associated with entries of the template of the current coding block are calculated based on the first reference line and the first intra prediction mode.

[0109] At (S1230), a first template matching cost value associated with the first combination is calculated by unequal weighting of the template entry delta values.

[0110] At (S1240), the combinations of reference lines and intra-prediction modes are sorted according to template matching cost values ​​respectively associated with the combinations, the template matching cost values ​​associated with the combinations including a first template matching cost value associated with a first combination.

[0111] At (S1250), a particular combination is selected from the sorted combinations, for example, the particular combination is selected based on rate-distortion optimization.

[0112] At (S1260), a signal indicating the particular combination is coded in a bitstream carrying coded information for the picture.

[0113] In some embodiments, the template for the current coding block includes neighboring samples to the current coding block (e.g., in reference line 0) as entries. A reference template in a first reference line corresponding to the template for the current coding block is determined based on the first intra-prediction mode, and the reference template includes reference entries that respectively correspond to the entries of the template.

[0114] In some embodiments, the first template entry delta value is calculated as a difference between a first entry among the entries of the template and a first reference entry among the reference entries of the reference template, and in some examples, a weight for weighting the first template entry delta value is determined based on at least the first entry and the first reference entry.

[0115] In some examples, the weight for weighting the first template entry delta value is determined based on the first entry, the first reference entry, and one or more neighboring reference entries of the first reference entry.

[0116] In some examples, the weight for weighting the first template entry delta value is determined based on the first template entry delta value and one or more template entry delta values ​​associated with neighboring entries of the template.

[0117] In some examples, the weight for weighting the first template entry delta value is determined based on at least a first gradient value at the first entry and / or a second gradient value at the first reference entry.

[0118] In some examples, some of the neighboring samples in reference line 0 are used to calculate the template matching cost value. In some examples, the template matching cost value is calculated according to the top template (e.g., including the top neighboring samples) of the current coding block. In other examples, the template matching cost value is calculated according to the left template (e.g., including the left neighboring samples) of the current coding block.

[0119] In some embodiments, a subset of the combinations is determined based on a predefined range of template matching cost values, and a particular combination is selected from the subset of the combinations.

[0120] In some examples, a first subset of the combinations having first template matching cost values ​​within a predefined range is determined. In examples, in response to a number of first combinations in the first subset of the combinations being less than a particular number specifying a number requirement for the number of available combinations, one or more second combinations having second template matching cost values ​​outside the predefined range are selected, the first subset and the one or more second combinations forming the subset of the combinations, and the number of combinations in the subset equals the particular number.

[0121] In another example, in response to a first number of combinations in the first subset of combinations being greater than a specific number specifying a number requirement for the number of available combinations, the subset of combinations is selected from the first subset of combinations, and the number of combinations in the subset is equal to the specific number. For example, the subset of combinations having the lowest template matching cost value is selected from the first subset. In another example, the subset of combinations having the closest reference line to the template is selected from the first subset.

[0122] In some examples, the template is in the second reference line, the projection point of the template entry is at a fractional position between two samples in the first reference line, and the reference entry associated with the template entry is generated according to interpolation based on at least the two samples.

[0123] The process then proceeds to (S1299) and ends.

[0124] Process 1200 may be adapted as appropriate. Steps of process 1200 may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0125] 13 shows a flowchart illustrating a process (1300) according to an embodiment of the present disclosure. The process (1300) may be used in a video decoder. In various embodiments, the process (1300) is performed by a processing circuit, such as a processing circuit performing the functions of the video decoder (110), a processing circuit performing the functions of the video decoder (210), etc. In some embodiments, the process (1300) is implemented by software instructions, such that the processing circuit performs the process (1300) when it executes the software instructions. The process (1300) begins at (S1301) and proceeds to (S1310).

[0126] At (S1310), a combination of reference lines and intra prediction modes for intra prediction of a current coding block in a picture is determined. In an example, a first candidate list of reference lines is determined, and a second candidate list of intra prediction modes is determined. Then, a combination of reference lines and intra prediction modes is determined. Each combination of the combinations includes a reference line among the reference lines and an intra prediction mode among the intra prediction modes. For example, a first combination of the combinations includes a first reference line among the reference lines and a first intra prediction mode among the intra prediction modes.

[0127] At (S1320), for a first combination, template entry delta values ​​respectively associated with entries of the template of the current coding block are calculated based on the first reference line and the first intra prediction mode.

[0128] At (S1330), a first template matching cost value associated with the first combination is calculated by unequal weighting of the template entry delta values.

[0129] At (S1340), the combinations of the reference lines and the intra-prediction modes are sorted according to template matching cost values ​​respectively associated with the combinations, the template matching cost values ​​associated with the combinations including a first template matching cost value associated with a first combination.

[0130] At (S1350), a particular combination is selected from the sorted combinations. In some examples, a signal indicating an index of a particular combination among the sorted combinations is decoded, and the particular combination is selected based on the decoded signal.

[0131] At (S1360), the current coding block is reconstructed according to the specific combination including the reference line and the intra prediction mode.

[0132] In some embodiments, the template for the current coding block includes neighboring samples to the current coding block as entries, and a reference template in a first reference line corresponding to the template for the current coding block is determined based on a first intra-prediction mode, and the reference template includes reference entries respectively corresponding to the entries of the template.

[0133] In some embodiments, the first template entry delta value is calculated as a difference between a first entry among the entries of the template and a first reference entry among the reference entries of the reference template, the first reference entry corresponding to the first entry. In some examples, a weight for weighting the first template entry delta value is determined based on at least the first entry and the first reference entry.

[0134] In some examples, the weight for weighting the first template entry delta value is determined based on the first entry, the first reference entry, and one or more neighboring reference entries of the first reference entry.

[0135] In some examples, the weight for weighting the first template entry delta value is determined based on the first template entry delta value and one or more template entry delta values ​​associated with neighboring entries of the template.

[0136] In some examples, the weight for weighting the first template entry delta value is determined based on at least a first gradient value at the first entry and / or a second gradient value at the first reference entry.

[0137] In some examples, a portion of the neighboring samples in reference line 0 is used to calculate the template matching cost value. In some examples, the template matching cost value is calculated according to the top template (e.g., including the top neighboring samples) of the current coding block. In other examples, the template matching cost value is calculated according to the left template (e.g., including the left neighboring samples) of the current coding block.

[0138] In some embodiments, the subset of combinations is determined based on a predefined range of template matching cost values, and a particular combination is selected from the subset of combinations.

[0139] In some examples, a first subset of the combinations having first template matching cost values ​​within a predefined range is determined. In examples, in response to a number of first combinations in the first subset of the combinations being less than a particular number specifying a number requirement for the number of available combinations, one or more second combinations having second template matching cost values ​​outside the predefined range are selected, the first subset and the one or more second combinations forming the subset of the combinations, and the number of combinations in the subset equals the particular number.

[0140] In another example, in response to a first number of combinations in the first subset of combinations being greater than a specific number specifying a number requirement for the number of available combinations, the subset of combinations is selected from the first subset of combinations, and the number of combinations in the subset is equal to the specific number. For example, the subset of combinations having the lowest template matching cost value is selected from the first subset. In another example, the subset of combinations having the closest reference line to the template is selected from the first subset.

[0141] In some examples, the template is in the second reference line, the projection point of the template entry is at a fractional position between two samples in the first reference line, and the reference entry associated with the template entry is generated according to an interpolation based on at least the two samples.

[0142] The process then proceeds to (S1399) and ends.

[0143] Process 1300 may be adapted as appropriate. Steps of process 1300 may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0144] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 14 illustrates a computer system (1400) suitable for implementing certain embodiments of the disclosed subject matter.

[0145] Computer software can be coded in any suitable machine code or computer language that can be subjected to mechanisms such as assembly, compilation, linking, etc. to generate code containing instructions that can be executed by one or more central processing units (CPUs), graphics processing units (GPUs), etc. directly or through interpretation, microcode execution, etc.

[0146] The instructions may be executable by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, devices for the Internet of Things, and the like.

[0147] 14 for computer system 1400 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components described in the exemplary embodiment of computer system 1400.

[0148] The computer system 1400 may include certain human interface input devices. Such human interface input devices may respond to input by one or more users through, for example, tactile input (e.g., keyboard, swipe, dataglove motion), audio input (e.g., voice, claps), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0149] The input human interface devices may include one or more of a keyboard (1401), a mouse (1402), a trackpad (1403), a touchscreen (1410), a data glove (not shown), a joystick (1405), a microphone (1406), a scanner (1407), and a camera (1408) (only one of each is shown).

[0150] The computer system 1400 may also include certain human interface output devices that may stimulate one or more of the user's senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1410), data gloves (not shown), or joystick (1405), although haptic feedback devices that do not function as input devices may also exist), audio output devices (e.g., speakers (1409), headphones (not shown)), visual output devices (e.g., screens (1410) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without haptic feedback capability, some of which are capable of outputting two-dimensional visual output or output in more than three dimensions by means of stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0151] The computer system (1400) may also include human-accessible storage devices and their associated media, such as CD / DVD or similar media (1421), CD / DVD ROM / RW (1420), including thumb drives (1422), removable hard disks or solid state drives (1423), legacy magnetic media, such as tape and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices, such as security dongles (not shown), and the like.

[0152] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0153] The computer system 1400 may also include interfaces 1454 to one or more communications networks 1455. Networks may be, for example, wireless, wireline, or optical. Networks may also be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet; wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, and the like; TV wireline or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; and vehicle and factory networks including CAN bus. Certain networks generally require an external network interface adapter attached to a particular general-purpose digital port or peripheral bus 1449 (e.g., a USB port on the computer system 1400). Others are generally integrated into the core of the computer system 1400 by attachment to a system bus as described below (e.g., an Ethernet network interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1400) can communicate with other entities. Such communication can be one-way receive-only (e.g., broadcast TV) or one-way transmit-only (e.g., a CAN bus to a specific CAN bus device), or it can be two-way to other computer systems, for example, using a local or wide-area digital network. Specific protocols or protocol stacks can be used with each of the networks and network interfaces described above.

[0154] The above-mentioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core 1440 of the computer system 1400 .

[0155] The core (1440) may include one or more central processing units (CPUs) (1441), graphics processing units (GPUs) (1442), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1443), task-specific hardware accelerators (1444), graphics adapters (1450), etc. These devices may be connected through a system bus (1448), along with read-only memory (ROM) (1445), random access memory (RAM) (1446), internal mass storage devices such as internal non-user-accessible hard drives, SSDs, etc. (1447). In some computer systems, the system bus (1448) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached to the core's system bus (1448) directly or through a peripheral bus (1449). In an example, a display 1410 may be connected to a graphics adapter 1450. Architectures for peripheral buses include PCI, USB, and the like.

[0156] The CPU (1441), GPU (1442), FPGA (1443), and accelerator (1444) can execute specific instructions that, in combination, can constitute the above-mentioned computer code. The computer code can be stored in ROM (1445) or RAM (1446). Temporary data can also be stored in RAM (1446), while persistent data can be stored, for example, in an internal mass storage device (1447). Rapid storage and retrieval from any of the memory devices can be enabled through the use of cache memory. Cache memory can be closely associated with one or more of the CPU (1441), GPU (1442), mass storage device (1447), ROM (1445), RAM (1446), etc.

[0157] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0158] By way of example, and not limitation, a computer system having the architecture (1400), and in particular the core (1440), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with the user-accessible mass storage devices previously introduced, in addition to specific storage of the core (1440) that is non-transitory in nature, such as the core's internal mass storage device (1447) or ROM (1445). Software implementing various embodiments of the present disclosure can be stored on such devices and executable by the core (1440). The computer-readable media can include one or more memory devices or chips, depending on particular needs. Software can cause the cores (1440), and specifically the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform particular processes or portions of particular processes described herein, including defining data structures stored in RAM (1446) and modifying such data structures according to software-defined processes. Additionally or alternatively, the computer system can provide functionality as a result of hardwired or otherwise embodied logic in circuitry (e.g., accelerators (1444)) that can operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software can encompass logic, where appropriate, and vice versa. References to computer-readable media can encompass circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both, where appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0159] The use of "at least one of" within this disclosure is intended to include any one or combination of the listed elements. For example, at least one of A, B, or C; at least one of A, B, and C; at least one of A, B, and / or C; and at least one of A through C is intended to include A only, B only, C only, or any combination thereof.

[0160] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be apparent to those skilled in the art that numerous systems and methods will be possible which embody the principles of the present disclosure and are therefore within its spirit and scope, even if not explicitly shown or described herein.

[0161] [Incorporated by reference] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 388,909, filed July 13, 2022, and entitled "Template matching for multiple reference line intra prediction," and U.S. Patent Application No. 18 / 207,587, filed June 8, 2023, and entitled "TEMPLATE MATCHING FOR MULTIPLE REFERENCE LINE INTRA PREDICTION," the disclosures of which are incorporated herein by reference in their entireties.

Claims

1. 1. A method of video decoding performed by a video decoder, comprising: receiving a video bitstream including a current coding block in a picture; determining that the current coding block is coded in a template-based multiple reference line intra-prediction (TMRL) mode; determining a combination of reference lines and intra prediction modes for intra prediction of the current coding block, wherein a first combination among the combinations includes a first reference line among the reference lines and a first intra prediction mode among the intra prediction modes, and the first reference line is a non-adjacent reference line; For the first combination, calculating template entry delta values ​​respectively associated with entries of a template of the current coding block based on the first reference line and the first intra prediction mode, wherein the template of the current coding block is in an adjacent reference line; calculating a first template matching cost value associated with the first combination by weighting each of the template entry delta values; ordering the combinations of the reference lines and the intra-prediction modes according to template matching cost values ​​respectively associated with the combinations, the template matching cost values ​​associated with the combinations including the first template matching cost value associated with the first combination; selecting a particular combination from the ordered combinations; reconstructing the current coding block according to the particular combination; and The template for the current coding block includes neighboring samples to the current coding block as the entries, and the step of calculating the template entry delta values ​​includes: determining a reference template in the first reference line corresponding to the template of the current coding block based on the first intra prediction mode, the reference template including reference entries each corresponding to the entry of the template; calculating a first template entry delta value as a difference between a first entry among the entries of the template and a first reference entry among the reference entries of the reference template, the first reference entry corresponding to the first entry, and the first template entry delta value being one of the template entry delta values; determining a weight for weighting the first template entry delta value based on at least the first entry and the first reference entry; method.

2. determining the weight for weighting the first template entry delta value based on the first entry, the first reference entry, and one or more neighboring reference entries of the first reference entry; The method of claim 1.

3. determining the weight for weighting the first template entry delta value based on the first template entry delta value and one or more template entry delta values ​​associated with neighboring entries of the template; The method of claim 1.

4. determining the weight for weighting the first template entry delta value based on at least a first gradient value at the first entry and / or a second gradient value at the first reference entry; The method of claim 1.

5. calculating the first template matching cost value associated with the first combination based on a portion of the template. The method of claim 1.

6. calculating the first template matching cost value based on an upper template including upper neighboring samples of the current coding block; The method of claim 5.

7. calculating the first template matching cost value based on a left template including left neighboring samples of the current coding block; The method of claim 6.

8. determining a subset of the combinations based on a predefined range of the template matching cost values; selecting the particular combination from the subset of combinations; The method of claim 1 further comprising:

9. selecting a first subset of the combinations having a first template matching cost value within the predefined range; selecting one or more second combinations having second template matching cost values ​​outside the predefined range in response to a number of first combinations in the first subset of the combinations being less than a specific number specifying a number requirement for available combinations, the first subset and the one or more second combinations forming the subset of the combinations, the number of combinations in the subset being equal to the specific number; The method of claim 8 further comprising:

10. selecting a first subset of the combinations having a first template matching cost value within the predefined range; selecting the subset of combinations from the first subset of combinations in response to a first number of combinations in the first subset of combinations being greater than a specific number specifying a number requirement for available combinations, the number of combinations in the subset being equal to the specific number; The method of claim 8 further comprising:

11. The step of selecting the subset of the combinations comprises: selecting the subset of the combinations having the lowest template matching cost value; The method of claim 10.

12. The step of selecting the subset of the combinations comprises: selecting the subset of the combinations having the closest reference line to the template. The method of claim 10.

13. The template is within a second reference line, and the method further comprises: determining that the projection point of the template entry is at a fractional position between two samples in the first reference line; generating a reference entry associated with said entry of said template according to interpolation based on at least said two samples; The method of claim 1 further comprising:

14. The step of selecting the specific combination includes: and decoding an index from the bitstream together with the picture encoded in the bitstream, the index indicating the particular combination from the ordered combinations. The method of claim 1.

15. 15. A method according to claim 1, further comprising: a processing circuit configured to execute software instructions which, when executed by the processing circuit, cause the processing circuit to perform the method of any one of claims 1 to 14. Device.

16. 1. A method of video encoding performed by a video encoder, comprising: determining a combination of reference lines and intra prediction modes for intra prediction of a current coding block in a picture, wherein a first combination among the combinations includes a first reference line among the reference lines and a first intra prediction mode among the intra prediction modes, and the first reference line is a non-adjacent reference line; For the first combination, calculating template entry delta values ​​respectively associated with entries of a template of the current coding block based on the first reference line and the first intra prediction mode, wherein the template of the current coding block is in an adjacent reference line; calculating a first template matching cost value associated with the first combination by weighting each of the template entry delta values; sorting the combinations of the reference lines and the intra-prediction modes according to template matching cost values ​​respectively associated with the combinations, the template matching cost values ​​associated with the combinations including the first template matching cost value associated with the first combination; selecting a particular combination from the sorted combinations; encoding a signal indicative of said particular combination; and The template for the current coding block includes neighboring samples to the current coding block as the entries, and the step of calculating the template entry delta values ​​includes: determining a reference template in the first reference line corresponding to the template of the current coding block based on the first intra prediction mode, the reference template including reference entries each corresponding to the entry of the template; calculating a first template entry delta value as a difference between a first entry among the entries of the template and a first reference entry among the reference entries of the reference template, the first reference entry corresponding to the first entry, and the first template entry delta value being one of the template entry delta values; determining a weight for weighting the first template entry delta value based on at least the first entry and the first reference entry; method.

17. A method for a video encoder to generate and store a bitstream by video encoding, comprising: encoding a picture including the current coding block into a bitstream; storing the bitstream on a non-transitory storage medium; The step of encoding the picture into a bitstream comprises: determining a combination of reference lines and intra prediction modes for intra prediction of the current coding block, wherein a first combination among the combinations includes a first reference line among the reference lines and a first intra prediction mode among the intra prediction modes, and the first reference line is a non-adjacent reference line; For the first combination, calculating template entry delta values ​​respectively associated with entries of a template of the current coding block based on the first reference line and the first intra prediction mode, wherein the template of the current coding block is in an adjacent reference line; calculating a first template matching cost value associated with the first combination by weighting each of the template entry delta values; sorting the combinations of the reference lines and the intra-prediction modes according to template matching cost values ​​respectively associated with the combinations, the template matching cost values ​​associated with the combinations including the first template matching cost value associated with the first combination; selecting a particular combination from the sorted combinations; encoding a signal indicative of said particular combination into said bitstream carrying coded information of said picture; Including, The template for the current coding block includes neighboring samples to the current coding block as the entries, and the step of calculating the template entry delta values ​​includes: determining a reference template in the first reference line corresponding to the template of the current coding block based on the first intra prediction mode, the reference template including reference entries each corresponding to the entry of the template; calculating a first template entry delta value as a difference between a first entry among the entries of the template and a first reference entry among the reference entries of the reference template, the first reference entry corresponding to the first entry, and the first template entry delta value being one of the template entry delta values; determining a weight for weighting the first template entry delta value based on at least the first entry and the first reference entry; method.

Images