Derivation of partitioning patterns based on template matching

Abstract

A reference block is determined for a current block in a current picture from a plurality of candidate reference blocks based on template matching (TM) costs of the plurality of candidate reference blocks. The TM cost indicates a difference between a template of the current block and each reference template of the plurality of candidate reference blocks. Samples of the determined reference block are classified into a plurality of sample classes. A partitioning pattern for the current block is derived from a plurality of predetermined partitioning patterns based on the determined reference block. The derived partitioning pattern indicates a plurality of partitions of the current block. Each of the plurality of classes of samples of the determined reference block corresponds to each of the plurality of partitions of the current block. The current block is reconstructed based on the derived partitioning pattern of the current block.

Description

Technical Field

[0001] [Related Applications] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 416,407, "Template Matching Based Partitioning Pattern Derivation", filed on October 14, 2022, and U.S. Patent Application No. 18 / 379,624, "TEMPLATE MATCHING BASED PARTITIONING PATTERN DERIVATION", filed on October 12, 2023. The disclosures of the foregoing applications are hereby incorporated by reference in their entirety.

[0002] [Technical Field] This disclosure generally describes embodiments related to video coding.

Background Art

[0003] The background description provided herein is for the purpose of generally presenting the background of the present disclosure. The research of the presently named inventors, to the extent it is not considered prior art at the time of filing in the context of the research described in this background chapter, is not to be regarded as either expressly or implicitly admitted as prior art to the present disclosure, in the same manner as aspects of the description that may not be considered prior art at the time of filing.

[0004] Image / video compression helps transfer image / video data across various devices, storage, and networks with minimal loss of quality. In some cases, video codec techniques can compress video based on spatial and temporal redundancy. For example, a video codec can use a technique called intra-prediction, which can compress images based on spatial redundancy. For example, intra-prediction can use reference data from the current picture being reconstructed for sample prediction. In another example, a video codec can use a technique called inter-prediction, which can compress images based on temporal redundancy. For example, inter-prediction can use motion compensation to predict samples in the current picture from previously reconstructed pictures. Motion compensation is represented by motion vectors (MV). [Overview of the project]

[0005] The aspects of the disclosure include methods and apparatus for video coding / decoding. In some examples, the apparatus for video decoding includes processing circuits.

[0006] According to one aspect of this disclosure, a method for video decoding is provided. In this method, a video bitstream containing the current block in the current picture is received. A reference block is determined for the current block from a plurality of candidate reference blocks based on the template matching (TM) costs of the plurality of candidate reference blocks. The TM cost represents the difference between the template of the current block and the reference templates of each of the plurality of candidate reference blocks. The samples of the determined reference block are classified into a plurality of sample classes. A partitioning pattern of the current block is derived from a predetermined plurality of partitioning patterns based on the determined reference block. The derived partitioning pattern represents a plurality of partitions of the current block. Each of the plurality of classes of samples of the determined reference block corresponds to each of the plurality of partitions of the current block. The current block is reconstructed based on the derived partitioning pattern of the current block.

[0007] In one embodiment, multiple candidate reference blocks are from the current picture and one of the reference pictures of the current block. The sample of the determined reference block is the reconstructed sample.

[0008] In one example, the initial reference block is determined based on motion vector information contained in the received video bitstream. Multiple candidate reference blocks are determined within the search range of the initial reference block. The TM cost between the reference template of each of the multiple candidate reference blocks and the template of the current block is determined. From the multiple candidate reference blocks, the reference block corresponding to the smallest TM cost among the determined TM costs between the reference template of each of the multiple candidate reference blocks and the template of the current block is determined.

[0009] In one example, the samples of the determined reference block are classified into a first sample class and a second sample class based on a binary image segmentation algorithm.

[0010] In one example, the samples of the determined reference block are classified into a first sample class with sample values ​​greater than a threshold, and a second sample class with sample values ​​less than a threshold. The threshold is either the mean or the median of the samples.

[0011] In one example, the determined reference block samples are clustered into one or more sample classes using a clustering method.

[0012] In one example, the template region adjacent to the determined reference block is determined. The samples of the determined reference block are classified into one or more sample classes in edge detection based on the sample values ​​of the template region adjacent to the determined reference block. The one or more sample classes include at least a first class containing samples at the edges of the determined reference block and a second class containing samples in the internal region of the determined reference block.

[0013] In one example, the template region includes one of the following: (i) multiple rows of neighbor samples above the determined reference block, (ii) multiple columns of neighbor samples to the left of the determined reference block, or (iii) a region in one or more reconstructed neighbor blocks of the determined reference block.

[0014] In one example, a sample of the determined reference block is classified based on each of several partitioning boundary candidates. Several partitioning boundary candidates for the current block are determined, and each of the several partitioning boundary candidates for the current block corresponds to each of the several partitioning boundary candidates for the determined reference block. The TM cost is determined between each of the several partitioning boundary candidates for the sample of the determined reference block and the corresponding one of the several partitioning boundary candidates for the sample of the current block. From the several partitioning boundary candidates for the current block, the partitioning boundary corresponding to the smallest TM cost among the determined TM costs is determined. Based on the determined partitioning boundary, several partitions of the current block are determined.

[0015] In one example, the binary mask is determined based on either the mean or median of the reconstructed sample of the current block's template. Based on the binary mask, the dominant sample group in the reconstructed sample of the current block's template is determined. Based on the binary mask, the dominant sample group in the sample of each of the candidate reference blocks' reference templates is determined. The TM cost between the dominant group in the reconstructed sample of the current block's template and the dominant group in the sample of each of the candidate reference blocks' reference templates is determined. From the candidate reference blocks, the reference block corresponding to the smallest TM cost among the determined TM costs between the dominant group in the reconstructed sample of the current block's template and the dominant group in the sample of each of the candidate reference blocks' reference templates is determined.

[0016] In one example, multiple candidate reference blocks are determined based on multiple motion vectors currently associated with the block.

[0017] In one example, the TM cost between the reference template of each of the multiple candidate reference blocks and the template of the current block is determined. From the multiple candidate reference blocks, the candidate reference block corresponding to the smallest TM cost among the determined TM costs between the reference template of each of the multiple candidate reference blocks and the template of the current block is determined. Based on the determined candidate reference block, a motion vector (MV) is determined. The motion vector indicates the offset between the current block and the determined candidate reference block. Based on the determined MV and the motion vector difference (MVD), an adjusted MV is determined. Based on the adjusted MV, a reference block is determined.

[0018] In one example, multiple candidate reference blocks are determined in the current picture. The TM cost between the reference template of each of the multiple candidate reference blocks and the template of the current block is determined. From the multiple candidate reference blocks, the candidate reference block corresponding to the smallest TM cost among the determined TM costs between the reference template of each of the multiple candidate reference blocks and the template of the current block is determined. A block vector (BV) is determined based on the determined candidate reference block, and the BV indicates the offset between the current block and the determined candidate reference block. The reference block is determined based on the determined BV.

[0019] In another aspect of this disclosure, a device is provided. The device includes a processing circuit. The processing circuit can be configured to perform one of the described methods of video decoding / encoding. For example, the processing circuit is configured to receive a video bitstream containing the current block in the current picture. The processing circuit is configured to determine a reference block for the current block from a plurality of candidate reference blocks based on the template matching (TM) cost of the plurality of candidate reference blocks. The TM cost represents the difference between the template of the current block and the reference template of each of the plurality of candidate reference blocks. The processing circuit is configured to classify the samples of the determined reference block into a plurality of sample classes. The processing circuit is configured to derive a partitioning pattern for the current block from a predetermined plurality of partitioning patterns based on the determined reference block. The derived partitioning pattern represents a plurality of partitions of the current block. Each of the plurality of classes of samples of the determined reference block corresponds to each of the plurality of partitions of the current block. The processing circuit is configured to reconstruct the current block based on the derived partitioning pattern of the current block.

[0020] Aspects of this disclosure also provide non-temporary computer-readable media that, when executed by a computer, causes the computer to perform one of the described methods for video decoding / encoding. [Brief explanation of the drawing]

[0021] Further features, characteristics, and various advantages of the subject matter of disclosure will become clearer from the detailed description and accompanying drawings below.

[0022] [Figure 1] This is a schematic diagram of an exemplary block diagram of a communication system (100).

[0023] [Figure 2] This is a schematic diagram of an exemplary block diagram of a decoder.

[0024] [Figure 3] It is a schematic diagram of an exemplary block diagram of an encoder.

[0025] [Figure 4] Exemplary geometric partition modes (GPMs) grouped by the same angle are shown.

[0026] [Figure 5A] Exemplary GPMs with intra prediction and inter prediction are shown. [Figure 5B] Exemplary GPMs with intra prediction and inter prediction are shown. [Figure 5C] Exemplary GPMs with intra prediction and inter prediction are shown.

[0027] [Figure 5D] Exemplary GPMs with intra prediction and intra prediction are shown.

[0028] [Figure 6A] An exemplary first template of a block according to some embodiments of the present disclosure is shown.

[0029] [Figure 6B] An exemplary second template of a block according to some embodiments of the present disclosure is shown. [[ID=四十二]]

[0030] [Figure 7] A flowchart showing an overview of decoding processing according to some embodiments of the present disclosure is shown.

[0031] [Figure 8] A flowchart showing an overview of encoding processing according to some embodiments of the present disclosure is shown.

[0032] [Figure 9] It is a schematic diagram of a computer system according to one embodiment. [Modes for carrying out the invention]

[0033] Figure 1 shows a block diagram of a video processing system (100) in several examples. The video processing system (100) is a video encoder and video decoder in a streaming environment, which is an example of the application of the subject matter of disclosure. The subject matter of disclosure is equally applicable to, for example, video conferencing, digital TV, streaming services, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc., and other video-enabled applications.

[0034] The video processing system (100) includes a capture subsystem (113) which may include a video source (101), such as a digital camera, that generates, for example, an uncompressed video picture stream (102). In one example, the video picture stream (102) includes samples captured by the digital camera. The video picture stream (102) is shown in thick lines to emphasize its high data volume compared to encoded video data (104) (or encoded video bitstream) and may be processed by an electronic device (120) which includes a video encoder (103) coupled with the video source (101). The video encoder (103) may include hardware, software, or a combination thereof, and may enable or implement aspects of the subject matter of disclosure as detailed below. The encoded video data (104) (or encoded video bitstream) is shown in thin lines to emphasize its low data volume compared to the video picture stream (102) and may be stored in a streaming server (105) for future use. One or more streaming client subsystems, such as client subsystems (106) and (108) in Figure 1, can access a streaming server (105) to read copies (107) and (109) of encoded video data (104). Client subsystem (106) may include a video decoder (110) within, for example, an electronic device (130). The video decoder (110) decodes the input copy (107) of the encoded video data and generates an output video picture stream (111) that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., video bitstreams) may be encoded according to a specific video coding / compression standard. An example of these standards is ITU-T Recommendation H.265. For example, a video coding standard under development is informally known as VVC (Versatile Video Coding). The subject of disclosure may be used in the context of VVC.

[0035] It should be noted that electronic devices (120) and (130) may include other components (not shown). For example, electronic device (120) may include a video decoder (not shown), and electronic device (130) may also include a video encoder (not shown).

[0036] Figure 2 shows an exemplary block diagram of a video decoder (210). The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., a receiving circuit). In the example of Figure 1, the video decoder (210) can be used instead of the video decoder (110).

[0037] The receiver (231) can receive one or more coded video sequences, for example, contained in a bitstream, which are decoded by the video decoder (210). In embodiments, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device that stores coded video data. The receiver (231) may receive coded video data together with other data, for example, coded audio data and / or auxiliary data streams that may be transferred to their respective usage entities (not shown). The receiver (231) may isolate the coded video sequences from other data. To eliminate network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter, "Parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). Alternatively, a buffer memory (not shown) may exist outside the video decoder (210) (not shown). Further, a buffer memory (not shown) may exist outside the video decoder (210), for example, to eliminate network jitter, or in addition to another buffer memory (215) inside the video decoder (210), for example, to handle playback timing. When the receiver (231) receives data controllably from a storage / transmission device with sufficient bandwidth, or from an isosynchronous network, the buffer memory (215) may not be necessary or can be made small. For use in best-effort packet networks such as the Internet, the buffer memory (215) may be required, may be relatively large, advantageously adaptive in size, and may be implemented at least partially in the operating system or a similar element (not shown) outside the video decoder (210).

[0038] The video decoder (210) may include a parser (220) to reconstruct symbols (221) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (210), and information for controlling rendering devices, such as a renderer (212) (e.g., a display screen), which may not be an integrated part of the electronic device (230) but may be coupled to the electronic device (230), as shown in Figure 2. The control information for the rendering device may be in the form of SEI (Supplemental Enhancement Information) messages or VUI (Video Usability Information) parameter set fragments (not shown). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, context-dependent or non-context-dependent arithmetic coding, etc. The parser (220) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to that group. Subgroups may include GOP (Groups of Picture), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (220) may also extract information such as transformation coefficients, quantization parameter values, motion vectors, etc., from the coded video sequence.

[0039] The parser (220) may perform an entropy decoding / parse operation on the video sequence received from the buffer memory (215) to generate a symbol (221).

[0040] The reconstruction of symbol (221) may include multiple different units, depending on the type of coded video picture or part thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. Which units are included and how they are included can be controlled by group control information parsed by parser (220) from the coded video sequence. The flow of such subgroup control information between parser (220) and the following multiple units is not shown for clarity.

[0041] Beyond the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into numerous functional units, as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with each other and be at least partially integrated. However, for the purpose of illustrating the subject of this disclosure, the following conceptual subdivision into functional units is appropriate.

[0042] The first unit is the scaler / inverse unit 251. The scaler / inverse unit (251) receives control information from the parser (220) as symbols (221), including quantized transformation coefficients and which transformation to use, block size, quantization coefficients, quantization scaling matrix, etc. The scaler / inverse unit (251) can output a block containing sample values ​​that can be input to the aggregator (255).

[0043] In some cases, the output samples of the scaler / inverse unit (251) may relate to intracoded blocks. Intracoded blocks are blocks that do not use prediction information from previously reconstructed pictures but can use prediction information from portions reconstructed before the current picture. Such prediction information can be provided by the intrapicture prediction unit (252). In some cases, the intrapicture prediction unit (252) generates a block of the same size and shape as the block being reconstructed, using the surrounding already reconstructed information fetched from the current picture buffer (258). The current picture buffer (258) buffers, for example, the reconstructed current picture partially and / or completely. In some cases, the aggregator (255) adds the prediction information generated by the intraprediction unit (252) to the output sample information provided by the scaler / inverse unit (251) on a sample-by-sample basis.

[0044] In other cases, the output samples of the scaler / inverse unit (251) may be associated with an interconnected, and possibly motion-compensated, block. In such cases, the motion-compensated prediction unit (253) can access the reference picture memory (257) to fetch samples to be used for prediction. After motion-compensating the fetched samples according to the symbols (221) associated with the block, these samples may be added to the output of the scaler / inverse unit (251) by the aggregator (255) to generate output sample information (in this case, called residual samples or residual signals). The addresses in the reference picture memory (257) from which the motion-compensated prediction unit (253) fetches prediction samples can be controlled by the motion vectors available to the motion-compensated prediction unit (253), for example, in the form of symbols (221) which may have X, Y, and reference picture components. Motion compensation may include interpolation of sample values ​​fetched from the reference picture memory (257) when the exact motion vectors of subsamples are in use, motion vector prediction mechanisms, etc.

[0045] The output samples of the aggregator (255) can undergo various loop filtering techniques in the loop filter unit (256). Video compression techniques may include in-loop filtering techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream), which are made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression may respond not only to previously reconstructed and loop-filtered sample values, but also to metadata obtained during decoding of earlier portions (in decoding order) of the coded picture or coded video sequence.

[0046] The output of the loop filter unit (256) may be a sample stream that can be output to the renderer (212) and stored in the reference picture memory (257) for use in future interpicture prediction.

[0047] A specific coded picture, once fully reconfigured, can be used as a reference picture for future predictions. For example, once the coded picture corresponding to the current picture is fully reconfigured and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) can become part of the reference picture memory (257) and a fresh current picture buffer can be reallocated before starting the reconfiguration of subsequent coded pictures.

[0048] The video decoder (210) may perform decoding operations in accordance with a standard such as ITU-T Rec.H.265 or a specified video compression technology. The coded video sequence may follow the syntax specified by the video compression technology or standard in use, in the sense that the coded video sequence conforms to both the video compression technology or standard and the profile documented in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as tools that are only usable under the profile. Also, compliance may require that the complexity of the coded video sequence be within the limits set by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further restricted through the HRD (Hypothetical Reference Decoder) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0049] In the embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be included as a portion of the coded video sequence. The additional data may be used by the video decoder (210) to correctly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) extension layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0050] Figure 3 shows an exemplary block diagram of a video encoder (303). The video encoder (303) is included in the electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmitting circuit). The video encoder (303) can be used in place of the video encoder (103) in the example of Figure 1.

[0051] The video encoder (303) may receive video samples from a video source (301) (not part of the electronic device (320) in the example in Figure 3) that can capture video images to be coded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0052] The video source (301) may provide a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream of any appropriate bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any appropriate sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media delivery system, the video source (301) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a series of individual pictures that give motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels. Each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. The following description will focus on samples.

[0053] According to the embodiment, the video encoder (303) may encode and compress the pictures of the source video sequence into a coded video sequence (343) in real time or under any other required time constraints. Implementing an appropriate coding speed is one function of the control unit (350). In some embodiments, the control unit (350) controls and is functionally coupled to other functional units, which are described later. The couplings are not shown for clarity. Parameters set by the control unit (350) may include rate control-related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique, ...), picture size, GOP (group of pictures) layout, maximum motion vector search range, etc. The control unit (350) may be configured to have other appropriate functions related to the video encoder (303) optimized for a particular system design.

[0054] In some embodiments, the video encoder (303) is configured to operate within a coding loop. In a very simplified explanation, one example might include a source coder (330) (responsible for generating symbols, such as a symbol stream, based on the input and reference pictures to be coded), and a (local) decoder (333) built into the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a similar manner to that created by a (remote) decoder. The reconstructed sample stream (sample data) is fed into the reference picture memory (334). The contents of the reference picture memory (334) are also bit-accurate between the local and remote encoders, when decoding the symbol stream yields bit-accurate results independently of the decoder location (local or remote). In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the decoder "sees" when using predictions during decoding, as reference picture samples. This fundamental principle of reference picture synchronization (and the resulting drift, for example, when synchronization cannot be maintained due to channel errors) is used similarly in several related technologies.

[0055] The operation of a “local” decoder (333) may be the same as that of a “remote” decoder, such as the video decoder (210) detailed above in relation to Figure 2. However, as also briefly referring to Figure 2, since symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropy coder (345) and parser (220) may be lossless, the entropy decoding section of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333).

[0056] In the embodiments, the decoder techniques present in the decoder, excluding analysis / entropy decoding, are present in the corresponding encoder in the same or substantially the same functional form. Therefore, the subject matter of the disclosed information focuses on the decoder operation. A description of the encoder techniques can be omitted, as they are the reverse of the decoder techniques from which they are comprehensively described. More detailed descriptions in specific areas are provided below.

[0057] During operation, in some examples, the source coder (330) may perform motion-compensated predictive coding. This predictively codes the input picture by referencing one or more previously coded pictures from a video sequence designated as “reference pictures”. In this method, the coding engine (332) codes the difference between the pixel blocks of the input picture and the pixel blocks of the reference picture which may be selected as the predictive criterion for the input picture.

[0058] The local video decoder (333) may decode the coded video data of a picture that may be designated as a reference picture, based on symbols generated by the source coder (330). The operation of the coding engine (332) may, advantageously, be lossy. When the coded video data can be decoded by a video decoder (not shown in Figure 3), the reconstructed video sequence may, as a standard, be a copy of the source video sequence with some errors. The local video decoder (333) may duplicate the decoding process that may be performed by the video decoder on the reference picture, resulting in a reconstructed reference picture to be stored in the reference picture memory (334). Thus, the video encoder (303) may store a copy of the reconstructed reference picture that has the same content as the reconstructed reference picture obtained by the far-end video decoder (if there are no transmission errors).

[0059] The predictor (335) may perform a predictive search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) may search the reference picture memory (334) for sample data (such as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., which could serve as appropriate predictive criteria for the new picture. The predictor (335) may operate sample block-pixel block by sample block to find appropriate predictive criteria. In some examples, the input picture may have predictive criteria drawn from multiple reference pictures stored in the reference picture memory (334), as determined by the search results obtained by the predictor (335).

[0060] The control unit (350) may manage the coding operations of the source coder (330), including, for example, setting parameters and subgroup parameters used for encoding video data.

[0061] The outputs of all the aforementioned functional units may undergo entropy coding in the entropy coder (345). The entropy coder (345) converts the symbols generated by the various functional units into coded video sequences by applying lossless compression to the symbols according to techniques such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0062] The transmitter (340) may buffer the coded video sequence generated by the entropy coder (345) in preparation for transmission over a communication channel (360), which may be a hardware / software link to a storage device capable of storing coded video data. The transmitter (340) may merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0063] The control unit (350) may manage the operation of the video encoder (303). During coding, the control unit (350) may assign each coded picture a specific coded picture type that may affect the coding techniques that can be applied to each picture. For example, a picture may often be assigned as one of the following picture types:

[0064] An intra-picture (I-picture) may be coded and decoded without using any other pictures in the sequence as a source for prediction. Some video codecs allow different types of intra-pictures, including, for example, IDR (Independent Decoder Refresh) pictures.

[0065] A predictive picture (P-picture) may, in most cases, be a picture that can be coded and decoded using intra-prediction or inter-prediction with motion vectors and reference indices to predict the sample values ​​of each block.

[0066] A bidirectionally predictive picture (B-picture) may be coded and decoded using intra-prediction or inter-prediction with two motion vectors and a reference index to predict the sample values ​​for each block. Similarly, a multi-predictive picture may use two or more reference pictures and associated metadata for the reconstruction of a single block.

[0067] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block may be coded. Blocks may be coded predictively by references to other (already coded) blocks, determined by the coding assignments applied to each picture in the block. For example, blocks of picture I may be coded non-predictively, or they may be coded predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be coded predictively via spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks of picture B may be coded predictively via spatial prediction or temporal prediction by referencing one or two previously coded reference pictures.

[0068] The video encoder (303) may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Rec.H.265. In these operations, the video encoder (303) may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. The coded video data may, therefore, conform to the syntax specified by the video coding technique or standard being used.

[0069] In one embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the encoded video sequence. The additional data may include time / space / SNR extension layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0070] Video may be captured as multiple source pictures (video pictures) in a time series. Intra-picture prediction (sometimes abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes (temporal or other) correlations between pictures. In one example, a particular picture being encoded / decoded is called the current picture and is partitioned into blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension to identify the reference picture if multiple reference pictures are in use.

[0071] In some embodiments, bi-prediction techniques can be used for interpicture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both of which are earlier in the video in decoding order than the current picture (but may be earlier and later in display order, respectively). Blocks in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Blocks can be predicted by combining the first and second reference blocks.

[0072] Furthermore, merge mode techniques can be used in interpicture prediction to improve coding efficiency.

[0073] According to some embodiments of this disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed within units of blocks. For example, according to the HEVC standard, pictures in a video-picture sequence are partitioned into coding tree units (CTUs) for compression. CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Typically, a CTU contains three coding tree blocks (CTBs), i.e., one lumen CTB and two chroma CTBs. Each CTU can be recursively quad-tree partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be partitioned into one 64x64 pixel CU, or four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, such as inter-prediction type or intra-prediction type. A CU is divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Typically, each PU contains a chroma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed within the units of the prediction block. Using a chroma prediction block as an example of a prediction block, the prediction block contains a matrix of values ​​(e.g., chroma values) for pixels such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0074] It should be noted that the video encoders (103) and (303), and the video decoders (110) and (210) can be implemented using any suitable technique. In one embodiment, the video encoders (103) and (303), and the video decoders (110) and (210) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303), and the video decoders (110) and (210) can be implemented using one or more processors that execute software instructions.

[0075] This disclosure includes embodiments relating to deriving partition patterns using template matching.

[0076] Geometric partitioning mode (GPM) can be applied to inter-prediction, as in VVC. Geometric partitioning mode can be signaled using a flag (e.g., CU level flag) as a type of merge mode. Other types of merge modes include normal merge mode, merge motion vector differences (MMVD) mode, combined inter and intra prediction (CIIP) mode, and subblock merge mode. In one example, w×h=2 m ×2 n For each possible CU of size, 64 partitions are supported by the GPM, where m,n ∈ {3...6}, excluding 8×64 and 6×48.

[0077] Using GPM, a CU can be divided into two parts by geometrically arranged straight lines (e.g., (404)). The position of the dividing line can be mathematically derived from the angle and offset parameters of a particular partition. Figure 4 shows an exemplary GPM partition grouped by the same angle. As shown in Figure 4, each CU (e.g., (402)) can contain each group of partition lines. Each partition line (e.g., (404)) can indicate the partitioning method and correspond to each offset. A group of partition lines in each CU can contain up to four partition lines, each corresponding to each angle. Each part of the geometric partition within a CU can be interpreted using its own motion. For each partition, a uni-prediction is permitted. For example, each part (each partition) may have one motion vector and one reference index. By applying uni-prediction motion constraints, it can be ensured that two motion-compensated predictions are required for each CU, similar to conventional bi-prediction.

[0078] When a geometric partitioning mode is used for the current CU, a geometric partitioning index (e.g., angle and offset) indicating the partitioning mode of the geometric partition, and two merge indices (one for each partition) can be further signaled. The maximum GPM candidate size can be explicitly notified in the Sequence Parameter Set (SPS), specifying syntactic binarization of the GPM merge index. After each geometric partition has been predicted, the sample values ​​along the geometric partition edges can be adjusted using a fusion process with adaptive weights. Thus, a predicted signal for the entire CU based on the GPM can be obtained. In other prediction modes, transformation and quantization processes can be applied to the entire CU. Furthermore, the motion field of the CU predicted using the geometric partitioning mode can be stored.

[0079] Geometric partition modes can be stored in motion fields. For example, Mv1 from the first part of the geometric partition, Mv2 from the second part of the geometric partition, and the combined Mv of Mv1 and Mv2 can be stored in the motion fields of the geometric partition modes coded in CU. The type of motion vector stored for each individual position in the motion field can be determined by equation (1) as follows:

number

number

[0080] Template matching can be applied to GPM. When GPM mode is enabled for CU, a CU-level flag can be signaled to indicate whether TM applies to both geometric partitions. Motion information for each geometric partition can be refined using TM. When TM is selected, a template can be constructed using left, top, or left and top adjacent samples according to the partition angle, as shown in Table 1. The motion can then be refined by minimizing the difference between the current template and the template in the reference image using the same search pattern in merge mode with the half-pel interpolation filter disabled. Table 1 shows examples of templates for the first and second geometric partitions in GPM, where A represents the use of top samples, L represents the use of left samples, and L+A represents the use of both left and top samples. [Table 1] Example of a partition template in GPM [Table 1]

[0081] In one embodiment, the GPM candidate list can be structured as follows: (1) Interleaved List-0MV candidates and List-1MV candidates can be directly derived from the regular merge candidate list, and List-0MV candidates may have a higher priority than List-1MV candidates. A pruning method with an adaptive threshold based on the current CU size can be applied to remove redundant MV candidates. (2) Furthermore, the interleaved List-1MV candidates and List-0MV candidates can be directly derived from the regular merge candidate list, and List-1MV candidates can have a higher priority than List-0MV candidates. The same pruning method with adaptive thresholds can also be applied to remove redundant MV candidates. (3) 0MV candidates can be padded until the GPM candidate list is full.

[0082] GPM-MMVD and GPM-TM can be enabled exclusively for a single GPM CU. For example, the GPM-MMVD syntax can be signaled first. If both GPM-MMVD control flags are equal to false, the GPM-TM flag can be signaled to indicate that GPM-MMVD is disabled for the two GPM partitions and whether template matching applies to the two GPM partitions. Otherwise, if at least one GPM-MMVD flag is equal to true, the value of the GPM-TM flag can be inferred to be false.

[0083] When GPM with inter and intra prediction is applied, the final predicted samples can be generated by weighting inter-predicted samples, predicted samples, and intra-predicted samples for each region separated by the GPM. Inter-predicted samples can be derived by the inter-GPM. Intra-predicted samples can be derived by an intra-prediction mode (IPM) candidate list and an index signaled from the encoder. The IPM candidate list size can be predefined as 3. Available IPM candidates can include parallel-angle modes with respect to the GPM block boundaries (e.g., parallel mode), perpendicular-angle modes with respect to the GPM block boundaries (e.g., perpendicular mode), and planar modes, which can be shown in Figures 5A, 5B, and 5C, respectively. Furthermore, as shown in Figure 5D, GPM with intra and intra-prediction can be restricted to reduce signaling overhead for IPM and avoid increasing the size of the intra-prediction circuit on the hardware decoder. Furthermore, coding performance can be further improved by introducing direct motion vectors and IPM storage on the GPM fusion region.

[0084] In decoder-side intra-mode derivation (DIMD) and neighbor-mode-based IPM derivation, parallel modes can be registered first. Therefore, if the same IPM candidate is not in the IPM candidate list, up to two IPM candidates derived by the DIMD method and / or from neighbor blocks can be registered. In neighbor-mode-based IPM derivation, up to five locations can be defined to determine the available neighbor blocks. However, as shown in Table 2, the five locations can be limited by the angle of the GPM block boundary to which GPM with template matching (GPM-TM) is applied. As shown in Table 2, the locations of the available neighbor blocks for IPM candidate derivation can be defined based on the angle of the GPM block boundary. A and L indicate the upper and left sides of the predicted block. [Table 2] Location of available neighboring blocks for IPM candidate derivation [Table 2] In one embodiment, GPM-intra can be combined with GPM with merge motion vector difference (GPM-MMVD). In one embodiment, template-based intra-mode derivation (TIMD) can be used to derive IPM candidates for GPM-intra, further improving coding performance. In one embodiment, parallel modes can be registered first, and then TIMD, DIMD, and adjacent block IPM candidates can be registered thereafter.

[0085] Current GPM designs use straight lines as partitioning boundaries, limiting the possibilities to a restricted set of predefined partitioning patterns. This does not always model the most efficient partitioning patterns, including irregular ones. Motion vectors can be used to identify blocks within a reference or current picture, and the reconfiguration values ​​of the identified blocks can be used to derive adaptive partitioning patterns. However, in relevant examples, additional motion vector signaling may be required, which can be costly in specifying partitioning patterns.

[0086] This disclosure can provide the derivation of partitioning patterns based on template matching. In this disclosure, a template can refer to a sample in the vicinity of a block, such as a neighboring sample above, to the left, to the right, and / or below the block. Exemplary templates can be shown in Figures 6A and 6B. As shown in Figure 6A, template (604) can include both an upper neighbor reconstructed sample (606) and a left neighbor reconstructed sample (608) of the current block (602). Template (604) can also include a reconstructed sample (610) in the upper left corner of the current block (602). As shown in Figure 6B, template (611) can include an upper template (614) above the current block (612) and a left template (616) to the left of the current block (612). In one embodiment, if a right or lower sample of the current block in the current frame has already been reconstructed, the right or lower sample can be used as a template.

[0087] This disclosure allows for the identification of reference blocks within a reference picture using template matching. Using the identified reference blocks, a geometric partitioning pattern can be derived for the current block. Each partitioning in the derived geometric partitioning pattern for the current block can be subjected to different behaviors. For example, multiple partitionings of the current block can be determined based on the derived geometric partitioning pattern. The derived geometric partitioning pattern can divide the samples of the current block into multiple groups. Each of the multiple partitions can correspond to each group of samples in the current block.

[0088] In one example, a referenced block can be determined for the current block from among several candidate referenced blocks, based on the template matching (TM) cost of those candidate referenced blocks. The TM cost can represent the difference between the template of the current block and the reference template of each of the candidate referenced blocks. The determined referenced block samples (e.g., reconstructed samples) can be classified into several reconstructed sample classes. Multiple partitions of the current block can be determined corresponding to the multiple reconstructed sample classes of the determined referenced block. Each of the multiple sample classes of the determined referenced block can correspond to each of the multiple partitions of the current block.

[0089] In one embodiment, the template of the current block can be constructed using neighbor samples such as the left, above, or left and above neighbor samples of the current block. The template of a candidate reference block in a reference picture can be constructed using neighbor samples such as the left, above, or left and above neighbor samples of the reference block. Among the reference blocks, the reference block that minimizes the cost between the current template (e.g., the template of the current block) and the reference template (e.g., the template of the reference block) can be identified. Using the identified reference block corresponding to the minimized cost, the partitioning pattern of the current block can be derived.

[0090] As an example, current block template examples can be seen in Figures 6A and 6B.

[0091] In one example, the initial reference block can be determined based on motion vector information contained in the received video bitstream. Multiple candidate reference blocks can be determined within the search range of the initial reference block. The TM cost between the reference template of each of the multiple candidate reference blocks and the template of the current block can be determined. From the multiple candidate reference blocks, the reference block corresponding to the smallest TM cost among the determined TM costs between the reference template of each of the multiple candidate reference blocks and the template of the current block can be determined.

[0092] In one embodiment, a partitioning pattern can be derived using reconstruction samples of identified blocks, such as blocks identified based on template matching.

[0093] In one embodiment, a binary image segmentation algorithm can be applied to identified blocks. The output binary map of the binary image segmentation algorithm can be used as a partitioning pattern.

[0094] In one example, the reconstructed samples of the determined reference block are classified into a first reconstructed sample class and a second reconstructed sample class based on a binary image segmentation algorithm.

[0095] In one embodiment, a threshold for the reconstructed samples of the identified block can be calculated first. The identified block can be divided into two partitions. The first partition may contain samples with values ​​(or sample values) greater than (or greater than) the threshold, and the second partition may contain samples with values ​​(or sample values) less than (or less than) the threshold. In one example, the threshold may be the mean value of the reconstructed samples of the identified block. In one example, the threshold may be the median value of the reconstructed samples of the identified block.

[0096] In one example, reconstructed samples of a determined reference block can be classified into a first reconstructed sample class with sample values ​​greater than a threshold and a second reconstructed sample class with sample values ​​less than a threshold. The threshold can be based on one or more sample values ​​of the identified block. In one example, the threshold can be either the mean or median of the reconstructed samples.

[0097] In one embodiment, a clustering method can be applied to reconstructed samples of identified blocks to derive two or more classes of samples. Samples associated with each class may constitute one of several partitions.

[0098] In one example, the clustering method may include a hierarchical model, a partitioning model, a density-based model, a model-based model, and a grid-based model, or any other suitable clustering method.

[0099] In one embodiment, an edge detection method can be applied to classify reconstructed samples of identified blocks into two or more sample classes. Samples associated with each class may constitute one of several partitions.

[0100] In one example, based on edge detection, reconstruction samples of identified blocks can be classified into edge groups and internal groups. Edge groups may include reconstruction samples at the edges of the identified block, while internal groups may include reconstruction samples within the internal regions of the identified block. Internal regions may be enclosed by the edges of the identified block.

[0101] In one example, the template region adjacent to the determined reference block can be determined. The reconstruction samples of the determined reference block can be classified into one or more reconstruction sample classes in edge detection based on the sample values ​​of the template region adjacent to the determined reference block. One or more reconstruction sample classes may include at least a first reconstruction class containing samples at the edges of the determined reference block and a second reconstruction class containing samples in the internal region of the determined reference block.

[0102] For example, the edge detection method may include the Sobel method, the Canny method, the Prewitt method, the Roberts method, fuzzy logic, or any other suitable edge detection method.

[0103] For example, edge detection could include various mathematical methods (such as Sobel's method, Canny's method, Prewitt's method, Roberts' method, and fuzzy logic methods) aimed at identifying edges and / or curves in a digital image where the image brightness changes abruptly or has discontinuities.

[0104] For example, the template region used for edge detection can be expanded to multiple rows or columns of the sample, or to one or more reconstructed neighbor blocks.

[0105] In one example, the template regions adjacent to the determined reference block can be determined. The reconstructed samples of the determined reference block can be classified into one or more reconstructed sample classes in edge detection based on the sample values ​​of the template regions adjacent to the determined reference block.

[0106] In one embodiment, if a reconstructed sample is available, the width or height of the template region used for edge detection can be greater than the current block width or height. For example, the template region may have a size equal to twice the block width or twice the block height.

[0107] For example, the template region may include one of the following: (i) multiple rows of neighbor samples above the determined reference block, (ii) multiple columns of neighbor samples to the left of the determined reference block, or (iii) a region in one or more reconstructed neighbor blocks of the determined reference block.

[0108] In one embodiment, multiple partitioning boundary candidates can be applied to the reconstructed sample of an identified block. A partitioning boundary candidate that minimizes a predefined cost score can be derived as the GPM-mode partitioning boundary for the current block. In one example, multiple partitions of the current block can be determined based on the derived partitioning boundary. The derived partitioning boundary can divide the sample of the current block into multiple groups. Each of the multiple partitions can correspond to each group of the sample of the current block.

[0109] In one example, a reconfiguration sample of a determined reference block can be classified based on each of several partitioning boundary candidates. Several partitioning boundary candidates are determined for the current block, and each of these candidates can correspond to each of the determined partitioning boundary candidates for the reference block. The TM cost can be determined between each of the partitioning boundary candidates for the reconfiguration sample of the determined reference block and the corresponding one of the partitioning boundary candidates for the sample of the current block. From the partitioning boundary candidates of the current block, the partitioning boundary corresponding to the smallest TM cost among the determined TM costs can be determined. Based on the determined partitioning boundary, several partitions of the current block can be determined.

[0110] In one embodiment, a binary mask can be derived based on the current template, and a template matching operation can be applied to both the current template and the reference template using the derived binary mask to identify blocks in the reference picture. Using the identified blocks, a geometric partitioning pattern of the current blocks can be derived, and each partitioning can be subjected to a different movement.

[0111] In one example, a binary mask can define a region of interest (ROI) of an image (or block). In a binary mask, an image pixel (or sample) with a first mask pixel value (e.g., 1) can belong to the ROI (or majority group). An image pixel with a second mask pixel value (e.g., 0) can belong to the background (or minority group). In one example, an image's ROI may include samples of the image with sample values ​​greater than the image's threshold sample value. The image's threshold sample value can be the mean sample value, the image's median sample value, or the image's average sample value.

[0112] The binary mask can be determined based on the reconstructed sample values ​​of the current block's template. In one example, one of the mean and median values ​​of the current block's template reconstructed sample is used to determine the binary mask. The dominant sample group in the current block's template reconstructed sample can be determined based on the frequency or range of sample values. For example, the dominant sample group can be determined based on the binary mask. Based on the binary mask, the dominant sample group in the sample of each reference template of multiple candidate reference blocks can be determined. The TM cost between the dominant group in the current block's template reconstructed sample and the dominant group in the sample of each reference template of multiple candidate reference blocks can be determined. From the multiple candidate reference blocks, the reference block corresponding to the smallest TM cost among the determined TM costs between the dominant group in the current block's template reconstructed sample and the dominant group in the sample of each reference template of multiple candidate reference blocks can be determined.

[0113] In one embodiment, a threshold for reconstructed samples of the current template can be calculated first. At least one binary mask can be derived by labeling each sample of the current template based on whether each sample is greater than (or equal to) the threshold. When the template matching cost is calculated, samples with a dominant label can be used, and the remaining samples may not be considered for the calculation of the template matching cost. A dominant label indicating a large group may indicate a label value (e.g., 0 or 1) associated with more (or fewer) samples than another label value.

[0114] In one example, the dominant label may indicate the dominant group (or majority group) of reconstructed samples in the current template. For instance, if there are more reconstructed samples with sample values ​​above a threshold than those with sample values ​​below a threshold, the dominant group may include the reconstructed samples with sample values ​​above a threshold.

[0115] The threshold can be derived from one or more values ​​of the reconstructed samples of the current template. In one embodiment, the threshold may be the mean value of the reconstructed samples of the current template. In another embodiment, the threshold may be the median value of the reconstructed samples of the current template.

[0116] In one embodiment, the MV between the current template and the reference template can be used for motion compensation of the associated geometric partitions. The MV can indicate the offset between the dominant group in the current template and the dominant group in the reference template, according to the binary mask.

[0117] In one embodiment, the partitioning pattern of the current block can be derived using a partitioning pattern that can be associated with an identified block coded by GPM, or that can be derived using a template matching method in a reference picture.

[0118] In one embodiment, template matching can be applied using motion vectors associated with the current block to identify candidate reference blocks for template matching.

[0119] In one example, multiple candidate reference blocks can be determined based on multiple motion vectors currently associated with a block.

[0120] In one embodiment, motion vector differences (MVDs) can be signaled on top of template matching to identify reference blocks. MVDs can be coded using a simplified MVD coding scheme such as MMVD. Thus, reference blocks can be determined based on an MV equal to the sum of the determined template and the TM determined by the MVD.

[0121] In one example, the TM cost between each of the reference templates of multiple candidate reference blocks and the template of the current block can be determined. From the multiple candidate reference blocks, the reference block corresponding to the smallest TM cost among the determined TM costs between each of the reference templates of the multiple candidate reference blocks and the template of the current block can be determined. Based on the determined candidate reference block, a motion vector (MV) can be determined. The motion vector can indicate the offset between the current block and the determined candidate reference block. Based on the determined MV and the motion vector difference (MVD), an adjusted MV can be determined. Based on the adjusted MV, a reference block can be determined.

[0122] In one embodiment, template matching can be applied to identify blocks within the current picture, and a partitioning pattern can be derived using the identified blocks. Candidate reference blocks can be determined within the already reconfigured region of the current picture.

[0123] In one embodiment, the derived BV (and / or corresponding BV offset) pointing to a reference block may have an integer sample (e.g., luma sample) resolution, similar to the block vector (BV) in intra-block copy (IBC) mode.

[0124] In one embodiment, the BV (and / or corresponding BV offset) pointing to the reference block may be a subpixel resolution.

[0125] In one example, multiple candidate reference blocks can be determined in the current picture. The TM cost between the reference template of each of the multiple candidate reference blocks and the template of the current block can be determined. From the multiple candidate reference blocks, the reference block corresponding to the smallest TM cost among the determined TM costs between the reference template of each of the multiple candidate reference blocks and the template of the current block can be determined. A block vector (BV) can be determined based on the determined candidate reference block, and the BV can indicate the offset between the current block and the determined candidate reference block. The reference block is determined based on the determined BV.

[0126] In one embodiment, additional MVs can be signaled to indicate the starting position of identified blocks within a reference picture. Furthermore, template matching can be applied to the starting position to find the best-matching identified block. The identified blocks can then be used to derive a partitioning pattern.

[0127] Figure 7 shows a flowchart illustrating an overview of process (700) according to one embodiment of the present disclosure. Process (700) can be used in a video decoder. In various embodiments, process (700) is performed by a processing circuit such as a processing circuit that performs the functions of a video decoder (110), a processing circuit that performs the functions of a video decoder (210), and so on. In some embodiments, process (700) is implemented by a software instruction, and therefore, when a processing circuit executes a software instruction, the processing circuit performs process (700). The process starts at (S701) and proceeds to (S710).

[0128] (S710) receives the video bitstream, which includes the current block in the current picture.

[0129] In (S720), the reference block is determined for the current block from among several candidate reference blocks, based on the template matching (TM) costs of the candidate reference blocks. The TM cost represents the difference between the template of the current block and the reference template of each of the candidate reference blocks.

[0130] In (S730), the determined sample of the reference block is classified into several sample classes.

[0131] In (S740), the partitioning pattern of the current block is derived from a predetermined set of partitioning patterns based on the determined reference block. The derived partitioning pattern represents the set of partitions of the current block. Each of the set of samples of the determined reference block corresponds to each of the set of partitions of the current block.

[0132] In (S750), the current block is reconfigured based on the derived partitioning pattern of the current block.

[0133] In one embodiment, multiple candidate reference blocks are from the current picture and one of the reference pictures of the current block. The sample of the determined reference block is the reconstructed sample.

[0134] In one example, the initial reference block is determined based on motion vector information contained in the received video bitstream. Multiple candidate reference blocks are determined within the search range of the initial reference block. The TM cost between the reference template of each of the multiple candidate reference blocks and the template of the current block is determined. From the multiple candidate reference blocks, the reference block corresponding to the smallest TM cost among the determined TM costs between the reference template of each of the multiple candidate reference blocks and the template of the current block is determined.

[0135] In one example, the samples of the determined reference block are classified into a first sample class and a second sample class based on a binary image segmentation algorithm.

[0136] In one example, the samples of the determined reference block are classified into a first sample class with sample values ​​greater than a threshold, and a second sample class with sample values ​​less than a threshold. The threshold is either the mean or the median of the samples.

[0137] In one example, the determined reference block samples are clustered into one or more sample classes using a clustering method.

[0138] In one example, the template region adjacent to the determined reference block is determined. The samples of the determined reference block are classified into one or more sample classes in edge detection based on the sample values ​​of the template region adjacent to the determined reference block. The one or more sample classes include at least a first class containing samples at the edges of the determined reference block and a second class containing samples in the internal region of the determined reference block.

[0139] In one example, the template region includes one of the following: (i) multiple rows of neighbor samples above the determined reference block, (ii) multiple columns of neighbor samples to the left of the determined reference block, or (iii) a region in one or more reconstructed neighbor blocks of the determined reference block.

[0140] In one example, a sample of the determined reference block is classified based on each of several partitioning boundary candidates. Several partitioning boundary candidates for the current block are determined, and each of the several partitioning boundary candidates for the current block corresponds to each of the several partitioning boundary candidates for the determined reference block. The TM cost is determined between each of the several partitioning boundary candidates for the sample of the determined reference block and the corresponding one of the several partitioning boundary candidates for the sample of the current block. From the several partitioning boundary candidates for the current block, the partitioning boundary corresponding to the smallest TM cost among the determined TM costs is determined. Based on the determined partitioning boundary, several partitions of the current block are determined.

[0141] In one example, the binary mask is determined based on either the mean or median of the reconstructed sample of the current block's template. Based on the binary mask, the dominant sample group in the reconstructed sample of the current block's template is determined. Based on the binary mask, the dominant sample group in the sample of each of the candidate reference blocks' reference templates is determined. The TM cost between the dominant group in the reconstructed sample of the current block's template and the dominant group in the sample of each of the candidate reference blocks' reference templates is determined. From the candidate reference blocks, the reference block corresponding to the smallest TM cost among the determined TM costs between the dominant group in the reconstructed sample of the current block's template and the dominant group in the sample of each of the candidate reference blocks' reference templates is determined.

[0142] In one example, multiple candidate reference blocks are determined based on multiple motion vectors currently associated with the block.

[0143] In one example, the TM cost between the reference template of each of the multiple candidate reference blocks and the template of the current block is determined. From the multiple candidate reference blocks, the candidate reference block corresponding to the smallest TM cost among the determined TM costs between the reference template of each of the multiple candidate reference blocks and the template of the current block is determined. Based on the determined candidate reference block, a motion vector (MV) is determined. The motion vector indicates the offset between the current block and the determined candidate reference block. Based on the determined MV and the motion vector difference (MVD), an adjusted MV is determined. Based on the adjusted MV, a reference block is determined.

[0144] In one example, multiple candidate reference blocks are determined in the current picture. The TM cost between the reference template of each of the multiple candidate reference blocks and the template of the current block is determined. From the multiple candidate reference blocks, the candidate reference block corresponding to the smallest TM cost among the determined TM costs between the reference template of each of the multiple candidate reference blocks and the template of the current block is determined. A block vector (BV) is determined based on the determined candidate reference block, and the BV indicates the offset between the current block and the determined candidate reference block. The reference block is determined based on the determined BV.

[0145] Next, the process proceeds to (S799) and terminates.

[0146] Process (700) can be appropriately adapted. The steps of process (700) can be changed and / or omitted. Additional steps can be added. Any appropriate implementation order can be used.

[0147] Figure 8 shows a flowchart illustrating an overview of process (800) according to one embodiment of the present disclosure. Process (800) can be used in a video encoder. In various embodiments, process (800) is performed by a processing circuit such as a processing circuit that performs the functions of a video encoder (103), a processing circuit that performs the functions of a video encoder (303), and so on. In some embodiments, process (800) is implemented by a software instruction, and so the processing circuit performs process (800) when it executes the software instruction. The process starts at (S801) and proceeds to (S810).

[0148] In (S810), a reference block is determined from among the multiple candidate reference blocks of the current block in the current picture based on the TM cost of the multiple candidate reference blocks. Here, the TM cost represents the difference between the template of the current block and the reference template of each of the multiple candidate reference blocks.

[0149] In (S820), the reconstructed sample of the reference block determined is classified into several reconstructed sample classes.

[0150] In (S830), multiple partitions of the current block corresponding to multiple reconstructed sample classes of the determined reference block are determined. Here, each of the multiple reconstructed sample classes of the determined reference block corresponds to each of the multiple partitions of the current block.

[0151] (S840) The current block is reconfigured based on the multiple partitions determined for the current block.

[0152] Next, the process proceeds to (S899) and terminates.

[0153] Process (800) can be appropriately adapted. The steps of process (800) can be changed and / or omitted. Additional steps can be added. Any appropriate implementation order can be used.

[0154] The above-described technology can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 9 shows a computer system (900) suitable for implementing a particular embodiment of the subject matter of this disclosure.

[0155] Computer software can be coded using any suitable machine code or computer language that can be processed by mechanisms such as assembly, compilation, and linking to generate code containing instructions that can be executed directly or through interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0156] The instructions can be executed on various computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, Internet of Things devices, etc.

[0157] The components shown in Figure 9 of the computer system (900) are illustrative and do not imply any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Furthermore, the configuration of the components should not be construed as having any dependencies or requirements relating to any one or combination of the components shown in the exemplary embodiments of the computer system (900).

[0158] The computer system (900) may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users, for example, through sensory input (e.g., keystrokes, swipes, data grab actions), voice input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media that do not necessarily need to be directly related to conscious human input, such as voice (e.g., conversation, music, ambient sounds), images (e.g., scanned images, photographic images taken from a digital camera), or video (e.g., including 2D video, 3D video, and stereoscopic video).

[0159] The input human interface device may include one or more of the following (only one is shown): keyboard (901), mouse (902), trackpad (903), touchscreen (910), data grab (not shown), joystick (905), microphone (906), scanner (907), and camera (908).

[0160] The computer system (900) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through sensory output, sound, light, and smell / taste. Such human interface output devices may include sensory output devices (e.g., touchscreen (910), data grab (not shown), or joystick (905) (for sensory feedback, however, there may also be sensory feedback devices that do not function as input devices), sound output devices (e.g., speaker (909), headphones (not shown)), visual output devices (e.g., screen (910), CRT screen, LCD screen, plasma screen, OLED screen, each having or not having touchscreen input capability, each having or not having sensory feedback capability, some of which may be capable of outputting two-dimensional visual output or three-dimensional or more output through means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), and printers (not shown))).

[0161] The computer system (900) may also include human-accessible storage devices, and related media such as optical media including CD / DVD ROM / RW (920) with media such as CD / DVD (921), thumb drives (922), removable hard drives or solid drives (923), legacy magnetic media such as tapes and floppy disks (not shown), and devices based on dedicated ROM / ASIC / PLD such as security dongles (not shown).

[0162] Those skilled in the art should also understand that the term “computer-readable medium” as used in connection with the subject matter of this disclosure does not include a transmission medium, carrier wave, or other transient signal.

[0163] The computer system (900) may also include an interface (954) to one or more communication networks (955). These networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, urban, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicle and industrial networks including CANBus, etc. Certain networks generally require an external network interface attached to a specific general-purpose data port or peripheral bus (949) (e.g., a USB port on the computer system (900)). Others are generally integrated into the core of the computer system (900) by attachment to a system bus, as described later (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). These networks can be used by the computer system (900) to communicate with other entities. Such communications can be unidirectional (e.g., broadcast television), unidirectional (e.g., CANbus to a specific CANbus device), or bidirectional (e.g., bidirectional to other computer systems using a local or wide-area digital network). Specific protocols and protocol stacks may be used in each of the aforementioned networks and network interfaces.

[0164] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core (940) of the computer system (900).

[0165] The core (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), dedicated programmable processing units in the form of FPGAs (943), hardware accelerators for specific tasks (944), graphics adapters (950), etc. These devices may be connected via a system bus (948) along with read-only memory (ROM) (945), random access memory (946), and internal mass storage devices (947) such as internal, user-inaccessible hard drives, SSDs, etc. In some computer systems, the system bus (948) is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals can be attached directly to the core's system bus (948) or via a peripheral bus (949). In the example, a screen (910) can be connected to the graphics adapter (950). The architecture of the peripheral bus includes PCI, USB, etc.

[0166] The CPU (941), GPU (942), FPGA (943), and accelerator (944) can execute specific instructions that, when combined, can generate the aforementioned computer code. This computer code can be stored in ROM (945) or RAM (946). Temporary data can also be stored in RAM (946), while permanent data can be stored, for example, in a built-in mass storage device (947). High-speed storage and retrieval to any of the memory devices can be enabled through the use of cache memory that may be closely associated with one or more of the CPU (941), GPU (942), mass storage device (947), ROM (945), RAM (946), etc.

[0167] Computer-readable media may contain computer code for performing actions performed by various computers. The media and computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type well known and available to those skilled in the computer software field.

[0168] As an example and not limited thereto, a computer system (900) having an architecture, and specifically a core (940), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be specific storage devices of the core 940 having non-transient characteristics, such as a core-integrated mass storage device 947 or ROM 945, and media associated with user-accessible mass storage devices as described above. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (940). The computer-readable media may include one or more memory devices or chips, depending on the specific needs. The software can cause the core (940) and specifically the processor (including a CPU, GPU, FPGA, etc.) therein to execute specific processes or specific parts of specific processes described herein, including defining and modifying data structures stored in RAM (946) according to software-defined processes. As an addition or alternative, a computer system may provide functionality as a result of a logic hardwired or other circuit implementation (e.g., an accelerator (944)) that can operate together with or in place of the software to perform the specific processes or specific parts of the specific processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuits (such as integrated circuits (ICs)) that house software for execution, circuits that implement logic for execution, or both. This disclosure includes any appropriate combination of hardware and software.

[0169] The use of “at least one” or “one” in this disclosure is intended to include any one or combination of the elements described. For example, references to at least one of A, B, or C; at least one of A, B, and C; at least one of A, B, and / or C; and at least one of A-C are intended to include A only, B only, C only, or any combination thereof. References to one of A or B, and one of A and B are intended to include A or B or (A and B). The use of “one” does not exclude any combination of the enumerated elements where applicable, such as when the elements are not mutually exclusive.

[0170] While this disclosure describes several exemplary embodiments, alternatives, substitutions, and various equivalents exist and are included within the scope of this disclosure. As will be apparent to those skilled in the art, numerous systems and methods can be devised to implement the principles of this disclosure and thus fall within the spirit and scope of this disclosure, although these are not expressly shown or described herein.

Claims

1. A method for video decoding, The steps include receiving a video bitstream containing the current block in the current picture, A step of determining a reference block for the current block from among the multiple candidate reference blocks based on the template matching (TM) costs of the multiple candidate reference blocks, wherein the TM cost represents the difference between the template of the current block and the reference template of each of the multiple candidate reference blocks. The steps include classifying the determined reference block samples into multiple sample classes, A step of deriving a partitioning pattern for the current block based on the determined reference block from a predetermined plurality of partitioning patterns, wherein the derived partitioning pattern represents a plurality of partitions of the current block, and each of the plurality of sample classes of the determined reference block corresponds to each of the plurality of partitions of the current block. The steps include: reconfiguring the current block based on the derived partitioning pattern of the current block; A method that includes this.

2. The aforementioned multiple candidate reference blocks are from the current picture and one of the reference pictures of the current block. The method according to claim 1, wherein the sample of the determined reference block is a reconstructed sample.

3. The step of determining the reference block from the plurality of candidate reference blocks further includes: The steps include determining an initial reference block based on motion vector information contained in the received video bitstream, The steps include determining the plurality of candidate reference blocks that are within the search range of the initial reference block, The steps include determining the TM cost between each of the reference templates of the plurality of candidate reference blocks and the template of the current block, The steps include determining the reference block corresponding to the minimum TM cost among the determined TM costs between the reference template of each of the candidate reference blocks and the template of the current block, from among the plurality of candidate reference blocks, The method according to claim 2, including the method described in claim 2.

4. The step of classifying the sample of the determined reference block is further: The method according to claim 1, further comprising the step of classifying the determined reference block samples into a first sample class and a second sample class based on a binary image segmentation algorithm.

5. The step of classifying the sample of the determined reference block is further: The method according to claim 1, comprising the step of classifying the determined reference block samples into a first sample class having sample values ​​greater than a threshold and a second sample class having sample values ​​less than a threshold, wherein the threshold is one of the mean value of the samples and the median value of the samples.

6. The step of classifying the sample of the determined reference block is further: The method according to claim 1, comprising the step of clustering the determined reference block samples into one or more sample classes.

7. The step of classifying the sample of the determined reference block is further: The steps include determining a template region adjacent to the aforementioned determined reference block, In edge detection, a step of classifying the samples of the determined reference block into one or more sample classes based on the sample values ​​of the template region adjacent to the determined reference block, wherein the one or more sample classes include at least a first class that includes samples at the edges of the determined reference block and a second class that includes samples in the internal region of the determined reference block. The method according to claim 1, including the method described in claim 1.

8. The method according to claim 7, wherein the template region includes one of (i) a plurality of rows of neighbor samples above the determined reference block, (ii) a plurality of columns of neighbor samples to the left of the determined reference block, or (iii) a region in one or more reconstructed neighbor blocks of the determined reference block.

9. The step of classifying the determined reference block samples further includes the step of classifying the determined reference block samples based on each of a plurality of partitioning boundary candidates, The step of deriving the partitioning pattern of the current block is: A step of determining a plurality of partitioning boundary candidates for the current block, wherein each of the plurality of partitioning boundary candidates for the current block corresponds to each of the plurality of partitioning boundary candidates for the determined reference block, The steps include determining the TM cost between each of the plurality of partitioning boundary candidates for the sample of the determined reference block and the corresponding one of the plurality of partitioning boundary candidates for the sample of the current block, The steps include determining a partitioning boundary corresponding to the minimum TM cost among the determined TM costs from the plurality of partitioning boundary candidates of the current block, The steps include determining the plurality of partitions of the current block based on the determined partitioning boundary, The method according to claim 1, further comprising:

10. The step of determining a reference block from the aforementioned multiple candidate reference blocks is: The steps include determining a binary mask based on one of the mean and median values ​​of the reconstruction samples of the current block template, A step of determining the dominant sample group in the reconstructed sample of the template of the current block based on the binary mask, The steps include determining the dominant sample group in the samples of each of the multiple candidate reference blocks based on the binary mask, The steps include determining the TM cost between the dominant group in the reconstructed sample of the current block template and the dominant group in the sample of each of the candidate reference blocks' reference templates, The steps include determining, from the plurality of candidate reference blocks, the reference block corresponding to the minimum TM cost among the determined TM costs between the dominant group in the reconstructed sample of the template of the current block and the dominant group in the sample of each of the candidate reference blocks' reference templates, The method according to claim 1, including the method described in claim 1.

11. The step of determining a reference block from the aforementioned multiple candidate reference blocks is: The method according to claim 1, further comprising the step of determining a plurality of candidate reference blocks based on a plurality of motion vectors associated with the current block.

12. The step of determining a reference block from the aforementioned multiple candidate reference blocks is: The steps include determining the TM cost between each of the reference templates of the plurality of candidate reference blocks and the template of the current block, The steps include determining a candidate reference block from the plurality of candidate reference blocks that corresponds to the minimum TM cost among the determined TM costs between the reference template of each of the plurality of candidate reference blocks and the template of the current block, A step of determining a motion vector (MV) based on the determined candidate reference block, wherein the motion vector indicates an offset between the current block and the determined candidate reference block. The steps include determining the MV adjusted based on the aforementioned determined MV and motion vector difference (MVD), The steps include determining the reference block based on the adjusted MV, The method according to claim 1, further comprising:

13. The aforementioned multiple candidate reference blocks are determined within the current picture. The step of determining a reference block from the aforementioned multiple candidate reference blocks is: The steps include determining the TM cost between each of the reference templates of the plurality of candidate reference blocks and the template of the current block, The steps include determining a candidate reference block from the plurality of candidate reference blocks that corresponds to the minimum TM cost among the determined TM costs between the reference template of each of the plurality of candidate reference blocks and the template of the current block, A step of determining a block vector (BV) based on the determined candidate reference block, wherein the BV indicates an offset between the current block and the determined candidate reference block. The steps include determining the reference block based on the BV determined above, The method according to claim 2, further comprising:

14. It is a device, It includes a processing circuit, and the processing circuit is Currently receiving a video bitstream containing the current block in the picture, Based on the template matching (TM) cost of multiple candidate reference blocks, a reference block is determined for the current block from the multiple candidate reference blocks, and the TM cost represents the difference between the template of the current block and each of the reference templates of the multiple candidate reference blocks. The samples of the reference blocks determined above are classified into several sample classes. A partitioning pattern for the current block is derived from a predetermined plurality of partitioning patterns based on the determined reference block, the derived partitioning pattern represents a plurality of partitions of the current block, and each of the plurality of sample classes of the determined reference block corresponds to each of the plurality of partitions of the current block. Based on the derived partitioning pattern of the current block, the current block is reconfigured. A device configured in such a way.

15. The aforementioned multiple candidate reference blocks are from the current picture and one of the reference pictures of the current block. The apparatus according to claim 14, wherein the sample of the determined reference block is a reconstructed sample.

16. The aforementioned processing circuit is The initial reference block is determined based on the motion vector information contained in the received video bitstream. Determine the plurality of candidate reference blocks that are within the search range of the initial reference block, Determine the TM cost between each of the reference templates of the plurality of candidate reference blocks and the template of the current block. From the plurality of candidate reference blocks, the reference block corresponding to the smallest TM cost among the determined TM costs between the reference template of each of the plurality of candidate reference blocks and the template of the current block is determined. The apparatus according to claim 15, configured as described above.

17. The aforementioned processing circuit is The apparatus according to claim 14, configured to classify the determined reference block samples into a first sample class and a second sample class based on a binary image segmentation algorithm.

18. The aforementioned processing circuit is The apparatus according to claim 14, wherein the samples of the determined reference block are classified into a first sample class having sample values ​​greater than a threshold and a second sample class having sample values ​​less than a threshold, and the threshold is configured to be one of the mean value of the samples and the median value of the samples.

19. The aforementioned processing circuit is The apparatus according to claim 14, wherein the clustering method is configured to cluster the samples of the determined reference block into one or more sample classes.

20. The aforementioned processing circuit is Determine the template region adjacent to the aforementioned determined reference block, The apparatus according to claim 14, wherein, in edge detection, the samples of the determined reference block are classified into one or more sample classes based on the sample values ​​of the template region adjacent to the determined reference block, and the one or more sample classes are configured to include at least a first class that includes samples at the edges of the determined reference block and a second class that includes samples in the internal region of the determined reference block.

21. A method of video encoding, A step of determining a reference block for the current block in the current picture from among multiple candidate reference blocks based on the template matching (TM) costs of multiple candidate reference blocks, wherein the TM cost represents the difference between the template of the current block and the reference template of each of the multiple candidate reference blocks. The steps include classifying the reconstructed samples of the determined reference block into multiple reconstructed sample classes, A step of determining a plurality of partitions of the current block corresponding to the plurality of reconstructed sample classes of the determined reference block, wherein each of the plurality of reconstructed sample classes of the determined reference block corresponds to each of the plurality of partitions of the current block. The steps of encoding the current block based on the determined plurality of partitions of the current block, A method that includes this.

Images