Merge index sorting based on template matching for Geometric Partition Mode (GPM)

Abstract

The method includes: dividing the current block into a first partition and a second partition; dividing the template sample of the current block into a first template region adjacent to the first partition and a second template region adjacent to the second partition; determining a plurality of first candidate reference blocks for the first partition; determining a plurality of second candidate reference blocks for the second partition; and reordering at least one of the plurality of first candidate reference blocks and the plurality of second candidate reference blocks based on the size of the first template region of the template sample and the size of the second template region of the template sample. The current block is reconstructed based on the received index value and based on the reordered at least one of the plurality of first candidate reference blocks and the plurality of second candidate reference blocks.

Description

[Technical Field]

[0001] [Related applications] This application claims priority to U.S. Patent Application No. 18 / 241,058, "TEMPLATE-MATCHING BASED MERGE INDEX REORDERING FOR GEOMETRIC PARTITION MODE (GPM)," filed August 31, 2023, which in turn claims priority to U.S. Provisional Patent Application No. 63 / 416,401, "Template-Matching Based Merge Index Reordering For Geometric Partition Mode (GPM)," filed October 14, 2022. The disclosures of the aforementioned applications are incorporated herein by reference in their entirety.

[0002] [Technical field] This disclosure generally describes embodiments related to video coding. [Background technology]

[0003] The background information provided herein is intended to provide a general overview of the background to this disclosure. The research of the inventors named herein, to the extent of the research described in this background section, is not expressly or implicitly recognized as prior art to this disclosure, as is the case with respect to any form of description that may not be considered prior art at the time of filing.

[0004] Image / video compression helps transfer image / video data across various devices, storage, and networks with minimal loss of quality. In some cases, video codec techniques can compress video based on spatial and temporal redundancy. For example, a video codec can use a technique called intra-prediction, which can compress images based on spatial redundancy. For example, intra-prediction can use reference data from the current picture being reconstructed for sample prediction. In another example, a video codec can use a technique called inter-prediction, which can compress images based on temporal redundancy. For example, inter-prediction can use motion compensation to predict samples in the current picture from previously reconstructed pictures. Motion compensation is generally represented by motion vectors (MV). [Overview of the project]

[0005] Aspects of this disclosure include methods and apparatus for video coding / decoding. In some examples, the apparatus for video decoding includes processing circuits.

[0006] A part of this disclosure provides a method for video decoding to be performed in a video decoder. In this method, a video bitstream is received, which includes a current block in the current frame and a template sample of the current block. The template sample includes an upper template along the top of the current block and a left template along the left side of the current block. The current block is divided into a first partition and a second partition, and the template sample is divided into a first template region adjacent to the first partition and a second template region adjacent to the second partition. A plurality of first candidate reference blocks are determined for the first partition, and a plurality of second candidate reference blocks are determined for the second partition. Each first candidate reference block has a first candidate template corresponding to the shape of the first template region, and each second candidate reference block has a second candidate template corresponding to the shape of the second template region. At least one of the plurality of first candidate reference blocks and the plurality of second candidate reference blocks is sorted based on the size of the first template region and the size of the second template region of the template sample. The current block is reconstructed based on the index value received in the bitstream and on a reordered sequence of at least one of a plurality of first candidate reference blocks and a plurality of plurality of second candidate reference blocks.

[0007] In one example, multiple first candidate reference blocks in the first partition of the current block and multiple second candidate reference blocks in the second partition of the current block are determined based on candidate reference blocks indicated by motion vectors (MVs) in the first and second reference lists.

[0008] In one example, multiple first candidate reference blocks in the first partition of the current block and multiple second candidate reference blocks in the second partition of the current block are determined based on one of the blocks in the same-position reference frame, indicated by non-adjacent blocks and temporal motion vectors.

[0009] In one embodiment, the template-matching (TM) difference between each of the first template regions of the template sample and the first candidate templates is determined. The TM difference between each of the second template regions of the template sample and the second candidate templates is determined. Multiple first candidate reference blocks for the first partition are sorted in ascending order of the TM difference between the first template region and the first candidate templates of the template sample. Multiple second candidate reference blocks for the second partition are sorted in ascending order of the TM difference between the second template region and the second candidate templates of the template sample.

[0010] In one embodiment, the upper template is divided into a first part and a second part, the first template area of ​​the template sample includes the left template and the first part of the upper template, and the second template area of ​​the template sample includes the second part of the upper template. Based on whether the number of samples in the sample rows of the first part of the template above is greater than the threshold, multiple first-choice reference blocks for the first partition are sorted. Based on whether the number of samples in the sample rows of the second part of the template above is greater than the threshold, multiple second-choice reference blocks for the second partition are sorted.

[0011] In one embodiment, the upper template is divided into a first part and a second part, the first template area of ​​the template sample includes the left template and the first part of the upper template, and the second template area of ​​the template sample includes the second part of the upper template. Based on the fact that the width of the sample rows in the first part of the template above is greater than the threshold, multiple first-choice reference blocks for the first partition are sorted. Based on the fact that the width of the sample rows in the second part of the template above is greater than the threshold, multiple second-choice reference blocks for the second partition are sorted.

[0012] In one embodiment, the left template is divided into a first part and a second part, the first template area of ​​the template sample includes the upper template and the first part of the left template, and the second template area of ​​the template sample includes the second part of the left template. Based on whether the number of samples in the sample column of the first part of the left template is greater than the threshold, multiple first-choice reference blocks for the first partition are ordered. Based on whether the number of samples in the sample column of the second part of the left template is greater than the threshold, multiple second-choice reference blocks for the second partition are sorted.

[0013] In one embodiment, the left template is divided into a first part and a second part, the first template area of ​​the template sample includes the upper template and the first part of the left template, and the second template area of ​​the template sample includes the second part of the left template. Based on whether the height of the sample column in the first part of the left template is greater than the threshold, multiple first-choice reference blocks for the first partition are sorted. Based on whether the height of the sample column in the second part of the left template is greater than the threshold, multiple second-choice reference blocks for the second partition are sorted.

[0014] In one embodiment, the TM difference between one of the undivided upper template or the undivided left template and each corresponding region of the first candidate template is determined based on a first template region of a template sample containing one of the undivided upper template or the undivided left template, and a second template region of a template sample containing the other of the divided upper template or the undivided left template. Furthermore, multiple first candidate reference blocks of the first partition are sorted in ascending order of the TM differences between one of the undivided upper template or the undivided left template and the corresponding region of the first candidate template.

[0015] In one example, based on a plurality of candidate partitioning methods, a plurality of candidate partitions of the current block are determined. For each of the plurality of candidate partitions, a plurality of candidate reference blocks are determined. A TM difference between the template sample of the current block and the template regions of the plurality of candidate reference blocks of the plurality of candidate partitions is determined. Based on the TM difference between the template sample of the current block and the template regions of the plurality of reference blocks of the current block, the plurality of candidate reference blocks are sorted in ascending order.

[0016] In one example, a video bitstream includes a plurality of indexes indicating a subset of the sorted plurality of candidate reference blocks, and the subset of the sorted plurality of candidate reference blocks includes candidate reference blocks having the first N smallest TM differences.

[0017] According to another aspect of the present disclosure, a device is provided. The device includes a processing circuit. The processing circuit can be configured to execute any of the described methods of video decoding / encoding.

[0018] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to execute a method for video decoding / encoding.

Brief Description of the Drawings

[0019] Further features, characteristics, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

[0020] [Figure 1] It is a schematic diagram of an exemplary block diagram of a communication system (100).

[0021] [Figure 2] It is a schematic diagram of an exemplary block diagram of a decoder.

[0022] [Figure 3] It is a schematic diagram of an exemplary block diagram of an encoder.

[0023] [Figure 4] This shows an exemplary partition angle distribution for geometrically partitioned mode (GPM) interpretation.

[0024] [Figure 5] This shows exemplary candidate partition lines associated with the GPM partition angle.

[0025] [Figure 6] This is a schematic diagram of the blending process at partition edges associated with GPM.

[0026] [Figure 7] The present disclosure provides a first example of a merge index sorting based on template matching for GPM according to several embodiments of this disclosure.

[0027] [Figure 8] A second example of a merge index sorting based on template matching for GPM, according to some embodiments of this disclosure, is shown.

[0028] [Figure 9] A third example of a merge index sorting based on template matching for GPM, according to several embodiments of this disclosure, is presented.

[0029] [Figure 10] A flowchart illustrating the decryption process according to several embodiments of this disclosure is shown.

[0030] [Figure 11] A flowchart illustrating the coding process according to several embodiments of this disclosure is shown.

[0031] [Figure 12] This is a schematic diagram of an exemplary computer system according to one embodiment. [Modes for carrying out the invention]

[0032] Figure 1 shows a block diagram of a video processing system (100) in several examples. The video processing system (100) is a video encoder and video decoder in a streaming environment, which is an example of the application of the subject matter of disclosure. The subject matter of disclosure is equally applicable to, for example, video conferencing, digital TV, streaming services, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc., and other video-enabled applications.

[0033] The video processing system (100) includes a capture subsystem (113) which may include a video source (101), such as a digital camera, that generates, for example, an uncompressed video picture stream (102). In one example, the video picture stream (102) includes samples captured by the digital camera. The video picture stream (102) is shown in thick lines to emphasize its high data volume compared to encoded video data (104) (or encoded video bitstream) and may be processed by an electronic device (120) which includes a video encoder (103) coupled with the video source (101). The video encoder (103) may include hardware, software, or a combination thereof, and may enable or implement aspects of the subject matter of disclosure as detailed below. The encoded video data (104) (or encoded video bitstream) is shown in thin lines to emphasize its low data volume compared to the video picture stream (102) and may be stored in a streaming server (105) for future use. One or more streaming client subsystems, such as client subsystems (106) and (108) in Figure 1, can access a streaming server (105) to read copies (107) and (109) of encoded video data (104). Client subsystem (106) may include a video decoder (110) within, for example, an electronic device (130). The video decoder (110) decodes the input copy (107) of the encoded video data and generates an output video picture stream (111) that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., video bitstreams) may be encoded according to a specific video coding / compression standard. An example of these standards is ITU-T Recommendation H.265. For example, a video coding standard under development is informally known as VVC (Versatile Video Coding). The subject of disclosure may be used in the context of VVC.

[0034] It should be noted that electronic devices (120) and (130) may include other components (not shown). For example, electronic device (120) may include a video decoder (not shown), and electronic device (130) may also include a video encoder (not shown).

[0035] Figure 2 shows an exemplary block diagram of a video decoder (210). The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., a receiving circuit). In the example of Figure 1, the video decoder (210) can be used instead of the video decoder (110).

[0036] The receiver (231) can receive one or more coded video sequences, for example, contained in a bitstream, which are decoded by the video decoder (210). In embodiments, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device that stores coded video data. The receiver (231) may receive coded video data together with other data, for example, coded audio data and / or auxiliary data streams that may be transferred to their respective usage entities (not shown). The receiver (231) may isolate the coded video sequences from other data. To eliminate network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter, "Parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). Alternatively, a buffer memory (not shown) may exist outside the video decoder (210) (not shown). Further, a buffer memory (not shown) may exist outside the video decoder (210), for example, to eliminate network jitter, or in addition to another buffer memory (215) inside the video decoder (210), for example, to handle playback timing. When the receiver (231) receives data controllably from a storage / transmission device with sufficient bandwidth, or from an isosynchronous network, the buffer memory (215) may not be necessary or can be made small. For use in best-effort packet networks such as the Internet, the buffer memory (215) may be required, may be relatively large, advantageously adaptive in size, and may be implemented at least partially in the operating system or a similar element (not shown) outside the video decoder (210).

[0037] The video decoder (210) may include a parser (220) to reconstruct symbols (221) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (210), and information for controlling rendering devices, such as a renderer (212) (e.g., a display screen), which may not be an integrated part of the electronic device (230) but may be coupled to the electronic device (230), as shown in Figure 2. The control information for the rendering device may be in the form of SEI (Supplemental Enhancement Information) messages or VUI (Video Usability Information) parameter set fragments (not shown). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, context-dependent or non-context-dependent arithmetic coding, etc. The parser (220) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to that group. Subgroups may include GOP (Groups of Picture), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (220) may also extract information such as transformation coefficients, quantization parameter values, motion vectors, etc., from the coded video sequence.

[0038] The parser (220) may perform an entropy decoding / parse operation on the video sequence received from the buffer memory (215) to generate a symbol (221).

[0039] The reconstruction of symbol (221) may include multiple different units, depending on the type of coded video picture or part thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. Which units are included and how they are included can be controlled by group control information parsed by parser (220) from the coded video sequence. The flow of such subgroup control information between parser (220) and the following multiple units is not shown for clarity.

[0040] Beyond the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into numerous functional units, as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with each other and be at least partially integrated. However, for the purpose of illustrating the subject of this disclosure, the following conceptual subdivision into functional units is appropriate.

[0041] The first unit is the scaler / inverse unit 251. The scaler / inverse unit (251) receives control information from the parser (220) as symbols (221), including quantized transformation coefficients and which transformation to use, block size, quantization coefficients, quantization scaling matrix, etc. The scaler / inverse unit (251) can output a block containing sample values ​​that can be input to the aggregator (255).

[0042] In some cases, the output samples of the scaler / inverse unit (251) may relate to intracoded blocks. Intracoded blocks are blocks that do not use prediction information from previously reconstructed pictures but can use prediction information from portions reconstructed before the current picture. Such prediction information can be provided by the intrapicture prediction unit (252). In some cases, the intrapicture prediction unit (252) generates a block of the same size and shape as the block being reconstructed, using the surrounding already reconstructed information fetched from the current picture buffer (258). The current picture buffer (258) buffers, for example, the reconstructed current picture partially and / or completely. In some cases, the aggregator (255) adds the prediction information generated by the intraprediction unit (252) to the output sample information provided by the scaler / inverse unit (251) on a sample-by-sample basis.

[0043] In other cases, the output samples of the scaler / inverse unit (251) may be associated with an interconnected, and possibly motion-compensated, block. In such cases, the motion-compensated prediction unit (253) can access the reference picture memory (257) to fetch samples to be used for prediction. After motion-compensating the fetched samples according to the symbols (221) associated with the block, these samples may be added to the output of the scaler / inverse unit (251) by the aggregator (255) to generate output sample information (in this case, called residual samples or residual signals). The addresses in the reference picture memory (257) from which the motion-compensated prediction unit (253) fetches prediction samples can be controlled by the motion vectors available to the motion-compensated prediction unit (253), for example, in the form of symbols (221) which may have X, Y, and reference picture components. Motion compensation may include interpolation of sample values ​​fetched from the reference picture memory (257) when the exact motion vectors of subsamples are in use, motion vector prediction mechanisms, etc.

[0044] The output samples of the aggregator (255) can undergo various loop filtering techniques in the loop filter unit (256). Video compression techniques may include in-loop filtering techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream), which are made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression may respond not only to previously reconstructed and loop-filtered sample values, but also to metadata obtained during decoding of earlier portions (in decoding order) of the coded picture or coded video sequence.

[0045] The output of the loop filter unit (256) may be a sample stream that can be output to the renderer (212) and stored in the reference picture memory (257) for use in future interpicture prediction.

[0046] A specific coded picture, once fully reconfigured, can be used as a reference picture for future predictions. For example, once the coded picture corresponding to the current picture is fully reconfigured and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) can become part of the reference picture memory (257) and a fresh current picture buffer can be reallocated before starting the reconfiguration of subsequent coded pictures.

[0047] The video decoder (210) may perform decoding operations in accordance with a standard such as ITU-T Rec.H.265 or a specified video compression technology. The coded video sequence may follow the syntax specified by the video compression technology or standard in use, in the sense that the coded video sequence conforms to both the video compression technology or standard and the profile documented in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as tools that are only usable under the profile. Also, compliance may require that the complexity of the coded video sequence be within the limits set by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further restricted through the HRD (Hypothetical Reference Decoder) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0048] In the embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be included as a portion of the coded video sequence. The additional data may be used by the video decoder (210) to correctly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) extension layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0049] Figure 3 shows an exemplary block diagram of a video encoder (303). The video encoder (303) is included in the electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmitting circuit). The video encoder (303) can be used in place of the video encoder (103) in the example of Figure 1.

[0050] The video encoder (303) may receive video samples from a video source (301) (not part of the electronic device (320) in the example in Figure 3) that can capture video images to be coded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0051] The video source (301) may provide a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream of any appropriate bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any appropriate sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media delivery system, the video source (301) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a series of individual pictures that give motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels. Each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. The following description will focus on samples.

[0052] According to the embodiment, the video encoder (303) may encode and compress the pictures of the source video sequence into a coded video sequence (343) in real time or under any other required time constraints. Implementing an appropriate coding speed is one function of the control unit (350). In some embodiments, the control unit (350) controls and is functionally coupled to other functional units, which are described later. The couplings are not shown for clarity. Parameters set by the control unit (350) may include rate control-related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique, ...), picture size, GOP (group of pictures) layout, maximum motion vector search range, etc. The control unit (350) may be configured to have other appropriate functions related to the video encoder (303) optimized for a particular system design.

[0053] In some embodiments, the video encoder (303) is configured to operate within a coding loop. In a very simplified explanation, one example might include a source coder (330) (responsible for generating symbols, such as a symbol stream, based on the input and reference pictures to be coded), and a (local) decoder (333) built into the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a similar manner to that created by a (remote) decoder. The reconstructed sample stream (sample data) is fed into the reference picture memory (334). The contents of the reference picture memory (334) are also bit-accurate between the local and remote encoders, when decoding the symbol stream yields bit-accurate results independently of the decoder location (local or remote). In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the decoder "sees" when using predictions during decoding, as reference picture samples. This fundamental principle of reference picture synchronization (and the resulting drift, for example, when synchronization cannot be maintained due to channel errors) is used similarly in several related technologies.

[0054] The operation of a “local” decoder (333) may be the same as that of a “remote” decoder, such as the video decoder (210) detailed above in relation to Figure 2. However, as also briefly referring to Figure 2, since symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropy coder (345) and parser (220) may be lossless, the entropy decoding section of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333).

[0055] In the embodiments, the decoder techniques present in the decoder, excluding analysis / entropy decoding, are present in the corresponding encoder in the same or substantially the same functional form. Therefore, the subject matter of the disclosed information focuses on the decoder operation. A description of the encoder techniques can be omitted, as they are the reverse of the decoder techniques from which they are comprehensively described. More detailed descriptions in specific areas are provided below.

[0056] During operation, in some examples, the source coder (330) may perform motion-compensated predictive coding. This predictively codes the input picture by referencing one or more previously coded pictures from a video sequence designated as “reference pictures”. In this method, the coding engine (332) codes the difference between the pixel blocks of the input picture and the pixel blocks of the reference picture which may be selected as the predictive criterion for the input picture.

[0057] The local video decoder (333) may decode the coded video data of a picture that may be designated as a reference picture, based on symbols generated by the source coder (330). The operation of the coding engine (332) may, advantageously, be lossy. When the coded video data can be decoded by a video decoder (not shown in Figure 3), the reconstructed video sequence may, as a standard, be a copy of the source video sequence with some errors. The local video decoder (333) may duplicate the decoding process that may be performed by the video decoder on the reference picture, resulting in a reconstructed reference picture to be stored in the reference picture memory (334). Thus, the video encoder (303) may store a copy of the reconstructed reference picture that has the same content as the reconstructed reference picture obtained by the far-end video decoder (if there are no transmission errors).

[0058] The predictor (335) may perform a predictive search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) may search the reference picture memory (334) for sample data (such as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., which could serve as appropriate predictive criteria for the new picture. The predictor (335) may operate sample block-pixel block by sample block to find appropriate predictive criteria. In some examples, the input picture may have predictive criteria drawn from multiple reference pictures stored in the reference picture memory (334), as determined by the search results obtained by the predictor (335).

[0059] The control unit (350) may manage the coding operations of the source coder (330), including, for example, setting parameters and subgroup parameters used for encoding video data.

[0060] The outputs of all the aforementioned functional units may undergo entropy coding in the entropy coder (345). The entropy coder (345) converts the symbols generated by the various functional units into coded video sequences by applying lossless compression to the symbols according to techniques such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0061] The transmitter (340) may buffer the coded video sequence generated by the entropy coder (345) in preparation for transmission over a communication channel (360), which may be a hardware / software link to a storage device capable of storing coded video data. The transmitter (340) may merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0062] The control unit (350) may manage the operation of the video encoder (303). During coding, the control unit (350) may assign each coded picture a specific coded picture type that may affect the coding techniques that can be applied to each picture. For example, a picture may often be assigned as one of the following picture types:

[0063] An intra-picture (I-picture) may be encoded and decoded without using any other pictures in the sequence as a source for prediction. Some video codecs allow different types of intra-pictures, including, for example, IDR (Independent Decoder Refresh) pictures.

[0064] A predictive picture (P-picture) may, in most cases, be a picture that can be coded and decoded using intra-prediction or inter-prediction with motion vectors and reference indices to predict the sample values ​​of each block.

[0065] A bidirectionally predictive picture (B-picture) may be coded and decoded using intra-prediction or inter-prediction with two motion vectors and a reference index to predict the sample values ​​for each block. Similarly, a multi-predictive picture may use two or more reference pictures and associated metadata for the reconstruction of a single block.

[0066] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block may be coded. Blocks may be coded predictively by references to other (already coded) blocks, determined by the coding assignments applied to each picture in the block. For example, blocks of picture I may be coded non-predictively, or they may be coded predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be coded predictively via spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks of picture B may be coded predictively via spatial prediction or temporal prediction by referencing one or two previously coded reference pictures.

[0067] The video encoder (303) may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Rec.H.265. In these operations, the video encoder (303) may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. The coded video data may, therefore, conform to the syntax specified by the video coding technique or standard being used.

[0068] In one embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the encoded video sequence. The additional data may include time / space / SNR extension layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0069] Video may be captured as multiple source pictures (video pictures) in a time series. Intra-picture prediction (sometimes abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes (temporal or other) correlations between pictures. In one example, a particular picture being encoded / decoded is called the current picture and is partitioned into blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension to identify the reference picture if multiple reference pictures are in use.

[0070] In some embodiments, bi-prediction techniques can be used for interpicture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both of which are earlier in the video in decoding order than the current picture (but may be earlier and later in display order, respectively). Blocks in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Blocks can be predicted by combining the first and second reference blocks.

[0071] Furthermore, merge mode techniques can be used in interpicture prediction to improve coding efficiency.

[0072] According to some embodiments of this disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed within units of blocks. For example, according to the HEVC standard, pictures in a video-picture sequence are partitioned into coding tree units (CTUs) for compression. CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Typically, a CTU contains three coding tree blocks (CTBs), i.e., one lumen CTB and two chroma CTBs. Each CTU can be recursively quad-tree partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be partitioned into one 64x64 pixel CU, or four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, such as inter-prediction type or intra-prediction type. A CU is divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Typically, each PU contains a chroma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed within the units of the prediction block. Using a chroma prediction block as an example of a prediction block, the prediction block contains a matrix of values ​​(e.g., chroma values) for pixels such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0073] It should be noted that the video encoders (103) and (303), and the video decoders (110) and (210) can be implemented using any suitable technique. In one embodiment, the video encoders (103) and (303), and the video decoders (110) and (210) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303), and the video decoders (110) and (210) can be implemented using one or more processors that execute software instructions.

[0074] This disclosure includes aspects relating to sorting the merge index of merge MVs associated with partitions in geometric partition mode (GPM) based on template matching processing.

[0075] GPM in VVCs and similar systems can be applied to inter-prediction. GPM can only be applied to 8x8 or larger CUs. GPM can be signaled, for example, using CU-level flags and processed as a merge mode. Other merge modes, such as those in VVCs, include normal merge mode, merge motion vector differences (MMVD) mode, combined inter-intraprediction (CIIP) mode, and sub-block merge mode.

[0076] When GPM mode is used, a CU can be evenly divided into two geometric partitions using one of 64 different partitioning methods, distinguished by 24 angles (e.g., non-uniform quantized between 0° and 360°) and up to four edges relative to the center of the CU. Figure 4 shows exemplary supporting angles in VVC. Figure 5 shows possible (or candidate) partition edges for an angle with index 3. For example, four partition edges can be provided for an angle with index 3. Each geometric partition of the CU can be interpreted using its own motion. Single prediction may be allowed only for each partition; that is, each partition can have one motion vector and one reference index. Single prediction motion constraints can ensure that only two motion-compensated predictions are required for each CU, as with conventional biprediction.

[0077] When GPM is currently used for CU, signals indicating the geometric partition index and two merge indices (one for each partition) can be further signaled. A numerical value for the maximum GPM candidate size can be explicitly signaled at the slice level, specifying the syntax binarization of the GPM merge index. After each of the two geometric partitions has been predicted, the sample values ​​along the geometric partition edges can be further refined using blending with adaptive weights. The blend intensity of the GPM can be shown in Figure 6. As shown in Figure 6, the blend intensity (or blend region width) can be fixed for all different content. The weighting values ​​for the blend mask can be given by the ramp function of equation (1) as follows:

number

number

[0078] The blended results can be signaled to the entire CU as a prediction signal, and, as with other prediction modes, transformation and quantization processes can be further applied to the entire CU. Furthermore, the motion field of the CU predicted using GPM can be stored.

[0079] As in VVC, CU motion information can be stored in 4x4 units. The stored motion information can be used for MV prediction and for constructing the merge list of the next coded CU. In GPM, three types of motion information can be stored spanned in 4x4 units. The stored motion information can include two geometric partitions P0 and P1, and unidirectional MVs of the two geometric partitions. The blend region between P0 and P1 can be predicted by the motion information from these two partitions. This allows GPM motion information to be stored according to the partitions.

[0080] GPM movement information can be signaled as a merge mode. To avoid additional memory bandwidth access, single predictions are allowed only for each partition of the GPM. However, regular merge candidates can be single or bipredictive and cannot be used directly as a GPM merge list. To minimize implementation complexity, an index parity-based method has been proposed to directly extract GPM merge candidates from a regular merge list without pruning. For example, for a candidate with an even value in the GPM merge index, MV0 from reference list 0, which has the corresponding regular merge index, can be used as the GPM merge candidate. If MV0 is unavailable, MV1 from reference list 1 can be used instead. Conversely, for an odd value in the GPM merge index, MV1 can be selected as the default GPM merge candidate.

[0081] To further improve the compression efficiency of the VVC standard, JVET-U0100 and EE2 were planned to be held between the 21st and 22nd JVET meetings to evaluate extended compression tools that go beyond the capabilities of VVC. In JVET-U0100 and EE2, template matching (TM) was proposed to refine motion on the decoder side. In TM mode, motion can be refined by constructing a template from left and top neighbor reconstruction samples and finding the closest match between the template of the current picture and the corresponding template of the reference frame.

[0082] In some embodiments, the TM can be applied to a GPM such as JVET-V117. Once the CU is coded in the GPM, it can be determined whether each motion of the geometric partition is refined using the TM. If the TM is selected, a template is constructed using left and top neighbor samples, and the motion is refined by finding the best match between the template and the reference region in the reference frame that has the same template pattern. The refined motion can be used to perform motion compensation for the geometric partition and can further be stored in the motion field.

[0083] Template matching (TM) can be used in GPMs, as in ECMs, to sort GPM partitioning modes according to the TM cost of each GPM partitioning mode. However, the two merge indices signaled within the GPM may be configured in a fixed order. The signal cost of these two merge indices can be high.

[0084] In this disclosure, merge indexes associated with GPM partitions can be sorted based on the TM cost of the merge indexes according to a template matching process.

[0085] In one embodiment, for a given geometric partitioning mode, the list of candidate geometric partitions is sorted in ascending order of TM costs, using the template matching cost of the available templates, which are currently above and / or to the left of the CU, for all possible MVs from the GPM merge candidates.

[0086] In one embodiment, for a given geometric partitioning mode, the list of candidate geometric partitions defined by the geometric partitioning mode can be sorted by applying the template matching (TM) cost of the currently available templates of the CU to all possible MVs of the GPM merge candidate, and using the TM costs in ascending order. The currently available templates of the CU may be reconfigured templates located above and / or to the left of the CU.

[0087] In one embodiment, as shown in Figure 7, the upper template has two partitions TT P0 and TT P1 It can be divided into TT. P0 The left template is used to sort the merge index of geometric partition P0 using TM cost in ascending order, and TT P1 This is used to sort the merge index of partition P1.

[0088] In one example, as shown in Figure 7, the current CU(702) can be partitioned into a first partition P0 and a second partition P1 according to a GPM partitioning edge (or partition line) (704). The current CU(702) can have an upper template (706) along the top of the current CU(702) and a left template (708) along the left side of the current CU(702). The upper template (706) is divided into two partitions TT by the partition line (704). P0 (or Part 1) and TT P1 It can be divided into (or two parts). TT P0And the left template can be used as the first template area to reorder the merge index of the geometric partition P0 using the TM cost in ascending order, TT P1 can be used as the second template area to reorder the merge index of partition P1. The merge index of partition P0 can indicate the candidate MVs of partition P0. Each of the candidate MVs of partition P0 can indicate each of the first candidate reference blocks of partition P0. The merge index of partition P1 can indicate the candidate MVs of partition P1. Each of the candidate MVs of partition P1 can indicate each of the first candidate reference blocks of partition P1. Each merge index can correspond to a candidate MV (or candidate reference block). Based on the TM cost, the candidate MVs (or candidate reference blocks) can be reordered. For example, TT P0 Based on the TM cost (or TM difference) between the left template (708) and the corresponding template of the candidate reference blocks of partition P0, the candidate MVs (or candidate reference blocks) of partition P0 can be reordered.

[0089] In one aspect, all MVs from the reference list 0 and reference list 1 of the normal merge candidates can be used as the MVs of the geometric partition for merge index reordering. All MVs are sorted in ascending order using the TM cost, and only the N candidates with the minimum TM cost are used and signaled in the GPM merge list with merge index reordering using the TM cost in ascending order.

[0090] In one example, all MVs from the normal merge candidate reference lists 0 (L0) and 1 (L1) can be used as MVs for geometric partitions for merge index sorting. MVs from L0 and / or L1 can be sorted in ascending order, for example, using TM costs. TM costs can represent the difference between the GPM partition template and the template of the MV corresponding to the GPM partition. In one example, only the N candidates with the smallest TM costs can be used, and signaling can be performed with a GPM merge list with sorted merge indices using ascending TM costs.

[0091] In one embodiment, MVs from non-adjacent blocks and temporal motion vectors from co-referenced frames can be used as MVs for sorting the GPM merge candidate list. For example, MVs from non-adjacent blocks and / or temporal motion vectors from co-referenced frames can be used as MVs for sorting the GPM merge candidate list.

[0092] In one embodiment, a sorting method based on template matching, such as adaptive reordering of merge candidates with template matching (ARMC-TM), is first applied to the merge candidate list, and the GPM candidate derivation process can be performed based on the sorted merge candidate list.

[0093] In one embodiment, sorting based on template matching can be applied to the merge candidate list first, and GPM candidates can be derived based on the sorted merge candidate list.

[0094] In one embodiment, the configuration of the GPM merge candidate list for one partition (e.g., P0) can be based on a sorted merge candidate list, while another partition (e.g., P1) can continue to use the unsorted candidate list based on the TM.

[0095] For example, the configuration of the GPM merge candidate list for one partition (e.g., P0) can be based on a sorted merge candidate list, while another partition (e.g., P1) can continue to use an unsorted candidate list based on the TM.

[0096] In one embodiment, for a given geometric partitioning mode, if the corresponding template size is smaller than a threshold S and this threshold S is not negative, then merge index sorting by using template matching for the geometric partition is not permitted. Figures 8 and 9 show two examples where template matching cannot be applied to the template of geometric partition P1 when the threshold S is 4. In Figures 8 and 9, each square in the template represents a pixel unit.

[0097] In one embodiment, for a given geometric partitioning mode, if the template size of the geometric partition template is smaller than a threshold S, merge index sorting by template matching is not permitted for the geometric partition. The threshold S may be a non-negative value such as 0 or another positive integer (e.g., 4).

[0098] The constraints on merge index sorting when using TM can be seen, for example, in Figures 8 and 9. Figures 8 and 9 show two examples where template matching cannot be applied to the template of geometric partition P1 when the threshold S is 4. In Figures 8 and 9, each square in the template can represent a pixel unit.

[0099] As shown in Figure 8, the current CU(802) can include an upper template (806) and a left template (808). The current CU(802) can be partitioned into a first partition P0 and a second partition P1 by a GPM partition edge (or partition line) (804). The upper template (806) can also be partitioned into a first part (806A) and (806B). The first part (806A) of the upper template (806) and the left template (808) can function as the template (or first template region) of the first partition P0, and the second part (806B) of the upper template (806) can function as the template (or second template region) of the second partition P1. Since the size of the second part (806B) is smaller than a threshold (e.g., 4), template matching may not apply to the template of the geometric partition P1.

[0100] As shown in Figure 9, the current CU(902) can include an upper template (906) and a left template (908). The current CU(902) can be partitioned into a first partition P0 and a second partition P1 by a GPM partition edge (or partition line) (904). The left template (908) can also be partitioned into a first part (908A) and (908B). The first part (908A) and the upper template (906) can function as a template (or first template region) for the first partition P0, and the second part (908B) can function as a template (or second template region) for the second partition P1. Since the size of the second part (908B) is smaller than a threshold (e.g., 4), template matching may not apply to the template of the geometric partition P1.

[0101] In one embodiment, the template size for a given geometric partitioning mode means the total number of samples within the corresponding template size of the geometric partition. In another embodiment, the template size for a given geometric partitioning mode means the width or height of the corresponding upper template or left template, regardless of the number of rows in the upper template or the number of columns in the left template.

[0102] In one example, the template size can indicate the total number of samples in the corresponding template of the geometric partition. For example, as shown in Figure 8, the first part (806A) of the upper template (806) may have a template size between 7 and 8, while the second part (806A) of the upper template (806) may have a template size of less than 1.

[0103] In one example, the template size can indicate the width or height of the corresponding upper template (e.g., (806)) or left template (e.g., (808)), regardless of the number of rows in the upper template or the number of columns in the left template. For example, as shown in Figure 8, the width of the first part (806A) of the upper template (806) can indicate the template size of the first part (806A). In Figure 9, the height of the first part (908A) of the left template (908) can indicate the template size of the first part (908A).

[0104] In one aspect, template matching is not permitted if one or more of the following conditions are true: (1) The size of the inner row of the upper template or the inner column of the left template of the corresponding geometric partition is less than or equal to the threshold. (2) The size of one of the rows in the upper template or one of the columns in the left template of the corresponding geometric partition is less than or equal to the threshold. (3) The size of all rows in the upper template or all columns in the left template of the corresponding geometric partition is less than or equal to the threshold.

[0105] In one example, template matching may not be permitted if one or more of the following conditions are true: (1) The size of an inner row (or inner line) in the upper template of the corresponding geometric partition or an inner column in the left template is less than or equal to a threshold. (2) The size of one of the rows (or lines) in the upper template of the corresponding geometric partition or one of the columns in the left template is less than or equal to a threshold. (3) The size of all rows (or lines) in the upper template of the corresponding geometric partition or all columns in the left template is less than or equal to a threshold.

[0106] In one embodiment, the configuration of the GPM merge list can only be applied to partitions that have at least one template with the full size of the block width or height (unaffected by GPM partitioning), and only complete templates are used for sorting based on template matching of candidate partitions. In one embodiment, if a partition does not have a complete adjacent template, the corresponding GPM candidate list cannot be sorted.

[0107] For example, the configuration of a GPM merge list can only be applied to partitions that have at least one template with a full size. Therefore, the width or height of the template is full and is not affected by (or not divided by) GPM partitioning. Thus, only full templates are used for sorting candidates based on template matching of the corresponding partition.

[0108] For example, if a partition does not have a complete adjacency template, the GPM candidate list corresponding to the partition cannot be sorted.

[0109] In one embodiment, the GPM merge list configuration for geometric partitions to which template matching cannot be applied remains the same as the GPM merge list configuration in VVC or current ECM software.

[0110] For example, if a GPM merge list configuration for a geometric partition to which template matching cannot be applied is not applicable, the GPM merge list configuration remains the same as the GPM merge list configuration in the associated codec (or software), such as VVC or ECM software.

[0111] In one embodiment, template matching is applied to all possible combinations of geometric partitioning modes and two (or more) merge indices of two (or more) geometric partitions, and then to indices indicating which geometric partitioning modes and motion vectors from the GPM merge list are used, sorted in ascending order based on template matching costs.

[0112] In one embodiment, template matching can be applied to all possible combinations of (i) geometric partitioning modes and (ii) two (or more) merge indices of two (or more) geometric partitions determined by the geometric partitioning mode. For example, a GPM can have 64 partitioning modes, each partitioning mode can have two geometric partitions, and each geometric partition can have N merge indices. Thus, the number of merge index combinations can be equal to 64 × 2 × N. Each merge index can further correspond to each merge candidate (or reference block). The merge indices can be sorted based on template matching costs in a predefined order (e.g., ascending order) to indicate which geometric partitioning modes and motion vectors (indicated by the merge indices) from the GPM merge list can be used.

[0113] In one embodiment, GPM merge candidates may be, but are not limited to, regular merge candidates, non-adjacent candidates, TMVP candidates, etc. For example, GPM merge candidates may be, but are not limited to, regular merge candidates, non-adjacent candidates, temporal motion vector prediction (TMVP) candidates, etc.

[0114] In one embodiment, only the n indices with the minimum TM cost are selected, where n is a non-zero value and is less than the number of all possible combinations of geometric partitioning modes and the two merge indices of the two geometric partitions. In one embodiment, the value of n can be signaled with high-level syntax such as VPS, SPS, PPS, and slice headers. In one example, the indices are signaled to indicate which of the n indices is currently used to reconstruct the block.

[0115] In one embodiment, instead of signaling both the GPM merge index and the GPM partition mode, only one GPM index is signaled, which includes information about which GPM merge candidate is used for each GPM partition and which GPM partition mode is used for the current block. In one example, instead of signaling both the GPM merge index and the GPM partition mode, one GPM index can be signaled.

[0116] Figure 10 shows a flowchart illustrating an overview of process (1000) according to one embodiment of the present disclosure. Process (1000) can be used in a video decoder. In various embodiments, process (1000) is performed by a processing circuit such as a processing circuit that performs the functions of a video decoder (110), a processing circuit that performs the functions of a video decoder (210), and so on. In some embodiments, process (1000) is implemented by a software instruction, and therefore, when the processing circuit executes the software instruction, the processing circuit performs process (1000). The process starts at (S1001) and proceeds to (S1010).

[0117] In (S1010), a video bitstream is received that includes the current block in the current frame and a template sample of the current block. The template sample includes an upper template along the top of the current block and a left template along the left side of the current block. The current block is divided into a first partition and a second partition, and the template sample is divided into a first template region adjacent to the first partition and a second template region adjacent to the second partition.

[0118] For example, for a given geometric partitioning mode, the list of candidate geometric partitions is sorted in ascending order of TM costs using the template matching costs of available templates that are currently above the CU (e.g., upper template) and / or to the left of the CU (e.g., left template) for all possible MVs from the GPM merge candidates. Furthermore, as shown in Figure 7, the upper template has two partition TTs. P0 and TT P1 It can be divided into TT. P0 The left template is used as the first template region to sort the merge index of the geometric partition P0 using the TM cost in ascending order. P1 This is used as a second template area to sort the merge index of partition P1.

[0119] In (S1020), multiple first candidate reference blocks are determined for the first partition, and multiple second candidate reference blocks are determined for the second partition. Each first candidate reference block has a first candidate template corresponding to the shape of the first template region, and each second candidate reference block has a second candidate template corresponding to the shape of the second template region.

[0120] For example, all MVs from reference list 0 and / or reference list 1 of the normal merge candidate can be used as MVs for geometric partitions for merge index sorting. All MVs are sorted using TM cost in ascending order, and only the N candidates with the smallest TM cost are used and signaled in the GPM merge list with merge index sorting, using TM cost in ascending order. All MVs from reference list 0 (L0) and reference list 1 (L1) of the normal merge candidate can be used as MVs for geometric partitions for merge index sorting. MVs from L0 and / or L1 can be sorted using TM cost in an order such as ascending. TM cost can represent the difference between the GPM partition template and the template of the MV corresponding to the GPM partition.

[0121] In (S1030), at least one of the multiple first candidate reference blocks and the multiple second candidate reference blocks is sorted based on the size of the first template area of ​​the template sample and the size of the second template area of ​​the template sample.

[0122] For example, if, for a given geometric partitioning mode, the size of the corresponding template (e.g., template region) is smaller than a threshold S, and this threshold S is not negative, then merge index sorting by using template matching for the geometric partition is not permitted. Figures 8 and 9 show two examples where template matching cannot be applied to the template of geometric partition P1 when the threshold S is 4. In Figures 8 and 9, each square in the template represents a pixel unit. For example, as shown in Figure 8, the current CU (802) can be partitioned into a first partition P0 and a second partition P1 by a GPM partitioning edge (or partition line) (804). The upper template (806) can also be partitioned into first parts (806A) and (806B). The first part (806A) of the upper template (806) and the left template (808) can correspond to the first template region of the first partition P0, and the second part (806B) of the upper template (806) can correspond to the second template region of the second partition P1. Since the size of the second part (806B) is smaller than the threshold (e.g., 4), template matching may not be applied to the template for geometric partition P1.

[0123] In (S1040), the current block is reconstructed based on the index value received in the bitstream and on a sorted order of at least one of a plurality of first candidate reference blocks and a plurality of second candidate reference blocks. In the example, all MVs are sorted using TM cost in ascending order, and only the N candidates with the smallest TM cost are signaled using the merge index value (e.g., indicating the index value) in a sorted GPM merge list, using TM cost in ascending order. In one example, instead of signaling both the GPM merge index and the GPM partition mode, one GPM index may be signaled. The single signaled GPM index may contain information about which GPM merge candidates are used for each GPM partition and which GPM partition mode is used for the current block.

[0124] In one example, multiple first candidate reference blocks in the first partition of the current block and multiple second candidate reference blocks in the second partition of the current block are determined based on candidate reference blocks indicated by motion vectors (MVs) in the first and second reference lists.

[0125] In one example, multiple first candidate reference blocks in the first partition of the current block and multiple second candidate reference blocks in the second partition of the current block are determined based on one of the blocks in the same-position reference frame, indicated by non-adjacent blocks and temporal motion vectors.

[0126] In one embodiment, the template-matching (TM) difference between each of the first template regions of the template sample and the first candidate templates is determined. The TM difference between each of the second template regions of the template sample and the second candidate templates is determined. Multiple first candidate reference blocks for the first partition are sorted in ascending order of the TM difference between the first template region and the first candidate templates of the template sample. Multiple second candidate reference blocks for the second partition are sorted in ascending order of the TM difference between the second template region and the second candidate templates of the template sample.

[0127] In one embodiment, the upper template is divided into a first part and a second part, the first template area of ​​the template sample includes the left template and the first part of the upper template, and the second template area of ​​the template sample includes the second part of the upper template. Based on whether the number of samples in the sample rows of the first part of the template above is greater than the threshold, multiple first-choice reference blocks for the first partition are sorted. Based on whether the number of samples in the sample rows of the second part of the template above is greater than the threshold, multiple second-choice reference blocks for the second partition are sorted.

[0128] In one embodiment, the upper template is divided into a first part and a second part, the first template area of ​​the template sample includes the left template and the first part of the upper template, and the second template area of ​​the template sample includes the second part of the upper template. Based on the fact that the width of the sample rows in the first part of the template above is greater than the threshold, multiple first-choice reference blocks for the first partition are sorted. Based on the fact that the width of the sample rows in the second part of the template above is greater than the threshold, multiple second-choice reference blocks for the second partition are sorted.

[0129] In one embodiment, the left template is divided into a first part and a second part, the first template area of ​​the template sample includes the upper template and the first part of the left template, and the second template area of ​​the template sample includes the second part of the left template. Based on whether the number of samples in the sample column of the first part of the left template is greater than the threshold, multiple first-choice reference blocks for the first partition are ordered. Based on whether the number of samples in the sample column of the second part of the left template is greater than the threshold, multiple second-choice reference blocks for the second partition are sorted.

[0130] In one embodiment, the left template is divided into a first part and a second part, the first template area of ​​the template sample includes the upper template and the first part of the left template, and the second template area of ​​the template sample includes the second part of the left template. Based on whether the height of the sample column in the first part of the left template is greater than the threshold, multiple first-choice reference blocks for the first partition are sorted. Based on whether the height of the sample column in the second part of the left template is greater than the threshold, multiple second-choice reference blocks for the second partition are sorted.

[0131] In one embodiment, the TM difference between one of the undivided upper template or the undivided left template and each corresponding region of the first candidate template is determined based on a first template region of a template sample containing one of the undivided upper template or the undivided left template, and a second template region of a template sample containing the other of the divided upper template or the undivided left template. Furthermore, multiple first candidate reference blocks of the first partition are sorted in ascending order of the TM differences between one of the undivided upper template or the undivided left template and the corresponding region of the first candidate template.

[0132] In one example, multiple candidate partitions are determined for the current block based on multiple candidate partitioning methods. For each of the multiple candidate partitions, multiple candidate reference blocks are determined. The TM difference between the template sample of the current block and the template area of ​​each of the multiple candidate reference blocks in the multiple candidate partitions is determined. Based on the TM difference between the template sample of the current block and the template areas of the multiple reference blocks in the current block, the multiple candidate reference blocks are sorted in ascending order.

[0133] In one example, a video bitstream contains multiple indices that represent a sorted subset of candidate reference blocks, and each sorted subset of candidate reference blocks contains candidate reference blocks that have the first N smallest TM difference.

[0134] Next, the process proceeds to (S1099) and terminates.

[0135] Process (1000) can be appropriately adapted. The steps of process (1000) can be changed and / or omitted. Additional steps can be added. Any appropriate implementation order can be used.

[0136] Figure 11 shows a flowchart illustrating an overview of process (1100) according to one embodiment of the present disclosure. Process (1100) can be used in a video encoder. In various embodiments, process (1100) is performed by a processing circuit such as a processing circuit that performs the functions of a video encoder (103), a processing circuit that performs the functions of a video encoder (303), and so on. In some embodiments, process (1100) is implemented by a software instruction, and so the processing circuit performs process (1100) when it executes the software instruction. The process starts at (S1101) and proceeds to (S1110).

[0137] In (S1110), multiple candidate first reference blocks are determined for the first partition of the current block, and multiple candidate second reference blocks are determined for the second partition of the current block. The current block is coded in geometric partition mode (GPM) in which the current block is partitioned into a first partition and a second partition by partition lines. The current block includes a template sample which includes an upper template adjacent to the top of the current block and a left template adjacent to the left of the current block. The template sample is divided by partition lines into a first template region adjacent to the first partition and a second template region adjacent to the second partition. Each of the multiple candidate first reference blocks in the first partition has its own candidate first template. Each of the multiple candidate second reference blocks in the second partition has its own candidate second template.

[0138] An exemplary embodiment related to step (S1110) can be shown in Figure 7. As shown in Figure 7, the current CU (702) can be partitioned into a first partition P0 and a second partition P1 according to a GPM partition edge (or partition line) (704). The current CU (702) can have an upper template (706) above the current CU (702) and a left template (708) to the left of the current CU (702). The upper template (706) is divided into two partitions TT by the partition line (704). P0 and TT P1 It can be divided into TT. P0 And the left template (708) can be used to sort the merge index of geometric partition P0 using TM cost in ascending order, TT P1 This can be used to sort the merge index of partition P1.

[0139] In (S1120), at least one of the multiple candidate first reference blocks in the first partition and the multiple candidate second reference blocks in the second partition are sorted based on the size of the first template area of ​​the template sample and the size of the second template area of ​​the template sample.

[0140] For example, as shown in Figure 8, the current CU (802) and GPM partition edge (or partition line) (804) can partition into a first partition P0 and a second partition P1. The upper template (806) can also be partitioned into first parts (806A) and (806B). The first part (806A) of the upper template (806) and the left template (808) can function as a template for the first partition P0, and the second part (806B) of the upper template (806) can function as a template for the second partition P1. Since the size of the second part (806B) is smaller than a threshold (e.g., 4), template matching may not apply to the template of the geometric partition P1.

[0141] In (S1130), the current block is encoded based on at least one sorted set of multiple candidate first reference blocks for the first partition and multiple candidate second reference blocks for the second partition. In one example, only the n indices with the minimum TM cost may be selected, where n is a non-zero value and is less than the number of all possible combinations of geometric partition modes and two merge indices of two geometric partitions. In one example, the value of n can be signaled with high-level syntax such as VPS, SPS, PPS, slice headers. In one example, the indices are signaled to indicate which of the n indices is used to reconstruct the current block.

[0142] Next, the process proceeds to (S1199) and terminates.

[0143] Process (1100) can be appropriately adapted. The steps of process (1100) can be changed and / or omitted. Additional steps can be added. Any appropriate implementation order can be used.

[0144] The techniques described above can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 12 shows a computer system (1200) suitable for implementing a particular embodiment of the subject matter of this disclosure.

[0145] Computer software can be coded using any suitable machine code or computer language that can be processed by mechanisms such as assembly, compilation, and linking to generate code containing instructions that can be executed directly or through interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0146] The instructions can be executed on various computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, Internet of Things devices, etc.

[0147] The components shown in Figure 12 of the computer system (1200) are illustrative and do not imply any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Furthermore, the configuration of the components should not be construed as having any dependencies or requirements relating to any one or combination of the components shown in the exemplary embodiments of the computer system (1200).

[0148] The computer system (1200) may include a specific human interface input device. Such a human interface input device may respond to input from one or more human users, for example, through sensory input (e.g., keystrokes, swipes, data grab actions), voice input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface device may also be used to capture certain media that do not necessarily need to be directly related to conscious human input, such as voice (e.g., conversation, music, ambient sounds), images (e.g., scanned images, photographic images taken from a digital camera), or video (e.g., including 2D video, 3D video, and stereoscopic video).

[0149] The input human interface device may include one or more of the following (only one is shown): keyboard (1201), mouse (1202), trackpad (1203), touchscreen (1210), data grab (not shown), joystick (1205), microphone (1206), scanner (1207), and camera (1208).

[0150] The computer system (1200) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through sensory output, sound, light, and smell / taste. Such human interface output devices may include sensory output devices (e.g., touchscreen (1210), data grab (not shown), or joystick (1205) (for sensory feedback, however, there may also be sensory feedback devices that do not function as input devices), sound output devices (e.g., speaker (1209), headphones (not shown)), visual output devices (e.g., screen (1210), CRT screen, LCD screen, plasma screen, OLED screen, each having or not having touchscreen input capability, each having or not having sensory feedback capability, some of which may be capable of outputting two-dimensional visual output or three-dimensional or more output through means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), and printers (not shown))).

[0151] The computer system (1200) may also include human-accessible storage devices, and related media such as optical media including CD / DVD ROM / RW (1220) with media such as CD / DVD (1221), thumb drives (1222), removable hard drives or solid drives (1223), legacy magnetic media such as tapes and floppy disks (not shown), and devices based on dedicated ROM / ASIC / PLD such as security dongles (not shown).

[0152] Those skilled in the art should also understand that the term “computer-readable medium” as used in connection with the subject matter of this disclosure does not include a transmission medium, carrier wave, or other transient signal.

[0153] The computer system (1200) may also include an interface (1254) to one or more communication networks (1255). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, urban, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicle and industrial networks including CANBus, etc. Certain networks generally require an external network interface connected to a specific general-purpose data port or peripheral bus (1249) (e.g., a USB port on the computer system (1200)). Others are generally integrated into the core of the computer system (1200) by connection to a system bus, as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using these networks, the computer system (1200) can communicate with other entities. Such communication may be unidirectional (e.g., broadcast television), unidirectional (e.g., CANbus to a specific CANbus device), or bidirectional (e.g., local or wide-area digital network). Specific protocols and protocol stacks may be used in each of the aforementioned networks and network interfaces.

[0154] The aforementioned human interface device, human-accessible storage device, and network interface can be mounted on the core (1240) of the computer system (1200).

[0155] The core (1240) may include one or more central processing units (CPUs) (1241), graphics processing units (GPUs) (1242), dedicated programmable processing units in the form of FPGAs (1243), hardware accelerators for specific tasks (1244), graphics adapters (1250), etc. These devices may be connected via a system bus (1248) along with read-only memory (ROM) (1245), random access memory (1246), and internal mass storage devices such as internal, user-inaccessible hard drives, SSDs, etc. (1247). In some computer systems, the system bus (1248) is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals can be attached directly to the core's system bus (1248) or via a peripheral bus (1249). In the example, a screen (1210) can be connected to the graphics adapter (1250). Peripheral device bus architectures include PCI, USB, etc.

[0156] The CPU (1241), GPU (1242), FPGA (1243), and accelerator (1244) can execute specific instructions that, when combined, can generate the aforementioned computer code. This computer code can be stored in ROM (1245) or RAM (1246). Temporary data can also be stored in RAM (1246), while permanent data can be stored, for example, in a built-in mass storage device (1247). High-speed storage and retrieval to any of the memory devices can be enabled through the use of cache memory that may be closely associated with one or more of the CPU (1241), GPU (1242), mass storage device (1247), ROM (1245), RAM (1246), etc.

[0157] Computer-readable media may contain computer code for performing actions performed by various computers. The media and computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type well known and available to those skilled in the computer software field.

[0158] As an example and not limited thereto, a computer system (1200) having an architecture, and specifically a core (1240), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be specific storage devices of the core (1240) with non-transient characteristics, such as a core-integrated mass storage device (1247) or ROM (1245), and media associated with the user-accessible mass storage devices described above. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1240). The computer-readable media may include one or more memory devices or chips, depending on the specific requirements. The software can cause the core (1240) and specifically the processor (including a CPU, GPU, FPGA, etc.) therein to execute specific processes or specific parts of specific processes described herein, including defining and modifying data structures stored in RAM (1246) according to software-defined processes. As an addition or alternative, a computer system may provide functionality as a result of a logic hardwired or other circuit implementation (e.g., an accelerator (1244)) that can operate together with or in place of the software to perform the specific processes or specific parts of the specific processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuits (such as integrated circuits (ICs)) that house software for execution, circuits that implement logic for execution, or both. This disclosure includes any appropriate combination of hardware and software.

[0159] The use of “at least one” or “one” in this disclosure is intended to include any one or combination of the elements described. For example, references to at least one of A, B, or C; at least one of A, B, and C; at least one of A, B, and / or C; and at least one of A-C are intended to include A only, B only, C only, or any combination thereof. References to one of A or B, and one of A and B are intended to include A or B or (A and B). The use of “one” does not exclude any combination of the enumerated elements where applicable, such as when the elements are not mutually exclusive.

[0160] While this disclosure describes several exemplary embodiments, alternatives, substitutions, and various equivalents exist and are included within the scope of this disclosure. As will be apparent to those skilled in the art, numerous systems and methods can be devised to implement the principles of this disclosure and thus fall within the spirit and scope of this disclosure, although these are not expressly shown or described herein.

Claims

1. A video decoding method performed in a video decoder, A step of receiving a video bitstream containing a current block in the current frame and a template sample of the current block, wherein the template sample includes an upper template along the top of the current block and a left template along the left side of the current block, the current block is partitioned into a first partition and a second partition, and the template sample is divided into a first template region adjacent to the first partition and a second template region adjacent to the second partition. A step of determining a plurality of first candidate reference blocks for the first partition and a plurality of second candidate reference blocks for the second partition, wherein each first candidate reference block has a first candidate template corresponding to the shape of the first template region, and each second candidate reference block has a second candidate template corresponding to the shape of the second template region. The steps of rearranging at least one of the plurality of first candidate reference blocks and the plurality of second candidate reference blocks based on the size of the first template region of the template sample and the size of the second template region of the template sample, The steps of reconstructing the current block based on the index value received in the video bitstream and based on a sorted version of at least one of the plurality of first candidate reference blocks and the plurality of second candidate reference blocks, A method that includes this.

2. The aforementioned decision-making step is: The method according to claim 1, further comprising the step of determining the plurality of first candidate reference blocks in the first partition of the current block and the plurality of second candidate reference blocks in the second partition of the current block, based on candidate reference blocks indicated by motion vectors (MVs) in the first and second reference lists.

3. The aforementioned sorting step is, The steps include determining the template matching (TM) difference between the first template region of the template sample and each of the first candidate templates, and the TM difference between the second template region of the template sample and each of the second candidate templates, (i) sorting the plurality of first candidate reference blocks for the first partition based on the ascending order of the TM difference between the first template region of the template sample and the first candidate template, and (ii) sorting the plurality of second candidate reference blocks for the second partition based on the ascending order of the TM difference between the second template region of the template sample and the second candidate template, The method according to claim 1, further comprising:

4. The aforementioned sorting step is, Based on the fact that the upper template is divided into a first part and a second part, that the first template region of the template sample includes the left template and the first part of the upper template, and that the second template region of the template sample includes the second part of the upper template, The steps of sorting the plurality of first candidate reference blocks for the first partition based on the fact that the number of samples in the sample rows of the first part of the above template is greater than a threshold, The steps of sorting the plurality of second candidate reference blocks for the second partition based on the fact that the number of samples in the sample row of the second part of the upper template is greater than the threshold, The method according to claim 1, further comprising:

5. The aforementioned sorting step is, Based on the fact that the upper template is divided into a first part and a second part, that the first template region of the template sample includes the left template and the first part of the upper template, and that the second template region of the template sample includes the second part of the upper template, The steps of sorting the plurality of first candidate reference blocks for the first partition based on the fact that the width of the sample rows in the first part of the above template is greater than a threshold, The steps of sorting the plurality of second candidate reference blocks for the second partition based on the fact that the width of the sample row in the second part of the upper template is greater than the threshold, The method according to claim 1, further comprising:

6. The aforementioned sorting step is, Based on the fact that the left template is divided into a first part and a second part, that the first template region of the template sample includes the upper template and the first part of the left template, and that the second template region of the template sample includes the second part of the left template, The steps of sorting the plurality of first candidate reference blocks for the first partition based on the fact that the number of samples in the sample column of the first part of the left template is greater than a threshold, The steps of sorting the plurality of second candidate reference blocks for the second partition based on the fact that the number of samples in the sample column of the second part of the left template is greater than the threshold, The method according to claim 1, further comprising:

7. The aforementioned sorting step is, Based on the fact that the left template is divided into a first part and a second part, that the first template region of the template sample includes the upper template and the first part of the left template, and that the second template region of the template sample includes the second part of the left template, The steps of sorting the plurality of first candidate reference blocks for the first partition based on the fact that the height of the sample column in the first part of the left template is greater than a threshold, The steps of sorting the plurality of second candidate reference blocks for the second partition based on the fact that the height of the sample column in the second part of the left template is greater than the threshold, The method according to claim 1, further comprising:

8. The aforementioned sorting step further, Based on the fact that the first template region of the template sample includes either the undivided upper template or the undivided left template, and the second template region of the template sample includes the other of the divided upper template or the left template, A step of determining the template matching (TM) difference between one of the undivided upper template or the undivided left template and the corresponding regions of the first candidate template, The steps of sorting the plurality of first candidate reference blocks for the first partition based on the ascending order of the TM difference between one of the undivided upper template or the undivided left template and the corresponding region of the first candidate template, The method according to claim 1, further comprising:

9. A step of determining a plurality of candidate partitions of the current block based on a plurality of candidate partitioning methods, The steps include determining a plurality of candidate reference blocks for each of the plurality of candidate partitions, The steps include determining the TM difference between the template sample of the current block and the template region of each of the multiple candidate reference blocks of the multiple candidate partitions, The steps include sorting the plurality of candidate reference blocks in ascending order based on the TM difference between the template sample of the current block and the template region of the plurality of candidate reference blocks of the current block, The method according to claim 3, further comprising:

10. The method according to claim 9, wherein the video bitstream includes a plurality of indices indicating a sorted subset of the plurality of candidate reference blocks, the sorted subset of the plurality of candidate reference blocks includes candidate reference blocks having the first N minimum TM difference.

11. A device, wherein the device includes a processing circuit, and the processing circuit is A video bitstream is received that includes the current block in the current frame and a template sample of the current block, wherein the template sample includes an upper template along the top of the current block and a left template along the left side of the current block, the current block is partitioned into a first partition and a second partition, and the template sample is divided into a first template region adjacent to the first partition and a second template region adjacent to the second partition. Determine a plurality of first candidate reference blocks for the first partition and a plurality of second candidate reference blocks for the second partition, wherein each first candidate reference block has a first candidate template corresponding to the shape of the first template region, and each second candidate reference block has a second candidate template corresponding to the shape of the second template region. Based on the size of the first template region of the template sample and the size of the second template region of the template sample, at least one of the plurality of first candidate reference blocks and the plurality of second candidate reference blocks is rearranged. The current block is reconstructed based on the index value received in the video bitstream and based on the sorted order of at least one of the plurality of first candidate reference blocks and the plurality of second candidate reference blocks. A device that is configured in such a way.

12. The aforementioned processing circuit is The apparatus according to claim 11, configured to determine the plurality of first candidate reference blocks in the first partition of the current block and the plurality of second candidate reference blocks in the second partition of the current block, based on candidate reference blocks indicated by motion vectors (MVs) in the first and second reference lists.

13. The aforementioned processing circuit is Determine the template matching (TM) difference between the first template region of the template sample and each of the first candidate templates, and the TM difference between the second template region of the template sample and each of the second candidate templates. (i) sort the plurality of first candidate reference blocks for the first partition based on the ascending order of the TM difference between the first template region and the first candidate template of the template sample, and (ii) sort the plurality of second candidate reference blocks for the second partition based on the ascending order of the TM difference between the second template region and the second candidate template of the template sample. The apparatus according to claim 11, configured as described above.

14. The aforementioned processing circuit is Based on the fact that the upper template is divided into a first part and a second part, that the first template region of the template sample includes the left template and the first part of the upper template, and that the second template region of the template sample includes the second part of the upper template, Based on the fact that the number of samples in the sample row of the first part of the above template is greater than the threshold, the plurality of first candidate reference blocks for the first partition are rearranged, Based on the fact that the number of samples in the sample rows of the second part of the above template is greater than the threshold, the plurality of second candidate reference blocks for the second partition are sorted. The apparatus according to claim 11, configured as described above.

15. The aforementioned processing circuit is Based on the fact that the upper template is divided into a first part and a second part, that the first template region of the template sample includes the left template and the first part of the upper template, and that the second template region of the template sample includes the second part of the upper template, Based on the fact that the width of the sample rows in the first part of the above template is greater than the threshold, the plurality of first candidate reference blocks for the first partition are rearranged. Based on the fact that the width of the sample rows in the second part of the above template is greater than the threshold, the plurality of second candidate reference blocks for the second partition are sorted. The apparatus according to claim 11, configured as described above.

16. The aforementioned processing circuit is Based on the fact that the left template is divided into a first part and a second part, that the first template region of the template sample includes the upper template and the first part of the left template, and that the second template region of the template sample includes the second part of the left template, Based on the fact that the number of samples in the sample column of the first part of the left template is greater than the threshold, the plurality of first candidate reference blocks for the first partition are rearranged. Based on the fact that the number of samples in the sample column of the second part of the left template is greater than the threshold, the plurality of second candidate reference blocks for the second partition are sorted. The apparatus according to claim 11, configured as described above.

17. The aforementioned processing circuit is Based on the fact that the left template is divided into a first part and a second part, that the first template region of the template sample includes the upper template and the first part of the left template, and that the second template region of the template sample includes the second part of the left template, Based on the fact that the height of the sample column in the first part of the left template is greater than the threshold, the plurality of first candidate reference blocks for the first partition are rearranged. Based on the fact that the height of the sample column in the second part of the left template is greater than the threshold, the plurality of second candidate reference blocks for the second partition are sorted. The apparatus according to claim 11, configured as described above.

18. The aforementioned processing circuit is Based on the fact that the first template region of the template sample includes either the undivided upper template or the undivided left template, and the second template region of the template sample includes the other of the divided upper template or the left template, Determine the template matching (TM) difference between one of the undivided upper template or the undivided left template and the corresponding regions of the first candidate template. The plurality of first candidate reference blocks for the first partition are sorted in ascending order of the TM difference between one of the undivided upper template or the undivided left template and the corresponding region of the first candidate template. The apparatus according to claim 11, configured as described above.

19. A video encoding method performed in a video encoder, A step of determining a plurality of candidate first reference blocks for a first partition of a current block, and a plurality of candidate second reference blocks for a second partition of the current block, wherein the current block is coded by a geometric partitioning mode (GPM) in which the current block is partitioned into the first partition and the second partition by partition lines, the current block includes a template sample including an upper template adjacent to the upper side of the current block and a left template adjacent to the left side of the current block, the template sample is divided by the partition lines into a first template region adjacent to the first partition and a second template region adjacent to the second partition, each of the plurality of candidate first reference blocks in the first partition has its own first candidate template, and each of the plurality of candidate second reference blocks in the second partition has its own second candidate template, The steps of rearranging at least one of the plurality of candidate first reference blocks in the first partition and the plurality of candidate second reference blocks in the second partition based on the size of the first template region of the template sample and the size of the second template region of the template sample, The steps of encoding the current block based on at least one reordered set of the plurality of candidate first reference blocks for the first partition and the plurality of candidate second reference blocks for the second partition, A method that includes this.

Images