Method, apparatus and program for video decoding in a video decoder

Abstract

Aspects of the present disclosure provide a method and apparatus for video decoding. The apparatus includes a processing circuit. The processing circuit determines whether at least one sample in a first template of a current block is included in an intra-coded block that is a spatial neighboring block of the current block. The first template includes one or more samples of a top neighboring block and a left neighboring block of the current block. If at least one sample in the first template is included in the intra-coded block and a template matching (TM) mode is applied to the current block, at least one sample is excluded from the first template to generate a second template for the current block. The processing circuit determines a motion vector (MV) of the current block based on the second template for the current block.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Application No. 63 / 281,772, "Constrained Template Matching," filed November 22, 2021, which claims the benefit of priority to U.S. Patent Application No. 17 / 942,036, "CONSTRAINED TEMPLATE MATCHING," filed September 9, 2022. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] This disclosure generally describes embodiments related to video coding. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context for the present disclosure. The inventors' work, to the extent that it is described in this background section, and aspects of the description that may not be admitted as prior art at the time of filing, are not admitted expressly or implicitly as prior art to the present disclosure.

[0004] Uncompressed digital images and / or video can include a series of pictures, each with spatial dimensions of, for example, 1920 x 1080 luma samples and associated chroma samples. The series of pictures can have a fixed or variable picture rate (informally known as a frame rate) of, for example, 60 pictures per second, or 60 Hz. Uncompressed images and / or video have specific bitrate requirements. For example, 1080p60 4:2:0 video (1920 x 1080 luma sample resolution at a 60 Hz frame rate) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 GBytes of storage space.

[0005] One goal of image and / or video coding and decoding may be reducing redundancy in the input image and / or video signal through compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by two or more orders of magnitude. While the description herein uses video encoding / decoding as an illustrative example, the same techniques can be applied to image encoding / decoding in a similar manner without departing from the spirit of this disclosure. Both lossless and lossy compression, and combinations thereof, can be employed. Lossless compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signal is small enough to make the reconstructed signal useful for its intended use. For video, lossy compression is widely adopted. The amount of acceptable distortion depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect that the higher the tolerable / acceptable distortion, the higher the compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transform processing, quantization, and entropy coding.

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture may be an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, are used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and video session, or as a still image. Samples of intra-blocks undergo a transform, and the transform coefficients may be quantized before entropy coding. Intra-prediction can be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and AC coefficients after the transform, the fewer bits required for a given quantization step size to represent the block after entropy coding.

[0008] For example, traditional intra-coding, used in MPEG-2 generation coding techniques, does not use intra-prediction. However, some new video compression techniques include techniques that attempt to perform prediction, for example, based on surrounding sample data and / or metadata obtained during encoding / decoding of a data block. Such techniques are hereinafter referred to as "intra-prediction" techniques. Note that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from a reference picture.

[0009] Intra-prediction can take many different forms. When two or more of such techniques can be used in a given video coding technique, the particular technique in use can be coded as a particular intra-prediction mode that uses the particular technique. In certain cases, an intra-prediction mode can have sub-modes and / or parameters that can be coded separately or included in a mode codeword that defines the prediction mode used. The codeword used for a given mode, sub-mode, and / or parameter combination can affect coding efficiency via intra-prediction, and therefore can also affect the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain modes of intra prediction were introduced in H.264, improved in H.265, and further refined in new coding techniques such as the Joint Search Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Neighboring sample values ​​of already available samples can be used to form a predictor block. The sample values ​​of neighboring samples are copied to the predictor block according to their direction. A reference to the direction in use can be coded in the bitstream, or it can be predicted itself.

[0011] Referring to FIG. 1A, depicted at the bottom right is a subset of nine known predictor directions from the 33 possible predictor directions defined in H.265 (corresponding to the 33 angular modes out of the 35 intra modes). The point where the arrows converge (101) represents the sample being predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right at an angle of 45 degrees from horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left of sample (101) at an angle of 22.5 degrees from horizontal.

[0012] 1A, a square block (104) of 4x4 samples (indicated by a thick dashed line) is shown at the top left. The square block (104) contains 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions within the block (104). Because the block is 4x4 samples in size, S44 is at the bottom right. Also shown are reference samples, which follow a similar numbering scheme. The reference samples are labeled with "R" and their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, the prediction samples are adjacent to the block being reconstructed, so negative values ​​do not need to be used.

[0013] Intra-picture prediction can work by copying reference sample values ​​from neighboring samples indicated by the signaled prediction direction. For example, assume that the coded video bitstream includes signaling for this block indicating a prediction direction consistent with the arrow (102), i.e., the sample is predicted from the sample to the upper right at a 45-degree angle from the horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, the values ​​of multiple reference samples may be combined, for example by interpolation, to calculate a reference sample, especially when the direction is not evenly divisible by 45 degrees.

[0015] The number of possible directions has increased as video coding technology has evolved. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and specific entropy coding techniques are used to represent these likely directions with a small number of bits, accepting a certain penalty for less likely directions. Furthermore, in some cases, the direction itself can be predicted from neighboring directions used in neighboring, already decoded blocks.

[0016] Figure 1B shows a schematic diagram (110) showing 65 intra-prediction directions with JEM to illustrate the increasing number of prediction directions over time.

[0017] The mapping of intra-prediction direction bits, which represent directions within a coded video bitstream, may vary depending on the video coding technique. Such mappings may range from simple direct mappings to complex adaptive schemes including codewords, most probable modes, and similar techniques. However, in most cases, there may be certain directions that are statistically less likely to occur within the video content than certain other directions. Because the goal of video compression is to reduce redundancy, these less likely directions are represented with more bits than more likely directions in well-performing video coding techniques.

[0018] Image and / or video coding and decoding can be performed using inter-picture prediction with motion compensation. Motion compensation is a lossy compression technique that may refer to a technique used to predict a newly reconstructed picture or portion of a picture after blocks of sample data from a previously reconstructed picture or portion thereof (reference picture) are spatially shifted in a direction indicated by a motion vector (hereinafter, MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, with the third dimension being an indication of the reference picture in use (the third dimension may indirectly be a temporal dimension).

[0019] In some video compression techniques, the MV applicable to a particular area of ​​sample data can be predicted from other MVs, e.g., from MVs associated with other areas of sample data that are spatially adjacent to the area being reconstructed and precede that MV in decoding order. Doing so can significantly reduce the amount of data required to code the MV, thereby eliminating redundancy and increasing compression ratios. For example, when coding an input video signal derived from a camera (known as natural video), MV prediction can work effectively because there is a statistical likelihood that areas larger than the area to which a single MV is applicable move in similar directions and, therefore, in some cases, can be predicted using similar motion vectors derived from MVs of neighboring areas. As a result, the detected MV for a given area is similar or identical to the MV predicted from surrounding MVs, which, after entropy coding, can be represented with fewer bits than would be used if the MV were coded directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from an original signal (i.e., a sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating a predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms provided by H.265, the one described with reference to Figure 2 is a technique hereafter referred to as "spatial merging".

[0021] Referring to Figure 2, a current block (201) contains samples that the encoder discovered during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of directly coding its MV, the MV can be derived from metadata associated with one or more reference pictures, e.g., from the most recent reference picture (in decoding order), using the MV associated with any one of five surrounding samples, denoted A0, A1, and B0, B1, B2 (202-206, respectively). In H.265, MV prediction can use predictors from the same reference picture used by neighboring blocks. Summary of the Invention [Means for solving the problem]

[0022] Aspects of the present disclosure provide methods and apparatuses for video encoding and decoding. In some examples, the apparatus for video decoding includes a processing circuit. The processing circuit is configured to decode prediction information for a current block in a current picture from a video bitstream. The prediction information indicates a template matching (TM) mode for the current block. The processing circuit determines whether at least one sample in an initial template (also referred to as a first template) for the current block is included in an intra-coded block that is a spatial neighbor of the current block. The initial template for the current block includes samples of one or more upper neighboring blocks of the current block and / or samples of one or more left neighboring blocks of the current block. If at least one sample in the initial template is included in an intra-coded block, the processing circuit determines a motion vector (MV) for the current block based on (i) an initial MV of the current block without applying the TM mode, or (ii) a current template (also referred to as a second template) that is generated based on the initial MV of the current block and omits at least one sample in the initial template included in the intra-coded block. The processing circuit reconstructs the current block based on the determined MV.

[0023] In one embodiment, a method is performed to determine whether at least one sample in a first template for a current block is included in an intra-coded block that is a spatial neighbor of the current block. The first template for the current block may include one or more samples from (1) one or more top-neighboring blocks of the current block and (2) one or more left-neighboring blocks of the current block. In one example, the one or more top-neighboring blocks are top-neighboring blocks. In one example, the one or more left-neighboring blocks are left-neighboring blocks. In response to determining that at least one sample in the first template is included in an intra-coded block, at least one sample may be excluded from the first template to generate a second template for the current block. The MV of the current block may be determined based on the second template for the current block.

[0024] In one example, in response to (i) a determination that at least one sample in the first template is included in an intra-coded block, and (ii) TM mode is applied to the current block, at least one sample can be excluded from the first template to generate a second template for the current block.

[0025] In one embodiment, at least one sample in the initial template of the current block is determined to be included in an intra-coded block, and the processing circuit determines the MV based on the initial MV of the current block without applying the TM mode.

[0026] In one embodiment, the initial template for the current block includes an initial top template including samples from one or more top-neighboring blocks and an initial left template including samples from one or more left-neighboring blocks. At least one sample in the initial template for the current block is determined to be included in an intra-coded block. If one or more left-neighboring blocks of the current block do not include an intra-coded block, the processing circuit determines the MV for the current block using a TM mode based on the initial MV and the current template including the initial left template. If one or more top-neighboring blocks of the current block do not include an intra-coded block, the processing circuit determines the MV for the current block using a TM mode based on the initial MV and the current template including the initial top template.

[0027] In one embodiment, at least one sample in the initial template for the current block is determined to be included in an intra-coded block, and the processing circuit determines the MV for the current block using a TM mode based on the initial MV and the current template for the current block.

[0028] In one embodiment, the processing circuit generates at least one interpolated sample by interpolation based on one or more samples in a current template, the at least one interpolated sample corresponding to at least one sample in an initial template included in an intra-coded block, the current template for the current block further including the at least one interpolated sample.

[0029] In one embodiment, reference samples in the reference template corresponding to the current template of the current block are located at sub-pixel positions in the reference picture. The processing circuit generates an interpolated reference template in the reference picture using interpolation with a filter having less than four taps in the reference template. The processing circuit determines the MV of the current block using a TM mode based on the initial MV, the current template of the current block, and the interpolated reference template.

[0030] In one example, a reference template in a reference picture omits at least one reference sample in the reference picture that corresponds to at least one sample in an initial template of a current block included in an intra-coded block. The processing circuit generates a padded sample corresponding to one of the at least one omitted reference sample in the reference picture by copying a nearest reference sample in the reference picture. The processing circuit generates an interpolated reference template in the reference picture by interpolation based on the padded sample.

[0031] In one example, the reference template in the reference picture omits at least one reference sample in the reference picture that corresponds to at least one sample in the initial template of the current block included in the intra-coded block, and the processing circuit generates the interpolated reference template in the reference picture by interpolation based on the omitted at least one reference sample in the reference picture.

[0032] In one embodiment, reference samples in an initial reference template corresponding to an initial template for a current block are located at sub-pixel positions in the reference picture. The processing circuit generates an intermediate reference template in the reference picture using interpolation on the initial reference template. The processing circuit generates a reference template corresponding to a current template for the current block based on the intermediate reference template and a mask. The shape of the mask can be determined based on the current template for the current block.

[0033] In one embodiment, at least one sample in the initial template included in the intra-coded block includes (i) one or more left samples of the samples of one or more left-neighboring blocks, or (ii) one or more top samples of the samples of one or more top-neighboring blocks. If at least one sample in the initial template includes one or more left samples, the current template further includes one or more extended left samples adjacent to the samples of the one or more left-neighboring blocks included in the current template. The one or more extended left samples are not included in an intra-coded block in the current picture.

[0034] If at least one sample in the initial template includes one or more top samples, the current template further includes one or more extended top samples that are adjacent to samples of one or more top-neighboring blocks included in the current template, where the one or more extended top samples are not included in an intra-coded block in the current picture.

[0035] In one example, the number of samples in the current template is 150% of the number of samples in the initial template.

[0036] In one example, the number of samples in the current template is 200% of the number of samples in the initial template.

[0037] Aspects of the present disclosure also provide a non-transitory computer-readable storage medium storing a program executable by at least one processor to implement a method for video decoding.

[0038] Further features, nature and various advantages of the disclosed subject matter will become apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0039] [Figure 1A] FIG. 2 is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B] FIG. 1 is a diagram of an exemplary intra-prediction direction. [Figure 2] FIG. 2 shows an example of a current block (201) and surrounding samples. [Figure 3] FIG. 3 is a schematic diagram of an exemplary block diagram of a communication system (300). [Figure 4] FIG. 4 is a schematic diagram of an exemplary block diagram of a communication system (400). [Figure 5] FIG. 2 is a schematic diagram of an exemplary block diagram of a decoder. [Figure 6] FIG. 2 is a schematic diagram of an exemplary block diagram of an encoder. [Figure 7] FIG. 2 is a block diagram of an exemplary encoder. [Figure 8] FIG. 2 is a block diagram of an exemplary decoder. [Figure 9] FIG. 1 is a diagram illustrating an example of template matching (TM). [Figure 10] FIG. 10 is a diagram illustrating an example of a TM in affine merge mode. [Figure 11] FIG. 1 is a diagram illustrating an example of a TM. [Figure 12A] FIG. 1 is a diagram illustrating an example of a TM. [Figure 12B] FIG. 1 is a diagram illustrating an example of a TM. [Figure 13] FIG. 1 illustrates an example process for obtaining a reference template in a reference picture. [Figure 14] FIG. 10 is a diagram illustrating an example of extending a current template in a TM. [Figure 15] 1 is a flowchart outlining an encoding process according to one embodiment of the present disclosure. [Figure 16] 1 is a flowchart outlining a decoding process according to one embodiment of the present disclosure. [Figure 17] FIG. 1 is a schematic diagram of a computer system, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0040] Figure 3 shows an exemplary block diagram of a communication system (300). The communication system (300) includes multiple terminal devices that may communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) implement unidirectional transmission of data. For example, the terminal device (310) may code video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) may receive the coded video data from the network (350), decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. Unidirectional data transmission may be common in media serving applications, etc.

[0041] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that implement bidirectional transmission of coded video data, for example, during a video conference. In the case of bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) may code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) may also receive coded video data transmitted by the other of the terminal devices (330) and (340), decode the coded video data to recover the video pictures, and display the video pictures on an accessible display device according to the recovered video data.

[0042] In the example of FIG. 3 , terminal devices 310, 320, 330, and 340 are illustrated as a server, a personal computer, and a smartphone, respectively, although the principles of the present disclosure need not be so limited. Embodiments of the present disclosure apply with laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. Network 350 represents any number of networks that convey coded video data between terminal devices 310, 320, 330, and 340, including, for example, wired and / or wireless communication networks. Communication network 350 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network 350 may not be important to the operation of the present disclosure, unless otherwise described herein below.

[0043] 4 shows a video encoder and video decoder in a streaming environment as an example of an application for the disclosed subject matter, which may be equally applicable to other video-enabled applications including, for example, video conferencing, digital television, streaming services, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0044] The streaming system may include a capture subsystem (413), which may include a video source (401), such as a digital camera, that generates a stream of uncompressed video pictures (402). In one example, the stream of video pictures (402) includes samples taken by the digital camera. The stream of video pictures (402), shown in bold to emphasize its large amount of data compared to the encoded video data (404) (or coded video bitstream), may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or coded video bitstream), shown in thin to emphasize its small amount of data compared to the stream of video pictures (402), may be stored on a streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include a video decoder (410), for example, within an electronic device (430). The video decoder (410) decodes the input copy (407) of the encoded video data and creates an output stream (411) of video pictures that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) may be encoded according to a particular video coding / compression standard. Examples of such standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC).The disclosed subject matter can be used in the context of VVC.

[0045] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0046] 5 shows an exemplary block diagram of a video decoder (510). The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., receiving circuitry). The video decoder (510) can be used in place of the video decoder (410) in the example of FIG.

[0047] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In one embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of the other coded video sequences. The coded video sequences are received from a channel (501), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (531) receives the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective using entities (not shown). The receiver (531) may separate the coded video sequences from other data. To combat network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter, "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In other applications, the buffer memory (515) may be external to the video decoder (510) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (510), for example, to combat network jitter, plus another buffer memory (515) internal to the video decoder (510), for example, to handle playout timing. When the receiver (531) is receiving data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may be unnecessary or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (515) may be used, may be relatively large, may advantageously be adaptively sized, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (510).

[0048] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (510) and information for potentially controlling a rendering device, such as a rendering device (512) (e.g., a display screen) that is not an integral part of the electronic device (530) but may be coupled to the electronic device (530), as shown in FIG. 5. The control information for the rendering device may be in the form of a supplemental enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles, including variable length coding with or without context dependency, Huffman coding, arithmetic coding, etc. The parser (520) may extract, from the coded video sequence, a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (520) may also extract from the coded video sequence information such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0049] The parser (520) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (515) to create symbols (521).

[0050] The reconstruction of the symbols (521) can involve several different units, depending on the type of video picture or portion thereof being coded (e.g., inter-picture and intra-picture, inter-block and intra-block, etc.), as well as other factors. Which units are involved and how can be controlled by the parser (520) through subgroup control information parsed from the coded video sequence. The flow of such subgroup control information between the parser (520) and the following units is not depicted for illustrative purposes.

[0051] Beyond the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into several functional units, as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0052] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients as well as control information from the parser (520) as symbols (521), including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. The scalar / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0053] In some cases, the output samples of the scaler / inverse transform unit (551) may relate to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates blocks of the same size and shape as the block being reconstructed using surrounding, already reconstructed information fetched from the current picture buffer (558). The current picture buffer (558), for example, buffers the partially reconstructed and / or fully reconstructed current picture. The aggregator (555) may optionally append the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a sample-by-sample basis.

[0054] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to an inter-coded, potentially motion-compensated, block. In such cases, the motion-compensated prediction unit (553) may access a reference picture memory (557) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (521) related to the block, these samples may be added by the aggregator (555) to the output of the scalar / inverse transform unit (551) (in this case, referred to as residual samples or a residual signal) to generate output sample information. The addresses in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches prediction samples may be controlled by motion vectors available to the motion-compensated prediction unit (553), for example, in the form of symbols (521) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, and the like.

[0055] The output samples of the aggregator (555) may undergo various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filtering techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and provided to the loop filter unit (556) as symbols (521) from the parser (520). Video compression may also be performed in response to meta-information obtained during decoding of the coded picture or a previous portion (in decoding order) of the coded video sequence, or in response to previously reconstructed and loop-filtered sample values.

[0056] The output of the loop filter unit (556) may be a sample stream that can be output to a rendering device (512) and stored in a reference picture memory (557) for use in future inter-picture prediction.

[0057] Once fully reconstructed, a particular coded picture can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be relocated before beginning reconstruction of a subsequent coded picture.

[0058] The video decoder (510) may perform decoding operations according to a given video compression technology or standard, such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technology or standard being used, in the sense that the coded video sequence adheres to both the syntax of the video compression technology or standard and the profile documented in the video compression technology or standard. Specifically, a profile may select certain tools from among all tools available in the video compression technology or standard as the only tools available under that profile. Compliance may also require that the complexity of the coded video sequence be within a range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further constrained by a hypothetical reference decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0059] In one embodiment, the receiver (531) may receive additional (redundant) data with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0060] 6 shows an example block diagram of a video encoder (603). The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG.

[0061] The video encoder (603) may receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of FIG. 6) that may capture video images to be coded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0062] The video source (601) may provide a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. Video data may be provided as multiple individual pictures that convey motion when viewed in sequence. The pictures themselves are organized as a spatial array of pixels, and each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art can readily understand the relationship between pixels and samples. The following description focuses on samples.

[0063] According to one embodiment, the video encoder (603) may code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other required time constraints. Enforcing the appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units described below, which are not depicted for illustrative purposes. Parameters set by the controller (650) may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions for the video encoder (603) optimized for a particular system design.

[0064] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an overly simplified explanation, in one example, the coding loop can include a source coder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to that which a (remote) decoder would also create. The reconstructed sample stream (sample data) is input to a reference picture memory (634). Because decoding of the symbol stream produces bit-exact results regardless of the location (local or remote) of the decoder, the contents of the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronization (the resulting drift when synchronization cannot be maintained due to, for example, channel errors) is also used in several related techniques.

[0065] The operation of the "local" decoder (633) may be the same as the operation of a "remote" decoder, such as the video decoder (510), already described in detail above in conjunction with Figure 5. Referring also briefly to Figure 5, however, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515), and the parser (520), may not be fully implemented in the local decoder (633).

[0066] In one embodiment, the decoder technology, excluding analysis / entropy decoding, present in the decoder is present in the corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on the operation of the decoder. The description of the encoder technology can be omitted, since it is the reverse of the decoder technology described generically. Only in certain areas is more detailed description required and is provided below.

[0067] In operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0068] The local video decoder (633) may decode coded video data of pictures that may be designated as reference pictures based on symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the coded video data is decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in the reference picture memory (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures that would be obtained by the far-end video decoder (without transmission errors).

[0069] The predictor (635) may perform the predictive search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., that may serve as suitable predictive references for the new picture. The predictor (635) may operate on sample block by pixel block to find a suitable predictive reference. In some cases, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory (634), as determined by search results obtained by the predictor (635).

[0070] The controller (650) may manage the coding operations of the source coder (630), including, for example, setting parameters and subgroup parameters used to encode the video data.

[0071] The output of all of the aforementioned functional units may undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0072] The transmitter (640) buffers the coded video sequence created by the entropy coder (645) and prepares it for transmission over a communication channel (660), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) may merge the coded video data from the video encoder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0073] The controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) assigns each coded picture a particular coded picture type, which may affect the coding techniques that may be applied to the respective picture. For example, pictures are often assigned as one of the following picture types:

[0074] An intra picture (I-picture) may be coded and decoded without using other pictures in a sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0075] A predictive picture (P picture) may be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0076] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0077] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to the block's respective picture. For example, blocks of an I-picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0078] The video encoder (603) may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In doing so, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0079] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0080] Video may be captured in time sequence as multiple source pictures (video pictures). Intra-picture prediction (often abbreviated as intra-prediction) uses spatial correlation within a given picture, while inter-picture prediction uses correlation (temporal or other) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture may be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0081] In some embodiments, bi-prediction techniques can be used for inter-picture prediction. According to bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which are before the decoding order of the current picture in the video (but their display orders may be past and future, respectively). A block in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and by a second motion vector pointing to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.

[0082] Furthermore, merge mode techniques can be used for inter-picture prediction to improve coding efficiency.

[0083] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction or intra-picture prediction, is performed on a block-by-block basis. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree-decomposed into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one CU of 64x64 pixels, four CUs of 32x32 pixels, or 16 CUs of 16x16 pixels. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter-prediction type or an intra-prediction type. A CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values) of 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0084] 7 shows an example diagram of a video encoder (703). The video encoder (703) is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video picture in a sequence of video pictures and encode the processed block into a coded picture that is part of a coded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) of the example of FIG. 4.

[0085] In an HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8x8 samples. The video encoder (703) determines whether the processing block is optimally coded using intra-mode, inter-mode, or bi-predictive mode, for example, using rate-distortion optimization. When the processing block is to be coded in intra-mode, the video encoder (703) may encode the processing block into a coded picture using intra-prediction techniques, and when the processing block is to be coded in inter-mode or bi-predictive mode, the video encoder (703) may encode the processing block into a coded picture using inter-prediction techniques or bi-prediction techniques, respectively. In certain video coding techniques, the merge mode may be an inter-picture prediction sub-mode in which a motion vector is derived from one or more motion vector predictors without the aid of coded motion vector components outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the mode of the processing block.

[0086] In the example of Figure 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), which are coupled together as shown in Figure 7.

[0087] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in reference pictures (e.g., blocks in previous and subsequent pictures), generate inter-prediction information (e.g., inter-coding techniques, motion vectors, description of redundant information through merge mode information), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that is decoded based on the coded video information.

[0088] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), possibly compare the block with previously coded blocks in the same picture, generate transformed quantized coefficients, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). In one example, the intra encoder (722) also calculates intra prediction results (e.g., prediction blocks) based on the intra prediction information and reference blocks in the same picture.

[0089] The general-purpose controller (721) is configured to determine general-purpose control data and control other components of the video encoder (703) based on the general-purpose control data. In one example, the general-purpose controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general-purpose controller (721) controls the switch (726) to select the intra-mode result used by the residual calculator (723) and controls the entropy encoder (725) to select intra-prediction information and include the intra-prediction information in the bitstream. When the mode is inter mode, the general-purpose controller (721) controls the switch (726) to select the inter-prediction result used by the residual calculator (723) and controls the entropy encoder (725) to select inter-prediction information and include the inter-prediction information in the bitstream.

[0090] The residual calculator (723) is configured to calculate a difference (residual data) between a received block and a prediction result selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain to generate transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intra-encoder (722) and the inter-encoder (730) as appropriate. For example, the inter-encoder (730) can generate decoded blocks based on the decoded residual data and inter-prediction information, and the intra-encoder (722) can generate decoded blocks based on the decoded residual data and intra-prediction information. In some examples, the decoded blocks are appropriately processed to generate a decoded picture, which can be buffered in a memory circuit (not shown) and used as a reference picture.

[0091] The entropy encoder (725) is configured to format the bitstream to include the encoded blocks. The entropy encoder (725) is configured to include various information in the bitstream in accordance with an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include in the bitstream general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information. It should be noted that, according to the disclosed subject matter, when coding a block in a merged sub-mode of either an inter mode or a bi-prediction mode, no residual information is present.

[0092] 8 shows an example diagram of a video decoder (810). The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) is used in place of the video decoder (410) of the example of FIG. 4.

[0093] In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter-decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-decoder (872), which are coupled together as shown in Figure 8.

[0094] The entropy decoder (871) can be configured to reconstruct, from a coded picture, specific symbols representing the syntax elements of which the coded picture is composed. Such symbols can include, for example, prediction information (e.g., intra-prediction information or inter-prediction information) that may identify the mode in which the block is coded (e.g., intra-mode, inter-mode, bi-prediction mode, inter-mode and bi-prediction mode of merged or other submodes, etc.) and specific samples or metadata used for prediction by the intra decoder (872) or inter decoder (880), respectively. The symbols can also include, for example, residual information in the form of quantized transform coefficients. In one example, when the prediction mode is an inter-mode or bi-prediction mode, the inter-prediction information is provided to the inter decoder (880), and when the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra decoder (872). The residual information can undergo inverse quantization and be provided to the residual decoder (873).

[0095] The inter decoder (880) is configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0096] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0097] The residual decoder (873) is configured to perform inverse quantization to extract inverse quantized transform coefficients and process the inverse quantized transform coefficients to transform the residual information from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantization parameters (QPs)), which may be provided by the entropy decoder (871) (this may be only a small amount of control information, so a data path is not depicted).

[0098] The reconstruction module (874) is configured to combine, in the spatial domain, the residual information output by the residual decoder (873) and the prediction results (possibly output by the inter-prediction module or the intra-prediction module) to form reconstructed blocks that may become part of a reconstructed picture, which may become part of the reconstructed video. It should be noted that other appropriate operations, such as a deblocking operation, may be performed to improve visual quality.

[0099] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0100] VVC can use various inter-prediction modes. For an inter-predicted CU, motion parameters can include MVs, one or more reference picture indexes, a reference picture list usage index, and additional information about specific coding features to be used for inter-predicted sample generation. Motion parameters can be signaled explicitly or implicitly. When a CU is coded in skip mode, the CU is associated with a PU and cannot have significant residual coefficients, coded motion vector deltas, MV differences (e.g., MVDs), or reference picture indexes. A merge mode can be specified, and the motion parameters of the current CU are obtained from neighboring CUs, including spatial candidates, temporal candidates, and / or additional schedules (e.g., schedules introduced in VVC). The merge mode can be applied not only to skip mode but also to inter-predicted CUs. In one example, an alternative to the merge mode is explicit transmission of motion parameters, in which the MVs, corresponding reference picture indexes of each reference picture list, a reference picture list usage flag, and other information are explicitly signaled for each CU.

[0101] In one embodiment, such as VVC, the VVC Test Model (VTM) reference software includes one or more refined inter-predictive coding tools, including enhanced merge prediction, merge motion vector differential (MMVD) mode, adaptive motion vector prediction with symmetric MVD signaling (AMVP) mode, affine motion compensation prediction, sub-block based temporal motion vector prediction (SbTMVP), adaptive motion vector resolution (AMVR), motion field storage (1 / 16 luma sample MV storage and 8x8 motion field compression), bi-prediction with CU-level weights (BCW), bidirectional optical flow (BDOF), prediction refinement using optical flow (PROF), decoder-side motion vector refinement (DMVR), combined inter and intra prediction (CIIP), geometric partitioning mode (GPM), etc.

[0102] Template matching techniques can be used in video / image coding. To further improve the compression efficiency of the VVC standard, for example, template matching (TM) can be used to refine the MV. In one example, TM is used on the decoder side. In TM mode, the MV is refined by constructing a template for a block in the current picture to be reconstructed (e.g., the current block), and the closest match between the template of the block in the current picture and multiple templates in the reference picture can be determined. In one embodiment, the template of the block in the current picture can include the left-neighboring reconstructed sample of the block and the upper-neighboring reconstructed sample of the block. Template matching techniques can be used in video / image coding beyond VVC.

[0103] Figure 9 shows an example of template matching (900). As shown in Figure 9, template matching can be used to derive motion information for a current CU (901) by determining the closest match between a template (921) for the current CU (901) in a current picture (910) and a template (925) in a reference picture (911). The template (921) for the current CU (901) can have any suitable shape and any suitable size.

[0104] In one embodiment, the template (921) of the current CU (901) includes an upper template (922) and a left template (923). Each of the upper template (922) and the left template (923) can have any suitable shape and any suitable size.

[0105] The top template (922) can include samples in one or more top-neighboring blocks of the current CU (901). In one example, the top template (922) includes four rows of samples in one or more top-neighboring blocks of the current CU (901). The left template (923) can include samples in one or more left-neighboring blocks of the current CU (901). In one example, the left template (923) includes four columns of samples in one or more left-neighboring blocks of the current CU (901).

[0106] The template (925) in the reference picture (911) corresponds to the template (921) in the current picture (910). In one embodiment, the initial MV (902) points from the current CU (901) to the reference block (903) in the reference picture (911). The template (925) in the reference picture (911) and the template (921) in the current picture (910) may have the same shape and size. For example, the template (925) includes a top template (926) in the reference picture (911) and a left template (927) in the reference picture (911). The top template (926) may include samples in one or more top neighboring blocks of the reference block (903). The left template (927) may include samples in one or more left neighboring blocks of the reference block (903).

[0107] The TM cost may be determined based on the template (921) and the template (925). The TM cost may indicate a match between the template (921) and the template (925). The optimized MV may be determined based on a search around the initial MV (902) of the current CU (901) within a search range (915). The search range (915) within the reference picture (911) may include a [-L, L] pixel range, where L is a positive integer such as 8 (e.g., 8 samples). For example, a difference (e.g., [0, 1]) is determined based on the search range (915), and an intermediate MV is determined by adding the initial MV (902) and the difference (e.g., [0, 1]). Based on the intermediate MV, an intermediate reference block within the reference picture (911) and a corresponding template may be determined. The TM cost may be determined based on the template (921) within the reference picture (911) and the intermediate template. The TM cost can correspond to a difference (e.g., [0,0], [0,1], etc., corresponding to the initial MV (902)) determined based on the search range (915). In one example, the difference corresponding to the minimum TM cost is selected, and the optimized MV is the sum of the difference corresponding to the minimum TM cost and the initial MV (902).

[0108] The TM can be applied in affine merge mode and can be called an affine TM. Figure 10 shows an example of a TM (1000) in affine merge mode. The template (1021) of a current block (e.g., current CU) (1001) can correspond to a template in the TM (e.g., template (921) in Figure 9) applied to a translational motion model. The reference template (1025) of a reference block in a reference picture can include multiple sub-blocks (e.g., 4x4 sub-blocks) pointed to by control point motion vector (CPMV)-derived motion vectors of neighboring sub-blocks (e.g., A0-A3 and L0-L3 as shown in Figure 10) at CU boundaries.

[0109] The search process of a TM applied in affine merge mode can start from CPMV0 while keeping other CPMVs constant (e.g., (i) CPMV1 if a four-parameter model is used, or (ii) CPMV1 and CPMV2 if a six-parameter model is used). The search can be performed in the horizontal and vertical directions. In one example, the search continues diagonally only if the zero vector is not the best difference vector found from the horizontal and vertical searches. The affine TM can repeat the same search process for CPMV1. The affine TM can repeat the same search process for CPMV2 if a six-parameter model is used. If the zero vector is not the best difference vector from the previous iteration and the search process has iterated less than three times, the entire search process can be restarted from refined CPMV0 based on the refined CPMV.

[0110] In related template matching methods, a template for a current block (e.g., template (921) or (1021)) is generated from reconstructed neighboring pixels of the current block. In some examples, an additional dependency occurs because the TM of the current block cannot be performed until all neighboring pixels used in the template of the current block are reconstructed. The additional dependency may be undesirable, for example, for hardware in which the inter-pipeline and intra-pipeline are separated. In some examples, the template for the current block includes samples in neighboring blocks that are intra-coded (at the decoder side) or intra-coded (at the encoder side). Thus, on the decoder side, if the intra-coded neighboring blocks have not been decoded when the current block is being reconstructed, the samples in the neighboring blocks are not available. Therefore, reconstruction of the current block cannot be performed until the intra-coded neighboring blocks are decoded. The dependency of the template of the current block on the intra-coded neighboring blocks may delay coding of the current block. The present disclosure includes embodiments that can remove the above-mentioned dependency, thereby reducing such delay.

[0111] This disclosure includes embodiments related to constrained template matching. According to one embodiment of this disclosure, the derivation of a template for a block in a current picture (e.g., the current block) is constrained so that the TM for that block does not depend on intra-coded spatially neighboring blocks.

[0112] The template for the current block in the current picture can be generated without using samples (or pixels) from intra-coded blocks in the current picture, so that at the decoder side, the decoder does not need to wait for the intra-coded blocks to be decoded before reconstructing the current block with TM, which can increase the decoding speed.

[0113] In one embodiment, prediction information for a current block in a current picture may be decoded from a video bitstream. The prediction information may indicate a TM mode for the current block. It may be determined whether samples in an initial template for the current block are included in an intra-coded block that is a spatial neighbor of the current block. The initial template for the current block may include samples of one or more top-neighboring blocks of the current block and / or samples of one or more left-neighboring blocks of the current block. In response to determining that samples in the initial template are included in an intra-coded block, the MV of the current block may be determined (i) based on the initial MV of the current block without applying the TM mode, or (ii) based on the initial MV of the current block and a current template that is generated based on the initial template and omits samples in the initial template that are included in the intra-coded block. The current block may be reconstructed based on the determined MV.

[0114] Figure 11 shows an example of template matching (1100). The current CU (901), template (921), initial MV (902), template (925), current picture (910), and reference picture (911) in Figure 11 are described in Figure 9. The template (921) is an initial template for the current block (e.g., the current CU (901)). In one embodiment, it is determined that the samples in the initial template (e.g., the template (921)) for the current block (e.g., (901)) are included in an intra-coded block. Based on the determination that the samples in the initial template are included in an intra-coded block, the MV can be determined based on the initial MV (e.g., the initial MV (902)) for the current block (e.g., the current CU (901)) without applying TM mode. In one example, the MV is the initial MV (e.g., the initial MV (902)) for the current block (e.g., the current CU (901)).

[0115] 11 , in one example, the top template (922) includes sub-templates (922(1) to 922(3)), and the left template (923) includes sub-templates (923(1) to 923(3)). The sub-templates (922(1)), (922(3)), (923(1)), and (923(3)) do not include samples in intra-coded blocks of the current picture (910). The sub-templates (922(2)) and (923(2)) include samples in intra-coded blocks in the current picture (910). In one example, the MV is an initial MV (e.g., initial MV (902)) of the current block (e.g., current CU (901)), and the TM is not applied to the current CU (901).

[0116] In one example, the samples (or pixels) used to generate a template (e.g., 921) of a current block (e.g., 901) are checked. If any of the pixels in the template (e.g., 921) of a current block (e.g., 901) belongs to an intra-coded block, and if the template (e.g., 921) includes a top template (e.g., 922) and a left template (e.g., 923), the template (e.g., 921) of the current block (e.g., 901) is marked as unavailable.

[0117] In one example, the top template (e.g., (922)) and the left template (e.g., (923)) are generated separately. The TM may be used when the top template (e.g., (922)) and / or the left template (e.g., (923)) are available. In one example, the top template (e.g., (922)) is available if none of the pixels in the top template (e.g., (922)) belong to an intra-coded block. In one example, the left template (e.g., (923)) is available if none of the pixels in the left template (e.g., (923)) belong to an intra-coded block in the current picture (910).

[0118] 11 , in one example, an initial template (e.g., (921)) of a current block (e.g., (901)) includes an initial top template (e.g., (922)) including samples of one or more top-neighboring blocks and an initial left template (e.g., (923)) including samples of one or more left-neighboring blocks. It is determined that the samples in the initial template (e.g., (921)) of the current block are included in an intra-coded block. In response to one or more left-neighboring blocks of the current block (e.g., (901)) not including an intra-coded block, the MV of the current block can be determined using a TM mode based on the initial MV (e.g., (902)) and the current template (e.g., (923)) including the initial left template (e.g., (923)). In response to one or more upper neighboring blocks of a current block (e.g., (901)) that do not include an intra-coded block, the MV of the current block can be determined using a TM mode based on an initial MV (e.g., (902)) and a current template (e.g., (922)) that includes an initial upper template (e.g., (922)).

[0119] In one embodiment, the samples (or pixels) used to generate an initial template (template 921) for a current block (e.g., 901) are checked. Samples belonging to intra-coded blocks can be excluded from the current template for the current block. After excluding samples from intra-coded blocks from the current template, corresponding samples in the reference template can be excluded when calculating the matching error between the current template and the reference template.

[0120] FIG. 12A shows an example of a TM (1200A), in which the current CU (901), initial MV (902), current picture (910), reference picture (911), sub-templates (922(1)), (922(3)), (923(1)), and (923(3)) of FIG. 12A are described in FIG. 9 and FIG. 11. Referring to FIG. 11 and FIG. 12A, the initial template for the current CU (901) is template (921). Samples in the initial template (921) for the current CU (901) are determined to be included in an intra-coded block. The MV for the current CU (901) can be determined using a TM mode based on the initial MV (902) and the current template (1221) for the current CU (901). In one example, samples in the initial template (921) included in an intra-coded block include samples in sub-templates (922(2)) and (923(2)). The current template (1221) of the current CU (901) may include sub-templates (922(1)), (922(3)), (923(1)), and (923(3)). The current template (1221) of the current CU (901) omits samples in sub-templates (922(2)) and (923(2)). In one example, the current template (1221) includes samples in an inter-coded block and / or an intra block copy (IBC) coded block.

[0121] Referring again to FIG. 11 , in the reference picture (911), the top template (926) may include sub-templates (926(1) to (926(3))) corresponding to the sub-templates (922(1) to (922(3)), and the left template (927) may include sub-templates (927(1) to (927(3))) corresponding to the sub-templates (923(1) to (923(3)). Referring to FIG. 12A , the reference template (1225) may correspond to the current template (1221), and the reference template (1225) includes sub-templates (926(1)), (926(3)), (927(1)), and (927(3)). The reference template (1225) omits sub-templates (926(2)) and (927(2)).

[0122] In one example, current template 1221 includes top template 1222 and left template 1223. Top template 1222 includes sub-templates 922(1) and 922(3). Left template 1223 includes sub-templates 923(1) and 923(3). Reference template 1225 includes top template 1226 and left template 1227. Top template 1226 includes sub-templates 926(1) and 926(3). Left template 1227 includes sub-templates 927(1) and 927(3).

[0123] The current template of the current CU (901) can include any samples in the initial template of the current CU (901) that are not included in an intra-coded block. The current template of the current CU (901) can omit any samples in the initial template of the current CU (901) that are included in an intra-coded block. Figure 12B shows an example of a TM (1200B), in which the current CU (901), initial MV (902), current picture (910), reference picture (911), top templates (922) and (926), and sub-templates (923(2)) through (922(3)) and (927(2)) through (923(3)) of Figure 12B are described in Figures 9 and 11.

[0124] Referring to Figures 11 and 12B, the initial template for the current CU (901) is the template (921). Samples in the initial template (921) included in the intra-coded block include samples in the sub-template (923(1)). The current template (1321) for the current CU (901) may include the top template (922) and sub-templates (923(2)) to (923(3)). The current template (1321) for the current CU (901) omits samples in the sub-template (923(1)). The reference template (1325) may correspond to the current template (1321), and includes the top template (926) and sub-templates (927(2)) to (927(3)). The reference template (1325) omits the sub-template (927(1)).

[0125] In one example, omitted pixels (e.g., intra-coded pixels) in a current template (e.g., current template (1221) or (1321)) are interpolated based on available pixels in the current template. In one example, interpolated samples may be generated by interpolation based on one or more samples in the current template (e.g., current template (1221) or (1321)). The interpolated samples may correspond to (replace) samples in an initial template (e.g., (921)) included in the intra-coded block. The current template for the current block may further include interpolated samples, which may replace samples in the initial template (e.g., (921)) that were omitted from the current template.

[0126] Omitted pixels (eg, intra-coded pixels) in a reference template (eg, reference template (1225) or (1325)) may be interpolated based on available pixels in the reference template.

[0127] In one embodiment, a reference sample in a reference template (e.g., reference template (1225) or (1325)) corresponding to a current template (e.g., current template (921)) of a current block (e.g., (901)) is placed at a sub-pixel position in a reference picture (e.g., (911)). In this case, an interpolated reference template in the reference picture can be generated using interpolation with a filter having less than four taps in the reference template. The MV of the current block can be determined using a TM mode based on the initial MV, the current template of the current block, and the interpolated reference template.

[0128] In one example, a reference template in a reference picture omits a reference sample in the reference picture that corresponds to a sample in the initial template of a current block included in an intra-coded block. A padded sample that corresponds to (replaces) one of the omitted reference samples in the reference picture can be generated by copying the nearest reference sample in the reference picture. An interpolated reference template in the reference picture can be generated by interpolation based on the padded sample.

[0129] In one example, the reference template (e.g., reference template (1225) or (1325)) is at a sub-pixel position. In this case, interpolation can be performed on the reference template so that the interpolated reference template is at an integer position. To reduce complexity, interpolation can be performed on the reference template using a filter with fewer than four taps (e.g., a two-tap bilinear interpolation filter). If a portion of the reference template is omitted due to the exclusion of an intra-coded block, pixel padding can be performed by copying the nearest pixel and performing interpolation. In one example, no more than two pixels are padded in the required direction.

[0130] In one embodiment, the reference template in the reference picture omits reference samples in the reference picture that correspond to samples in the initial template of the current block contained in the intra-coded block, in which case the interpolated reference template in the reference picture can be generated by interpolation based on the omitted reference samples in the reference picture.

[0131] In one example, pixel padding is not used when the reference template is at a sub-pixel location. In this case, all pixels in the reference picture (e.g., (911)) are available, so interpolation can be based on pixels that should be excluded from the matching process.

[0132] In various examples, when pixels in the current template are reconstructed spatial neighbors at integer pixel locations, the current template does not need to be interpolated.

[0133] In one example, interpolation of the entire reference template in the reference picture is performed at sub-pixel positions. A mask may be used for the reference template to exclude pixels in the reference template that correspond to pixels belonging to intra-coded blocks in the current template. The shape and size of the mask may be based on the shape and size of the current template.

[0134] Figure 13 shows an example process (1300) for obtaining a reference template in a reference picture. In one embodiment, reference samples in an initial reference template (e.g., template (925)) corresponding to an initial template (e.g., (921)) of a current block (e.g., (901)) are located at sub-pixel positions in the reference picture (e.g., (911)). Figure 9 describes the template (925), template (921), current CU (901), and reference picture (911). For convenience of explanation, the template (921) and current CU (901) are not shown in Figure 13. An intermediate reference template (1425) in the reference picture can be generated using interpolation in the initial reference template (e.g., (925)).

[0135] The mask 1415 includes regions 1415(1) through 1415(4) and regions 1415(5) through 1415(6). In one example, the pixel values ​​in regions 1415(1) through 1415(4) are "1," and the pixel values ​​in regions 1415(5) through 1415(6) are "0." Regions 1415(1) through 1415(4) may be referred to as non-zero regions, and regions 1415(5) through 1415(6) may be referred to as zero regions. In one example, the shape of the non-zero regions 1415(1) through 1415(4) of the mask 1415 is identical to the shape of the current template (e.g., 1221) of the current block. In one example, the size of the non-zero regions (1415(1)) to (1415(4)) of the mask (1415) is the same as the size of the current template (e.g., (1221)) of the current block.

[0136] Based on the intermediate reference template (1425) and the mask (1415), a reference template (1435) corresponding to the current template of the current block (e.g., the current template (1221) in FIG. 12A) can be generated. For example, the intermediate reference template (1425) includes sub-templates (1425(1)) through (1425(6)). The non-zero regions (1415(1)) through (1415(4)) in the mask (1415) correspond to the sub-templates (1425(1)), (1425(3)), (1425(4)), and (1425(6)). The reference template (1435) is generated by applying the mask (1415) to the intermediate reference template (1425). In one example, the reference template (1435) includes sub-templates (1435(a)) through (1435(b)). Sub-templates 1435(a)-1435(b) may correspond to sub-templates 1425(1), 1425(3), 1425(4), and 1425(6). In one example, sub-templates 1435(a)-1435(b) are sub-templates 1425(1), 1425(3), 1425(4), and 1425(6), respectively.

[0137] The mask (1415) may include additional non-zero regions that do not overlap with the template (925), where pixels within the additional non-zero regions have a value of "1." The mask (1415) may include additional zero regions that do not overlap with the template (925), where pixels within the additional zero regions have a value of "0." The additional non-zero regions and the additional zero regions do not affect the reference template (1435).

[0138] In one embodiment, when an intra-coded block is in the top or left template area of ​​the current picture, the intra-coded block may be excluded from the current template. The current template may be extended to another part of the current picture. The size of the current template may be increased.

[0139] In one embodiment, samples in an initial template (e.g., template (921)) included in an intra-coded block include (i) one or more left samples of samples of one or more left-neighboring blocks, or (ii) one or more top samples of samples of one or more top-neighboring blocks. If samples in the initial template (e.g., template (921)) include one or more left samples, the current template further includes one or more extended left samples adjacent to the samples of one or more left-neighboring blocks included in the current template. In one example, the one or more extended left samples are not included in an intra-coded block in the current picture. If samples in the initial template include one or more top samples, the current template may further include one or more extended top samples adjacent to the samples of one or more top-neighboring blocks included in the current template. The one or more extended top samples are not included in an intra-coded block in the current picture.

[0140] Figure 14 shows an example of extending a current template in a TM (1400). The current CU (901), initial MV (902), current picture (910), reference picture (911), top template (922) and sub-templates (923(1)), (923(3)), top template (926) and sub-templates (927(1)), (927(3)) in Figure 14 are described in Figures 9 and 11.

[0141] Referring to Figures 11 and 14, the samples in the sub-template (923(2)) of the initial template (921) are within an intra-coded block. The samples in the top template (922) and the samples of the sub-templates (923(1)) and (923(3)) in the initial template (921) are not within an intra-coded block. The current template (1421) of the current CU (901) can include the top template (922) and the sub-templates (923(1)) and (923(3)). The current template (1421) is the sub-template (923(1)), which is an extension of each of the sub-templates (923(1)) and (923(3)) in the current template (1421). e (1)) and (923 e (3)) can further include a sub-template (923 e Subtemplate (923(1)) is adjacent to the left of subtemplate (923(1)). e The sub-template (923(1)) may contain one or more extended left samples adjacent to the samples of the sub-template (923(1)). e Subtemplate (923(3)) is adjacent to the left of subtemplate (923(3)). e The sub-template (923(3)) may include one or more extended left samples adjacent to the samples of the sub-template (923(3)). e (1)) and (923 e The samples in (3)) are not included in the intra-coded block.

[0142] In one example, the number of samples in the current template (eg, (1421)) is 150% of the number of samples in the initial template (eg, (921)).

[0143] In one example, the number of samples in the current template (eg, (1421)) is 200% of the number of samples in the initial template (eg, (921)).

[0144] The explanation using FIG. 14 can be appropriately adapted when the sub-template in the upper template (922) is omitted from the current template.

[0145] 15 shows a flowchart outlining an encoding process (1500) according to one embodiment of the present disclosure. In various embodiments, the process (1500) is performed by processing circuitry, such as processing circuitry within terminal devices (310), (320), (330), and (340), or processing circuitry performing the functions of a video encoder (e.g., (403), (603), (703)). In some embodiments, the process (1500) is implemented by software instructions, and thus, the processing circuitry performs the process (1500) when it executes the software instructions. The process begins at (S1501) and proceeds to (S1510).

[0146] At (S1510), it is determined whether a sample in an initial template for a current block in a current picture is included in a block to be intra-coded. In one example, the intra-coded block is a spatially neighboring block of the current block. The initial template for the current block may include samples of one or more top-neighboring blocks of the current block and / or samples of one or more left-neighboring blocks of the current block. In one example, the intra-coded block is intra-coded. In one example, the intra-coded block is not intra-coded.

[0147] In (S1520), in response to determining that samples in the initial template are included in the block to be intra-coded, the motion vector (MV) of the current block can be determined (i) based on the initial MV of the current block without applying template matching (TM) mode, or (ii) based on the initial MV of the current block and a current template generated based on the initial template and omitting samples included in the block to be intra-coded.

[0148] In one embodiment, the samples in the initial template of the current block are determined to be included in an intra-coded block, in which case the MV can be determined based on the initial MV of the current block without applying the TM mode.

[0149] In one embodiment, the initial template for the current block includes an initial top template including samples of one or more top-neighboring blocks and an initial left template including samples of one or more left-neighboring blocks. The samples of the initial template for the current block are determined to be included in intra-coded blocks. If one or more left-neighboring blocks of the current block do not include intra-coded blocks, the MV for the current block can be determined using a TM mode based on the initial MV and the current template including the initial left template. If one or more top-neighboring blocks of the current block do not include intra-coded blocks, the MV for the current block can be determined using a TM mode based on the initial MV and the current template including the initial top template.

[0150] In one embodiment, a sample in the initial template (e.g., (921)) of the current block is determined to be included in an intra-coded block. In this case, the MV of the current block can be determined using a TM mode based on the initial MV and the current template (e.g., (1221) or (1321)) of the current block, as described in Figures 12A-12B.

[0151] In one example, as described in FIG. 14 , the samples in the initial template included in the intra-coded block include (i) one or more left samples of one or more left-neighboring blocks and / or (ii) one or more top samples of one or more top-neighboring blocks. If the samples in the initial template include one or more left samples, the current template may further include one or more extended left samples adjacent to the samples of the one or more left-neighboring blocks included in the current template. The one or more extended left samples are not included in the intra-coded block in the current picture. If the samples in the initial template include one or more top samples, the current template further includes one or more extended top samples adjacent to the samples of the one or more top-neighboring blocks included in the current template. The one or more extended top samples are not included in the intra-coded block in the current picture. The number of samples in the current template may be greater than the number of samples in the initial template. The number of samples in the current template may be 150% or 200% of the number of samples in the initial template.

[0152] In one example, the interpolated samples are generated by interpolation based on one or more samples in the current template. The interpolated samples may correspond to (replace) samples in an initial template included in an intra-coded block. The current template for the current block may further include the interpolated samples.

[0153] In one example, the reference samples in the reference template corresponding to the current template of the current block are located at sub-pixel positions in the reference picture. The interpolated reference template in the reference picture can be generated using interpolation with a filter having less than four taps in the reference template. The MV of the current block can be determined using a TM mode based on the initial MV, the current template of the current block, and the interpolated reference template.

[0154] In one example, a reference template in a reference picture omits a reference sample in the reference picture that corresponds to a sample in the initial template of a current block included in an intra-coded block. A padded sample that corresponds to (replaces) one of the omitted reference samples in the reference picture can be generated by copying the nearest reference sample in the reference picture. An interpolated reference template in the reference picture can be generated by interpolation based on the padded sample.

[0155] In one example, the reference template in the reference picture omits reference samples corresponding to samples in the initial template of the current block included in the intra-coded block, and the interpolated reference template in the reference picture can be generated by interpolation based on the omitted reference samples in the reference picture.

[0156] In one example, a reference sample in an initial reference template corresponding to an initial template of a current block is located at a sub-pixel position in a reference picture. In this case, an intermediate reference template in the reference picture can be generated using interpolation in the initial reference template. A reference template corresponding to a current template of a current block can be generated based on the intermediate reference template and a mask. The shape of the mask can be determined based on the current template of the current block.

[0157] At (S1530), the current block can be coded based on the determined MV.

[0158] In (S1540), prediction information of the current block can be coded. In one example, the prediction information indicates a TM mode for the current block. In one example, the prediction information includes information indicating an initial MV (e.g., initial MV (902)) used in the TM mode.

[0159] The process (1500) proceeds to (S1599) and ends.

[0160] The process (1500) can be adapted to various scenarios as appropriate, and the steps within the process (1500) can be adjusted accordingly. One or more of the steps within the process (1500) can be adapted, omitted, repeated, and / or combined. Any suitable order can be used to perform the process (1500). Additional steps can be added.

[0161] 16 shows a flowchart outlining a decoding process (1600) according to one embodiment of the present disclosure. In various embodiments, the process (1600) is performed by a processing circuit, such as a processing circuit within the terminal devices (310), (320), (330), and (340), a processing circuit that performs the functions of the video encoder (403), a processing circuit that performs the functions of the video decoder (410), a processing circuit that performs the functions of the video decoder (510), or a processing circuit that performs the functions of the video encoder (603). In some embodiments, the process (1600) is implemented by software instructions, and thus, the processing circuit performs the process (1600) when the processing circuit executes the software instructions. The process begins at (S1601) and proceeds to (S1610).

[0162] At (S1610), prediction information for a current block in a current picture may be decoded from a video bitstream. The prediction information may indicate a template matching (TM) mode for the current block.

[0163] At (S1620), it is determined whether samples in the initial template (or first template) of the current block are included in an intra-coded block. In one example, the intra-coded block is a spatially neighboring block of the current block. The initial template of the current block (e.g., template (921)) may include samples of one or more upper neighboring blocks of the current block and / or samples of one or more left neighboring blocks of the current block.

[0164] In (S1630), in one example, if it is determined that a sample in the initial template is included in an intra-coded block, the motion vector (MV) of the current block can be determined based on the initial MV of the current block without applying the TM mode. For example, referring to Figure 11, if a sample in the sub-template (922(2)) is included in an intra-coded block, the MV of the current block (e.g., the current CU (901)) can be determined based on the initial MV (902) of the current block without applying the TM mode.

[0165] Alternatively, if it is determined in (S1630) that a sample in the initial template is included in an intra-coded block, the MV of the current block can be determined based on the initial MV of the current block and the current template (or a second template), as described in Figures 11, 12A, and 12B. The current template can be generated based on the initial template, and the samples in the initial template that are included in the intra-coded block can be omitted.

[0166] In one embodiment, the initial template for the current block includes an initial top template including samples of one or more top-neighboring blocks and an initial left template including samples of one or more left-neighboring blocks. The samples in the initial template for the current block are determined to be included in intra-coded blocks. In this case, if the left-neighboring block of the initial left template does not include an intra-coded block, the MV for the current block can be determined using a TM mode based on the initial MV and the current template, which is the initial left template. If one or more top-neighboring blocks for the current block do not include an intra-coded block, the MV for the current block can be determined using a TM mode based on the initial MV and the current template, which includes the initial top template.

[0167] In one embodiment, a sample in the initial template (e.g., (921)) of the current block is determined to be included in an intra-coded block. In this case, the MV of the current block can be determined using a TM mode based on the initial MV and the current template (e.g., (1221) or (1321)) of the current block, as described in Figures 12A-12B.

[0168] In one example, as described in FIG. 14 , the samples in the initial template included in the intra-coded block include (i) one or more left samples of one or more left-neighboring blocks and / or (ii) one or more top samples of one or more top-neighboring blocks. If the samples in the initial template include one or more left samples, the current template may further include one or more extended left samples adjacent to the samples of the one or more left-neighboring blocks included in the current template. The one or more extended left samples are not included in the intra-coded block in the current picture. If the samples in the initial template include one or more top samples, the current template further includes one or more extended top samples adjacent to the samples of the one or more top-neighboring blocks included in the current template. The one or more extended top samples are not included in the intra-coded block in the current picture. The number of samples in the current template may be greater than the number of samples in the initial template. The number of samples in the current template may be 150% or 200% of the number of samples in the initial template.

[0169] In one example, the interpolated samples are generated by interpolation based on one or more samples in the current template. The interpolated samples may correspond to (replace) samples in an initial template included in an intra-coded block. The current template for the current block may further include the interpolated samples.

[0170] In one example, the reference sample in the reference template corresponding to the current template of the current block is located at a sub-pixel position in the reference picture. In this case, an interpolated reference template in the reference picture can be generated using interpolation with a filter having less than four taps in the reference template. The MV of the current block can be determined using a TM mode based on the initial MV, the current template of the current block, and the interpolated reference template.

[0171] In one example, a reference template in a reference picture omits a reference sample corresponding to a sample in an initial template of a current block included in an intra-coded block. A padded sample corresponding to (replacing) one of the omitted reference samples in the reference picture can be generated by copying the nearest reference sample in the reference picture. An interpolated reference template in the reference picture can be generated by interpolation based on the padded sample.

[0172] In one example, a reference template in a reference picture omits reference samples that correspond to samples in the initial template of a current block included in an intra-coded block, in which case an interpolated reference template in the reference picture can be generated by interpolation based on the omitted reference samples in the reference picture.

[0173] In one example, reference samples in an initial reference template corresponding to an initial template for a current block are located at sub-pixel positions in a reference picture. An intermediate reference template in a reference picture can be generated using interpolation in the initial reference template. A reference template corresponding to a current template for a current block can be generated based on the intermediate reference template and a mask. The shape of the mask can be determined based on the current template for the current block.

[0174] At (S1640), the current block can be reconstructed based on the determined MV.

[0175] The process (1600) proceeds to (S1699) and ends.

[0176] The process (1600) can be adapted appropriately to various scenarios, and the steps within the process (1600) can be adjusted accordingly. One or more of the steps within the process (1600) can be adapted, omitted, repeated, and / or combined. Any suitable order can be used to perform the process (1600). Additional steps can be added.

[0177] In one embodiment, it is determined whether at least one sample in an initial template (or a first template) for a current block is included in an intra-coded block that is a spatial neighboring block of the current block. The first template for the current block may include one or more samples from (1) one or more top-neighboring blocks of the current block and (2) one or more left-neighboring blocks of the current block. In one example, the one or more top-neighboring blocks are top-neighboring blocks. In one example, the one or more left-neighboring blocks are left-neighboring blocks. In response to determining that at least one sample in the first template is included in an intra-coded block, at least one sample may be excluded from the first template to generate a current template (or a second template) for the current block. The MV of the current block may be determined based on the second template for the current block.

[0178] In one example, in response to (i) a determination that at least one sample in the first template is included in an intra-coded block, and (ii) TM mode is applied to the current block, at least one sample can be excluded from the first template to generate a second template for the current block.

[0179] The embodiments of the present disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), the encoder, and the decoder may be implemented by processing circuitry (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0180] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 17 illustrates a computer system (1700) suitable for implementing certain embodiments of the subject matter of this disclosure.

[0181] Computer software may be coded using any suitable machine code or computer language that can be subjected to assembly, compilation, linking, or similar mechanisms to produce code containing instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc. directly, or through interpretation and microcode execution, etc.

[0182] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.

[0183] 17 for computer system (1700) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1700).

[0184] The computer system (1700) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), and video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0185] The input human interface devices may include one or more (only one of each shown) of a keyboard (1701), a mouse (1702), a trackpad (1703), a touchscreen (1710), a data glove (not shown), a joystick (1705), a microphone (1706), a scanner (1707), and a camera (1708).

[0186] The computer system (1700) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1710), data gloves (not shown), or joystick (1705), although some haptic feedback devices may not function as input devices), audio output devices (such as speakers (1709), headphones (not shown), etc.), visual output devices (such as screens (1710), including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capability and each with or without haptic feedback capability, some of which may be capable of two-dimensional visual output or three-dimensional or higher-dimensional output via means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0187] The computer system (1700) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (1720) with media such as CD / DVD (1721), thumb drives (1722), removable hard drives or solid state drives (1723), legacy magnetic media such as tape and floppy disks (not shown), and dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0188] Additionally, those skilled in the art should understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0189] The computer system (1700) also includes an interface (1754) to one or more communication networks (1755). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet; cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, and the like; television wired or wireless wide-area digital networks including cable, satellite, and terrestrial television; and vehicular and industrial networks including CAN Bus. Certain networks generally require an external network interface adapter connected to a particular general-purpose data port or peripheral bus (1749) (e.g., a USB port on the computer system (1700)); others are generally integrated into the core of the computer system (1700) by connection to a system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1700) can communicate with other entities. Such communication may be unidirectional, receive only (e.g., broadcast TV), unidirectional transmit only (e.g., from the CANbus to a particular CANbus device), or bidirectional, e.g., communication with other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of those networks and network interfaces, as described above.

[0190] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (1740) of the computer system (1700).

[0191] The core (1740) may include one or more central processing units (CPUs) (1741), graphics processing units (GPUs) (1742), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1743), hardware accelerators for specific tasks (1744), graphics adapters (1750), etc. These devices may be connected via a system bus (1748), along with read-only memory (ROM) (1745), random access memory (1746), and internal mass storage devices (1747) such as internal hard drives or SSDs that are not user accessible. In some computer systems, the system bus (1748) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1748) or via a peripheral bus (1749). In one example, a screen (1710) may be connected to the graphics adapter (1750). Architectures for peripheral buses include PCI, USB, etc.

[0192] The CPU (1741), GPU (1742), FPGA (1743), and accelerator (1744) can execute specific instructions that may combine to form the above-mentioned computer code. The computer code can be stored in ROM (1745) or RAM (1746). Transient data can also be stored in RAM (1746), while persistent data can be stored in, for example, internal mass storage device (1747). The use of cache memory, which can be closely associated with one or more CPUs (1741), GPUs (1742), mass storage devices (1747), ROM (1745), RAM (1746), etc., allows for fast storage and retrieval from any of the memory devices.

[0193] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0194] By way of example and not limitation, a computer system having the architecture (1700), and in particular the core (1740), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage devices, as introduced above, as well as media associated with specific storage devices of the core (1740) that are non-transitory in nature, such as the core's internal mass storage device (1747) or ROM (1745). Software implementing various embodiments of the present disclosure can be stored on such devices and executed by the core (1740). The computer-readable media may include one or more memory devices or chips, depending on particular needs. The software enables the core (1740), and in particular the processor (including a CPU, GPU, FPGA, etc.) within the core (1740), to perform certain operations, or certain portions of certain operations, described herein, including defining data structures stored in RAM (1746) and modifying such data structures according to software-defined operations. Additionally, or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1744)) that may operate in place of or in conjunction with software to perform particular processes, or particular portions of particular processes, described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, as appropriate. The present disclosure encompasses any appropriate combination of hardware and software. Appendix A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplemental Extended Information VUI: Video Usability Information GOP: Group of Pictures TU: Conversion unit PU: Prediction Unit CTU: Coding Tree Unit CTB: coding tree block PB: Predicted Block HRD: Hypothetical Reference Decoder SNR: Signal-to-Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Area SSD: Solid State Drive IC: Integrated Circuit CU: Coding Unit RD: Rate Distortion

[0195] While this disclosure has described several exemplary embodiments, there are modifications, substitutions, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. [Explanation of symbols]

[0196] 101 Samples 102 Arrow 103 Arrow 104 Square Blocks 201 Current Block 202 Surrounding Samples 203 Surrounding Samples 204 Surrounding Samples 205 Surrounding Samples 206 Surrounding Samples 300 Communication Systems 310 Terminal Devices 320 terminal devices 330 Terminal Devices 340 Terminal Devices 350 Network 400 Communication Systems 401 Video Source 402 Video Picture Stream 403 Video Encoder 404 encoded video data 405 Streaming Server 406 Client Subsystem 407 Copy 408 Client Subsystem 409 Copy 410 Video Decoder 411 Video Picture Output Stream 412 Display 413 Capture Subsystem 420 Electronic Devices 430 Electronic Devices 501 Channel 510 Video Decoder 512 rendering device 515 buffer memory 520 Parser 521 Symbol 530 Electronic Devices 531 Receiver 551 Scaler / Descaler Unit 552 Intra-picture prediction unit 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter Unit 557 Reference Picture Memory 558 Current Picture Buffer 601 Video Sources 603 Video Encoder 620 Electronic Devices 630 Source Coder 632 Coding Engine 633 Local Video Decoder 634 Reference Picture Memory 635 Predictors 640 Transmitter 643 coded video sequence 645 Entropy Coder 650 Controller 660 Communication Channels 703 Video Encoder 721 General-purpose controller 722 Intra Encoder 723 Residual Calculator 724 Residual Encoder 725 Entropy Encoder 726 Switch 728 Residual Decoder 730 Interencoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Reconstruction Module 880 Interdecoder 900 Template Matching 901 Current Coding Unit (CU) 902 Initial Motion Vector (MV) 903 Reference Block 910 Current Picture 911 Reference Picture 915 Search Range 921 Templates 922 Upper Template 922(1) Subtemplate 922(2) Sub-template 922(3) Subtemplate 923(1) Subtemplate 923(2) Sub-template 923(3) Subtemplate 923 Left Template 925 Templates 926 Upper Template 926(1) Subtemplate 926(2) Sub-template 926(3) Sub-template 927 Left Template 927(1) Subtemplate 927(2) Sub-template 927(3) Sub-template 1000 Template Matching(TM) 1001 Current Block 1021 Template 1025 Reference Template 1100 Template Matching 1200A TM 1200B TM 1221 Current Template 1222 Upper Template 1223 Left Template 1225 Reference Template 1226 Upper Template 1227 Left Template 1300 processes 1321 Current Template 1325 Reference Template 1400 TM 1415 Mask 1415(1) Area 1415(2) Area 1415(3) Area 1415(4) Area 1415(5) Area 1415(6) Area 1421 Current Template 1425 Intermediate Reference Template 1425(1) Subtemplate 1425(2) Subtemplate 1425(3) Subtemplate 1425(4) Subtemplate 1425(5) Subtemplate 1425(6) Subtemplate 1435 Reference Template 1435(a) Subtemplate 1435(b) Subtemplate 1500 processes 1700 Computer Systems 1701 Keyboard 1702 Mouse 1703 Trackpad 1705 Joystick 1706 Microphone 1707 Scanner 1708 Camera 1709 Speaker 1710 Touchscreen 1720 CD / DVD ROM / RW 1721 Medium 1722 thumb drive 1723 Solid State Drive 1740 cores 1741 Central Processing Unit (CPU) 1742 Graphics Processing Unit (GPU) 1743 Field Programmable Gate Area (FPGA) 1744 Hardware Accelerator 1745 Read-Only Memory (ROM) 1746 Random Access Memory (RAM) 1747 Internal mass storage 1748 System Bus 1749 General Purpose Data Port or Peripheral Bus 1750 graphics adapter 1754 Interface 1755 Communication Network

Claims

1. decoding prediction information for a current block in a current picture from a video bitstream, the prediction information indicating a template matching (TM) mode for the current block; determining whether at least one sample in a first template of the current block is included in an intra-coded block that is a spatial neighboring block of the current block, wherein the first template of the current block includes samples from one or more of: (1) a top neighboring block of the current block; and (2) a left neighboring block of the current block; In response to (i) determining that the at least one sample in the first template is included in the intra-coded block, and (ii) that the TM mode is applied to the current block, excluding the at least one sample from the first template to generate a second template for the current block; determining a motion vector (MV) for the current block based on the second template for the current block; reconstructing the current block based on the determined MV; 1. A method for video decoding in a video decoder, comprising:

2. The at least one sample in the first template of the current block is determined to be included in the intra-coded block; determining the MV includes determining the MV based on an initial MV of the current block without applying the TM mode. The method of claim 1.

3. the first template for the current block includes an initial top template including samples of the top neighboring block and an initial left template including samples of the left neighboring block; The at least one sample in the first template of the current block is determined to be included in the intra-coded block; determining the motion vector (MV) in response to the left neighboring block of the current block that does not include the intra-coded block includes determining the motion vector of the current block using the TM mode based on an initial motion vector (MV) and the second template that includes the initial left template; determining the motion vector (MV) in response to the top neighboring block of the current block that does not include the intra-coded block includes determining the motion vector of the current block using the TM mode based on the initial motion vector (MV) and the second template that includes the initial top template. The method of claim 1.

4. The at least one sample in the first template of the current block is determined to be included in the intra-coded block; determining the MV includes determining the MV of the current block using the TM mode based on an initial MV and the second template of the current block; The method of claim 1.

5. the method comprising generating at least one interpolated sample by interpolation based on one or more samples in the second template, the at least one interpolated sample corresponding to the at least one sample in the first template included in the intra-coded block; the second template of the current block further comprises the at least one interpolated sample. The method of claim 4.

6. a reference sample in a reference template corresponding to the second template of the current block is placed at a sub-pixel position in a reference picture; the method further comprising generating an interpolated reference template in the reference picture using interpolation with a filter having less than four taps in the reference template; determining the motion vector includes determining the motion vector of the current block using the TM mode based on the initial motion vector, the second template of the current block, and the interpolated reference template; The method of claim 4.

7. The reference template in the reference picture omits at least one reference sample in the reference picture that corresponds to the at least one sample in the first template of the current block included in the intra-coded block; the method comprising generating a padded sample corresponding to one of the at least one omitted reference sample in the reference picture by copying a nearest reference sample in the reference picture; generating the interpolated reference template comprises generating the interpolated reference template in the reference picture by the interpolation based on the padded samples. The method of claim 6.

8. The reference template in the reference picture omits at least one reference sample in the reference picture that corresponds to the at least one sample in the first template of the current block included in the intra-coded block; generating the interpolated reference template includes generating the interpolated reference template in the reference picture by the interpolation based on the omitted at least one reference sample in the reference picture. The method of claim 6.

9. a reference sample in an initial reference template corresponding to the first template of the current block is placed at a sub-pixel position in a reference picture; The method comprises: generating an intermediate reference template in the reference picture using interpolation in the initial reference template; generating a reference template corresponding to the second template of the current block based on the intermediate reference template and a mask, the shape of the mask being determined based on the second template of the current block; The method of claim 4 further comprising:

10. the at least one sample in the first template included in the intra-coded block includes (i) one or more left samples of the samples of the left-neighboring block, or (ii) one or more top samples of the samples of the top-neighboring block; In response to the at least one sample in the first template including the one or more left samples, the second template further includes one or more extended left samples adjacent to the samples of the left-neighboring block included in the second template, wherein the one or more extended left samples are not included in an intra-coded block in the current picture; In response to the at least one sample in the first template including the one or more top samples, the second template further includes one or more extended top samples adjacent to the samples of the top-neighboring block included in the second template, wherein the one or more extended top samples are not included in an intra-coded block in the current picture. The method of claim 4.

11. The method of claim 10 , wherein the number of samples in the second template is 150% of the number of samples in the first template.

12. The method of claim 10 , wherein the number of samples in the second template is 200% of the number of samples in the first template.

13. A processing circuit configured to implement the method of any one of claims 1 to 12.

1. An apparatus for video decoding, comprising:

14. A program comprising instructions which, when executed by at least one processor, cause the processor to perform the method of any one of claims 1 to 12.

Images