Template availability for template matching tools

Abstract

Example techniques for encoding and decoding video data are disclosed. An example apparatus for encoding and decoding video data includes one or more processors. The one or more processors are configured to determine a current template for a current block of video data. The one or more processors are configured to determine a potential reference block for a current block in a picture of video data. The one or more processors are configured to determine that the potential reference block and a reference template of the potential reference block are fully within the picture, the reference template having the same shape as the current template. The one or more processors are configured to perform template matching between the reference template and the current template based on both the potential reference block and the reference template being completely within the picture, and to encode the current block based on the template matching.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 632,979, filed April 11, 2024, and U.S. Provisional Application No. 63 / 496,378, filed April 14, 2023, the entire contents of each of which are incorporated herein by reference. U.S. Patent Application No. 18 / 632,979, filed April 11, 2024, claims the benefit of U.S. Provisional Application No. 63 / 496,378, filed April 14, 2023. Technical Field

[0002] This disclosure relates to video encoding and video decoding. Background Technology

[0003] Digital video capabilities can be integrated into a wide variety of devices, including digital televisions, digital live broadcasting systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite wireless phones, so-called "smartphones," video conferencing equipment, video streaming devices, and more. Digital video devices implement video codec technologies (such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), ITU-T H.266 / Versatile Video Coding (VVC), and extensions to these standards) as well as proprietary video codecs / formats (such as AV1 (AOMedia Video 1) developed by the Open Media Alliance). By implementing these video encoding and decoding technologies, video devices can more effectively send, receive, encode, decode, and / or store digital video information.

[0004] Video coding and decoding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove inherent redundancy in video sequences. For block-based video coding and decoding, a video slice (e.g., a video picture or a portion of a video picture) can be divided into video blocks, which may also be referred to as coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in a slice of intra-frame coding and decoding (I) of pictures are encoded using spatial prediction relative to reference samples in adjacent blocks of the same picture. Video blocks in a slice of inter-frame coding and decoding (P or B) of pictures can use either spatial prediction relative to reference samples in adjacent blocks of the same picture or temporal prediction relative to reference samples in other reference pictures. Pictures may be referred to as frames, and reference pictures may be referred to as reference frames. Summary of the Invention

[0005] In general, this disclosure describes techniques for template matching in video encoding and decoding. Current template matching search algorithms are overly restrictive in terms of search regions close to image boundaries and reconstructed region boundaries (e.g., boundaries of already reconstructed regions). The techniques described herein can improve the quality of encoded and decoded video by addressing these shortcomings of current template matching search algorithms. For example, the techniques described herein can provide additional potential reference blocks that take into account potential reference blocks and / or cases where the current block may be close to image boundaries or reconstructed region boundaries.

[0006] In one example, a method includes: determining a current template for a current block of video data; determining a potential reference block for the current block in an image of the video data; determining that the potential reference block and its reference template are completely within the image, the reference template having the same shape as the current template; performing template matching between the reference template and the current template based on the fact that both the potential reference block and the reference template are completely within the image; and encoding / decoding the current block based on the template matching.

[0007] In another example, a device includes: one or more memories configured to store video data; and one or more processors implemented in a circuit and communicatively coupled to the one or more memories, the one or more processors being configured to: determine a current template of a current block of video data; determine a potential reference block for the current block in a picture of the video data; determine that the potential reference block and its reference template are completely within the picture, the reference template having the same shape as the current template; perform template matching between the reference template and the current template based on the fact that both the potential reference block and the reference template are completely within the picture; and encode / decode the current block based on the template matching.

[0008] In another example, a computer-readable storage medium is encoded with instructions that, when executed, cause one or more processors to: determine a current template for a current block of video data; determine a potential reference block for the current block in a picture of the video data; determine that the potential reference block and its reference template are entirely within the picture, the reference template having the same shape as the current template; perform template matching between the reference template and the current template based on the fact that both the potential reference block and the reference template are entirely within the picture; and encode or decode the current block based on the template matching.

[0009] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will become clear from the specification, drawings, and claims. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating an example video encoding and decoding system that can perform the techniques of this disclosure.

[0011] Figure 2 This is a conceptual diagram showing the search area for template matching within an example frame.

[0012] Figure 3 This is a conceptual diagram illustrating an example of template matching performed over a search region around an initial motion vector (MV).

[0013] Figure 4 This is a tabular diagram illustrating the search patterns of Adaptive Motion Vector Resolution (AMVR) and the merging modes under AMVR.

[0014] Figure 5 This is a conceptual diagram showing reference samples of the template in the example template and reference image.

[0015] Figure 6 It is a conceptual diagram showing the template and reference points of a block when the sub-blocks are moving using the motion information of the current block's sub-blocks.

[0016] Figure 7 It is a tabular diagram showing the templates of the first and second geometric partitions, where A indicates the use of the upper sample points, L indicates the use of the left sample points, and L+A indicates the use of both the left and upper sample points.

[0017] Figure 8 is a conceptual diagram showing an example intra-block copy (IBC) reference area depending on the current CU location.

[0018] Figure 9 This is a conceptual diagram showing a sample template section.

[0019] Figures 10A-10C This is a conceptual diagram illustrating an example of image boundary processing that depends on the template type.

[0020] Figure 11 This is a conceptual diagram illustrating the boundary treatment of an alternative example image for an L-shaped example.

[0021] Figure 12 This is a conceptual diagram illustrating an example of filling unavailable samples by copying values ​​from available samples.

[0022] Figure 13 This is a flowchart illustrating an example template matching technique according to one or more aspects of this disclosure.

[0023] Figure 14 This is a block diagram illustrating an example video encoder that can perform the techniques of this disclosure.

[0024] Figure 15 This is a block diagram illustrating an example video decoder that can perform the techniques disclosed herein.

[0025] Figure 16 This is a flowchart illustrating an example method for encoding the current block according to the technology of this disclosure.

[0026] Figure 17 This is a flowchart illustrating an example method for decoding the current block according to the technology of this disclosure. Detailed Implementation

[0027] For some video content, current template matching search algorithms may impose excessive restrictions on the search area near image boundaries and reconstructed region boundaries. By limiting potential matching templates to these finite search areas near image boundaries or reconstructed region boundaries, template matching search algorithms exclude better matches in some cases. This can unduly limit encoding / decoding efficiency and / or the quality of decoded images. The techniques described in this paper can improve encoding / decoding efficiency and / or the quality of encoded / decoded video by addressing these shortcomings of current template matching search algorithms.

[0028] Figure 1 This is a block diagram illustrating an example video encoding and decoding system 100 capable of performing the techniques of this disclosure. The techniques of this disclosure are generally directed to encoding and / or decoding video data. In general, video data includes any data used for processing video. Therefore, video data can include raw, unencoded video, encoded video, decoded (e.g., reconstructed) video, and video metadata, such as signaling data.

[0029] like Figure 1As shown, in this example, system 100 includes source device 102, which provides encoded video data to be decoded and displayed by destination device 116. Specifically, source device 102 provides video data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 can be or include any of a wide variety of devices, such as desktop computers, laptop computers, mobile devices, tablet computers, set-top boxes, mobile phones (such as smartphones), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, etc. In some cases, source device 102 and destination device 116 may be equipped for wireless communication and therefore may be referred to as wireless communication devices.

[0030] exist Figure 1 In the example, source device 102 includes a video source 104, a memory 106, a video encoder 200, and an output interface 108. Destination device 116 includes an input interface 122, a video decoder 300, a memory 120, and a display device 118. According to this disclosure, the video encoder 200 of source device 102 and the video decoder 300 of destination device 116 can be configured to apply techniques for template matching. Thus, source device 102 represents an example of a video encoding device, while destination device 116 represents an example of a video decoding device. In other examples, the source device and destination device may include other components or arrangements. For example, source device 102 may receive video data from an external video source, such as an external camera. Similarly, destination device 116 may interface with an external display device, rather than including an integrated display device.

[0031] like Figure 1 The system 100 shown is merely an example. Generally, any digital video encoding and / or decoding device can perform techniques for template matching. The source device 102 and destination device 116 are merely examples of such encoding / decoding devices, where the source device 102 generates encoded / decoded video data for transmission to the destination device 116. This disclosure refers to a “encoding / decoding” device as a device that performs data encoding / decoding (encoding and / or decoding). Thus, video encoder 200 and video decoder 300 represent examples of encoding / decoding devices, specifically examples of a video encoder and a video decoder, respectively. In some examples, the source device 102 and destination device 116 can operate in a substantially symmetrical manner, such that each of the source device 102 and destination device 116 includes video encoding and decoding components. Therefore, system 100 can support one-way or two-way video transmission between the source device 102 and destination device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0032] Generally, video source 104 represents a video data source (i.e., raw, unencoded video data) and provides a continuous series of pictures (also called “frames”) of video data to video encoder 200, which encodes the data of the pictures. Video source 104 of source device 102 may include a video capture device (such as a camera), a video archive containing previously captured raw video, and / or a video feed interface receiving video from a video content provider. As a further alternative, video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes the captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange the pictures from the received order (sometimes called “display order”) into an encoding / decoding order for encoding and decoding. Video encoder 200 may generate a bitstream comprising encoded video data. The source device 102 can then output the encoded video data to a computer-readable medium 110 via the output interface 108 for reception and / or retrieval, for example, by the input interface 122 of the destination device 116.

[0033] The memory 106 of source device 102 and the memory 120 of destination device 116 represent general-purpose memory. In some examples, memories 106 and 120 may store raw video data, such as raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106 and 120 may store software instructions executable by, for example, video encoder 200 and video decoder 300. Although memories 106 and 120 are shown as separate from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store encoded video data, such as video data output from video encoder 200 and input to video decoder 300. In some examples, portions of memories 106 and 120 may be allocated as one or more video buffers, for example, to store raw decoded and / or encoded video data.

[0034] Computer-readable medium 110 can represent any type of medium or device capable of transmitting encoded video data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium that enables source device 102 to transmit encoded video data directly to destination device 116 in real time, for example, via a radio frequency network or a computer-based network. According to communication standards such as wireless communication protocols, output interface 108 can modulate the transmitted signal including the encoded video data, and input interface 122 can demodulate the received transmitted signal. The communication medium can include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network, such as a local area network, a wide area network, or a global network (such as the Internet). The communication medium can include a router, switch, base station, or any other device that can be used to facilitate communication from source device 102 to destination device 116.

[0035] In some examples, source device 102 can output encoded data to storage device 112 from output interface 108. Similarly, destination device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0036] In some examples, source device 102 may output encoded video data to file server 114 or another intermediate storage device that may store the encoded video data generated by source device 102. Destination device 116 may access the stored video data from file server 114 via streaming or download.

[0037] File server 114 can be any type of server device capable of storing encoded video data and transferring such encoded video data to destination device 116. File server 114 can represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a hypertext transfer protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and / or a network attached storage (NAS) device. File server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real Time Streaming Protocol (RTSP), and HTTP Dynamic Streaming.

[0038] Destination device 116 can access encoded video data from file server 114 via any standard data connection, including an internet connection. This may include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on file server 114. Input interface 122 can be configured to operate according to any one or more of the various protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data.

[0039] Output interface 108 and input interface 122 may represent a wireless transmitter / receiver, a modem, a wired network component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 may be configured to transmit data (such as encoded video data) according to cellular communication standards such as 4G, 4G-LTE (Long Term Evolution), Advanced LTE, 5G, etc. In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 may be configured to transmit data (such as encoded video data) according to other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee™), the Bluetooth™ standard, etc. In some examples, source device 102 and / or destination device 116 may include their respective system-on-a-chip (SoC) devices. For example, source device 102 may include a SoC device to perform functions attributed to video encoder 200 and / or output interface 108, and destination device 116 may include a SoC device to perform functions attributed to video decoder 300 and / or input interface 122.

[0040] The technology disclosed herein can be applied to video encoding and decoding that supports any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission (such as HTTP-based Dynamic Adaptive Streaming (DASH)), digital video encoded onto data storage media, decoding digital video stored on data storage media, or other applications.

[0041] The input interface 122 of the destination device 116 receives an encoded video bitstream from a computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200 and used by the video decoder 300, such as syntax elements having values ​​describing the characteristics and / or processing of video blocks or other encoding / decoding units (e.g., stripes, pictures, picture groups, sequences, etc.). The display device 118 displays decoded images of the decoded video data to the user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0042] Although not in Figure 1 As shown, however, in some examples, the video encoder 200 and the video decoder 300 may each be integrated with the audio encoder and / or audio decoder, and may include appropriate MUX-DEMUX units or other hardware and / or software to handle multiplexed streams that include both audio and video in a common data stream.

[0043] The video encoder 200 and video decoder 300 can each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When these technologies are implemented in part in software, the device can store software instructions in a suitable non-transitory computer-readable medium and execute those instructions in hardware using one or more processors to perform the technologies of this disclosure. Each of the video encoder 200 and video decoder 300 can be included in one or more encoders or decoders, and either encoder or decoder can be integrated as part of a combined encoder / decoder (CODEC) in the respective device. Devices including the video encoder 200 and / or video decoder 300 can implement the video encoder 200 and / or video decoder 300 in processing circuitry such as integrated circuits and / or microprocessors. Such devices can be wireless communication devices, such as cellular phones, or any other type of device described herein.

[0044] Video encoder 200 and video decoder 300 may operate according to video codec standards such as ITU-T H.265 (also known as High Efficiency Video Codec (HEVC)) or its extensions (such as Multi-View and / or Scalable Video Codec Extensions). Alternatively, video encoder 200 and video decoder 300 may operate according to other proprietary or industry standards such as ITU-T H.266 (also known as Multi-Functional Video Codec (VVC)). In other examples, video encoder 200 and video decoder 300 may operate according to proprietary video codecs / formats such as AOMedia Video 1 (AV1), extensions to AV1, and / or successors to AV1 (e.g., AV2). In other examples, video encoder 200 and video decoder 300 may operate according to other proprietary formats or industry standards. However, the techniques disclosed herein are not limited to any particular codec standard or format. Generally, video encoder 200 and video decoder 300 may be configured to perform the techniques of this disclosure in combination with any template-matching video codec technique.

[0045] Generally, the video encoder 200 and video decoder 300 can perform block-based encoding and decoding of images. The term "block" typically refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used during encoding and / or decoding). For example, a block may include a two-dimensional matrix of samples of luminance and / or chrominance data. Generally, the video encoder 200 and video decoder 300 can encode and decode video data represented in YUV (e.g., Y, Cb, Cr) format. That is, the video encoder 200 and video decoder 300 can encode and decode luminance and chrominance components, rather than encoding and decoding the red, green, and blue (RGB) data of image samples, where the chrominance components may include chrominance components of red and blue hues. In some examples, the video encoder 200 converts the received RGB format data to a YUV representation before encoding, and the video decoder 300 converts the YUV representation to RGB format. Alternatively, preprocessing and postprocessing units (not shown) may perform these conversions.

[0046] This disclosure generally relates to the encoding and decoding of images (e.g., encoding and decoding), including the process of encoding or decoding image data. Similarly, this disclosure may relate to the encoding and decoding of blocks of images (e.g., predictive encoding and / or residual encoding and decoding), including the process of encoding or decoding block data. Encoded video bitstreams typically include a series of syntax element values ​​representing encoding / decoding decisions (e.g., encoding / decoding modes) and image blocks. Therefore, references to encoding or decoding images or blocks should generally be understood as encoding or decoding the values ​​of the syntax elements that form the images or blocks.

[0047] HEVC defines various blocks, including codec units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video codec (such as a video encoder 200) partitions a codec tree unit (CTU) into CUs according to a quadtree structure. That is, the video codec partitions the CTU and CU into four equal, non-overlapping squares, and each node of the quadtree has zero or four child nodes. Nodes without child nodes can be called "leaf nodes," and the CU of such leaf nodes can include one or more PUs and / or one or more TUs. The video codec can further partition PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents a partition of a TU. In HEVC, PUs represent inter-frame prediction data, while TUs represent residual data. CUs with intra-frame prediction include intra-frame prediction information, such as intra-frame mode indication.

[0048] As another example, video encoder 200 and video decoder 300 can be configured to operate according to VVC. According to VVC, the video codec (such as video encoder 200) segments an image into multiple CTUs. Video encoder 200 can segment CTUs according to tree structures such as quadtree-binary tree (QTBT) structures or multi-type tree (MTT) structures. The QTBT structure eliminates the concept of multiple segmentation types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels: a first level of segmentation based on quadtree segmentation and a second level of segmentation based on binary tree segmentation. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to CUs.

[0049] In the MTT partitioning structure, blocks can be partitioned using quadtree (QT), binary tree (BT), and one or more triple tree (TT) partitions (also known as ternary tree (TT)). A triple tree partition or ternary tree partition is a partition that splits a block into three sub-blocks. In some examples, a triple tree partition or ternary tree partition divides a block into three sub-blocks without splitting the original block through a center. The partitioning types in MTT (e.g., QT, BT, and TT) can be symmetric or asymmetric.

[0050] When operating according to the AV1 codec, the video encoder 200 and video decoder 300 can be configured to encode and decode video data block by block. In AV1, the largest codec block that can be processed is called a superblock. In AV1, a superblock can be 128×128 luma samples or 64×64 luma samples. However, in subsequent video codec formats (e.g., AV2), superblocks can be defined by different (e.g., larger) luma sample sizes. In some examples, the superblock is the upper level of a block quadtree. The video encoder 200 can further divide the superblock into smaller codec blocks. The video encoder 200 can divide the superblock and other codec blocks into smaller blocks using square or non-square partitioning. Non-square blocks can include N / 2×N, N×N / 2, N / 4×N, and N×N / 4 blocks. The video encoder 200 and video decoder 300 can perform separate prediction and transform processes for each codec block.

[0051] AV1 also defines video data tiles. A tile is a rectangular array of superblocks that can be encoded and decoded independently of other tiles. That is, the video encoder 200 and video decoder 300 can encode and decode the codec blocks within a tile separately without using video data from other tiles. However, the video encoder 200 and video decoder 300 can perform filtering across tile boundaries. Tiles can be uniform or non-uniform in size. Tile-based encoding and decoding can support parallel processing and / or multithreading for encoder and decoder implementations.

[0052] In some examples, the video encoder 200 and the video decoder 300 may use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, the video encoder 200 and the video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luma component and another QTBT / MTT structure for the two chroma components (or two QTBT / MTT structures for the respective chroma components).

[0053] The video encoder 200 and the video decoder 300 can be configured to use quadtree segmentation, QTBT segmentation, MTT segmentation, superblock segmentation or other segmentation structures.

[0054] In some examples, a CTU includes a coding tree block (CTB) for luma samples, two corresponding CTBs for chroma samples of an image with three sample arrays, or a CTB for samples of a monochrome image or an image encoded using three separate color planes and a syntax structure for encoding and decoding the samples. For a given value of N, a CTB can be an N×N sample block such that dividing the components into CTBs is a partition. A component is an array (one luma and two chroma) of three arrays that make up a 4:2:0, 4:2:2, or 4:4:4 color format image, or a single sample of one of those three arrays, or an array that makes up a monochrome image, or a single sample of that array. In some examples, for certain values ​​of M and N, the coding block is an M×N sample block such that dividing the CTB into coding blocks is a partition.

[0055] Blocks (e.g., CTUs or CUs) can be grouped in various ways within an image. As an example, a brick can refer to a rectangular area of ​​a CTU row within a specific slice in an image. A slice can be a rectangular CTU area within a specific slice column or a specific slice row in an image. A slice column is a rectangular CTU area with a height equal to the image height and a width specified by a syntax element (e.g., as in an image parameter set). A slice row is a rectangular CTU area with a height specified by a syntax element (e.g., as in an image parameter set) and a width equal to the image width.

[0056] In some examples, a slice can be divided into multiple bricks, each brick potentially including one or more CTU rows within the slice. A slice that is not divided into multiple bricks can also be referred to as a brick. However, bricks that are a proper subset of a slice may not be referred to as slices. Bricks in an image can also be arranged into strips. A strip can be an integer number of bricks in an image that can be exclusively contained within a single network abstraction layer (NAL) unit. In some examples, a strip may include multiple complete slices or a sequence of consecutive complete bricks from a single slice.

[0057] This disclosure uses "N×N" and "N by N" interchangeably to refer to the sample size of a block (such as a CU or other video block) in the vertical and horizontal dimensions, for example, 16×16 samples or 16 by 16 samples. Generally, a 16×16 CU has 16 samples in the vertical direction (y = 16) and 16 samples in the horizontal direction (x = 16). Similarly, an N×N codec block typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU can be arranged in rows and columns. Furthermore, a CU does not necessarily have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU may include N×M samples, where M is not necessarily equal to N.

[0058] The video encoder 200 encodes video data of the CU (Cubic Array) representing prediction and / or residual information, as well as other information. The prediction information indicates how the CU will be predicted to form a prediction block of the CU. The residual information typically represents the point-by-point difference between the CU samples before encoding and the prediction block.

[0059] To predict the Cues (CUs), the video encoder 200 typically forms prediction blocks of the CUs through inter-frame prediction or intra-frame prediction. Inter-frame prediction generally refers to predicting the CU based on data from previously encoded images, while intra-frame prediction generally refers to predicting the CU based on data from previously encoded images of the same frame. To perform inter-frame prediction, the video encoder 200 can generate prediction blocks using one or more motion vectors. The video encoder 200 can typically perform motion search to identify reference blocks that closely match the CU (e.g., in terms of the difference between the CU and a reference block). The video encoder 200 can compute difference metrics such as sum of absolute difference (SAD), sum of squared difference (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculations to determine whether the reference block closely matches the current CU. In some examples, the video encoder 200 can use unidirectional or bidirectional prediction to predict the current CU.

[0060] Some examples of VVC also provide affine motion compensation modes, which can be viewed as inter-frame prediction modes. In affine motion compensation modes, the video encoder 200 can determine two or more motion vectors representing non-translational motion, such as zoom in or out, rotational motion, perspective motion, or other irregular motion types.

[0061] To perform intra-frame prediction, the video encoder 200 can select an intra-frame prediction mode to generate prediction blocks. Some examples of VVC provide 67 intra-frame prediction modes, including various directional modes as well as planar and DC modes. Generally, the video encoder 200 selects an intra-frame prediction mode that describes the samples of the current block (e.g., a block of a CU) based on the samples of the current block's neighboring blocks. Assuming the video encoder 200 encodes and decodes the CTU and CU in raster scan order (from left to right, from top to bottom), these samples can typically be located above, to the upper left, or to the left of the current block within the same frame.

[0062] The video encoder 200 encodes data representing the prediction mode of the current block. For example, for inter-frame prediction modes, the video encoder 200 can encode data indicating which of the various available inter-frame prediction modes is used, along with the motion information of the corresponding mode. For example, for unidirectional or bidirectional inter-frame prediction, the video encoder 200 can encode motion vectors using advanced motion vector prediction (AMVP) or merging modes. The video encoder 200 can use similar modes to encode motion vectors for affine motion compensation modes.

[0063] AV1 includes two common techniques for encoding and decoding blocks of video data. These two common techniques are intra-frame prediction (e.g., intra-frame prediction or spatial prediction) and inter-frame prediction (e.g., inter-frame prediction or temporal prediction). In the context of AV1, when using intra-frame prediction modes to predict blocks of the current frame of video data, the video encoder 200 and video decoder 300 do not use video data from other frames of the video data. For most intra-frame prediction modes, the video encoder 200 encodes the blocks of the current frame based on the difference between sample values ​​in the current block and predicted values ​​generated based on reference samples in the same frame. The video encoder 200 determines the predicted values ​​generated based on the reference samples based on the intra-frame prediction mode.

[0064] After prediction (such as intra-frame or inter-frame prediction of blocks), the video encoder 200 can compute residual data for the blocks. Residual data (such as residual blocks) represents the sample-by-sample difference between predicted blocks formed using the corresponding prediction mode. The video encoder 200 can apply one or more transforms to the residual blocks to produce transformed data in the transform domain rather than the sample domain. For example, the video encoder 200 can apply a discrete cosine transform (DCT), integer transform, wavelet transform, or conceptually similar transforms to the residual video data. Furthermore, the video encoder 200 can apply secondary transforms after the first transform, such as mode-dependent non-separable secondary transform (MDNSST), signal-dependent transform, Karhunen-Loeve transform (KLT), etc. The video encoder 200 produces transform coefficients after applying one or more transforms.

[0065] As described above, after any transform that produces the transform coefficients, the video encoder 200 can perform quantization on the transform coefficients. Quantization generally refers to a process in which transform coefficients are quantized to minimize the amount of data used to represent them, thereby providing further compression. By performing the quantization process, the video encoder 200 can reduce the bit depth associated with some or all of the transform coefficients. For example, the video encoder 200 can round an n-bit value down to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder 200 can perform a bit-right shift of the value to be quantized.

[0066] After quantization, the video encoder 200 can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix including the quantized transform coefficients. The scan can be designed to place transform coefficients with higher energy (and therefore lower frequency) before the vector and transform coefficients with lower energy (and therefore higher frequency) after the vector. In some examples, the video encoder 200 can utilize a predefined scan order to scan the quantized transform coefficients to generate a serialized vector, and then entropy encode the quantized transform coefficients of the vector. In other examples, the video encoder 200 can perform an adaptive scan. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 200 can entropy encode the one-dimensional vector (e.g., using context-adaptive binary arithmetic coding, CABAC). The video encoder 200 can also entropy encode the values ​​of syntax elements describing metadata associated with the encoded video data for use by the video decoder 300 when decoding the video data.

[0067] To perform CABAC, the video encoder 200 can assign context within a context model to the symbols to be transmitted. The context may involve, for example, whether the neighboring values ​​of a symbol are zero. Probability determination can be based on the context assigned to the symbols.

[0068] The video encoder 200 can further generate (e.g., in picture headers, block headers, strip headers) syntax data (such as block-based syntax data, picture-based syntax data, and sequence-based syntax data) or other syntax data (such as sequence parameter set (SPS), picture parameter set (PPS), or video parameter set (VPS)) to the video decoder 300. The video decoder 300 can similarly decode this syntax data to determine how to decode the corresponding video data.

[0069] In this way, the video encoder 200 can generate a bitstream that includes encoded video data (e.g., syntax elements describing the segmentation of an image into blocks (e.g., CUs) and prediction and / or residual information for the blocks). Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0070] Generally, the video decoder 300 performs the inverse of the process performed by the video encoder 200 to decode the encoded video data of the bitstream. For example, the video decoder 300 can use CABAC to decode the values ​​of syntax elements of the bitstream in a manner substantially similar to but inverse of the CABAC process of the video encoder 200. Syntax elements can define segmentation information used to segment the image into superblocks and codec blocks. Syntax elements can define segmentation information used to segment the image into CTUs, and segment each CTU according to the corresponding segmentation structure (such as a QTBT structure) to define the CU of the CTU. Syntax elements can further define prediction information and residual information for blocks of video data (e.g., CUs).

[0071] The residual information can be represented, for example, by quantized transform coefficients. The video decoder 300 can inversely quantize and inversely transform the quantized transform coefficients of a block to reconstruct the residual block of that block. The video decoder 300 uses the prediction mode (intra-frame or inter-frame prediction) notified by signaling and associated prediction information (e.g., motion information for inter-frame prediction) to form a prediction block of the block. The video decoder 300 can then combine the prediction block and the residual block (on a sample-by-sample basis) to reconstruct the original block. The video decoder 300 can perform additional processing, such as performing a deblocking process to reduce visual artifacts along block boundaries.

[0072] This disclosure generally relates to certain information such as "signaling notification," including syntax elements. The term "signaling notification" generally refers to the communication of values ​​for syntax elements and / or other data used to decode encoded video data. That is, the video encoder 200 can signal the values ​​of syntax elements in the bitstream. Generally, signaling notification refers to the generation of values ​​in the bitstream. As described above, the source device 102 can transmit the bitstream to the destination device 116 substantially in real-time or non-real-time, such as when syntax elements are stored in the storage device 112 for later retrieval by the destination device 116.

[0073] According to the technology disclosed herein, a method includes: determining a current template for a current block of video data; determining a potential reference block for the current block in an image of the video data; determining that the potential reference block and its reference template are completely within the image, the reference template having the same shape as the current template; performing template matching between the reference template and the current template based on the fact that both the potential reference block and the reference template are completely within the image; and encoding / decoding the current block based on the template matching.

[0074] In another example, a device includes: one or more memories configured to store video data; and one or more processors implemented in a circuit and communicatively coupled to the one or more memories, the one or more processors being configured to: determine a current template of a current block of video data; determine a potential reference block for the current block in a picture of the video data; determine that the potential reference block and its reference template are completely within the picture, the reference template having the same shape as the current template; perform template matching between the reference template and the current template based on the fact that both the potential reference block and the reference template are completely within the picture; and encode / decode the current block based on the template matching.

[0075] In another example, a computer-readable storage medium is encoded with instructions that, when executed, cause one or more processors to: determine a current template for a current block of video data; determine a potential reference block for the current block in a picture of the video data; determine that the potential reference block and its reference template are entirely within the picture, the reference template having the same shape as the current template; perform template matching between the reference template and the current template based on the fact that both the potential reference block and the reference template are entirely within the picture; and encode or decode the current block based on the template matching.

[0076] This disclosure relates to various techniques (e.g., template types, fusion, etc.) used in template matching (TM) related tools. The disclosed techniques can be applied to any existing video codec, such as HEVC (High Efficiency Video Codec), VVC (Multi-Functional Video Codec), Essential Video Coding (EVC), Joint Exploration Model (JEM), or high efficiency codec tools (e.g., ECM (Enhanced Compression Model)) in future video codec standards.

[0077] This section discusses the technologies related to HEVC, JEM, and VVC as disclosed. Video codec standards include ITU-T H.261, ISO / IEC MPEG-1 Visible, ITU-T H.262 or ISO / IEC MPEG-2 Visible, ITU-T H.263, ISO / IEC MPEG-4 Visible, and ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), including its Scalable Video Coding (SVC) extension and Multi-view Video Coding (MVC) extension.

[0078] In addition, new video codec standards (i.e., High Efficiency Video Coding (HEVC) or ITU-T H.265, including its Range Extension, Multi-View Extension (MV-HEVC), and Scalable Extension (SHVC)) have been developed by the Joint Collaboration Team on Video Coding (JCT-VC), the ITU-T Video Coding Experts Group (VCEG), and the Joint Collaboration Team on 3D Video Coding Extension Development (JCT-3V) of the ISO / IEC Motion Picture Experts Group (MPEG).

[0079] The latest HEVC draft specification (hereinafter referred to as HEVC WD) is available from phenix.int-evry.fr / jct / doc_end_user / documents / 14_Vienna / wg11 / JCTVC-N1003-v1.

[0080] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) are investigating the potential need for standardization of future video codec technologies with compression capabilities significantly exceeding the current HEVC standard (including its current extensions and recent extensions for screen content coding and high dynamic range coding). These groups are working collaboratively on this exploration, known as the Joint Video Exploration Team (JVET), to evaluate compression technology designs proposed by their experts in the field. The latest version of the reference software (i.e., VVC Test Model 10 (VTM 10)) can be downloaded from vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM. The draft specification for Multi-Functional Video Codec (VVC) can be referred to as JVET-T2001. The algorithmic description of Multi-Functional Video Codec and Test Model 10 (VTM 10.0) can be referred to as JVET-T2002. Alternatively, you can use Alliance for OpenMedia (AOM) video codec formats, such as AV1 and AV2.

[0081] Intra-frame template matching prediction (intra-frame TMP) is a special intra-frame prediction mode that copies a prediction block (e.g., the best prediction block) from the reconstructed portion of the current frame, where the left or top template of the L-shape of the reconstructed portion matches the current template. For a predefined search range, the video encoder (e.g., video encoder 200) searches for the template most similar to the current template in the reconstructed portion of the current frame and uses the corresponding block as the prediction block. The video encoder then signals the use of this mode and performs the same prediction operation at the video decoder (e.g., video decoder 300).

[0082] Figure 2 This is a conceptual diagram illustrating the template matching search region within an example frame. A prediction signal is generated by matching the causally adjacent block to the left or above the L-shape of the current block 260 with another block (e.g., reference block 262) in a predefined search region that includes search regions R1, R2, R3, or R4.

[0083] In some examples, the search area can be different Figure 2 As shown in the example. For example, R1 can include the upper CTU, R2 can include the CTU to the left and below the current block, R3 can include the left CTU above the current block and the left portion of the current CTU, and R4 can include the right portion of the current CTU.

[0084] The L-shaped template includes a left template section, a top template section, and a top-left template section, as discussed below. Figure 9 As shown. The video encoder 200 or video decoder 300 can use the sum of absolute differences (SAD) as a cost function to determine the matching template. For example, the video encoder 200 or video decoder 300 can determine the template with the lowest SAD as the matching template based on the template of the current block.

[0085] For example, within each region, the video decoder 300 can search for the template with the minimum SAD relative to the current template and use the corresponding block of the template as the prediction block (e.g., reference block). For example, the video decoder 300 can select a reference block from multiple potential reference blocks as the potential reference block with the template that has the lowest template matching cost.

[0086] The size of the region (SearchRange_w, SearchRange_h) can be set to be proportional to the block size (BlkW, BlkH), thus performing a fixed number of SAD comparisons per pixel. That is:

[0087] SearchRange_w = a * BlkW

[0088] SearchRange_h = a * BlkH

[0089] Here, "a" is a constant that controls the gain / complexity tradeoff. In practice, "a" is usually equal to 5.

[0090] In one example, the minimum value of the region size (SearchRange_w, SearchRange_h) can be set to:

[0091] SearchRange_w = min(minSize, a * BlkW)

[0092] SearchRange_h = min(minSize, a * BlkH)

[0093] Here, "minSize" is the minimum size of the search region. In practice, "minSize" is usually equal to 64.

[0094] For CUs with a width and height of 64 or less, the intra-frame template matching tool can be enabled. The maximum CU size used for intra-frame template matching can be configurable.

[0095] When decoder-side intra-mode derivation (DIMD) is not used for the current CU, the intra-TMP mode can be notified at the CU level via a dedicated flag. For example, video encoder 200 can signal the intra-TMP mode to video decoder 300 via a dedicated flag.

[0096] Search area ( Figure 2 The template matching search (R1 to R4) can be subsampled by integer factors. This reduces the template matching search by, for example, a factor of 1 / 4. After finding the best template match, a multi-stage refinement process can be performed, in which another template matching search is performed around the best match with a reduced search range, the reduced search range being constrained to the region in the set R1, R2, R3, R4 in which the best match was found in the first stage. The refined search range can be defined as min(w,h) / 2, where w and h are the width and height of the current CU. For example, a video encoder 200 or a video decoder 300 can perform a multi-stage refinement process.

[0097] Figure 3 This is a conceptual diagram illustrating an example of template matching performed over a search region around an initial motion vector (MV). Inter-frame template matching (InterTM) is a decoder-side MV derivation technique used to refine the motion information of the current CU 390 by finding the closest match between the current template 392 in the current image 396 (e.g., the top and / or left adjacent block of the current CU) and a block in the reference image 398 that has a reference template 394 of the same size as the current template 392. Figure 3As shown, within the [–8, +8] pixel search range, a better MV can be searched around the initial motion of the current CU 390. The template matching technique in Chen et al.'s "Description of SDR, HDR and 360° video coding technology proposal by Qualcomm and Technicolor – low and high complexity versions" (Joint Video Exploration Group (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 10th meeting: San Diego, USA, April 10-20, 2018, document JVET-J0021) can be used with the following modifications: the search step size can be determined based on the adaptive motion vector resolution (AMVR) mode, and InterTM can be cascaded with the bilateral matching process in the merging mode.

[0098] Figure 4 This is a tabular diagram illustrating AMVR's search styles and merging modes under AMVR. The video encoder 200 or video decoder 300 can use, for example... Figure 4 The described search patterns are used to perform AMVR. Figure 4 The cell with " "This indicates the specific search style supported by the mode. In AMVP mode, motion vector predictor (MVP) candidates can be determined based on template matching error to select the MVP candidate that achieves the minimum difference between the current block template (also referred to as the current template in this paper) and the reference block template (also referred to as the reference template in this paper). Then, InterTM is performed only on that specific MVP candidate for MV refinement. InterTM refines the MVP candidate by using an iterative diamond search over a search range of [–8, +8] pixels, starting with full-pixel motion vector difference (MVD) precision (or 4 pixels for 4-pixel AMVR mode). The AMVP candidate can be further refined by using a cross-shaped search with full-pixel MVD precision (or 4 pixels for 4-pixel AMVR mode), followed by sequential half-pixel and quarter-pixel searches depending on the AMVR mode, such as..." Figure 4The description is shown in Table 1. This search process ensures that the MVP candidate maintains the same MV accuracy as indicated by the AMVR mode after the TM process. During the search process, the search terminates if the difference between the previous minimum cost and the current minimum cost in the iteration is less than a threshold equal to the area of ​​the block.

[0099] In merge mode, similar search techniques can be applied to merge candidates indicated by the merge index. As shown in Table 1, InterTM can proceed down to 1 / 8 pixel MVD precision, or skip those beyond half-pixel MVD precision, depending on whether an alternative interpolation filter (used when AMVR is in half-pixel mode) is used based on the motion information of the merge. Furthermore, when TM mode is enabled, template matching can function as a separate process between block-based bilateral matching (BM) and sub-block-based BM methods, or as an additional MV refinement process, depending on whether BM can be enabled based on its enabling conditions.

[0100] Template matching (TM) can be used to adaptively reorder merging candidates. Reordering techniques can be applied to regular merging mode, TM merging mode, and affine merging mode (excluding subblock-based temporal motion vector prediction (SbTMVP) candidates). For TM merging mode, merging candidates can be reordered before the refinement process. These reordering techniques can be performed by the video encoder 200 or the video decoder 300.

[0101] The video encoder 200 or video decoder 300 can construct an initial merging candidate list based on a given inspection order, such as spatial motion vector predictor (TMVP) candidates, temporal TMVP candidates, non-nearest neighbor candidates, HMVP candidates, paired candidates, virtual merging candidates, etc. The candidates in the initial list can then be divided into several subgroups. For Template Matching (TM) merging mode and / or Decoder-Side Motion Vector Refinement (DMVR) mode, each merging candidate in the initial list can first be refined using TM / multi-pass DMVR. The merging candidates in each subgroup can be reordered to generate a reordered merging candidate list, and the reordering can be based on the template matching-based cost value. The index of the selected merging candidate in the reordered merging candidate list can be signaled from the video encoder 200 to the video decoder 300. For simplicity, merging candidates in the last subgroup instead of the first subgroup may not be reordered. During the construction of the candidate list for merging motion vectors, zero candidates from the adaptive reordering of merge candidate (ARMC) reordering process can be excluded. For regular merging and TM merging modes, the subgroup size can be set to 5. For affine merging mode, the subgroup size can be set to 3.

[0102] The template matching cost of merge candidates during the reordering process can be measured by the SAD between the samples of the current block's template and their corresponding reference samples. The template comprises a set of reconstructed samples adjacent to the current block. Reference samples of the template are located using the motion information of the merge candidates. When merge candidates utilize bidirectional prediction, the reference samples of the merge candidate's template are also generated through bidirectional prediction, such as... Figure 5 As shown.

[0103] Figure 5 This is a conceptual diagram showing reference samples of the template in the example template and reference image. For example... Figure 5 As shown in the example, a video codec (e.g., video encoder 200 or video decoder 300) is encoding / decoding (e.g., encoding or decoding) the current block 500 of the current image 502. The current block 500 is associated with a merge candidate list that includes one or more bidirectional prediction merge candidates. Each bidirectional prediction merge candidate includes a list 0 motion vector and a list 1 motion vector. The list 0 motion vector indicates the position within a reference image in a first list of reference images (e.g., list 0). The list 1 motion vector indicates the position within a reference image in a second list of reference images (e.g., list 1). Figure 5In the example, the motion vector of list 0, selected as a merge candidate, indicates the location in reference image 504. The motion vector of list 1, selected as a merge candidate, indicates the location in reference image 506. The video encoder 200 or video decoder 300 can then use the template 508 of the current block 500 to search for regions near the indicated locations in reference images 504 and 506 of template 510 (i.e., reference samples of templates in reference lists 0 and 1). In this way, the video encoder 200 or video decoder 300 is able to identify reference blocks 512 and 514 of the current block 500 that are potentially more similar to the reference blocks at the locations indicated by the motion vectors of list 0 and list 1.

[0104] When using multi-pass DMVR to derive a refined motion for the initial merge candidate list, only the first pass of the multi-pass DMVR (e.g., PU level) can be applied during reordering. When using template matching to derive the refined motion, the template size can be set to 1. In some examples, when the block is flat and its width is greater than twice its height, or when the block is narrow and its height is greater than twice its width, only the top or left template can be used during TM motion refinement. TM can be extended to perform 1 / 16 pixel MVD precision. The first four merge candidates can be reordered with refined motion in TM merge mode.

[0105] Figure 6 This is a conceptual diagram illustrating an example template and reference sample points of a block when using the motion information of its sub-blocks. For sub-block-based merging candidates with a sub-block size equal to Wsub × Hsub, the upper template includes several sub-templates 602 of size Wsub × 1 (which can together form the upper reference template), and the left template includes several sub-templates 604 of size 1 × Hsub (which can together form the left reference template). Figure 6 As shown, the motion information of the sub-blocks (sub-block AG) in the first row and first column of the current block 606 is used to derive the reference sample points of each sub-template.

[0106] During the reordering process, if the cost difference between a candidate and its predecessor is worse than (e.g., less than) the Lambda (λ) value, such as |D1-D2| < λ, then the video encoder 200 or the video decoder 300 may treat the candidate as redundant, where D1 and D2 are the costs obtained during the first ARMC sorting, and λ is the Lagrange parameter used in the rate distortion (RD) criterion at the video encoder 200.

[0107] For example, video encoder 200 or video decoder 300 can execute an algorithm for the reordering process, which can work as follows: Determine the minimum cost difference between a candidate and its predecessor among all candidates in the list. If the minimum cost difference is greater than or equal to (e.g., greater than or equal to) λ, the list is considered sufficiently diverse, and reordering stops. If the minimum cost difference is less than (e.g., less than) λ, the candidate is considered redundant, and the candidate is moved further down the list. This further down the list is the first position where the candidate is sufficiently diverse compared to its predecessor. The algorithm can stop after a finite number of iterations (if the minimum cost difference is not less than λ).

[0108] This algorithm can be applied to regular merging mode, TM merging mode, BM merging mode, and affine merging mode. Similar algorithms can be applied to merging modes that utilize motion vector difference (MMVD) and to sign MVD prediction methods that also use ARMC for reordering.

[0109] The λ value can be set to be equal to the rate-distortion criterion used to select the best merging candidate for the low-latency configuration at video encoder 200, and equal to the λ value corresponding to another quantization parameter (QP) used for the random access configuration. The set of λ values ​​corresponding to the QP offset for each signaling notification can be provided in the SPS, or in the strip header of the QP offset that does not exist in the SPS.

[0110] The ARMC design also applies to the AMVP pattern, where AMVP candidates are reordered based on TM cost. For template matching (TM-AMVP) of advanced motion vector prediction patterns, an initial AMVP candidate list can be constructed, which is then refined based on TM to construct a more detailed AMVP candidate list. Furthermore, MVP candidates with TM costs exceeding a threshold equal to five times the cost of the first MVP candidate can be skipped.

[0111] Note that when wrap-around motion compensation is enabled, the MV candidate can be clipped using the wrap-around offset taken into account.

[0112] Figure 7 It is a tabular diagram showing the templates of the first and second geometric partitions, where A indicates the use of the upper sample points, L indicates the use of the left sample points, and L+A indicates the use of both the left and upper sample points.

[0113] The video encoder 200 or video decoder 300 can apply template matching to geometric partitioning mode (GPM). When GPM mode is enabled for a CU, the video encoder 200 can signal a CU-level flag to indicate whether template matching (TM) is applied to two geometric partitions. TM can be used to refine the motion information of each geometric partition. When TM is selected, a template can be constructed using adjacent samples from the left, top, or left and top, based on the partitioning angle, such as... Figure 7 As shown in Table 2. Then, the motion can be refined by minimizing the difference between the current template and the template in the reference image using the same search style of the merging mode while disabling the half-pixel interpolation filter.

[0114] The GPM candidate list can be constructed as follows: Interleaved lists of 0MV candidates and 1MV candidates are derived directly from the rule-merging candidate list, where list 0MV candidates have a higher priority than list 1MV candidates. A pruning method with an adaptive threshold based on the current CU size is applied to remove redundant MV candidates. Further, interleaved lists of 1MV candidates and 0MV candidates are derived directly from the rule-merging candidate list, where list 1MV candidates have a higher priority than list 0MV candidates. The same pruning method with an adaptive threshold is also applied to remove redundant MV candidates. Zero-MV candidates are filled until the GPM candidate list is full.

[0115] The Geometric Segmentation Mode - Merging Mode Using Motion Vector Difference (GPM-MMVD) and Geometric Segmentation Mode - Template Matching (GPM-TM) can be exclusively enabled for only one GPM CU. This can be achieved by first signaling the GPM-MMVD syntax. For example, the video encoder 200 can signal the GPM-MMVD syntax. When both GPM-MMVD control flags are equal to "false" (e.g., GPM-MMVD is disabled for two GPM partitions), the GPM-TM flag can be signaled to indicate whether template matching is applied to both GPM partitions. For example, the video encoder 200 can signal the GPM-TM flag. Otherwise (at least one GPM-MMVD flag is equal to "true"), the value of the GPM-TM flag can be inferred as "false". For example, if no GPM-TM flag is signaled, the video decoder 300 can infer that the GPM-TM flag is "false".

[0116] The video encoder 200 or video decoder 300 can use template matching in Intra Block Copy (IBC) for both Intra Block Copy (IBC) merging mode and IBC AMVP mode. The IBC-TM merging list can be modified compared to the list used in regular IBC merging mode, allowing candidates to be selected based on a pruning technique utilizing motion distance between candidates, as in regular TM merging mode. Ending zero motion fulfillment can be replaced by motion vectors to the left (-W, 0), up (0, -H), and top left (-W, -H), where W is the width of the current CU and H is the height of the current CU.

[0117] In Intra-Block Copy-Template Matching (IBC-TM) merging mode, template matching techniques can be used to refine selected candidates before rate-distortion optimization (RDO) or the decoding process. For example, video encoder 200 or video decoder 300 can refine selected candidates. IBC-TM merging mode can compete with regular IBC merging mode and can signal the TM merging flag.

[0118] In the IBC-TM AMVP mode, up to three candidates can be selected from the IBC-TM merging list. Each of these three selected candidates can be refined using template matching techniques and ranked according to their resulting template matching costs. Then, as usual, only the first two candidates are considered during motion estimation.

[0119] Figures 8A-8D This is a conceptual diagram illustrating an example IBC reference region that depends on the current location of the codec unit (CU). Figures 8A-8D In this context, the shaded block is "causal" relative to the current block 800A, 800B, 800C, or 800D (collectively referred to as "current block 800"). In other words, the shaded block is encoded or decoded before the current block 800. Blocks marked with an "X" are reference regions that can be used relative to the current block 800 for IBC TM. Template matching refinement for IBC-TM merging and AMVP modes is fairly straightforward because the IBC motion vector is constrained to be (i) integers and (ii) within the reference region, such as... Figures 8A-8D As shown. Therefore, in IBC-TM merge mode, all thinning is performed with integer precision, while in IBC-TM AMVP mode, they are performed with integer or 4-pixel precision depending on the AMVR value. This thinning only accesses samples that are not interpolated. In both cases, the thinning motion vector and the template used in each thinning step can (and in some examples must) adhere to the constraints of the reference region.

[0120] To improve the encoding and decoding efficiency of template matching and / or the quality of encoded video data, template matching according to the techniques disclosed herein may include using different template types, storing more candidates, and / or applying fusion to combine these different candidates found through different template matching techniques, rather than using only one template matching style and process.

[0121] Current search algorithms may impose excessive restrictions on search regions near image boundaries and reconstructed region boundaries, potentially limiting potential reference blocks and thus limiting potential music video (MV) candidates for some video content. The technique disclosed herein expands the potential selection by addressing the problem of search regions near image boundaries and reconstructed region boundaries.

[0122] For example, current search algorithms might test the top-left portion, then the left portion, and then the top portion to determine the availability of each such portion. If the current search algorithm finds the top-left portion unavailable, for example because it might cross an image boundary or a reconstructed region boundary, the algorithm might terminate the search, thus excluding potential reference blocks, or it might continue examining the top portion without examining the left portion. However, there are cases where the top-left portion might be unavailable while the left portion might be available, or vice versa. The techniques disclosed herein include treating blocks with such templates as potential reference blocks. While such techniques may be more computationally expensive because more potential reference blocks may be examined, these techniques can lead to better reference blocks for predicting the movement of the current block, thereby ultimately improving encoding / decoding quality and / or encoding / decoding efficiency.

[0123] For simplicity, the TM discussed below may refer to intra-frame template matching, inter-frame template matching, merged candidate adaptive reordering using template matching (ARMC-TM), and / or IBC template matching. The disclosed techniques may be used alone or in any combination.

[0124] Figure 9 This is a conceptual diagram illustrating an example template portion. For example, the current block 900 may have a template including a top portion 904, a top-left portion 902, and / or a left portion 906. The top portion 904 may be a template or a portion thereof directly above the current block 900. The left portion 906 may be a template or a portion thereof directly to the left of the current block 900. The current block 900 may have a height H and a width W. The top portion 904 may have a height T, and the left portion 906 may have a width T. The top-left portion may be directly to the left of the top portion 904 and directly above the left portion 906, and has a height T and a width T. Thus, the top-left portion 902 can be said to be above and to the left of the current block 900.

[0125] In some examples, template availability is determined based on the availability of adjacent blocks. For example, video encoder 200 or video decoder 300 may determine which template portions (top portion, left portion, and / or top-left portion) are available based on the availability of adjacent blocks.

[0126] For example, the left template portion (e.g., left portion 906) is used when all neighboring CUs to the left of the current block are available. For instance, video encoder 200 or video decoder 300 can use left portion 906 when all neighboring CUs to the left of the current block 900 are available. In one example, at least one block adjacent to the top image boundary but not adjacent to the left image boundary has a left template portion. In this example, video encoder 200 or video decoder 300 can use left portion 906. In the example existing design, the left template portion of this block is considered unavailable.

[0127] For example, the upper template portion (e.g., upper portion 904) is used when all neighboring CUs above the current block are available. For example, the video encoder 200 or video decoder 300 can use upper portion 904 when all neighboring CUs above the current block 900 are available. In one example, at least one block adjacent to the left image boundary but not adjacent to the upper image boundary has an upper template portion. In this example, the video encoder 200 or video decoder 300 can use upper portion 904.

[0128] For example, when the adjacent CU to the upper left of the current block is available, the upper left template portion (e.g., upper left portion 902) is used. For example, when the adjacent CU to the upper left of the current block 900 is available, the video encoder 200 or the video decoder 300 can use the upper left portion 902.

[0129] Similarly, when the upper right and / or lower left portions are available, the template can be extended to include these portions. For example, video encoder 200 or video decoder 300 can extend the template to include the upper right and / or lower left portions. The upper right portion will include the samples to the right of the upper portion 904. The lower left portion will include the samples of the lower left portion 906. For example, if the lower left CU of the current block 900 is available, the left portion 906 can be extended downwards (e.g., towards the bottom of the image). This left portion can be referred to as the extended left portion. In one example, at least one block not adjacent to the image boundary has a template that includes the extended left portion.

[0130] For example, if the CU to the upper right of the current block 900 is available, the upper portion 904 can extend to the right. This upper portion can be referred to as an extended upper portion. In one example, at least one block that is not adjacent to the image boundary has a template that includes an extended upper portion.

[0131] TM cannot be applied when no template portion is available.

[0132] The reference block search region of the TM is now discussed. To address image boundary handling, in the example, during template matching search, the reference block can (e.g., should) be located within a predefined search area within the image. For example, using a reference block outside or partially outside the predefined search area within the image can be penalized or prohibited. Furthermore, constraints can be imposed: a reference block, or a reference block including its template, can (e.g., should or must) be entirely within the image. For example, video encoder 200 or video decoder 300 can perform template matching with reference blocks located within a predefined search area within the image. In some examples, video encoder 200 or video decoder 300 can perform template matching with the potential reference block if both the potential reference block and its template are entirely within the image (e.g., do not cross image boundaries).

[0133] Figures 10A-10C This is a conceptual diagram illustrating an example of image boundary processing depending on the template type. Figure 10A In the current block 1000, there is an L-shaped template 1010 comprising a top portion, a left portion, and a top-left portion. Similarly, reference blocks 1002 and 1004 have L-shaped templates 1012 and 1014, respectively. Figure 10B In this context, current block 1020 has an upper shape template 1030 including the upper portion. Similarly, reference blocks 1022 and 1024 have upper shape templates 1032 and 1034, respectively. Figure 10C In the current block 1040, there is a left-shaped template 1050 that includes the left side portion. Similarly, reference blocks 1042 and 1044 have left-shaped templates 1052 and 1054, respectively.

[0134] In one example, when the template shape includes a left template portion, a top left template portion, and a top template portion (e.g., as shown below) Figure 10A When the top-left sample point of the reference block is (as shown) or at least two of these parts, the top-left sample point of the reference block may (e.g., should) have an offset of at least T in both the vertical and horizontal directions from the image boundary (which in some examples may be referred to as the image boundary line), where T represents the template size. For example, in Figure 10A In the image, reference block 1002 has a vertical offset of T from the top image boundary 1016 and a horizontal offset greater than T from the left image boundary 1018. Reference block 1004 has a vertical offset greater than T from the top image boundary 1016 and a horizontal offset of T from the left image boundary 1018. Both reference blocks 1002 and 1004 can be used as potential reference blocks for the current block 1000.

[0135] In one example, when the template shape only includes the upper template portion (e.g.) Figure 10B As shown), the top-left sample point of the reference block can (e.g., should) have an offset of at least T from the image boundary (e.g., top image boundary 1036) only in the vertical direction. In one example, at least one searched reference block is adjacent to the left image boundary (e.g., left image boundary 1038) but not adjacent to the top image boundary (e.g., top image boundary 1036), whereas in the example existing design, this reference block is excluded from the search area. For example, in Figure 10B In the image, reference blocks 1022 and 1024 both have a vertical offset of at least T from the top image boundary 1036. Thus, both reference blocks 1022 and 1024 can be used as potential reference blocks for the current block 1020.

[0136] In one example, when the template shape only includes the left template portion (e.g.) Figure 10C When (as shown), the top-left sample point of the reference block can (e.g., should) have an offset of at least T in the horizontal direction from the image boundary (e.g., left image boundary 1058). In one example, at least one searched reference block (e.g., reference block 1042) is adjacent to the top image boundary (e.g., top image boundary 1056), but not adjacent to the left image boundary (e.g., left image boundary 1058), whereas in the example existing design, this reference block is excluded from the search area. For example, in Figure 10C In the image, reference blocks 1042 and 1044 both have a horizontal offset of at least T from the left image boundary 1058. Thus, reference blocks 1042 and 1044 can both be used as potential reference blocks for the current block 1040.

[0137] Figure 11 This is a conceptual diagram illustrating the boundary treatment of an alternative example image for the L-shaped template example. Figure 11 In the example, the current block 1100 has an L-shaped template 1110 that includes an upper portion, a left portion, and an upper left portion. The reference block 1102 has an L-shaped template 1112. The L-shaped template 1112 may include unavailable samples of the left portion 1104 on the other side of the left image boundary (e.g., left image boundary 1118).

[0138] In another example, only reference blocks without their templates can (e.g., should) be located within the image. In other words, if the template (or a portion thereof) of the reference block is located outside the image, the video encoder 200 or video decoder 300 can still use such a reference block. In this case, the video encoder 200 or video decoder 300 can deduce the unusable template portion by filling the left portion 1104 with available samples, such as... Figure 11As shown. In one example, a reference block with an L-shaped template can be adjacent to the left or top image boundary, while the existing design in the example excludes this block from the search area.

[0139] We will now discuss the handling of reconstructed region boundaries. In the example existing design, reference blocks with samples below or to the right of the current block's lower right (also referred to as the bottom right) sample in the current CTU are excluded from the search region.

[0140] The video encoder 200 or video decoder 300 can determine the search region. For example, the search region can be determined based on the block decoding order; if adjacent regions have already been decoded, such regions can be used for searching. In one example, all blocks that are wholly or partially located within the current CTU and completely belong to the reconstructed region are included in the search region. In one example, in the disclosed technique, reconstructed regions relative to the lower left (also referred to as the lower left) of the current block and reconstructed regions relative to the upper right of the current block can be included in the search region (e.g., if these regions have already been reconstructed based on the encoding / decoding order), whereas in the current example intra-frame template matching design, such regions are excluded from the search.

[0141] Now let's discuss the refinement of the search. In the existing example design, the TM search is a multi-step process: in the first stage, a search is performed in a composite region including R1, R2, R3, and R4, such as... Figure 2 As shown. The vertical and horizontal offsets of the reference block are constrained to predefined thresholds by absolute values. A subsampling search is applied in the first step by selecting a search step size greater than 1, and the optimal block vector (BV) that minimizes the search cost is identified. In subsequent stages, the search region is constrained to an optimal region pointed to by the optimal BV obtained from the subsampling search in sets R1, R2, R3, and R4. In the next step, a refinement search with a step size of 1 sample point is performed around the optimal BV in a predefined neighborhood. However, since the neighborhood is constrained to be located within the optimal region, the position of the neighborhood in other regions is not checked when it crosses a region boundary (e.g., between R2 and R3), which is suboptimal, especially for samples where the optimal BV points near the region boundary.

[0142] In one example, the refinement stage search region is not constrained to be performed only within one of regions R1, R2, R3, and R4; instead, it can be performed using a combination of regions R1, R2, R3, and R4. For example, the video encoder 200 or the video decoder 300 can perform template matching refinement searches across multiple regions. In one example, the constraint that the neighborhood should only be within one of these regions is removed.

[0143] In one example, the final determined best match has the lowest template cost within a predefined region of rectangle [x-dx1, x+dx2]*[y-dy1, y+dy2], where samples (x-dx1,y) and (x+dx2,y) can belong to different search regions, whereas in the example's existing design, these samples must come from a single search range. In one example, video encoder 200 or video decoder 300 can determine a list of matches sorted by template matching cost based on samples from multiple search regions.

[0144] The unavailable reference block samples and padding of TM will now be discussed. The video encoder 200 or video decoder 300 can utilize the padding techniques for TM described herein.

[0145] Figure 12 This is a conceptual diagram illustrating an example of filling unavailable samples by copying values ​​from available samples. In one example, reference block samples that are unavailable due to the position of the current block 1200 (e.g., in one example, the reference block overlaps with the current block) are recursively derived by copying samples from the reference block at positions corresponding to their positions in the current block, as follows: Figure 12 As shown, each sample is represented by a number, and samples with the same number in the hashed, patterned, unreconstructed region of the current block are copied from reconstructed samples (represented using a diagonal pattern) that have the same sample number. For example, a video encoder 200 or a video decoder 300 can perform this padding.

[0146] With the block vector (BVx, BVy) and the current position (x, y) in the reference block, unusable values ​​can be copied from the position (x-BVx, y-BVy). The same method is used to derive unusable values ​​when the reference position is in the current block 1200: (x-2*BVx, y-2*BVy), etc.

[0147] In another example, other padding techniques can be applied, such as copying the nearest available sample point, or any other padding technique can be used. In yet another example, any image boundary padding technique can be used to deduce the unavailable sample points of the reference block located outside the image boundary.

[0148] Figure 13 This is a flowchart illustrating an example template matching technique according to one or more aspects of this disclosure. A video encoder 200 or a video decoder 300 can determine the current template (1300) of the current block within a picture of video data. For example, the video encoder 200 or the video decoder 300 can generate the current template of the current block based on the availability of neighboring blocks.

[0149] The video encoder 200 or video decoder 300 can identify potential reference blocks (1302) for the current block in the image. For example, the video encoder 200 or video decoder 300 can determine which blocks are available within the search range to identify one or more potential reference blocks.

[0150] Video encoder 200 or video decoder 300 can determine that the potential reference block and its reference template are completely within the image, and the reference template has the same shape as the current template (1304). For example, video encoder 200 or video decoder 300 can determine the reference template of a reference block whose shape matches the current template. Video encoder 200 or video decoder 300 can determine that the potential reference block and its reference template are completely within the image. If all samples of a certain type of the potential reference block (e.g., all luma samples) are within the image, then the potential reference block is likely completely within the image. If all samples of a certain type of the reference template (e.g., all luma samples) are within the image, then the reference template is likely completely within the image. For example, video encoder 200 or video decoder 300 can determine that neither the potential reference block nor the reference template crosses any image boundary.

[0151] Since both the potential reference block and the reference template are entirely within the image, the video encoder 200 or the video decoder 300 can perform template matching between the reference template and the current template (1306). For example, the video encoder 200 or the video decoder 300 can perform template matching by calculating the template matching cost associated with the reference template.

[0152] The video encoder 200 or video decoder 300 can encode or decode the current block based on template matching (1308). For example, when encoding or decoding the current block, the video encoder 200 or video decoder 300 can use a potential reference block with the lowest template matching cost as a reference block for the current block. In another example, the video encoder 200 or video decoder 300 can construct a list of potential reference blocks ordered using template matching costs and can signal the index of the reference block in the list for encoding or decoding the current block. In yet another example, when encoding or decoding the current block, the video encoder 200 or video decoder 300 can use multiple reference blocks from the list.

[0153] In some examples, the potential reference block is a first potential reference block, and the reference template is a first reference template. In some examples, the video encoder 200 or the video decoder 300 may determine that at least one of the second potential reference block or the second reference template of the second potential reference block is not entirely within the picture. In some examples, based on the fact that at least one of the second potential reference block or the second reference template is not entirely within the picture, the video encoder 200 or the video decoder 300 may not perform template matching between the second reference template and the current template of the current block of video data. In some examples, the video encoder 200 or the video decoder 300 may encode or decode the current block using a reference block different from the second potential reference block. For example, based on the fact that one or both of the second potential reference block or the second reference template crosses one or more picture boundaries, the video encoder 200 or the video decoder 300 may exclude the second potential reference block as a reference block for the current block.

[0154] In some examples, the reference template includes at least two of a left portion, a top-left portion, or a top portion. In some examples, the top-left sample of the potential reference block is offset vertically from the top image boundary by the width of at least one of the top-left or top portions, and horizontally offset from the left image boundary by the width of at least one of the left or top-left portions.

[0155] In some examples, the reference template only includes the left portion. In some examples, the top-left sample of the potential reference block is offset horizontally from the left image boundary by the width of the left portion. In some examples, the potential reference block is adjacent to the top image boundary but not to the left image boundary.

[0156] In some examples, the reference template only includes the upper portion. In some examples, the top-left sample of the potential reference block is offset vertically from the top image boundary by the width of the upper portion. In some examples, the potential reference block is adjacent to the left image boundary but not to the top image boundary.

[0157] In some examples, a potential reference block is one of multiple potential reference blocks. In some examples, the video encoder 200 or the video decoder 300 can determine a search region for the image for multiple potential reference blocks to include all blocks, wholly or partially, located within the current codec tree unit (CTU) of the video data for complete reconstruction. In some examples, the video encoder 200 or the video decoder 300 can determine that a potential reference block is within the search region.

[0158] In some examples, the search area includes one of multiple search areas. In some examples, the video encoder 200 or video decoder 300 can perform a refined template matching search on the results of the template matching search. In some examples, the refined template matching search involves searching within multiple search areas.

[0159] In some examples, the current template includes at least one portion. In some examples, as part of determining the current template, the video encoder 200 or the video decoder 300 may determine at least one portion based on the availability of neighboring blocks of the current block.

[0160] In some examples, at least one portion includes the left side. In some examples, as part of determining at least one portion, the video encoder 200 or the video decoder 300 may determine that all neighboring blocks to the left of the current block are available. In some examples, the current block includes a block adjacent to the top image boundary but not adjacent to the left image boundary.

[0161] In some examples, at least one portion includes the upper portion. In some examples, as part of determining at least one portion, the video encoder 200 or the video decoder 300 may determine that all neighboring blocks above the current block are available. In some examples, the current block includes a block that is not adjacent to the top image boundary but is adjacent to the left image boundary.

[0162] In some examples, at least one portion includes the upper left portion. In some examples, as part of determining at least one portion, the video encoder 200 or the video decoder 300 may determine that adjacent blocks to the left and above the current block are available.

[0163] In some examples, the video encoder 200 or the video decoder 300 can determine that adjacent blocks to the left and below the current block are available. In some examples, based on the availability of adjacent blocks to the left and below the current block, the video encoder 200 or the video decoder 300 can extend the left portion downwards such that the bottom of the left portion is lower than the bottom of the current block in the vertical direction.

[0164] In some examples, the video encoder 200 or video decoder 300 can determine that adjacent blocks above and to the right of the current block are available. In some examples, based on the availability of adjacent blocks above and to the right of the current block, the video encoder 200 or video decoder 300 can extend the upper portion to the right, such that the right edge of the upper portion is further to the right in the horizontal direction than the right edge of the current block.

[0165] In some examples, encoding / decoding includes decoding. In other examples, encoding / decoding includes encoding.

[0166] Figure 14This is a block diagram illustrating an example video encoder 200 that can perform the techniques of this disclosure. Figure 14 This disclosure is provided for illustrative purposes and should not be construed as a limitation on the techniques extensively illustrated and described herein. For illustrative purposes, this disclosure describes a video encoder 200 based on VVC and HEVC techniques. However, the techniques of this disclosure can be implemented by video encoding devices configured for other video codec standards and video codec formats, such as AV1 and its successors.

[0167] exist Figure 14 In the example, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy encoding unit 220. Any one or all of the video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, DPB 218, and entropy encoding unit 220 can be implemented in one or more processors or in processing circuitry. For example, the units of the video encoder 200 can be implemented as one or more circuit or logic elements, as part of hardware circuitry or as part of a processor, ASIC, or FPGA. Furthermore, the video encoder 200 may include additional or alternative processors or processing circuitry to perform these and other functions.

[0168] The video data storage device 230 can store video data to be encoded by the components of the video encoder 200. The video encoder 200 can obtain data from, for example, a video source 104 (…). Figure 1The video encoder 200 receives video data stored in video data memory 230. DPB 218 can act as a reference picture memory storing reference video data for use by the video encoder 200 when predicting subsequent video data. Video data memory 230 and DPB 218 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 230 and DPB 218 can be provided by the same memory device or separate memory devices. In various examples, video data memory 230 can be on-chip with other components of the video encoder 200, as shown, or off-chip relative to those components.

[0169] In this disclosure, references to video data memory 230 should not be construed as being limited to memory inside video encoder 200 (unless specifically described otherwise) or memory outside video encoder 200 (unless specifically described otherwise). Rather, references to video data memory 230 should be understood as a reference memory that stores video data received by video encoder 200 for encoding (e.g., video data of the current block to be encoded). Figure 1 The memory 106 can also provide temporary storage for the outputs from the various units of the video encoder 200.

[0170] It shows Figure 14 The various units help to understand the operations performed by the video encoder 200. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. A fixed-function circuit is a circuit that provides a specific function and has predefined operations that can be performed. A programmable circuit is a circuit that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For example, a programmable circuit can execute software or firmware to cause the programmable circuit to operate in a manner defined by the instructions of the software or firmware. A fixed-function circuit can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more units can be different circuit blocks (fixed-function or programmable), and in some examples, one or more units can be integrated circuits.

[0171] The video encoder 200 may include an arithmetic logic unit (ALU), an elementary function unit (EFU), digital circuits, analog circuits, and / or a programmable core formed by programmable circuits. In an example where the operation of the video encoder 200 is performed using software executed by programmable circuits, memory 106 ( Figure 1 The video encoder 200 may store instructions (e.g., object code) of the software received and executed by the video encoder 200, or another memory (not shown) within the video encoder 200 may store such instructions.

[0172] The video data storage unit 230 is configured to store received video data. The video encoder 200 can retrieve images of the video data from the video data storage unit 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data storage unit 230 can be the raw video data to be encoded.

[0173] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra-frame prediction unit 226. The mode selection unit 202 may include additional functional units to perform video prediction based on other prediction modes. As an example, the mode selection unit 202 may include a palette unit, an intra-frame block copying unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, etc.

[0174] The mode selection unit 202 typically coordinates multiple coding passes to test combinations of coding parameters and the resulting rate-distortion values. Coding parameters may include the CTU-CU split, the prediction mode of the CU, the transformation type of the CU residual data, and the quantization parameters of the CU residual data. The mode selection unit 202 can ultimately select a combination of coding parameters that yields a better rate-distortion value than other tested combinations.

[0175] The video encoder 200 can segment images retrieved from the video data storage 230 into a series of CTUs and encapsulate one or more CTUs within a strip. The mode selection unit 202 can segment the CTUs of the image according to a tree structure (such as the MTT structure, QTBT structure, superblock structure, or quadtree structure described above). As mentioned above, the video encoder 200 can form one or more CUs by segmenting CTUs according to a tree structure. Such CUs can also generally be referred to as "video blocks" or "blocks".

[0176] Generally, mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-prediction unit 226) to generate predicted blocks for the current block (e.g., the current CU, or the overlapping portion of PU and TU in HEVC). For example, for inter-frame prediction of the current block, motion estimation unit 222 may perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previously encoded / decoded pictures stored in DPB 218). Specifically, motion estimation unit 222 may calculate values ​​representing the similarity between the potential reference block and the current block (e.g., based on sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), etc.). Motion estimation unit 222 may typically perform these calculations using the sample-by-sample difference between the current block and the considered reference blocks. Motion estimation unit 222 may identify reference blocks with the lowest values ​​produced by these calculations, the lowest value indicating the reference block that most closely matches the current block.

[0177] Motion estimation unit 222 can generate one or more motion vectors (MVs), where MVs define the position of a reference block in the reference image relative to the current block position in the current image. Motion estimation unit 222 can then provide the motion vectors to motion compensation unit 224. For example, for unidirectional inter-frame prediction, motion estimation unit 222 can provide a single motion vector, while for bidirectional inter-frame prediction, motion estimation unit 222 can provide two motion vectors. Motion compensation unit 224 can then use the motion vectors to generate prediction blocks. For example, motion compensation unit 224 can use the motion vectors to retrieve data for the reference blocks. As another example, if the motion vectors have fractional sample precision, motion compensation unit 224 can interpolate the values ​​of the prediction blocks according to one or more interpolation filters. Furthermore, for bidirectional inter-frame prediction, motion compensation unit 224 can retrieve data for two reference blocks identified by corresponding motion vectors and combine the retrieved data (e.g., by sample-by-sample averaging or weighted averaging).

[0178] When operating according to the AV1 video codec format, the motion estimation unit 222 and the motion compensation unit 224 can be configured to encode the codec blocks of the video data (e.g., both the luma codec block and the chroma codec block) using translational motion compensation, affine motion compensation, overlapped block motion compensation (OBMC), and / or composite inter-frame intra-frame prediction.

[0179] As another example, for intra-prediction or intra-prediction codec, intra-prediction unit 226 can generate a prediction block based on samples adjacent to the current block. For example, in directional mode, intra-prediction unit 226 can typically mathematically combine the values ​​of adjacent samples and fill these calculated values ​​across the current block in a defined direction to produce a prediction block. As another example, in DC mode, intra-prediction unit 226 can calculate the average of the adjacent samples of the current block and generate a prediction block to include this resulting average for each sample of the prediction block.

[0180] When operating according to the AV1 video codec format, the intra-prediction unit 226 can be configured to encode codec blocks (e.g., both luma and chroma codec blocks) of video data using directional intra-prediction, non-directional intra-prediction, recursive filter intra-prediction, chroma-from-luma (CFL) prediction, intra-block copying (IBC), and / or palette modes. The mode selection unit 202 may include additional functional units to perform video prediction according to other prediction modes. The mode selection unit 202 (e.g., motion estimation unit 222, motion compensation unit 224, and / or intra-prediction unit 226) can perform... Figure 13 Template matching technology.

[0181] Mode selection unit 202 provides the prediction block to residual generation unit 204. Residual generation unit 204 receives the original, uncoded version of the current block from video data memory 230 and the prediction block from mode selection unit 202. Residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block of the current block. In some examples, residual generation unit 204 can also determine the difference between sample values ​​in the residual block to generate a residual block using residual differential pulse code modulation (RDPCM). In some examples, residual generation unit 204 can be formed using one or more subtractor circuits that perform binary subtraction.

[0182] In the example where mode selection unit 202 divides a CU into PUs, each PU can be associated with a luma prediction unit and a corresponding chroma prediction unit. Video encoder 200 and video decoder 300 can support PUs of various sizes. As mentioned above, the size of a CU can refer to the size of its luma codec block, and the size of a PU can refer to the size of the luma prediction unit of the PU. Assuming a specific CU size is 2N×2N, video encoder 200 can support PU sizes of 2N×2N or N×N for intra-frame prediction and symmetrical PU sizes of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. Video encoder 200 and video decoder 300 can also support asymmetric segmentation for PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0183] In the example where mode selection unit 202 does not further divide the CU into PUs, each CU can be associated with a luma codec block and a corresponding chroma codec block. As mentioned above, the size of the CU can refer to the size of the luma codec block of the CU. The video encoder 200 and the video decoder 300 can support CU sizes of 2N×2N, 2N×N, or N×2N.

[0184] For other video coding and decoding techniques such as intra-block copy mode, affine mode, and linear model (LM) mode, as examples, mode selection unit 202 generates a prediction block for the current block to be encoded via a corresponding unit associated with the coding and decoding technique. In some examples, such as palette mode coding and decoding, mode selection unit 202 may not generate a prediction block, but instead generate syntax elements indicating how the block should be reconstructed based on a selected palette. In this mode, mode selection unit 202 may provide these syntax elements to entropy coding unit 220 for encoding.

[0185] As described above, the residual generation unit 204 receives video data of the current block and the corresponding prediction block. The residual generation unit 204 then generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

[0186] Transform processing unit 206 applies one or more transformations to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transformations to the residual block to form the transform coefficient block. For example, transform processing unit 206 may apply discrete cosine transform (DCT), direction transform, KL transform (KLT), or conceptually similar transformations to the residual block. In some examples, transform processing unit 206 may perform multiple transformations on the residual block, such as primary and secondary transformations (e.g., rotation transformations). In some examples, transform processing unit 206 does not apply any transformations to the residual block.

[0187] When operating according to AV1, the transform processing unit 206 can apply one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). The transform processing unit 206 can apply various transforms to the residual block to form the transform coefficient block. For example, the transform processing unit 206 can apply a combination of horizontal / vertical transforms, which may include the Discrete Cosine Transform (DCT), the Asymmetric Discrete Sine Transform (ADST), the Inverted ADST (e.g., the Reverse ADST), and the Identity Transform (IDTX). When using the Identity Transform, the transform is skipped in either the vertical or horizontal direction. In some examples, transform processing may be skipped altogether.

[0188] Quantization unit 208 can quantize the transform coefficients in the transform coefficient block to produce a quantized transform coefficient block. Quantization unit 208 can quantize the transform coefficients of the transform coefficient block based on the quantization parameter (QP) value associated with the current block. Video encoder 200 (e.g., via mode selection unit 202) can adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may introduce information loss; therefore, the accuracy of the quantized transform coefficients may be lower than that of the original transform coefficients generated by transform processing unit 206.

[0189] The inverse quantization unit 210 and the inverse transform processing unit 212 can apply inverse quantization and inverse transform to the quantized transform coefficient block, respectively, to reconstruct the residual block based on the transform coefficient block. The reconstruction unit 214 can generate a reconstructed block corresponding to the current block based on the reconstructed residual block and the prediction block generated by the mode selection unit 202 (although potentially with some degree of distortion). For example, the reconstruction unit 214 can add the samples of the reconstructed residual block to the corresponding samples in the prediction block generated by the mode selection unit 202 to generate the reconstructed block.

[0190] Filter unit 216 can perform one or more filtering operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce block artifacts along the edges of the CU. In some examples, the operation of filter unit 216 can be skipped.

[0191] When operating according to AV1, filter unit 216 can perform one or more filtering operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce block artifacts along the edges of the CU. In other examples, filter unit 216 can apply a constrained directional enhancement filter (CDEF) after deblocking and can include the application of an inseparable nonlinear low-pass directional filter based on the estimated edge orientation. Filter unit 216 can also include a loop recovery filter applied after CDEF and can include a separable symmetric normalized Wiener filter or a dual self-guided filter.

[0192] The video encoder 200 stores reconstructed blocks in the DPB 218. For example, in an example where the filter unit 216 is not operated, the reconstruction unit 214 may store reconstructed blocks in the DPB 218. In an example where the filter unit 216 is operated, the filter unit 216 may store filtered reconstructed blocks in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 may retrieve a reference image formed by the reconstructed (and potentially filtered) blocks from the DPB 218 to perform inter-frame prediction of blocks in subsequent encoded images. Furthermore, the intra-frame prediction unit 226 may use the reconstructed blocks in the DPB 218 of the current image to perform intra-frame prediction of other blocks in the current image.

[0193] Generally, entropy coding unit 220 can entropy code syntax elements received from other functional components of video encoder 200. For example, entropy coding unit 220 can entropy code quantized transform coefficient blocks from quantization unit 208. As another example, entropy coding unit 220 can entropy code predictive syntax elements (e.g., motion information for inter-frame prediction or intra-frame mode information for intra-frame prediction) from mode selection unit 202. Entropy coding unit 220 can perform one or more entropy coding operations on syntax elements, another example of video data, to generate entropy-coded data. For example, entropy coding unit 220 can perform context-adaptive variable length coding (CAVLC), CABAC, variable-to-variable (V2V) length coding, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE), exponential Golomb coding, or another type of entropy coding operation on the data. In some examples, entropy coding unit 220 can operate in a bypass mode where entropy coding of syntax elements is not performed.

[0194] The video encoder 200 can output a bitstream containing the entropy-encoded syntax elements required to reconstruct strips or images. Specifically, the entropy coding unit 220 can output the bitstream.

[0195] According to AV1, entropy coding unit 220 can be configured as a symbol-to-symbol adaptive multi-symbol arithmetic codec. The syntax elements in AV1 include an N-element alphabet, and the context (e.g., a probability model) includes a set of N probabilities. Entropy coding unit 220 can store the probabilities as an n-bit (e.g., 15-bit) cumulative distribution function (CDF). Entropy coding unit 220 can perform recursive scaling to update the context using an update factor based on the alphabet size.

[0196] The above operations are described at the block level. This description should be understood as operations on luma codec blocks and / or chroma codec blocks. As mentioned above, in some examples, the luma codec block and chroma codec block are the luma and chroma components of the CU. In some examples, the luma codec block and chroma codec block are the luma and chroma components of the PU.

[0197] In some examples, for the chroma codec block, it is not necessary to repeat the operations performed for the luma codec block. As an example, it is not necessary to repeat the process of identifying the motion vector (MV) and reference image of the luma codec block to identify the MV and reference image of the chroma block. Instead, the MV of the luma codec block can be scaled to determine the MV of the chroma block, and the reference image can be the same. As another example, the intra-frame prediction process can be the same for both the luma and chroma codec blocks.

[0198] Video encoder 200 represents an example of a device configured to encode video data, the device including: one or more memories configured to store video data; and one or more processors implemented in circuitry and communicatively coupled to the one or more memories, the one or more processors being configured to: determine a current template of a current block of video data; determine a potential reference block for the current block in a picture of the video data; determine that the potential reference block and its reference template are completely within the picture, the reference template having the same shape as the current template; perform template matching between the reference template and the current template based on the fact that both the potential reference block and the reference template are completely within the picture; and encode the current block based on the template matching.

[0199] The video encoder 200 also represents an example of a device configured to encode video data, the device including: a memory configured to store video data; and one or more processing units implemented in a circuit and configured to: determine at least one portion of a template based on the availability of neighboring blocks of video data, the neighboring blocks being adjacent to the current block of video data; perform template matching based on the template; and encode the current block based on the template matching.

[0200] The video encoder 200 also represents an example of a device configured to encode video data, the device including: a memory configured to store video data; and one or more processing units implemented in a circuit and configured to: determine a search range; determine that a reference block of the video data is completely within the search range; perform template matching between the reference block and the current block of the video data based on the fact that the reference block is completely within the search range; and encode the current block based on the template matching.

[0201] The video encoder 200 also represents an example of a device configured to encode video data, the device including: a memory configured to store video data; and one or more processing units implemented in a circuit and configured to: determine a search region based on a region that has already been decoded; perform template matching based on the search region; and encode the current block of video data based on the template matching.

[0202] The video encoder 200 also represents an example of a device configured to encode video data, the device including: a memory configured to store video data; and one or more processing units implemented in a circuit and configured to: perform a template matching search; perform a refined template matching search on the result of the template matching search, wherein the refined template matching search includes searching in multiple search regions; and encode the current block of video data based on the refined template matching search.

[0203] The video encoder 200 also represents an example of a device configured to encode video data, the device including: a memory configured to store video data; and one or more processing units implemented in a circuit and configured to: determine that samples of a reference block of video data are unavailable; copy samples from corresponding positions in the current block of video data based on the unavailability of samples of the reference block; and encode the current block based on the reference block.

[0204] Figure 15 A block diagram of an example video decoder 300 that can perform the techniques of this disclosure is shown. Figure 15 This disclosure is provided for illustrative purposes and is not intended to limit the techniques broadly exemplified and described herein. For illustrative purposes, this disclosure describes a video decoder 300 based on VVC and HEVC technologies. However, the techniques of this disclosure can be implemented by video codec devices configured for other video codec standards.

[0205] exist Figure 15 In the example, the video decoder 300 includes a coded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, and a DPB 314. Any or all of the CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 can be implemented in one or more processors or in processing circuitry. For example, units of the video decoder 300 can be implemented as one or more circuit or logic elements, as part of hardware circuitry or as part of a processor, ASIC, or FPGA. Furthermore, the video decoder 300 may include additional or alternative processors or processing circuitry to perform these and other functions.

[0206] The prediction processing unit 304 includes a motion compensation unit 316 and an intra-prediction unit 318. The prediction processing unit 304 may include additional units to perform predictions according to other prediction modes. As an example, the prediction processing unit 304 may include a palette unit, an intra-block copying unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, the video decoder 300 may include more, fewer, or different functional components. The prediction processing unit 304 (e.g., motion compensation unit 316 and / or intra-prediction unit 318) can perform... Figure 13 Template matching technology.

[0207] When operating according to AV1, motion compensation unit 316 can be configured to decode the codec blocks of video data (e.g., both luma and chroma codec blocks) using translational motion compensation, affine motion compensation, OBMC, and / or composite inter-intra-frame prediction as described above. Intra-frame prediction unit 318 can be configured to decode the codec blocks of video data (e.g., both luma and chroma codec blocks) using directional intra-frame prediction, non-directional intra-frame prediction, recursive filtered intra-frame prediction, CFL, IBC, and / or palette modes as described above.

[0208] CPB memory 320 can store video data to be decoded by components of video decoder 300, such as encoded video bitstreams. The video data stored in CPB memory 320 can be obtained from, for example, computer-readable medium 110 (…). Figure 1 The CPB memory 320 may include a CPB that stores encoded video data (e.g., syntax elements) from the encoded video bitstream. Additionally, the CPB memory 320 may store video data other than the syntax elements of the encoded / decoded pictures, such as temporary data representing the output from various units of the video decoder 300. The DPB 314 typically stores decoded pictures that the video decoder 300 may output and / or use as reference video data when decoding subsequent data or pictures from the encoded video bitstream. The CPB memory 320 and DPB 314 may be formed from any of a variety of memory devices, such as DRAM (including SDRAM), MRAM, RRAM, or other types of memory devices. The CPB memory 320 and DPB 314 may be provided by the same memory device or separate memory devices. In various examples, the CPB memory 320 may be on-chip with other components of the video decoder 300, or off-chip relative to those components.

[0209] Additionally or alternatively, in some examples, the video decoder 300 can be generated from the memory 120 ( Figure 1The memory 120 can retrieve encoded and decoded video data from the memory. In other words, the memory 120 can store data as discussed above regarding the CPB memory 320. Similarly, when some or all of the functions of the video decoder 300 are implemented in software to be executed by the processing circuitry of the video decoder 300, the memory 120 can store instructions executed by the video decoder 300.

[0210] It shows Figure 15 The various units shown aid in understanding the operations performed by the video decoder 300. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Figure 14 Similarly, a fixed-function circuit is a circuit that provides a specific function and has a predefined set of operations that can be performed. A programmable circuit is a circuit that can be programmed to perform various tasks and provide flexible functionality within the operations it can perform. For example, a programmable circuit can execute software or firmware to cause it to operate in a manner defined by the instructions in the software or firmware. A fixed-function circuit can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is typically immutable. In some examples, one or more units may be different circuit blocks (fixed-function or programmable), and in some examples, one or more units may be integrated circuits.

[0211] The video decoder 300 may include an ALU, an EFU, digital circuitry, analog circuitry, and / or a programmable core formed by programmable circuitry. In an example where the operation of the video decoder 300 is performed by software executed on the programmable circuitry, on-chip or off-chip memory may store instructions (e.g., object code) of the software received and executed by the video decoder 300.

[0212] Entropy decoding unit 302 can receive encoded video data from the CPB and perform entropy decoding on the video data to reproduce the syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filter unit 312 can generate decoded video data based on the syntax elements extracted from the bitstream.

[0213] Generally, the video decoder 300 reconstructs the image on a block-by-block basis. The video decoder 300 can perform the reconstruction operation on each block individually (where the block currently being reconstructed (i.e., decoded) can be referred to as the "current block").

[0214] Entropy decoding unit 302 can entropy decode the syntax elements of the quantized transform coefficients defining the quantized transform coefficient block, as well as transform information such as quantization parameters (QP) and / or (multiple) transform mode indications. Inverse quantization unit 306 can use the QP associated with the quantized transform coefficient block to determine the degree of quantization, and similarly determine the degree of inverse quantization to be applied by inverse quantization unit 306. Inverse quantization unit 306 can, for example, perform a bit-left shift operation to inverse quantize the quantized transform coefficients. Thus, inverse quantization unit 306 can form a transform coefficient block including the transform coefficients.

[0215] After the inverse quantization unit 306 forms the transform coefficient block, the inverse transform processing unit 308 can apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, the inverse transform processing unit 308 can apply inverse DCT, inverse integer transform, inverse KL transform (KLT), inverse rotation transform, inverse direction transform, or another inverse transform to the transform coefficient block.

[0216] Furthermore, prediction processing unit 304 generates prediction blocks based on prediction information syntax elements entropy-decoded by entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is predicted inter-frame, motion compensation unit 316 can generate prediction blocks. In this case, the prediction information syntax elements may indicate a reference image from which the reference block is retrieved in DPB 314, and a motion vector identifying the position of the reference block in the reference image relative to the current block position in the current image. Motion compensation unit 316 can generally be configured in a manner substantially similar to that of motion compensation unit 224 ( Figure 14 The method described is used to perform the inter-frame prediction process.

[0217] As another example, if the prediction information syntax element indicates that the current block is intra-predicted, then intra-prediction unit 318 can generate a prediction block according to the intra-prediction mode indicated by the prediction information syntax element. Similarly, intra-prediction unit 318 can generally be configured in a manner substantially similar to that of intra-prediction unit 226 ( Figure 14 The intra-prediction process is performed in the manner described. The intra-prediction unit 318 can retrieve data of neighboring samples of the current block from the DPB 314.

[0218] Reconstruction unit 310 can reconstruct the current block using prediction blocks and residual blocks. For example, reconstruction unit 310 can add the samples of the residual block to the corresponding samples of the prediction block to reconstruct the current block.

[0219] Filter unit 312 can perform one or more filtering operations on the reconstructed block. For example, filter unit 312 can perform a deblocking operation to reduce block artifacts along the edges of the reconstructed block. The operation of filter unit 312 is not necessarily performed in all examples.

[0220] The video decoder 300 can store reconstructed blocks in the DPB 314. For example, in an example where the filter unit 312 is not operated, the reconstruction unit 310 can store the reconstructed blocks in the DPB 314. In an example where the filter unit 312 is operated, the filter unit 312 can store the filtered reconstructed blocks in the DPB 314. As described above, the DPB 314 can provide reference information to the prediction processing unit 304, such as samples of the current image for intra-frame prediction and previously decoded images for subsequent motion compensation. Furthermore, the video decoder 300 can output the decoded image (e.g., decoded video) from the DPB 314 for subsequent display on a display device (e.g., [unclear]). Figure 1 It is displayed on the display device 118.

[0221] In this manner, video decoder 300 represents an example of a video decoding device, comprising: one or more memories configured to store video data; and one or more processors implemented in circuitry and communicatively coupled to the one or more memories, the one or more processors being configured to: determine a current template of a current block of video data; determine a potential reference block for the current block in a picture of the video data; determine that the potential reference block and its reference template are entirely within the picture, the reference template having the same shape as the current template; perform template matching between the reference template and the current template based on the fact that both the potential reference block and the reference template are entirely within the picture; and decode the current block based on the template matching.

[0222] The video decoder 300 also represents an example of a video decoding device, including: a memory configured to store video data; and one or more processing units implemented in a circuit and configured to: determine at least one portion of a template based on the availability of neighboring blocks of video data, the neighboring blocks being adjacent to the current block of video data; perform template matching based on the template; and decode the current block based on the template matching.

[0223] The video decoder 300 also represents an example of a device configured to decode video data, the device including: a memory configured to store video data; and one or more processing units implemented in a circuit and configured to: determine a search range; determine that a reference block of the video data is completely within the search range; perform template matching between the reference block and the current block of the video data based on the fact that the reference block is completely within the search range; and decode the current block based on the template matching.

[0224] The video decoder 300 also represents an example of a device configured to decode video data, the device including: a memory configured to store video data; and one or more processing units implemented in a circuit and configured to: determine a search region based on a region that has already been decoded; perform template matching based on the search region; and decode the current block of video data based on the template matching.

[0225] The video decoder 300 also represents an example of a device configured to decode video data, the device including: a memory configured to store video data; and one or more processing units implemented in a circuit and configured to: perform a template matching search; perform a refined template matching search on the result of the template matching search, wherein the refined template matching search includes searching in multiple search regions; and decode the current block of video data based on the refined template matching search.

[0226] The video decoder 300 also represents an example of a device configured to decode video data, the device including: a memory configured to store video data; and one or more processing units implemented in a circuit and configured to: determine that samples of a reference block of video data are unavailable; copy samples from corresponding positions in the current block of video data based on the unavailability of samples of the reference block; and decode the current block based on the reference block.

[0227] Figure 16 This is a flowchart illustrating an example method for encoding the current block according to the technology of this disclosure. The current block may be or includes the current CU. Although for video encoder 200 ( Figure 1 and Figure 14 The description is provided, but it should be understood that other devices can be configured to perform similar actions. Figure 16 The method.

[0228] In this example, video encoder 200 initially predicts the current block (350). For example, video encoder 200 may form a prediction block for the current block. When forming the prediction block for the current block, video encoder 200 may perform... Figure 13The template matching technique is used. The video encoder 200 can then compute the residual block of the current block (352). To compute the residual block, the video encoder 200 can compute the difference between the original uncoded block and the predicted block of the current block. The video encoder 200 can then transform the residual block and quantize the transform coefficients of the residual block (354). Next, the video encoder 200 can scan the quantized transform coefficients of the residual block (356). During or after the scan, the video encoder 200 can entropy encode the transform coefficients (358). For example, the video encoder 200 can encode the transform coefficients using CAVLC or CABAC. The video encoder 200 can then output the entropy-encoded data of the block (360).

[0229] Figure 17 This is a flowchart illustrating an example method for decoding a current block of video data according to the technology of this disclosure. The current block may be or includes the current CU. Although for video decoder 300 ( Figure 1 and Figure 15 The description is provided, but it should be understood that other devices can be configured to perform similar actions. Figure 17 The method.

[0230] The video decoder 300 can receive entropy-coded data of the current block, such as entropy-coded prediction information corresponding to the current block and entropy-coded data of the transform coefficients of the residual block (370). The video decoder 300 can perform entropy decoding on the entropy-coded data to determine the prediction information of the current block and the transform coefficients of the reconstructed residual block (372). The video decoder 300 can predict the current block (374), for example, by calculating the prediction block of the current block using the intra-frame or inter-frame prediction mode indicated by the prediction information of the current block. For example, when predicting the current block, the video decoder 300 can use... Figure 13 The template matching technique is used. The video decoder 300 can then perform an inverse scan (376) on the reconstructed transform coefficients to create a block of quantized transform coefficients. The video decoder 300 can then inverse quantize the transform coefficients and apply an inverse transform to the transform coefficients to produce a residual block (378). The video decoder 300 can then perform final decoding on the current block by combining the predicted block and the residual block (380).

[0231] The following numbered clauses illustrate one or more aspects of the devices and technologies described in this disclosure.

[0232] Clause 1A: A method for encoding and decoding video data, the method comprising: determining at least a portion of a template based on the availability of neighboring blocks of the video data, the neighboring blocks being adjacent to a current block of the video data; performing template matching based on the template; and encoding and decoding the current block based on the template matching.

[0233] Clause 2A: The method according to Clause 1A, wherein at least one portion includes a left portion, and wherein determining at least one portion includes determining that all neighboring blocks to the left of the current block are available.

[0234] Clause 3A: The method described in Clause 2A, wherein the current block includes blocks that are adjacent to the top image boundary but not adjacent to the left image boundary.

[0235] Clause 4A: The method according to any one of Clauses 1A-3A, wherein at least one portion includes an upper portion, and wherein determining at least one portion includes determining that all neighboring blocks above the current block are available.

[0236] Clause 5A: The method described in Clause 4A, wherein the current block includes blocks that are not adjacent to the top image boundary but are adjacent to the left image boundary.

[0237] Clause 6A: The method according to any one of Clauses 1A-5A, wherein at least one portion includes an upper left portion, and wherein determining at least one portion includes determining that adjacent blocks to the left and above the current block are available.

[0238] Clause 7A: The method according to any one of Clauses 1A-6A further includes: determining that adjacent blocks to the left and below the current block are available; and extending the left portion downward based on the availability of adjacent blocks to the left and below the current block.

[0239] Clause 8A: The method according to any one of Clauses 1A-7A further includes: determining that adjacent blocks above and to the right of the current block are available; and extending the upper portion to the right based on the availability of adjacent blocks above and to the right of the current block.

[0240] Clause 9A: A method for encoding and decoding video data, the method comprising: determining a search range; determining that a reference block of the video data is entirely within the search range; performing template matching between the reference block and a current block of the video data based on the fact that the reference block is entirely within the search range; and encoding and decoding the current block based on the template matching.

[0241] Clause 10A: The method described in Clause 9A further includes determining that the reference block and the template of the reference block are entirely within the image, and wherein performing template matching is also based on the fact that the reference block and the template of the reference block are entirely within the image.

[0242] Clause 11A: The method according to Clause 9A or Clause 10A, wherein the template of the reference block includes at least two of the left portion, the upper left portion, or the upper portion, and wherein the upper left sample point of the reference block is offset from the picture boundary by the size of the template in both the vertical and horizontal directions.

[0243] Clause 12A: The method described in Clause 9A or Clause 10A, wherein the template of the reference block includes only the left side portion, and wherein the top left sample point of the reference block is offset from the picture boundary by the size of the template in the horizontal direction.

[0244] Clause 13A: The method described in Clause 12A, wherein a searched reference block for template matching is adjacent to the top image boundary but not adjacent to the left image boundary.

[0245] Clause 14A: The method described in Clause 9A or Clause 10A, wherein the template of the reference block includes only the upper portion, and wherein the upper left sample point of the reference block is offset from the picture boundary by the size of the template in the vertical direction.

[0246] Clause 15A: The method described in Clause 14A, wherein a searched reference block for template matching is adjacent to the left image boundary but not adjacent to the upper image boundary.

[0247] Clause 16A: The method described in Clause 9A further includes: determining that the template of the reference block is not entirely within the image; and filling in unusable template portions based on the fact that the template of the reference block is not entirely within the image.

[0248] Clause 17A: The method according to Clause 16A, wherein the reference block has an L-shaped template and is adjacent to at least one of the left picture boundary or the top picture boundary.

[0249] Clause 18A: A method for encoding and decoding video data, the method comprising: determining a search region based on a region that has already been decoded; performing template matching based on the search region; and encoding and decoding the current block of video data based on the template matching.

[0250] Clause 19A: The method described in Clause 18A, wherein the search area includes all blocks, wholly or partially located within the current block and which have been completely reconstructed.

[0251] Clause 20A: A method for encoding and decoding video data, the method comprising: performing a template matching search; performing a refined template matching search on the result of the template matching search, wherein the refined template matching search includes searching in multiple search regions; and encoding and decoding a current block of video data based on the refined template matching search.

[0252] Clause 21A: The method described in Clause 20A further includes determining the best match with the lowest template cost based on samples in multiple search regions.

[0253] Clause 22A: A method for encoding and decoding video data, the method comprising: determining that samples of a reference block of video data are unavailable; copying samples from corresponding positions in a current block of video data based on the unavailability of samples of the reference block; and encoding and decoding the current block based on the reference block.

[0254] Clause 23A: The method according to any one of Clauses 1A-22A, wherein encoding / decoding includes decoding.

[0255] Clause 24A: The method according to any one of Clauses 1A-23A, wherein encoding / decoding includes encoding.

[0256] Clause 25A: An apparatus for encoding and decoding video data, the apparatus comprising one or more components for performing a method according to any one of Clauses 1A-24A.

[0257] Clause 26A: The device described in Clause 25A, wherein one or more components include one or more processors implemented in a circuit.

[0258] Clause 27A: The device pursuant to any one of Clauses 25A and 26A further includes a memory for storing video data.

[0259] Clause 28A: The device pursuant to any one of Clauses 25A-27A further includes a display configured to display decoded video data.

[0260] Clause 29A: Equipment under any one of Clauses 25A-28A, wherein the equipment includes one or more of a camera, computer, mobile device, broadcast receiver equipment or set-top box.

[0261] Clause 30A: The device pursuant to any one of Clauses 25A-29A, wherein the device includes a video decoder.

[0262] Clause 31A: The device pursuant to any one of Clauses 25A-30A, wherein the device includes a video encoder.

[0263] Clause 32A: A computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to perform the method according to any one of Clauses 1A-24A.

[0264] Clause 33A: An apparatus for encoding video data, the apparatus comprising: components for performing the method according to any one of Clauses 1A-24A.

[0265] Clause 1B: A method for encoding and decoding video data, the method comprising: determining a current template of a current block of video data; determining a potential reference block for the current block in a picture of the video data; determining that the potential reference block and a reference template of the potential reference block are completely within the picture, the reference template having the same shape as the current template; performing template matching between the reference template and the current template based on the fact that both the potential reference block and the reference template are completely within the picture; and encoding and decoding the current block based on the template matching.

[0266] Clause 2B: The method according to Clause 1B, wherein the potential reference block is a first potential reference block, and wherein the reference template is a first reference template, and wherein the method further comprises: determining that at least one of a second potential reference block or a second reference template of the second potential reference block is not entirely within the picture; and based on the fact that at least one of the second potential reference block or the second reference template is not entirely within the picture, not performing template matching between the second reference template and the current template of the current block of video data; and encoding / decoding the current block using a reference block different from the second potential reference block.

[0267] Clause 3B: The method according to Clause 1B or Clause 2B, wherein the reference template includes at least two of a left portion, an upper left portion, or an upper portion, and wherein the upper left sample of the potential reference block is offset from the top image boundary by the width of at least one of the upper left portion or the upper portion in the vertical direction, and is offset from the left image boundary by the width of at least one of the left portion or the upper left portion in the horizontal direction.

[0268] Clause 4B: The method described in Clause 1B or Clause 2B, wherein the reference template comprises only the left portion, and wherein the top left sample of the potential reference block is offset horizontally from the left image boundary by the width of the left portion, and wherein the potential reference block is adjacent to the top image boundary but not adjacent to the left image boundary.

[0269] Clause 5B: The method described in Clause 1B or Clause 2B, wherein the reference template includes only the upper portion, and wherein the upper left sample of the potential reference block is offset from the top image boundary by the width of the upper portion in the vertical direction, and wherein the potential reference block is adjacent to the left image boundary but not adjacent to the top image boundary.

[0270] Clause 6B: The method according to any one of Clauses 1B-5B, wherein the potential reference block is one of a plurality of potential reference blocks, further comprising: determining a search region of an image for the plurality of potential reference blocks to include all blocks, wholly or partially reconstructed, located within the current codec tree unit (CTU) of the video data; and determining the potential reference block within the search region.

[0271] Clause 7B: The method according to Clause 6B, wherein the search area includes one of a plurality of search areas, and the method further includes performing a refined template match search on the results of the template match search, wherein the refined template match search includes searching in the plurality of search areas.

[0272] Clause 8B: The method according to any one of Clauses 1B-7B, wherein the current template comprises at least one portion, and wherein determining the current template comprises determining at least one portion based on the availability of neighboring blocks of the current block.

[0273] Clause 9B: The method according to Clause 8B, wherein at least one portion includes a left portion, and wherein determining at least one portion includes determining that all neighboring blocks to the left of the current block are available, and wherein the current block includes blocks adjacent to the top image boundary but not adjacent to the left image boundary.

[0274] Clause 10B: The method according to Clause 8B, wherein at least one portion includes an upper portion, and wherein determining at least one portion includes determining that all neighboring blocks above the current block are available, and wherein the current block includes blocks that are not adjacent to the top image boundary but are adjacent to the left image boundary.

[0275] Clause 11B: The method according to Clause 8B, wherein at least one portion includes the upper left portion, and wherein determining at least one portion includes determining that adjacent blocks to the left and above the current block are available.

[0276] Clause 12B: The method described under Clause 8B or Clause 9B further comprises: determining that adjacent blocks to the left and below the current block are available; and, based on the availability of adjacent blocks to the left and below the current block, extending the left portion downward such that the bottom of the left portion is lower than the bottom of the current block in the vertical direction.

[0277] Clause 13B: The method described in accordance with Clause 8B or Clause 10B further includes: determining that adjacent blocks above and to the right of the current block are available; and based on the availability of adjacent blocks above and to the right of the current block, extending the upper portion to the right such that the right edge of the upper portion is further to the right in the horizontal direction than the right edge of the current block.

[0278] Clause 14B: The method described in accordance with Clauses 1B-13B, wherein encoding / decoding includes decoding.

[0279] Clause 15B: The method described in accordance with Clauses 1B-13B, wherein encoding / decoding includes encoding.

[0280] Clause 16B: An apparatus for encoding and decoding video data, the apparatus comprising: one or more memories configured to store video data; and one or more processors implemented in circuitry and communicatively coupled to the one or more memories, the one or more processors being configured to: determine a current template of a current block of video data; determine a potential reference block for the current block in a picture of the video data; determine that the potential reference block and its reference template are entirely within the picture, the reference template having the same shape as the current template; perform template matching between the reference template and the current template based on the fact that both the potential reference block and the reference template are entirely within the picture; and encode and decode the current block based on the template matching.

[0281] Clause 17B: The device according to Clause 16B, wherein the current template comprises at least one portion, and wherein, as part of determining the current template, one or more processors are configured to determine at least one portion based on the availability of neighboring blocks of the current block.

[0282] Clause 18B: The device described in Clause 16B or Clause 17B, wherein encoding / decoding includes decoding, and wherein the device further includes a display configured to display decoded video data.

[0283] Clause 19B: The device described in Clause 16B or Clause 17B, wherein encoding / decoding includes encoding, and wherein the device further includes a camera configured to capture video data.

[0284] Clause 20B: A computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to: determine a current template for a current block of video data; determine a potential reference block for the current block in a picture of the video data; determine that the potential reference block and its reference template are entirely within the picture, the reference template having the same shape as the current template; perform template matching between the reference template and the current template based on the fact that both the potential reference block and the reference template are entirely within the picture; and encode or decode the current block based on the template matching.

[0285] It should be recognized that, depending on the example, certain actions or events of any technique described herein may be performed in a different order, and may be added, combined, or omitted entirely (e.g., not all described actions or events are necessary for the practice of the technique). Furthermore, in some examples, actions or events may be performed concurrently (e.g., through multithreading, interrupt handling, or multiprocessor) rather than sequentially.

[0286] In one or more examples, the described functionality may be implemented using hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored on or transmitted on a computer-readable medium as one or more instructions or code, and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium (such as a data storage medium) or a communication medium (including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol). In this way, a computer-readable medium may generally correspond to (1) a tangible, non-transitory computer-readable storage medium or (2) a communication medium (such as a signal or carrier wave). A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0287] By way of example and not limitation, such computer-readable storage media may include one or more of RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. The disks and optical discs used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. The above combinations should also be included within the scope of computer-readable media.

[0288] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Accordingly, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein can be provided in dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, these techniques can be implemented entirely within one or more circuit or logic elements.

[0289] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but they do not necessarily need to be implemented by different hardware units. Rather, as described above, various units can be combined with appropriate software and / or firmware in a codec hardware unit, or provided by a collection of interoperable hardware units, including one or more processors as described above.

[0290] Various examples have been described. These and other examples are within the scope of the appended claims.

Claims

1. A method for encoding and decoding video data, the method comprising: Determine the current template of the current block of the video data; Identify potential reference blocks for the current block within the images of the video data; The potential reference block and its reference template are determined to be completely within the image, and the reference template has the same shape as the current template; Based on the fact that both the potential reference block and the reference template are completely within the image, template matching is performed between the reference template and the current template; as well as The current block is encoded and decoded based on the template matching.

2. The method of claim 1, wherein the potential reference block is a first potential reference block, and wherein the reference template is a first reference template, and wherein the method further comprises: It is determined that at least one of the second potential reference block or the second reference template of the second potential reference block is not entirely within the image; as well as If at least one of the second potential reference block or the second reference template is not completely within the image, template matching between the second reference template and the current template of the current block of the video data is not performed; as well as The current block is encoded and decoded using a reference block different from the second potential reference block.

3. The method of claim 1, wherein the reference template comprises at least two of a left portion, an upper left portion, or an upper portion, and wherein the upper left sample point of the potential reference block is offset from the top image boundary by the width of at least one of the upper left portion or the upper portion in the vertical direction, and is offset from the left image boundary by the width of at least one of the left portion or the upper left portion in the horizontal direction.

4. The method of claim 1, wherein the reference template comprises only the left portion, and wherein the upper left sample of the potential reference block is offset horizontally from the left image boundary by the width of the left portion, and wherein the potential reference block is adjacent to the top image boundary but not adjacent to the left image boundary.

5. The method of claim 1, wherein the reference template comprises only the upper portion, and wherein the upper left sample point of the potential reference block is offset from the top image boundary by the width of the upper portion in the vertical direction, and wherein the potential reference block is adjacent to the left image boundary but not adjacent to the top image boundary.

6. The method of claim 1, wherein the potential reference block is one of a plurality of potential reference blocks, the method further comprising: Determine the search region of the image for the plurality of potential reference blocks to include all or part of all blocks located within the current codec tree unit (CTU) of the video data that are fully reconstructed; as well as The potential reference block is determined within the search area.

7. The method of claim 6, wherein the search region includes one of a plurality of search regions, and the method further includes performing a refined template matching search on the results of the template matching search, wherein the refined template matching search includes searching in the plurality of search regions.

8. The method of claim 1, wherein the current template comprises at least one portion, and wherein determining the current template comprises determining the at least one portion based on the availability of neighboring blocks of the current block.

9. The method of claim 8, wherein the at least one portion includes a left portion, and wherein determining the at least one portion includes determining that all neighboring blocks to the left of the current block are available, and wherein the current block includes blocks adjacent to the top image boundary but not adjacent to the left image boundary.

10. The method of claim 8, wherein the at least one portion includes an upper portion, and wherein determining the at least one portion includes determining that all neighboring blocks above the current block are available, and wherein the current block includes a block that is not adjacent to the top image boundary but is adjacent to the left image boundary.

11. The method of claim 8, wherein the at least one portion includes an upper left portion, and wherein determining the at least one portion includes determining that adjacent blocks to the left and above the current block are available.

12. The method according to claim 8, further comprising: Determine that the adjacent blocks to the left and below the current block are available; as well as Based on the availability of adjacent blocks to the left and below the current block, extend the left portion downwards such that the bottom of the left portion is lower than the bottom of the current block in the vertical direction.

13. The method of claim 8, further comprising: Determine that the adjacent blocks above and to the right of the current block are available; as well as Based on the availability of adjacent blocks above and to the right of the current block, the upper portion is extended to the right such that the right edge of the upper portion is further to the right in the horizontal direction than the right edge of the current block.

14. The method of claim 1, wherein the encoding / decoding includes decoding.

15. The method of claim 1, wherein the encoding / decoding includes encoding.

16. An apparatus for encoding and decoding video data, the apparatus comprising: One or more memories are configured to store the video data; as well as One or more processors, implemented in a circuit and communicatively coupled to the one or more memories, are configured to: Determine the current template of the current block of the video data; Identify potential reference blocks for the current block within the images of the video data; The potential reference block and its reference template are determined to be completely within the image, and the reference template has the same shape as the current template; Based on the fact that both the potential reference block and the reference template are completely within the image, template matching is performed between the reference template and the current template; as well as The current block is encoded and decoded based on the template matching.

17. The device of claim 16, wherein the current template comprises at least one portion, and wherein, as part of determining the current template, the one or more processors are configured to determine the at least one portion based on the availability of neighboring blocks of the current block.

18. The device of claim 16, wherein encoding / decoding includes decoding, and wherein the device further includes a display configured to display decoded video data.

19. The device of claim 16, wherein encoding and decoding include encoding, and wherein the device further includes a camera configured to capture the video data.

20. A computer-readable storage medium having instructions stored thereon, said instructions, when executed, causing one or more processors to: Determine the current template for the current block of video data; Identify potential reference blocks for the current block within the images of the video data; The potential reference block and its reference template are determined to be completely within the image, and the reference template has the same shape as the current template; Based on the fact that both the potential reference block and the reference template are completely within the image, template matching is performed between the reference template and the current template; as well as The current block is encoded and decoded based on the template matching.

Images