Multiple transform selection for intra block copy and intra template matching in video coding

Abstract

A method of decoding video data includes receiving a block of video data encoded using an Intra Block Copy (IBC) mode or an Intra Template Matching Prediction (TMP) mode; determining, using a multiple transform selection (MTS) process, a transform pair of the plurality of transform pairs for decoding the block; and decoding the block using the transform pair.

Description

[0001] This application claims the benefit of U.S. Patent Application No. 18 / 631,465, filed April 10, 2024, and U.S. Provisional Patent Application No. 63 / 495,976, filed April 13, 2023, the entire contents of which are incorporated herein by reference. U.S. Patent Application No. 18 / 631,465, filed April 10, 2024, claims the benefit of U.S. Provisional Patent Application No. 63 / 495,976, filed April 13, 2023. Technical Field

[0002] This disclosure relates to video encoding and video decoding. Background Technology

[0003] Digital video functionality can be integrated into a wide variety of devices, including digital televisions, digital live broadcast systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio phones, so-called "smartphones," video conferencing equipment, video streaming devices, and more. Digital video devices implement video decoding technologies, such as those defined by standards such as MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10, Advanced Video Decoding (AVC), ITU-T H.265 / High-Efficiency Video Decoding (HEVC), ITU-T H.266 / Variety Video Decoding (VVC), and extensions to these standards, as well as proprietary video codecs / formats (such as those described in AOMedia Video 1 (AV1) developed by the Open Media Alliance). By implementing such video decoding technology, video devices can more effectively send and receive digital video information, encode, decode, and / or store digital video information.

[0004] Video decoding techniques include spatial (intra-frame picture) prediction and / or temporal (inter-frame picture) prediction to reduce or remove inherent redundancy in video sequences. For block-based video decoding, video slices (e.g., video pictures or portions of video pictures) can be segmented into video blocks, which may also be referred to as decoding tree units (CTUs), decoding units (CUs), and / or decoding nodes. Video blocks in an intra-frame decoding (I) slice of a picture are encoded using spatial prediction relative to reference samples in adjacent blocks within the same picture. Video blocks in an inter-frame decoding (P or B) slice of a picture can use spatial prediction with respect to reference samples in adjacent blocks within the same picture or temporal prediction with respect to reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame. Summary of the Invention

[0005] Generally, this disclosure describes techniques for video decoding. Specifically, the techniques described herein relate to transform decoding. This disclosure describes a Multiple Transform Selection (MTS) technique for intra block copy (IBC) mode and an MTS scheme for intra template matching prediction (TMP) mode. The video decoder can determine one of several transform pairs to use when decoding blocks decoded in IBC and / or TMP modes. The techniques disclosed herein can improve decoding efficiency in Enhanced Compression Model (ECM) based on Multifunctional Video Decoding (VVC / H.266). The techniques described herein can be used in other advanced video codecs, including HEVC and extensions to next-generation video decoding standards.

[0006] In one example, this disclosure describes a method for decoding video data, the method comprising: receiving a block of video data encoded using IBC mode or intra-frame TMP mode; using an MTS procedure to determine a transform pair from a plurality of transform pairs for decoding the block; and using the transform pair to decode the block.

[0007] In another example, this disclosure describes an apparatus configured to decode video data, the apparatus comprising: a memory configured to receive a block of video data; and processing circuitry in communication with the memory, the processing circuitry being configured to: receive a block of video data encoded using an IBC mode or an intra-frame TMP mode; determine, using an MTS process, a plurality of transform pairs for decoding the block; and decode the block using the transform pair.

[0008] In another example, this disclosure describes a method for encoding video data, the method comprising: receiving a block of video data to be encoded using an IBC mode or an intra-frame TMP mode; using an MTS procedure to determine a transform pair from a plurality of transform pairs for decoding the block; and using the transform pair to encode the block.

[0009] In another example, this disclosure describes an apparatus configured to encode video data, the apparatus comprising: a memory configured to receive a block of video data; and processing circuitry in communication with the memory, the processing circuitry being configured to: receive a block of video data to be encoded using an IBC mode or an intra-frame TMP mode; determine, using an MTS process, a transform pair from a plurality of transform pairs for decoding the block; and encode the block using the transform pair.

[0010] Details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims. Attached Figure Description

[0011] Figure 1 This is a block diagram illustrating an exemplary video encoding and decoding system that can perform the techniques of this disclosure.

[0012] Figure 2 This is a conceptual diagram illustrating an example of IBC.

[0013] Figure 3 This is a conceptual diagram illustrating an example of template matching over the search region surrounding the initial motion vector.

[0014] Figure 4 This is a conceptual diagram of an exemplary intra-frame template matching search region.

[0015] Figure 5 This is a conceptual diagram illustrating an exemplary IBC reference region depending on the current CU location.

[0016] Figure 6 This is a block diagram illustrating an exemplary video encoder that can perform the techniques of this disclosure.

[0017] Figure 7 This is a block diagram illustrating an exemplary video decoder that can perform the techniques of this disclosure.

[0018] Figure 8 This is a flowchart illustrating an exemplary method for encoding the current block according to the technology of this disclosure.

[0019] Figure 9 This is a flowchart illustrating an exemplary method for decoding the current block according to the technology of this disclosure.

[0020] Figure 10 This is a flowchart illustrating another exemplary method for encoding the current block according to the technology of this disclosure.

[0021] Figure 11 This is a flowchart illustrating another exemplary method for decoding the current block according to the technology of this disclosure. Detailed Implementation

[0022] Transform decoding is a technique used in video decoding to compress video data. For example, a video encoder can use a prediction process (e.g., inter-frame prediction) to generate residual blocks of video data. The residuals can then be transformed from the spatial domain to the frequency domain using horizontal and vertical transforms (e.g., transform pairs). The application of such transform pairs produces transform coefficients that typically require fewer bits to represent than the residuals. These transform coefficients can then be quantized and entropy-decoded to further compress the data. In some examples, a Type 2 Discrete Cosine Transform (DCT-2) can be used in the video codec to perform both the forward and inverse transforms.

[0023] In addition to the DCT-2 transform already used in HEVC, the Multiple Transform Selection (MTS) scheme is also used for residual decoding of inter-frame decoding blocks in other exemplary codecs such as VVC. In one example, the inter-frame MTS mode involves using multiple transforms selected from multiple transform kernels, including a type 8 DCT (DCT-8) and a type 7 Discrete Sine Transform (DST-7). Both transform kernels can be applied to both vertical and horizontal transforms, corresponding to four different possible transform combinations.

[0024] In some examples, the use of inter-frame MTS is restricted to certain decoding modes. This disclosure describes techniques related to transform decoding, including MTS techniques for IBC mode and TMP. A video decoder can determine one of multiple transform pairs to use when decoding a block decoded in IBC and / or TMP modes. For example, a video decoder can receive a video data block encoded using IBC mode or intra-frame TMP mode, use the MTS process to determine the transform pair from multiple transform pairs used for decoding the block, and use that transform pair to decode the block.

[0025] The techniques disclosed herein can improve decoding efficiency in Enhanced Compression Model (ECM) based on Multifunctional Video Decoding (VVC / H.266). The techniques described herein can be used in other advanced video codecs, including HEVC and extensions to next-generation video decoding standards.

[0026] Figure 1 This is a block diagram illustrating an exemplary video encoding and decoding system 100 capable of performing the transformation decoding techniques of this disclosure. The techniques of this disclosure generally relate to decoding (encoding and / or decoding) video data. Typically, video data includes any data used for processing video. Therefore, video data can include raw, unencoded video, encoded video, decoded (e.g., reconstructed) video, and video metadata (such as signaling data).

[0027] like Figure 1As shown in the example, system 100 includes source device 102, which provides encoded video data to be decoded and displayed by target device 116. Specifically, source device 102 provides video data to target device 116 via computer-readable medium 110. Source device 102 and target device 116 can be or include any of a variety of devices, such as desktop computers, laptop computers, mobile devices, tablet computers, set-top boxes, mobile phones (such as smartphones), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, etc. In some cases, source device 102 and target device 116 may be equipped for wireless communication and may therefore be referred to as wireless communication devices.

[0028] exist Figure 1 In the example, source device 102 includes a video source 104, memory 106, video encoder 200, and output interface 108. Target device 116 includes an input interface 122, video decoder 300, memory 120, and display device 118. According to this disclosure, the video encoder 200 of source device 102 and the video decoder 300 of target device 116 can be configured to apply transform decoding technology. Therefore, source device 102 represents an example of a video encoding device, while target device 116 represents an example of a video decoding device. In other examples, the source device and target device may include other components or arrangements. For example, source device 102 may receive video data from an external video source, such as an external camera. Similarly, target device 116 may interface with an external display device, rather than including an integrated display device.

[0029] like Figure 1 The system 100 shown is merely an example. Generally, any digital video encoding and / or decoding device can perform transformation decoding techniques. The source device 102 and the target device 116 are simply examples of such decoding devices, where the source device 102 generates decoded video data to be sent to the target device 116. This disclosure refers to a “decoding” device as a device that performs decoding (encoding and / or decoding) on ​​data. Thus, video encoder 200 and video decoder 300 represent examples of decoding devices, and more specifically, examples of video encoders and video decoders, respectively. In some examples, the source device 102 and the target device 116 can operate in a substantially symmetrical manner, such that each of the source device 102 and the target device 116 includes video encoding and decoding components. Therefore, system 100 can support one-way or two-way video transmission between the source device 102 and the target device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0030] Typically, video source 104 represents a video data source (i.e., raw, unencoded video data) and provides a series of consecutive pictures (also called “frames”) of video data to video encoder 200, which encodes the data of the pictures. Video source 104 of source device 102 may include video capture devices such as cameras, video archives containing previously captured raw video, and / or video feed interfaces for receiving video from video content providers. Alternatively, video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes the captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange the pictures from the receiving order (sometimes referred to as “display order”) to the decoding order for decoding. Video encoder 200 may generate a bitstream comprising encoded video data. The source device 102 can then output the encoded video data to a computer-readable medium 110 via the output interface 108 for reception and / or retrieval by, for example, the input interface 122 of the target device 116.

[0031] The memory 106 of source device 102 and the memory 120 of target device 116 represent general-purpose memory. In some examples, memories 106 and 120 may store raw video data, such as raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106 and 120 may store software instructions that can be executed by, for example, video encoder 200 and video decoder 300. Although memories 106 and 120 are shown separately from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store, for example, encoded video data output from video encoder 200 and input to video decoder 300. In some examples, portions of memories 106 and 120 may be allocated as one or more video buffers to store, for example, raw, decoded, and / or encoded video data.

[0032] Computer-readable medium 110 can represent any type of medium or device capable of transmitting encoded video data from source device 102 to target device 116. In one example, computer-readable medium 110 represents a communication medium enabling source device 102 to transmit encoded video data directly to target device 116 in real time, for example, via a radio frequency network or a computer-based network. Depending on communication standards (such as wireless communication protocols), output interface 108 can modulate the transmitted signal including the encoded video data, and input interface 122 can demodulate the received transmitted signal. The communication medium can include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium can include a router, switch, base station, or any other device that may facilitate communication from source device 102 to target device 116.

[0033] In some examples, source device 102 can output encoded data from output interface 108 to storage device 112. Similarly, target device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0034] In some examples, source device 102 may output encoded video data to file server 114 or another intermediate storage device that may store the encoded video data generated by source device 102. Target device 116 may access the stored video data from file server 114 via streaming or download.

[0035] File server 114 can be any type of server device capable of storing encoded video data and sending such encoded video data to target device 116. File server 114 can represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or One-Way File Transfer Protocol (FLUTE)), a Content Delivery Network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and / or a Network Attached Storage (NAS) device. File server 114 may additionally or alternatively implement one or more HTTP streaming protocols (such as HTTP-based Dynamic Adaptive Streaming (DASH), HTTP Real-Time Streaming (HLS), Real-Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, etc.).

[0036] Target device 116 can access encoded video data from file server 114 via any standard data connection, including an internet connection. This can include wireless channels (e.g., Wi-Fi connections), wired connections (e.g., Digital Subscriber Line (DSL), cable modems, etc.), or combinations of both suitable for accessing encoded video data stored on file server 114. Input interface 122 can be configured to operate according to any or more of the various protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data.

[0037] Output interface 108 and input interface 122 can represent a wireless transmitter / receiver, a modem, a wired network connectivity component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 can be configured to transmit data (such as encoded video data) according to cellular communication standards such as 4G, 4G-LTE (Long Term Evolution), Advanced LTE, 5G, etc. In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 can be configured to operate according to other wireless standards (such as the IEEE 802.11 specification, the IEEE 802.15 specification, or, for example, ZigBee). TM Bluetooth TMThe system transmits data, such as encoded video data, using standards, etc. In some examples, source device 102 and / or target device 116 may include corresponding system-on-chip (SoC) devices. For example, source device 102 may include an SoC device to perform functionality associated with video encoder 200 and / or output interface 108, and target device 116 may include an SoC device to perform functionality associated with video decoder 300 and / or input interface 122.

[0038] The technology disclosed herein can be applied to video decoding that supports any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission, dynamic adaptive streaming (DASH) based on HTTP, digital video encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0039] The input interface 122 of the target device 116 receives an encoded video bitstream from a computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200 (which is also used by the video decoder 300), such as syntax elements having values ​​describing the characteristics and / or processing of video blocks or other decoded units (e.g., slices, pictures, picture groups, sequences, etc.). The display device 118 displays a decoded image of the decoded video data to the user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0040] although Figure 1 Not shown, but in some examples, the video encoder 200 and the video decoder 300 may each be integrated with the audio encoder and / or the audio decoder, and may include appropriate multiplexing-demultiplexing (MUX-DEMUX) units or other hardware and / or software to handle multiplexed streams that include both audio and video in a common data stream.

[0041] The video encoder 200 and video decoder 300 can each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is implemented in part in software, the device may store instructions for software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the technology of this disclosure. Each of the video encoder 200 and video decoder 300 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder / decoder (CODEC) in the respective device. Devices including the video encoder 200 and / or video decoder 300 may implement the video encoder 200 and / or video decoder 300 in processing circuitry such as integrated circuits and / or microprocessors. Such devices may be wireless communication devices, such as cellular phones, or any other type of device described herein.

[0042] The video encoder 200 and video decoder 300 may operate according to a video decoding standard such as ITU-T H.265, also known as High Efficiency Video Decoding (HEVC) or its extensions, such as Multi-View and / or Scalable Video Decoding Extensions. Alternatively, the video encoder 200 and video decoder 300 may operate according to other proprietary or industry standards, such as ITU-T H.266, also known as Multi-Functional Video Decoding (VVC). In other examples, the video encoder 200 and video decoder 300 may operate according to proprietary video codecs / formats such as AOMedia Video 1 (AV1), extensions to AV1, and / or successors to AV1 (e.g., AV2). In other examples, the video encoder 200 and video decoder 300 may operate according to other proprietary formats or industry standards. However, the techniques disclosed herein are not limited to any particular decoding standard or format. Typically, the video encoder 200 and video decoder 300 may be configured to perform the techniques of this disclosure in combination with any video decoding technique using multiple transform cores.

[0043] Typically, video encoder 200 and video decoder 300 can perform block-based decoding of images. The term "block" generally refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used in the encoding and / or decoding process). For example, a block may include a two-dimensional matrix of illuminance and / or chrominance data samples. Typically, video encoder 200 and video decoder 300 can decode video data represented in YUV (e.g., Y, Cb, Cr) format. That is, video encoder 200 and video decoder 300 can decode illuminance and chrominance components, rather than decoding the red, green, and blue (RGB) data of a sample of an image, where the chrominance components may include both red and blue chrominance components. In some examples, video encoder 200 converts the received RGB formatted data to a YUV representation before encoding, and video decoder 300 converts the YUV representation to RGB format. Alternatively, preprocessing and postprocessing units (not shown in the figures) may perform these conversions.

[0044] This disclosure can generally relate to decoding (e.g., encoding and decoding) an image to include the process of encoding or decoding the data of the image. Similarly, this disclosure can relate to decoding blocks of an image to include the process of encoding or decoding the data of the block, such as prediction and / or residual decoding. Encoded video bitstreams typically include a series of values ​​representing decoding decisions (e.g., decoding modes) and syntax elements that divide the image into blocks. Therefore, references to decoding an image or block should generally be understood as the decoded values ​​of the syntax elements used to form the image or block.

[0045] HEVC defines various blocks, including decoding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video decoder (such as a video encoder 200) partitions the decoding tree unit (CTU) into CUs according to a quadtree structure. That is, the video decoder partitions the CTU and CU into four identical non-overlapping squares, and each node of the quadtree has zero or four child nodes. Nodes without child nodes are called "leaf nodes," and the CU of such leaf nodes can include one or more PUs and / or one or more TUs. The video decoder can further partition the PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents a partition of the TU. In HEVC, the PU represents inter-frame prediction data, while the TU represents residual data. CUs with intra-frame prediction include intra-frame prediction information, such as intra-frame mode indication.

[0046] As another example, video encoder 200 and video decoder 300 can be configured to operate according to VVC. According to VVC, the video decoder (such as video encoder 200) segments the image into multiple CTUs. Video encoder 200 can segment CTUs according to a tree structure, such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure removes the concept of multiple segmentation types, such as the CU, PU of HEVC, and the spacing between TUs. The QTBT structure includes two levels: a first level for segmentation based on quadtree segmentation, and a second level for segmentation based on binary tree segmentation. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to the CU.

[0047] In the MTT partitioning structure, blocks can be partitioned using quadtree (QT) partitioning, binary tree (BT) partitioning, and one or more types of ternary tree (TT) partitioning (also known as tripartite (TT)) partitioning. A ternary or tripartite partition is a partition in which a block is divided into three sub-blocks. In some examples, a ternary or tripartite partition divides a block into three sub-blocks without dividing the original block through a center. The partitioning types in MTT (e.g., QT, BT, and TT) can be symmetric or asymmetric.

[0048] When operating according to the AV1 codec, the video encoder 200 and video decoder 300 can be configured to decode video data in blocks. In AV1, the largest decoded block that can be processed is called a superblock. In AV1, a superblock can be 128×128 luminance samples or 64×64 luminance samples. However, in subsequent video decoding formats (e.g., AV2), superblocks can be defined by different (e.g., larger) luminance sample sizes. In some examples, the superblock is the top level of a block quadtree. The video encoder 200 can also divide the superblock into smaller decoded blocks. The video encoder 200 can divide the superblock and other decoded blocks into smaller blocks using square or non-square partitioning. Non-square blocks can include N / 2×N, N×N / 2, N / 4×N, and N×N / 4 blocks. The video encoder 200 and video decoder 300 can perform separate prediction and transform processes for each of the decoded blocks.

[0049] AV1 also defines video data tiles. A tile is a rectangular array of superblocks that can be decoded independently of other tiles. That is, the video encoder 200 and the video decoder 300 can encode and decode the decoding blocks within a tile separately, without using video data from other tiles. However, the video encoder 200 and the video decoder 300 can perform filtering across tile boundaries. The size of the tiles can be uniform or non-uniform. Tile-based decoding enables parallel processing and / or multithreading of the encoder and decoder implementations.

[0050] In some examples, the video encoder 200 and the video decoder 300 may use a single QTBT or MTT structure to represent each of the illuminance and chromaticity components, while in other examples, the video encoder 200 and the video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the illuminance component and another QTBT / MTT structure for the two chromaticity components (or two QTBT / MTT structures for the respective chromaticity components).

[0051] The video encoder 200 and the video decoder 300 can be configured to use quadtree segmentation, QTBT segmentation, MTT segmentation, superblock segmentation or other segmentation structures.

[0052] In some examples, a CTU includes a decoded tree block (CTB) of luminance samples, two corresponding CTBs of chrominance samples of an image with three sample arrays, or a CTB of samples of a monochrome image or an image decoded using three independent color planes and a syntax structure for decoding the samples. A CTB can be an N×N block of samples for some value of N, such that dividing the components into CTBs is a segmentation. A component is an array or a single sample from one of three arrays (luminance and two chrominance), which together form an image in a 4:2:0, 4:2:2, or 4:4:4 color format; or an array or a single sample of an array that makes up a monochrome image. In some examples, a decoded block is an M×N block of samples for some values ​​of M and N, such that dividing the CTB into decoded blocks is a segmentation.

[0053] Blocks (e.g., CTUs or CUs) can be grouped in various ways within an image. As an example, a brick can refer to a rectangular area of ​​a row of CTUs within a specific tile in an image. A tile can be a rectangular area of ​​a CTU within a specific tile column and a specific tile row in an image. A tile column refers to a rectangular area of ​​a CTU with a height equal to the height of the image and a width specified by a syntax element (e.g., a width as in the image parameter set). A tile row refers to a rectangular area of ​​a CTU with a height specified by a syntax element (e.g., a height as in the image parameter set) and a width equal to the width of the image.

[0054] In some examples, a tile can be divided into multiple bricks, each of which may include one or more CTU rows within the tile. A tile that is not divided into multiple bricks can also be called a brick. However, bricks that are a true subset of a tile cannot be called a tile. Bricks in an image can also be arranged as slices. A slice can be an integer number of bricks in the image, and they can be proprietaryly contained in a single Network Abstraction Layer (NAL) unit. In some examples, a slice consists of a continuous sequence of consecutive bricks from multiple complete tiles or a single tile.

[0055] This disclosure uses “N×N” and “N multiplied by N” interchangeably to refer to the sample size of a block (such as a CU or other video block) with respect to vertical and horizontal dimensions, for example, 16×16 samples or 16 multiplied by 16 samples. Typically, a 16×16 CU will have 16 samples in the vertical direction (y=16) and 16 samples in the horizontal direction (x=16). Similarly, an N × N CU typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU may be arranged in columns and rows. Furthermore, a CU does not necessarily have the same number of samples in the horizontal and vertical directions. For example, a CU may include N×M samples, where M does not necessarily equal N.

[0056] The video encoder 200 encodes video data containing representation prediction and / or residual information, as well as other information, of the CU. Prediction information indicates how the CU will be predicted to form a prediction block of the CU. Residual information typically represents the sample-by-sample difference between the CU and the prediction block before encoding.

[0057] To predict a CU, the video encoder 200 can typically form a predicted block of the CU via inter-frame prediction or intra-frame prediction. Inter-frame prediction generally refers to predicting the CU from data of a previously decoded image, while intra-frame prediction generally refers to predicting the CU from previously decoded data of the same image. To perform inter-frame prediction, the video encoder 200 can use one or more motion vectors to generate the predicted block. The video encoder 200 can typically perform a motion search to identify, for example, a reference block that closely matches the CU in terms of the difference between the CU and a reference block. The video encoder 200 can use sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculations to compute a difference metric to determine whether the reference block closely matches the current CU. In some examples, the video encoder 200 can use unidirectional or bidirectional prediction to predict the current CU.

[0058] Some examples of VVC also provide affine motion compensation modes, which can be viewed as inter-frame prediction modes. In affine motion compensation mode, the video encoder 200 can determine two or more motion vectors representing non-translational motion (such as zooming in or out, rotation, perspective motion, or other irregular motion types).

[0059] To perform intra-frame prediction, the video encoder 200 can select an intra-frame prediction mode to generate prediction blocks. Some examples of VVC provide sixty-seven intra-frame prediction modes, including various directional modes as well as planar and DC modes. Typically, the video encoder 200 selects an intra-frame prediction mode from which samples of the current block are predicted, describing samples adjacent to the current block (e.g., a block of a CU). Assuming the video encoder 200 decodes the CTU and CU in raster scan order (left to right, top to bottom), such samples can typically be located above, to the upper left, or to the left of the current block within the same frame.

[0060] The video encoder 200 encodes data representing the prediction mode of the current block. For example, for inter-frame prediction modes, the video encoder 200 may encode data indicating which of the various available inter-frame prediction modes is used and the motion information for the corresponding mode. For example, for unidirectional or bidirectional inter-frame prediction, the video encoder 200 may use Advanced Motion Vector Prediction (AMVP) or merging modes to encode motion vectors. The video encoder 200 may use similar modes to encode motion vectors for affine motion compensation modes.

[0061] AV1 includes two common techniques for encoding and decoding blocks of video data. These two common techniques are intra-frame prediction (e.g., intra-frame prediction or spatial prediction) and inter-frame prediction (e.g., inter-frame prediction or temporal prediction). In the context of AV1, when using intra-frame prediction modes to predict blocks of video data for the current frame, the video encoder 200 and video decoder 300 do not use video data from other frames of the video data. For most intra-frame prediction modes, the video encoder 200 encodes blocks of the current frame based on the difference between sample values ​​in the current block and predicted values ​​generated from reference samples in the same frame. The video encoder 200 determines the predicted values ​​generated from the reference samples based on the intra-frame prediction mode.

[0062] After block prediction (such as intra-frame prediction or inter-frame prediction), the video encoder 200 can compute residual data for the block. Residual data (such as residual blocks) represents the sample-by-sample difference between the block and the predicted block formed using the corresponding prediction mode. The video encoder 200 can apply one or more transforms to the residual blocks to produce transformed data in the transform domain rather than the sample domain. For example, the video encoder 200 can apply Discrete Cosine Transform (DCT), integer transform, wavelet transform, or conceptually similar transforms to the residual video data. Additionally, the video encoder 200 can apply secondary transforms (such as Mode Correlated Inseparable Secondary Transform (MDNSST), Signal Correlation Transform, Karhunen-Loeve Transform (KLT), etc.) after a primary transform. The video encoder 200 produces transform coefficients after applying one or more transforms.

[0063] As noted above, after any transform that produces the transform coefficients, the video encoder 200 can perform quantization of the transform coefficients. Quantization generally refers to the process of quantizing transform coefficients to potentially reduce the amount of data used to represent them, thereby providing further compression. By performing the quantization process, the video encoder 200 can reduce the bit depth associated with some or all of the transform coefficients. For example, the video encoder 200 can round an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder 200 can perform a bit-wise right shift of the value to be quantized.

[0064] After quantization, the video encoder 200 can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix including the quantized transform coefficients. The scan can be designed to place higher-energy (and therefore lower-frequency) transform coefficients at the beginning of the vector and lower-energy (and therefore higher-frequency) transform coefficients at the end. In some examples, the video encoder 200 can utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector, and then entropy-encode the quantized transform coefficients of the vector. In other examples, the video encoder 200 can perform an adaptive scan. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 200 can entropy-encode the one-dimensional vector, for example, according to context-adaptive binary arithmetic decoding (CABAC). The video encoder 200 can also entropy-encode the values ​​of syntax elements describing metadata associated with the encoded video data for use by the video decoder 300 to decode the video data.

[0065] To perform CABAC, the video encoder 200 can assign context within a context model to the symbols to be transmitted. Context may involve, for example, whether the neighboring values ​​of a symbol are zero. Probability determination can be based on the context assigned to the symbols.

[0066] The video encoder 200 may also, for example, generate syntax data for the video decoder 300 in image headers, block headers, slice headers, or other syntax data (such as sequence parameter sets (SPS), picture parameter sets (PPS), or video parameter sets (VPS)). This syntax data may include block-based syntax data, image-based syntax data, and sequence-based syntax data. The video decoder 300 may similarly decode such syntax data to determine how to decode the corresponding video data.

[0067] In this way, the video encoder 200 can generate a bitstream that includes encoded video data (e.g., syntax elements describing the segmentation of images to blocks (e.g., CUs) and prediction and / or residual information for the blocks). Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0068] Typically, the video decoder 300 performs the inverse of the process performed by the video encoder 200 to decode the encoded video data of the bitstream. For example, the video decoder 300 can use CABAC to decode the values ​​of the syntax elements of the bitstream in a manner substantially similar to but inverse of the CABAC encoding process of the video encoder 200. Syntax elements can define segmentation information for dividing the image into CTUs and the segmentation of each CTU according to a corresponding segmentation structure (such as a QTBT structure) to define the CUs of the CTUs. Syntax elements can also define prediction and residual information for video data blocks (e.g., CUs).

[0069] The residual information can be represented, for example, by quantized transform coefficients. The video decoder 300 can inverse-quantize and inverse-transform the quantized transform coefficients of the block to reconstruct the residual block of the block. The video decoder 300 uses a signaled prediction mode (intra-frame or inter-frame prediction) and associated prediction information (e.g., motion information for inter-frame prediction) to form a predicted block of the block. The video decoder 300 can then (sample-by-sample) combine the predicted block with the residual block to reconstruct the initial block. The video decoder 300 can perform additional processing (such as performing a deblocking procedure) to reduce visual artifacts along block boundaries.

[0070] This disclosure can generally refer to "signaling" certain information (such as syntax elements). The term "signaling" can generally refer to transmitting values ​​of syntax elements and / or other data used for decoding encoded video data. That is, the video encoder 200 can signal the values ​​of syntax elements in the bitstream. Generally, signaling refers to generating values ​​in the bitstream. As mentioned above, the source device 102 can transmit the bitstream to the target device 116 substantially in real time, or not in real time, such as when syntax elements are stored in the storage device 112 for later retrieval by the target device 116.

[0071] Generally, this disclosure describes techniques for video decoding. Specifically, the techniques described herein relate to transform decoding. This disclosure describes a Multiple Transform Selection (MTS) technique for Intra-Block Copy (IBC) mode and an MTS scheme for Intra-Template Match Prediction (TMP) mode. The techniques disclosed herein can improve decoding efficiency in Enhanced Compression Models (ECM) based on Multifunctional Video Decoding (VVC / H.266). The techniques described herein can be used in other advanced video codecs, including HEVC and extensions to next-generation video decoding standards.

[0072] As will be explained in more detail below, the video decoder 300 can be configured to: receive a block of video data encoded using IBC mode or intra-frame TMP mode; use the MTS process to determine a transform pair from a plurality of transform pairs for decoding the block; and use the transform pair to decode the block.

[0073] Conversely, the video encoder 200 can be configured to: receive a block of video data to be encoded using IBC mode or intra-frame TMP mode; use an MTS process to determine a transform pair from a plurality of transform pairs for decoding the block; and use the transform pair to encode the block.

[0074] Overview of Transformation-Related Tools

[0075] Prior to HEVC, exemplary video decoding standards used only fixed separable transforms, with DCT-2 used both vertically and horizontally. In HEVC, in addition to DCT-2, DST-7 (Type-7 DCT) is also used as a fixed separable transform for 4×4 blocks. Depending on the block type (e.g., INTRA (intra-frame), INTER (inter-frame), or IBC), different transform combinations can be considered. One technique for considering several transforms for a block is called Multiple Transform Selection (MTS). In MTS, the video encoder 200 can test multiple different transform pairs and select one by signaling concurrently. Note that MTS was previously called Adaptive Multiple Transform (AMT). The video encoder 200 indicates the transform pair to use by signaling an index (hereinafter referred to as MTSIDx), which is an integer value.

[0076] MTSIdx signaling

[0077] In one version of ECM, MTSIdx values ​​of 0 and 1 were considered special cases. For example, MTSIdx=0 indicated that a DCT-2 transform should be used (e.g., both horizontal and vertical DCT-2 transforms), while MTSIdx=1 indicated that the transform was skipped. That is, the block was decoded without any transform.

[0078] First, the video decoder 300 decodes the transform skip flag (trSkip), which indicates whether a transform is applied (e.g., trSkip = true) or not (e.g., trSkip = false). If trSkip = false, the video decoder 300 further decodes the mts_flag, which indicates whether the MTS transform is used (e.g., mts_flag = true) or not (e.g., mts_flag = false). If mts_flag = false, the video decoder 300 uses the DCT-2 transform. If mts_flag = true, the video decoder 300 decodes MTSIDx into truncated binary code. The application of the truncated binary code uses knowledge of the maximum possible number of candidate transform pairs (denoted here as nCands). The value of nCands can be adaptively chosen (e.g., for intra-frame blocks, including TMP), or it can be constant (e.g., for inter-frame blocks).

[0079] In one example of ECM, inter-frame blocks and intra-frame blocks use the same MTSIDx signaling, where the index refers to different transform combinations for inter-frame blocks (“interMTS”) and intra-frame blocks (“intraMTS”). For inter-frame blocks, the value of nCands is set to 4. For intra-frame blocks, the value of nCands is adaptively set to 1, 4, or 6 (“adaptive intraMTS”). IBC blocks only support DCT-2 and transform skipping (i.e., in the current ECM, IBCs do not use MTS for decoding).

[0080] Symbolic prediction

[0081] The sign of the transform coefficients can be predicted as part of the sign prediction algorithm. In an exemplary ECM, the prediction region is adaptively expanded to 32×32, and the number of predicted signs is configurable.

[0082] IBC mode

[0083] Figure 2 This is a conceptual diagram illustrating an example of Intra-Block Copy (IBC) mode. IBC mode is one of the decoding tools for screen content. For the current decoding unit (CU) 260, a video codec using IBC (such as video encoder 200 or video decoder 300) will search multiple block vectors and find the best matching block in the reference region for prediction, such as... Figure 2As shown. Then, a prediction for the selected block vector is generated, and the difference between the selected block vector and the predicted block vector (called the block vector difference (BVD)) is signaled in the bitstream. For example, the video encoder 200 can generate the selected block vector and the BVD and signal the BVD in the bitstream to the video decoder 300.

[0084] Template matching (TM) is a decoder-side MV derivation technique used to refine the motion information of a current CU by finding the closest match between a template in the current image (e.g., the top and / or left adjacent blocks of the current CU, such as the upper current template 262A and the left current template 262B) and a template in a reference image (e.g., reference block 270) (e.g., the upper reference template 272A and / or the left reference template 272B). For example, the closest match could be the closest size match. TM can be applied to both AMVP and regular merge modes, referred to as TM-AMVP and TM-MRG modes, respectively. Similarly, the same TM technique can be applied to IBC AMVP and IBC merge modes. The only difference between inter-frame TM and IBC TM is that the reference image for IBC TM is the reconstructed region (decoded region 252) in the current frame (current frame 250). Therefore, the reference block of the current CU 260 with IBC TM can be a reference block within the decoded region 252 (instead of the non-decoded region 254), such as reference block 270. Without loss of generality, this disclosure generally uses "MV" or "motion" to indicate that the current CU is an inter-frame block and has motion information or that the current CU is an IBC block and has block vector information.

[0085] Figure 3 This is a conceptual diagram illustrating an example of template matching over a search region surrounding the initial motion vector. (Example:) Figure 3As shown, within the [–8, +8] pixel search range, a better MV than the initial MV 390 is searched around the current CU's initial MV 390. For example, the video encoder 200 or the video decoder 300 can perform such a search. The template matching technique in the "Description of SDR, HDR and 360° video coding technology proposal by Qualcomm and Technicolor – low and high complexity versions" presented by Chen et al. at the 10th meeting of the Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, San Diego, April 10-20, 2018 (JVET-J0021) can be used with the following modifications: the search step size can be determined based on the AMVR mode, and the TM can be cascaded with the bilateral matching process in the merge mode.

[0086] The video encoder 200 or video decoder 300 can operate in TM-AMVP mode. In TM-AMVP mode, MVP candidates are determined based on template matching error to select the MVP candidate with the smallest difference between the current block template and the reference block template. TM can then be performed only for that specific MVP candidate to facilitate MV refinement. TM can refine the MVP candidate by using an iterative diamond search starting with full-pixel MVD precision (or 4 pixels in 4-pixel AMVR mode) within a search range of [–8, +8] pixels. The AMVP candidate can be further refined by using a cross search with full-pixel MVD precision (or 4 pixels in 4-pixel AMVR mode), followed by sequential half-pixel and quarter-pixel cross searches according to the AMVR mode, as shown in Table 1 below. This search process ensures that the MVP candidate retains the same MV precision as indicated by the AMVR mode after the TM process. During the search process, if the difference between the previous minimum cost and the current minimum cost in a single iteration is less than or equal to a threshold of the block area, the search process can terminate.

[0087] Table 1 – Search Patterns for AMVR and Merge Patterns with AMVR

[0088]

[0089] In TM-MRG merging mode, a similar search technique is applied to the merging candidates indicated by the merging index. Video encoder 200 or video decoder 300 can operate in TM-MRG merging mode. As shown in Table 1, TM can be performed up to 1 / 8 pixel MVD accuracy, or skip accuracies exceeding half-pixel MVD accuracy, depending on whether an alternative interpolation filter (used when AMVR is in half-pixel mode) is used based on the merging motion information. Furthermore, when TM mode is enabled, template matching can operate as a separate process between block-based and sub-block-based bilateral matching (BM) methods, or as an additional MV refinement process, depending on whether BM is enabled according to its enable condition check.

[0090] TMP mode

[0091] Intra-Template Matching Prediction (IntraTMP) mode is a special intra-frame prediction mode in which the video encoder 200 or video decoder 300 copies the best prediction block from the reconstructed portion of the current frame that matches the current template. For a predefined search range, the video encoder 200 searches for the template most similar to the current template in the reconstructed portion of the current frame and uses the corresponding block as the prediction block. The video encoder 200 then signals the use of this mode, and the video decoder 300 performs the same prediction operation.

[0092] Figure 4 This is a conceptual diagram of an exemplary intra-frame template matching search region. A video encoder 200 or a video decoder 300 can generate a prediction signal. This is achieved by matching the L-shaped causal neighboring blocks of the current block 400 with... Figure 4 The prediction signal is generated by matching another block in a predefined search region, which includes R1 402 (current CTU), R2 404 (top left CTU), R3 406 (top CTU), and R4 408 (left CTU). SAD can be used as a cost function. Within each region (R1 402, R2 404, R3 406, and R4 408), for example, the video decoder 300 searches for the template with the minimum SAD relative to the template of the current block 400, and uses the corresponding block of the template with the minimum SAD relative to the template of the current block 400 as the prediction block. For example, the video decoder 300 can determine that the matching block 410 is the prediction block of the current block 400.

[0093] The dimensions of all regions (SearchRange_w, SearchRange_h) can be set to be proportional to the block dimensions (BlkW, BlkH) so that each pixel has a fixed number of SAD comparisons. That is:

[0094] SearchRange_w = 5 * BlkW

[0095] SearchRange_h = 5 * BlkH

[0096] For CUs with a width and height of 64 or less, the intra-template matching tool can be enabled. The maximum CU size used for intra-template matching can be configurable. When the decoder-side intra-mode derivation (DIMD) is not used for the current CU, the intra-template matching prediction mode is signaled at the CU level via a dedicated flag.

[0097] Template matching can be used in both IBC merge mode and IBC AMVP mode in IBC, which are referred to as IBC-TM-AMVP and IBC-TM-MRG, respectively. Video encoder 200 or video decoder 300 can use these modes.

[0098] In IBC-TM-MRG, the merge list is modified compared to that used in the regular IBC merge mode, so that, like in the regular TM merge mode, candidates are selected based on a pruning method with motion distances between them. The end-zero motion is replaced by motion vectors to the left (-W, 0), up (0, -H), and top-left (-W, -H), where W is the width of the current CU and H is the height of the current CU. Additionally, template matching is used to refine the selected candidates before RDO or the decoding process. The IBC-TM-MRG mode competes with the regular IBC merge mode and signals the TM merge flag.

[0099] In the IBC-TM-AMVP mode, a maximum of three candidates are selected from the IBC-TM-MRG merge list. Each of these three selected candidates is refined using template matching and ranked according to the template matching cost generated. Then, only the top two candidates are considered during motion estimation.

[0100] Figure 5 This is a conceptual diagram illustrating an exemplary IBC reference region depending on the current CU position. Template matching refinement for both IBC-TM merging and AMVP modes is quite straightforward because the IBC motion vector is constrained to be (i) an integer and (ii) within the reference region, as... Figure 5 As shown. Specifically, in Figure 5 In the examples, each small square corresponds to a CU. A group of four CUs with dark borders may correspond to a CTU. In some examples, each CTU can be 128×128 in size, while each CU can be 64×64 in size. Furthermore, Figure 5The diagram shows the current CU 500 at different locations within the CTU. The shaded CUs represent the causal relationship of the current CU 500. Depending on the location of the current CU 500, different CUs are available for use in the IBC. Figure 5 In this context, CUs marked with "X" cannot be used with IBCs.

[0101] Example

[0102] This describes techniques for using MTS in IBC blocks, and new techniques for using MTS in intra-frame blocks using TMP mode (intra-frame TMP). Decoding efficiency is improved by adding additional transform combinations selected for IBC and / or TMP decoding blocks. In some examples, the MTS techniques disclosed herein can improve the decoding efficiency of natural content video data when decoding it using IBC or TMP modes.

[0103] Natural content video data refers to video data that captures real-world scenes, including real objects, people, landscapes, and animals. A typical characteristic of this type of video data is its organic, unmanipulated nature, including realistic textures, continuous motion, dynamic color gamut, varying lighting conditions, and depth and perspective. In contrast, screen content video data primarily refers to content generated by a computer screen or electronic display, including text, graphics, and animation. Screen content typically features high contrast, sharp edges, uniform colors, and often includes static images or text, which differs from the smoother and more textured appearance of natural content.

[0104] MTSIdx decoding process of IBC block

[0105] The video decoder 300 can first decode the transform skip flag of the block being decoded. If the transform skip flag is true (e.g., equal to 1), the video decoder 300 can apply transform skipping to the block. If the transform skip flag is false (e.g., equal to 0), the video decoder 300 uses the context to decode the first bin of MTSIdx. The first bin of MTSIdx indicates whether DCT-2 is used. If the first bin is true, the video decoder 300 determines the separable DCT-2 (vertical) / DCT-2 (horizontal) of the IBC decoded block. If the first bin is false, the video decoder 300 can determine that the MTS core (e.g., transform pair) is derived by decoding the remaining two bins of MTSIdx, assuming nCands=4 and equal probability (e.g., truncated unary decoding). The remaining two frequency modules of the decoded MTSIdx={2,3,4,5} (0=DCT, 1=TrSkip) refer to a set of transform combinations used for inter-block (e.g., interMTS).

[0106] If the width and height of the transform unit block are both greater than 16:

[0107] i. MTSIdx=2: DST7 / DST7

[0108] ii. MTSIdx=3: DCT8 / DST7

[0109] iii. MTSIdx=4: DST7 / DCT8

[0110] iv. MTSIdx=5: DCT8 / DCT8

[0111] If the width and height of the transform unit block are both less than or equal to 16:

[0112] i. MTSIdx=2: KLT0 / KLT0

[0113] ii. MTSIdx=3: KLT1 / KLT0

[0114] iii. MTSIdx=4: KLT0 / KLT1

[0115] iv. MTSIdx=5:KLT1 / KLT1.

[0116] Restrictions on certain IBC modes

[0117] In one example, the MTSIdx decoding process described above can be used for certain IBC modes.

[0118] Case 1: MTS is used only for IBC blocks encoded using AMVP mode. In this case, the condition must be met: for a given IBC decoded block, the merge flag is set to false.

[0119] Case 2: MTS is also used for IBC blocks using AMVP and merge mode.

[0120] MTS for intra-frame blocks with TMP mode

[0121] For intra-coded blocks using TMP mode, the transform selection is aligned with the IBC mode, as described above. Therefore, for intra-TMP mode, an interMTS with nCands=4 is used instead of an adaptive intra-MTS.

[0122] Extend

[0123] The MTSIdx decoding process and the selected transformations mentioned above can be extended, while using TMP mode to maintain alignment between IBC decoded blocks and intra-frame decoded blocks.

[0124] Case 1: Instead of using a constant number of candidate transform pairs (where nCands=4), the maximum value of MTSIDx is adaptively derived, for example, 1, 2, or 4. For example, the number of candidate transform pairs for MTS can be based on the sum of coefficients in the transform block compared to a threshold. The selection of transform pairs is accomplished through the interMTS kernel described above. If it is determined that fewer than 4 transform pairs should be used, the video encoder 200 and video decoder 300 can truncate the list by discarding later transform pairs shown above.

[0125] Scenario 2: Instead of using the interMTS core, decoder-side intra-mode derivation (DIMD) is introduced for the IBC decoded block to derive the intra-mode index. In this case, the intraMTS core can be selected from:

[0126] a. Same predefined options as intra-frame decoding blocks using intraMTS

[0127] b. A separate set of options, which differ from the options previously used for intra-frame decoding blocks.

[0128] Case 3: Instead of using truncated unary codes to decode the candidate number or MTSIdx frequency blocks with equal probability, a dedicated context is used to perform CABAC decoding on the two frequency blocks separately.

[0129] Therefore, in light of the above, the video decoder 300 can be configured to receive video data blocks encoded using either Intra-Block Copy (IBC) mode or Intra-Template Match Prediction (TMP) mode. The video decoder 300 can use a Multiple Transform Selection (MTS) process to determine which transform pair from a plurality of transform pairs is used to decode the block, and decode the block using the transform pair. In one example, the video data block is encoded using IBC mode, and the IBC mode is AMVP IBC mode. In another example, the video data block is encoded using IBC mode, and the IBC mode is either AMVP IBC mode or merged IBC mode. In either of the above examples, the video data is natural content video data.

[0130] To determine the transform pairs among multiple transform pairs, the video decoder 300 is also configured to receive the MTS index and determine the transform pairs based on the MTS index (e.g., as described above). In another example, to determine the transform pairs among multiple transform pairs, the video decoder 300 is also configured to receive the MTS index and determine the transform pairs based on the MTS index and the block size.

[0131] The multiple transform pairs may include a vertical 2-Discrete Cosine Transform (DCT-2), a horizontal DCT-2, and four other transform pairs. In other examples, the number of other transform pairs may be variable, and the video decoder 300 may determine the number of multiple transform pairs. To determine the number of multiple transform pairs, the video decoder 300 may be configured to use context-adaptive binary arithmetic decoding and a dedicated context to decode the syntax element indicating the number of multiple transform pairs. In another example, to determine the number of multiple transform pairs, the video decoder 300 may be configured to determine the intra-mode of the block based on a decoder-side intra-mode derivation (DIMD) process, and determine the number of multiple transform pairs based on the intra-mode.

[0132] The video encoder 200 can perform operations opposite to those of the video decoder 300. For example, the video encoder 200 can receive a video data block to be encoded using IBC mode or intra-frame TMP mode, use the MTS process to determine the transform pair among multiple transform pairs for decoding the block, and encode the block using that transform pair. The video encoder 200 can encode the MTS index indicating the transform pair.

[0133] Figure 6 This is a block diagram illustrating an exemplary video encoder 200 that can perform the techniques disclosed herein. Figure 6 This disclosure is provided for illustrative purposes and should not be construed as a limitation on the techniques extensively illustrated and described herein. For illustrative purposes, this disclosure describes the video encoder 200 in accordance with the techniques of VVC and HEVC. However, the techniques of this disclosure can be implemented by video encoding devices configured for other video decoding standards and video decoding formats, such as AV1 and its successors.

[0134] exist Figure 6In the example, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy encoding unit 220. Any or all of the video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, DPB 218, and entropy encoding unit 220 can be implemented in one or more processors or processing circuits. For example, units of the video encoder 200 can be implemented as one or more circuit or logic elements, as part of hardware circuitry or as part of a processor, ASIC, or FPGA. Furthermore, the video encoder 200 may include additional or alternative processors or processing circuits to perform these and other functions.

[0135] The video data storage device 230 can store video data encoded by components of the video encoder 200. The video encoder 200 can retrieve data from, for example, a video source 104 (…). Figure 1 The video encoder 200 receives video data stored in video data memory 230. DPB 218 can act as a reference picture memory, storing reference video data for prediction of subsequent video data by the video encoder 200. Video data memory 230 and DPB 218 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 230 and DPB 218 can be provided by the same memory device or separate memory devices. In various examples, video data memory 230 can be on-chip with other components of the video encoder 200, as shown, or off-chip relative to those components.

[0136] In this disclosure, reference to video data memory 230 should not be construed as limited to memory within video encoder 200 (unless specifically described as such) or memory outside video encoder 200 (unless specifically described as such). Rather, reference to video data memory 230 should be understood as a reference memory storing video data received by video encoder 200 for encoding (e.g., video data of the current block to be encoded). Figure 1 The memory 106 can also provide temporary storage for the outputs from various units of the video encoder 200.

[0137] Figure 6Various units are shown to aid in understanding the operations performed by the video encoder 200. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. A fixed-function circuit is a circuit that provides specific functionality and predefines the operable operations. A programmable circuit is a circuit that can be programmed to perform various tasks and provide flexible functionality in the operable operations. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by instructions from software or firmware. A fixed-function circuit can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is typically immutable. In some examples, one or more of the units can be different circuit blocks (fixed-function or programmable), and in some examples, one or more of the units can be integrated circuits.

[0138] The video encoder 200 may include an arithmetic logic unit (ALU), an essential function unit (EFU), digital circuitry, analog circuitry, and / or a programmable core, all formed by programmable circuitry. In an example where the operation of the video encoder 200 is performed using software executed by programmable circuitry, memory 106 ( Figure 1 The video encoder 200 may store instructions (e.g., object code) of the software received and executed by the video encoder 200, or another memory (not shown) within the video encoder 200 may store such instructions.

[0139] The video data storage unit 230 is configured to store received video data. The video encoder 200 can retrieve images of the video data from the video data storage unit 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data storage unit 230 can be the raw video data to be encoded.

[0140] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra-frame prediction unit 226. The mode selection unit 202 may include additional functional units to perform video prediction based on other prediction modes. As an example, the mode selection unit 202 may include a palette unit, an intra-block copying unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, etc.

[0141] The mode selection unit 202 typically coordinates multiple coding passes to test combinations of coding parameters and the resulting rate-distortion values ​​for such combinations. Coding parameters may include the CTU-CU split, the prediction mode for the CU, the transformation type of the residual data for the CU, the quantization parameters of the residual data for the CU, and so on. The mode selection unit 202 can ultimately select a combination of coding parameters that has a better rate-distortion value than other tested combinations.

[0142] The video encoder 200 can segment images retrieved from the video data storage 230 into a series of CTUs and encapsulate one or more CTUs within a slice. The mode selection unit 202 can segment the CTUs of the image according to a tree structure such as the MTT structure, QTBT structure, superblock structure, or quadtree structure described above. As described above, the video encoder 200 can form one or more CUs from segmenting CTUs according to a tree structure. This CU can also generally be referred to as a "video block" or "block".

[0143] Typically, mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-frame prediction unit 226) to generate predicted blocks for the current block (e.g., the current CU, or the overlapping portion of PU and TU in HEVC). For inter-frame prediction of the current block, motion estimation unit 222 may perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previously decoded pictures stored in DPB 218). Specifically, motion estimation unit 222 may calculate values ​​representing the similarity between the potential reference block and the current block, for example, based on sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), etc. Motion estimation unit 222 may typically perform these calculations using sample-by-sample differences between the current block and the considered reference blocks. Motion estimation unit 222 may identify reference blocks with the minimum values ​​produced by these calculations, thereby indicating the reference block that most closely matches the current block.

[0144] Motion estimation unit 222 can generate one or more motion vectors (MVs) whose position relative to the current block in the current image defines the position of a reference block in the reference image. Motion estimation unit 222 can then provide the motion vectors to motion compensation unit 224. For example, for unidirectional inter-frame prediction, motion estimation unit 222 can provide a single motion vector, while for bidirectional inter-frame prediction, motion estimation unit 222 can provide two motion vectors. Motion compensation unit 224 can then use the motion vectors to generate prediction blocks. For example, motion compensation unit 224 can use the motion vectors to retrieve data for the reference blocks. As another example, if the motion vectors have fractional-sample precision, motion compensation unit 224 can interpolate the values ​​of the prediction blocks according to one or more interpolation filters. Furthermore, for bidirectional inter-frame prediction, motion compensation unit 224 can retrieve data for two reference blocks identified by corresponding motion vectors and combine the retrieved data, for example, via a sample-wise averaging or weighted averaging.

[0145] When operating according to the AV1 video decoding format, the motion estimation unit 222 and the motion compensation unit 224 can be configured to encode the decoded blocks of video data (e.g., both luma and chroma decoded blocks) using translational motion compensation, affine motion compensation, overlap block motion compensation (OBMC), and / or composite inter-intra-frame prediction.

[0146] As another example, for intra-prediction or intra-prediction decoding, intra-prediction unit 226 can generate a prediction block from samples adjacent to the current block. For example, in directional mode, intra-prediction unit 226 can mathematically approximate the values ​​of adjacent samples and fill in these calculated values ​​in a defined direction across the current block to generate a prediction block. As another example, in DC mode, intra-prediction unit 226 can calculate the average of samples adjacent to the current block and generate a prediction block to include this resulting average for each sample in the prediction block.

[0147] When operating according to the AV1 video decoding format, the intra-prediction unit 226 can be configured to encode decoded blocks of video data (e.g., both luma and chroma decoded blocks) using directional intra-prediction, non-directional intra-prediction, recursive filter intra-prediction, chroma-from-luma (CFL) prediction, intra-block copying (IBC), and / or palette modes. The mode selection unit 202 may include additional functional units to perform video prediction based on other prediction modes.

[0148] Mode selection unit 202 provides a prediction block to residual generation unit 204. Residual generation unit 204 receives the original, uncoded version of the current block from video data memory 230 and the prediction block from mode selection unit 202. Residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block of the current block. In some examples, residual generation unit 204 may also determine the difference between sample values ​​in the residual block to generate a residual block using residual differential pulse code modulation (RDPCM). In some examples, one or more subtractor circuits performing binary subtraction may be used to form residual generation unit 204.

[0149] In the example where mode selection unit 202 divides a CU into PUs, each PU can be associated with a luma prediction unit and a corresponding chroma prediction unit. Video encoder 200 and video decoder 300 can support PUs of various sizes. As indicated above, the size of a CU can refer to the size of the luma decoding block of the CU, while the size of a PU can refer to the size of the luma prediction unit of the PU. Assuming a particular CU size is 2N×2N, video encoder 200 can support PU sizes of 2N×2N or N×N for intra-frame prediction, and symmetrical PU sizes of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. Video encoder 200 and video decoder 300 can also support asymmetric segmentation of PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0150] In the example where mode selection unit 202 does not further divide the CU into PUs, each CU can be associated with a luminance decoding block and a corresponding chrominance decoding block. As shown above, the size of the CU can refer to the size of the luminance decoding block of the CU. The video encoder 200 and the video decoder 300 can support CU sizes of 2N×2N, 2N×N, or N×2N.

[0151] For other video decoding techniques such as intra-block copy mode decoding, affine mode decoding, and linear model (LM) mode decoding (as some examples), mode selection unit 202 generates a prediction block of the current block being encoded via a corresponding unit associated with the decoding technique. In some examples, such as palette mode decoding, mode selection unit 202 may not generate a prediction block, but instead generate syntax elements indicating how the block should be reconstructed based on a selected palette. In such modes, mode selection unit 202 can provide these syntax elements to entropy coding unit 220 for encoding.

[0152] As described above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. The residual generation unit 204 then generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

[0153] Transform processing unit 206 applies one or more transforms to the residual block to produce a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transforms to the residual block to form the transform coefficient block. For example, transform processing unit 206 may apply Discrete Cosine Transform (DCT), Oriented Transform, Karhunen-Loeve Transform (KLT), or conceptually similar transforms to the residual block. In some examples, transform processing unit 206 may perform multiple transforms on the residual block, such as primary and secondary transforms, such as rotation transforms. In some examples, transform processing unit 206 does not apply any transforms to the residual block. Transform processing unit 206 may apply any of the aforementioned MTS techniques when decoding video data blocks encoded using IBC and / or TMP modes.

[0154] When operating according to AV1, transform processing unit 206 may apply one or more transforms to the residual block to produce a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transforms to the residual block to form the transform coefficient block. For example, transform processing unit 206 may apply a combination of horizontal / vertical transforms, which may include the Discrete Cosine Transform (DCT), the Asymmetric Discrete Sine Transform (ADST), the inverted ADST (e.g., the reversed ADST), and the Identity Transform (IDTX). When using the Identity Transform, the transform is skipped in either the vertical or horizontal direction. In some examples, transform processing may be skipped.

[0155] Quantization unit 208 can quantize the transform coefficients in a transform coefficient block to produce a quantized transform coefficient block. Quantization unit 208 can quantize the transform coefficients of the transform coefficient block based on the quantization parameter (QP) value associated with the current block. Video encoder 200 (e.g., via mode selection unit 202) can adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization can introduce information loss, and therefore, the quantized transform coefficients can have lower precision than the original transform coefficients generated by transform processing unit 206.

[0156] The inverse quantization unit 210 and the inverse transform processing unit 212 can apply inverse quantization and inverse transform to the quantized transform coefficient block, respectively, to reconstruct the residual block based on the transform coefficient block. The inverse transform processing unit 210 can apply any of the aforementioned MTS techniques when decoding video data blocks encoded using IBC and / or TMP modes. The reconstruction unit 214 can generate a reconstructed block corresponding to the current block based on the reconstructed residual block and the prediction block generated by the mode selection unit 202 (although it may have some degree of distortion). For example, the reconstruction unit 214 can add samples of the reconstructed residual block to corresponding samples of the prediction block generated from the mode selection unit 202 to generate the reconstructed block.

[0157] Filter unit 216 can perform one or more filtering operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce block artifacts along the edges of the CU. In some examples, the operation of filter unit 216 can be skipped.

[0158] When operating according to AV1, filter unit 216 can perform one or more filtering operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce block artifacts along the edges of the CU. In other examples, filter unit 216 can apply a constrained directional enhancement filter (CDEF), which can be applied after deblocking, and can include the application of a non-separable, nonlinear, low-pass directional filter based on the estimated edge direction. Filter unit 216 can also include a loop recovery filter applied after CDEF, and can include a separable symmetric normalized Wiener filter or a dual self-guided filter.

[0159] The video encoder 200 stores reconstructed blocks in the DPB 218. For example, in an example where the filter unit 216 is not operated, the reconstruction unit 214 may store the reconstructed blocks in the DPB 218. In an example where the filter unit 216 is operated, the filter unit 216 may store the filtered reconstructed blocks in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 may retrieve a reference picture formed by the reconstructed (and possibly filtered) blocks from the DPB 218 to perform inter-frame prediction of blocks in subsequently encoded pictures. Additionally, the intra-frame prediction unit 226 may use the reconstructed blocks of the current picture in the DPB 218 to perform intra-frame prediction of other blocks in the current picture.

[0160] Typically, entropy coding unit 220 can entropy-encode syntax elements received from other functional components of video encoder 200. For example, entropy coding unit 220 can entropy-encode quantized transform coefficient blocks from quantization unit 208. As another example, entropy coding unit 220 can entropy-encode predictive syntax elements (e.g., motion information for inter-frame prediction or intra-frame mode information for intra-frame prediction) from mode selection unit 202. Entropy coding unit 220 can perform one or more entropy coding operations on syntax elements (another example of video data) to produce entropy-encoded data. For example, entropy coding unit 220 can perform context-adaptive variable-length decoding (CAVLC), CABAC, variable-to-variable (V2V) length decoding, syntax-based context-adaptive binary arithmetic decoding (SBAC), probabilistic interval segmented entropy (PIPE) decoding, indexed-Golomb coding, or another type of entropy coding operation on the data. In some examples, entropy coding unit 220 can operate in bypass mode, where syntax elements are not entropy-encoded.

[0161] The video encoder 200 can output a bitstream containing entropy-coded syntax elements required to reconstruct slices or blocks of images. Specifically, the entropy coding unit 220 can output a bitstream.

[0162] According to AV1, entropy coding unit 220 can be configured as a symbol-to-symbol adaptive multi-symbol arithmetic decoder. The syntax elements in AV1 include an alphabet with N elements, and the context (e.g., a probability model) includes a set of N probabilities. Entropy coding unit 220 can store the probabilities as an n-bit (e.g., 15-bit) cumulative distribution function (CDF). Entropy coding unit 220 can perform recursive scaling to update the context using an update factor based on the alphabet size.

[0163] The block description refers to the operations described above. This description should be understood as the operations applied to the luma decoding block and / or chroma decoding block. As described above, in some examples, the luma decoding block and chroma decoding block are the luma and chroma components of the CU. In some examples, the luma decoding block and chroma decoding block are the luma and chroma components of the PU.

[0164] In some examples, it is unnecessary to repeat the operations performed on the luma decoded block for the chroma decoded block. As an example, it is unnecessary to repeat the operations of identifying the motion vector (MV) and reference image of the luma decoded block to identify the MV and reference image of the chroma block. More precisely, the MV of the luma decoded block can be scaled to determine the MV of the chroma block, and the reference image can be the same. As another example, the intra-frame prediction process can be the same for both the luma and chroma decoded blocks.

[0165] Video encoder 200 represents an example of a device configured to encode video data, the device including: a memory configured to store video data; and one or more processing units implemented in circuitry and configured to: receive a block of video data to be encoded using IBC mode or intra-frame TMP mode; determine, using an MTS process, a transform pair from a plurality of transform pairs for decoding the block; and encode the block using the transform pair.

[0166] Figure 7 This is a block diagram illustrating an exemplary video decoder 300 that can perform the techniques disclosed herein. Figure 7 This disclosure is provided for illustrative purposes and is not intended to limit the techniques extensively illustrated and described herein. For illustrative purposes, this disclosure describes the video decoder 300 based on VVC and HEVC techniques. However, the techniques of this disclosure can be implemented by video decoding devices configured for other video decoding standards.

[0167] exist Figure 7 In the example, the video decoder 300 includes a decoded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, and a DPB 314. Any or all of the CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 can be implemented in one or more processors or processing circuits. For example, units of the video decoder 300 can be implemented as one or more circuit or logic elements, as part of hardware circuitry or as part of a processor, ASIC, or FPGA. Furthermore, the video decoder 300 may include additional or alternative processors or processing circuits to perform these and other functions.

[0168] The prediction processing unit 304 includes a motion compensation unit 316 and an intra-frame prediction unit 318. The prediction processing unit 304 may include additional units that perform predictions based on other prediction modes. As an example, the prediction processing unit 304 may include a palette unit, an intra-block copy unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, the video decoder 300 may include more, fewer, or different functional components.

[0169] When operating according to AV1, motion compensation unit 316 can be configured to decode blocks of video data (e.g., both luma and chroma decoded blocks) using translational motion compensation, affine motion compensation, OBMC, and / or composite inter-intra-frame prediction, as described above. Intra-frame prediction unit 318 can be configured to encode blocks of video data (e.g., both luma and chroma decoded blocks) using directional intra-frame prediction, non-directional intra-frame prediction, recursive filter intra-frame prediction, CFL, IBC, and / or palette mode, as described above.

[0170] CPB memory 320 can store video data (such as encoded video bitstreams) to be decoded by components of video decoder 300. The video data stored in CPB memory 320 can, for example, be retrieved from computer-readable medium 110 (…). Figure 1 The CPB memory 320 may include a CPB that stores encoded video data (e.g., syntax elements) from the encoded video bitstream. Furthermore, the CPB memory 320 may store video data other than the syntax elements of the decoded picture (such as temporary data representing the output from various units of the video decoder 300). The DPB 314 typically stores decoded pictures that the video decoder 300 may output and / or use as reference video data when decoding subsequent data or pictures from the encoded video bitstream. The CPB memory 320 and DPB 314 may be formed from any of a variety of memory devices, such as DRAM including SDRAM, MRAM, RRAM, or other types of memory devices. The CPB memory 320 and DPB 314 may be provided by the same memory device or separate memory devices. In various examples, the CPB memory 320 may be on-chip with other components of the video decoder 300, or off-chip relative to those components.

[0171] Additionally or alternatively, in some examples, the video decoder 300 can be generated from the memory 120 ( Figure 1 The decoded video data can be retrieved. That is, memory 120 can store the data discussed above through CPB memory 320. Similarly, when some or all of the functionality of video decoder 300 is implemented in software for execution by the processing circuitry of video decoder 300, memory 120 can store instructions to be executed by video decoder 300.

[0172] Figure 7 The various units shown are illustrated to aid in understanding the operations performed by the video decoder 300. This unit can be implemented as a fixed-function circuit, a programmable circuit, or a combination thereof. Similar to... Figure 6Fixed-function circuits are circuits that provide specific functionality and have predefined executable operations. Programmable circuits are circuits that can be programmed to perform various tasks and provide flexible functionality in their executable operations. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by instructions from software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is typically immutable. In some examples, one or more of the units may be different circuit blocks (fixed-function or programmable), and in some examples, one or more of the units may be integrated circuits.

[0173] The video decoder 300 may include an ALU, an EFU, digital circuitry, analog circuitry, and / or a programmable core formed by programmable circuitry. In an example where the operation of the video decoder 300 is performed by software executed on the programmable circuitry, on-chip or off-chip memory may store instructions (e.g., object code) of the software received and executed by the video decoder 300.

[0174] Entropy decoding unit 302 can receive encoded video data from the CPB and perform entropy decoding on the video data to reproduce the syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filter unit 312 can generate decoded video data based on the syntax elements extracted from the bitstream.

[0175] Typically, the video decoder 300 reconstructs the image block by block. The video decoder 300 can perform the reconstruction operation on each block individually (where the currently reconstructed (i.e., decoded) block can be referred to as the "current block").

[0176] Entropy decoding unit 302 can entropy decode the syntax elements of the quantized transform coefficients that define the quantized transform coefficient block, as well as transform information such as quantization parameters (QP) and / or (one or more) transform mode indications. Inverse quantization unit 306 can use the QP associated with the quantized transform coefficient block to determine the quantization level, and similarly determine the inverse quantization level for application by inverse quantization unit 306. Inverse quantization unit 306 can, for example, perform a bit-left shift operation to inverse quantize the quantized transform coefficients. Inverse quantization unit 306 can thereby form a transform coefficient block including the transform coefficients.

[0177] After the inverse quantization unit 306 forms the transform coefficient block, the inverse transform processing unit 308 can apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, the inverse transform processing unit 308 can apply inverse DCT, inverse integer transform, inverse Karhunen-Loeve transform (KLT), inverse rotation transform, inverse direction transform, or another inverse transform to the transform coefficient block. The inverse transform processing unit 308 can apply any of the above-described MTS techniques when decoding video data blocks encoded using IBC and / or TMP modes.

[0178] Furthermore, prediction processing unit 304 generates a prediction block based on the prediction information syntax elements entropy decoded by entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is predicted inter-frame, motion compensation unit 316 can generate a prediction block. In this case, the prediction information syntax elements can indicate a reference picture in DPB 314 (from which the reference block is retrieved) and a motion vector that identifies the position of the reference block in the reference picture relative to the current block in the current picture. Motion compensation unit 316 can generally be substantially similar to that of motion compensation unit 224 ( Figure 6 The inter-frame prediction process is performed in the manner described.

[0179] As another example, if the prediction information syntax element indicates that the current block is intra-predicted, then intra-prediction unit 318 can generate a prediction block according to the intra-prediction mode indicated by the prediction information syntax element. Furthermore, intra-prediction unit 318 can generally be substantially similar to intra-prediction unit 226 ( Figure 6 The intra-prediction process is performed in the manner described. The intra-prediction unit 318 can retrieve data of neighboring samples of the current block from the DPB 314.

[0180] Reconstruction unit 310 can use the prediction block and the residual block to reconstruct the current block. For example, reconstruction unit 310 can add samples from the residual block to the corresponding samples from the prediction block to reconstruct the current block.

[0181] Filter unit 312 can perform one or more filtering operations on the reconstructed block. For example, filter unit 312 can perform a deblocking operation to reduce block artifacts along the edges of the reconstructed block. The operation of filter unit 312 is not necessarily performed in all examples.

[0182] The video decoder 300 can store reconstructed blocks in the DPB 314. For example, in an example where the operation of the filter unit 312 is not performed, the reconstruction unit 310 can store the reconstructed blocks in the DPB 314. In an example where the operation of the filter unit 312 is performed, the filter unit 312 can store the filtered reconstructed blocks in the DPB 314. As discussed above, the DPB 314 can provide reference information such as the current image for intra-frame prediction and samples of previously decoded images for subsequent motion compensation to the prediction processing unit 304. Furthermore, the video decoder 300 can output decoded images (e.g., decoded video) from the DPB 314 for subsequent rendering in applications such as... Figure 1 The display device 118 is on the display device.

[0183] In this manner, video decoder 300 represents an example of a video decoding device, which includes: a memory configured to store video data; and one or more processing units implemented in a circuit and configured to: receive a block of video data encoded using IBC mode or intra-frame TMP mode; determine, using an MTS process, a transform pair from a plurality of transform pairs for decoding the block; and decode the block using the transform pair.

[0184] Figure 8 This is a flowchart illustrating an exemplary method for encoding a current block according to the technology of this disclosure. The current block may be or includes the current CU. Although regarding video encoder 200 ( Figure 1 and Figure 6 This description is provided, but it should be understood that other devices can be configured to perform similar functions. Figure 8 The method of the method.

[0185] In this example, the video encoder 200 first predicts the current block (350). For example, the video encoder 200 may form a prediction block for the current block. The video encoder 200 may then compute the residual block for the current block (352). To compute the residual block, the video encoder 200 may compute the difference between the original uncoded block of the current block and the prediction block. The video encoder 200 may then transform the residual block and quantize the transform coefficients of the residual block (354). Next, the video encoder 200 may scan the quantized transform coefficients of the residual block (356). During or after the scan, the video encoder 200 may entropy encode the transform coefficients (358). For example, the video encoder 200 may encode the transform coefficients using CAVLC or CABAC. The video encoder 200 may then output the entropy-encoded data of the block (360).

[0186] Figure 9This is a flowchart illustrating an exemplary method for decoding a current block of video data according to the technology of this disclosure. The current block may be or includes the current CU. Although regarding the video decoder 300 ( Figure 1 and Figure 7 This description is provided, but it should be understood that other devices can be configured to perform similar functions. Figure 9 The method of the method.

[0187] The video decoder 300 can receive entropy-coded data of the current block (such as entropy-coded prediction information and entropy-coded data corresponding to the transform coefficients of the residual block of the current block) (370). The video decoder 300 can entropy decode the entropy-coded data to determine the prediction information of the current block and reproduce the transform coefficients of the residual block (372). The video decoder 300 can predict the current block, for example, using intra-frame or inter-frame prediction indicated by the prediction information of the current block (374), to compute the prediction block of the current block. The video decoder 300 can then inversely scan the reproduced transform coefficients (376) to produce a block of quantized transform coefficients. The video decoder 300 can then inversely quantize the transform coefficients and apply an inverse transform to the transform coefficients to produce the residual block (378). The video decoder 300 can finally decode the current block by combining the prediction block and the residual block (380).

[0188] Figure 10 This is a flowchart illustrating another exemplary method for encoding the current block according to the technology of this disclosure. Figure 10 The technology can be performed by one or more components of the video encoder 200 (including the transform processing unit 206 and / or the inverse transform processing unit 212).

[0189] In one example of this disclosure, video encoder 200 may be configured to receive a block of video data (1000) to be encoded using either Intra-Block Copy (IBC) mode or Intra-Template Match Prediction (TMP) mode. In one example, the video data is natural content video data. Video encoder 200 may use a Multiple Transform Selection (MTS) procedure to determine (1010) a plurality of transform pairs for decoding the block, and encode the block using the transform pairs (1020). The plurality of transform pairs may include a Vertical 2 Discrete Cosine Transform (DCT-2), a Horizontal DCT-2, and four other transform pairs. In one example, video encoder 200 may also be configured to encode an MTS index indicating a transform pair.

[0190] In one example of this disclosure, a video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode. In another example of this disclosure, the video data block is encoded using an IBC mode, and the IBC mode is either an AMVP IBC mode or a merged IBC mode.

[0191] In another example of this disclosure, the video encoder 200 may also be configured to encode syntax elements indicating the number of the plurality of transform pairs using context-adaptive binary arithmetic decoding and a dedicated context pair. In yet another example, the video encoder 200 may be configured to determine the intra-mode of the block based on a decoder-side intra-mode derivation (DIMD) process, and to determine the number of the plurality of transform pairs based on the intra-mode.

[0192] Figure 11 This is a flowchart illustrating another exemplary method for decoding the current block according to the technology of this disclosure. Figure 11 The technology can be performed by one or more components of the video decoder 300, including the inverse transform processing unit 308.

[0193] In one example of this disclosure, the video decoder 300 can be configured to receive a block of video data encoded using either an intra-block copy (IBC) mode or an intra-template match prediction (TMP) mode (1100). In one example, the video data is natural content video data. In one example, the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode. In another example, the video data block is encoded using an IBC mode, and the IBC mode is either an AMVP IBC mode or a merged IBC mode. The video decoder 300 can also be configured to use a multiple transform selection (MTS) process to determine (1110) a transform pair from a plurality of transform pairs for decoding the block, and to decode the block using that transform pair (1120).

[0194] In one example, to determine a transform pair among multiple transform pairs, the video decoder 300 is configured to receive an MTS index and determine the transform pair based on the MTS index. In another example, to determine a transform pair among multiple transform pairs, the video decoder 300 is configured to receive an MTS index and determine the transform pair based on the MTS index and the block size.

[0195] In one example, the multiple transform pairs include a vertical 2-Discrete Cosine Transform (DCT-2), a horizontal DCT-2, and four other transform pairs. The video decoder 300 can also be configured to determine the number of multiple transform pairs. In one example, to determine the number of multiple transform pairs, the video decoder 300 can be configured to use context-adaptive binary arithmetic decoding and a dedicated context to decode the syntax element indicating the number of multiple transform pairs. In another example, to determine the number of multiple transform pairs, the video decoder 300 is configured to determine the intra-mode of the block based on a decoder-side intra-mode derivation (DIMD) process, and determine the number of multiple transform pairs based on the intra-mode.

[0196] The following numbered clauses illustrate one or more aspects of the devices and technologies described in this disclosure.

[0197] Aspect 1A. A method for decoding video data, the method comprising: determining a plurality of transformations for decoding a block that is also decoded using an intra-block copy mode; and decoding the block using the determined transformation.

[0198] Aspect 2A. The method according to aspect 1A, wherein the intra-block copy mode is an AMVP mode.

[0199] Aspect 3A. The method according to aspect 1A, wherein the intra-block copy mode is a merge mode.

[0200] Aspect 4A. A method for decoding video data, the method comprising: determining a plurality of transforms for decoding a block that is also decoded using an intra-frame template matching pattern; and decoding the block using the determined transforms.

[0201] Aspect 5A. The method according to any one of Aspects 1A to 4A, wherein decoding includes decoding.

[0202] Aspect 6A. The method according to any one of aspects 1A to 4A, wherein decoding includes encoding.

[0203] Aspect 7A. An apparatus for decoding video data, the apparatus comprising one or more components for performing the method according to any one of aspects 1A to 6A.

[0204] Aspect 8A. The device according to aspect 7A, wherein the one or more components include one or more processors implemented in a circuit.

[0205] Aspect 9A. The device according to any one of aspects 7A and 8A further includes a memory for storing video data.

[0206] Aspect 10A. The device according to any one of aspects 7A to 9A further includes a display configured to display decoded video data.

[0207] Aspect 11A. The device according to any one of aspects 7A to 10A, wherein the device includes one or more of the following: a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0208] Aspect 12A. The device according to any one of aspects 7A to 11A, wherein the device includes a video decoder.

[0209] Aspect 13A. The device according to any one of aspects 7A to 12A, wherein the device includes a video encoder.

[0210] Aspect 14A. A computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to perform the method according to any one of aspects 1A to 6A.

[0211] Aspect 1B. A method for decoding video data, the method comprising: receiving a block of video data encoded using an intra-block copy (IBC) mode or an intra-template matching prediction (TMP) mode; determining, using a multiple transform selection (MTS) procedure, a transform pair from a plurality of transform pairs for decoding the block; and decoding the block using the transform pair.

[0212] Aspect 2B. The method according to aspect 1B, wherein determining the transformation pair among the plurality of transformation pairs comprises: receiving an MTS index; and determining the transformation pair based on the MTS index.

[0213] Aspect 3B. The method according to aspect 1B, wherein determining the transform pair among the plurality of transform pairs comprises: receiving an MTS index; and determining the transform pair based on the MTS index and the size of the block.

[0214] Aspect 4B. The method according to any one of Aspects 1B to 3B, wherein the plurality of transformation pairs includes a vertical type 2 discrete cosine transform (DCT-2), a horizontal DCT-2, and four other transformation pairs.

[0215] Aspect 5B. The method according to any one of aspects 1B to 4B, further comprising: determining the number of the plurality of transformation pairs.

[0216] Aspect 6B. The method according to aspect 5B, wherein determining the number of the plurality of transform pairs comprises: decoding a syntax element indicating the number of the plurality of transform pairs using context-adaptive binary arithmetic decoding and a dedicated context pair.

[0217] Aspect 7B. The method according to aspect 5B, wherein determining the number of the plurality of transform pairs comprises: determining the intra-mode of the block based on a decoder-side intra-mode derivation (DIMD) process; and determining the number of the plurality of transform pairs based on the intra-mode.

[0218] Aspect 8B. The method according to any one of Aspects 1B to 7B, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode.

[0219] Aspect 9B. The method according to any one of Aspects 1B to 7B, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode or a merged IBC mode.

[0220] Aspect 10B. The method according to any one of aspects 1B to 9B, wherein the video data is natural content video data.

[0221] Aspect 11B. An apparatus configured to decode video data, the apparatus comprising: a memory configured to receive a block of video data; and processing circuitry in communication with the memory, the processing circuitry being configured to: receive a block of video data encoded using an intra-block copy (IBC) mode or an intra-template matching prediction (TMP) mode; determine, using a multiple transform selection (MTS) process, a transform pair from a plurality of transform pairs for decoding the block; and decode the block using the transform pair.

[0222] Aspect 12B. The apparatus according to aspect 11B, wherein, in order to determine the transformation pair among the plurality of transformation pairs, the processing circuitry is further configured to: receive an MTS index; and determine the transformation pair based on the MTS index.

[0223] Aspect 13B. The apparatus according to aspect 11B, wherein, in order to determine the transformation pair among the plurality of transformation pairs, the processing circuitry is further configured to: receive an MTS index; and determine the transformation pair based on the MTS index and the size of the block.

[0224] Aspect 14B. The apparatus according to any one of aspects 11B to 13B, wherein the plurality of transformation pairs includes a vertical 2-discrete cosine transform (DCT-2), a horizontal DCT-2, and four other transformation pairs.

[0225] Aspect 15B. The apparatus according to any one of aspects 11B to 14B, wherein the processing circuit is further configured to: determine the number of the plurality of transformation pairs.

[0226] Aspect 16B. The apparatus according to aspect 15B, wherein, in order to determine the number of the plurality of transform pairs, the processing circuitry is further configured to: decode the syntax elements indicating the number of the plurality of transform pairs using context-adaptive binary arithmetic decoding and a dedicated context pair.

[0227] Aspect 17B. The apparatus according to aspect 15B, wherein, in order to determine the number of the plurality of transform pairs, the processing circuitry is further configured to: determine the intra-mode of the block based on a decoder-side intra-mode derivation (DIMD) process; and determine the number of the plurality of transform pairs based on the intra-mode.

[0228] Aspect 18B. The apparatus according to any one of aspects 11B to 17B, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode.

[0229] Aspect 19B. The apparatus according to any one of aspects 11B to 17B, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode or a merged IBC mode.

[0230] Aspect 20B. The apparatus according to any one of aspects 11B to 19B, wherein the video data is natural content video data.

[0231] Aspect 21B. A method for encoding video data, the method comprising: receiving a block of video data to be encoded using an intra-block copy (IBC) mode or an intra-template matching prediction (TMP) mode; determining, using a multiple transform selection (MTS) procedure, a transform pair from a plurality of transform pairs for decoding the block; and encoding the block using the transform pair.

[0232] Aspect 22B. The method according to aspect 21B further includes: encoding the MTS index indicating the transform pair.

[0233] Aspect 23B. The method according to any one of aspects 21B to 22B, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode.

[0234] Aspect 24B. The method according to any one of aspects 21B to 22B, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode or a merged IBC mode.

[0235] Aspect 25B. The method according to any one of aspects 21B to 24B, wherein the video data is natural content video data.

[0236] Aspect 26B. An apparatus configured to encode video data, the apparatus comprising: a memory configured to receive a block of video data; and processing circuitry in communication with the memory, the processing circuitry being configured to: receive a block of video data to be encoded using an intra-block copy (IBC) mode or an intra-template matching prediction (TMP) mode; determine, using a multiple transform selection (MTS) process, a transform pair from a plurality of transform pairs for decoding the block; and encode the block using the transform pair.

[0237] Aspect 27B. The apparatus according to aspect 26B, wherein the processing circuitry is further configured to encode the MTS index indicating the transform pair.

[0238] Aspect 28B. The apparatus according to any one of aspects 26B to 27B, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode.

[0239] Aspect 29B. The apparatus according to any one of aspects 26B to 27B, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode or a merged IBC mode.

[0240] Aspect 30B. The apparatus according to any one of aspects 26B to 29B, wherein the video data is natural content video data.

[0241] It should be recognized that, depending on the example, certain actions or events of any technique described herein may be performed in a different order, or may be added, combined, or omitted entirely (e.g., not all described actions or events are necessary for technical practice). Furthermore, in some examples, actions or events may be performed concurrently, for example, through multithreaded processing, interrupt handling, or multiple processors, rather than sequentially.

[0242] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, including any medium that facilitates, for example, the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium (such as a signal or carrier wave). A data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0243] By way of example, and not limitation, such computer-readable storage media may include one or more of the following: RAM, ROM, EEPROM, CD-ROM or other optical disc storage, magnetic disk storage media or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, the definition of medium includes coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, if instructions are transmitted from a website, server, or other remote source using coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient, tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically reproduce data magnetically, while optical discs reproduce data optically by means of lasers. The above combinations should also be included within the scope of computer-readable media.

[0244] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Therefore, the terms "processor" and "processing circuitry" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding or incorporated into combined codecs. Moreover, the techniques can be fully implemented in one or more circuit or logic elements.

[0245] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but they do not necessarily need to be implemented by different hardware units. More precisely, as described above, various units can be combined in a codec hardware unit, or provided by a collection of interoperable hardware units (including one or more processors as described above) combined with suitable software and / or firmware.

[0246] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for decoding video data, the method comprising: Receive video data blocks encoded using Intra-Block Copy (IBC) mode or Intra-Template Match Prediction (TMP) mode; The Multiple Transform Selection (MTS) process is used to determine the transform pair among multiple transform pairs used for decoding the block; as well as The block is decoded using the transformation.

2. The method of claim 1, wherein determining the transformation pair among the plurality of transformation pairs comprises: Receive MTS index; as well as The transformation pair is determined based on the MTS index.

3. The method of claim 1, wherein determining the transformation pair among the plurality of transformation pairs comprises: Receive MTS index; as well as The transformation pair is determined based on the MTS index and the size of the block.

4. The method of claim 1, wherein the plurality of transformation pairs includes a vertical 2-discrete cosine transform (DCT-2), a horizontal DCT-2, and four other transformation pairs.

5. The method according to claim 1, further comprising: Determine the number of the plurality of transformation pairs.

6. The method of claim 5, wherein determining the number of the plurality of transformation pairs comprises: The context-adaptive binary arithmetic decoding and dedicated context are used to decode the number of syntax elements indicating the plurality of transformation pairs.

7. The method of claim 5, wherein determining the number of the plurality of transformation pairs comprises: The intra-frame mode of the block is determined based on the decoder-side intra-frame mode derivation (DIMD) process; as well as The number of the plurality of transform pairs is determined based on the intra-frame mode.

8. The method of claim 1, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode.

9. The method of claim 1, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode or a merged IBC mode.

10. The method of claim 1, wherein the video data is natural content video data.

11. An apparatus configured to decode video data, the apparatus comprising: A memory configured to receive blocks of video data; as well as The processing circuitry communicating with the memory is configured to: Receive the video data block encoded using Intra-Block Copy (IBC) mode or Intra-Template Match Prediction (TMP) mode; The Multiple Transform Selection (MTS) process is used to determine the transform pair among multiple transform pairs used for decoding the block; as well as The block is decoded using the transformation.

12. The apparatus of claim 11, wherein, in order to determine the transformation pair among the plurality of transformation pairs, the processing circuit is further configured to: Receive MTS index; and The transformation pair is determined based on the MTS index.

13. The apparatus of claim 11, wherein, in order to determine the transformation pair among the plurality of transformation pairs, the processing circuit is further configured to: Receive MTS index; and The transformation pair is determined based on the MTS index and the size of the block.

14. The apparatus of claim 11, wherein the plurality of transformation pairs comprises a vertical 2-discrete cosine transform (DCT-2), a horizontal DCT-2, and four other transformation pairs.

15. The apparatus of claim 11, wherein the processing circuit is further configured to: Determine the number of the plurality of transformation pairs.

16. The apparatus of claim 15, wherein, in order to determine the number of the plurality of transformation pairs, the processing circuit is further configured to: The context-adaptive binary arithmetic decoding and dedicated context are used to decode the number of syntax elements indicating the plurality of transformation pairs.

17. The apparatus of claim 15, wherein, in order to determine the number of the plurality of transformation pairs, the processing circuit is further configured to: The intra-frame mode of the block is determined based on the decoder-side intra-frame mode derivation (DIMD) process; and The number of the plurality of transform pairs is determined based on the intra-frame mode.

18. The apparatus of claim 11, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode.

19. The apparatus of claim 11, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode or a merged IBC mode.

20. The apparatus of claim 11, wherein the video data is natural content video data.

21. A method for encoding video data, the method comprising: Receive video data blocks to be encoded using Intra-Block Copy (IBC) mode or Intra-Template Match Prediction (TMP) mode; The Multiple Transform Selection (MTS) process is used to determine the transform pair among multiple transform pairs used for decoding the block; as well as The block is encoded using the transformation.

22. The method of claim 21, further comprising: The MTS index indicating the transform pair is encoded.

23. The method of claim 21, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode.

24. The method of claim 21, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode or a merged IBC mode.

25. The method of claim 21, wherein the video data is natural content video data.

26. An apparatus configured to encode video data, the apparatus comprising: A memory configured to receive blocks of video data; as well as The processing circuitry communicating with the memory is configured to: Receive video data blocks to be encoded using Intra-Block Copy (IBC) mode or Intra-Template Match Prediction (TMP) mode; The Multiple Transform Selection (MTS) process is used to determine the transform pair among multiple transform pairs used for decoding the block; as well as The block is encoded using the transformation.

27. The apparatus of claim 26, wherein the processing circuitry is further configured to: The MTS index indicating the transform pair is encoded.

28. The apparatus of claim 26, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode.

29. The apparatus of claim 26, wherein the video data block is encoded using an IBC mode, and the IBC mode is an AMVP IBC mode or a merged IBC mode.

30. The apparatus of claim 26, wherein the video data is natural content video data.

Images