Joint component neural network-based filtering during video coding

Abstract

An exemplary device for decoding video data includes a memory configured to store the video data and one or more processors implemented in a circuit, the one or more processors configured to: apply a downsampling convolutional neural network layer to first color components of a block of the video data, the first color components of the block having a first size, where applying the downsampling convolutional neural network layer to the first color components generates downsampled first color components having a second size smaller than the first size; filter the second color components having the second size to form filtered second color components; concatenate the downsampled first color components with the filtered second color components to form concatenated color components; and filter the concatenated color components to form filtered concatenated components including the filtered downsampled first color components.

Description

Technical Field

[0001]

[0001] This application claims the benefit of U.S. Patent Application No. 17 / 493,543, filed Oct. 4, 2021, and U.S. Provisional Application No. 63 / 087,784, filed Oct. 5, 2020, the entire contents of each of which are incorporated herein by reference. U.S. Patent Application No. 17 / 493,543, filed Oct. 4, 2021, claims the benefit of U.S. Provisional Patent Application No. 63 / 087,784, filed Oct. 5, 2020.

[0002]

[0002] This disclosure relates to video coding, including video encoding and video decoding.

Background Art

[0003]

[0003] Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular phones or satellite radiotelephones, so-called “smartphones,” video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.26High Efficiency Video Coding (HEVC), and extensions to such standards. By implementing such video coding techniques, video devices can transmit, receive, encode, decode, and / or store digital video information more efficiently.

[0004]

[0004] Video coding techniques include spatial (in-picture) prediction and / or temporal (between-picture) prediction to reduce or eliminate redundancy inherent in video sequences. In block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be divided into video blocks, which may also be called coding tree units (CTUs), coding units (CUs), and / or coding nodes. A video block in an intra-coding (I) slice of a picture is coded using spatial predictions for reference samples in adjacent blocks within the same picture. A video block in an inter-coding (P or B) slice of a picture may use spatial predictions for reference samples in adjacent blocks within the same picture, or temporal predictions for reference samples in other reference pictures. A picture may be called a frame, and a reference picture may be called a reference frame. [Overview of the project]

[0005]

[0005] Generally speaking, this disclosure describes techniques for filtering decoded pictures that may be distorted. The filtering process may be based on neural network techniques. The filtering process may be used in the context of advanced video codecs such as extensions to ITU-T H.266 / General-Purpose Video Coding (VVC) and subsequent generations of video coding standards, and any other video codecs.

[0006]

[0006] In one example, a method for filtering decoded video data is to apply a downsampling convolutional neural network layer to the first color component of a block of video data, where the first color component of the block has a first size, and applying the downsampling convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; to form a filtered second color component, the second color component having a second size is filtered; to form a filtered second color component, the downsampled first color component is concatenated with the filtered second color component; and the filtered downsampled first color component is concatenated with the filtered first color component. This includes filtering concatenated color components to form a filtered concatenated component that includes a component.

[0007]

[0007] In another example, a device for filtering decoded video data includes a memory configured to store video data and one or more processors implemented in the circuit, the one or more processors being configured to: apply a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, herein, applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; filter the second color component having a second size to form a filtered second color component; concatenate the downsampled first color component with the filtered second color component to form a concatenated color component; and filter the concatenated color component to form a filtered concatenated component containing the filtered downsampled first color component.

[0008]

[0008] In another example, a computer-readable storage medium stores instructions which, when executed, cause a processor to: apply a downsampled convolutional neural network layer to a first color component of a block of video data, where the first color component of the block has a first size, and applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; filter the second color component having the second size to form a filtered second color component; concatenate the downsampled first color component with the filtered second color component to form a concatenated color component; and filter the concatenated color component to form a filtered concatenated component containing the filtered downsampled first color component.

[0009]

[0009] In another example, a device for filtering decoded video data includes means for applying a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, wherein applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; means for filtering the second color component having a second size to form a filtered second color component; means for concatenating the downsampled first color component with the filtered second color component to form a concatenated color component; and means for filtering the concatenated color component to form a filtered concatenated component containing the filtered downsampled first color component.

[0010]

[0010] Details of one or more examples are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0011] [Figure 1]

[0011] A block diagram showing an exemplary video coding and decoding system capable of implementing the techniques of the present disclosure. [Figure 2]

[0012] A conceptual diagram illustrating a hybrid video coding framework. [Figure 3]

[0013] A conceptual diagram showing a hierarchical prediction structure using a Group of Pictures (GOP) size of 16. [Figure 4A]

[0014] A conceptual diagram showing an exemplary joint component convolutional neural network (CNN)-based filter having a residual network (ResNet) containing residual blocks. [Figure 4B] A conceptual diagram showing an exemplary joint component convolutional neural network (CNN)-based filter having a residual network (ResNet) containing residual blocks. [Figure 5A]

[0015] A conceptual diagram illustrating an exemplary quad-tree binary tree (QTBT) structure. [Figure 5B] A conceptual diagram showing the corresponding coding tree unit (CTU). [Figure 6]

[0016] A block diagram showing an exemplary video encoder capable of implementing the techniques of this disclosure. [Figure 7]

[0017] A block diagram illustrating an exemplary video decoder capable of implementing the techniques of this disclosure. [Figure 8]

[0018] A block diagram illustrating an example of a joint component CNN filter using the technique described herein. [Figure 9]

[0019] Conceptual diagram showing an exemplary residual processing unit [Figure 10]

[0020] Block diagram showing another example of joint component CNN filter design according to the techniques of the present disclosure [Figure 11]

[0021] Flowchart showing an exemplary method for filtering a decoded block according to the techniques of the present disclosure [Figure 12]

[0022] Flowchart showing an exemplary method for filtering a decoded block according to the techniques of the present disclosure [Figure 13]

[0023] Flowchart showing an exemplary method for filtering decoded video data according to the techniques of the present disclosure [Figure 14] ]>

[0024] Flowchart showing an exemplary method for filtering decoded video data according to the techniques of the present disclosure

Mode for Carrying Out the Invention

[0012]

[0025] Video coding standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, and ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), High Efficiency Video Coding (HEVC) or ITU-T H.265 (including its range extensions, multi-view extensions (MV-HEVC), and scalable extensions (SHVC)). Another exemplary video coding standard is Versatile Video Coding (VVC) or ITU-T H.266, which was developed by the Joint Video Expert Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). Version 1 of the VVC specification, hereinafter referred to as "VVC FDIS", is available from http: / / phenix.int-evry.fr / jvet / doc_end_user / documents / 19_Teleconference / wg11 / JVET-S2001-v17.zip.

[0013]

[0026] FIG. 1 is a block diagram showing an exemplary video encoding and decoding system 100 in which the techniques of the present disclosure may be implemented. The techniques of the present disclosure generally are directed to coding (encoding and / or decoding) video data. Generally, video data includes any data for processing video. Thus, video data may include raw uncoded video, encoded video, decoded (e.g., reconstructed) video, and video metadata such as signaling data.

[0014]

[0027] As shown in Figure 1, system 100 includes, in this example, a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116. In particular, the source device 102 provides video data to the destination device 116 via a computer-readable medium 110. The source device 102 and the destination device 116 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, and so on. In some cases, the source device 102 and the destination device 116 may be equipped for wireless communication and are therefore sometimes referred to as wireless communication devices.

[0015]

[0028] In the example in Figure 1, the source device 102 includes a video source 104, a memory 106, a video encoder 200, and an output interface 108. The destination device 116 includes an input interface 122, a video decoder 300, a memory 120, and a display device 118. According to this disclosure, the video encoder 200 of the source device 102 and the video decoder 300 of the destination device 116 may be configured to apply techniques for filtering video data using a joint component neural network-based filtering process. Thus, the source device 102 represents an example of a video encoding device, and the destination device 116 represents an example of a video decoding device. In other examples, the source and destination devices may include other components or arrangements. For example, the source device 102 may receive video data from an external video source, such as an external camera. Similarly, the destination device 116 may interface with an external display device rather than including an integrated display device.

[0016]

[0029] The system 100 shown in Figure 1 is merely an example. In general, any digital video coding and / or decoding device may employ techniques for filtering video data using a joint-component neural network-based filtering process. Source device 102 and destination device 116 are merely examples of coding devices such that source device 102 generates coded video data for transmission to destination device 116. In this disclosure, the term “coding” device refers to a device that performs coding (encoding and / or decoding) of data. Thus, video encoder 200 and video decoder 300 represent examples of coding devices, in particular, a video encoder and a video decoder, respectively. In some examples, source device 102 and destination device 116 may operate substantially symmetrically such that each of source device 102 and destination device 116 includes video coding components and video decoding components. Thus, system 100 may support one-way or two-way video transmission between source device 102 and destination device 116 for, for example, video streaming, video playback, video broadcasting, or video telephony.

[0017]

[0030] Generally, the video source 104 represents a source of video data (i.e., raw, uncoded video data) and provides the video encoder 200 with a sequence of pictures (also called "frames") of video data, which the video encoder 200 encodes the data for the pictures. The video source 104 of the source device 102 may include a video capture device, such as a video camera, a video archive containing previously captured raw video, and / or a video feed interface for receiving video from a video content provider. As a further alternative, the video source 104 may generate computer graphics-based data, or a combination of live video, archived video, and computer-generated video, as source video. In each case, the video encoder 200 encodes the captured video data, pre-captured video data, or computer-generated video data. The video encoder 200 may rearrange the pictures from the order they are received (sometimes called the "display order") to the coding order for encoding. The video encoder 200 may generate a bitstream containing the encoded video data. The source device 102 may then output encoded video data onto a computer-readable medium 110 via the output interface 108 for reception and / or retrieval by the input interface 122 of the destination device 116, for example.

[0018]

[0031] Memory 106 of source device 102 and memory 120 of destination device 116 represent general-purpose memory. In some examples, memories 106 and 120 may store raw video data, for example, raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106 and 120 may store, for example, software instructions executable by video encoder 200 and video decoder 300, respectively. Although memories 106 and 120 are shown separately from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store encoded video data, for example, output from video encoder 200 and input to video decoder 300. In some examples, portions 106 and 120 of memory may be allocated as one or more video buffers, for example, to store raw decoded and / or encoded video data.

[0019]

[0032] The computer-readable medium 110 may represent any type of medium or device capable of transferring encoded video data from the source device 102 to the destination device 116. For example, the computer-readable medium 110 may represent a communication medium that enables the source device 102 to directly transmit encoded video data to the destination device 116 in real time, for example, over a radio frequency network or a computer-based network. According to a communication standard such as a wireless communication protocol, the output interface 108 may modulate the transmission signal containing the encoded video data, and the input interface 122 may demodulate the received transmission signal. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful to facilitate communication from the source device 102 to the destination device 116.

[0020]

[0033] In some examples, the source device 102 may output encoded data to the storage device 112 via the output interface 108. Similarly, the destination device 116 may access encoded data from the storage device 112 via the input interface 122. The storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray® disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0021]

[0034] In some examples, the source device 102 may output encoded video data to a file server 114 or another intermediate storage device capable of storing the encoded video data generated by the source device 102. The destination device 116 may access the stored video data from the file server 114 via streaming or download.

[0022]

[0035] The file server 114 can be any type of server device capable of storing encoded video data and transmitting that encoded video data to the destination device 116. The file server 114 may represent a web server (for example, for a website), a server configured to provide file transfer protocol services (such as the File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol), a Content Delivery Network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and / or a Network Attached Storage (NAS) device. The file server 114 may, in addition or alternatively, implement one or more HTTP streaming protocols, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real-Time Streaming Protocol (RTSP), or HTTP Dynamic Streaming.

[0023]

[0036] The destination device 116 may access encoded video data from the file server 114 through any standard data connection, including an internet connection. This may include wireless channels (e.g., Wi-Fi® connection), wired connections (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both, which are suitable for accessing encoded video data stored in the file server 114. The input interface 122 may be configured to operate according to one or more of the various protocols described above for retrieving or receiving media data from the file server 114, or any other such protocols for retrieving media data.

[0024]

[0037] The output interface 108 and input interface 122 may represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet® card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where the output interface 108 and input interface 122 include wireless components, the output interface 108 and input interface 122 may be configured to transfer data such as encoded video data according to cellular communication standards such as 4G, 4G-LTE® (Long-Term Evolution), LTE Advanced, or 5G. In some examples where the output interface 108 includes a wireless transmitter, the output interface 108 and input interface 122 may be configured to transfer data such as encoded video data according to other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee®), or the Bluetooth® standard. In some examples, the source device 102 and / or destination device 116 may include their respective system-on-chip (SoC) devices. For example, the source device 102 may include an SoC device for performing functions associated with the video encoder 200 and / or the output interface 108, and the destination device 116 may include an SoC device for performing functions associated with the video decoder 300 and / or the input interface 122.

[0025]

[0038] The techniques of this disclosure can be applied to video coding that supports any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission such as Dynamic Adaptive Streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0026]

[0039] The input interface 122 of the destination device 116 receives an encoded video bitstream from a computer-readable medium 110 (e.g., a communication medium, a storage device 112, a file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200, which is also used by the video decoder 300, such as syntax elements having values ​​that describe the characteristics and / or processing of video blocks or other encoded units (e.g., slices, pictures, groups of pictures, sequences, etc.). The display device 118 displays the decoded picture of the decoded video data to the user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0027]

[0040] Although not shown in Figure 1, in some examples, the video encoder 200 and video decoder 300 may be integrated with an audio encoder and / or audio decoder, respectively, and may include a suitable MUX-DEMUX unit or other hardware and / or software to handle multiplexed streams containing both audio and video in a common data stream. Where applicable, the MUX-DEMUX unit may comply with the ITU H.223 Multiplexer Protocol or other protocols such as the User Datagram Protocol (UDP).

[0028]

[0041] The video encoder 200 and video decoder 300 can each be implemented as one or more suitable encoder and / or decoder circuits, or any combination thereof, including one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, etc. When the technique is partially implemented in software, the device may store instructions for the software in a suitable non-temporary computer-readable medium and execute those instructions in hardware using one or more processors to implement the technique of the Disclosure. Each of the video encoder 200 and video decoder 300 may comprise one or more encoders or decoders, any of which may be integrated as part of a composite encoder / decoder (CODEC) in each device. A device comprising the video encoder 200 and / or video decoder 300 may include an integrated circuit, a microprocessor, and / or a wireless communication device such as a cellular telephone.

[0029]

[0042] The video encoder 200 and video decoder 300 may operate in accordance with video coding standards such as ITU-T H.265, also known as High Efficiency Video Coding (HEVC), or its extensions such as the Multiview and / or Scalable Video Coding Extension. Alternatively, the video encoder 200 and video decoder 300 may operate in accordance with other proprietary or industry standards, such as General Purpose Video Coding (VVC). A draft of the VVC standard is described in Bross et al., "Versatile Video Coding (Draft 9)," JVET-R2001-v8 (hereinafter, "VVC Draft 9"), 18th meeting of the Joint Video Expert Team (JVET) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11, April 15-24. However, the techniques of this disclosure are not limited to any particular coding standard.

[0030]

[0043] Generally, the video encoder 200 and video decoder 300 can perform block-based coding of pictures. The term “block” generally refers to a structure containing data to be processed (e.g., to be encoded, decoded, or otherwise used in the encoding and / or decoding process). For example, a block may contain a two-dimensional matrix of samples of luminance and / or chrominance data. Generally, the video encoder 200 and video decoder 300 can code video data represented in YUV (e.g., Y, Cb, Cr) format. That is, rather than coding red, green, and blue (RGB) data for the samples of a picture, the video encoder 200 and video decoder 300 can code luminance and chrominance components, where the chrominance component may include both red and blue chrominance components. In some examples, the video encoder 200 converts the received RGB format data to a YUV representation before encoding, and the video decoder 300 converts the YUV representation to RGB format. Alternatively, pre-processing and post-processing units (not shown) may perform these conversions.

[0031]

[0044] This disclosure may refer to coding a picture (e.g., encoding and decoding) to include, in general, the process of encoding or decoding the data of a picture. Similarly, this disclosure may refer to coding a block of a picture to include, for example, the process of encoding or decoding data about a block, for example, predictive and / or residual coding. An encoded video bitstream generally contains a set of values ​​for syntax elements that represent coding decisions (e.g., coding modes) and divisions of the picture into blocks. Thus, references to coding a picture or a block should generally be understood as coding values ​​for the syntax elements that make up the picture or block.

[0032]

[0045] HEVC defines various blocks, including coding units (CUs), prediction units (PUs), and transformation units (TUs). According to HEVC, a video coder (such as video encoder 200) divides the coding tree units (CTUs) into CUs according to a quad-tree structure. That is, the video coder divides the CTUs and CUs into four equal, non-overlapping squares, and each node in the quad-tree has either zero or four child nodes. Nodes without child nodes are sometimes called "leaf nodes," and the CU of such a leaf node may contain one or more PUs and / or one or more TUs. The video coder may further divide the PUs and TUs. For example, in HEVC, the residual quad-tree (RQT) represents a division of the TUs. In HEVC, PUs represent intra-predicted data, while TUs represent residual data. Intra-predicted CUs contain intra-predicted information, such as intra-mode indications.

[0033]

[0046] As another example, a video encoder 200 and a video decoder 300 may be configured to operate according to VVC. According to VVC, a video coder (such as the video encoder 200) divides a picture into multiple coding tree units (CTUs). The video encoder 200 may divide the CTUs according to a tree structure, such as a quad-tree binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure eliminates the concept of multiple division types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels: a first level divided according to quad-tree division and a second level divided according to binary tree division. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to coding units (CUs).

[0034]

[0047] In an MTT partitioning structure, blocks can be partitioned using quad-tree (QT) partitions, binary-tree (BT) partitions, and one or more types of triple-tree (TT) (also called terminally-tree (TT)) partitions. A triple-tree or terminally-tree partition is a partition in which a block is split into three sub-blocks. In some examples, a triple-tree or terminally-tree partition divides a block into three sub-blocks without splitting the original block through a center. The partitioning types in an MTT (e.g., QT, BT, and TT) can be symmetrical or asymmetrical.

[0035]

[0048] In some examples, the video encoder 200 and video decoder 300 may use a single QTBT or MTT structure to represent the luminance component and the chrominance component, respectively, while in other examples, the video encoder 200 and video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luminance component and another QTBT / MTT structure for both chrominance components (or two QTBT / MTT structures for each chrominance component).

[0036]

[0049] The video encoder 200 and video decoder 300 may be configured to use a quad-tree segment, QTBT segment, MTT segment, or other segmentation structure that conforms to HEVC. For illustrative purposes, the description of the techniques of this disclosure will be presented in relation to the QTBT segment. However, it should be understood that the techniques of this disclosure may also be applicable to video coders configured to use a quad-tree segment, or similarly other types of segmentation.

[0037]

[0050] In some examples, the CTU includes a coding tree block (CTB) of a lumen sample, two corresponding CTBs of a chroma sample for a picture with three sample arrays, or a CTB of a sample for a monochrome picture or a picture coded using three separate color planes and syntax structures used to code the sample. The CTB can be an N×N block of samples for some value N such that the division of components into the CTB is a partition. The components can be an array or a single sample from one of the three arrays (lumen and two chroma) for a picture in a 4:2:0, 4:2:2, or 4:4:4 color format, or an array or a single sample of an array for a picture in a monochrome format. In some examples, the coding block is an M×N block of samples for some values ​​M and N such that the division of the CTB into the coding block is a partition.

[0038]

[0051] Blocks (e.g., CTUs or CUs) can be grouped in various ways within a picture. For example, a brick may refer to a rectangular area of ​​a row of CTUs within a particular tile in a picture. A tile can be a rectangular area of ​​CTUs within a particular tile column and a particular tile row in a picture. A tile column refers to a rectangular area of ​​CTUs having a height equal to the height of the picture and a width specified by a syntax element (e.g., in a picture parameter set). A tile row refers to a rectangular area of ​​CTUs having a height specified by a syntax element (e.g., in a picture parameter set) and a width equal to the width of the picture.

[0039]

[0052] In some examples, a tile may be divided into multiple bricks, each of which may contain one or more CTU rows. A tile that is not divided into multiple bricks may also be called a brick. However, a brick that is a true subset of a tile may not be called a tile.

[0040]

[0053] Bricks within a picture can also be placed within a slice. A slice can be an integer number of bricks in a picture, which may be contained exclusively within a single Network Abstraction Layer (NAL) unit. In some examples, a slice may contain either several complete tiles or just a continuous sequence of complete bricks of a single tile.

[0041]

[0054] This disclosure may use "N×N (NxN)" and "N×N (N by N)" interchangeably to refer to the sample dimensions of a block (such as a CU or other video block) with respect to vertical and horizontal dimensions, for example, 16×16 samples or 16×16 samples. Generally, a 16×16 CU has 16 samples vertically (y=16) and 16 samples horizontally (x=16). Similarly, an N×N CU generally has N samples vertically and N samples horizontally, where N represents a non-negative integer. Samples in a CU can be arranged in rows and columns. Furthermore, a CU does not necessarily have to have the same number of samples horizontally as vertically. For example, a CU may have N×M samples, where M is not necessarily equal to N.

[0042]

[0055] The video encoder 200 encodes video data for CUs, which represent prediction and / or residual information, as well as other information. The prediction information indicates how the CU should be predicted in order to form a prediction block for the CU. The residual information generally represents the sample-by-sample difference between the CU sample before encoding and the prediction block.

[0043]

[0056] To predict a CU, the video encoder 200 may generally form prediction blocks for the CU through inter-prediction or intra-prediction. Inter-prediction generally refers to predicting the CU from data of a previously coded picture, while intra-prediction generally refers to predicting the CU from data of the same picture that was coded before. To perform inter-prediction, the video encoder 200 may generate prediction blocks using one or more motion vectors. The video encoder 200 may generally perform motion search to identify a reference block that perfectly matches the CU with respect to the difference between the CU and the reference block. The video encoder 200 may compute a difference metric using absolute difference sum (SAD), squared difference sum (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculations to determine whether the reference block currently perfectly matches the CU. In some examples, the video encoder 200 may predict the current CU using unidirectional or bidirectional prediction.

[0044]

[0057] Some examples of VVC also offer an affine motion compensation mode, which can be considered an interpredictive mode. In affine motion compensation mode, the video encoder 200 may determine two or more motion vectors representing non-translational motion, such as zooming in or out, rotation, perspective motion, or other anomalous motion types.

[0045]

[0058] To perform intra-prediction, the video encoder 200 may select an intra-prediction mode to generate a prediction block. Several examples of VVCs offer 67 intra-prediction modes, including various directional modes, as well as planar and DC modes. Generally, the video encoder 200 selects an intra-prediction mode that describes adjacent samples to the current block (e.g., a block of CU) from which samples of the current block should be predicted. Such samples could generally be above, to the upper left, or to the left of the current block in the same picture as the current block, assuming the video encoder 200 codes CTU and CU in raster scan order (left to right, top to bottom).

[0046]

[0059] The video encoder 200 encodes data representing the prediction mode for the current block. For example, in interprediction mode, the video encoder 200 may encode data representing which of the various available interprediction modes is used, as well as motion information for the corresponding mode. For example, in unidirectional or bidirectional interprediction, the video encoder 200 may encode motion vectors using advanced motion vector prediction (AMVP) or merge mode. The video encoder 200 may use similar modes to encode motion vectors for affine motion compensation mode.

[0047]

[0060] Following predictions such as intra-prediction or inter-prediction of a block, the video encoder 200 may compute residual data for the block. Residual data, such as residual blocks, represents the sample-by-sample difference between the block and the predicted block for the block, formed using the corresponding prediction mode. The video encoder 200 may apply one or more transformations to the residual blocks to generate transformation data in the transformation region rather than the sample region. For example, the video encoder 200 may apply a discrete cosine transform (DCT), integer transform, wavelet transform, or a conceptually similar transform to the residual video data. Furthermore, the video encoder 200 may apply a quadratic transform such as a mode-dependent non-separable secondary transform (MDNSST), signal-dependent transform, or Carunen-Löwe ​​transform (KLT) following the first transformation. The video encoder 200 generates transformation coefficients following the application of one or more transformations.

[0048]

[0061] As described above, following any transformation to generate the transformation coefficients, the video encoder 200 may perform quantization of the transformation coefficients. Quantization generally refers to the process of further compression in which the transformation coefficients are quantized to reduce the amount of data used to represent them as much as possible. By performing the quantization process, the video encoder 200 may reduce the bit depth associated with some or all of the transformation coefficients. For example, the video encoder 200 may round an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder 200 may perform a bitwise right shift of the value to be quantized.

[0049]

[0062] Following quantization, the video encoder 200 may scan the transformation coefficients to generate a one-dimensional vector from a two-dimensional matrix containing the quantized transformation coefficients. The scan may be designed to place higher-energy (and therefore lower-frequency) coefficients at the beginning of the vector and lower-energy (and therefore higher-frequency) transformation coefficients at the end. In some examples, the video encoder 200 may utilize a predefined scan order to scan the quantized transformation coefficients to generate a serialized vector, and then entropically encode the quantized transformation coefficients of the vector. In other examples, the video encoder 200 may perform adaptive scanning. After scanning the quantized transformation coefficients to form a one-dimensional vector, the video encoder 200 may entropically encode the one-dimensional vector, for example, according to context-adaptive binary arithmetic coding (CABAC). The video encoder 200 may also entropically encode values ​​for syntax elements describing metadata associated with the encoded video data for use by the video decoder 300 when decoding the video data.

[0050]

[0063] To perform CABAC, the video encoder 200 may assign a context from the context model to the symbol to be transmitted. The context may relate, for example, to whether the adjacent value of the symbol is zero. Probability decisions may be based on the context assigned to the symbol.

[0051]

[0064] The video encoder 200 may further generate syntax data for the video decoder 300, such as block-based syntax data, picture-based syntax data, and sequence-based syntax data, in other syntax data such as picture headers, block headers, slice headers, or sequence parameter sets (SPS), picture parameter sets (PPS), or video parameter sets (VPS). The video decoder 300 may similarly decode such syntax data to determine how the corresponding video data should be decoded.

[0052]

[0065] In this way, the video encoder 200 can generate a bitstream containing syntax elements that describe the encoded video data, for example, the division of a picture into blocks (e.g., CUs), and predictive and / or residual information for the blocks. Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0053]

[0066] Generally, the video decoder 300 performs the reverse process of what the video encoder 200 did to decode the encoded video data of the bitstream. For example, the video decoder 300 may decode values ​​for syntax elements of the bitstream using CABAC in a substantially similar manner to, but in reverse, the CABAC encoding process of the video encoder 200. The syntax elements may define partitioning information for partitioning a picture into CTUs, and partitions for each CTU, following a corresponding partitioning structure such as a QTBT structure, in order to define the CUs of the CTUs. The syntax elements may further define prediction and residual information for blocks of video data (e.g., CUs).

[0054]

[0067] Residual information may be represented, for example, by quantized transformation coefficients. The video decoder 300 may dequantize and inverse transform the quantized transformation coefficients of a block in order to reconstruct the residual block for the block. The video decoder 300 uses a signaled prediction mode (intra or inter-prediction) and associated prediction information (for example, motion information for inter-prediction) to form a prediction block for the block. The video decoder 300 may then combine the prediction block and the residual block (sample by sample) to reconstruct the original block. The video decoder 300 may perform additional processing, such as performing a deblocking process to reduce visual artifacts along the block boundaries.

[0055]

[0068] This disclosure may generally refer to “signaling” certain information, such as syntax elements. The term “signaling” may generally refer to the communication of values ​​about syntax elements and / or other data used to decode encoded video data. That is, the video encoder 200 may signal values ​​about syntax elements in the bitstream. Generally, signaling refers to generating values ​​in the bitstream. As described above, the source device 102 may transfer the bitstream to the destination device 116 in substantially real time, or non-real time, such as when storing the syntax elements in the storage device 112 for later retrieval by the destination device 116.

[0056]

[0069] Figure 2 is a conceptual diagram illustrating the hybrid video coding framework. Video coding standards since H.261 have been based on the so-called hybrid video coding principle shown in Figure 2. The term "hybrid" refers to the combination of two means for reducing redundancy in a video signal, namely, combining predictive and transformative coding with the quantization of predictive residuals. Predictive and transformative coding reduce redundancy in the video signal by decorrelating, while quantization reduces the data in the transformative coefficient representation by lowering their precision, ideally by removing only irrelevant details. This hybrid video coding design principle is also used in two recent standards, ITU-T H.265 / HEVC and ITU-T H.266 / VVC.

[0057]

[0070] As shown in Figure 2, a modern hybrid video coder 130 generally includes block segmentation, motion-compensated or inter-picture prediction, intra-picture prediction, transformation, quantization, entropy coding, and post / in-loop filtering. In the example in Figure 2, the video coder 130 includes a totalization unit 134, a transformation unit 136, a quantization unit 138, an entropy coding unit 140, an inverse quantization unit 142, an inverse transformation unit 144, a totalization unit 146, a loop filter unit 148, a decoded picture buffer (DPB) 150, an intra-prediction unit 152, an inter-prediction unit 154, and a motion estimation unit 156.

[0058]

[0071] Generally, when a video coder 130 encodes video data, it can receive input video data 132. Block partitioning is used to divide the received picture (image) of the video data into smaller blocks for operation in the prediction and transformation process. Early video coding standards used a fixed block size, typically 16x16 samples. Recent standards such as HEVC and VVC employ a tree-based partitioning structure to provide flexible partitioning.

[0059]

[0072] The motion estimation unit 156 and the inter-prediction unit 154 can predict the input video data 132 from, for example, the previously decoded DPB 150 data. Motion-compensated or inter-picture prediction utilizes the redundancy present between (and thus "inter") pictures in the video sequence. All modern video codecs use block-based motion compensation, which, according to block-based motion compensation, allows predictions to be obtained from one or more previously decoded pictures, i.e., reference pictures. The corresponding area for generating inter-predictions is indicated by motion information, which includes motion vectors and reference picture indices.

[0060]

[0073] The summing unit 134 may calculate residual data as the difference between the input video data 132 and the predicted data from the intra-prediction unit 152 or the inter-prediction unit 154. The summing unit 134 provides the residual block to the transformation unit 136, which applies one or more transformations to the residual block to generate a transformation block. The quantization unit 138 quantizes the transformation block to form quantized transformation coefficients. The entropy coding unit 140 entropy encodes the quantized transformation coefficients, as well as other syntax elements such as motion information and intra-prediction information, to generate an output bitstream 158.

[0061]

[0074] Meanwhile, the inverse quantization unit 142 inversely quantizes the quantized transformation coefficients, and the inverse transformation unit 144 inversely transforms the transformation coefficients to reconstruct the residual blocks. The summing unit 146 combines the residual blocks with the prediction blocks (for each sample) to generate decoded blocks of video data. The loop filter unit 148 applies one or more filters (for example, at least one of a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter) to the decoded blocks to generate filtered decoded blocks.

[0062]

[0075] Blocks of video data, such as CTU and CU, can actually contain multiple color components, for example, a luminance or "luma" component, a blue chrominance or "chroma" component, and a red chrominance (chroma) component. The luma component may have a greater spatial resolution than the chroma component, and one of the chroma components may have a greater spatial resolution than the other chroma component. Alternatively, the luma component may have a greater spatial resolution than the chroma component, and the two chroma components may have equal spatial resolution to each other. For example, in a 4:2:2 format, the luma component may be twice as large as the chroma component horizontally and equal to the chroma component vertically. As another example, in a 4:2:0 format, the luma component may be twice as large as the chroma component horizontally and vertically. The various behaviors discussed above can generally be applied individually to each of the rumer and chroma components (however, some coding information, such as motion information and intra-prediction direction, can be determined for the rumer component and inherited by the corresponding chroma component).

[0063]

[0076] According to the techniques of this disclosure, a loop filter unit 148 may receive from a summing unit 146 a first color component having a first size (e.g., luminance or "luma" component) and a second color component having a second size smaller than the first size (e.g., blue or red chrominance or "chroma" component). A common block of video data may contain both the first and second color components. The loop filter unit 148 may be configured to apply a downsampled convolutional neural network layer to the first color component of a block of video data to generate a downsampled first color component having a second size, i.e., downsample the first color component to the size of the second color component. In one example, the first color component may be a luma component and the second color component may be one of two chroma components. In another example, the first color component may be a first chroma component and the second color component may be a second chroma component.

[0064]

[0077] In yet another example, the loop filter unit 148 may receive a lumar component and each of both chroma components, the lumar component having a first size, the first chroma component having a second size smaller than the first size, and the second chroma component having a third size smaller than the second size. In this example, the loop filter unit may apply a downsampled convolutional neural network filter to both the lumar component and the first chroma component to generate a downsampled lumar component having a third size and a downsampled first chroma component having a third size.

[0065]

[0078] The loop filter unit 148 may also filter the second color component using, for example, a convolutional neural network filter to form a filtered second color component. The loop filter unit 148 may then concatenate the downsampled first color component with the filtered second color component to form a concatenated color component. The loop filter unit 148 may then filter the concatenated color component using, for example, a convolutional neural network filter to form a filtered concatenated component that includes a filtered downsampled first color component.

[0066]

[0079] In particular, the first and second color components may originally be stored in separate arrays or matrices. To concatenate the color components, the loop filter unit 148 may form a single array or matrix that is twice the width or twice the height of the individual color components. The loop filter unit 148 may then store a sample of the first color component in a first region of the newly formed array or matrix, and a sample of the second color component in a second neighboring region of the newly formed array or matrix. When three color components are used (for example, luma, blue-hue chroma, and red-hue chroma), the loop filter unit 148 may form a single array or matrix that is three times the width or height of the individual color components, and store a sample of each of the three color components in the respective regions of the newly formed array or matrix.

[0067]

[0080] As mentioned above, the loop filter unit 148 can receive the lumar and chroma components, respectively. The loop filter unit 148 may filter both the downsampled first and second chroma components using, for example, a convolutional neural network filter to produce a filtered downsampled first chroma component and a filtered second chroma component. The loop filter unit 148 may then concatenate the downsampled lumar component with the filtered downsampled first chroma component and the filtered second chroma component to form a concatenated color component. The loop filter unit 148 may then filter the concatenated color component (including the downsampled lumar component, the filtered downsampled first chroma component, and the filtered second chroma component) using, for example, a convolutional neural network filter.

[0068]

[0081] After filtering the concatenated color components, the loop filter unit 148 may further upsample the filtered, downsampled first color component back to the first size (i.e., the original size of the first color component). If the loop filter unit 148 also downsamples the second color component (for example, from the second size to the third size), the loop filter unit 148 may also upsample the filtered second color component back to the second size (i.e., the original size of the second color component).

[0069]

[0082] Figure 3 is a conceptual diagram showing a hierarchical prediction structure 166 using a group of pictures (GOP) size 16. In recent video codecs, a hierarchical prediction structure within the group of pictures (GOP) is applied to improve coding efficiency.

[0070]

[0083] Referring again to Figure 2, in-picture prediction leverages spatial redundancy present within the picture (and thus "intra") by deriving predictions about blocks from spatially neighboring (reference) samples that have already been coded / decoded. The most recent video codecs, including AVC, HEVC, and VVC, use directional angle prediction, DC prediction, and plane (or planar) prediction.

[0071]

[0084] Hybrid video coding standards apply a block transform to the prediction residual (regardless of whether the prediction residual comes from inter-picture prediction or intra-picture prediction). Early standards, including H.261, H.262, and H.263, employ the discrete cosine transform (DCT). HEVC and VVC apply more transform kernels in addition to the DCT to handle different statistics in a given video signal.

[0072]

[0085] Quantization aims to reduce the precision of input values ​​in order to decrease the amount of data required to represent an input value or set of input values. In hybrid video coding, quantization is typically applied to individual transformed residual samples, or transformation coefficients, resulting in integer coefficient levels. In recent video coding standards, the step size is derived from so-called quantization parameters (QPs) that control fidelity and bitrate. A larger step size results in a lower bitrate but also degrades quality, which can lead to, for example, video pictures exhibiting blocking artifacts and blurred details.

[0073]

[0086] Context-Adaptive Binary Arithmetic Coding (CABAC) is a form of entropy coding used in modern video codecs such as AVC, HEVC, and VVC due to its high efficiency.

[0074]

[0087] Post- / in-loop filtering is a filtering process (or a combination of such processes) applied to a reconstructed picture to reduce coding artifacts. The input to the filtering process is typically the reconstructed picture, which is a combination of the reconstructed residual signal (including quantization errors) and the prediction. As shown in Figure 2, the reconstructed picture after in-loop filtering is stored and used as a reference for inter-picture predictions of subsequent pictures. Coding artifacts are primarily determined by QP, and therefore QP information is commonly used in the design of the filtering process. In HEVC, in-loop filtering includes deblocking filtering and sample-adaptive offset (SAO) filtering. The VVC standard introduced adaptive loop filtering (ALF) as a third filter. The ALF filtering process is shown below.

[0075] R'(i,j)=R(i,j)+((Σ k≠0 Σ l≠0 f(k,l)×K(R(i+k,j+l)-R(i,j),c(k,l))+64)>>7) (1) Here, R(i,j) is the set of samples before the filtering process, and R'(i,j) is the sample value after the filtering process. f(k,l) represents the filter coefficients, K(x,y) is the clipping function, and c(k,l) represents the clipping parameters. The variables k and l are:

[0076]

number

[0077] and

[0078]

number

[0079] It fluctuates between and , where L represents the filter length. The clipping function K(x,y) = min(y,max(-y,x)), which corresponds to the function Clip3(-y,y,x). The clipping operation makes ALF more efficient by introducing nonlinearity by reducing the influence of neighboring sample values ​​that are excessively different from the current sample value. In VVC, filtering parameters can be signaled in the bitstream and can be selected from a predefined set of filters. The ALF filtering process can also be summarized using the following equation:

[0080] R'(i,j)=R(i,j)+ALF_residual_ouput(R) (2)

[0088] Figures 4A and 4B are conceptual diagrams illustrating exemplary joint-component convolutional neural network (CNN)-based filters having a residual network (ResNet) that includes residual blocks. Specifically, Figure 4A depicts a system 170 that includes a convolutional neural network (convolutional NN) filter 172, a concatenation unit 174, a series of residual blocks 176, residual blocks 178A, 178B (residual block 178), and channel-unit summation units 179A, 179B (channel-unit summation unit 179). Figure 4B depicts an exemplary residual block 180. Either residual block 176 or 178 may contain components similar to those of residual block 180. In the example of Figure 4B, residual block 180 includes convolutional NN filters 182 and 186, a rectified linear unit (ReLU) layer 184, and a summation unit 188.

[0081]

[0089] In the example in Figure 4A, the convolutional NN filter 172 upsamples the chroma components (i.e., the blue (Cb) and red (Cr) components) to the size of the corresponding luminance (luma (Y)) component. The coupling unit 174 couples the upsampled chrominance sample with the luma sample so that it is used as input to the NN-based filter formed by the residual block 176. The filtering process performed by the residual block 176 can also be generalized as follows:

[0082] R'(i,j)=R(i,j)+NN_filter_residual_ouput(R) (3) The loop filters of the video encoder 200 and video decoder 300 may include components similar to those of system 170 in Figure 4A. The model structure and model parameters of the NN-based filter can be predefined and stored in the video encoder 200 and video decoder 300. Additionally or alternatively, the video encoder 200 may signal data representing the NN-based filter in the bitstream.

[0083]

[0090] After residual block 176 applies an NN-based filter to the concatenated components, residual blocks 178A and 178B may apply channel-specific filtering to individual color components, and then channel-unit summation units 179A and 179B may separate the components into their respective color channels Y, Cb, and Cr.

[0084]

[0091] In the example in Figure 4B, the convolutional NN filter 182 may first filter the received video data samples. The convolutional NN filters 182 and 186 may be 3 × 3 × K × K convolutional NN filters. The residual block 180 may also be called the “residual processing unit”. The ReLU layer 184 may provide an activation function that, for each sample, outputs the sample value itself if the sample value is greater than 0, and otherwise outputs 0 as the sample value if the input sample value is less than 0. That is, the ReLU activation function can be summarized as follows:

[0085]

number

[0086]

[0092] The convolutional NN filter 186 may then filter the resulting sample values ​​from the ReLU layer 184 and provide the filtered sample values ​​to the summing unit 188. The summing unit 188 may add the filtered sample values ​​to the corresponding input sample values.

[0087]

[0093] This disclosure acknowledges that, in some cases, upsampling chroma blocks to the size of the corresponding ruma blocks may result in an increase in the size of the neural network and thus an increase in computational complexity.

[0088]

[0094] The video encoder 200 and video decoder 300 in Figure 1, or the video coder 130 in Figure 2, may be configured to perform any or all of the following techniques, individually or in any combination, by the techniques of the present disclosure. Generally, the video encoder 200, video decoder 300, and video coder 130 may first apply convolutional neural network (CNN) layers separately to some or all of the color components to match the sizes of the color components. The video encoder 200 and video decoder 300 may set the size of the output neural (or tensor) block to be equal to the size of the smallest color component block among all the color components. The video encoder 200 and video decoder 300 may then apply a joint convolutional NN layer to all components of the same size. As a specific and general useful application example, in the case of a video sequence in a 4:2:0 color format, the chroma component block is downsampled using a convolutional NN layer with stride 2 in both the horizontal and vertical directions. The output of the convolutional NN layer may have the same block size as the chroma component. Next, the video encoder 200 and video decoder 300 can apply a joint convolutional NN layer to all color component data as input.

[0089]

[0095] Figures 5A and 5B are conceptual diagrams showing an exemplary quad-tree binary tree (QTBT) structure 190 and its corresponding coding tree unit (CTU) 192. Solid lines represent quad-tree splitting, and dotted lines represent binary tree splitting. Each split (i.e., non-leaf) node in the binary tree is signaled with a flag to indicate which splitting type (i.e., horizontal or vertical) is used, where in this example, 0 indicates horizontal splitting and 1 indicates vertical splitting. In quad-tree splitting, the quad-tree node splits a block horizontally and vertically into four sub-blocks of equal size, so there is no need to indicate the splitting type. Therefore, the video encoder 200 can encode and the video decoder 300 can decode syntax elements (such as splitting information) for the region tree level (i.e., solid lines) of the QTBT structure 190 and syntax elements (such as splitting information) for the prediction tree level (i.e., dashed lines) of the QTBT structure 190. The video encoder 200 can encode and the video decoder 300 can decode video data such as prediction and transformation data for CUs represented by terminal leaf nodes of the QTBT structure 190.

[0090]

[0096] Generally, the CTU192 in Figure 5B can be associated with parameters that define the size of the blocks corresponding to the nodes of the QTBT structure 190 at the first and second levels. These parameters may include the CTU size (representing the size of the CTU192 in the sample), the minimum quad tree size (MinQTSize, representing the minimum allowable quad tree leaf node size), the maximum binary tree size (MaxBTSize, representing the maximum allowable binary tree root node size), the maximum binary tree depth (MaxBTDepth, representing the maximum allowable binary tree depth), and the minimum binary tree size (MinBTSize, representing the minimum allowable binary tree leaf node size).

[0091]

[0097] The root node of a QTBT structure corresponding to a CTU may have four child nodes at the first level of the QTBT structure, each of which may be subdivided according to a quad-tree partition. That is, a node at the first level is either a leaf node (without child nodes) or has four child nodes. An example of QTBT structure 190 represents a node that includes a parent node and child nodes with solid lines for branching. If a node at the first level is not larger than the maximum allowable binary tree root node size (MaxBTSize), the node may be further subdivided by its respective binary tree. Binary tree splitting of a node may be repeated until the nodes resulting from the split reach the minimum allowable binary tree leaf node size (MinBTSize) or the maximum allowable binary tree depth (MaxBTDepth). An example of QTBT structure 190 represents a node with dashed lines for branching. Binary tree leaf nodes are called coding units (CUs), and CUs are used for prediction (e.g., in-picture or between-picture prediction) and transformation without further subdivision. As explained above, CU is sometimes called a "video block" or "block".

[0092]

[0098] In one example of a QTBT partition structure, the CTU size is set to 128×128 (a chroma sample and two corresponding 64×64 chroma samples), MinQTSize is set to 16×16, MaxBTSize is set to 64×64, MinBTSize (for both width and height) is set to 4, and MaxBTDepth is set to 4. The quadtree partition is first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes ranging from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If a quadtree leaf node is 128×128, it is not further split by the binary tree because its size exceeds MaxBTSize (i.e., 64×64 in this example). In other cases, the quadtree leaf node may be further partitioned by the binary tree. Therefore, a quad-tree leaf node is also the root node for a binary tree and has a binary tree depth of 0. When the binary tree depth reaches MaxBTDepth (4 in this example), no further splitting is allowed. A binary tree node with a width equal to MinBTSize (4 in this example) implies that no further vertical splitting (i.e., splitting of width) is allowed for that binary tree node. Similarly, a binary tree node with a height equal to MinBTSize implies that no further horizontal splitting (i.e., splitting of height) is allowed for that binary tree node. As mentioned above, leaf nodes of a binary tree are called CUs and are further processed according to prediction and transformation without further division.

[0093]

[0099] Figure 6 is a block diagram illustrating an exemplary video encoder 200 capable of implementing the techniques of this disclosure. Figure 6 is provided for illustrative purposes only and should not be considered to limit the techniques broadly illustrated and described in this disclosure. For illustrative purposes, this disclosure describes the video encoder 200 in the context of video coding standards such as the ITU-T H.265 / HEVC video coding standard and the VVC video coding standard under development. However, the techniques of this disclosure are not limited to these video coding standards and are generally applicable to other video coding and decoding standards.

[0094]

[0100] In the example shown in Figure 6, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a conversion processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse conversion processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy coding unit 220. Any or all of the video data memory 230, the mode selection unit 202, the residual generation unit 204, the conversion processing unit 206, the quantization unit 208, the inverse quantization unit 210, the inverse conversion processing unit 212, the reconstruction unit 214, the filter unit 216, the DPB 218, and the entropy coding unit 220 may be implemented in one or more processors or processing circuits. For example, a unit of the video encoder 200 may be implemented as one or more circuits or logic elements, as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. Furthermore, the video encoder 200 may include additional or alternative processors or processing circuits to perform these and other functions.

[0095]

[0101] The video data memory 230 can store video data to be encoded by the components of the video encoder 200. The video encoder 200 can receive video data to be stored in the video data memory 230 from, for example, a video source 104 (Figure 1). The DPB 218 can act as a reference picture memory that stores reference video data for use in predicting subsequent video data by the video encoder 200. The video data memory 230 and the DPB 218 can be formed by any of various memory devices, including DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM®), or other types of memory devices. The video data memory 230 and the DPB 218 can be provided by the same memory device or by separate memory devices. In various examples, the video data memory 230 may be on-chip with the other components of the video encoder 200, as shown in the figure, or off-chip relative to those components.

[0096]

[0102] In this disclosure, references to video data memory 230 should not be interpreted as being limited to memory inside the video encoder 200 unless specifically described so, nor should they be interpreted as being limited to memory outside the video encoder 200 unless specifically described so. Rather, references to video data memory 230 should be understood as reference memory that stores video data received by the video encoder 200 for encoding (for example, video data for the current block to be encoded). Memory 106 in Figure 1 may also provide temporary storage of outputs from various units of the video encoder 200.

[0097]

[0103] The various units in Figure 6 are shown to help understand the operations performed by the video encoder 200. The units may be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Fixed-function circuits refer to circuits that provide a specific function and are preset in terms of the operations they may perform. Programmable circuits refer to circuits that can be programmed to perform various tasks and to provide flexible functionality in the operations they may perform. For example, a programmable circuit may execute software or firmware that operates the programmable circuit in a manner defined by software or firmware instructions. Fixed-function circuits may execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more of the units may be separate circuit blocks (fixed-function or programmable), and in some examples, one or more units may be integrated circuits.

[0098]

[0104] The video encoder 200 may include a programmable core formed from an arithmetic logic unit (ALU), an EFU, digital circuits, analog circuits, and / or programmable circuits. In an example where the operation of the video encoder 200 is carried out using software executed by the programmable circuits, memory 106 (Figure 1) may store software instructions (e.g., object code) that the video encoder 200 receives and executes, or another memory (not shown) within the video encoder 200 may store such instructions.

[0099]

[0105] The video data memory 230 is configured to store the received video data. The video encoder 200 can retrieve a picture of the video data from the video data memory 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data memory 230 may be raw video data to be encoded.

[0100]

[0106] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra-prediction unit 226. The mode selection unit 202 may include additional functional units for performing video prediction according to other prediction modes. For example, the mode selection unit 202 may include a pallet unit, an intra-block copy unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, and the like.

[0101]

[0107] The mode selection unit 202 generally coordinates multiple coding paths to test combinations of coding parameters and the resulting rate-distortion values ​​for such combinations. The coding parameters may include the division of the CTU to the CU, the prediction mode for the CU, the transformation type for the residual data of the CU, and the quantization parameters for the residual data of the CU. The mode selection unit 202 may ultimately select a combination of coding parameters that has a better rate-distortion value than other tested combinations.

[0102]

[0108] The video encoder 200 divides the picture retrieved from the video data memory 230 into a series of CTUs, and may encapsulate one or more CTUs within a slice. The mode selection unit 202 may divide the CTUs of a picture according to a tree structure, such as the HEVC QTBT structure or quad-tree structure described above. As described above, the video encoder 200 may form one or more CUs from dividing the CTUs according to a tree structure. Such CUs are sometimes generally referred to as “video blocks” or “blocks”.

[0103]

[0109] In general, the mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-prediction unit 226) to generate a predicted block for the current block (e.g., the current CU, or in HEVC, the overlapping portion of PU and TU). For intra-prediction of the current block, the motion estimation unit 222 may perform a motion search to identify one or more perfectly matching reference blocks in one or more reference pictures (e.g., one or more previously coded pictures stored in the DPB 218). In particular, the motion estimation unit 222 may calculate a value representing how similar the potential reference blocks are to the current block, for example, according to the sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), etc. The motion estimation unit 222 may generally perform these calculations using sample-by-sample differences between the current block and the reference blocks under consideration. The motion estimation unit 222 can identify the reference block with the lowest value obtained from these calculations, which indicates the reference block that best matches the current block.

[0104]

[0110] The motion estimation unit 222 may form one or more motion vectors (MVs) that define the position of a reference block in a reference picture relative to the position of a current block in the current picture. The motion estimation unit 222 may then provide the motion vectors to the motion compensation unit 224. For example, in unidirectional interpretation, the motion estimation unit 222 may provide a single motion vector, while in bidirectional interpretation, the motion estimation unit 222 may provide two motion vectors. The motion compensation unit 224 may then use the motion vectors to generate predicted blocks. For example, the motion compensation unit 224 may use the motion vectors to extract data for a reference block. As another example, if the motion vectors have partial sample accuracy, the motion compensation unit 224 may interpolate values ​​for the predicted block according to one or more interpolation filters. Furthermore, in bidirectional interpretation, the motion compensation unit 224 may extract data for the two reference blocks identified by each motion vector and combine the extracted data, for example, through sample-wise averaging or weighted averaging.

[0105]

[0111] As another example, for intra-prediction, or intra-prediction coding, the intra-prediction unit 226 may generate a prediction block from samples adjacent to the current block. For example, in directional mode, the intra-prediction unit 226 may mathematically combine the values ​​of adjacent samples and populate these calculated values ​​in a direction defined across the current block to generate a prediction block. As another example, in DC mode, the intra-prediction unit 226 may calculate the average of adjacent samples relative to the current block and generate a prediction block so that each sample of the prediction block contains this obtained average.

[0106]

[0112] The mode selection unit 202 provides the prediction block to the residual generation unit 204. The residual generation unit 204 receives a raw, uncoded version of the current block from the video data memory 230 and the prediction block from the mode selection unit 202. The residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block for the current block. In some examples, the residual generation unit 204 may also determine the difference between sample values ​​in the residual block to generate the residual block using residual difference pulse code modulation (RDPCM). In some examples, the residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.

[0107]

[0113] In an example where the mode selection unit 202 divides a CU into PUs, each PU may be associated with a lumar prediction unit and a corresponding chroma prediction unit. The video encoder 200 and video decoder 300 may support PUs of various sizes. As shown above, the size of a CU may refer to the size of the lumar coding block of the CU, and the size of a PU may refer to the size of the lumar prediction unit of the PU. Assuming that the size of a particular CU is 2N×2N, the video encoder 200 may support PU sizes of 2N×2N or N×N for intra-prediction, and symmetric PU sizes of 2N×2N, 2N×N, N×2N, N×N, or similar for inter-prediction. The video encoder 200 and video decoder 300 may also support asymmetric divisions of PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-prediction.

[0108]

[0114] In an example where the mode selection unit 202 does not further subdivide the CUs into PUs, each CU may be associated with a ruma coding block and a corresponding chroma coding block. As described above, the size of a CU may refer to the size of the ruma coding block of the CU. The video encoder 200 and video decoder 300 may support CU sizes of 2N×2N, 2N×N, or N×2N.

[0109]

[0115] In some examples, such as intra-block copy mode coding, affine mode coding, and linear model (LM) mode coding, the mode selection unit 202 generates a predicted block for the current block being coded via the respective unit associated with the coding technique. In some examples, such as palette mode coding, the mode selection unit 202 may not generate a predicted block, but instead may generate syntax elements indicating the form in which the block should be reconstructed based on the selected palette. In such modes, the mode selection unit 202 may provide these syntax elements to be coded to the entropy coding unit 220.

[0110]

[0116] As described above, the residual generation unit 204 receives video data for the current block and the corresponding predicted block. The residual generation unit 204 then generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the predicted block and the current block.

[0111]

[0117] The transformation processing unit 206 applies one or more transformations to the residual block to generate blocks of transformation coefficients (referred to herein as “transformation coefficient blocks”). The transformation processing unit 206 may apply various transformations to the residual block to form the transformation coefficient blocks. For example, the transformation processing unit 206 may apply a discrete cosine transform (DCT), a directional transform, a Carunenlebe transform (KLT), or a conceptually similar transform to the residual block. In some examples, the transformation processing unit 206 may perform multiple transformations on the residual block, such as linear and quadratic transformations, including a rotation transform. In some examples, the transformation processing unit 206 does not apply any transformations to the residual block.

[0112]

[0118] The quantization unit 208 may quantize the transformation coefficients in the transformation coefficient block in order to generate a quantized transformation coefficient block. The quantization unit 208 may quantize the transformation coefficients of the transformation coefficient block according to the quantization parameter (QP) value associated with the current block. The video encoder 200 may adjust the degree of quantization applied to the transformation coefficient block associated with the current block by adjusting the QP value associated with the CU (for example, via the mode selection unit 202). Quantization may result in a loss of information, and therefore the quantized transformation coefficients may have lower precision than the original transformation coefficients generated by the transformation processing unit 206.

[0113]

[0119] The inverse quantization unit 210 and the inverse transform processing unit 212 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficient block in order to reconstruct the residual block from the transform coefficient block. The reconstruction unit 214 may generate a reconstructed block corresponding to the current block (with some potential distortion) based on the reconstructed residual block and the predicted block generated by the mode selection unit 202. For example, the reconstruction unit 214 may add samples from the reconstructed residual block to the corresponding samples from the predicted block generated by the mode selection unit 202 in order to generate the reconstructed block.

[0114]

[0120] The filter unit 216 may perform one or more filtering operations on the reconstructed blocks. For example, the filter unit 216 may perform a deblocking operation to reduce blocking artifacts along the edges of the CU. The operation of the filter unit 216 may be skipped in some examples. The filter unit 216 may be configured to perform various techniques of the present disclosure, for example, to resize color components to the smallest color component size and then apply a convolutional neural network (CNN) filter 232 to the resized color components.

[0115]

[0121] In particular, the filter unit 216 may reduce the size of the chroma component to the smaller of the two chroma components by applying one or more of the CNN filters 232, rather than upsampling the chroma component to the size of the corresponding chroma component. That is, the filter unit 216 may receive a chroma component having a first size and a chroma component having a second size smaller than the first size. The filter unit 216 may downsample the chroma component to the second size by applying the convolutional NN filter of the CNN filter 232 to the chroma component. The filter unit 216 may filter the chroma component to form a filtered chroma component. The filter unit 216 may then concatenate the downsampled chroma component with the filtered chroma component to form a concatenated color component, and then filter the concatenated color component to form a filtered concatenated component containing a filtered downsampled first color component. Furthermore, the filter unit 216 can upsample the filtered rumor component of the second size back to the first size and then store the rumor component in the DPB 218.

[0116]

[0122] In particular, the first and second color components may originally be stored in separate arrays or matrices. To concatenate the color components, the filter unit 216 may form a single array or matrice with twice the width or height of the individual color components. The filter unit 216 may then store a sample of the first color component in a first region of the newly formed array or matrice, and a sample of the second color component in a second neighboring region of the newly formed array or matrice. When three color components are used (for example, luma, blue-hue chroma, and red-hue chroma), the filter unit 216 may form a single array or matrice with three times the width or height of the individual color components, and store a sample of each of the three color components in the respective regions of the newly formed array or matrice.

[0117]

[0123] The video encoder 200 stores the reconstructed blocks in the DPB 218. For example, in cases where the operation of the filter unit 216 is not required, the reconstruction unit 214 may store the reconstructed blocks in the DPB 218. In cases where the operation of the filter unit 216 is required, the filter unit 216 may store the filtered reconstructed blocks in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 may retrieve a reference picture formed from the reconstructed (and potentially filtered) blocks from the DPB 218 to interpret blocks of the picture to be encoded later. Furthermore, the intraprediction unit 226 may use the reconstructed blocks in the DPB 218 of the current picture to intrapret other blocks in the current picture.

[0118]

[0124] In general, the entropy coding unit 220 can entropy code syntax elements received from other functional components of the video encoder 200. For example, the entropy coding unit 220 can entropy code quantized transformation coefficient blocks from the quantization unit 208. As another example, the entropy coding unit 220 can entropy code prediction syntax elements from the mode selection unit 202 (e.g., motion information for inter-prediction, or intra-mode information for intra-prediction). The entropy coding unit 220 can perform one or more entropy coding operations on syntax elements, which are another example of video data, to generate entropy coded data. For example, the entropy coding unit 220 may perform context-adaptive variable-length coding (CAVLC) operation, CABAC operation, variable-to-variable (V2V) length coding operation, syntax-based context-adaptive binary arithmetic coding (SBAC) operation, probability interval partitioned entropy (PIPE) coding operation, exponential Golomb coding operation, or another type of entropy coding operation on the data. In some examples, the entropy coding unit 220 may operate in a bypass mode in which syntax elements are not entropically coded.

[0119]

[0125] The video encoder 200 may output a bitstream containing entropy coding syntax elements required to reconstruct the slice or block of the picture. In particular, the entropy coding unit 220 may output a bitstream.

[0120]

[0126] The behavior described above is described in relation to blocks. Such descriptions should be understood as behavior for rumacoding blocks and / or chromacoding blocks. As described above, in some examples the rumacoding block and chromacoding block are the ruma and chroma components of the CU. In some examples the rumacoding block and chromacoding block are the ruma and chroma components of the PU.

[0121]

[0127] In some cases, actions performed for a rumacoding block do not need to be repeated for a chromacoding block. For example, actions to identify the motion vector (MV) and reference picture for a rumacoding block do not need to be repeated to identify the MV and reference picture for a chromablock. Rather, the MV for the rumacoding block may be scaled to determine the MV for the chromablock, and the reference picture may be the same. In another example, an intra-prediction process may be the same for both rumacoding and chromacoding blocks.

[0122]

[0128] Figure 7 is a block diagram showing an exemplary video decoder 300 capable of implementing the techniques of this disclosure. Figure 7 is provided for illustrative purposes only and is not intended to limit the techniques broadly illustrated and described in this disclosure. For illustrative purposes, this disclosure describes the video decoder 300 in accordance with the VVC and HEVC (ITU-T H.265) techniques. However, the techniques of this disclosure may be implemented by video coding devices configured to other video coding standards.

[0123]

[0129] In the example shown in Figure 7, the video decoder 300 includes a coded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transformation processing unit 308, a reconstruction unit 310, a filter unit 312, and a decoded picture buffer (DPB) 314. Any or all of the CPB memory 320, the entropy decoding unit 302, the prediction processing unit 304, the inverse quantization unit 306, the inverse transformation processing unit 308, the reconstruction unit 310, the filter unit 312, and the DPB 314 may be implemented in one or more processors or processing circuits. For example, the units of the video decoder 300 may be implemented as one or more circuits or logic elements, as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. Furthermore, the video decoder 300 may include additional or alternative processors or processing circuits to perform these and other functions.

[0124]

[0130] The prediction processing unit 304 includes a motion compensation unit 316 and an intra-prediction unit 318. The prediction processing unit 304 may include additional units for performing predictions according to other prediction modes. For example, the prediction processing unit 304 may include a pallet unit, an intra-block copy unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, and the like. In other examples, the video decoder 300 may include more, fewer, or different functional components.

[0125]

[0131] The CPB memory 320 may store video data, such as an encoded video bitstream, to be decoded by the components of the video decoder 300. The video data stored in the CPB memory 320 may be obtained, for example, from a computer-readable medium 110 (Figure 1). The CPB memory 320 may include a CPB that stores encoded video data (e.g., syntax elements) from the encoded video bitstream. The CPB memory 320 may also store video data other than syntax elements of an encoded picture, such as temporary data representing outputs from various units of the video decoder 300. The DPB 314 generally stores decoded pictures that the video decoder 300 may output and / or use as reference video data when decoding subsequent data or pictures from the encoded video bitstream. The CPB memory 320 and DPB 314 may be formed by any of a variety of memory devices, including DRAM (including synchronous dynamic random access memory) (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The CPB memory 320 and DPB 314 may be provided by the same memory device or by separate memory devices. In various examples, the CPB memory 320 may be on-chip along with the other components of the video decoder 300, or it may be off-chip relative to those components.

[0126]

[0132] As an addition or alternative, in some examples, the video decoder 300 may retrieve encoded video data from memory 120 (Figure 1). That is, memory 120 may store data together with CPB memory 320 as described above. Similarly, memory 120 may store instructions to be executed by the video decoder 300 when some or all of the functions of the video decoder 300 are implemented in software to be executed by the processing circuit of the video decoder 300.

[0127]

[0133] The various units shown in Figure 7 are presented to help understand the operations performed by the video decoder 300. The units may be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Similar to Figure 6, fixed-function circuits refer to circuits that provide a specific function and are preset in terms of the operations they may perform. Programmable circuits refer to circuits that can be programmed to perform various tasks and to provide flexible functionality in the operations they may perform. For example, a programmable circuit may execute software or firmware that operates the programmable circuit in a manner defined by software or firmware instructions. A fixed-function circuit may execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more units may be separate circuit blocks (fixed-function or programmable), and in some examples, one or more units may be integrated circuits.

[0128]

[0134] The video decoder 300 may include a programmable core formed from an ALU, EFU, digital circuitry, analog circuitry, and / or programmable circuitry. In an example where the operation of the video decoder 300 is performed by software running on the programmable circuitry, on-chip or off-chip memory may store software instructions (e.g., object code) that the video decoder 300 receives and executes.

[0129]

[0135] The entropy decoding unit 302 can receive encoded video data from the CPB and entropy decode the video data to reconstruct the syntax elements. The prediction processing unit 304, the inverse quantization unit 306, the inverse transformation processing unit 308, the reconstruction unit 310, and the filter unit 312 can generate decoded video data based on the syntax elements extracted from the bitstream.

[0130]

[0136] Generally, the video decoder 300 reconstructs the picture block by block. The video decoder 300 can perform the reconstruction operation individually for each block (wherein the block currently being reconstructed, i.e., the block currently being decoded, is sometimes called the "current block").

[0131]

[0137] The entropy decoding unit 302 can entropy decode transformation information such as syntax elements that define the quantized transformation coefficients of the quantized transformation coefficient block, as well as quantization parameters (QP) and / or (one or more) transformation mode indications. The inverse quantization unit 306 may use the QP associated with the quantized transformation coefficient block to determine the degree of quantization and, similarly, the degree of inverse quantization that the inverse quantization unit 306 should apply. The inverse quantization unit 306 may perform, for example, a bitwise left shift operation to inverse quantize the quantized transformation coefficients. The inverse quantization unit 306 may thereby form a transformation coefficient block containing the transformation coefficients.

[0132]

[0138] After the inverse quantization unit 306 has formed a transformation coefficient block, the inverse transformation processing unit 308 may apply one or more inverse transformations to the transformation coefficient block to generate a residual block associated with the current block. For example, the inverse transformation processing unit 308 may apply an inverse DCT, an inverse integer transformation, an inverse Carunenlebe transformation (KLT), an inverse rotation transformation, an inverse direction transformation, or another inverse transformation to the transformation coefficient block.

[0133]

[0139] Furthermore, the prediction processing unit 304 generates prediction blocks according to prediction information syntax elements entropy-decoded by the entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is interpredicted, the motion compensation unit 316 may generate prediction blocks. In this case, the prediction information syntax elements may indicate a reference picture in the DPB 314 from which the reference block should be extracted, as well as a motion vector identifying the location of the reference block in the reference picture relative to the location of the current block in the current picture. The motion compensation unit 316 may generally perform the interprediction process in a manner substantially similar to that described with respect to the motion compensation unit 224 (Figure 6).

[0134]

[0140] As another example, if the prediction information syntax element indicates that the current block is intra-predicted, the intra-prediction unit 318 may generate a predicted block according to the intra-prediction mode indicated by the prediction information syntax element. In this case as well, the intra-prediction unit 318 may perform the intra-prediction process in a manner that is generally substantially the same as that described with respect to the intra-prediction unit 226 (Figure 6). The intra-prediction unit 318 may retrieve adjacent sample data for the current block from the DPB 314.

[0135]

[0141] The reconstruction unit 310 may reconstruct the current block using the predicted block and the residual block. For example, the reconstruction unit 310 may add the samples of the residual block to the corresponding samples of the predicted block in order to reconstruct the current block.

[0136]

[0142] The filter unit 312 may perform one or more filtering operations on the reconstructed block. For example, the filter unit 312 may perform a deblocking operation to reduce blocking artifacts along the edges of the reconstructed block. The operation of the filter unit 312 is not necessarily performed in all examples. The filter unit 312 may be configured to apply the techniques of the present disclosure, for example, by resizing a color component to the smallest color component size and then applying a convolutional neural network (CNN) filter 322 to the resized color component.

[0137]

[0143] In particular, the filter unit 312 may reduce the size of the chroma component to the smaller of the two chroma components by applying one or more of the CNN filters 322, rather than upsampling the chroma component to the size of the corresponding chroma component. That is, the filter unit 312 may receive a chroma component having a first size and a chroma component having a second size smaller than the first size. The filter unit 312 may downsample the chroma component to the second size by applying the convolutional NN filter of the CNN filter 322 to the chroma component. The filter unit 312 may filter the chroma component to form a filtered chroma component. The filter unit 312 may then concatenate the downsampled chroma component with the filtered chroma component to form a concatenated color component, and then filter the concatenated color component to form a filtered concatenated component containing a filtered downsampled first color component. Furthermore, the filter unit 312 can upsample the filtered lumar component of the second size back to the first size and then store the lumar component in the DPB218.

[0138]

[0144] In particular, the first and second color components may originally be stored in separate arrays or matrices. To concatenate the color components, the filter unit 312 may form a single array or matrice with twice the width or height of the individual color components. The filter unit 312 may then store a sample of the first color component in a first region of the newly formed array or matrice, and a sample of the second color component in a second neighboring region of the newly formed array or matrice. When three color components are used (for example, luma, blue-hue chroma, and red-hue chroma), the filter unit 312 may form a single array or matrice with three times the width or height of the individual color components, and store a sample of each of the three color components in the respective regions of the newly formed array or matrice.

[0139]

[0145] The video decoder 300 may store the reconstructed blocks in the DPB 314. For example, in cases where the filter unit 312 does not operate, the reconstruction unit 310 may store the reconstructed blocks in the DPB 314. In cases where the filter unit 312 operates, the filter unit 312 may store the filtered reconstructed blocks in the DPB 314. As described above, the DPB 314 may provide the prediction processing unit 304 with reference information, such as a sample of the current picture for intra-prediction and a previously decoded picture for subsequent motion compensation. Furthermore, the video decoder 300 may output the decoded picture from the DPB 314 for subsequent presentation on a display device such as the display device 118 in Figure 1.

[0140]

[0146] Figure 8 is a block diagram showing an example of a joint component convolutional neural network (CNN) filter unit 330 using the technique of the present disclosure. The CNN filter in Figure 8 may correspond to the CNN filter 232 in Figure 6, which is performed by the filter unit 216, or the CNN filter 322 in Figure 7, which is performed by the filter unit 312.

[0141]

[0147] In this example, filter unit 330 (for example, filter unit 216 or 312) includes CNN filters 332, 334, 340, 342, and 344, a concatenation unit 336, hidden layers 338A~338N (hidden layer 338), and channel adders 346, 348, and 350. CNN filter 332 may downsample the luminance (luma) component using a convolutional neural network (CNN) layer with a stride of 2 in both the horizontal and vertical directions. That is, CNN filter 332 may apply a 3×3×M convolutional neural network layer filter with a stride of (2,2) to the luma component, and then apply a PReLU activation function to the filtered sample of the luma component.

[0142]

[0148] CNN filter 334 can then filter the chrominance components (i.e., Cb and Cr) and apply a PReLU activation function to the filtered samples of the chroma components. The concatenation unit 336 can then concatenate the resulting output of the downsampled luma component with the output of CNN filter 334 to form a concatenated color component. The hidden layer 338 can then use the concatenated color component as input. In the final layer, CNN filter 340 can upsample the output from the hidden layer 338 and then feed the upsampled data back into CNN filter 340 to generate a luma component output. Similarly, CNN filters 342 and 344 can filter the corresponding chroma components of the output of the hidden layer 338. The respective channel adders 346, 348, and 350 can separate the concatenated color component into its respective luma and chroma components.

[0143]

[0149] The blocks of the hidden layer 338 in Figure 8 can be combinations of filters or functions. For example, each block of the hidden layer 338 may contain a convolutional NN filter plus an activation function, or it may be a residual processing unit as shown in Figure 9.

[0144]

[0150] Figure 9 is a conceptual diagram showing an exemplary residual processing unit 360. The residual processing unit 360 may correspond to one or more blocks of the hidden layer 338 in Figure 8. In this example, the residual processing unit 360 includes a first 3×3×K×K convolution layer 362, a parametric rectified linear unit (PReLU) layer 364, a second 3×3×K×K convolution layer 366, and a total unit 368. The first 3×3×K×K convolution layer 362 may apply a first CNN filter to the received samples, the PReLU layer 364 may apply a PReLU activation function to the filtered samples, and the second 3×3×K×K convolution layer 366 may apply a second CNN filter to the filtered samples from the PReLU layer 364. Ultimately, a total of 368 units can be used to combine the filtered samples with the received input samples.

[0145]

[0151] Figure 10 is a block diagram showing another example of a joint component CNN filter unit 330' using the technique of the present disclosure. The CNN filter unit 330' in Figure 10 may correspond to the CNN filter 232 performed by the filter unit 216 in Figure 6, or the CNN filter 322 performed by the filter unit 312 in Figure 7.

[0146]

[0152] The exemplary CNN filter unit 330' in Figure 10 is substantially similar to the CNN filter unit 330 in Figure 8. However, in this example, the CNN filter unit 330' includes a block joiner unit 370 instead of the CNN filter 340 of the CNN filter unit 330 in Figure 8. The block joiner unit 370 can form a 2N×2N lumen block output by a set of hidden layers by joining four N×N blocks which are the outputs of the last convolutional NN layer, and the last convolutional NN layer can also be a "ConvTranspose2d / De-convolution" operation with a stride of 2 in both the horizontal and vertical directions.

[0147]

[0153] Figure 11 is a flowchart illustrating an exemplary method for encoding a current block using the technique of the present disclosure. A current block may comprise a current CU. While the video encoder 200 (Figures 1 and 3) is described, it should be understood that other devices may be configured to perform a similar method to that shown in Figure 11.

[0148]

[0154] In this example, the video encoder 200 first predicts the current block (380). For example, the video encoder 200 may form a predicted block for the current block. The video encoder 200 may then compute the residual block for the current block (382). To compute the residual block, the video encoder 200 may compute the difference between the original uncoded block and the predicted block for the current block. The video encoder 200 may then transform and quantize the coefficients of the residual block (384). Next, the video encoder 200 may scan the quantized transformed coefficients of the residual block (386). During or following the scan, the video encoder 200 may entropy encode the coefficients (388). For example, the video encoder 200 may encode the coefficients using CAVLC or CABAC. The video encoder 200 may then output the entropy encoded data of the block (390).

[0149]

[0155] The video encoder 200 may also decode the current block after encoding it (for example, in inter or intra prediction mode) to use the decoded version of the current block as reference data for data to be coded later. Thus, the video encoder 200 may inversely quantize and inversely transform the coefficients to reconstruct the residual block (392). The video encoder 200 may combine the residual block with the prediction block to form the decoded block (394).

[0150]

[0156] According to the techniques of the present disclosure, the video encoder 200 may filter and refine the decoded blocks (396). For example, the video encoder 200 may downsample the luminance component using, for example, a convolutional neural network filter. The video encoder 200 may also filter the chrominance component and then concatenate the downsampled lumen component with the filtered chromen component. The video encoder 200 may further filter the concatenated color component to form a filtered concatenated component containing a filtered downsampled first color component. The video encoder 200 may further filter the concatenated color component and then upsample the filtered lumen component to its original size. The video encoder 200 may then store the filtered decoded blocks in the DPB 218 (398).

[0151]

[0157] Thus, the method in Figure 11 represents an example of a method for filtering decoded video data, the method comprising: applying a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, where applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; filtering the second color component having the second size to form a filtered second color component; concatenating the downsampled first color component with the filtered second color component to form a concatenated color component; and filtering the concatenated color component.

[0152]

[0158] Figure 12 is a flowchart illustrating an exemplary method for decoding a current block using the technique of the present disclosure. The current block may comprise a current CU. While the video decoder 300 (Figures 1 and 4) is described in reference, it should be understood that other devices may be configured to perform a similar method to that shown in Figure 12.

[0153]

[0159] The video decoder 300 may receive entropy-encoded data about the current block, such as entropy-encoded prediction information and entropy-encoded data about the coefficients of the residual block corresponding to the current block (400). The video decoder 300 may entropy-decode the entropy-encoded data to determine the prediction information for the current block and to reconstruct the coefficients of the residual block (402). The video decoder 300 may predict the current block, for example, using an intra-prediction or inter-prediction mode indicated by the prediction information for the current block, in order to compute a prediction block for the current block (404). The video decoder 300 may then back-scan the reconstructed coefficients to create a block of quantized transformation coefficients (406). The video decoder 300 may then inverse-quantize and inverse-transform the quantized transformation coefficients to generate a residual block (408). The video decoder 300 may finally decode the current block by combining the prediction block and the residual block (410).

[0154]

[0160] The video decoder 300 may also filter the decoded video data, for example, using a convolutional neural network filter, as discussed above by the techniques of the present disclosure (412). For example, the video decoder 300 may downsample the luminance component, for example, using a convolutional neural network filter. The video decoder 300 may also filter the chrominance component and then concatenate the downsampled lumen component with the filtered chromen component. The video decoder 300 may further filter the concatenated color component. The video decoder 300 may further filter the concatenated color component and then upsample the filtered lumen component to its original size. The video decoder 300 may then store the filtered decoded block in the DPB218 (414).

[0155]

[0161] Thus, the method in Figure 12 represents an example of a method for filtering decoded video data, the method comprising: applying a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, herein, applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; filtering the second color component having the second size to form a filtered second color component; concatenating the downsampled first color component with the filtered second color component to form a concatenated color component; and filtering the concatenated color component.

[0156]

[0162] Figure 13 is a flowchart illustrating an exemplary method for filtering decoded video data using the technique of the present disclosure. A video encoder 200, a video decoder 300, or a video coder 130 may implement the method of Figure 13. For illustrative purposes, the method of Figure 13 will be described with respect to the video decoder 300.

[0157]

[0163] First, the video decoder 300 can receive and decode video data. The decoded video data may include a luminance (luma) component and two chrominance (chroma) components (e.g., Cb and Cr data). In this example, it is assumed that each of these components has a different size. That is, the luma component has a first size, the first chroma component has a second size, the second chroma component has a third size, the second size is assumed to be smaller than the first size, and the third size is assumed to be smaller than the second size. For example, the video data in this example may have a 4:2:0 format.

[0158]

[0164] The video decoder 300 may apply a downsampled CNN layer to the chroma component (420) to generate a downsampled chroma component having a third size. The video decoder 300 may also apply a downsampled CNN layer (or a different downsampled CNN layer) to the first chroma component (422) to generate a first downsampled chroma component having a third size. The video decoder 300 may then filter the first downsampled chroma component and the second chroma component using, for example, a CNN filter (424).

[0159]

[0165] The video decoder 300 may then concatenate the lumens and chroma components (426) to produce concatenated color components. The video decoder 300 may then filter the concatenated color components using, for example, a CNN filter (such as the hidden layer 338 in Figures 8 and 10) (428).

[0160]

[0166] The video decoder 300 may then upsample the lumen component of the filtered concatenated color components to a first size (430) and upsample the first chroma component of the filtered concatenated color components to a second size (432). The video decoder 300 may then store the components in the DPB 314 and output the components (434).

[0161]

[0167] Thus, the method in Figure 13 represents an example of a method for filtering decoded video data, the method comprising: applying a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, where applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; filtering the second color component having the second size to form a filtered second color component; concatenating the downsampled first color component with the filtered second color component to form a concatenated color component; and filtering the concatenated color component.

[0162]

[0168] Figure 14 is a flowchart illustrating an exemplary method for filtering decoded video data using the technique of the present disclosure. A video encoder 200, a video decoder 300, or a video coder 130 may implement the method of Figure 14. For illustrative purposes, the method of Figure 14 will be described with respect to the video decoder 300.

[0163]

[0169] First, the video decoder 300 can receive and decode video data. The decoded video data may include a luminance (luma) component and two chrominance (chroma) components (e.g., Cb and Cr data). In this example, it is assumed that the luma component has a first size, and the first and second chroma components have a second size, with the second size being smaller than the first size. For example, the video data in this example may be in 4:2:0 format or 4:2:2 format.

[0164]

[0170] The video decoder 300 may apply a downsampled CNN layer to the chroma component (440) to generate a downsampled chroma component having a second size. The video decoder 300 may then filter the first chroma component and the second chroma component using, for example, a CNN filter (442).

[0165]

[0171] The video decoder 300 may then concatenate the lumens and chroma components (444) to produce concatenated color components. The video decoder 300 may then filter the concatenated color components using, for example, a CNN filter (such as the hidden layer 338 in Figures 8 and 10) (446).

[0166]

[0172] The video decoder 300 may then upsample the lumen component of the filtered concatenated color components to a first size (448). The video decoder 300 may then store the components in the DPB 314 and output the components (450).

[0167]

[0173] Thus, the method in Figure 14 represents an example of a method for filtering decoded video data, the method comprising: applying a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, where applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; filtering the second color component having the second size to form a filtered second color component; concatenating the downsampled first color component with the filtered second color component to form a concatenated color component; and filtering the concatenated color component.

[0168]

[0174] Some examples of the techniques of this disclosure are summarized in the following clauses.

[0169]

[0175] Clause 1: A method for filtering decoded video data, comprising: applying a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, wherein applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size; filtering the downsampled first color component using one or more convolutional neural network layer filters; and filtering a second color component of a block of video data using one or more convolutional neural network layer filters.

[0170]

[0176] Clause 2: The method according to Clause 1, further comprising upsampling a filtered, downsampled first color component to a first size.

[0171]

[0177] Clause 3: The method according to Clause 1, further comprising combining two or more filtered downsampled blocks of a first color component, each containing a filtered downsampled block, to produce an upsampled first color component having a first size.

[0172]

[0178] Clause 4: The method according to Clause 3, wherein the first size comprises 2N × 2N, and two or more filtered downsampled blocks of the first color component comprise four N × N filtered downsampled blocks of the first color component.

[0173]

[0179] Clause 5: The method according to any one of Clauses 1 to 4, wherein the downsampling convolutional neural network layer comprises a 3x3xM convolutional neural network layer filter with a stride of 2.

[0174]

[0180] Clause 6: The method according to any one of Clauses 1 to 5, wherein one or more convolutional neural network layer filters include a residual processing unit.

[0175]

[0181] Clause 7: The method according to Clause 6, wherein the residual processing unit comprises a first 3×3×K×K convolutional layer, a PReLU layer, and a second 3×3×K×K convolutional layer.

[0176]

[0182] Clause 8: The method according to any one of Clauses 1 to 7, further comprising concatenating a downsampled first color component with a second color component.

[0177]

[0183] Clause 9: The method according to any one of Clauses 1 to 8, further comprising filtering out the third color component of a block of video data using one or more convolutional neural network layer filters.

[0178]

[0184] Clause 10: The method according to Clause 9, wherein the second size comprises the smaller of the size of the second color component and the size of the third color component.

[0179]

[0185] Clause 11: The method according to any one of Clauses 1 to 10, wherein the first color component comprises a luminance component and the second color component comprises either a blue hue chrominance component or a red hue chroma chrominance component.

[0180]

[0186] Clause 12: A device for decoding video data, comprising one or more means for carrying out the method described in any of Clauses 1 to 11.

[0181]

[0187] Clause 13: The device according to Clause 12, wherein one or more means comprises one or more processors implemented in the circuit.

[0182]

[0188] Clause 14: The device described in Clause 12, further comprising a display configured to display decoded video data.

[0183]

[0189] Clause 15: A device as described in Clause 12, comprising one or more of a camera, a computer, a mobile device, a broadcast receiver device, and a set-top box.

[0184]

[0190] Clause 16: The device described in Clause 12, further comprising memory configured to store video data.

[0185]

[0191] Clause 17: A computer-readable storage medium storing instructions, wherein the instructions, when executed, cause a processor to perform any of the methods described in Clauses 1 through 11.

[0186]

[0192] Clause 18: A method for filtering decoded video data, comprising: applying a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, wherein applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; filtering the second color component having the second size to form a filtered second color component; concatenating the downsampled first color component with the filtered second color component to form a concatenated color component; and filtering the concatenated color component.

[0187]

[0193] Clause 19: The method according to Clause 18, further comprising upsampling a filtered, downsampled first color component to a first size.

[0188]

[0194] Clause 20: The method according to Clause 18, further comprising combining two or more filtered downsampled blocks of a first color component, including filtered downsampled blocks, to produce an upsampled first color component having a first size.

[0189]

[0195] Clause 21: The method according to Clause 20, wherein the first size comprises 2N × 2N, and two or more filtered downsampled blocks of the first color component comprise four N × N filtered downsampled blocks of the first color component.

[0190]

[0196] Clause 22: The method according to Clause 18, wherein the downsampling convolutional neural network layer comprises a 3x3xM convolutional neural network layer filter with a stride of 2.

[0191]

[0197] Clause 23: The method according to Clause 18, wherein one or more convolutional neural network layer filters include a residual processing unit.

[0192]

[0198] Clause 24: The method according to Clause 23, wherein the residual processing unit comprises a first 3×3×K×K convolutional layer, a PReLU layer, and a second 3×3×K×K convolutional layer.

[0193]

[0199] Clause 25: The method according to Clause 18, further comprising filtering a third color component of a block of video data using one or more convolutional neural network layer filters.

[0194]

[0200] Clause 26: The method according to Clause 25, wherein the second size comprises the smaller of the size of the second color component and the size of the third color component.

[0195]

[0201] Clause 27: The method according to Clause 18, wherein the first color component comprises a luminance component and the second color component comprises one of a blue-phase chrominance component and a red-phase chromochrominance component.

[0196]

[0202] Clause 28: The method according to Clause 18, wherein filtering the second color component is performed by using a convolutional neural network filter to filter the second color component.

[0197]

[0203] Clause 29: The method according to Clause 18, wherein filtering the concatenated color components is performed using a convolutional neural network filter.

[0198]

[0204] Clause 30: A device for decoding video data, comprising a memory configured to store video data, and one or more processors implemented in the circuit, wherein the one or more processors are configured to apply a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, wherein applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; filter the second color component having the second size to form a filtered second color component; concatenate the downsampled first color component with the filtered second color component to form a concatenated color component; and filter the concatenated color component.

[0199]

[0205] Clause 31: The device according to Clause 30, wherein one or more processors are further configured to upsample a filtered, downsampled first color component to a first size.

[0200]

[0206] Clause 32: The device according to Clause 30, wherein one or more processors are further configured to combine two or more filtered downsampled blocks of a first color component, each containing a filtered downsampled block, in order to produce an upsampled first color component having a first size.

[0201]

[0207] Clause 33: The device according to Clause 32, wherein the first size comprises 2N × 2N, and two or more filtered downsampled blocks of the first color component comprise four N × N filtered downsampled blocks of the first color component.

[0202]

[0208] Clause 34: The device according to Clause 30, wherein the downsampling convolutional neural network layer comprises a 3x3xM convolutional neural network layer filter with a stride of 2.

[0203]

[0209] Clause 35: The device according to Clause 30, wherein one or more convolutional neural network layer filters comprise a residual processing unit.

[0204]

[0210] Clause 36: The device according to Clause 35, wherein the residual processing unit comprises a first 3×3×K×K convolutional layer, a PReLU layer, and a second 3×3×K×K convolutional layer.

[0205]

[0211] Clause 37: The device according to Clause 30, further configured to filter a third color component of a block of video data using one or more processors with one or more convolutional neural network layer filters.

[0206]

[0212] Clause 38: The device according to Clause 37, wherein the second size comprises the smaller of the size of the second color component and the size of the third color component.

[0207]

[0213] Clause 39: The device according to Clause 30, wherein the first color component comprises a luminance component and the second color component comprises either a blue-phase chrominance component or a red-phase chromochrominance component.

[0208]

[0214] Clause 40: The device according to Clause 30, wherein one or more processors are configured to filter a second color component using a convolutional neural network filter.

[0209]

[0215] Clause 41: The device according to Clause 30, wherein one or more processors are configured to filter concatenated color components using a convolutional neural network filter.

[0210]

[0216] Clause 42: The device described in Clause 30, further comprising a display configured to display video data corresponding to concatenated color components.

[0211]

[0217] Clause 43: The device described in Clause 30, comprising one or more of the following: a camera, a computer, a mobile device, a broadcast receiver device, and a set-top box.

[0212]

[0218] Clause 44: A computer-readable storage medium storing instructions, wherein, when the instructions are executed, the processor causes the processor to: apply a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, wherein applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; filter the second color component having the second size to form a filtered second color component; concatenate the downsampled first color component with the filtered second color component to form a concatenated color component; and filter the concatenated color component.

[0213]

[0219] Clause 45: The computer-readable storage medium described in Clause 44, further comprising instructions for causing a processor to upsample a filtered, downsampled first color component to a first size.

[0214]

[0220] Clause 46: The computer-readable storage medium according to Clause 44, further comprising instructions for causing a processor to combine two or more filtered downsampled blocks of a first color component, each containing a filtered downsampled block, in order to produce an upsampled first color component having a first size.

[0215]

[0221] Clause 47: The computer-readable storage medium according to Clause 46, wherein the first size comprises 2N × 2N, and two or more filtered downsampled blocks of the first color component comprise four N × N filtered downsampled blocks of the first color component.

[0216]

[0222] Clause 48: A computer-readable storage medium according to Clause 44, wherein the downsampling convolutional neural network layer comprises a 3x3xM convolutional neural network layer filter with a stride of 2.

[0217]

[0223] Clause 49: A computer-readable storage medium according to Clause 44, wherein one or more convolutional neural network layer filters comprise a first 3×3×K×K convolutional layer, a PReLU layer, and a second 3×3×K×K convolutional layer.

[0218]

[0224] Clause 50: A computer-readable storage medium according to Clause 44, further configured to filter a third color component of a block of video data using one or more processors with one or more convolutional neural network layer filters.

[0219]

[0225] Clause 51: The computer-readable storage medium according to Clause 50, wherein the second size comprises the smaller of the size of the second color component and the size of the third color component.

[0220]

[0226] Clause 52: The computer-readable storage medium according to Clause 44, wherein the first color component comprises a luminance component and the second color component comprises either a blue-phase chrominance component or a red-phase chromochrominance component.

[0221]

[0227] Clause 53: A computer-readable storage medium according to Clause 44, wherein the instruction for causing the processor to filter a second color component includes an instruction for causing the processor to filter a second color component using a convolutional neural network filter.

[0222]

[0228] Clause 54: A computer-readable storage medium according to Clause 44, wherein the instruction for causing the processor to filter a second color component includes an instruction for causing the processor to filter a color component concatenated using a convolutional neural network filter.

[0223]

[0229] Clause 55: A device for filtering decoded video data, comprising: means for applying a downsampled convolutional neural network layer to a first color component of a block of video data; wherein the first color component of the block has a first size, wherein applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; means for filtering the second color component having a second size to form a filtered second color component; means for concatenating the downsampled first color component with the filtered second color component to form a concatenated color component; and means for filtering the concatenated color component.

[0224]

[0230] Clause 56: The device according to Clause 55, wherein the means for filtering a second color component comprises means for filtering a second color component using a convolutional neural network filter.

[0225]

[0231] Clause 57: The device according to Clause 55, wherein the means for filtering concatenated color components comprises means for filtering concatenated color components using a convolutional neural network filter.

[0226]

[0232] Clause 58: A method for filtering decoded video data, comprising: applying a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, wherein applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; filtering the second color component having the second size to form a filtered second color component; concatenating the downsampled first color component with the filtered second color component to form a concatenated color component; and filtering the concatenated color component.

[0227]

[0233] Clause 59: The method according to Clause 58, further comprising upsampling a filtered, downsampled first color component to a first size.

[0228]

[0234] Clause 60: The method according to Clause 58, further comprising combining two or more filtered downsampled blocks of a first color component, including filtered downsampled blocks, to produce an upsampled first color component having a first size.

[0229]

[0235] Clause 61: The method according to Clause 60, wherein the first size comprises 2N × 2N, and two or more filtered downsampled blocks of the first color component comprise four N × N filtered downsampled blocks of the first color component.

[0230]

[0236] Clause 62: The method according to any one of Clauses 58 to 61, wherein the downsampling convolutional neural network layer comprises a 3 × 3 × M convolutional neural network layer filter with a stride of 2.

[0231]

[0237] Clause 63: The method according to any one of Clauses 58 to 62, wherein one or more convolutional neural network layer filters comprises a residual processing unit.

[0232]

[0238] Clause 64: The method according to Clause 63, wherein the residual processing unit comprises a first 3×3×K×K convolutional layer, a PReLU layer, and a second 3×3×K×K convolutional layer.

[0233]

[0239] Clause 65: The method according to any one of Clauses 58 to 64, further comprising filtering a third color component of a block of video data using one or more convolutional neural network layer filters.

[0234]

[0240] Clause 66: The method according to Clause 65, wherein the second size comprises the smaller of the size of the second color component and the size of the third color component.

[0235]

[0241] Clause 67: The method according to any one of Clauses 58 to 66, wherein the first color component comprises a luminance component and the second color component comprises one of a blue-phase chrominance component and a red-phase chromochrominance component.

[0236]

[0242] Clause 68: The method according to any one of Clauses 58 to 67, wherein filtering the second color component comprises filtering the second color component using a convolutional neural network filter.

[0237]

[0243] Clause 69: The method according to any one of Clauses 58 to 68, wherein filtering the concatenated color components is performed using a convolutional neural network filter.

[0238]

[0244] Clause 70: A device for decoding video data, comprising a memory configured to store video data, and one or more processors implemented in the circuit, wherein one or more processors are configured to apply a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, wherein applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; filter the second color component having the second size to form a filtered second color component; concatenate the downsampled first color component with the filtered second color component to form a concatenated color component; and filter the concatenated color component.

[0239]

[0245] Clause 71: The device according to Clause 70, wherein one or more processors are further configured to upsample a filtered, downsampled first color component to a first size.

[0240]

[0246] Clause 72: The device according to Clause 70, wherein one or more processors are further configured to combine two or more filtered downsampled blocks of a first color component, each containing a filtered downsampled block, in order to produce an upsampled first color component having a first size.

[0241]

[0247] Clause 73: The device according to Clause 72, wherein the first size comprises 2N × 2N, and two or more filtered downsampled blocks of the first color component comprise four N × N filtered downsampled blocks of the first color component.

[0242]

[0248] Clause 74: The device according to any one of Clauses 70 to 73, wherein the downsampling convolutional neural network layer comprises a 3x3xM convolutional neural network layer filter with a stride of 2.

[0243]

[0249] Clause 75: The device according to any one of Clauses 70 to 74, wherein one or more convolutional neural network layer filters comprises a residual processing unit.

[0244]

[0250] Clause 76: The device according to Clause 75, wherein the residual processing unit comprises a first 3×3×K×K convolutional layer, a PReLU layer, and a second 3×3×K×K convolutional layer.

[0245]

[0251] Clause 77: The device according to any one of Clauses 70 to 76, wherein one or more processors are further configured to filter a third color component of a block of video data using one or more convolutional neural network layer filters.

[0246]

[0252] Clause 78: The device according to Clause 77, wherein the second size comprises the smaller of the size of the second color component and the size of the third color component.

[0247]

[0253] Clause 79: The device according to any one of Clauses 70 to 78, wherein the first color component comprises a luminance component and the second color component comprises either a blue-phase chrominance component or a red-phase chromochrominance component.

[0248]

[0254] Clause 80: The device according to any one of Clauses 70 to 79, wherein one or more processors are configured to filter a second color component using a convolutional neural network filter.

[0249]

[0255] Clause 81: The device according to any one of Clauses 70 to 80, wherein one or more processors are configured to filter concatenated color components using a convolutional neural network filter.

[0250]

[0256] Clause 82: The device described in any of Clauses 70 to 81, further comprising a display configured to display video data corresponding to concatenated color components.

[0251]

[0257] Clause 83: A device as described in any of Clauses 70 to 82, comprising one or more of a camera, a computer, a mobile device, a broadcast receiver device, and a set-top box.

[0252]

[0258] Clause 84: A computer-readable storage medium storing instructions, wherein, when the instructions are executed, the processor causes the processor to: apply a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, wherein applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; filter the second color component having the second size to form a filtered second color component; concatenate the downsampled first color component with the filtered second color component to form a concatenated color component; and filter the concatenated color component.

[0253]

[0259] Clause 85: The computer-readable storage medium described in Clause 84, further comprising instructions for causing a processor to upsample a filtered, downsampled first color component to a first size.

[0254]

[0260] Clause 86: The computer-readable storage medium according to Clause 84, further comprising instructions for causing a processor to combine two or more filtered downsampled blocks of a first color component, each containing a filtered downsampled block, in order to produce an upsampled first color component having a first size.

[0255]

[0261] Clause 87: The computer-readable storage medium according to Clause 86, wherein the first size comprises 2N × 2N, and two or more filtered downsampled blocks of the first color component comprise four N × N filtered downsampled blocks of the first color component.

[0256]

[0262] Clause 88: A computer-readable storage medium according to any one of Clauses 84 to 87, wherein the downsampling convolutional neural network layer comprises a 3x3xM convolutional neural network layer filter with a stride of 2.

[0257]

[0263] Clause 89: A computer-readable storage medium according to any one of Clauses 84 to 88, wherein one or more convolutional neural network layer filters comprises a first 3×3×K×K convolutional layer, a PReLU layer, and a second 3×3×K×K convolutional layer.

[0258]

[0264] Clause 90: A computer-readable storage medium according to any one of Clauses 84 to 89, further configured to filter a third color component of a block of video data using one or more processors with one or more convolutional neural network layer filters.

[0259]

[0265] Clause 91: The computer-readable storage medium according to Clause 90, wherein the second size comprises the smaller of the size of the second color component and the size of the third color component.

[0260]

[0266] Clause 92: A computer-readable storage medium according to any one of Clauses 84 to 91, wherein the first color component comprises a luminance component and the second color component comprises either a blue-phase chrominance component or a red-phase chromochrominance component.

[0261]

[0267] Clause 93: A computer-readable storage medium according to any one of Clauses 84 to 92, wherein the instruction for causing the processor to filter a second color component comprises an instruction for causing the processor to filter a second color component using a convolutional neural network filter.

[0262]

[0268] Clause 94: A computer-readable storage medium according to any one of Clauses 84 to 93, wherein an instruction causing the processor to filter a second color component includes an instruction causing the processor to filter a color component concatenated using a convolutional neural network filter.

[0263]

[0269] Clause 95: A device for filtering decoded video data, comprising: means for applying a downsampled convolutional neural network layer to a first color component of a block of video data; wherein the first color component of the block has a first size, wherein applying the downsampled convolutional neural network layer to the first color component produces a downsampled first color component having a second size smaller than the first size; means for filtering the second color component having a second size to form a filtered second color component; means for concatenating the downsampled first color component with the filtered second color component to form a concatenated color component; and means for filtering the concatenated color component.

[0264]

[0270] Clause 96: The device according to Clause 95, wherein the means for filtering a second color component comprises means for filtering a second color component using a convolutional neural network filter.

[0265]

[0271] Clause 97: The device according to either Clause 95 or 96, wherein the means for filtering concatenated color components comprises means for filtering concatenated color components using a convolutional neural network filter.

[0266]

[0272] In accordance with the examples above, it should be noted that some of the actions or events of any of the techniques described herein may be performed in different sequences, added, merged, or completely excluded (for example, not all described actions or events are necessarily required for the practice of this technique). Furthermore, in some examples, actions or events may be performed not sequentially, but simultaneously, for example, through multithreading, interrupt handling, or across multiple processors.

[0267]

[0273] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted through a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include computer-readable storage media corresponding to tangible media such as data storage media, or communication media including any medium that enables the transfer of computer programs from one location to another, for example, according to a communication protocol. Thus, the computer-readable medium may generally correspond to (1) non-transient tangible computer-readable storage media, or (2) communication media such as signals or carrier waves. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described herein. A computer program product may include computer-readable media.

[0268]

[0274] As an example, and not an limitation, such computer-readable storage media may include RAM, ROM, EEPROM®, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. Any connection is also appropriately referred to as computer-readable media. For example, if instructions are transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but instead refer to non-temporary tangible storage media. As used herein, the terms "disk" and "disc" include Compact Disc (CD), LaserDisc® (disc), Optical Disc (disc), Digital Multipurpose Disc (disc) (DVD), Floppy Disk (disk), and Blu-ray Disc (disc), where a disk typically reproduces data magnetically and a disc reproduces data optically using a laser. Any combination of the above should also be included within the scope of computer-readable media.

[0269]

[0275] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the terms “processor” and “processing circuit” as used herein may refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some embodiments, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a composite codec. Moreover, the techniques can be adequately implemented in one or more circuits or logic elements.

[0270]

[0276] The techniques of this disclosure can be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). While various components, modules, or units have been described in this disclosure to highlight the functional aspects of devices configured to implement the disclosed techniques, these components, modules, or units do not necessarily require implementation by different hardware units. Rather, as described above, the various units, along with suitable software and / or firmware, may be combined in a codec hardware unit, including one or more processors described above, or provided by a set of interoperable hardware units.

[0271]

[0277] Various examples were described. These and other examples fall within the scope of the following claims. The invention described in the original claims of this application is listed below. [C1] A method for filtering decoded video data, Applying a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, and hereby applying the downsampled convolutional neural network layer to the first color component generates a downsampled first color component having a second size smaller than the first size. To form a filtered second color component, the second color component having the second size is filtered, In order to form concatenated color components, the downsampled first color component is concatenated with the filtered second color component, Filtering the concatenated color component in order to form a filtered concatenated component that includes a filtered downsampled first color component. A method for providing this. [C2] The method according to C1, further comprising upsampling the filtered downsampled first color component to the first size. [C3] The method according to C1, further comprising combining two or more filtered downsampled blocks of the first color component, which include the filtered downsampled first color component, in order to produce an upsampled first color component having the first size. [C4] The method according to C3, wherein the first size comprises 2N × 2N, and the two or more filtered downsampled first color components of the first color component comprises four N × N filtered downsampled blocks of the first color component. [C5] The method according to C1, wherein the downsampling convolutional neural network layer comprises a 3×3×M convolutional neural network layer filter with a stride of 2. [C6] The method according to C1, wherein the convolutional neural network layer includes a residual processing unit. [C7] The method according to C6, wherein the residual processing unit comprises a first 3×3×K×K convolutional layer, a PReLU layer, and a second 3×3×K×K convolutional layer. [C8] The method according to C1, further comprising filtering the third color component of the block of video data using the convolutional neural network layer. [C9] The method according to C8, wherein the second size comprises the smaller of the size of the second color component and the size of the third color component. [C10] The method according to C1, wherein the first color component comprises a luminance component, and the second color component comprises one of a blue-phase chrominance component and a red-phase chromochrominance component. [C11] The method according to C1, wherein filtering the second color component is performed using a convolutional neural network filter. [C12] The method according to C1, wherein filtering the concatenated color components is performed using a convolutional neural network filter. [C13] A device for decoding video data, A memory configured to store video data, The circuit comprises one or more processors implemented in the circuit, and the one or more processors Applying a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, and hereby applying the downsampled convolutional neural network layer to the first color component generates a downsampled first color component having a second size smaller than the first size. To form a filtered second color component, the second color component having the second size is filtered, In order to form concatenated color components, the downsampled first color component is concatenated with the filtered second color component, Filtering the concatenated color component in order to form a concatenated color component that includes a filtered downsampled first color component. A device configured to perform the following actions. [C14] The device according to C13, wherein one or more processors are further configured to upsample the filtered downsampled first color component to the first size. [C15] The device according to C13, wherein the one or more processors are further configured to combine two or more filtered downsampled blocks of the first color component, including the filtered downsampled first color component, in order to generate an upsampled first color component having the first size. [C16] The device according to C15, wherein the first size comprises 2N × 2N, and the two or more filtered downsampled blocks of the first color component comprises four N × N filtered downsampled blocks of the first color component. [C17] The device according to C13, wherein the downsampling convolutional neural network layer comprises a 3×3×M convolutional neural network layer filter with a stride of 2. [C18] The device according to C13, wherein the convolutional neural network layer comprises a residual processing unit. [C19] The device according to C18, wherein the residual processing unit comprises a first 3×3×K×K convolutional layer, a PReLU layer, and a second 3×3×K×K convolutional layer. [C20] The device according to C13, wherein one or more processors are further configured to filter a third color component of the block of video data using the convolutional neural network layer. [C21] The device according to C20, wherein the second size comprises the smaller of the size of the second color component and the size of the third color component. [C22] The device according to C13, wherein the first color component comprises a luminance component, and the second color component comprises one of a blue-phase chrominance component and a red-phase chromochrominance component. [C23] The device according to C13, wherein one or more processors are configured to filter the second color component using a convolutional neural network filter. [C24] The device according to C13, wherein one or more processors are configured to filter the concatenated color components using a convolutional neural network filter. [C25] The device according to C13, further comprising a display configured to display video data corresponding to the aforementioned concatenated color components. [C26] A device as described in C13, comprising one or more of a camera, a computer, a mobile device, a broadcast receiver device, and a set-top box. [C27] A computer-readable storage medium storing instructions, wherein, when the instructions are executed, the instructions are to apply a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, and applying the downsampled convolutional neural network layer to the first color component generates a downsampled first color component having a second size smaller than the first size. To form a filtered second color component, the second color component having the second size is filtered, In order to form concatenated color components, the downsampled first color component is concatenated with the filtered second color component, Filtering the concatenated color component in order to form a filtered concatenated component that includes a filtered downsampled first color component. A computer-readable storage medium that allows a processor to perform certain operations. [C28] The computer-readable storage medium according to C27, further comprising an instruction causing the processor to upsample the filtered, downsampled first color component to the first size. [C29] The computer-readable storage medium according to C27, further comprising instructions for causing the processor to combine two or more filtered downsampled blocks of the first color component, including the filtered downsampled first color component, in order to generate an upsampled first color component having the first size. [C30] The computer-readable storage medium according to C29, wherein the first size comprises 2N × 2N, and the two or more filtered downsampled blocks of the first color component comprises four N × N filtered downsampled blocks of the first color component. [C31] The computer-readable storage medium according to C27, wherein the downsampling convolutional neural network layer comprises a 3 × 3 × M convolutional neural network layer filter with a stride of 2. [C32] The computer-readable storage medium according to C27, wherein the convolutional neural network layer comprises a first 3×3×K×K convolutional layer, a PReLU layer, and a second 3×3×K×K convolutional layer. [C33] The computer-readable storage medium according to C27, further comprising instructions for causing the processor to filter the third color component of the block of video data using the convolutional neural network layer. [C34] The computer-readable storage medium according to C33, wherein the second size comprises the smaller of the size of the second color component and the size of the third color component. [C35] The computer-readable storage medium according to C27, wherein the first color component comprises a luminance component, and the second color component comprises one of a blue-phase chrominance component and a red-phase chromochrominance component. [C36] The computer-readable storage medium according to C27, wherein the instruction causing the processor to filter the second color component comprises an instruction causing the processor to filter the second color component using a convolutional neural network filter. [C37] The computer-readable storage medium according to C27, wherein the instruction causing the processor to filter the second color component comprises an instruction causing the processor to filter the connected color components using a convolutional neural network filter. [C38] A device for filtering decoded video data, A means for applying a downsampled convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, and applying the downsampled convolutional neural network layer to the first color component generates a downsampled first color component having a second size smaller than the first size. Means for filtering the second color component having the second size in order to form a filtered second color component, Means for concatenating the downsampled first color component with the filtered second color component in order to form concatenated color components, means for filtering the linked color components A device equipped with the following features. [C39] The device according to C38, wherein the means for filtering the second color component comprises means for filtering the second color component using a convolutional neural network filter. [C40] The device according to C38, wherein the means for filtering the concatenated color components comprises means for filtering the concatenated color components using a convolutional neural network filter.

Claims

1. A method for filtering decoded video data, Applying a downsampled first convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, and hereby applying the downsampled first convolutional neural network layer to the first color component generates a downsampled first color component having a second size smaller than the first size. Filtering a second color component having the second size in order to form a second color component filtered using a second convolutional neural network layer, wherein the second convolutional neural network layer is different from the downsampling first convolutional neural network layer. Here, the first color component is a luminance component, and the second color component is one of the blue chrominance component and the red chromochrominance component. In order to form concatenated color components, the downsampled first color component is concatenated with the filtered second color component, Filtering the concatenated color component in order to form a filtered concatenated component that includes a first color component that has been filtered downsampled using at least a third convolutional neural network layer, and A method for providing this.

2. The method according to claim 1, further comprising upsampling the filtered downsampled first color component to the first size.

3. The method according to claim 1, further comprising combining two or more filtered downsampled blocks of the first color component, which include the filtered downsampled first color component, in order to generate an upsampled first color component having the first size.

4. The method according to claim 3, wherein the first size comprises 2N × 2N, and the two or more filtered downsampled first color components of the first color component comprises four N × N filtered downsampled blocks of the first color component.

5. The method according to claim 1, wherein the downsampling first convolutional neural network layer comprises a 3 × 3 × M convolutional neural network layer filter with a stride of 2.

6. The method according to claim 1, wherein the convolutional neural network layer comprises a residual processing unit, and the residual processing unit comprises a first 3×3×K×K convolutional layer, a PReLU layer, and a second 3×3×K×K convolutional layer.

7. The method according to claim 1, further comprising filtering the third color component of the block of video data using the convolutional neural network layer.

8. The method according to claim 7, wherein the second size comprises the smaller of the size of the second color component and the size of the third color component.

9. A device for decoding video data, A memory configured to store video data, The circuit comprises one or more processors implemented in the circuit, and the one or more processors Applying a downsampled first convolutional neural network layer to a first color component of a block of video data, wherein the first color component of the block has a first size, and hereby applying the downsampled first convolutional neural network layer to the first color component generates a downsampled first color component having a second size smaller than the first size. Filtering a second color component having the second size in order to form a second color component filtered using a second convolutional neural network layer, wherein the second convolutional neural network layer is different from the downsampling first convolutional neural network layer. Here, the first color component is a luminance component, and the second color component is one of the blue chrominance component and the red chromochrominance component. In order to form concatenated color components, the downsampled first color component is concatenated with the filtered second color component, Filtering the concatenated color component in order to form a concatenated color component that includes a first color component that has been filtered using at least a third convolutional neural network layer and downsampled; A device configured to perform the following actions.

10. The device according to claim 9, wherein one or more processors are further configured to upsample the filtered downsampled first color component to the first size.

11. The device according to claim 9, wherein the one or more processors are further configured to combine two or more filtered downsampled blocks of the first color component, each containing the filtered downsampled first color component, in order to generate an upsampled first color component having the first size.

12. The device according to claim 11, wherein the first size comprises 2N × 2N, and the two or more filtered downsampled blocks of the first color component comprises four N × N filtered downsampled blocks of the first color component.

13. The device according to claim 9, wherein the downsampling first convolutional neural network layer comprises a 3 × 3 × M convolutional neural network layer filter with a stride of 2.

14. The device according to claim 9, wherein the convolutional neural network layer comprises a residual processing unit, and the residual processing unit comprises a first 3×3×K×K convolutional layer, a PReLU layer, and a second 3×3×K×K convolutional layer.

15. A non-temporary computer-readable storage medium storing instructions, wherein, when an instruction is executed, the storage medium causes a processor to perform the method described in any one of claims 1 to 8.

Images