Neural network-based in-loop filter architecture with separable convolution and multi-scale enhancement for video coding

Abstract

An apparatus for decoding video data receives a picture of the video data; reconstructing a block of a picture of the video data to generate a reconstructed block; and performing a neural network (NN)-based filtering process on the reconstructed block to generate a filtered block, where the NN-based filtering process includes performing a plurality of separable convolutions in parallel with a point-by-point input convolution.

Description

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 495,732, filed April 12, 2023, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to video encoding and video decoding. BACKGROUND

[0003] Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio telephones, so-called “smart phones,” video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), ITU-T H.266 / Versatile Video Coding (VVC), and extensions of such standards, as well as proprietary video codecs / formats developed by Open Media Alliance (OMA), such as AOMedia Video 1 (AV1). By implementing such video coding techniques, video devices can more efficiently send, receive, encode, decode, and / or store digital video information.

[0004] Video coding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) can be partitioned into video blocks, which can also be referred to as coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction relative to other, neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture can use spatial or temporal prediction relative to other blocks in the same picture or relative to blocks in other pictures. SUMMARY

[0005] This disclosure describes simplifications that can be applied to neural network (NN) based filtering techniques while also preserving coding quality. For example, according to the techniques of this disclosure, a video coder can be configured to perform an NN based filtering process that includes performing multiple separable convolutions in parallel with point-wise input convolutions. Application of these simplifications can reduce processing performed by one or more processors to perform NN based filtering with relatively minimal loss in coding quality. In this way, the techniques of this disclosure can improve the performance of video coding devices. Also, these techniques can enable more devices to perform NN based filtering, thereby generally improving the field of video coding.

[0006] According to one example of the disclosure, a method of decoding video data includes receiving a picture of video data; reconstructing a block of the picture of video data to generate a reconstructed block; and performing a neural network (NN) based filtering process on the reconstructed block to generate a filtered block, wherein the NN based filtering process includes performing multiple separable convolutions in parallel with point-wise input convolutions.

[0007] According to one example of the disclosure, a device for decoding video data includes a memory configured to store video data and one or more processors implemented in circuitry and configured to: receive a picture of video data; reconstruct a block of the picture of video data to generate a reconstructed block; and perform a neural network (NN) based filtering process on the reconstructed block to generate a filtered block, wherein the NN based filtering process includes performing multiple separable convolutions in parallel with point-wise input convolutions.

[0008] A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receive a picture of video data; reconstruct a block of the picture of video data to generate a reconstructed block; and perform a neural network (NN) based filtering process on the reconstructed block to generate a filtered block, wherein the NN based filtering process includes performing multiple separable convolutions in parallel with point-wise input convolutions.

[0009] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 FIG. 1 is a block diagram illustrating an example video encoding and decoding system that can perform the techniques of this disclosure.

[0011] Figure 2 FIG. 2 is a block diagram illustrating an example hybrid video coding framework.

[0012] Figure 3 is a conceptual diagram illustrating a layered prediction structure using a picture group (GOP) size of 16.

[0013] Figure 4 is a block diagram illustrating an example convolutional neural network (CNN) based filter with four layers.

[0014] Figure 5 is a block diagram illustrating an example CNN based filter with padded input samples and supplemental data.

[0015] Figure 6 is a block diagram illustrating another example CNN based filter with padded input samples and supplemental data.

[0016] Figure 7 is a block diagram illustrating an example attention residual block of the neural network filter of Figure 6

[0017] Figure 8 is a block diagram illustrating an example of the spatial attention layer of Figure 6

[0018] Figure 9 is a block diagram illustrating an example of a simplified CNN based filter architecture with padded input samples and supplemental data using a residual block structure.

[0019] Figure 10 is a block diagram illustrating an example residual block structure of the example of Figure 9

[0020] Figure 11 is a block diagram illustrating an example of a simplified CNN based filter architecture with padded input samples and supplemental data using a filter block structure.

[0021] Figure 12 is a block diagram illustrating an example filter block structure of the example of Figure 11

[0022] Figure 13 is a block diagram illustrating an example of multi-dimensional convolution decomposition.

[0023] Figure 14 is a block diagram illustrating an example of multi-dimensional convolution decomposition.

[0024] Figure 15 An example architecture for a residual block is shown.

[0025] Figure 16 An example architecture for a residual block is shown.

[0026] Figure 17A and Figure 17B An example architecture for a residual block is shown.​​​​

[0027] Figure 18 and Figure 19 An example of a residual block architecture that does not include a 1x1 convolution branch is shown.

[0028] Figure 20 An example architecture for a residual block is shown.

[0029] Figure 21 An example architecture that utilizes convolution is shown.

[0030] Figure 22 FIG. 1 is a block diagram illustrating an example video encoder that can perform the techniques of this disclosure.

[0031] Figure 23 FIG. 1 is a block diagram illustrating an example video encoder that can perform the techniques of this disclosure.

[0032] Figure 24 FIG. 1 is a block diagram illustrating an example video encoder that can perform the techniques of this disclosure.

[0033] Figure 25 FIG. 1 is a block diagram illustrating an example video encoder that can perform the techniques of this disclosure.

[0034] Figure 26 FIG. 1 is a block diagram illustrating an example video encoder that can perform the techniques of this disclosure. DETAILED DESCRIPTION

[0035] Video coding (e.g., video encoding and / or video decoding) often involves predicting a block of video data from already coded blocks of video data in the same picture (e.g., intra prediction) or different pictures (e.g., inter prediction). In some cases, a video encoder also computes residual data by comparing a predicted block to an original block. Thus, the residual data represents the difference between the predicted block and the original block. To reduce the number of bits needed to signal the residual data, the video encoder transforms and quantizes the residual data and signals the transformed and quantized residual data in an encoded bitstream. The compression achieved by the transform and quantization process can be lossy, meaning that the transform and quantization process can introduce distortion into the decoded video data.

[0036] A video decoder decodes the residual data and adds the residual data to the predicted block to produce a reconstructed video block that more closely matches the original video block than the predicted block alone. Due to the loss introduced by the transform and quantization of the residual data, the first reconstructed block can have distortion or artifacts. One common type of artifact or distortion is referred to as blocking artifacts, where the boundaries of the blocks used to code the video data are visible.

[0037] To further improve the quality of decoded video, a video decoder can perform one or more filtering operations on a reconstructed video block. Examples of these filtering operations include deblocking filtering, sample adaptive offset (SAO) filtering, and adaptive loop filtering (ALF). Parameters for these filtering operations can be determined by a video encoder and explicitly signaled in an encoded video bitstream, or can be implicitly determined by a video decoder without needing to be explicitly signaled in the encoded video bitstream. This disclosure describes techniques related to convolutional neural network (CNN)-based in-loop filtering (or, more generally, neural network (NN)-based filtering).

[0038] Research has shown that neural network (NN)-based filtering techniques can improve decoded and / or regenerated video data well. NN-based filtering techniques can be very complex and require significant processing power to effectively perform.

[0039] This disclosure describes simplifications that can be applied to NN-based filtering techniques while still maintaining coding quality. For example, according to techniques of this disclosure, a video coder can be configured to perform a NN-based filtering process that includes performing multiple separable convolutions in parallel with point-wise input convolutions. Application of these simplifications can reduce processing performed by one or more processors to perform NN-based filtering with relatively minimal loss in coding quality. In this way, techniques of this disclosure can improve the performance of video coding devices. Also, these techniques can enable more devices to perform NN-based filtering, thereby generally improving the field of video coding.

[0040] As used in this disclosure, the term video coding generally refers to video encoding or video decoding. Similarly, the term video coder can generally refer to a video encoder or a video decoder. Moreover, certain techniques described in this disclosure with respect to video decoding can also apply to video encoding, and vice versa. For example, video encoders and video decoders are often configured to perform identical or reciprocal processes. Moreover, a video encoder typically performs video decoding (also referred to as reconstruction) as part of the process of determining how to encode video data. For example, a video encoder can perform deblocking filtering on decoded video blocks in order to determine whether a particular encoding scheme produces a desirable rate-distortion tradeoff, and also so that the video encoder can perform motion estimation using the same blocks available to the video decoder when the video decoder performs motion compensation.

[0041] Figure 1is a block diagram illustrating an example video encoding and decoding system 100 that can perform the techniques of this disclosure. The techniques of this disclosure generally relate to coding (encoding and / or decoding) video data, including filtering of video data using NN-based techniques. Generally, video data includes any data for processing video. Thus, video data can include raw, uncoded video, coded video, decoded (e.g., reconstructed) video, and video metadata, such as signaling data.

[0042] As shown in Figure 1 In this example, the system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116. In particular, the source device 102 provides the video data to the destination device 116 via a computer-readable medium 110. The source device 102 and the destination device 116 can be or include any of a wide variety of devices, including desktop computers, notebook (i.e., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, broadcast receiver devices, and the like. In some cases, the source device 102 and the destination device 116 can be equipped for wireless communication, and thus can be referred to as wireless communication devices.

[0043] In Figure 1 the example of FIG. 1, the source device 102 includes a video source 104, a memory 106, a video encoder 200, and an output interface 108. The destination device 116 includes an input interface 122, a video decoder 300, a memory 120, and a display device 118. In accordance with this disclosure, the video encoder 200 of the source device 102 and the video decoder 300 of the destination device 116 can be configured to apply the techniques for NN-based video coding and filtering. Thus, the source device 101 represents an example of a video encoding device, while the destination device 116 represents an example of a video decoding device. In other examples, a source device and a destination device can include other components or arrangements. For example, the source device 102 can receive video data from an external video source, such as an external camera. Likewise, the destination device 116 can interface with an external display device, rather than include an integrated display device.

[0044] As shown in Figure 1The illustrated system 100 is merely one example. In general, any digital video encoding and / or decoding device can perform the techniques for NN-based video coding and filtering. Source device 102 and destination device 116 are merely examples of such coding devices that generate coded video data from a source device 102 for transmission to a destination device. This disclosure refers to a “coding” device as a device that performs coding (encoding and / or decoding) of data. Thus, video encoder 200 and video decoder 300 represent examples of coding devices, in particular, a video encoder and a video decoder, respectively. In some examples, source device 102 and destination device 116 can operate in a substantially symmetrical manner, such that each of source device 102 and destination device 116 includes video encoding and decoding components. Hence, system 100 can support one-way or two-way video transmission between source device 102 and destination device 116, e.g., for video streaming, video playback, video broadcasting, or video telephony.

[0045] In general, video source 104 represents a source of video data (i.e., raw, unencoded video data) and provides a sequence of pictures (also referred to as “frames”) of the video data to video encoder 200, which encodes data for the pictures. Video source 104 of source device 102 can include a video capture device, such as a video camera, a video archive containing previously captured raw video, and / or a video feed interface to receive video from a video content provider. As a further alternative, video source 104 can generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes the captured, pre-captured, or computer-generated video data. Video encoder 200 can rearrange the pictures from the received order (sometimes referred to as “display order”) into the coding order for coding. Video encoder 200 can generate a bitstream including encoded video data. Source device 102 can then output the encoded video data via output interface 108 onto computer-readable medium 110 for reception and / or retrieval by, e.g., input interface 122 of destination device 116.

[0046] Memory 106 of source device 102 and memory 120 of destination device 116 represent general storage memory. In some examples, memories 106, 120 can store raw video data, e.g., raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106, 120 can store software instructions executable by, e.g., video encoder 200 and video decoder 300, respectively. Although memories 106 and 120 are shown as separate from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 can also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106, 120 can store encoded video data, e.g., output from video encoder 200 and input to video decoder 300. In some examples, portions of memories 106, 120 can be allocated as one or more video buffers, e.g., to store raw, decoded, and / or encoded video data.

[0047] Computer-readable medium 110 can represent any type of medium or device capable of transporting the encoded video data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium to enable source device 102 to transmit encoded video data directly to destination device 116 in real-time, e.g., via a radio frequency network or computer-based network. Output interface 108 can modulate a transmission signal including the encoded video data, and input interface 122 can demodulate the received transmission signal, according to a communication standard, such as a wireless communication protocol. The communication medium can comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet. The communication medium can include routers, switches, base stations, or any other equipment that can be useful to facilitate communication from source device 102 to destination device 116.

[0048] In some examples, source device 102 can output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 can include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

[0049] In some examples, source device 102 can output encoded video data to file server 114 or another intermediate storage device, which can store the encoded video data generated by source device 102. Destination device 116 can access stored video data from file server 114 via streaming or download.

[0050] File server 114 can be any type of server device capable of storing encoded video data and transmitting that encoded video data to destination device 116. File server 114 can represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a hypertext transfer protocol (HTTP) server, a multimedia broadcast multicast service (MBMS) or enhanced MBMS (eMBMS) server, and / or a network attached storage (NAS) device. Additionally or alternatively, file server 114 can implement one or more HTTP streaming protocols, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, and / or the like.

[0051] Destination device 116 can access encoded video data from file server 114 through any standard data connection, including an Internet connection. This can include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., digital subscriber line (DSL), a cable modem, etc.), or a combination of both that is suitable for accessing encoded video data stored on file server 114. Input interface 122 can be configured to operate according to any one or more of various protocols

[0052] Output interface 108 and input interface 122 can represent wireless transmitters / receivers, modems, wired networking components (e.g., Ethernet cards), wireless communication components operating according to any of a variety of IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 comprise wireless components, output interface 108 and input interface 122 can be configured to transmit and / or receive data (such as encoded video data) according to a cellular communication standard, such as 4G, 4G-LTE (Long-Term Evolution), LTE Advanced, 5G, or the like. In some examples where output interface 108 comprises a wireless transmitter, output interface 108 and input interface 122 can be configured to transmit and / or receive data (such as encoded video data) according to other wireless standards, such as an IEEE 802.11 specification, an IEEE 802.15 specification (e.g., ZigBee™), a Bluetooth™ standard, or the like. In some examples, source device 102 and / or destination device 116 can include respective system on a chip (SoC) devices. For example, source device 102 can include an SoC device that performs the functionality attributed to video encoder 200 and / or output interface 108, and destination device 116 can include an SoC device that performs the functionality attributed to video decoder 300 and / or input interface 122.

[0053] The techniques of this disclosure can be applied to video coding in support of any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions, such as Dynamic Adaptive Streaming over HTTP (DASH), digital video that is encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0054] Input interface 122 of destination device 116 receives an encoded video bitstream from computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, or the like). The encoded video bitstream can include signaling information defined by video encoder 200, which is also used by video decoder 300, such as syntax elements having values that describe properties and / or processing of video blocks or other coded units (e.g., slices, pictures, groups of pictures, sequences, or the like). Display device 118 displays decoded pictures of the decoded video data to a user. Display device 118 can represent any of a variety of display devices such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0055] Although not shown in FIG. 1, in some examples, source device 102 and / or destination device 116 can include multiple video encoders 200 and / or video decoders 300. For example, source device 102 can include a plurality of video encoders 200 for encoding video data simultaneously. Similarly, destination device 116 can include two or more video decoders 300 for decoding video data simultaneously. Figure 1Although not shown separately in FIG. 2, in some examples, video encoder 200 and video decoder 300 can each be integrated with an audio encoder and / or audio decoder, and can include appropriate MUX-DEMUX units, or other hardware and / or software, to handle multiplexed streams including both audio and video in a common data stream.

[0056] Video encoder 200 and video decoder 300 each can be implemented as any of a variety of suitable encoder and / or decoder circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic circuitry, software, hardware, firmware, or any combinations thereof. When the techniques are implemented partially in software, a device can store instructions for the software in a suitable, non- transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of video encoder 200 and video decoder 300 can be included in one or more encoders or decoders, either of which can be integrated as part of a combined encoder / decoder (CODEC) in a respective device. A device including video encoder 200 and / or video decoder 300 can implement video encoder 200 and / or video decoder 300 in a processing circuit, such as an integrated circuit and / or a microprocessor. Such a device can be a wireless communication device, such as a cellular phone or any other type of device described herein.

[0057] Video encoder 200 and video decoder 300 can operate according to a video coding standard, such as ITU-T H.265, also referred to as High Efficiency Video Coding (HEVC) or extensions thereto, such as the multi-view and / or scalable video coding extensions. Alternatively, video encoder 200 and video decoder 300 can operate according to other proprietary or industry standards, such as ITU-T H.266, also referred to as Versatile Video Coding (VVC). In other examples, video encoder 200 and video decoder 300 can operate according to a proprietary video codec / format, such as AOMedia Video 1 (AV1), extensions to AV1, and / or subsequent versions of AV1 (e.g., AV2). In other examples, video encoder 200 and video decoder 300 can operate according to other proprietary formats or industry standards. The techniques of this disclosure, however, are not limited to any particular coding standard or format. In general, video encoder 200 and video decoder 300 can be configured to perform the techniques of this disclosure in connection with any video coding technology that uses neural networks.

[0058] In general, video encoder 200 and video decoder 300 can perform block-based picture coding. The term “block” generally refers to a structure containing data to be processed (e.g., encoded, decoded, or otherwise used in the encoding and / or decoding process). For example, a block can include a two-dimensional matrix of samples of luma and / or chroma data. In general, video encoder 200 and video decoder 300 can code video data represented in a YUV (e.g., Y, Cb, Cr) format. That is, rather than coding red, green, and blue (RGB) data for samples of a picture, video encoder 200 and video decoder 300 can code luma and chroma components, where the chroma components can include red hue and blue hue chroma components. In some examples, video encoder 200 converts received data in an RGB format to a YUV representation prior to encoding, and video decoder 300 converts the YUV representation to the RGB format. Alternatively, pre- and post-processing units (not shown) can perform these conversions.

[0059] In general, this disclosure can relate to coding (e.g., encoding and decoding) of pictures to include processes of encoding or decoding data of pictures. Similarly, this disclosure can relate to coding of blocks of pictures to include processes of encoding or decoding data for blocks, e.g., prediction and / or residual coding. An encoded video bitstream generally includes a series of values for syntax elements representing coding decisions (e.g., coding modes) and partitioning of pictures into blocks. Accordingly, references to coding of pictures or blocks should generally be understood to refer to coding of values of syntax elements used to form the pictures or blocks.

[0060] HEVC defines various blocks, including coding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video coder (such as video encoder 200) partitions a coding tree unit (CTU) into CUs according to a quadtree structure. That is, the video coder partitions a CTU and CUs into four equal, non overlapping squares, and each node of the quadtree has either zero or four child nodes. Nodes with zero child nodes can be referred to as “leaf nodes,” and CUs of such leaf nodes can include one or more PUs and / or one or more TUs. Video coder can further partition PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents partitioning of TUs. In HEVC, PUs represent inter-predicted data, while TUs represent residual data. CUs that are intra-predicted include intra-prediction information, such as an intra-mode indication.

[0061] As another example, video encoder 200 and video decoder 300 can be configured to operate according to VVC. According to VVC, a video coder (such as video encoder 200) partitions a picture into multiple CTUs. Video encoder 200 can partition a CTU according to a tree structure such as a quad-tree binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure removes the multiple separation concepts between CUs, PUs, and TUs of HEVC. The QTBT structure includes two levels: a first level partitioned according to quad-tree partitioning, and a second level partitioned according to binary tree partitioning. The root node of the QTBT structure corresponds to a CTU. Leaf nodes of the binary tree correspond to CUs.

[0062] In the MTT partitioning structure, a block can be partitioned using quad-tree (QT) partitioning, binary tree (BT) partitioning, and one or more types of ternary tree (TT) (also referred to as triple tree (TT)) partitioning. A ternary tree or triple tree partition is a partitioning of a block into three sub-blocks. In some examples, a ternary tree or triple tree partition divides a block into three sub-blocks without partitioning the original block by a center. The partition types (e.g., QT, BT, and TT) in the MTT can be symmetric or asymmetric.

[0063] When operating according to the AV1 codec, video encoder 200 and video decoder 300 can be configured to code video data in blocks. In AV1, the largest block that can be processed is referred to as a superblock. In AV1, a superblock can be 128x128 luma samples or 64x64 luma samples. However, in subsequent video coding formats (e.g., AV2), a superblock can be defined by different (e.g., larger) luma sample sizes. In some examples, a superblock is the top level of a block quadtree. Video encoder 200 can further partition a superblock into smaller coding blocks. Video encoder 200 can partition superblocks and other coding blocks into smaller blocks using square or non-square partitions. Non-square blocks can include N / 2xN, NxN / 2, N / 4xN, and NxN / 4 blocks. Video encoder 200 and video decoder 300 can perform separate prediction and transform processes for each coding block.

[0064] AV1 also defines tiles of video data. A tile is a rectangular array of superblocks that can be coded independently of other tiles. That is, video encoder 200 and video decoder 300 can encode and decode coding blocks within a tile without using video data from other tiles. However, video encoder 200 and video decoder 300 can perform filtering across tile boundaries. The size of a tile can be uniform or non-uniform. Tile-based coding can enable parallel processing and / or multi-threading for encoder and decoder implementations.

[0065] In some examples, video encoder 200 and video decoder 300 can use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, image encoder 200 and image decoder 300 can use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luma component and another QTBT / MTT structure for the two chroma components (or two QTBT / MTT structures for the respective chroma components).

[0066] Video encoder 200 and video decoder 300 can be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, superblock partitioning, or other partitioning structures.

[0067] In some examples, a CTU includes a coding tree block (CTB) of luma samples, two corresponding CTBs of chroma samples of a picture having three sample arrays, or a CTB of samples of a monochrome picture or a picture coded using three separate color planes and syntax structures for coding samples. A CTB can be an NxN block of samples for some value of N such that the splitting of components to CTBs is a kind of partitioning. A component is an array or a single sample from one of the three arrays (luma and two chroma) that make up a picture in 4:2:0, 4:2:2, or 4:4:4 color formats, or an array or a single sample of an array that make up a picture in monochrome format. In some examples, a coding block is an MxN block of samples for some values of M and N such that the splitting of CTBs to coding blocks is a kind of partitioning.

[0068] Blocks (e.g., CTUs or CUs) can be grouped in various ways in a picture. As one example, a brick can refer to a rectangular region of CTU rows within a particular tile in a picture. A tile can be a rectangular region of CTUs within a particular tile column and a particular tile row in a picture. A tile column refers to a rectangular region of CTUs having a height equal to a height of the picture and a width specified by a syntax element (e.g., such as in a picture parameter set). A tile row refers to a rectangular region of CTUs having a height specified by a syntax element (e.g., such as in a picture parameter set) and a width equal to a width of the picture.

[0069] In some examples, a tile can be divided into multiple bricks, each of which can include one or more CTU rows within the tile. A tile that is not divided into multiple bricks can also be referred to as a brick. However, a brick that is a proper subset of a tile can not be referred to as a tile. Bricks in a picture can also be arranged in slices. A slice can be an integer number of bricks of a picture that can be uniquely contained in a single network abstraction layer (NAL) unit. In some examples, a slice includes a number of complete tiles, or only a contiguous sequence of complete bricks of one tile.

[0070] The present disclosure can use“NxN” and“N by N” interchangeably to refer to the sample dimensions of a block, such as a CU or other video block, in terms of vertical and horizontal dimensions, e.g., 16x16 samples or 16 by 16 samples. In general, a 16x16 CU will have 16 samples in the vertical direction (y = 16) and 16 samples in the horizontal direction (x = 16). Likewise, an NxN CU generally has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non negative integer value. The samples in a CU can be arranged in rows and columns. Moreover, a CU need not necessarily have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU can include NxM samples, where M is not necessarily equal to N.

[0071] Video encoder 200 encodes video data representing prediction and / or residual information for a CU, among other information. Prediction information indicates how to predict the CU in order to form a prediction block for the CU. Residual information generally represents a sample-by-sample difference between samples of the CU prior to encoding and the prediction block.

[0072] To predict a CU, video encoder 200 can generally form a prediction block for the CU through inter prediction or intra prediction. Inter prediction generally refers to predicting the CU from data of a previously coded picture, whereas intra prediction generally refers to predicting the CU from previously coded data of the same picture. To perform inter prediction, video encoder 200 can use one or more motion vectors to generate the prediction block. Video encoder 200 can generally perform a motion search to identify a reference block that closely matches the CU, e.g., according to a difference between the CU and the reference block. Video encoder 200 can calculate the difference metric using a sum of absolute difference (SAD), sum of squared difference (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculations to determine whether a reference block closely matches a current CU. In some examples, video encoder 200 can use uni -prediction or bi-prediction to predict a current CU.

[0073] Some examples of VVC also provide an affine motion compensation mode, which can be considered a type of inter prediction mode. In affine motion compensation mode, video encoder 200 can determine two or more motion vectors that represent non-translational motion, such as scaling or zooming, rotation, perspective motion, or other irregular types of motion.

[0074] To perform intra prediction, video encoder 200 can select an intra prediction mode to generate the prediction block. Some examples of VVC provide 67 intra prediction modes, which include various directional modes, as well as a planar mode and a DC mode. In general, video encoder 200 selects an intra prediction mode that describes neighboring samples of the current block (e.g., a block of a CU) from which to predict samples of the current block. Such samples can typically be above, above-left, or to the left of the current block in the same picture as the current block, assuming video encoder 200 is coding the CTU and CU in a raster scan order (from left to right, top to bottom).

[0075] Video encoder 200 encodes data representing a prediction mode for the current block. For example, for inter prediction modes, video encoder 200 can encode data representing which of various available inter prediction modes to use, as well as motion information for the corresponding mode. For example, for uni- or bi-prediction, video encoder 200 can encode motion vectors using advanced motion vector prediction (AMVP) or merge mode. Video encoder 200 can use similar modes to encode motion vectors for affine motion compensation modes.

[0076] AV1 includes two general techniques for encoding and decoding blocks of video data. The two general techniques are intra prediction (e.g., intra prediction or spatial prediction) and inter prediction (e.g., inter prediction or temporal prediction). In the context of AV1, when a block of a current frame of video data is predicted using an intra prediction mode, video encoder 200 and video decoder 300 do not use video data from other frames of the video data. For most intra prediction modes, video encoder 200 encodes the block of the current frame based on differences between sample values in the current block and prediction values generated from reference samples in the same frame. Video encoder 200 determines the prediction values generated from the reference samples based on the intra prediction mode.

[0077] Following prediction, such as intra prediction or inter prediction of a block, video encoder 200 can calculate residual data for the block. The residual data, such as a residual block, represents sample -wise differences between the block and a prediction block for the block formed using the corresponding prediction mode. Video encoder 200 can apply one or more transforms to the residual block to produce transformed data in a transform domain instead of the sample domain. For example, video encoder 200 can apply a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. Furthermore, video encoder 200 can apply a secondary transform, such as a mode-dependent non-separable secondary transform (MDNSST), a signal-dependent transform, a Karhunen-Loeve transform (KLT), and so on, following the first transform. Video encoder 200 produces transform coefficients following application of the one or more transforms.

[0078] As described above, following any transforms to produce transform coefficients, video encoder 200 can perform quantization of the transform coefficients. Quantization generally refers to a process that reduces the bit depth of transform coefficients to possibly reduce the amount of data used to represent the transform coefficients, providing further compression. By performing the quantization process, video encoder 200 can reduce the bit depth associated with some or all of the transform coefficients. For example, video encoder 200 can round n-bit values down to m-bit values during quantization, where n is greater than m. In some examples, to perform quantization, video encoder 200 can perform a bitwise right-shift of the values to be quantized.

[0079] Following quantization, video encoder 200 can scan the transform coefficients, producing a one-dimensional vector from the two-dimensional matrix including the quantized transform coefficients. The scan can be designed to place higher energy (and thus lower frequency) transform coefficients at the front of the vector, and lower energy (and thus higher frequency) transform coefficients at the back of the vector. In some examples, video encoder 200 can utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector, and then entropy encode the quantized transform coefficient values of the vector. In other examples, video encoder 200 can perform an adaptive scan. Following scanning of the quantized transform coefficients to form a one-dimensional vector, video encoder 200 can entropy encode the one-dimensional vector, e.g., according to context adaptive binary arithmetic coding (CABAC). Video encoder 200 can also entropy encode values for syntax elements describing metadata associated with the encoded video data for use by video decoder 300 when decoding the video data.

[0080] To perform CABAC, video encoder 200 can assign a context within a context model to a symbol to be transmitted. The context can relate to, for example, whether neighboring values of the symbol are zero-valued or not. The probability determination can be based on a context assigned to the symbol.

[0081] Video encoder 200 can also generate syntax data, such as block-based syntax data, picture-based syntax data, and sequence-based syntax data, or other syntax data such as a sequence parameter set (SPS), a picture parameter set (PPS), or a video parameter set (VPS), to video decoder 300, e.g., in a picture header, a block header, a slice header. Video decoder 300 can likewise decode such syntax data to determine how to decode corresponding video data.

[0082] In this way, video encoder 200 can generate a bitstream including encoded video data, e.g., syntax elements describing partitioning of a picture into blocks (e.g., CUs) and prediction and / or residual information for the blocks. Ultimately, video decoder 300 can receive the bitstream and decode the encoded video data.

[0083] In general, video decoder 300 performs a reciprocal process to that performed by video encoder 200 to decode the encoded video data of the bitstream. For example, video decoder 300 can decode values for syntax elements of the bitstream using CABAC in a manner substantially similar to, but reciprocal from, the CABAC encoding process of video encoder 200. The syntax elements can define partitioning information for partitioning a picture into CTUs, and partitioning each CTU according to a corresponding partition structure, such as a QTBT structure, to define CUs of the CTU. The syntax elements can further define prediction information and residual information for blocks (e.g., CUs) of the video data.

[0084] The residual information can be represented by, for example, quantized transform coefficients. Video decoder 300 can inverse quantize and inverse transform the quantized transform coefficients of a block to reproduce a residual block for the block. Video decoder 300 forms a prediction block for the block using the signaled prediction mode (intra prediction or inter prediction) and related prediction information (e.g., motion information for inter prediction). Video decoder 300 can then combine the prediction block and the residual block (on a sample-by-sample basis) to reproduce the original block. Video decoder 300 can perform additional processing such as performing a deblocking process to reduce visual artifacts along boundaries of the blocks.

[0085] The disclosure can generally relate to "signaling" particular information, such as syntax elements. The term "signaling" can generally refer to the communication of values for syntax elements and / or other data used to decode encoded video data. That is, video encoder 200 can signal values for syntax elements in a bitstream. Generally, signaling refers to generating values in a bitstream. As described above, source device 102 can transmit the bitstream to destination device 116 in substantially real time or non-real time, such as can occur when syntax elements are stored to storage device 112 for later retrieval by destination device 112.

[0086] The disclosure describes processes, techniques, and structures that can reduce the computational complexity and / or memory bandwidth requirements of neural network (NN)-based video coding tools. The example techniques described below relate to NN-aided in-loop filtering. However, the techniques of this disclosure are applicable to any NN-based video coding tool that uses input data with certain statistical properties. The techniques of this disclosure can be used in the context of advanced video codecs, such as extensions of VVC, next generation video coding standards, and / or any other video codec.

[0087] According to the techniques of this disclosure, video encoder 200 and video decoder 300 can be configured to perform NN-based video coding, including NN-based filtering using any combination of the techniques described below.

[0088] Figure 2 is a conceptual diagram showing a hybrid video coding framework. Video coding standards since H.261 have been based on the so-called hybrid video coding principle, which is shown in Figure 2 The term hybrid refers to the combination of two means for reducing redundancy in video signals, namely prediction and transform coding with quantization of the prediction residual. Prediction and transform reduce redundancy in video signals by decorrelation, while quantization reduces the data represented by the transform coefficients by lowering their precision, ideally by removing only uncorrelated details. This hybrid video coding design principle is also used for the two most recent standards, ITU-T H.265 / HEVC and ITU-T H.266 / VVC.

[0089] As shown in Figure 2 Modern hybrid video coders 130 typically perform block partitioning, motion-compensated prediction or inter-picture prediction, intra-picture prediction, transform, quantization, entropy coding, and post-loop / in-loop filtering. In Figure 2In the example of FIG. 1, video coder 130 includes a summation unit 134, a transform unit 136, a quantization unit 138, an entropy coding unit 140, an inverse quantization unit 142, an inverse transform unit 144, a summation unit 146, an in-loop filter unit 148, a decoded picture buffer (DPB) 150, an intra-prediction unit 152, an inter-prediction unit 154, and a motion estimation unit 156.

[0090] Generally, video coder 130, when encoding video data, can receive input video data 132. Block partitioning is used to divide a received picture (image) of video data into smaller blocks for operation of the prediction and transform processes. Early video coding standards used fixed block sizes, typically 16x16 samples. More recent standards, such as HEVC and VVC, employ tree-based partitioning structures to provide flexible partitioning.

[0091] Motion estimation unit 156 and inter-prediction unit 154 can predict input video data 132, e.g., from previously decoded data of DPB 150. Motion-compensated prediction or inter-picture prediction exploits redundancy existing between pictures of a video sequence (hence, “inter-picture”). According to block-based motion compensation used in modern video codecs, a prediction is obtained from one or more previously decoded pictures (i.e., reference pictures). The corresponding region used to generate the inter-predicted block is indicated by motion information including a motion vector and a reference picture index.

[0092] Summation unit 134 can compute residual data as a difference between input video data 132 and prediction data from intra-prediction unit 152 or inter-prediction unit 154. Summation unit 134 provides the residual block to transform unit 136, which applies one or more transforms to the residual block to produce a transform block. Quantization unit 138 quantizes the transform block to form quantized transform coefficients. Entropy coding unit 140 entropy-encodes the quantized transform coefficients, as well as other syntax elements such as motion information or intra-prediction information, to generate output bitstream 158.

[0093] Meanwhile, inverse quantization unit 142 inverse quantizes the quantized transform coefficients, and inverse transform unit 144 inverse transforms the transform coefficients to reproduce the residual block. Summation unit 146 combines (on a sample-by-sample basis) the residual block with the prediction block to produce a decoded block of video data. In-loop filter unit 148 applies one or more filters (e.g., at least one of a neural network-based filter, a neural network-based in-loop filter, a neural network-based post-in-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter) to the decoded block to produce a filtered decoded block.

[0094] According to the techniques of this disclosure, the neural network filtering unit of the loop filter unit 148 can receive data for a reconstructed picture of video data from the summing unit 146 and from one or more other units of the hybrid video coder 130, such as the transform unit 136, the quantization unit 138, the intra prediction unit 152, the inter prediction unit 154, the motion estimation unit 156, and / or one or more other filtering units within the loop filter unit 146. For example, the neural network filtering unit can receive data from a deblocking filter unit (also referred to as a "deblock unit") of the loop filter unit 148. For example, the neural network filtering unit can receive a boundary strength value that indicates whether a particular boundary is to be filtered for deblocking, and if so, the extent to which the boundary is to be filtered. For example, the boundary strength value can correspond to a number of samples on either side of the boundary to be modified and / or the extent to which the samples are to be modified.

[0095] In other examples, in addition to or instead of the boundary strength value, the neural network filtering unit can receive any or all of: coding unit (CU) partitioning data, prediction unit (PU) partitioning data, transform unit (TU) partitioning data, deblocking filter data, quantization parameter (QP) data, intra prediction data (e.g., reconstructed samples and / or prediction samples), inter prediction data (e.g., reconstructed samples and / or prediction samples), data representing a distance between a decoded picture and one or more reference pictures, or motion information for one or more decoded blocks of the decoded picture. The deblocking filter data can further include one or more of: whether a long filter or a short filter was used for deblocking, or whether a strong filter or a weak filter was used for deblocking. The data representing a distance between a decoded picture and a reference picture can be represented as a picture order count (POC) difference between POC values of the pictures.

[0096] A block of video data, such as a CTU or CU, can actually include multiple color components, e.g., a luminance or "luma" component, a blue hue chrominance or "chroma" component, and a red hue chrominance (chroma) component. The luma component can have a greater spatial resolution than the chroma components, and one of the chroma components can have a higher spatial resolution than the other chroma component. Alternatively, the luma component can have a greater spatial resolution than the chroma components, and both chroma components can have equal spatial resolution to each other. For example, in a 4:2:2 format, the luma component can be twice as large as the chroma components in the horizontal direction, and equal to the chroma components in the vertical direction. As another example, in a 4:2:0 format, the luma component can be twice as large as the chroma components in the horizontal direction and in the vertical direction. The various operations discussed above can generally be applied individually to each of the luma component and the chroma components (although certain coding information, such as motion information or intra prediction directions, can be determined for the luma component and inherited by the corresponding chroma components).

[0097] In recent video codecs, hierarchical prediction structures applied within a group of pictures (GOP) are used to improve coding efficiency. Figure 3 An example hierarchical prediction structure is shown with a group of pictures (GOP) size equal to 16.

[0098] Referring again to Figure 2 , intra-picture prediction exploits spatial redundancy present within a picture by deriving a prediction for a block from spatially neighboring (reference) samples that have already been coded / decoded. Directional angle prediction, DC prediction, and planar or plane prediction are used in the latest video codecs, including AVC, HEVC, and VVC.

[0099] Hybrid video coding standards apply block transforms to the prediction residuals, whether the prediction residuals are from inter-picture prediction or intra-picture prediction. In early standards, including H.261, H.262, and H.263, a discrete cosine transform (DCT) was employed. In HEVC and VVC, more transform kernels besides DCT were applied in order to take into account different statistics in specific video signals.

[0100] Quantization aims to reduce the precision of input values or sets of input values in order to reduce the amount of data required to represent these values. In hybrid video coding, quantization is typically applied to individual transformed residual samples (i.e., to transform coefficients), resulting in integer coefficient levels. In recent video coding standards, a step size is derived from a so-called quantization parameter (QP) that controls the trade-off between fidelity and bit rate. A larger step size reduces the bit rate, but also reduces the quality, which for example leads to blocky artifacts and blurry details in video pictures.

[0101] Entropy coding unit 140 can perform context adaptive binary arithmetic coding (CABAC) on the encoded video. CABAC is used in recent video codecs (e.g., AVC, HEVC, and VVC) due to its high efficiency.

[0102] Filtering unit 148 can perform post-loop or in-loop filtering. Post-loop / in-loop filtering is a filtering process (or a combination of such processes) applied to a reconstructed picture to reduce coding artifacts. The input to the filtering process is typically the reconstructed picture (or reconstructed blocks of the picture), which is a combination of a reconstructed residual signal (e.g., reconstructed samples) and a prediction (e.g., predicted samples), where the reconstructed samples include quantization errors. As Figure 2 indicated, the reconstructed picture after in-loop filtering is stored in decoded picture buffer (DPB) 150 and used as a reference for inter-picture prediction of subsequent pictures.

[0103] Coding artifacts are mainly determined by the QP. Therefore, QP information is typically used in the design of the filtering process. In HEVC, the in-loop filters include a deblocking filter and a sample adaptive offset (SAO) filter. In the VVC standard, an adaptive loop filter (ALF) is introduced as a third filter. The filtering process of the ALF is shown as follows: ,(1) wherein is the sample before the filtering process, is the sample value after the filtering process. denotes a filter coefficient, is a clipping function, and denotes a clipping parameter. The variables k and l vary between and , wherein L denotes the filter length. The clipping function corresponds to the function The clipping operation introduces nonlinearity to make ALF more efficient by reducing the impact of neighboring sample values that differ too much from the current sample value. In VVC, the filter parameters can be signaled in the bitstream and can be selected from a pre-defined filter set. The ALF filtering process can also be summarized with the following equation: (2).

[0104] The video encoder 200, the video decoder 300, and the hybrid video coder 130 can be configured to perform neural network (NN) based filtering for video coding. Figure 4 is a conceptual diagram showing a neural network based filter 170 with four layers. Various studies have shown that embedding a neural network (NN) into a hybrid video coding framework such as Figure 2 can improve compression efficiency. Neural networks have been used for intra prediction and inter prediction to improve prediction efficiency. Neural network based in-loop filtering is also a highlighted research topic in recent years. In some examples, the filtering process is applied as a post-filter. In such examples, the filtering process is applied to the output picture, and the unfiltered picture can be used as a reference picture.

[0105] In addition to existing filters, the NN based filter 170 can also be applied, such as a deblocking filter, sample adaptive offset (SAO), and / or adaptive loop filtering (ALF). The NN based filter can also be applied exclusively, where the NN based filter is designed to replace all existing filters. Additionally or alternatively, the NN based filter, such as the NN based filter 170, can be designed to complement, enhance, or replace any or all of the other filters.

[0106] Figure 4 An example of a convolutional neural network (CNN) based filter with four layers is shown. Figure 4 The NN based filtering process can take reconstructed samples as input, and the intermediate output is residual samples, which are added back to the input to refine the input samples. The NN filter can use all color components (e.g., Y, U, and V, or Y, Cb, and Cr, i.e., luma data 172A, blue hue chroma 172B, and red hue chroma 172C) as input 172 to exploit cross-component correlation. Different color components can share the same filter (including network structure and model parameters), or each component can have its own specific filter.

[0107] The filtering process can also be summarized as follows: (3) The model structure and model parameters of the NN-based filter can be predefined and stored at the video encoder 200 and the video decoder 300. The filter can also be signaled in the bitstream.

[0108] In Figure 4 examples, the NN-based filter can include a series of feature extraction layers followed by an output convolution. In Figure 4 examples, the feature extraction layers can include 3x3 conv layers followed by a parametric rectified linear unit (PReLU) layer. A convolution layer applies a convolution operation to input data, which involves a filter or kernel being slid over the input data (e.g., the reconstructed samples of the input 172) and computing a dot product at each location. The convolution operation essentially captures local patterns within the input data. For example, in the context of image processing, these patterns can be edges, textures, or other visual features. The filter or kernel is a small matrix of weights that are updated during a training process. By sliding this filter over the input data (or a feature map from a previous layer) and computing a dot product at each location, the convolution layer creates a feature map that encodes the spatial hierarchies and patterns detected in the input.

[0109] The output of a convolution layer is a set of feature maps that capture different aspects of the input data, each corresponding to a filter. As data passes through deeper layers of the network, the layers help the neural network learn more complex and abstract features. Figure 4 In the example named 3x3 conv 3x3x6x8, the first 3x3 indicates that the convolution layer has a 3x3 filter size (e.g., a 3x3 matrix). The 3x3x6x8 refers to both the input and output dimensions of the convolution layer, where 6 is the number of input channels and 8 is the number of output channels.

[0110] A PReLU layer is an activation function used in neural networks and was introduced as a variant of the ReLU (rectified linear unit) activation function. As described above, a convolution layer outputs a feature map representing features detected in the input, each corresponding to a filter. After a convolution layer, a PReLU layer applies a PReLU activation function to each element of the feature map produced by the convolution layer. For positive values, the PReLU layer acts like a standard ReLU, passing the values through. For negative values, instead of setting them to zero (e.g., as ReLU does), the PReLU layer allows small, linear, negative outputs. This can keep neurons active and maintain gradient flow, which is beneficial for learning in deep networks.

[0111] In summary, when a convolutional layer is followed by a PReLU layer, the convolutional layer first extracts features from the input data by learning a set of filters. The resulting feature maps are then passed through a PReLU activation function, which introduces nonlinearity and helps avoid the problem of dead neurons by allowing small gradients when the input is negative. This combination can be effective in learning complex patterns in the data while maintaining robust gradient flow, which is especially beneficial in deeper network architectures.

[0112] When applying NN-based filtering in video coding, the entire video signal (pixel data) can be divided into multiple processing units (e.g., 2D blocks), and each processing unit can be processed individually or combined with other information associated with the pixel block. Possible choices of processing units include frames, slices / tiles, CTUs, or any predefined or signaled shape and size. Typically, NN-based filtering is performed on reconstructed blocks of video data. Here, reconstructed blocks and samples can refer to decoded blocks produced by video decoder 300, as well as blocks reconstructed in the reconstruction loop of video encoder 200.

[0113] To further improve the performance of NN-based filtering, different types of input data can be jointly processed to produce filtered output. Input data can include, but is not limited to, reconstructed pixels / samples, predicted pixels / samples, pixels / samples after in-loop filters, partition structure information, deblocking parameters (e.g., boundary strength (BS)), QP values, slice or picture types, or filter applicability or coding mode maps. Input data can be provided at different granularities. Luma reconstructed and predicted samples can be provided at the original resolution, while chroma samples can be provided at a lower resolution (e.g., for 4:2:0 representation), or upsampled to the luma resolution to enable per-pixel representation. Similarly, QP, BS, partition, or coding mode information can be provided at a lower resolution, including cases with a single value per frame, slice, or processing block (e.g., QP). In other examples, QP, BS, partition, or coding mode information can be expanded (e.g., replicated) to enable per-pixel / sample representation.

[0114] Figure 5 An example of an architecture that utilizes supplemental data is shown in FIG. 5. Figure 5is a block diagram illustrating an example CNN-based filter with padded input samples and supplemental data. The NN-based filter 171 uses pixels / samples of a processing block combined with supplemental data as input 174. The input 174 can include 4 sub-blocks of interleaved luma samples (Yx4) 174A and associated blue hue chroma (U) data 174B and red hue chroma (V) data 174C. The supplemental data includes quantization parameter (QP) step size 176 and boundary strength (BS) 178. The area of input pixels / samples can be extended with 4 padded pixels / samples from each side. The resulting dimensions of the processing volume are (4+64+4)x(4+64+4)x(4Y+2UV+1QP+3BS).

[0115] With respect to the NN-based filter in Figure 4 The NN-based filter 171 can include two or more hidden layers with both lx l convolutions and Leaky ReLU layers, as opposed to the NN-based filter in PReLU layers. Similar to PReLU layers, LeakyReLU layers allow a small non-zero gradient to be output when the layer is not active. Instead of outputting zero for negative inputs, Leaky ReLU multiplies these inputs by a small constant. This small slope ensures that even neurons that were originally inactive can make a small contribution to the learning of the network, thus reducing the likelihood of the dead ReLU problem.

[0116] The video encoder 200, the video decoder 300, and the hybrid video coder 130 can be configured to implement NN-based filtering in a multi-mode design. To further improve the performance of NN-based filtering, a multi-mode solution can be designed. For example, for each processing unit, the video encoder 200 can select among a set of modes based on rate-distortion optimization, and the selection can be signaled in the bitstream. Different modes can include different NN models, different values of input information used as input to the NN model, and the like. In one example, the video encoder 200 and the video decoder 300 can use a NN-based filtering solution with multiple modes based on a single NN model by using different QP values as input to the NN model for different modes.

[0117] Examples of NN architectures will now be described. In one example, a NN-based filtering solution with multiple modes can be used, as described above. In Figure 6 The structure of the network is shown in Figure 6The NN-based filter of FIG. 1 includes a first portion that includes input 3x3 convolutional filters 510A-510E and a respective parametric rectified linear unit (PReLU) filter 512A-512E for each input to generate feature maps (e.g., a feature extraction portion of the NN filter). A concatenation unit 514 concatenates the feature maps and provides them to a fusion block 516 and a conversion block 522. Figure 6 The NN-based filter of FIG. 2 further includes a set of attention residual (AttRes) blocks 530A-530N; and a last portion (e.g., a tail) that includes a 3x3 convolutional filter 550, a PReLU filter 552, a 3x3 convolutional filter 554, and a pixel shuffle unit 556. The AttRes blocks can also be referred to as backbone blocks.

[0118] In the first portion (e.g., a feature extraction portion), different inputs are received, including quantization parameters (QP) 500, partition information (part) 502, boundary strength (BS) 504, prediction samples (pred) 506, and reconstructed samples (rec) 508. Respective 3x3 convolutional filters 510A-510E and PReLU filters 512A-512E convolve and activate the respective inputs to produce feature maps. A concatenation unit 514 then concatenates the feature maps. A fusion block 516 that includes a 1x1 convolutional filter 518 and a PReLU filter 520 fuses the concatenated feature maps. A conversion block that includes a 3x3 convolutional filter 524 and a PReLU filter 526 subsamples the fused input to produce an output 188. The output 188 is then fed through a set of attention residual blocks 530A-530N, which can include various numbers of attention residual blocks, such as 8. With respect to Figure 7 The attention blocks are further explained. The output 189 of the last attention residual block from the set of attention residual blocks 184 is fed to a last portion of the NN-based filter. In the last portion, a 3x3 convolutional filter 550, a PReLU filter 552, a 3x3 convolutional filter 554, and a pixel shuffle unit 556 process the output 189, and an addition unit 558 combines the result with the original reconstructed sample input 508. This ultimately forms a filtered output for presentation and storage as a reference for subsequent inter-prediction, for example, in a decoded picture buffer (DPB). In some examples, the last portion of the NN-based filter is a 1x1 convolutional filter. Figure 6 The NN-based filter of FIG. 1 uses 96 feature maps.

[0119] Figure 7 is a conceptual diagram illustrating Figure 6 an attention residual block of FIG. 2. That is, Figure 7 depicts an attention residual block 530, which can include a 3x3 convolutional filter 532, a PReLU filter 534, and a 1x1 convolutional filter 536. Figure 6components of the attention residual block 530A-530N. In this example, the attention residual block 530 includes a first 3x3 convolutional filter 532, a parametric rectified linear unit (PReLU) filter 534, a second 3x3 convolutional filter 536, an attention block 538, and a summation unit 540. The summation unit 540 combines the output of the attention block 538 and the output 188 originally received by the convolutional filter 532 to generate the output 189.

[0120] Figure 8 is a conceptual diagram illustrating an example spatial attention layer of Figure 7 Figure 8 As shown, the spatial attention layer of the attention residual block 530 includes a 3x3 convolutional filter 706, a PReLU filter 708, a 3x3 convolutional filter 710, a size expansion unit 712, a 3x3 convolutional filter 720, a PReLU filter 722, and a 3x3 convolutional filter 724. The 3x3 convolutional filter 706 receives an input 702 corresponding to the quantization parameter (QP) 500, the partition information (part) 502, the boundary strength (BS) 504, the prediction information (pred) 506, and the reconstructed block (rec) 508 of Figure 6 The 3x3 convolutional filter 720 receives the ZK 704. The output of the size expansion unit 712 and the 3x3 convolutional filter 724 are combined and then combined with the R value 730 to generate the S value 732. The S value 732 is then combined with the ZK value 704 to generate the output ZK+1 value 734.

[0121] In other examples, alternative designs of NN architectures can be used. For example, a larger number of low complexity residual blocks in the backbone of the filter of Figure 6 may be used, along with a reduced number of channels (feature maps) and removal of the attention modules. Such alternative convolutional neural network filter structures (e.g., for luma filtering) are illustrated in Figure 9 Figure 9 is a block diagram illustrating an example of a simplified CNN-based filter architecture with padded input samples and supplemental data.

[0122] Figure 9 ​​The NN-based filters include 3x3 convolutional filters 810A-810E and PReLU filters 812A-812E, which convolve the corresponding inputs (i.e., QP 800, Part 802, BS 804, Pred 806, and Rec 808) to generate feature maps (e.g., feature extraction parts). A cascade unit 814 concatenates the convolutional inputs (e.g., feature maps). A fusion block 816 then uses a 1x1 convolutional filter 818 and a PReLU filter 820 to fuse the cascaded feature maps. A transformation block 822 then uses a 3x3 convolutional filter 824 and a PReLU filter 826 to process the fused data.

[0123] In this example, the NN-based filter comprises a set 828 of residual blocks 830A-830N (also referred to as backbone blocks), where each residual block can be configured according to... Figure 10 The residual block structure 830 is used for construction, as described below. Residual blocks 830A-830N can be replaced... Figure 6 AttRes block 530A-530N. Figure 9 The example can be used for luminance (luma) filtering, although similar modifications can be made for chrominance (chroma) filtering as described below.

[0124] exist Figure 9 The number of residual blocks and channels included in set 828 can be configured differently. That is, N can be set to different values, and the number of channels in residual block structure 830 can be set to a number other than 160 to achieve different performance complexity trade-offs. These modifications can be used to perform chroma filtering for chroma channel processing.

[0125] The set 828 of residual blocks 830A-830N has N instances of residual block structure 830. In one example, N can be equal to 32, resulting in 32 residual block structures. Residual blocks 830A-830N can use 64 feature maps, which is relative to... Figure 6 The example uses a reduced number of 96 feature maps.

[0126] Figure 10 It shows Figure 9conceptual diagram of an example residual block structure 830. In this example, the residual block structure 830 includes a first 1x1 convolutional filter 832, which can increase the number of input channels to 160 before an activation layer (PReLU filter 834) processes the input channels. The PReLU filter 834 can thereby reduce the number of channels to 64 through this processing. Then, a second 1x1 convolutional filter 836 processes the reduced channels, followed by a 3x3 convolutional filter 838. Finally, a combination unit 840 can combine the output of the 3x3 convolutional filter 838 with the original input received by the residual block structure 830.

[0127] In yet another NN architecture, the residual blocks can be replaced by filter blocks (also referred to as backbone blocks), as shown in Figure 11 In this example, the bypass branches around all layers in each residual block are removed, as shown in Figure 12 The number of channels and the number of filter blocks are configurable, e.g., 64 channels, 24 filter blocks, with 160 channels before and after the activation.

[0128] Figure 11 is a conceptual diagram of another example filter block structure showing a set of attention residual blocks that can replace the set of attention residual blocks Figure 6 of FIG. 9. Figure 11 The NN-based filter of FIG. 9 includes 3x3 convolutional filters 1010A-1010E and PReLU filters 1012A-1012E, which convolve respective inputs (i.e., QP 1000, Part 1002, BS 1004, Pred 1006, and Rec 1008) to form feature maps (e.g., feature extraction portion). A concatenation unit 1014 concatenates the feature maps. A fusion block 1016 then fuses the concatenated inputs using a 1x1 convolutional filter 1018 and a PReLU filter 1020. A conversion block 1022 then processes the fused data using a 3x3 convolutional filter 1024 and a PReLU filter 1026.

[0129] In this example, the NN-based filter unit includes N filter blocks 1030A-1030N (also referred to as backbone blocks), each of which can have the structure of the filter block 1030 of FIG. 10, as described below. The filter block structure 1030 can be substantially similar to the residual block structure 830, except that the combination unit 840 is omitted from the filter block structure 1030, such that the input is not combined with the output. Instead, the output of each residual block structure can be fed directly to a subsequent block. Figure 12

[0130] ​The number of channels and the number of filter blocks can be configured. In one example, the parameters can be set to 64 channels and 32 filter blocks. The number of additional channels in each filter block 198 can be 160, as described above.

[0131] Figure 12 It is shown Figure 11 A conceptual diagram of an example filter block structure 1030. In this example, the residual block structure 1030 includes a first 1x1 convolutional filter 1032, which can increase the number of input channels to 160 before the activation layer (PReLU filter 1034) processes the input channels. The PReLU filter 1034 can thus reduce the number of channels to 64 through this processing. A second 1x1 convolutional filter 1036 then processes the reduced channels, followed by a 3x3 convolutional filter 1038. As described above, with Figure 10 Compared to the residual block structure 830, the filter block structure 1030 does not include a combination unit.

[0132] Convolutions with 3x3 kernels are popular in neural network-based filters. In the architecture described above, 3x3xNxM convolutions are utilized in multiple parts and blocks, where the 3x3 kernel slides in the spatial (2D) domain. However, multidimensional convolutions (such as 2D kernel convolutions) introduce significant complexity. According to the techniques of this disclosure, the video encoder 200 and the video decoder 300 can be configured to utilize separable convolutions instead of multidimensional convolutions (e.g., 3x3xNxM convolutions). For example, in any part of an neural network-based filter, two separable one-dimensional convolutions can be used instead of a 3x3 convolution. The use of separable convolutions can reduce computational complexity and memory bandwidth requirements.

[0133] To avoid overcomputation and reduce the parameter set derived from multidimensional convolutions (such as 3x3 convolutions (or 2D convolutional quantities with higher-dimensional kernels) in the CNN architecture or similar architectures described above), this disclosure describes techniques for configuring the video encoder 200 and video decoder 300 to utilize separable convolutions (e.g., 1D separable convolutions) generated by low-complexity approximations instead of multidimensional (e.g., 2D) convolutions that slide in spatial directions. While the techniques of this disclosure are described with reference to 3x3 convolutions (e.g., 4x4, 5x5, or larger), the decomposition techniques of this disclosure can be used for multidimensional convolutions of any size. Typically, multidimensional convolutions have a kernel size of n1 x n2 in the spatial dimension, where n1 and n2 are positive integers. The values ​​of n1 and n2 can be the same or different. Multidimensional convolutions can further have a size of K in the depth dimension (e.g., n1xn2xK). Furthermore, in the case of the number of output channels M, multidimensional convolutions can be represented as a 4-D tensor of n1xn2xKxM.

[0134] Video encoder 200 and video decoder 300 can be configured to perform multi-dimensional convolution decomposition. In one example of the disclosure, a low-rank convolution approximation decomposes a 3x3xMxN convolution into a pixel-wise convolution (1x1xMxR), two separable convolutions (3x1xRxR, 1x3xRxR), and another pixel-wise convolution (1x3xRxN). Here, R is the rank of the approximation, and can be used to adjust the performance / complexity of the approximation. The value of R can be an integer. In some examples, R can be derived as a function (ratio) of M or N or max(M,N). In some examples, R can be set equal to A* max(M,N), where A is less than 1 (e.g., 0.2, 0.5, 0.8), A is higher than 1 (e.g., 1.0, 1.2), or other values.

[0135] In a general example, a multi-dimensional convolution can be approximated by a plurality of separable convolutions by performing a first convolution of size n1x1 on the output of the first convolution and performing a second convolution of size 1xn2.

[0136] Figure 13 One example is shown of using separable convolutions to approximate a multi-dimensional convolution in a backbone block of a NN-based filter. In this example, the multi-dimensional convolution is a 2D 3x3 convolution. However, the techniques of the disclosure can be extended to convolutions of other dimensions.

[0137] Figure 13 An example is shown in which the backbone block is a residual block 830A from Figure 9 and Figure 10 of FIG. 8. Figure 13 An example is shown of decomposing a 3x3xKxK convolution 838 into a series of 1D and separable convolutions in residual block 1300. That is, video encoder 200 and video decoder 300 can be configured to perform residual block 1300, which includes performing a plurality of separable convolutions to approximate a multi-dimensional convolution. In the example of FIG. 13, Figure 13 The plurality of separable convolutions includes a 3x1xRxR separable convolution 1304 and a 1x3xRxR separable convolution 1306 in the example of FIG. 13. In other examples, the order of separable convolution 1304 and separable convolution 1306 can be switched.

[0138] Accordingly, in one example of the disclosure, to perform a plurality of separable convolutions to approximate a multi-dimensional convolution, video encoder 200 and video decoder 300 can be configured to perform a first 1x1 convolution (e.g., 1x1xKxR convolution 1302), perform a first separable convolution of the plurality of separable convolutions on an output of the first 1x2 convolution (e.g., 3x1xRxR separable convolution 1304), perform a second separable convolution of the plurality of separable convolutions on an output of the first separable convolution (e.g., 1x3xRxR separable convolution 1306), and perform a second 1x1 convolution on an output of the second separable convolution (e.g., 1x1xRxK separable convolution 1308). The number of output channels of the first 1x1 convolution 1302 is used to control the complexity approximation of the multi-dimensional convolution. The number of output channels of the separable convolutions 1304 and 1306 can be selected to control the complexity approximation of the multi-dimensional convolution. The separable convolutions 1304 and 1306 are performing a depthwise convolution operation.

[0139] In another example, to perform a plurality of separable convolutions to approximate a multi-dimensional convolution, video encoder 200 and video decoder 300 can receive an input, perform a 1x1xKxM convolution on the input, perform a PReLU layer on an output of the 1x1xKxM convolution, perform a 1x1xMxK convolution on an output of the PReLU, perform a 3x1xKxR separable convolution on an output of the 1x1xMxK convolution, perform a 1x3xRxK separable convolution on an output of the 3x1xKxR separable convolution, and perform a 1x1xRxK convolution on an output of the 1x3xRxR separable convolution.

[0140] In another example, Figure 13 An example is shown in which a 3x3xKxK convolution 838 is decomposed in the residual block 1300 into a series of 1D and separable convolutions, but in which the first 1D convolution is fused with another 1D convolution. That is, video encoder 200 and video decoder 300 can be configured to perform a residual block 1310 that includes performing a 1x1xMxR convolution 1320, which is a fusion of a 1x1xMxK convolution (e.g., convolution 836) and a 1x1xKxR convolution (e.g., convolution 1302). In this way, the decomposition of the multi-dimensional convolution can be further simplified. The fusion of the 1x1 convolutions in the residual block 1310 can be used in any case in which the decomposed convolutions are applied before or after another pixel-wise convolution (e.g., 1x1 fusion) without any nonlinearity or residual connection in between. The values of M and R can be selected to control the complexity and accuracy of the approximation.

[0141] In other examples of the disclosure, the NN-based filtering process includes a cascaded (e.g., sequentially utilized) application of the backbone block. For example, the backbone block can be applied to multiple different color components. In other examples, the NN-based filtering process includes a cascaded application of the backbone block applied in two or more parallel processing branches.

[0142] In one or more examples of the disclosure, performing multiple separable convolutions to approximate a multi-dimensional convolution in a backbone block of a NN-based filtering process includes applying an element-wise activation process as part of the multi-dimensional convolution. Examples of the element-wise activation process can include ReLU and PReLU functions. The PReLU function is an example of a parameter-controlled element-wise activation process.

[0143] Different algorithms for determining separable kernels to replace 2D or other multi-dimensional kernels can be used to determine the decomposition. In some examples, a Candecomp / Parafac (CP) tensor decomposition can be used. Examples of other decompositions suitable for use with the present disclosure can be found in V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, V. Lempitsky, Speeding up Convolutional Neural Networks Using Fine-tuned CP-Decomposition, ICLR 2015.

[0144] Alternative implementations and architectures will now be described. In some examples, different dimensional 2D convolutions (e.g., ZxY) or high dimensional convolution components can be used and replaced with corresponding separable convolutions 1xZ and Yx1.

[0145] Figure 14 is a block diagram illustrating another example of a multi-dimensional convolution decomposition. As Figure 14 shown, a 3x3xKxK convolution 838 of a residual block 830A is approximated by a 3x1xKxR convolution 1400 and then by a 1x3xRxK convolution 1410. In other examples, the locations of the convolution 1400 and the convolution can be switched. R is a regular rank of the decomposition. A lower rank means a greater complexity reduction. In comparison to the example of Figure 13 The video encoder 200 and the video decoder 300 can approximate a 3x3 convolution with two separable convolutions and not perform the leading or trailing 1x1 convolutions.

[0146] Aspects of integrating the described techniques into CNN architectures will now be described. While Figure 13 the example is with respect to a CNN architecture in Figure 9 andFigure 10 The residual blocks described in this disclosure are used to describe the process, but the techniques of this disclosure can be used with any multidimensional convolution found anywhere in a neural network-based filter. For example, in Figure 6-7 (Attention residual block 530A-N) and Figure 11-12 The 3x3 convolutions in the backbone blocks of (filter block 1030A-N) can also be approximated using multiple separable convolutions. Furthermore, the feature extraction portion of any NN-based filter described above (e.g., Figure 11 Convolutional layers 1010A-E and PReLU layers 1012A-E), fusion blocks (e.g., Figure 11 Block 1016), transformation block (e.g., Figure 11 Block 1022), backbone block (e.g., Figure 11 Block 1028) or tail (e.g., Figure 11 Any multidimensional convolution (e.g., a 3x3 convolution) shown in the convolution 1050, PReLU 1052, convolution 1054, and pixel shuffling 1056 can be used as described in the reference above. Figure 13 or Figure 14 The description uses multiple separable convolutions to approximate the data. Typically, the feature extraction portion of a neural network can include any layer configured to extract features from the input data. The tail of the neural network can include any number of layers at the end of the network before the output.

[0147] The aspects of decomposing the training will now be described. In some examples, the parameters of the separable convolution can be extracted from the training parameters of the 2D convolution. An example extraction model can be implemented as follows.

[0148] For CP decomposition, 3D (e.g., 3x3xk) convolution kernels It can be broken down into ,in It is the rank of the decomposition, and , These are kernels in a certain dimension. The kernel parameters can be optimized using a nonlinear optimization process (e.g., Gaussian-Newton). The squared error is used for calculation.

[0149] Alternatively, other processes inspired by stochastic gradient descent and machine learning practices exist. Consider multiple approximate 3x3 convolutions within a neural network model. These approximate convolutions can be optimized individually (e.g., sequentially trained to match the output feature maps of the original 3x3 convolutions with any accumulated approximation errors from previously approximate convolutions in the network), or jointly optimized across the entire network (e.g., by substituting approximate convolutions into the network and then optimizing the entire network or only all approximate convolutions end-to-end).

[0150] The 3x3 convolution approximation techniques described herein for CNN architectures (e.g., similar to Figure 13 and Figure 14 ) can be applied to different types of architectures and modules that employ 2D convolutions in sliding windows over spatial dimensions and receive supplemental information as input data. The application of the techniques described herein can be used not only in the body of a NN architecture, but also in the head blocks (e.g., feature extraction, fusion blocks, and / or conversion blocks) or tail of the architecture.

[0151] Another example of a CNN ILF architecture is a multi-mode CNN ILF with separable convolutions. Further complexity reduction of the CNN ILF architecture is achieved by utilizing separable convolutions instead of 2D convolutions (3x3). In JVET-AD00205, EE1 test 1.3.5, a low-rank convolution approximation decomposes a 3x3xMxN convolution into two separable convolutions (3x1xMxR, 1x3xRxN). This is applied to the residual blocks of the architecture described in JVET-AC0155. Here, R is the rank of the approximation, and the performance / complexity of the approximation can be eliminated.

[0152] In some examples, the architecture of Figure 14 with the decomposition shown in Figure 9 is implemented with parameters K = 64, M = 160, and R = 51, and the total number of 24 residual blocks results in a complexity of the network of 356.43 kMAC, and the number of parameters is 1.07M for the intra-luma model.

[0153] Convolution with 3x3 kernels is popular in NN-based filters. In the architectures described above, 3x3xNxM convolutions are utilized in multiple modules, where the 3x3 kernel slides in the spatial (2D) domain. However, 2D kernel convolutions introduce significant complexity compared to separable processing techniques that are typically used for filtering or transforms in video coding. As described above with respect to the multi-mode CNN ILF with separable convolutions, using separable convolutions instead of 2D allows some complexity reduction. However, as described above, the decomposition comes at the cost of certain coding performance degradation and lack of robustness to quantization of its parameters. Alternative solutions for the decomposition with better complexity / performance tradeoff and robustness to quantization errors are desired.

[0154] To avoid excessive computation and reduce the parameter set originating from 3x3 convolutions (or 2D convolution components of higher dimensional kernels) in the CNN architectures proposed in JVET-AC0155, JVET-Z0113, or similar documents, this disclosure proposes techniques that utilize large filters prior to activation and convolution kernel approximation. Examples of the approximation can include tensor decomposition or have group convolutions, which have low complexity compared to traditional 2D convolutions.

[0155] To improve Figure 9 the performance of the basic design (e.g., JVET-AC0155) shown in Figure 15 The proposed architecture of the residual block is shown in

[0156] Figure 15 An example of the proposed architecture of the residual block is shown. In this example, the residual block structure 1530 includes a first 1x1 convolutional filter 1532 before a first activation layer (PReLU filter 1534), and a 3x3 convolutional filter 1540 and a second activation layer (PReLU filter 1542) in parallel with the first 1x1 convolutional filter 1532 and the PReLU filter 1534. Then, a second 1x1 convolutional filter 1536 processes the combined output of the PReLU filter 1534 and the PReLU filter 1542, followed by a 3x3 convolutional filter 1538.

[0157] The separable decomposition of the 3x3 convolutional kernel can provide the desired complexity reduction. The parameters of the decomposition can control the amount of complexity reduction.

[0158] In Figure 16 some examples of the residual block shown, the 3x3 convolution is decomposed into a 3x1xC1xR convolution and followed by a 1x3xRx C2 convolution, where C1 and C2 are the number of input and output channels, respectively, and R is the rank of the approximation. The parameter R can be derived as R = C1xC2 / (C1+C2), and controls the complexity of the approximation.

[0159] In some examples, the residual block has Figure 16 The architecture of the residual block shown in Figure 9 is implemented with parameters R1 = 8, R2 = 44, M1 = 160, and M2 = 32, and a total number of 24 residual blocks, such that the complexity of the network is 358.43 kMAC, and the number of parameters is 1.07M for the intra luma model.

[0160] Figure 16 The proposed architecture is shown replacing the 3x3 convolutional block with 3x1 and 1x3 separable convolutions. In this example, the residual block structure 1630 includes a first 1x1 convolutional filter 1632 before a first activation layer (PReLU filter 1634), and a 3x3 convolutional filter 1640 and a second activation layer (PReLU filter 1642) in parallel with the first 1x1 convolutional filter 1632 and the PReLU filter 1634. Then, a second 1x1 convolutional filter 1636 processes the combined output of the PReLU filter 1634 and the PReLU filter 1642, followed by a 3x3 convolutional filter 1638. However, in Figure 16 the example of FIG. 16A, the 3x3 convolutional filter 1640 can be approximated using multiple separable convolutions, as shown by the 3x1 convolutional filter 1650 and the 1x3 convolutional filter 1652 in FIG. 16B. Figure 16 Similarly, the 3x3 convolutional filter 1638 can be approximated using multiple separable convolutions, as shown by the 3x1 convolutional filter 1660 and the 1x3 convolutional filter 1662 in FIG. 16C. Figure 16

[0161] Figure 17A and Figure 17B An example architecture of a residual block replacing the 3x3 convolution with a group convolution is shown. In Figure 17A the group convolution process is followed by a 1x1 convolution. Figure 17B Integrating group convolutions into residual blocks is included. In Figure 17A the example of FIG. 17, a 3x3xC1xR group convolution (followed by a 1x1xRxC2 convolution 1704) using n groups (1702_1 through 1702_N) approximates a 3x3 convolution, where C1 is the number of input channels and C2 is the number of output channels. For an input data volume of HxWxC1, each group operates on a volume of HxWx(C1 / n), and the output of the convolution can result in a volume of HxWxR for each group. The outputs of all groups are concatenated into data with dimensions HxWxR. In some examples, C1 = C2 = 64, n = 4, and R = 64, which would result in 4 groups of 16 features each. Additionally, C1 and R should be divisible by n as an integer value. The number of groups (the value of n) can be chosen in a range from 1 to C1. The process is shown in FIG. 17. Figure 17A

[0162] In Figure 17B ​​In the example, the residual block structure 1730 includes a first 1x1 convolutional filter 1732 preceding a first activation layer (PReLU filter 1734), and a 3x3 convolutional filter 1740 and a second activation layer (PReLU filter 1742) running in parallel with the first 1x1 convolutional filter 1732 and the PReLU filter 1734. Then, a second 1x1 convolutional filter 1736 processes the combined output of the PReLU filters 1734 and 1742, followed by the 3x3 convolutional filter 1738. Figure 17B In the example, with Figure 15 Conversely, the 3x3 convolutional filter 1740 can be approximated by a group convolution of n groups of 3x3xKxR1 1750 and a subsequent 1x1xR1xM2 convolutional filter 1752. Similarly, the 3x3 convolutional filter 1738 can be approximated by a group convolution of n groups of 3x3xKxR1 1760 and a subsequent 1x1xR2xK convolutional filter 1762.

[0163] In some examples, Figure 9 The architecture uses group convolutions in residual blocks (such as...) Figure 17B As shown, this is achieved by replacing two modules of a 3x3 convolution. To replace the 3x3xKxM2 convolution, the parameters of the group convolution are C1=C2=K=64, R1=32, n=4, and M2=32. To replace the 3x3xKxK convolution, the parameters of the group convolution are C1=C2=K=64, R2=64, and n=4. With a total of 24 residual block results, the network with group convolution utilization ( Figure 9 The complexity is evaluated to be equal to 366kMAC, and for the intra-frame luminance model, the number of parameters is 1.1M.

[0164] In other examples, R and n can be set to other values, and both R and n control the approximate complexity for different performance-complexity trade-offs.

[0165] In some examples, the architecture can be simplified by removing the 1x1 convolution and the associated PReLU module. Figure 18 and Figure 19 Examples of residual block architectures for separable decomposition and group convolution-based designs are shown respectively. For some applications and specific signal classes, the proposed architectures can offer a better performance-complexity tradeoff.

[0166] In some examples, having Figure 19 The residual block Figure 9 The architecture (with parameters R1=R2=64 and n=4) results in a network complexity of 369kMAC, and for the intra-frame luminance model, the number of parameters is 1142116.

[0167] Figure 18 and Figure 19 An example of a residual block architecture that terminates a 1x1 convolution branch is shown.

[0168] exist Figure 18 In the example, the residual block structure 1830 includes a 3x3 convolutional filter 1840 and an activation layer (PReLU filter 1842). A 1x1 convolutional filter 1836 then processes the output of the PReLU filter 1842, followed by a 3x3 convolutional filter 1838. Figure 18 In the example, multiple separable convolutions can be used (shown as...). Figure 18 The 3x3 convolution filter 1840 can be approximated by the 3x1 convolution filter 1850 and the 1x3 convolution filter 1852 in the example. Similarly, multiple separable convolutions (shown as...) can be used. Figure 18 The 3x1 convolution filter 1860 and the 1x3 convolution filter 1862 are used to approximate the 3x3 convolution filter 1838. Figure 18 In the example, box 1833 shows the removal of the 1x1 convolutional filter and activation layer, such as Figure 16 The 1x1 convolution filter 1630 and the PReLU filter 1634 are used in this context.

[0169] exist Figure 19 In the example, the residual block structure 1930 includes a 3x3 convolutional filter 1940 and an activation layer (PReLU filter 1942). A 1x1 convolutional filter 1936 then processes the output of the PReLU filter 1942, followed by a 3x3 convolutional filter 1938. Figure 19 In the example, multiple separable convolutions can be used (shown as...). Figure 19 The 3x3 convolution filter 1940 can be approximated by the 3x1 convolution filter 1950 and the 1x3 convolution filter 1952. Similarly, the 3x3 convolution filter 1938 can be approximated by a group convolution 1950 of n groups of 3x3xC1xR and a subsequent 1x1xRxC2 convolution filter 1952. Figure 19 In the example, box 1933 shows the removal of the 1x1 convolutional filter and activation layer, such as Figure 16 The 1x1 convolution filter 1630 and the PReLU filter 1634 are used in this context.

[0170] exist Figure 18 and Figure 19 In another example implementation of the technique, the first 3x3 convolution branch can be removed, leaving only the 1x1 convolution branch.

[0171] exist Figure 20In the example of FIG. 20, the residual block structure 2030 includes a first 1x1 convolutional filter 2032, and a second 3x1 convolutional filter 2050 and a third 1x3 convolutional filter 2052 in parallel with the first 1x1 convolutional filter 2032. A PReLU filter 2034 processes the combined output of the first 1x1 convolutional filter 2032 and the third 1x3 convolutional filter 2042. The output of the PReLU filter 2034 is then processed by a 1x1 convolutional filter 2036, a 1x3 convolutional filter 2038, and a 3x3 convolutional filter 2040.

[0172] In some examples, such as Figure 20 In the example of FIG. 20, the residual block structure 2030 includes a first 1x1 convolutional filter 2032, and a second 3x1 convolutional filter 2050 and a third 1x3 convolutional filter 2052 in parallel with the first 1x1 convolutional filter 2032. A PReLU filter 2034 processes the combined output of the first 1x1 convolutional filter 2032 and the third 1x3 convolutional filter 2042. The output of the PReLU filter 2034 is then processed by a 1x1 convolutional filter 2036, a 1x3 convolutional filter 2038, and a 3x3 convolutional filter 2040. Figure 20 In the example of FIG. 20, both 3x1 convolutions (e.g., 2050 and 2048) can be implemented with group convolutions of group size equal to n (e.g., n = 2 and low rank R1= 32, R2= 48, and K = 64).

[0173] In the example of FIG. 20, the 3x3 convolutional filter 2040 can be approximated using an n-group 3x3xKxR1 group convolution 2070 followed by a 1x1xR1xM2 convolutional filter 2072. Figure 17B Figure 15 In contrast to the example of FIG. 20, the 3x3 convolutional filter 1740 can be approximated using an n-group 3x3xKxR1 group convolution 1750 followed by a 1x1xR1xM2 convolutional filter 1752. Similarly, the 3x3 convolutional filter 1738 can be approximated using an n-group 3x3xKxR1 group convolution 1760 followed by a 1x1xR2xK convolutional filter 1762.

[0174] Further, different algorithms for replacing 2D kernels with lookup separable kernels can be employed. In some examples, examples of other decompositions applicable here are described in V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, V. Lempitsky, Speeding up Convolutional Neural Networks Using Fine-Tuned CP-Decomposition, ICLR 2015, available online at https: / / arxiv.org / pdf / 1412.6553.pdf, which is incorporated herein by reference.

[0175] Integration with CNN architectures will now be described. Utilizing the proposed decompositions and complexity reduction techniques in CNN architectures can be applied outside of filter blocks / residual blocks. Modules of 3x3xMxN convolutions of additionally implemented architectures can be decomposed by Figure 21 ​The remaining blocks of 2010A-2010E, 2024, 2050, and 2054 are marked. Figure 21 are operated in the same manner as described with respect to Figure 6 .

[0176] It has been determined that the 3x3 convolution approximation process presented herein for the CNN architecture of JVET-AC0155 can be applied to different types of architectures and modules that employ 2D convolutions in a sliding window over the spatial dimensions and receive supplemental information as input data. The application of the presented techniques can not only be used in the body of the NN architecture, but also in the head or tail blocks of the architecture.

[0177] The adoption of the techniques described herein in the NNVC architecture can reduce the computational complexity and memory bandwidth requirements. The examples described in this document relate to NN-aided loop filtering; however, these examples can also be applicable to any NN-based video coding tool that consumes input data with certain statistical properties, such as static content or sparse representation.

[0178] Figure 22 FIG. 2 is a block diagram illustrating an example video encoder 200 that can perform the techniques of this disclosure. Figure 22 is provided for purposes of explanation and should not be considered to be limiting of the technology as broadly exemplified and described in this disclosure. For purposes of explanation, this disclosure describes video encoder 200 according to the VVC and HEVC techniques. However, the technology of this disclosure can be performed by video encoding devices configured to other video coding standards and video coding formats, such as AV1 and subsequent versions of the AV1 video coding format.

[0179] In Figure 22In an example, video encoder 200 includes video data memory 230, mode select unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, decoded picture buffer (DPB) 218, and entropy encoding unit 220. Any or all of video data memory 230, mode select unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, DPB 218, and entropy encoding unit 220 can be implemented in one or more processors or in processing circuitry. For instance, the units of video encoder 200 can be implemented as one or more circuits or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Also, video encoder 200 can include additional or alternative processors or processing circuitry to perform these and other functions.

[0180] Video data memory 230 can store video data to be encoded by the components of video encoder 200. Video encoder 200 can receive the video data stored in video data memory 230 from, for example, video source 104 Figure 1 DPB 218 can function as a reference picture memory that stores reference video data for use in prediction of subsequent video data by video encoder 200. Video data memory 230 and DPB 218 can be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 230 and DPB 218 can be provided by the same memory device or separate memory devices. In various examples, video data memory 230 can be on-chip with other components of video encoder 200, as illustrated, or off-chip relative to those components.

[0181] In this disclosure, reference to video data memory 230 should not be interpreted as being limited to memory internal to video encoder 200 (unless specifically described as such) or memory external to video encoder 200 (unless specifically described as such). Rather, reference to video data memory 230 should be understood as reference to a memory that stores video data that video encoder 200 receives for encoding (e.g., video data for a current block to be encoded). Figure 1 Memory 106 of source device 102 can also provide temporary storage of the outputs from the various units of video encoder 200.

[0182] Figure 22 The various units of video encoder 200 are shown to help understand the operations performed by video encoder 200. The units can be implemented as fixed- function circuits, programmable circuits, or a combination thereof. Fixed-function circuits refer to circuits that provide particular functionality, and are preset on the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks, and provide flexible functionality in the operations that can be performed. For instance, programmable circuits can execute software or firmware that cause the programmable circuits to operate in the manner defined by instructions of the software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive parameters or output parameters), but the types of operations that the fixed-function circuits perform are generally immutable. In some examples, one or more of the units can be distinct circuit blocks (fixed-function or programmable), and in some examples, one or more of the units can be integrated circuits.

[0183] Video encoder 200 can include arithmetic logic units (ALUs), elementary function units (EFUs), digital circuits, analog circuits, and / or programmable cores formed from Figure 1 ) can store instructions (e.g., object code) for software that video encoder 200 receives and executes, or another memory within video encoder 200 (not shown) can store such instructions.

[0184] Video data memory 230 is configured to store video data that is received. Video encoder 200 can retrieve pictures of the video data from video data memory 230 and provide the video data to residual generation unit 204 and mode selection unit 202. Video data in video data memory 230 can be raw video data that is to be encoded.

[0185] Mode selection unit 202 includes motion estimation unit 222, motion compensation unit 224, and intra prediction unit 226. Mode selection unit 202 can include additional functional units to perform video prediction according to other prediction modes. As examples, mode selection unit 202 can include a palette unit, an intra-block copy unit (which can be part of motion estimation unit 222 and / or motion compensation unit 224), an affine unit, a linear model (LM) unit, etc.

[0186] The mode select unit 202 generally coordinates multiple encoding passes to test combinations of encoding parameters and resulting rate-distortion values for those combinations. The encoding parameters can include partitioning of CTUs into CUs, prediction modes for CUs, transform types for residual data of CUs, quantization parameters for residual data of CUs, and so on. The mode select unit 202 can ultimately select a combination of encoding parameters that has a better rate-distortion value than other tested combinations.

[0187] The video encoder 200 can partition a picture retrieved from the video data memory 230 into a series of CTUs, and encapsulate one or more CTUs within a slice. The mode select unit 202 can partition the CTUs of the picture according to a tree structure such as an MTT structure, a QTBT structure, a superblock structure, or the quadtree structure described above. As described above, the video encoder 200 can form one or more CUs from partitioning a CTU according to the tree structure. Such CUs are also commonly referred to as “video blocks” or “blocks,” in general.

[0188] In general, the mode select unit 202 also controls its components (e.g., the motion estimation unit 222, the motion compensation unit 224, and the intra prediction unit 226) to generate a prediction block for a current block (e.g., a current CU, or in HEVC, an overlapping portion of a PU and a TU). For inter prediction of a current block, the motion estimation unit 222 can perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previously coded pictures stored in the DPB 218). In particular, the motion estimation unit 222 can calculate a value that represents a degree of similarity of a potential reference block to the current block, e.g., according to a sum of absolute difference (SAD), a sum of squared difference (SSD), a mean absolute difference (MAD), a mean squared difference (MSD), and so on. The motion estimation unit 222 can generally perform these calculations using a sample-by-sample difference between the current block and the reference block under consideration. The motion estimation unit 222 can identify the reference block with the lowest value resulting from these calculations that indicates the closest match to the current block.

[0189] Motion estimation unit 222 can form one or more motion vectors (MVs) that define a position of a reference block in a reference picture relative to a position of a current block in a current picture. Motion estimation unit 222 can then provide the motion vector(s) to motion compensation unit 224. For example, for single prediction, motion estimation unit 222 can provide a single motion vector, while for bi-prediction, motion estimation unit 222 can provide two motion vectors. Motion compensation unit 224 can then use the motion vector(s) to generate a predicted block. For example, motion compensation unit 224 can use the motion vector(s) to retrieve data for a reference block. As another example, if the motion vector(s) have fractional sample precision, motion compensation unit 224 can interpolate values for the predicted block according to one or more interpolation filters. Moreover, for bi-prediction, motion compensation unit 224 can retrieve data for two reference blocks identified by the respective motion vectors and combine the retrieved data, e.g., by sample-wise averaging or weighted averaging.

[0190] When operating according to the AV1 video coding format, motion estimation unit 222 and motion compensation unit 224 can be configured to use translational motion compensation, affine motion compensation, overlapped block motion compensation (OBMC), and / or compound intra-inter prediction to encode coding blocks (e.g., both luma and chroma coding blocks) of video data.

[0191] As another example, for intra prediction or intra prediction coding, intra prediction unit 226 can generate a predicted block from samples neighboring the current block. For example, for directional modes, intra prediction unit 226 can mathematically combine values of the neighboring samples and fill these computed values across a defined direction of the current block to produce the predicted block. As another example, for DC mode, intra prediction unit 226 can compute an average of the neighboring samples of the current block and generate the predicted block to include this resulting average for each sample of the predicted block.

[0192] When operating according to the AV1 video coding format, intra prediction unit 226 can be configured to use directional intra prediction, non-directional intra prediction, recursive filter intra prediction, chroma from luma (CFL) prediction, intra block copy (IBC), and / or palette mode to encode coding blocks (e.g., both luma and chroma coding blocks) of video data. Mode selection unit 202 can include additional functional units to perform video prediction according to other prediction modes.

[0193] The mode select unit 202 provides the prediction block to a residual generation unit 204. The residual generation unit 204 receives the original, uncoded version of the current block from the video data memory 230 and the prediction block from the mode select unit 202. The residual generation unit 204 computes the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block for the current block. In some examples, the residual generation unit 204 can also determine the difference between sample values in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). In some examples, the residual generation unit 204 can use one or more subtractor circuits that perform binary subtraction to form.

[0194] In examples where the mode select unit 202 partitions the CU into PUs, each PU can be associated with a luma prediction unit and corresponding chroma prediction units. Video encoder 200 and video decoder 300 can support PUs having various sizes. As indicated above, the size of a CU can refer to the size of the CU's luma coding block, while the size of a PU can refer to the size of the PU's luma prediction unit. Assuming that a particular CU has a size of 2Nx2N, video encoder 200 can support PU sizes of 2Nx2N or NxN for intra prediction, and 2Nx2N, 2NxN, Nx2N, NxN, or similar symmetric PU sizes for inter prediction. Video encoder 200 and video decoder 300 can also support asymmetric partitioning for PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter prediction.

[0195] In examples where the mode select unit 202 does not further partition the CU into PUs, each CU can be associated with a luma coding block and corresponding chroma coding blocks. As described above, the size of a CU can refer to the size of the CU's luma coding block. Video encoder 200 and video decoder 300 can support CU sizes of 2Nx2N, 2NxN, or Nx2N.

[0196] For other video coding techniques, such as intra block copy mode coding, affine mode coding, and linear model (LM) mode coding, as some examples, the mode select unit 202 generates a prediction block for the current block being coded via the respective unit associated with the coding technique. In some examples, such as palette mode coding, the mode select unit 202 can not generate a prediction block, but rather generate syntax elements that indicate a way to reconstruct the block based on a selected palette. In such modes, the mode select unit 202 can provide the syntax elements to the entropy encoding unit 220 for encoding.

[0197] As described above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. The residual generation unit 204 then generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates a sample-by-sample difference between the prediction block and the current block.

[0198] The transform processing unit 206 applies one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a “transform coefficient block”). The transform processing unit 206 can apply various transforms to the residual block to form a transform coefficient block. For example, the transform processing unit 206 can apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform to the residual block. In some examples, the transform processing unit 206 can perform multiple transforms on the residual block, e.g., a primary transform and a secondary transform such as a rotation transform. In some examples, the transform processing unit 206 does not apply a transform to the residual block.

[0199] When operating according to AV1, the transform processing unit 206 can apply one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a “transform coefficient block”). The transform processing unit 206 can apply various transforms to the residual block to form a transform coefficient block. For example, the transform processing unit 206 can apply a horizontal / vertical transform combination, which can include a discrete cosine transform (DCT), an asymmetric discrete sine transform (ADST), a flipped ADST (e.g., ADST in reverse order), and an identity transform (IDTX). When the identity transform is used, the transform is skipped in one of the vertical or horizontal directions. In some examples, the transform processing can be skipped.

[0200] The quantization unit 208 can quantize the transform coefficients in the transform coefficient block to produce a quantized transform coefficient block. The quantization unit 208 can quantize the transform coefficients of the transform coefficient block according to a quantization parameter (QP) value associated with the current block. The video encoder 200 (e.g., via the mode selection unit 202) can adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization can introduce loss of information, thus quantized transform coefficients can have lower precision than the original transform coefficients produced by the transform processing unit 206.

[0201] Inverse quantization unit 210 and inverse transform processing unit 212 can apply inverse quantization and inverse transform, respectively, to the quantized transform coefficient block to reconstruct a residual block from the transform coefficient block. Reconstruction unit 214 can produce a reconstructed block corresponding to the current block (albeit potentially with some degree of distortion) based on the reconstructed residual block and the prediction block generated by mode selection unit 202. For example, reconstruction unit 214 can add samples of the reconstructed residual block to corresponding samples from the prediction block generated by mode selection unit 202 to generate the reconstructed block.

[0202] Filter unit 216 can perform one or more filter operations on the reconstructed block, including the CNN-based filtering techniques described herein. For example, filter unit 216 can also perform a deblocking operation to reduce blocking artifacts along the edges of the CU. In some examples, the operations of filter unit 216 can be skipped.

[0203] When operating according to AV1, filter unit 216 can perform one or more filter operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce blocking artifacts along the edges of the CU. In other examples, filter unit 216 can apply a constrained directional enhancement filter (CDEF), which can be applied after deblocking and can include applying a non-separable, nonlinear low-pass directional filter based on an estimated edge direction. Filter unit 216 can also include a loop restoration filter applied after CDEF and can include a separable, symmetric, normalized Wiener filter or a bi-directional self-guided filter.

[0204] Video encoder 200 stores the reconstructed block in DPB 218. For example, in examples in which the operations of filter unit 216 are not performed, reconstruction unit 214 can store the reconstructed block to DPB 218. In examples in which the operations of filter unit 216 are performed, filter unit 216 can store the filtered reconstructed block to DPB 218. Motion estimation unit 222 and motion compensation unit 224 can retrieve reference pictures formed from reconstructed (and potentially filtered) blocks from DPB 218 to inter-predict blocks of subsequent encoded pictures. Further, intra-prediction unit 226 can use reconstructed blocks of the current picture in DPB 218 to intra-predict other blocks in the current picture.

[0205] In general, entropy encoding unit 220 can entropy encode syntax elements received from other functional components of video encoder 200. For example, entropy encoding unit 220 can entropy encode quantized transform coefficient blocks from quantization unit 208. As another example, entropy encoding unit 220 can entropy encode prediction syntax elements (e.g., motion information for inter-prediction or intra-mode information for intra-prediction) from mode select unit 202. Entropy encoding unit 220 can perform one or more entropy encoding operations on syntax elements, as another example of video data, to generate encoded data. For example, entropy encoding unit 220 can perform a context- adaptive variable length coding (CAVLC) operation, a CABAC operation, a variable to variable (V2V) length coding operation, a syntax-based context-adaptive binary arithmetic coding (SBAC) operation, a Probability Interval Partitioning Entropy (PIPE) coding operation, an Exponential-Golomb coding operation, or another type of entropy coding operation. In some examples, entropy encoding unit 220 can operate in a bypass mode in which no entropy coding is performed on the syntax elements.

[0206] Video encoder 200 can output a bitstream that includes the entropy encoded syntax elements needed to reconstruct blocks of a slice or picture. In particular, entropy encoding unit 220 can output the bitstream.

[0207] According to AV1, entropy encoding unit 220 can be configured as a symbol-to-symbol adaptive multi-symbol arithmetic coder. Syntax elements in AV1 include an alphabet of N elements, and a context (e.g., a probability model) includes a set of N probabilities. Entropy encoding unit 220 can store the probabilities as n-bit (e.g., 15-bit) cumulative distribution functions (CDFs). Entropy encoding unit 220 can perform recursive scaling with an update factor based on the alphabet size to update the context.

[0208] The operations described above are described with respect to blocks. Such description should be understood to be operations for luma coding blocks and / or chroma coding blocks. As described above, in some examples, the luma coding blocks and the chroma coding blocks are luma components and chroma components of a CU. In some examples, the luma coding blocks and the chroma coding blocks are luma components and chroma components of a PU.

[0209] In some examples, operations performed with respect to luma coding blocks need not be repeated for chroma coding blocks. As one example, operations to identify a motion vector (MV) and a reference picture for a luma coding block need not be repeated for identifying an MV and a reference picture for a chroma block. Rather, the MV for the luma coding block can be scaled to determine the MV for the chroma block, and the reference picture can be the same. As another example, the intra-prediction process can be the same for luma coding blocks and chroma coding blocks.

[0210] Video encoder 200 represents an example of a device configured to encode video data, the device including a memory configured to store the video data, and one or more processing units implemented in circuitry and configured to perform the following operations: determining a prediction block for the current block; determining a residual block for the current block; determining a reconstruction block for the current block based on the prediction block and the residual block; and applying a neural network-based filter to the reconstruction block to determine a filtering block. The neural network-based filter may include one or more convolutions and / or two or more separable convolutions on the reconstruction block.

[0211] Figure 23 This is a block diagram illustrating an example video decoder 300 capable of performing the techniques described herein. Figure 23 This disclosure is provided for illustrative purposes and is not intended to limit the techniques as extensively illustrated and described herein. For illustrative purposes, this disclosure describes a video decoder 300 based on VVC and HEVC technologies. However, the techniques of this disclosure can be implemented by video decoding devices configured for other video decoding standards.

[0212] exist Figure 23 In the example, the video decoder 300 includes a decoded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, and a DPB 314. Any or all of the CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 can be implemented in one or more processors or in processing circuitry. For example, the units of the video decoder 300 can be implemented as one or more circuit or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Furthermore, the video decoder 300 may include additional or alternative processors or processing circuitry to perform these and other functions.

[0213] The prediction processing unit 304 includes a motion compensation unit 316 and an intra-frame prediction unit 318. The prediction processing unit 304 may include additional units that perform predictions based on other prediction modes. As an example, the prediction processing unit 304 may include a palette unit, an intra-block copy unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, the video decoder 300 may include more, fewer, or different functional components.

[0214] When operating according to AV1, the motion compensation unit 316 can be configured to use translational motion compensation, affine motion compensation, OBMC, and / or compound inter-intra prediction to decode coding blocks (e.g., both luma and chroma coding blocks) of video data, as described above. The intra prediction unit 318 can be configured to use directional intra prediction, non-directional intra prediction, recursive filter intra prediction, CFL, IBC, and / or palette mode to decode coding blocks (e.g., both luma and chroma coding blocks) of video data, as described above.

[0215] The CPB memory 320 can store video data, such as an encoded video bitstream, to be decoded by the components of the video decoder 300. The video data stored in the CPB memory 320 can be obtained, for example, from the computer- readable medium 110 Figure 1 ). The CPB memory 320 can include a CPB that stores encoded video data (e.g., syntax elements) from an encoded video bitstream. In addition, the CPB memory 320 can store video data other than syntax elements of coded pictures, such as temporary data representing outputs from the various units of the video decoder 300. The DPB 314 generally stores decoded pictures, which the video decoder 300 can output and / or use as reference video data when decoding subsequent data or pictures of the encoded video bitstream. The CPB memory 320 and the DPB 314 can be formed by any of a variety of memory devices, such as DRAM, including SDRAM, MRAM, RRAM, or other types of memory devices. The CPB memory 320 and the DPB 314 can be provided by the same memory device or separate memory devices. In various examples, the CPB memory 320 can be on-chip with other components of the video decoder 300, or off-chip relative to such components.

[0216] Additionally or alternatively, in some examples, the video decoder 300 can retrieve coded video data from the memory 120 Figure 1 ). That is, the memory 120 can store data as discussed above with respect to the CPB memory 320. Likewise, when some or all of the functionality of the video decoder 300 is implemented in software to be executed by processing circuitry of the video decoder 300, the memory 120 can store the instructions to be executed by the video decoder 300.

[0217] Figure 23 The various units shown in FIG. 3B are shown to assist with understanding the operations performed by the video decoder 300. The units can be implemented as fixed- function circuits, programmable circuits, or a combination thereof. Similar to the video encoder 200, the video decoder 300 can include additional details shown or not shown. Figure 22Fixed function circuits refer to circuits that provide specific functionality, and are pre- configured on the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks, and provide flexible functionality in the operations that can be performed. For instance, programmable circuits can execute software or firmware that cause the programmable circuits to operate in a manner defined by the instructions of the software or firmware. Fixed function circuits can execute software instructions (e.g., to receive parameters or output parameters), but the types of operations that the fixed function circuits perform are generally immutable. In some examples, one or more of the units can be distinct circuit blocks (fixed function or programmable), and in some examples one or more of the units can be integrated circuits.

[0218] Video decoder 300 can include ALUs, EFUs, digital circuits, analog circuits, and / or programmable cores formed from programmable circuits. In examples where the operations of video decoder 300 are performed by software executing on the programmable circuits, on-chip or off-chip memory can store instructions for the software that video decoder 300 receives and executes.

[0219] Entropy decoding unit 302 can receive encoded video data from a CPB, and entropy decode the video data to reproduce syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filter unit 312 can generate decoded video data based on the syntax elements extracted from the bitstream.

[0220] In general, video decoder 300 reconstructs a picture on a block-by-block basis. Video decoder 300 can perform reconstruction operations individually for each block (where the block that is currently being reconstructed (i.e., decoded) can be referred to as the “current block”).

[0221] Entropy decoding unit 302 can entropy decode syntax elements defining quantized transform coefficient blocks, as well as transform information such as quantization parameters (QPs) and / or transform mode indications. Inverse quantization unit 306 can use a QP associated with a quantized transform coefficient block to determine a degree of quantization, and likewise a degree of inverse quantization for inverse quantization unit 306 to apply. For instance, inverse quantization unit 306 can perform a bit- shift operation to de-quantize a quantized transform coefficient. Inverse quantization unit 306 can thereby form a transform coefficient block including transform coefficients.

[0222] After inverse quantization unit 306 forms a transform coefficient block, inverse transform processing unit 308 can apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, inverse transform processing unit 308 can apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve Transform (KLT), an inverse rotational transform, an inverse directional transform, or another inverse transform to the transform coefficient block.

[0223] Furthermore, prediction processing unit 304 generates the prediction block from the prediction information syntax elements entropy decoded by entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is inter-predicted, motion compensation unit 316 can generate the prediction block. In this case, the prediction information syntax elements can indicate a reference picture from which the reference block is retrieved from the DPB 314, and a motion vector that identifies the position of the reference block in the reference picture relative to the position of the current block in the current picture. Motion compensation unit 316 can generally perform the inter-prediction process in a manner substantially similar to the manner described with respect to motion compensation unit 224 Figure 22 ) described above.

[0224] As another example, if the prediction information syntax elements indicate that the current block is intra-predicted, intra-prediction unit 318 can generate the prediction block according to an intra-prediction mode indicated by the prediction information syntax elements. Again, intra-prediction unit 318 can generally perform the intra-prediction process in a manner substantially similar to the manner described with respect to intra-prediction module 226 Figure 22 ) described above. Intra-prediction unit 318 can retrieve data for neighboring samples of the current block from the DPB 314.

[0225] Reconstruction unit 310 can reconstruct the current block using the prediction block and the residual block. For example, reconstruction unit 310 can add the samples of the residual block to corresponding samples of the prediction block to reconstruct the current block.

[0226] Filter unit 312 can perform one or more filter operations on the reconstructed block, including the CNN-based filtering techniques described herein. For example, filter unit 312 can also perform a deblocking operation to reduce blocking artifact pseudo-ghosts along the edges of the reconstructed block. The operations of filter unit 312 need not be performed in all examples.

[0227] Video decoder 300 can store the reconstructed block in DPB 314. For example, in examples where operations of filter unit 312 are not performed, reconstructed unit 310 can store the reconstructed block to DPB 314. In examples where operations of filter unit 312 are performed, filter unit 312 can store the filtered reconstructed block to DPB 314. As described above, DPB 314 can provide reference information, such as samples of a current picture for intra prediction and previously decoded pictures for subsequent motion compensation, to prediction processing unit 304. Moreover, video decoder 300 can output decoded pictures (e.g., decoded video) from DPB 314 for subsequent presentation on a display device, such as display device 118 of FIG. 1. Figure 1

[0228] In this way, video decoder 300 represents an example of a video decoding device including a memory configured to store video data, and one or more processing units implemented in circuitry and configured to determine a prediction block for a current block, determine a residual block for the current block, determine a reconstructed block for the current block based on the prediction block and the residual block, and apply a NN-based filter to the reconstructed block to determine a filtered block. The NN-based filter can include one or more convolutions and / or two or more separable convolutions on the reconstructed block.

[0229] Figure 24 is a flowchart illustrating an example process for encoding a current block in accordance with the techniques of this disclosure. The current block can be or can include a current CU. Although described with respect to video encoder 200 Figure 1 and Figure 22 ), it should be understood that other devices can be configured to perform similar processes as the processes of Figure 24 .

[0230] In this example, video encoder 200 initially predicts a current block (350). For example, video encoder 200 can form a prediction block for the current block. Video encoder 200 can then calculate a residual block for the current block (352). To calculate the residual block, video encoder 200 can calculate a difference between an original, unencoded block and the prediction block for the current block. Video encoder 200 can then transform the residual block and quantize transform coefficients of the residual block (354). Next, video encoder 200 can scan the quantized transform coefficients of the residual block (356). During or after the scan, video encoder 200 can entropy encode the transform coefficients (358). For example, video encoder 200 can encode the transform coefficients using CAVLC or CABAC. Video encoder 200 can then output the entropy encoded data of the block (360). ​

[0231] Figure 25 FIG. 7 is a flowchart illustrating an example process for decoding a current block of video data, in accordance with the techniques of this disclosure. The current block can be or include a current CU. Although described with respect to video decoder 300 Figure 1 and Figure 23 ), it should be understood that other devices can be configured to perform similar processes to those of Figure 25 .

[0232] Video decoder 300 can receive entropy encoded data for the current block, such as entropy encoded prediction information and entropy encoded data for transform coefficients of a residual block corresponding to the current block (370). Video decoder 300 can entropy decode the entropy encoded data to determine prediction information for the current block and reproduce transform coefficients of a residual block (372). Video decoder 300 can predict the current block, e.g., using an intra- or inter-prediction mode as indicated by the prediction information for the current block, to calculate a prediction block for the current block (374). Video decoder 300 can then inverse scan the reproduced transform coefficients to create a block of quantized transform coefficients (376). Video decoder 300 can then inverse quantize the transform coefficients and apply an inverse transform to the transform coefficients to produce a residual block (378). Video decoder 300 can finally decode the current block by combining the prediction block and the residual block (380).

[0233] Figure 26 FIG. 7 is a flowchart illustrating an example process for decoding a current block of video data, in accordance with the techniques of this disclosure. The current block can be or include a current CU. Although described with respect to video decoder 300 Figure 1 and Figure 23 ), it should be understood that other devices can be configured to perform similar processes to those of Figure 26 . For example, the decoding loop of video encoder 200 can also perform the processes of Figure 26 .

[0234] In the example of Figure 26 , video decoder 300 receives a picture of video data (400). Video decoder 300 reconstructs a block of the picture of video data to generate a reconstructed block (402). Video decoder 300 performs a NN-based filter process on the reconstructed block to generate a filtered block, where the NN-based filter process includes performing a plurality of separable convolutions in parallel with point-wise input convolution (404).

[0235] The following numbered clauses illustrate one or more aspects of the devices and techniques described in this disclosure.

[0236] Clause 1A: A method of decoding video data, the method comprising: determining a prediction block for a current block; determining a residual block for the current block; determining a reconstructed block for the current block based on the prediction block and the residual block; applying a neural network (NN)-based filter to the reconstructed block to determine a filtered block.

[0237] Clause 2A: The method of clause 1A, wherein applying the NN-based filter to the reconstructed block comprises performing one or more convolutions on the reconstructed block.

[0238] Clause 3A: The method of clause 2A, wherein the one or more convolutions comprise two or more separable convolutions.

[0239] Clause 4A: The method of any of clauses 1A-3A, wherein the method of decoding is performed as part of a video encoding process.

[0240] Clause 5A: A device for coding video data, the device comprising one or more units for performing the method of any of clauses 1A-4A.

[0241] Clause 6A: The device of clause 5A, wherein the one or more units comprise one or more processors implemented in circuitry.

[0242] Clause 7A: The device of any of clauses 5A and 6A, further comprising a memory that stores the video data. Clause 8A: The device of any of clauses 5A-7A, further comprising: a display configured to display decoded video data.

[0243] Clause 9A: The device of any of clauses 5A-8A, wherein the device comprises one or more of: a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0244] Clause 10A: The device of any of clauses 5A-9A, wherein the device comprises a video decoder.

[0245] Clause 11A: The device of any of clauses 5A-10A, wherein the device comprises a video encoder.

[0246] Clause 12A: A computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to perform the method of any of clauses 1A-4A.

[0247] Clause 1B: A method of decoding video data, the method comprising: receiving a picture of video data; reconstructing a block of the picture of video data to generate a reconstructed block; and performing a neural network (NN)-based filtering process on the reconstructed block to generate a filtered block, wherein the NN-based filtering process comprises a pointwise input convolution in parallel with a plurality of separable convolutions.

[0248] Clause 2B: The method of clause 1B, wherein the pointwise input convolution comprises a 1x1 convolution.

[0249] Clause 3B: The method of clause 1B or 2B, wherein the plurality of separable convolutions comprises a 3x1 separable convolution and a 1x3 separable convolution.

[0250] Clause 4B: The method of any of clauses 1B-3B, wherein performing the NN-based filtering process on the reconstructed block to generate the filtered block comprises: performing a first parametric rectified linear unit (PReLU) layer on an output of the plurality of separable convolutions; performing a second PReLU layer on an output of the pointwise input convolution; and adding an output of the first PReLU to an output of the second PReLU to produce a combined PReLU output.

[0251] Clause 5B: The method of clause 4B, wherein performing the NN-based filtering process on the reconstructed block to generate the filtered block comprises: performing a second pointwise input convolution on the combined PReLU output.

[0252] Clause 6B: The method of clause 5B, wherein performing the NN-based filtering process on the reconstructed block to generate the filtered block comprises: performing a second plurality of separable convolutions on an output of the second pointwise input convolution.

[0253] Clause 7B: The method of clause 6B, wherein the second plurality of separable convolutions comprises a 3x1 separable convolution and a 1x3 separable convolution.

[0254] Clause 8B: The method of any of clauses 1B-7B, wherein the method of decoding is performed as part of a video encoding process.

[0255] Clause 9B: A device for decoding video data, the device comprising: a memory configured to store video data; one or more processors implemented in circuitry and configured to: receive a picture of video data; reconstruct a block of the picture of video data to generate a reconstructed block; and perform a neural network (NN)-based filtering process on the reconstructed block to generate a filtered block, wherein the NN-based filtering process comprises a pointwise input convolution in parallel with a plurality of separable convolutions.

[0256] Clause 10B: The device of clause 9B, wherein the point-wise input convolution comprises a 1x1 convolution.

[0257] Clause 11B: The device of clause 9B or 10B, wherein the multiple separable convolutions comprise a 3x1 separable convolution and a 1x3 separable convolution.

[0258] Clause 12B: The device of any of clauses 9B-11B, wherein to perform the NN-based filtering process on the reconstructed block to generate the filtered block, the one or more processors are further configured to: perform a first parametric rectified linear unit (PReLU) layer on outputs of the multiple separable convolutions; perform a second PReLU layer on an output of the point-wise input convolution; and add the output of the first PReLU to the output of the second PReLU to produce a combined PReLU output.

[0259] Clause 13B: The device of clause 12B, wherein to perform the NN-based filtering process on the reconstructed block to generate the filtered block, the one or more processors are further configured to: perform a second point-wise input convolution on the combined PReLU output.

[0260] Clause 14B: The device of clause 13B, wherein to perform the NN-based filtering process on the reconstructed block to generate the filtered block, the one or more processors are further configured to: perform a second multiple separable convolutions on an output of the second point-wise input convolution.

[0261] Clause 15B: The device of clause 14B, wherein the second multiple separable convolutions comprise a 3x1 separable convolution and a 1x3 separable convolution.

[0262] Clause 16B: The method of clause 15B, wherein the NN-based filtering process further comprises performing a group convolution.

[0263] Clause 17B: The device of any of clauses 9B-16B, wherein the device comprises one or more of: a camera, a computer, a mobile device, a broadcast receiver device, a display device, or a set-top box.

[0264] Clause 18B: The device of any of clauses 9B-17B, wherein the device comprises a video decoder.

[0265] Clause 19B: The device of any of clauses 9B-17B, wherein the device comprises a video encoder.

[0266] Clause 20B: A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receive a picture of video data; reconstruct a block of the picture of video data to generate a reconstructed block; and perform a neural network (NN) based filtering process on the reconstructed block to generate a filtered block, wherein the NN based filtering process includes performing a plurality of separable convolutions in parallel with point-wise input convolutions.

[0267] It is recognized that, in accordance with examples, a particular sequence of actions or events can be performed in a different order, added, merged, or omitted entirely (e.g., not all described actions or events are necessary for the practice of the described techniques). Moreover, in a particular example, actions or events can be performed concurrently (e.g., through multi-threading processing, interrupt processing, or multiple processors), rather than sequentially.

[0268] In one or more examples, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer- readable media generally can correspond to (1) tangible computer- readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this disclosure. A computer program product can include a computer-readable medium.

[0269] By way of example, and not limitation, such computer-readable storage media can comprise one or more of the following: RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any

[0270] Instructions can be executed by one or more processors, such as one or more DSPs, general purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor," as used herein can refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0271] The techniques of this disclosure can be implemented in a variety of devices or apparatuses including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described herein to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units can be combined in a codec hardware unit or provided by a collection of interoperating hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0272] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method of decoding video data, the method comprising: receiving a picture of video data; reconstructing a block of the picture of video data to generate a reconstructed block; and performing a neural network (NN)-based filtering process on the reconstructed block to generate a filtered block, wherein the NN-based filtering process includes performing a plurality of separable convolutions in parallel with a point-wise input convolution. The point-wise input convolution includes a 1x1 convolution.

2. The method of claim 1, wherein, The plurality of separable convolutions includes a 3x1 separable convolution and a 1x3 separable convolution.

3. The method of claim 1, wherein, Performing the NN-based filtering process on the reconstructed block to generate the filtered block includes:

4. The method of claim 1, wherein, performing a first parametric rectified linear unit (PReLU) layer on outputs of the plurality of separable convolutions; performing a second PReLU layer on an output of the point-wise input convolution; and adding outputs of the first PReLU and the second PReLU to produce a combined PReLU output. Performing the NN-based filtering process on the reconstructed block to generate the filtered block includes:

5. The method of claim 4, wherein, performing a second point-wise input convolution on the combined PReLU output. Performing the NN-based filtering process on the reconstructed block to generate the filtered block includes:

6. The method of claim 5, wherein, performing a second plurality of separable convolutions on an output of the second point-wise input convolution. The second plurality of separable convolutions includes a 3x1 separable convolution and a 1x3 separable convolution.

7. The method of claim 6, wherein, The method of decoding is performed as part of a video encoding process.

8. The method of claim 1, wherein, 9. A device for decoding video data, the device comprising: a memory configured to store video data; one or more processors implemented in circuitry and configured to: receive a picture of video data; reconstruct a block of the picture of video data to generate a reconstructed block; and perform a neural network (NN)-based filtering process on the reconstructed block to generate a filtered block, wherein the NN-based filtering process includes performing a plurality of separable convolutions in parallel with a point-wise input convolution. The point-wise input convolution includes a 1x1 convolution.

10. The apparatus of claim 9, wherein, The plurality of separable convolutions includes a 3x1 separable convolution and a 1x3 separable convolution.

11. The apparatus of claim 9, wherein, To perform the NN-based filtering process on the reconstructed block to generate the filtered block, the one or more processors are further configured to:

12. The apparatus of claim 9, wherein, perform a first parametric rectified linear unit (PReLU) layer on outputs of the plurality of separable convolutions; perform a second PReLU layer on an output of the point-wise input convolution; and add outputs of the first PReLU and the second PReLU to produce a combined PReLU output. To perform the NN-based filtering process on the reconstructed block to generate the filtered block, the one or more processors are further configured to: perform a second point-wise input convolution on the combined PReLU output.

13. The apparatus of claim 12, wherein, To perform the NN-based filtering process on the reconstructed block to generate the filtered block, the one or more processors are further configured to: perform a second plurality of separable convolutions on an output of the second point-wise input convolution.

14. The apparatus of claim 13, wherein, The second plurality of separable convolutions includes a 3x1 separable convolution and a 1x3 separable convolution. ​ 15. The apparatus of claim 14, wherein, ​ 16. The apparatus of claim 15, wherein, The NN-based filtering process further includes performing group convolutions.

17. The apparatus of claim 9, wherein, The device includes one or more of a camera, a computer, a mobile device, a broadcast receiver device, a display device, or a set-top box.

18. The apparatus of claim 9, wherein, The device includes a video decoder.

19. The apparatus of claim 9, wherein, The device includes a video encoder.

20. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform: receiving a picture of video data; reconstructing a block of the picture of video data to generate a reconstructed block; and performing a neural network (NN) based filtering process on the reconstructed block to generate a filtered block, wherein, The NN-based filtering process includes performing a plurality of separable convolutions in parallel with point-wise input convolutions.

Images