Method and apparatus for filtering processes for video coding

Abstract

An example apparatus for filtering decoded video data includes: a memory configured to store video data; and one or more processors implemented in circuitry and configured to: decode blocks of video data to form decoded blocks; apply a filter to the decoded blocks to form filtered blocks; multiply samples of the filtered blocks by a scaling factor to form thinned filtered blocks; and combine samples of the thinned filtered blocks with corresponding samples of the decoded blocks. The one or more processors may also encode the blocks prior to decoding them. The one or more processors may encode or decode, for example, the value of a syntax element representing a scaling factor in a picture header including the block.

Description

[0001] This application is a divisional application of patent application No. 202180063665.5, filed on September 24, 2021, entitled "Method and Apparatus for Filtering Process in Video Decoding". Technical Field

[0002] This disclosure relates to video encoding and video decoding. Background Technology

[0003] Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital live broadcast systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite wireless phones, so-called "smartphones," video conferencing equipment, video streaming devices, and more. Digital video devices implement video decoding technologies (such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 (Part 10, Advanced Video Decoding (AVC)), ITU-T H.265 / High Efficiency Video Decoding (HEVC), and extensions to such standards). By implementing such video decoding technologies, video devices can more efficiently send, receive, encode, decode, and / or store digital video information.

[0004] Video decoding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video decoding, video slices (e.g., video pictures or portions of video pictures) can be segmented into video blocks, which may also be referred to as decoding tree units (CTUs), decoding units (CUs), and / or decoding nodes. Video blocks in a slice of a picture that has been intra-decoded (I) are encoded using spatial prediction relative to reference samples in neighboring blocks within the same picture. Video blocks in a slice of a picture that has been inter-decoded (P or B) can use spatial prediction relative to reference samples in neighboring blocks within the same picture or temporal prediction relative to reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame. Summary of the Invention

[0005] In summary, this disclosure describes techniques for filtering video data. Specifically, it describes techniques applied to filtering processes for distorted images. The filtering processes described in this disclosure can be based on neural network techniques and can be used in the context of advanced video codecs, such as extensions to the Multi-Functional Video Decoding (VVC) standard, next-generation video decoding standards, and / or any other video codecs.

[0006] In one example, a method for filtering decoded video data includes: decoding blocks of video data to form decoded blocks; applying a filter to the decoded blocks to form filtered blocks; multiplying samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combining samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0007] In another example, an apparatus for filtering decoded video data includes: a memory configured to store the video data; and one or more processors implemented in a circuit and configured to: decode blocks of the video data to form decoded blocks; apply a filter to the decoded blocks to form filtered blocks; multiply samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combine samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0008] In another example, a computer-readable storage medium has instructions stored thereon that, when executed, cause a processor to: decode blocks of video data to form decoded blocks; apply a filter to the decoded blocks to form filtered blocks; multiply samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combine samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0009] In another example, an apparatus for filtering decoded video data includes: a unit for decoding blocks of video data to form decoded blocks; a unit for applying a filter to the decoded blocks to form filtered blocks; a unit for multiplying samples of the filtered blocks by a scaling factor to form refined filtered blocks; and a unit for combining samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0010] Details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0011] Figure 1 This is a block diagram illustrating an example video encoding and decoding system that can perform the techniques described in this disclosure.

[0012] Figure 2 This is a block diagram illustrating an example hybrid video decoding framework.

[0013] Figure 3 This is a conceptual diagram showing a hierarchical prediction structure with a group of pictures (GOP) size of 16.

[0014] Figure 4 This is a block diagram illustrating an example filter based on a convolutional neural network with four layers.

[0015] Figure 5A and Figure 5B This is a conceptual diagram showing an example quadtree binary tree (QTBT) structure and its corresponding decoding tree unit (CTU).

[0016] Figure 6 This is a block diagram illustrating an example video encoder that can perform the techniques described in this disclosure.

[0017] Figure 7 This is a block diagram illustrating an example video decoder that can perform the techniques described in this disclosure.

[0018] Figure 8 This is a flowchart illustrating an example method for encoding the current block according to the technology of this disclosure.

[0019] Figure 9 This is a flowchart illustrating an example method for decoding the current block according to the technology of this disclosure.

[0020] Figure 10 This is a flowchart illustrating an example method for encoding video data and filtering decoded video data according to the technology of this disclosure.

[0021] Figure 11 This is a flowchart illustrating an example method for filtering decoded video data according to the technology of this disclosure. Detailed Implementation

[0022] In summary, this disclosure describes techniques involving filtering decoded video data. Typically, a video decoder can decode blocks of video data, for example, by using inter-frame / intra-frame prediction to generate predictive blocks, and then combine the predictive blocks with decoded residual blocks. A video encoder can encode blocks of video data and then decode the blocks in a manner substantially similar to that of a decoder, such that any distortion introduced via the encoding process during predictive coding is also replicated at the video encoder.

[0023] Furthermore, video encoders and decoders can be configured to filter decoded blocks of video data. For example, video encoders and decoders can perform "in-loop" filtering in the same manner, such that the filtered decoded blocks of video data are used as a reference during subsequent predictive decoding (encoding or decoding). Such filters can include, for example, adaptive loop filters (ALF), neural network (NN) based filters, etc.

[0024] Predefined filters are typically built or trained on large amounts of video and image data in a database. While this construction process may result in filters optimized for general video data, there may be specific situations where the filter output can be improved for specific video data. This disclosure describes techniques that can be used to improve filtering of decoded video data.

[0025] In summary, the techniques of this disclosure include: decoding video data, applying filters to the decoded video data, and then applying a scaling factor to the filtered, decoded video data. The scaling factor can improve the filtering result. For example, a video encoder can determine the scaling factor, for example, using a rate-distortion optimization (RDO) process. The video encoder can determine the scaling factor for slices of video data, pictures of video data, sequences of pictures of video data, at the block level (e.g., decode tree unit (CTU)). Therefore, the video encoder can encode data representing the scaling factor. The video encoder can also encode data indicating whether filtering refinement is performed, according to the techniques of this disclosure. The video encoder can then apply the scaling factor to the filtered, decoded video data and store the resulting refined, filtered, decoded video data in a decoded picture buffer (DPB).

[0026] Similarly, the video decoder can receive data indicating whether to perform filtering refinement. When the data instructs the video decoder to perform filtering refinement, the video decoder can decode data representing the scaling factor (e.g., in the slice header, Adaptive Parameter Set (APS), Picture header, Picture Parameter Set (PPS), Block header (e.g., CTU header), etc.). The video decoder can then apply the scaling factor to the filtered and decoded video data and store the resulting refined filtered and decoded video data in the DPB. The video decoder can also output the refined filtered and decoded video data from the DPB as output decoded video data for display purposes.

[0027] In this way, scaling factors can be used to improve the filtered data. Therefore, the refined and filtered data can more accurately reflect the original video data initially received by the video encoder. Consequently, the refined and filtered data can improve the prediction of subsequent video data, thereby reducing the bitrate associated with the encoded video bitstream. Furthermore, the refined and filtered data can have reduced distortion, thus providing a more visually pleasing experience for users viewing the displayed video data.

[0028] Figure 1 This is a block diagram illustrating an example video encoding and decoding system 100 capable of performing the techniques of this disclosure. In general, the techniques of this disclosure relate to coding (encoding and / or decoding) video data. Typically, video data includes any data used for processing video. Therefore, video data can include raw, unencoded video, encoded video, decoded (e.g., reconstructed) video, and video metadata (e.g., signaling data).

[0029] like Figure 1 As shown, in this example, system 100 includes source device 102, which provides encoded video data to be decoded and displayed by destination device 116. Specifically, source device 102 provides the video data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 can include any device in a wide range of devices, including desktop computers, notebook computers (i.e., laptop computers), mobile devices, tablet computers, set-top boxes, mobile phones (such as smartphones), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, etc. In some cases, source device 102 and destination device 116 may be equipped for wireless communication and therefore may be referred to as wireless communication devices.

[0030] exist Figure 1In the example, source device 102 includes a video source 104, memory 106, video encoder 200, and output interface 108. Destination device 116 includes an input interface 122, video decoder 300, memory 120, and display device 118. According to this disclosure, the video encoder 200 of source device 102 and the video decoder 300 of destination device 116 can be configured to apply techniques for filtering video data. Therefore, source device 102 represents an example of a video encoding device, while destination device 116 represents an example of a video decoding device. In other examples, the source and destination devices may include other components or arrangements. For example, source device 102 may receive video data from an external video source such as an external camera. Similarly, destination device 116 may interface with an external display device, rather than including an integrated display device.

[0031] like Figure 1 The system 100 shown is merely an example. Typically, any digital video encoding and / or decoding device can perform techniques for filtering video data. Source device 102 and destination device 116 are merely examples of such decoding devices, where source device 102 generates encoded video data for transmission to destination device 116. In this disclosure, "decoding device" refers to a device that performs the decoding (e.g., encoding and / or decoding) of data. Thus, video encoder 200 and video decoder 300 represent examples of decoding devices (specifically, video encoder and video decoder). In some examples, source device 102 and destination device 116 may operate in a substantially symmetrical manner, such that each of source device 102 and destination device 116 includes video encoding and decoding components. Therefore, system 100 can support one-way or two-way video transmission between source device 102 and destination device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0032] Typically, video source 104 represents a source of video data (i.e., raw, unencoded video data) and a sequence of pictures (also referred to as "frames") that provide the video data in order to video encoder 200, which encodes the data used for the pictures. Video source 104 of source device 102 may include video capture devices such as cameras, video archive units containing previously captured raw video, and / or video feed interfaces for receiving video from video content providers. Alternatively, video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 may encode the captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange the pictures from their received order (sometimes referred to as "display order") to a decoding order for decoding. Video encoder 200 may generate a bitstream comprising the encoded video data. Then, the source device 102 can output the encoded video data to the computer-readable medium 110 via the output interface 108 so that it can be received and / or retrieved by, for example, the input interface 122 of the destination device 116.

[0033] The memory 106 of source device 102 and the memory 120 of destination device 116 represent general-purpose memory. In some examples, memories 106 and 120 may store raw video data, such as raw video from video source 104 and raw decoded video data from video decoder 300. Alternatively, memories 106 and 120 may store software instructions executable by, for example, video encoder 200 and video decoder 300. Although memories 106 and 120 are shown separately from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store, for example, encoded video data output from video encoder 200 and input to video decoder 300. In some examples, portions of memories 106 and 120 may be allocated as one or more video buffers, for example, to store raw decoded and / or encoded video data.

[0034] Computer-readable medium 110 can represent any type of medium or device capable of transmitting encoded video data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium that enables source device 102 to directly transmit encoded video data to destination device 116 in real time, for example, via a radio frequency network or a computer-based network. Output interface 108 can modulate the transmitted signal including encoded video data according to a communication standard such as a wireless communication protocol, and input interface 122 can demodulate the received transmitted information according to a communication standard such as a wireless communication protocol. The communication medium can include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium can include a router, switch, base station, or any other device that may be useful for facilitating communication from source device 102 to destination device 116.

[0035] In some examples, source device 102 can output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a wide variety of distributed or locally accessible data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0036] In some examples, source device 102 can output encoded video data to file server 114 or another intermediate storage device that can store the encoded video data generated by source device 102. Destination device 116 can access the stored video data from file server 114 via streaming or downloading.

[0037] File server 114 can be any type of server device capable of storing encoded video data and sending such encoded video data to destination device 116. File server 114 can represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or FLUTE-based file delivery protocol), a Content Delivery Network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or enhanced MBMS (eMBMS) server, and / or a Network Attached Storage (NAS) device. File server 114 can additionally or alternatively implement one or more HTTP streaming protocols, such as HTTP-based Dynamic Adaptive Streaming (DASH), HTTP Real-Time Streaming (HLS), Real-Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, etc.

[0038] Destination device 116 can access encoded video data from file server 114 via any standard data connection, including an internet connection. This may include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., Digital Subscriber Line (DSL), cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on file server 114. Input interface 122 can be configured to operate according to any one or more of the various protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data.

[0039] Output interface 108 and input interface 122 may represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 may be configured to transmit data (such as encoded video data) according to cellular communication standards (such as 4G, 4G-LTE (Long Term Evolution), improved LTE, 5G, etc.). In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 may be configured to transmit data (such as encoded video data) according to other wireless standards (such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee™), the Bluetooth™ standard, etc.). In some examples, source device 102 and / or destination device 116 may include corresponding system-on-chip (SoC) devices. For example, source device 102 may include a SoC device for performing functions assigned to video encoder 200 and / or output interface 108, and destination device 116 may include a SoC device for performing functions assigned to video decoder 300 and / or input interface 122.

[0040] The technology disclosed herein can be applied to video decoding to support any of a wide variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission (such as HTTP-based Dynamic Adaptive Streaming (DASH)), digital video encoded onto data storage media, decoding digital video stored on data storage media, or other applications.

[0041] The input interface 122 of the destination device 116 receives an encoded video bitstream from a computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200, such as syntax elements (which are also used by the video decoder 300), which have values ​​describing the characteristics and / or processing of video blocks or other decoding units (e.g., slices, pictures, picture groups, sequences, etc.). The display device 118 displays a decoded picture of the decoded video data to the user. The display device 118 may represent any of a wide variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0042] Despite Figure 1Not shown, but in some examples, the video encoder 200 and video decoder 300 may each be integrated with the audio encoder and / or audio decoder, and may include appropriate MUX-DEMUX units or other hardware and / or software to process multiplexed streams including both audio and video in a common data stream. Where applicable, the MUX-DEMUX unit may comply with the ITU H.223 multiplexer protocol or other protocols (such as User Datagram Protocol (UDP)).

[0043] The video encoder 200 and video decoder 300 can each be implemented as any of a wide variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is implemented in part in software, the device may store instructions for the software in a suitable non-transitory computer-readable medium, and may use one or more processors to execute the instructions in hardware to perform the contents of this disclosure. Each of the video encoder 200 and video decoder 300 may be included in one or more encoders or decoders, and either encoder or decoder may be integrated as part of a combined encoder / decoder (CODEC) in the respective device. Devices including the video encoder 200 and / or video decoder 300 may include integrated circuits, microprocessors, and / or wireless communication devices (such as cellular phones).

[0044] The video encoder 200 and video decoder 300 can operate according to video decoding standards such as ITU-T H.265 (also known as the High Efficiency Video Coding (HEVC) standard) or extensions thereof such as MultiView or Scalable Video Coding Extensions. Alternatively, the video encoder 200 and video decoder 300 can operate according to other proprietary or industry standards such as the ITU-T H.266 standard (also known as Versatile Video Coding (VVC)). A draft of the VVC standard is described in: Bross et al., “Versatile Video Coding (Draft 10)”, Joint Video Experts Group (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 18th meeting by teleconference, 22 June–1 July 2020, JVET-S2001-vA (hereinafter referred to as “VVC”). However, the technology of this disclosure is not limited to any particular decoding standard.

[0045] Typically, video encoder 200 and video decoder 300 can perform block-based decoding of images. The term "block" generally refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used during encoding and / or decoding). For example, a block may include a two-dimensional matrix of samples of luminance and / or chrominance data. Typically, video encoder 200 and video decoder 300 can decode video data represented in YUV (e.g., Y, Cb, Cr) format. That is, instead of decoding the red, green, and blue (RGB) data used for images, video encoder 200 and video decoder 300 can decode both luminance and chrominance components, where chrominance components may include both red hue and blue hue chrominance components. In some examples, video encoder 200 converts received RGB-formatted data to YUV representation before encoding, and video decoder 300 converts the YUV representation to RGB format. Alternatively, preprocessing and post-processing units (not shown) can perform these conversions.

[0046] In summary, this disclosure may relate to the decoding (e.g., encoding and decoding) of images to include processes of encoding or decoding image data. Similarly, this disclosure may relate to the decoding of blocks of images to include processes of encoding or decoding data used for blocks (e.g., prediction and / or residual decoding). Encoded video bitstreams typically include a series of values ​​for representing decoding decisions (e.g., decoding modes) and syntax elements that segment images into blocks. Therefore, references to decoding images or blocks should generally be understood as decoding the values ​​of syntax elements used to form images or blocks.

[0047] HEVC defines various blocks, including decoding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video decoder (such as a video encoder 200) partitions the decoding tree unit (CTU) into CUs based on a quadtree structure. That is, the video decoder partitions the CTU and CU into four equal, non-overlapping squares, and each node of the quadtree has zero or four child nodes. Nodes without child nodes can be called "leaf nodes," and a CU with such a leaf node can include one or more PUs and / or one or more TUs. The video decoder can further partition the PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents a partition of a TU. In HEVC, a PU represents inter-frame prediction data, while a TU represents residual data. CUs with intra-frame prediction include intra-frame prediction information, such as intra-frame mode indication.

[0048] As another example, video encoder 200 and video decoder 300 can be configured to operate according to VVC. According to VVC, the video decoder (such as video encoder 200) segments the image into multiple decoding tree units (CTUs). Video encoder 200 can segment CTUs according to a tree structure (such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure). The QTBT structure eliminates the concept of multiple segmentation types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels: a first level segmented according to quadtree segmentation and a second level segmented according to binary tree segmentation. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to decoding units (CUs).

[0049] In the MTT partitioning structure, blocks can be partitioned using quadtree (QT) partitioning, binary tree (BT) partitioning, and one or more types of ternary tree (TT) partitioning (also known as triplet (TT)). A ternary tree or triplet partitioning is a partition in which a block is divided into three sub-blocks. In some examples, a ternary tree or triplet partitioning divides a block into three sub-blocks without partitioning the original block through a center. The partitioning types in MTT (e.g., QT, BT, and TT) can be symmetric or asymmetric.

[0050] In some examples, the video encoder 200 and the video decoder 300 may use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, the video encoder 200 and the video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luma component and another QTBT / MTT structure for the two chroma components (or two QTBT / MTT structures for the respective chroma components).

[0051] The video encoder 200 and video decoder 300 can be configured to use per-HEVC quadtree segmentation, QTBT segmentation, MTT segmentation, or other segmentation structures. For illustrative purposes, a description of the techniques of this disclosure is given with respect to QTBT segmentation. However, it should be understood that the techniques of this disclosure can also be applied to video decoders configured to use quadtree segmentation or other types of segmentation.

[0052] In some examples, a CTU includes a decoded tree block (CTB) of luminance samples, two corresponding CTBs of chrominance samples of an image with three sample arrays, or a CTB of samples of a monochrome image or an image decoded using three separate color planes and a syntax structure for decoding the samples. A CTB can be an N×N block of samples (for some value of N) such that dividing a component into a CTB is a partition. A component is an array or a single sample of one of the three arrays (one luminance and two chrominance) that make up an image in a 4:2:0, 4:2:2, or 4:4:4 color format, or an array or a single sample of an array that makes up an image in monochrome format. In some examples, a decoded block is an M×N block of samples (for some values ​​of M and N) such that dividing a CTB into a decoded block is a partition.

[0053] Blocks (e.g., CTUs or CUs) can be grouped in various ways within an image. As an example, a brick-shaped block can refer to a rectangular area of ​​a row of CTUs within a specific slice in an image. A slice can be a rectangular area of ​​CTUs within a specific slice column and a specific slice row in an image. A slice column refers to a rectangular area of ​​CTUs with a height equal to the height of the image and a width specified by syntax elements (e.g., in a set of image parameters). A slice row refers to a rectangular area of ​​CTUs with a height specified by syntax elements (e.g., in a set of image parameters) and a width equal to the width of the image.

[0054] In some examples, a region can be divided into multiple brick blocks, each of which may include one or more CTU rows within the region. A region that is not divided into multiple brick blocks may still be referred to as a brick block. However, a brick block that is a true subset of a region may not be referred to as a region.

[0055] The bricks in an image can also be arranged as slices. A slice can be an integer number of bricks in the image, which can be exclusively contained in a single Network Abstraction Layer (NAL) unit. In some examples, a slice consists of multiple complete regions or a continuous sequence of complete bricks consisting of only one region.

[0056] This disclosure uses "NxN" and "N by N" interchangeably to refer to the sample size of a block (such as a CU or other video block) in the vertical and horizontal dimensions, for example, 16x16 samples or 16 by 16 samples. Typically, a 16x16 CU will have 16 samples in the vertical direction (y = 16) and 16 samples in the horizontal direction (x = 16). Similarly, an NxN CU typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU can be arranged in rows and columns. Furthermore, a CU does not necessarily need to have the same number of samples in the horizontal direction as it does in the vertical direction. For example, a CU can include NxM samples, where M is not necessarily equal to N.

[0057] The video encoder 200 encodes video data for use in predicting and / or residual information, as well as other information, for the CU. The prediction information indicates how the CU will be predicted to form a prediction block for the CU. The residual information typically represents the sample-by-sample difference between a sample of the CU before encoding and the prediction block.

[0058] To predict the Cubic Frame (CU), the video encoder 200 typically forms prediction blocks for the CU using either inter-frame prediction or intra-frame prediction. Inter-frame prediction generally refers to predicting the CU based on data from previously decoded images, while intra-frame prediction generally refers to predicting the CU based on data from previously decoded images of the same frame. To perform inter-frame prediction, the video encoder 200 can generate prediction blocks using one or more motion vectors. The video encoder 200 can typically perform motion search to identify, for example, reference blocks that closely match the CU in terms of the difference between the CU and a reference block. The video encoder 200 can calculate difference metrics using sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculations to determine whether the reference block closely matches the current CU. In some examples, the video encoder 200 can use unidirectional or bidirectional prediction to predict the current CU.

[0059] Some examples of VVC also provide an affine motion compensation mode, which can be considered an inter-frame prediction mode. In affine motion compensation mode, the video encoder 200 can determine two or more motion vectors representing non-translational motion (such as zooming in or out, rotation, perspective motion, or other irregular types of motion).

[0060] To perform intra-frame prediction, the video encoder 200 can select an intra-frame prediction mode to generate prediction blocks. Some examples of VVC provide sixty-seven intra-frame prediction modes, including various directional modes, as well as planar and DC modes. Typically, the video encoder 200 selects an intra-frame prediction mode that describes the samples of the current block (e.g., a block of a CU) to be predicted based on, which are the neighboring samples of the current block. Assuming the video encoder 200 decodes the CTU and CU in raster scan order (from left to right, from top to bottom), such samples can typically be located above, to the upper left, or to the left of the current block within the same image.

[0061] The video encoder 200 encodes data representing the prediction mode used for the current block. For example, for inter-frame prediction modes, the video encoder 200 may encode data indicating which of the various available inter-frame prediction modes is used, as well as motion information for the corresponding mode. For unidirectional or bidirectional inter-frame prediction, for example, the video encoder 200 may use Advanced Motion Vector Prediction (AMVP) or merging modes to encode motion vectors. The video encoder 200 may use similar modes to encode motion vectors used for affine motion compensation modes.

[0062] Following a prediction, such as intra-frame or inter-frame prediction of a block, the video encoder 200 can compute residual data for that block. The residual data (such as a residual block) represents the sample-by-sample difference between the block and the prediction block used to form the block, which is formed using the corresponding prediction mode. The video encoder 200 can apply one or more transforms to the residual block to produce transformed data in the transform domain rather than the sample domain. For example, the video encoder 200 can apply a Discrete Cosine Transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. Additionally, the video encoder 200 can apply a secondary transform after the first transform, such as a Mode-dependent Inseparable Quadratic Transform (MDNSST), a Signal-dependent Transform, a Karhunen-Loeve Transform (KLT), etc. The video encoder 200 produces transform coefficients after applying one or more transforms.

[0063] As described above, after any transformation to produce transform coefficients, the video encoder 200 can perform quantization of the transform coefficients. Quantization generally refers to the process in which the transform coefficients are quantized to potentially reduce the amount of data used to represent them, thereby providing further compression. By performing the quantization process, the video encoder 200 can reduce the bit depth associated with some or all of the transform coefficients. For example, the video encoder 200 can reduce the bit depth during quantization. n The value of a bit is rounded down to m The value of the bit, wheren Greater than m In some examples, in order to perform quantization, the video encoder 200 may perform a bitwise right shift of the value to be quantized.

[0064] After quantization, the video encoder 200 can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix including the quantized transform coefficients. The scan can be designed to place higher-energy (and therefore lower-frequency) transform coefficients before the vector and lower-energy (and therefore higher-frequency) transform coefficients after the vector. In some examples, the video encoder 200 can utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector, and then entropy-encode the quantized transform coefficients of the vector. In other examples, the video encoder 200 can perform an adaptive scan. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 200 can entropy-encode the one-dimensional vector, for example, according to Context Adaptive Binary Arithmetic Coding (CABAC). The video encoder 200 can also entropy-encode the values ​​of syntax elements used to describe metadata associated with the encoded video data for use by the video decoder 300 when decoding the video data.

[0065] To perform CABAC, the video encoder 200 can assign context from within a context model to the symbols to be transmitted. Context may involve, for example, whether the neighboring values ​​of a symbol are zero. Probability determination can be based on the context assigned to the symbol.

[0066] The video encoder 200 can also generate, for example, grammatical data (such as block-based grammatical data, image-based grammatical data, and sequence-based grammatical data), or other grammatical data (such as sequence parameter sets (SPS), image parameter sets (PPS), or video parameter sets (VPS)) destined for the video decoder 300 in image headers, block headers, and slice headers. Similarly, the video decoder 300 can decode such grammatical data to determine how to decode the corresponding video data.

[0067] In this way, the video encoder 200 can generate a bitstream that includes encoded video data, such as syntax elements describing the segmentation of an image into blocks (e.g., CUs) and prediction and / or residual information for those blocks. Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0068] Typically, the video decoder 300 performs a process reciprocal to that performed by the video encoder 200 to decode the encoded video data of the bitstream. For example, the video decoder 300 can use CABAC to decode the values ​​of syntax elements used for the bitstream in a manner substantially similar to, but reciprocal to, the CABAC encoding process of the video encoder 200. Syntax elements can define segmentation information for segmenting images into CTUs and for segmenting each CTU according to a corresponding segmentation structure (such as a QTBT structure) to define the CUs of the CTU. Syntax elements can also define prediction and residual information for blocks (e.g., CUs) of the video data.

[0069] Residual information can be represented, for example, by quantized transform coefficients. The video decoder 300 can inversely quantize and inversely transform the quantized transform coefficients of the block to regenerate a residual block for that block. The video decoder 300 uses a signal-informed prediction mode (intra-frame prediction or inter-frame prediction) and associated prediction information (e.g., motion information for inter-frame prediction) to form a prediction block for that block. The video decoder 300 can then combine the prediction block and the residual block (on a sample-by-sample basis) to regenerate the original block. The video decoder 300 can perform additional processing, such as performing a deblocking process to reduce visual artifacts along the block boundaries.

[0070] In summary, this disclosure may involve "signaling" certain information (such as syntax elements). The term "signaling" can generally refer to the transmission of values ​​for syntax elements and / or other data used to decode encoded video data. That is, video encoder 200 can signal values ​​for syntax elements in the bitstream. Generally, signaling refers to generating values ​​in the bitstream. As described above, source device 102 can transmit the bitstream to destination device 116 substantially in real time or not in real time (such as when syntax elements are stored in storage device 112 for later retrieval by destination device 116).

[0071] According to the technology of this disclosure, as will be explained in more detail below, the video encoder 200 and the video decoder 300 can be configured to perform one or more filtering techniques of this disclosure, including neural network-based filtering techniques.

[0072] Most video decoding standards since H.261 have been based on... Figure 2 The so-called hybrid video decoding principle is shown in the figure.

[0073] Figure 2This is a block diagram illustrating an example hybrid video decoding framework. The term hybrid refers to a combination of two methods to reduce redundancy in a video signal: prediction and transform decoding with quantization of the prediction residuals. However, prediction and transform reduce redundancy in the video signal by decorrelation, while quantization reduces the amount of data represented by the transform coefficients by decreasing their precision (ideally by removing only irrelevant details). This hybrid video coding design principle is also used in two recent standards: HEVC and VVC.

[0074] like Figure 2 As shown, a modern hybrid video decoder 130 typically includes block segmentation, motion compensation or inter-picture prediction, intra-picture prediction, transform, quantization, entropy decoding, and post-loop / intra-loop filtering. Figure 2 In the example, the video decoder 130 includes a summing unit 134, a transform unit 136, a quantization unit 138, an entropy decoding unit 140, an inverse quantization unit 142, an inverse transform unit 144, a summing unit 146, a loop filter unit 148, a decoded picture buffer (DPB) 150, an intra-frame prediction unit 152, an inter-frame prediction unit 154, and a motion estimation unit 156.

[0075] Typically, the video decoder 130 receives input video data 132 while encoding video data. Block segmentation is used to divide the received video data into smaller blocks for prediction and transformation processes. Early video decoding standards used a fixed block size, typically 16×16 samples. More recent standards (such as HEVC and VVC) employ tree-based segmentation structures to provide flexible segmentation.

[0076] Motion estimation unit 156 and inter-frame prediction unit 154 can predict input video data 132, for example, based on previously decoded data from DPB 150. Motion compensation or inter-picture prediction utilizes the redundancy present between (hence “inter-picture”) images in the video sequence. Predictions are obtained from one or more previously decoded images (i.e., reference images) based on block-based motion compensation used in all modern video codecs. The corresponding region used to generate inter-frame predictions is indicated by motion information including motion vectors and reference image indices.

[0077] The summing unit 134 can calculate the difference between the input video data 132 and the predicted data from the intra-frame prediction unit 152 or the inter-frame prediction unit 134 based on the residual data. The summing unit 134 provides the residual block to the transform unit 136, which applies one or more transforms to the residual block to generate a transform block. The quantization unit 138 quantizes the transform block to form quantized transform coefficients. The entropy decoding unit 140 entropy-encodes the quantized transform coefficients and other syntax elements (such as motion information or intra-frame prediction information) to generate an output bitstream 158.

[0078] Simultaneously, the inverse quantization unit 142 inverse quantizes the quantized transform coefficients, and the inverse transform unit 144 inverse transforms the transform coefficients to regenerate the residual block. The summation unit 146 (on a sample-by-sample basis) combines the residual block with the prediction block to produce a decoded block of video data. The loop filter unit 148 applies one or more filters (e.g., at least one of a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter) to the decoded block to produce a filtered decoded block.

[0079] According to the technology of this disclosure, the loop filter unit 148 can also be configured to refine the filtered and decoded blocks. For example, the loop filter unit 148 can apply a scaling factor to the filtered and decoded blocks to form refined filtered and decoded blocks. The loop filter unit 148 can then add the refined filtered and decoded blocks to the corresponding decoded blocks. The loop filter unit 148 can then store these final blocks in the DPB 150. The data in the DPB 150 can be used as reference data, for example, by the inter-frame prediction unit 154 or the intra-frame prediction unit 152, and can also be output as output video data 160.

[0080] Figure 3 This is a conceptual diagram showing a hierarchical prediction structure 170 with a group of pictures (GOP) size of 16.

[0081] In-picture prediction utilizes spatial redundancy present within a picture (hence the term "in-picture") by deriving predictions for blocks based on spatially adjacent (reference) samples that have already been decoded / decoded. Recent video codecs (including AVC, HEVC, and VVC) employ orientation angle prediction, DC prediction, and planar or plane-based prediction.

[0082] Some example hybrid video decoding standards apply block transforms to the prediction residuals (regardless of whether they originate from inter-picture or intra-picture predictions). In earlier standards (including H.261 / 262 / 263), the Discrete Cosine Transform (DCT) was used. In HEVC and VVC, more transform kernels are applied in addition to DCT to account for different statistics in a particular video signal.

[0083] Quantization aims to reduce the precision of input values ​​or sets of input values ​​in order to reduce the amount of data required to represent those values. In hybrid video decoding, quantization is typically applied to individual transformed residual samples (e.g., transform coefficients), resulting in integer coefficient levels. In recent video decoding standards, the stride is derived from a so-called quantization parameter (QP) that controls fidelity and bit rate. A larger stride reduces the bit rate but can also degrade quality; for example, this can cause video images to exhibit blockiness and blurred details.

[0084] Context-adaptive binary arithmetic decoding (CABAC) is used in recent video codecs such as AVC, HEVC and VVC due to its high efficiency.

[0085] Post-loop / intra-loop filtering is a filtering process (or a combination of such processes) applied to a reconstructed image to reduce decoding artifacts. The input to the filtering process is typically the reconstructed image, which is a combination of the reconstructed residual signal (including quantization errors) and predictions. Figure 2 As shown, the reconstructed image after in-loop filtering is stored and used as a reference for inter-image prediction of subsequent images. Decoding artifacts are primarily determined by the QP (Quality-Proofing Process). Therefore, QP information is typically used in the design of the filtering process. In HEVC, the in-loop filter includes deblocking filtering and Sample Adaptive Offset (SAO) filtering. In the VVC standard, an Adaptive Loop Filter (ALF) is introduced as a third filter. The ALF filtering process is shown below: (1) in These are samples before the filtering process. These are the sample values ​​after the filtering process. Represents the filter coefficients. It is a clipping function and This represents the limiting parameter. Variables k and l can be used... and The changes between them, among which L Indicates the filter length. Limiting function. Its corresponding function The clipping operation introduces nonlinearity to make ALF more efficient by reducing the influence of neighboring sample values ​​that differ too much from the current sample value. In VVC, filter parameters can be signaled in the bitstream and can be selected from a predefined set of filters. The ALF filtering process can also be summarized by the following equation: (2)

[0086] In some examples, it has been shown that embedding neural networks into hybrid video decoding frameworks can improve compression efficiency. Neural networks have been used in intra-frame prediction and inter-frame prediction modules to improve prediction efficiency. NN-based loop filtering has also been a research topic in recent years. In some examples, the filtering process is applied as a post-filter. In this case, the filtering process is only applied to the output image, and the unfiltered image is used as a reference image.

[0087] In addition to existing filters, neural network-based filters, such as deblocking filters, SAO, and / or ALF, can be applied. Neural network-based filters can also be specifically designed to replace all existing filters.

[0088] Figure 4 This is a block diagram illustrating an example filter based on a convolutional neural network with four layers. Figure 4 As shown, the NN-based filtering process 182 takes the reconstructed samples 180 as input, and the intermediate outputs are residual samples, which are added back to the input to refine the input samples, thus producing refined data 184. NN filters can use all color components as input to utilize cross-component correlations. Different components can share the same filters (including network structure and model parameters), or each component can have its own specific filters.

[0089] The filtering process can also be summarized as follows: (3)

[0090] The model structure and parameters of the NN-based filter can be predefined and stored in the video encoder 200 and video decoder 300. The filter can also be notified by signals in the bitstream.

[0091] Existing neural network-based filters used for video decoding may have problems. For example, predefined filters (e.g., neural network-based filters or ALF) may be trained on large datasets of videos and images. Such training sets may be optimal in general, but may not be optimal for specific distorted sequences.

[0092] This disclosure describes techniques described below that can improve decoding efficiency and / or reduce distortion when using neural network-based filters for video decoding. The itemized techniques described below can be applied individually. Alternatively, any combination of the techniques described below can be applied. The video encoder 200 and video decoder 300 can be configured individually or in any combination according to any of the various techniques described in this disclosure.

[0093] Return to reference Figure 1In one example of this disclosure, the video encoder 200 and video decoder 300 can be configured to apply a refinement process to the output residual of the filtering process, and then add the refined filter output residual to update the input samples. The proposed approach can be expressed as follows: (4) Where f() is a function applied to the refinement process of the filtered output residual for a specific filtering process. The video encoder 200 and the video decoder 300 can be configured to apply one or more of the following processes as the refinement process. a. As an example, the video encoder 200 and the video decoder 300 can modify the filtered value by multiplying the scaling factor by the filtered sample of the decoded block, and then add the refined filtered value to the decoded block to update the input sample. i. The video encoder 200 can quantize the scaling factor using an integer value with predefined bit precision. The video decoder 300 can inversely quantize this value using an integer value with predefined bit precision to determine the scaling factor. ii. The video encoder 200 can signal the value of the scaling factor in the bitstream. The video decoder 300 can determine the value of the scaling factor based on the data signaled in the bitstream. iii. The video encoder 200 can signal the scaling factor value in the bitstream as a syntax element in the slice header, picture header, picture parameter set (PPS), adaptive parameter set (APS), or any other high-level syntax element body. The video decoder 300 can determine the scaling factor based on the signaled data in the corresponding data structure. iv. The video encoder 200 can signal the value of the scaling factor as a block-level (e.g., at the CTU level) syntax element in the bitstream. The video decoder 300 can determine the scaling factor based on the data signaled at the block level. v. The video encoder 200 may signal a set of scaling factors in the bitstream, or the set of scaling factors may be predefined at both the video encoder 200 and the video decoder 300. The video encoder 200 may signal an index at a block (e.g., at the CTU level) to specify the value of the scaling factor. The video decoder 300 may use a table that maps index values ​​to the set of scaling factors to determine the scaling factor based on the index value. vi. The scaling factor can be derived separately for each color component. That is, the video encoder 200 can individually signal the scaling factor for each color component (e.g., the luminance component, the blue hue component, and the red hue component). The video decoder 300 can determine the scaling factor for each color component based on the signaled value. b. As another example, the video encoder 200 and the video decoder 300 can modify the filtered output residual by multiplying by a scaling factor and adding an offset to the scaled filtered value, and then add the refined filtered value to the input sample. i. The video encoder 200 can quantize the scaling factor using an integer value with predefined bit precision. The video decoder 300 can inversely quantize this value using an integer value with predefined bit precision to determine the scaling factor. ii. The video encoder 200 can quantize the offset using an integer value with predefined bit precision. The video decoder 300 can inversely quantize this value using an integer value with predefined bit precision to determine the offset. iii. The video encoder 200 can signal the values ​​of the scaling factor and offset in the bitstream. The video decoder 300 can determine the values ​​of the scaling factor and offset based on the data signaled in the bitstream. iv. The video encoder 200 can signal the values ​​of scaling factor and offset in the bitstream as syntax elements in a slice header, picture header, picture parameter set (PPS), adaptive parameter set (APS), or any other high-level syntax element body. The video decoder 300 can determine the scaling factor and offset based on the signaled data in the corresponding data structure. v. The values ​​of scaling factor and offset can be derived during the encoding process and signaled in the bitstream as syntax elements in slice headers, picture headers, adaptive parameter sets (APS), or any other high-level syntax element body. vi. The video encoder 200 can signal the values ​​of scaling factor and offset as block-level (e.g., at the CTU level) syntax elements in the bitstream. The video decoder 300 can determine the scaling factor and offset based on the data signaled at the block level. vii. The video encoder 200 may signal the scaling factor set and offset set in the bitstream, or the scaling factor set and offset set may be predefined at both the video encoder 200 and the video decoder 300. The video encoder 200 may signal index values ​​at the block level (e.g., at the CTU level) to specify the values ​​of the scaling factor and offset. The video decoder 300 may use a corresponding table mapping index values ​​to the scaling factor set and offset set to determine the scaling factor and offset based on the index values. viii. The scaling factor and offset can be derived separately for each color component. That is, the video encoder 200 can individually signal the scaling factor and offset for each color component (e.g., the luminance component, the blue hue component, and the red hue component). The video decoder 300 can determine the scaling factor and offset for each color component based on the values ​​signaled. c. The filtering process mentioned in this disclosure can be any filtering process that generates a residual output for updating the input samples. i. As an example, the filtering process mentioned in this section is based on NN filters. ii. As another example, the filtering process mentioned in this section is an NN-based loop filter applied in the context of a video codec. iii. As another example, the filtering process mentioned in this section is a NN-based post-filter applied in the context of a video codec. iv. As another example, the filtering process mentioned in this section is an adaptive in-loop filter applied in the context of a video codec. v. As another example, the filtering process mentioned in this section is a predefined adaptive in-loop filter applied in the context of a video codec. d. A signaling flag can be used to indicate whether refinement is applied to refine the filter output residuals. This flag can be signaled in the bitstream as a syntax element in the slice header, picture header, picture parameter set (PPS), adaptive parameter set (APS), or any other high-level syntax element body.

[0094] Therefore, the video encoder 200 and video decoder 300 can be configured to decode blocks of video data and then apply filters to the decoded blocks of video data. These filters can be one of the following: neural network-based filters, neural network-based loop filters, neural network-based post-loop filters, adaptive in-loop filters, or predefined adaptive in-loop filters. In some examples, the video encoder 200 and video decoder 300 can apply a combination of such filters to the decoded blocks of video data.

[0095] After filtering the decoded blocks (thus forming filtered blocks), the video encoder 200 and video decoder 300 can refine the filtered blocks, for example, by multiplying samples of the filtered blocks by a scaling factor to form refined filtered blocks. Then, the video encoder 200 and video decoder 300 can combine samples of the decoded blocks with corresponding samples of the refined filtered blocks. The video encoder 200 and video decoder 300 can execute the above formula (4) to form refined filtered blocks, and combine the refined filtered blocks with the decoded blocks to form a final decoded and filtered block, which is stored in a decoded picture buffer (DPB) and used for output.

[0096] Video encoder 200 can determine the scaling factor based on a rate-distortion optimization (RDO) process. For example, video encoder 200 can test a wide variety of different scaling factors and determine which of the tested scaling factors reduces distortion without excessively increasing the bitrate of the corresponding bitstream. Video encoder 200 can select the scaling factor with the best RDO performance as the scaling factor for a block (or a slice or picture containing a block). Video encoder 200 can then encode data representing the selected scaling factor, for example, as values ​​of syntax elements in slice headers, adaptive parameter sets (APS), picture headers, picture parameter sets (PPS), block headers (e.g., CTU headers), etc. Video decoder 300 can determine the scaling factor based on the encoded data representing the scaling factor.

[0097] In some examples, video encoder 200 and video decoder 300 can encode the values ​​of syntax elements that directly represent scaling factors. In some examples, video encoder 200 and video decoder 300 can be configured with a lookup table that maps index values ​​to scaling factors, and video encoder 200 can encode values ​​representing index values ​​corresponding to the selected scaling factor. Therefore, video decoder 300 can decode index values ​​and use the index values ​​and the lookup table to determine the scaling factor; that is, by determining the scaling factor to which the index value is mapped by the lookup table. The lookup table can also be referred to as an index table.

[0098] In some examples, the video encoder 200 can also determine whether to enable filter refinement, as discussed above. For instance, if none of the scaling factors produce a sufficient RDO value, the video encoder 200 can disable filter refinement, for example, by signaling the value of a syntax element indicating that filter refinement is disabled. The video decoder 300 can use the value of the syntax element to determine whether to perform filter refinement.

[0099] In some examples, the video encoder 200 can quantize the value representing the scaling factor using an integer value with predefined bit precision. Therefore, the video decoder 300 can inversely quantize the decoded value using an integer value with predefined bit precision to determine the scaling factor.

[0100] In another example of this disclosure, the output residuals of multiple filters are combined to update the input samples. As an example, the proposed method can be expressed as: (5) Here, f1() and f2() are functions applied to refine the filter output residuals for a specific filtering process. Combinations of more than one model can be summarized as follows: (6)

[0101] In other words, the video encoder 200 and the video decoder 300 may apply two or more (e.g., multiple) scaling factors and filters to the decoded block to form a corresponding refined and filtered block, and then combine the decoded block with the refined and filtered block, for example, according to either of the above formulas (5) or (6).

[0102] The video encoder 200 and the video decoder 300 can be configured to apply one or more of the following processes as a refinement process. a) As an example, the video encoder 200 and the video decoder 300 can modify the output residual of each filtering process, including multiplying by a scaling factor, and then combining the refined filtered block with the input sample. i) The video encoder 200 can quantize the scaling factor using an integer value with predefined bit precision. The video decoder 300 can inversely quantize the scaling factor using an integer value with predefined bit precision. ii) The video encoder 200 can derive the value of the scaling factor during the encoding process and signal these values ​​in the bitstream. The video decoder 300 can determine the scaling factor based on the signaled values. iii) The video encoder 200 may signal the value of the scaling factor as a syntax element in a slice header, picture header, picture parameter set (PPS), adaptive parameter set (APS), or any other high-level syntax element body. The video decoder 300 may determine the value of the scaling factor based on the corresponding data structure. iv) The video encoder 200 may signal the value of the scaling factor as a block-level (e.g., at the CTU level) syntax element in the bitstream. The video decoder 300 may determine the value of the scaling factor based on the signaled block-level syntax element. v) The video encoder 200 may signal a set of scaling factors in the bitstream, or the set of scaling factors may be predefined at both the video encoder 200 and the video decoder 300. The video encoder 200 may signal index values ​​at the block level (e.g., CTU level) to specify the values ​​of the scaling factors. vi) The video encoder 200 can signal the scaling factor for each color component separately, and the video decoder 300 can determine the scaling factor for each color component separately based on the signal signal value. b. As another example, the video encoder 200 and video decoder 300 can modify the output residual of each filtering process by multiplying by a corresponding scaling factor and adding an offset. The video encoder 200 and video decoder 300 can then add the refined, filtered data to the input samples. i) The video encoder 200 can quantize the scaling factor using an integer value with predefined bit precision. The video decoder 300 can inversely quantize the scaling factor using an integer value with predefined bit precision. ii) The video encoder 200 can quantize the offset using an integer value with predefined bit precision. The video decoder 300 can inversely quantize the offset using an integer value with predefined bit precision. iii) The video encoder 200 can derive the values ​​of the scaling factor and offset during the encoding process and signal these values ​​in the bitstream. The video decoder 300 can determine the scaling factor and offset based on the values ​​signaled. iv) The video encoder 200 can signal the values ​​of the scaling factor and offset as syntax elements in the slice header, picture header, picture parameter set (PPS), adaptive parameter set (APS), or any other high-level syntax element body. The video decoder 300 can determine the values ​​of the scaling factor and offset based on the corresponding data structure. v) To save on signaling overhead for scaling factors, predefined fixed scaling factors can be used at both the video encoder 200 and the video decoder 300. Examples of different scaling factors are shown below.

[0103] The video encoder 200 can select one of the scaling factors based on the RD cost, and then use a direct representation of the scaling factor or an index representing the scaling factor in the bitstream data to signal the optimal scaling factor (i.e., the scaling factor with the best RD cost). The index can be decoded into unary code or truncated binary code, or variable-length decoding based on the statistical properties of the scaling factor. When the scaling factor has the most probable specific value across the entire input sequence, unary code signaling can be selected as the signaling method, where the smallest index value can be assigned to the most probable scaling factor value. vi. The values ​​of scaling factors and offsets can be derived during the encoding process, and the values ​​of scaling factors and offsets can be signaled as block-level (e.g., CTU-level) syntax elements in the bitstream. vii. Signal a set of scaling factors and offsets in the bitstream or predefine them on the encoder and decoder side, and signal an index at a block (e.g., CTU level) to specify the values ​​of the scaling factors and offsets. viii. The scaling factor and offset can be derived separately for each color component. c) The filtering process mentioned in this disclosure can be any filtering process that generates a residual output for updating the input samples. i. As an example, the filtering process mentioned in this section is based on NN filters. ii. As another example, the filtering process mentioned in this section is an NN-based loop filter applied in the context of a video codec. iii. As another example, the filtering process mentioned in this section is a set of NN-based post-filters applied in the context of a video codec. iv. As another example, the filtering process mentioned in this section is an adaptive in-loop filter applied in the context of a video codec. v. As another example, the filtering process mentioned in this section is a predefined adaptive in-loop filter applied in the context of a video codec.

[0104] As an example, one of the proposed ideas can be implemented using the following equation: (7)

[0105] The "shift" value is a predefined positive integer. The shift value can be equal to any positive value. Typical shift values ​​are in the range of 4 to 8. The "shift" bits are used to quantize the scale_factor and to signal its value in the bitstream. The "offset" value can be set to zero. Alternatively, the offset value can be signaled in the bitstream. In this example, the same precision as the input sample is used for quantization. Quantify it.

[0106] As another example, N bits are used to quantize the scale_factor, where the value of N is less than a "shift". In this example, a higher precision than that of the input sample is used for quantization. Quantify it.

[0107] You can also apply a limiting operation to... The value is limited to the dynamic range of the original input sample.

[0108] As an example, one of the proposed ideas can be implemented using the following equation: (8)

[0109] The "shift" value in formula (8) is a predefined positive integer. The shift value can be equal to any positive value. Example shift values ​​can be in the range of 4 to 8. The "shift" bits are used to quantize scale_factor1 and scale_factor2, and to signal their values ​​in the bitstream. The "offset" value can be set to zero. In another example, the "offset" value can be signaled in the bitstream. In this example, the same precision as the input sample is used to quantize... and Quantify it.

[0110] As another example, N bits are used to quantize scale_factor1 and scale_factor2, where the value of N is less than a "shift". In this example, a higher precision than that of the input sample is used for quantization. Quantify it.

[0111] You can also apply a limiting operation to... The value is limited to the dynamic range of the original input sample.

[0112] Figure 5A and 5BThis is a conceptual diagram illustrating an example Quadtree Binary Tree (QTBT) structure 190 and its corresponding Decoding Tree Unit (CTU) 192. Solid lines represent quadtree splits, and dashed lines indicate binary tree splits. In each split (i.e., non-leaf) node of the binary tree, a flag is sent via signaling to indicate which split type (i.e., horizontal or vertical) is used, where, in this example, 0 indicates a horizontal split and 1 indicates a vertical split. For quadtree splits, since the quadtree node splits the block horizontally and vertically into four sub-blocks of equal size, there is no need to indicate the split type. Accordingly, the video encoder 200 can encode the following, and the video decoder 300 can decode the following: syntax elements (such as split information) for the region tree level (i.e., solid lines) of the QTBT structure 190, and syntax elements (such as split information) for the prediction tree level (i.e., dashed lines) of the QTBT structure 190. The video encoder 200 can encode video data (such as prediction and transform data) for a CU represented by the terminal leaf nodes of the QTBT structure 190, and the video decoder 300 can decode the video data.

[0113] generally, Figure 5B The CTU 192 can be associated with parameters that define the size of the blocks corresponding to the nodes at the first and second levels of the QTBT structure 190. These parameters can include the CTU size (representing the size of the CTU 192 in the sample), the minimum quadtree size (MinQTSize, which represents the minimum allowed quadtree leaf node size), the maximum binary tree size (MaxBTSize, which represents the maximum allowed binary tree root node size), the maximum binary tree depth (MaxBTDepth, which represents the maximum allowed binary tree depth), and the minimum binary tree size (MinBTSize, which represents the minimum allowed binary tree leaf node size).

[0114] The root node corresponding to the CTU in a QTBT structure can have four child nodes at the first level of the QTBT structure, where each child node can be partitioned according to a quadtree. That is, the node at the first level is a leaf node (with no child nodes) or has four child nodes. An example of QTBT structure 190 represents such a node as including a parent node and child nodes with solid-line branches. If the node at the first level is not larger than the maximum allowed binary tree root node size (MaxBTSize), the node can be further partitioned by the corresponding binary tree. The binary tree partitioning of a node can be iterated until the node resulting from the partition reaches the minimum allowed binary tree leaf node size (MinBTSize) or the maximum allowed binary tree depth (MaxBTDepth). An example of QTBT structure 190 represents such a node as having dashed-line branches. The binary tree leaf nodes are called decoding units (CUs), which are used for prediction (e.g., intra-image or inter-image prediction) and transformation without any further partitioning. As discussed above, CUs can also be referred to as “video chunks” or “blocks”.

[0115] In one example of a QTBT segmentation structure, the CTU size is set to 128x128 (luminance sample and two corresponding 64x64 chrominance samples), MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize (for both width and height) is set to 4, and MaxBTDepth is set to 4. First, a quadtree segmentation is applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes ranging from 16x16 (i.e., MinQTSize) to 128x128 (i.e., the CTU size). If a quadtree leaf node is 128x128, it will not be further split by the binary tree because this size exceeds MaxBTSize (i.e., 64x64 in this example). Otherwise, the quadtree leaf node will be further split by the binary tree. Therefore, the quadtree leaf node also serves as the root node of the binary tree, with a binary tree depth of 0. When the depth of a binary tree reaches MaxBTDepth (4 in this example), further splitting is not allowed. A binary tree node with a width equal to MinBTSize (4 in this example) means that further vertical splitting (i.e., width-based division) is not allowed for that binary tree node. Similarly, a binary tree node with a height equal to MinBTSize means that further horizontal splitting (i.e., height-based division) is not allowed for that binary tree node. As mentioned above, the leaf nodes of the binary tree are referred to as CUs and are further processed according to prediction and transformation without further splitting.

[0116] Figure 6This is a block diagram illustrating an example video encoder 200 that can perform the techniques described in this disclosure. Figure 6 This disclosure is provided for illustrative purposes and should not be construed as limiting the scope of the techniques extensively illustrated and described herein. For illustrative purposes, this disclosure describes the video encoder 200 based on VVC (ITU-T H.266, under development) and HEVC (ITU-TH.265) technologies. However, the technologies of this disclosure can be implemented by video encoding devices configured for other video decoding standards.

[0117] exist Figure 6 In the example, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy coding unit 220. Any or all of the video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, DPB 218, and entropy coding unit 220 can be implemented in one or more processors or in processing circuitry. For example, the units of the video encoder 200 can be implemented as one or more circuit or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Furthermore, the video encoder 200 may include additional or alternative processors or processing circuitry to perform these and other functions.

[0118] The video data storage device 230 can store video data to be encoded by the components of the video encoder 200. The video encoder 200 can obtain data from, for example, a video source 104 (…). Figure 1 The video data memory 230 receives video data stored in the video data memory 230. The DPB 218 can act as a reference picture memory, storing reference video data for use when the video encoder 200 predicts subsequent video data. The video data memory 230 and DPB 218 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The video data memory 230 and DPB 218 can be provided by the same memory device or separate memory devices. In various examples, the video data memory 230 can be on-chip (as shown) with other components of the video encoder 200, or off-chip relative to those components.

[0119] In this disclosure, reference to video data memory 230 should not be construed as limited to memory inside video encoder 200 (unless specifically described therein) or memory outside video encoder 200 (unless specifically described therein). Rather, reference to video data memory 230 should be understood as a reference memory that stores video data received by video encoder 200 for encoding (e.g., video data for the current block to be encoded). Figure 1 The memory 106 can also provide temporary storage for the outputs from the various units of the video encoder 200.

[0120] It shows Figure 6 The various units within this unit help to understand the operations performed by the video encoder 200. This unit can be implemented as a fixed-function circuit, a programmable circuit, or a combination thereof. A fixed-function circuit refers to a circuit that provides a specific function and is pre-configured regarding the operations that can be performed. A programmable circuit refers to a circuit that can be programmed to perform various tasks and provide flexible functionality in terms of the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. A fixed-function circuit can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is typically immutable. In some examples, one or more units within this unit can be different circuit blocks (fixed-function or programmable), and in some examples, one or more units within this unit can be integrated circuits.

[0121] The video encoder 200 may include an arithmetic logic unit (ALU), an essential function unit (EFU), digital circuitry, analog circuitry, and / or a programmable core, formed according to programmable circuitry. In an example where software executed by programmable circuitry is used to perform the operation of the video encoder 200, memory 106 ( Figure 1 The video encoder 200 may store instructions (e.g., target code) of the software received and executed by the video encoder 200, or another memory (not shown) within the video encoder 200 may store such instructions.

[0122] The video data storage unit 230 is configured to store received video data. The video encoder 200 can retrieve images of the video data from the video data storage unit 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data storage unit 230 can be the raw video data to be encoded.

[0123] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra-frame prediction unit 226. The mode selection unit 202 may include additional functional units that perform video prediction based on other prediction modes. As an example, the mode selection unit 202 may include a palette unit, an intra-block copy unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, etc.

[0124] Mode selection unit 202 typically coordinates multiple coding passes to test combinations of coding parameters and the resulting rate-distortion values ​​for such combinations. Coding parameters may include segmenting the CTU into CUs, the prediction mode for the CUs, the transform type for the residual data of the CUs, and the quantization parameters for the residual data of the CUs. Mode selection unit 202 can ultimately select a combination of coding parameters that yields a better rate-distortion value than other tested combinations.

[0125] The video encoder 200 can segment images retrieved from the video data storage 230 into a series of CTUs, and encapsulate one or more CTUs within slices. The mode selection unit 202 can segment the CTUs of the image according to a tree structure (such as the QTBT structure or quadtree structure of HEVC described above). As mentioned above, the video encoder 200 can form one or more CUs by segmenting CTUs according to a tree structure. Such CUs can also be referred to as "video blocks" or "blocks".

[0126] Typically, mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-prediction unit 226) to generate predicted blocks for the current block (e.g., the current CU, or the overlapping portion of PU and TU in HEVC). To perform inter-frame prediction for the current block, motion estimation unit 222 may perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previously decoded pictures stored in DPB 218). Specifically, motion estimation unit 222 may calculate values ​​representing the similarity between a potential reference block and the current block, for example, based on sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), etc. Motion estimation unit 222 may typically perform these calculations using sample-by-sample differences between the current block and the considered reference blocks. Motion estimation unit 222 may identify the reference block with the lowest value obtained from these calculations, indicating the reference block that most closely matches the current block.

[0127] Motion estimation unit 222 can generate one or more motion vectors (MVs) that define the position of a reference block in a reference image relative to the position of the current block in the current image. Motion estimation unit 222 can then provide the motion vectors to motion compensation unit 224. For example, for unidirectional inter-frame prediction, motion estimation unit 222 can provide a single motion vector, while for bidirectional inter-frame prediction, motion estimation unit 222 can provide two motion vectors. Motion compensation unit 224 can then use the motion vectors to generate prediction blocks. For example, motion compensation unit 224 can use the motion vectors to retrieve data from the reference blocks. As another example, if the motion vectors have fractional-sample precision, motion compensation unit 224 can interpolate the values ​​used for the prediction blocks according to one or more interpolation filters. Furthermore, for bidirectional inter-frame prediction, motion compensation unit 224 can retrieve data for two reference blocks identified by corresponding motion vectors and combine the retrieved data, for example, by per-sample averaging or weighted averaging.

[0128] As another example, for intra-prediction or intra-prediction decoding, intra-prediction unit 226 can generate a prediction block based on samples adjacent to the current block. For example, in directional mode, intra-prediction unit 226 can typically mathematically combine the values ​​of adjacent samples and fill these calculated values ​​across the current block in a defined direction to generate a prediction block. As another example, in DC mode, intra-prediction unit 226 can calculate the average of the adjacent samples of the current block and generate a prediction block to include the obtained average for each sample of the prediction block.

[0129] Mode selection unit 202 provides a prediction block to residual generation unit 204. Residual generation unit 204 receives the original, uncoded version of the current block from video data memory 230 and the prediction block from mode selection unit 202. Residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block for the current block. In some examples, residual generation unit 204 may also determine the difference between sample values ​​in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). In some examples, one or more subtractor circuits performing binary subtraction may be used to form residual generation unit 204.

[0130] In the example where mode selection unit 202 divides a CU into PUs, each PU can be associated with a luma prediction unit and a corresponding chroma prediction unit. Video encoder 200 and video decoder 300 can support PUs of various sizes. As noted above, the size of a CU can refer to the size of the luma decoding block of the CU, and the size of a PU can refer to the size of the luma prediction unit of the PU. Assuming a particular CU size is 2Nx2N, video encoder 200 can support PU sizes of 2Nx2N or NxN for intra-frame prediction, and 2Nx2N, 2NxN, Nx2N, NxN, or similar symmetrical PU sizes for inter-frame prediction. Video encoder 200 and video decoder 300 can also support asymmetric segmentation for PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter-frame prediction.

[0131] In an example where the mode selection unit 202 does not further divide the CU into PUs, each CU can be associated with a luminance decoding block and a corresponding chrominance decoding block. As mentioned above, the size of the CU can refer to the size of the luminance decoding block of the CU. The video encoder 200 and the video decoder 300 can support CU sizes of 2Nx2N, 2NxN, or Nx2N.

[0132] For other video decoding techniques (such as intra-block copy mode decoding, affine mode decoding, and linear model (LM) mode decoding, the mode selection unit 202 generates a prediction block for the current block being encoded via a corresponding unit associated with the decoding technique. In some examples (such as palette mode decoding), the mode selection unit 202 may not generate a prediction block, but instead generate syntax elements indicating how the block should be reconstructed based on the selected palette. In such a mode, the mode selection unit 202 can provide these syntax elements to the entropy coding unit 220 for encoding.

[0133] As described above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. Then, the residual generation unit 204 generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

[0134] Transform processing unit 206 applies one or more transformations to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 can apply various transformations to the residual block to form the transform coefficient block. For example, transform processing unit 206 can apply a discrete cosine transform (DCT), direction transform, Karhunen-Loeve transform (KLT), or conceptually similar transformations to the residual block. In some examples, transform processing unit 206 can perform multiple transformations on the residual block, such as primary and secondary transformations (e.g., rotation transformations). In some examples, transform processing unit 206 does not apply any transformations to the residual block.

[0135] Quantization unit 208 can quantize the transform coefficients in the transform coefficient block to produce a quantized transform coefficient block. Quantization unit 208 can quantize the transform coefficients of the transform coefficient block based on the quantization parameter (QP) value associated with the current block. Video encoder 200 (e.g., via mode selection unit 202) can adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may cause information loss, and therefore, the quantized transform coefficients may have lower accuracy compared to the original transform coefficients produced by transform processing unit 206.

[0136] The inverse quantization unit 210 and the inverse transform processing unit 212 can apply inverse quantization and inverse transform, respectively, to the quantized transform coefficient block to reconstruct the residual block based on the transform coefficient block. The reconstruction unit 214 can generate a reconstructed block corresponding to the current block (although potentially with some degree of distortion) based on the reconstructed residual block and the prediction block generated by the mode selection unit 202. For example, the reconstruction unit 214 can add samples from the reconstructed residual block to corresponding samples from the prediction block generated by the mode selection unit 202 to generate the reconstructed block.

[0137] Filter unit 216 can perform one or more filter operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce block artifacts along the edges of the CU / TU. In some examples, the operations of filter unit 216 can be skipped. Additionally, filter unit 216 can be configured to perform one or more filter techniques of this disclosure, including neural network-based filter techniques.

[0138] Specifically, filter unit 216 can be configured to apply at least one of a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter to a decoded block of video data to form one or more filtered decoded blocks. Filter unit 216 can then refine the filtered decoded blocks by applying one or more scaling factors to the filtered decoded blocks. For example, filter unit 216 can test one or more dynamically determined or predefined scaling factors for the filtered decoded blocks. For example, filter unit 216 can apply one of the formulas (3)-(8) discussed above to refine the filtered decoded blocks. Mode selection unit 202 can perform a rate-distortion optimization (RDO) process to calculate the RDO value of each scaling factor among the tested scaling factors, which represents the distortion and bit rate of the various scaling factors. Mode selection unit 202 can select the scaling factor that produces the optimal RDO value.

[0139] Additionally, the mode selection unit 202 can provide the entropy coding unit 220 with data representing an enable flag for filter refinement and data representing a scaling factor. Therefore, the entropy coding unit 220 can encode the flag (with a value indicating whether filter refinement using a scaling factor is enabled) and the value representing the scaling factor for a specific unit of video data (e.g., slice, image, CTU, etc.). For example, the entropy coding unit 220 can encode data directly representing the scaling factor or an index value representing the scaling factor in a lookup table (e.g., an index table).

[0140] In some examples, filter unit 216 may also apply the offset value to the scaled, filtered, decoded block to form a refined, filtered, decoded block. Mode selection unit 202 may similarly provide data representing the offset value to entropy coding unit 220, which may encode the offset value, for example, directly or using an index value.

[0141] Entropy coding unit 220 can encode data representing scaling factors and offset values ​​(when applied) in picture headers, picture parameter sets (PPS), slice headers and / or adaptive parameter sets (APS) or other such high-level syntax structures.

[0142] The video encoder 200 stores the reconstructed blocks (potentially filtered and refined) in the DPB 218. For example, in an example where the operation of filter unit 216 is not performed, reconstruction unit 214 may store the reconstructed blocks in the DPB 218. In an example where the operation of filter unit 216 is performed, filter unit 216 may store the refined and filtered reconstructed blocks in the DPB 218. Motion estimation unit 222 and motion compensation unit 224 may retrieve a reference picture formed from the reconstructed (and potentially filtered) blocks from the DPB 218 to perform inter-frame prediction of blocks in subsequently encoded pictures. Additionally, intra-frame prediction unit 226 may use the reconstructed blocks of the current picture in the DPB 218 to perform intra-frame prediction of other blocks in the current picture.

[0143] Typically, entropy coding unit 220 can entropy-encode syntax elements received from other functional components of video encoder 200. For example, entropy coding unit 220 can entropy-encode quantized transform coefficient blocks from quantization unit 208. As another example, entropy coding unit 220 can entropy-encode predictive syntax elements (e.g., motion information for inter-frame prediction or intra-frame mode information for intra-frame prediction) from mode selection unit 202. Entropy coding unit 220 can perform one or more entropy coding operations on syntax elements, another example of video data, to generate entropy-encoded data. For example, entropy coding unit 220 can perform context-adaptive variable-length coding (CAVLC), CABAC, variable-to-variable (V2V) length decoding, syntax-based context-adaptive binary arithmetic decoding (SBAC), probabilistic interval partitioned entropy (PIPE) decoding, exponential Golomb coding, or another type of entropy coding operation on the data. In some examples, entropy coding unit 220 can operate in a bypass mode where syntax elements are not entropy-encoded.

[0144] The video encoder 200 can output a bitstream that includes entropy-encoded syntax elements required for reconstructing slices or blocks of images. Specifically, the entropy coding unit 220 can output a bitstream.

[0145] The above operations pertain to block descriptions. Such descriptions should be understood as operations applied to luma decoding blocks and / or chroma decoding blocks. As mentioned above, in some examples, the luma decoding block and chroma decoding block are the luma and chroma components of the CU. In some examples, the luma decoding block and chroma decoding block are the luma and chroma components of the PU.

[0146] In some examples, it is not necessary to repeat the operations performed for the luma decoding block for the chroma decoding block. As an example, it is not necessary to repeat the operations used to identify the motion vector (MV) and reference image for the luma decoding block to identify the MV and reference image for the chroma block. Instead, the MV for the luma decoding block can be scaled to determine the MV for the chroma block, and the reference image can be the same. As another example, the intra-frame prediction process can be the same for both the luma and chroma decoding blocks.

[0147] Video encoder 200 represents an example of a device configured to filter decoded video data, the device including: a memory configured to store video data; and one or more processors implemented in circuitry and configured to: decode blocks of video data to form decoded blocks; apply filters to the decoded blocks to form filtered blocks; multiply samples of the filtered blocks by a scaling factor to form thinned filtered blocks; and combine samples of the thinned filtered blocks with corresponding samples of the decoded blocks.

[0148] Figure 7 This is a block diagram illustrating an example video decoder 300 capable of performing the techniques described herein. Figure 7 This disclosure is provided for illustrative purposes and does not limit the scope of the technologies extensively illustrated and described herein. For illustrative purposes, this disclosure describes a video decoder 300 based on VVC (ITU-T H.266) and HEVC (ITU-T H.265) technologies. However, the technologies of this disclosure can be implemented by video decoding devices configured for other video decoding standards.

[0149] exist Figure 7 In the example, the video decoder 300 includes a decoded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, and a decoded picture buffer (DPB) 314. Any or all of the CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 can be implemented in one or more processors or in processing circuitry. For example, units of the video decoder 300 can be implemented as one or more circuit or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Furthermore, the video decoder 300 may include additional or alternative processors or processing circuitry to perform these and other functions.

[0150] The prediction processing unit 304 includes a motion compensation unit 316 and an intra-frame prediction unit 318. The prediction processing unit 304 may include additional units that perform predictions based on other prediction modes. As an example, the prediction processing unit 304 may include a palette unit, an intra-block copy unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, the video decoder 300 may include more, fewer, or different functional components.

[0151] CPB memory 320 can store video data to be decoded by components of video decoder 300, such as encoded video bitstreams. For example, it can be stored from computer-readable medium 110 ( Figure 1 The video data stored in the CPB memory 320 is obtained. The CPB memory 320 may include a CPB that stores encoded video data (e.g., syntax elements) from the encoded video bitstream. Additionally, the CPB memory 320 may store video data other than the syntax elements of the decoded picture, such as temporary data representing the output from various units of the video decoder 300. The DPB 314 typically stores decoded pictures, which the video decoder 300 may output and / or use as reference video data when decoding subsequent data or pictures of the encoded video bitstream. The CPB memory 320 and DPB 314 may be formed from any of a variety of memory devices, such as DRAM, including SDRAM, MRAM, RRAM, or other types of memory devices. The CPB memory 320 and DPB 314 may be provided by the same memory device or separate memory devices. In various examples, the CPB memory 320 may be on-chip with other components of the video decoder 300, or off-chip relative to those components.

[0152] Alternatively or concurrently, in some examples, the video decoder 300 can be derived from the memory 120 ( Figure 1 The decoded video data is retrieved. In other words, memory 120 can utilize CPB memory 320 to store data, as discussed above. Similarly, when some or all of the functions of video decoder 300 are implemented in software to be executed by the processing circuitry of video decoder 300, memory 120 can store instructions to be executed by video decoder 300.

[0153] It shows Figure 7 The various units shown help to understand the operations performed by the video decoder 300. This unit can be implemented as a fixed-function circuit, a programmable circuit, or a combination thereof. Similar to... Figure 6Fixed-function circuits refer to circuits that provide a specific function and are pre-configured regarding the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in terms of the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is typically immutable. In some examples, one or more units in this unit may be different circuit blocks (fixed-function or programmable), and in some examples, one or more units in this unit may be integrated circuits.

[0154] The video decoder 300 may include an ALU, EFU, digital circuitry, analog circuitry, and / or a programmable core formed according to programmable circuitry. In an example where the operation of the video decoder 300 is performed by software executing on the programmable circuitry, on-chip or off-chip memory may store instructions (e.g., object code) of the software received and executed by the video decoder 300.

[0155] Entropy decoding unit 302 can receive encoded video data from the CPB and perform entropy decoding on the video data to regenerate syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filter unit 312 can generate decoded video data based on syntax elements extracted from the bitstream.

[0156] Typically, the video decoder 300 reconstructs the image on a block-by-block basis. The video decoder 300 can perform the reconstruction operation on each block individually (where the block currently being reconstructed (i.e., decoded) can be referred to as the "current block").

[0157] Entropy decoding unit 302 can entropy decode the syntax elements of the quantized transform coefficients that define the quantized transform coefficient block, as well as transform information such as quantization parameters (QP) and / or transform mode indications. Inverse quantization unit 306 can use the QP associated with the quantized transform coefficient block to determine the degree of quantization, and similarly, determine the degree of inverse quantization to be applied by inverse quantization unit 306. Inverse quantization unit 306 can, for example, perform a bitwise left shift operation to inverse quantize the quantized transform coefficients. Inverse quantization unit 306 can thus form a transform coefficient block including the transform coefficients.

[0158] After the inverse quantization unit 306 forms the transform coefficient block, the inverse transform processing unit 308 can apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, the inverse transform processing unit 308 can apply the inverse DCT, inverse integer transform, inverse Karhunen-Loeve transform (KLT), inverse rotation transform, inverse direction transform, or another inverse transform to the transform coefficient block.

[0159] Furthermore, the prediction processing unit 304 generates a prediction block based on the prediction information syntax elements entropy-decoded by the entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is inter-frame predicted, the motion compensation unit 316 can generate the prediction block. In this case, the prediction information syntax elements may indicate the reference picture from which the reference block is to be retrieved in the DPB 314, and a motion vector identifying the position of the reference block in the reference picture relative to the position of the current block in the current picture. The motion compensation unit 316 can typically be configured with respect to the motion compensation unit 224 ( Figure 6 The method described is essentially the same as the method used to perform the inter-frame prediction process.

[0160] As another example, if the prediction information syntax element indicates that the current block is intra-predicted, then intra-prediction unit 318 can generate a prediction block according to the intra-prediction mode indicated by the prediction information syntax element. Again, intra-prediction unit 318 can typically be configured with respect to intra-prediction unit 226 ( Figure 6 The intra-prediction process is performed in a manner substantially similar to that described above. The intra-prediction unit 318 can retrieve data from neighboring samples of the current block from the DPB 314.

[0161] Reconstruction unit 310 can reconstruct the current block using the prediction block and the residual block. For example, reconstruction unit 310 can reconstruct the current block by adding the samples of the residual block to the corresponding samples of the prediction block.

[0162] Filter unit 312 can perform one or more filter operations on the reconstructed block. For example, filter unit 312 can perform a deblocking operation to reduce block artifacts along the edges of the reconstructed block. It is not necessary to perform the operations of filter unit 312 in all examples. Additionally, filter unit 312 can be configured to perform one or more filter techniques of this disclosure, including neural network-based filter techniques.

[0163] Specifically, initially, the entropy decoding unit 302 can decode the value of a flag indicating whether a thinning filter should be applied. If a thinning filter is to be applied, the entropy decoding unit 302 can perform entropy decoding on data representing one or more scaling factors. For example, the entropy decoding unit 302 can directly perform entropy decoding on the scaling factor or on the index value corresponding to the scaling factor in a lookup table (e.g., an index table or a mapping table). The entropy decoding unit 302 can decode the flag and scaling factor from slice headers, image headers, image parameter sets (PPS), adaptive parameter sets (APS), or other high-level syntax structures.

[0164] Filter unit 312 can be configured to apply at least one of a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter to a decoded block of video data to form one or more filtered decoded blocks. Then, assuming refinement filtering is enabled, filter unit 312 can refine the filtered decoded blocks by applying one or more scaling factors to the filtered decoded blocks. For example, filter unit 312 can apply one of the formulas (3)-(8) discussed above to refine the filtered decoded blocks.

[0165] In some examples, filter unit 312 may also apply offset values ​​to scaled, filtered, decoded blocks to form refined, filtered, decoded blocks. Entropy decoding unit 302 may decode data representing offset values ​​(e.g., directly or by index values ​​of a lookup table (i.e., an index table or mapping table)).

[0166] Entropy decoding unit 302 can decode data representing scaling factors and offset values ​​(when applied) from picture headers, picture parameter sets (PPS), slice headers and / or adaptive parameter sets (APS) or other such high-level syntax structures.

[0167] The video decoder 300 can store the reconstructed blocks in the DPB 314. For example, in an example where the operation of the filter unit 312 is not performed, the reconstruction unit 310 can store the reconstructed blocks in the DPB 314. In an example where the operation of the filter unit 312 is performed, the filter unit 312 can store the filtered reconstructed blocks in the DPB 314. As discussed above, the DPB 314 can provide reference information (such as samples of the current image for intra-frame prediction and samples of previously decoded images for subsequent motion compensation) to the prediction processing unit 304. Furthermore, the video decoder 300 can output decoded images (e.g., decoded video) from the DPB 314 for use in applications such as... Figure 1The subsequent presentation on the display device 118.

[0168] In this manner, the video decoder 300 represents an example of a device for filtering decoded video data, the device comprising: a memory configured to store video data; and one or more processors implemented in circuitry and configured to: decode blocks of video data to form decoded blocks; apply filters to the decoded blocks to form filtered blocks; multiply samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combine samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0169] Figure 8 This is a flowchart illustrating an example method for encoding a current block according to the technology of this disclosure. The current block may include the current CU. Although regarding video encoder 200 ( Figure 1 and 6 The description is provided, but it should be understood that other devices can be configured to perform the same actions. Figure 6 Similar to the method.

[0170] In this example, the video encoder 200 initially predicts the current block (350). For example, the video encoder 200 may form a prediction block for the current block. Then, the video encoder 200 may compute a residual block for the current block (352). To compute the residual block, the video encoder 200 may compute the difference between the original unencoded block and the prediction block for the current block. Then, the video encoder 200 may transform the residual block and quantize the transform coefficients of the residual block (354). Next, the video encoder 200 may scan the quantized transform coefficients of the residual block (356). During or after the scan, the video encoder 200 may entropy encode the transform coefficients (358). For example, the video encoder 200 may use, for example, CAVLC or CABAC to entropy encode the transform coefficients. Then, the video encoder 200 may output the entropy-encoded data of the block (360).

[0171] The video encoder 200 can also decode the current block after encoding it, using the decoded version of the current block as reference data for subsequent decoded data (e.g., in inter-frame or intra-frame prediction modes). Therefore, the video encoder 200 can inversely quantize and inversely transform the coefficients to regenerate the residual block (362). The video encoder 200 can combine the residual block with the prediction block to form a decoded block (364). According to the techniques of this disclosure, the video encoder 200 can filter and refine the decoded block (366). The video encoder 200 can then store the filtered, decoded block in the DPB 218 (368).

[0172] In this way, Figure 8 The method represents an example of a method for filtering decoded video data, the method comprising: decoding blocks of video data to form decoded blocks; applying a filter to the decoded blocks to form filtered blocks; multiplying samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combining samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0173] Figure 9 This is a flowchart illustrating an example method for decoding a current block of video data according to the technology of this disclosure. The current block may include the current CU. Although regarding the video decoder 300 ( Figure 1 and 7 The description is provided, but it should be understood that other devices can be configured to perform the same actions. Figure 7 Similar to the method.

[0174] The video decoder 300 can receive entropy-encoded data for the current block (such as entropy-encoded prediction information and entropy-encoded data for the transform coefficients of the residual block corresponding to the current block) (370). The video decoder 300 can entropy decode the entropy-encoded data to determine the prediction information for the current block and regenerate the transform coefficients of the residual block (372). The video decoder 300 can predict the current block, for example, using an intra-frame or inter-frame prediction mode indicated by the prediction information for the current block (374), to compute a prediction block for the current block. The video decoder 300 can then inverse scan the regenerated transform coefficients (376) to create a block of quantized transform coefficients. The video decoder 300 can then inverse quantize the transform coefficients and apply the inverse transform to the transform coefficients to produce a residual block (378). Finally, the video decoder 300 can decode the current block by combining the prediction block and the residual block (380).

[0175] In this way, Figure 9 The method represents an example of a method for filtering decoded video data, the method comprising: decoding blocks of video data to form decoded blocks; applying a filter to the decoded blocks to form filtered blocks; multiplying samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combining samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0176] Figure 10 This is a flowchart illustrating an example method for encoding video data and filtering decoded video data according to the technology of this disclosure. Although Figure 10 The method is about Figure 1 and 6 This is interpreted by the video encoder 200, but other video encoding devices can be configured to perform this method or a similar method.

[0177] Initially, the video encoder 200 can encode blocks of video data and subsequently decode blocks of video data (390). Then, the video encoder 200 can filter the decoded blocks, for example, using one or more of the following: a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter (392).

[0178] Then, the video encoder 200 can use rate-distortion optimization (RDO) to test multiple dynamically determined or predefined scaling factors (394). For example, the video encoder 200 can apply each of the scaling factors to a filtered, decoded block and compare the resulting refined, filtered, decoded block with the original undecoded block to calculate an RDO value. Assuming that at least one of the scaling factors results in a refined, filtered, decoded block that produces better RDO performance than an unrefined, filtered, decoded block, the video encoder 200 can determine to enable refinement filtering according to the techniques of this disclosure. In some examples, the RDO process may also include a test offset applied in conjunction with the scaling factors. In some examples, the video encoder 200 may test multiple different filters and / or multiple different scaling factors (potentially with offsets), as described above.

[0179] The video encoder 200 can then determine a scaling factor (or set of scaling factors, potentially with an offset) among the scaling factors that has the optimal RDO value (396). The video encoder 200 can then apply the determined scaling factor (or set of scaling factors and / or offsets) to the filtered, decoded block (398). The video encoder 200 can then, for example, combine the refined, filtered block with the original decoded block on a sample-by-sample basis (400), and store the resulting block in DPB 218 (402).

[0180] Furthermore, the video encoder 200 can encode data representing the selected scaling factor, for example, in a slice header, picture header, picture parameter set (PPS), adaptive parameter set (APS), or other such high-level syntax structures (404). Additionally, the video encoder 200 can encode the value of a flag enabling filter refinement, as discussed above. In some examples, the video encoder 200 can encode data directly representing the scaling factor and / or offset, while in other examples, the video encoder 200 can encode the index value corresponding to the selected scaling factor and / or offset in the corresponding lookup table.

[0181] In this way, Figure 10 The method represents an example of a method for filtering decoded video data, the method comprising: decoding blocks of video data to form decoded blocks; applying a filter to the decoded blocks to form filtered blocks; multiplying samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combining samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0182] Figure 11 This is a flowchart illustrating an example method for filtering decoded video data according to the technology of this disclosure. Although Figure 11 The method is about Figure 1 and 7 This is interpreted by the video decoder 300, but other video decoding devices can be configured to perform this method or a similar method.

[0183] Initially, the video decoder 300 can encode blocks of video data and subsequently decode blocks of video data (410). Then, the video decoder 300 can filter the decoded blocks, for example, using one or more of the following: a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter (412).

[0184] Then, the video decoder 300 can decode the data with thinning filtering enabled according to the techniques of this disclosure (414). That is, the video decoder 300 can decode the syntax element indicating whether thinning filtering is enabled, and determine the value of the syntax element indicating that thinning filtering is enabled. Then, the video decoder 300 can decode the data indicating the scaling factor (416). For example, the data can directly represent the scaling factor, or the index value of a lookup table mapping index values ​​to scaling factors. In some examples, the video decoder 300 can decode the data representing a set of scaling factors and / or offsets. This data can be included in a slice header, picture header, picture parameter set (PPS), adaptive parameter set (APS), or other high-level syntax structures. The video decoder 300 can apply the determined scaling factor (or the set of scaling factors and / or offsets) to the filtered and decoded block (418) to produce a thinned and filtered block. The video decoder 300 can then combine the refined, filtered blocks with the original decoded blocks, for example, on a sample-by-sample basis (420), and store the resulting blocks in the DPB 314 (422). The video decoder 300 can also output the resulting video data from the DPB 314 for display, for example, to a display device (424).

[0185] In this way, Figure 11 The method represents an example of a method for filtering decoded video data, the method comprising: decoding blocks of video data to form decoded blocks; applying a filter to the decoded blocks to form filtered blocks; multiplying samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combining samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0186] Some example technologies of this disclosure are outlined in the following terms:

[0187] Clause 1: A method for decoding video data, the method comprising: reconstructing the video data; and applying a filter to the reconstructed video data, wherein applying the filter comprises: refining the filter output residual and adding the refined filter output residual to update the input samples of the filter.

[0188] Clause 2: The method according to Clause 1, wherein refining the filter output residual includes multiplying the filter output residual by a scaling factor.

[0189] Clause 3: The method according to Clause 1, wherein refining the filter output residual comprises: multiplying the filter output residual by a scaling factor and adding an offset.

[0190] Clause 4: The method according to any one of Clauses 1-3, wherein the filter is one of a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter.

[0191] Clause 5: The method according to any combination of Clauses 1-5, wherein refining the filter output residuals comprises: refining the filter output residuals of a plurality of filters.

[0192] Clause 6: The method according to any one of Clauses 1-5, wherein decoding includes encoding.

[0193] Clause 7: The method according to any one of Clauses 1-5, wherein decoding includes decoding.

[0194] Clause 8: An apparatus for decoding video data, said apparatus comprising one or more units for performing the method according to any one of Clauses 1-7.

[0195] Clause 9: The device according to Clause 8, wherein the one or more units include one or more processors implemented in a circuit.

[0196] Clause 10: The device according to any one of Clauses 8 and 9 further includes: a memory configured to store the video data.

[0197] Clause 11: The device according to any one of Clauses 8-10 further includes: a display configured to display decoded video data.

[0198] Clause 12: The device according to any one of Clauses 8-11, wherein the device includes one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0199] Clause 13: The device according to any one of Clauses 8-12, wherein the device includes a video decoder.

[0200] Clause 14: The device according to any one of Clauses 8-13, wherein the device includes a video encoder.

[0201] Clause 15: A computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to perform the method according to any one of Clauses 1-7.

[0202] Clause 16: A method for filtering decoded video data, the method comprising: decoding blocks of video data to form decoded blocks; applying a filter to the decoded blocks to form filtered blocks; multiplying samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combining samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0203] Clause 17: The method according to Clause 16 further includes: decoding the value of the syntax element representing the scaling factor.

[0204] Clause 18: The method according to Clause 17, wherein decoding the value of the syntax element comprises: decoding an image header of an image including the block, the image header including the syntax element representing the scaling factor.

[0205] Clause 19: The method according to Clause 17, wherein decoding the value of the syntax element comprises decoding at least one of: a slice header of a slice comprising the block, and the slice header comprising the syntax element; an adaptive parameter set (APS) for the slice, and the APS comprising the syntax element; or a block header of the block, and the block header comprising the syntax element.

[0206] Clause 20: The method according to any one of Clauses 17-19, wherein decoding the value of the syntax element comprises: decoding an index value mapped to the scaling factor in an index table, the method further comprising: using the index table to determine the scaling factor based on the index value.

[0207] Clause 21: The method according to any one of Clauses 17-19 further includes: inverse quantizing the value by means of an integer value having a predefined bit precision.

[0208] Clause 22: The method according to any one of Clauses 16-21, wherein the scaling factor includes a first scaling factor of a plurality of scaling factors, and wherein multiplying the sample of the filtered block by the scaling factor includes multiplying the sample of the filtered block by each of the scaling factors of the plurality of scaling factors.

[0209] Clause 23: The method according to any one of Clauses 16-22, wherein multiplying the sample of the filtered block by the scaling factor to form the refined filtered block further comprises: adding the offset to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

[0210] Clause 24: The method according to any one of Clauses 16-22, wherein multiplying the sample of the filtered block by the scaling factor to form the refined filtered block further comprises: determining an offset value; applying a bitwise shift to the offset value to form a shifted offset value; and adding the shifted offset value to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

[0211] Clause 25: The method according to any one of Clauses 16-24, wherein applying the filter comprises: applying at least one of a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter.

[0212] Clause 26: The method according to any one of Clauses 16-25 further comprises: determining, before multiplying the sample of the filtered block by the scaling factor, that a syntax element indicating whether to refine the filtered block has a value indicating that the filtered block should be refined.

[0213] Clause 27: The method according to Clause 26 further comprises: decoding the value of the syntax element indicating whether the filtered block is refined in at least one of the following: a slice header including a slice of the block, an image header including an image of the block, or an adaptive parameter set (APS) corresponding to the slice including the block.

[0214] Clause 28: The method according to any one of Clauses 16-27, wherein the filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, the method further comprising: applying a second filter to the decoded block to form a second filtered block; and multiplying samples of the second filtered block by a second scaling factor to form a second refined filtered block, wherein combining the samples of the first refined filtered block with the corresponding samples of the decoded block comprises: combining the samples of the first refined filtered block and the samples of the second refined filtered block with the corresponding samples of the decoded block.

[0215] Clause 29: The method according to any one of Clauses 16-27, wherein the filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, wherein applying the first filter to the decoded block includes: applying each of a plurality of filters including the first filter to the decoded block to form a corresponding plurality of filtered blocks including the first filtered block; and wherein multiplying the sample of the first filtered block by the first scaling factor includes: multiplying the sample of the corresponding plurality of filtered blocks including the first filtered block by a corresponding plurality of scaling factors including the first scaling factor to form a plurality of refined filtered blocks including the first refined filtered block, wherein combining the sample of the first refined filtered block with the corresponding sample of the decoded block includes: combining the sample of each of the plurality of refined filtered blocks with the corresponding sample of the decoded block.

[0216] Clause 30: The method according to any one of Clauses 16-29 further includes: encoding the current block before decoding the current block.

[0217] Clause 31: An apparatus for filtering decoded video data, the apparatus comprising: a memory configured to store video data; and one or more processors implemented in a circuit and configured to: decode blocks of video data to form decoded blocks; apply a filter to the decoded blocks to form filtered blocks; multiply samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combine samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0218] Clause 32: The device according to Clause 31, wherein the one or more processors are further configured to: decode the value of the syntax element representing the scaling factor.

[0219] Clause 33: The device according to Clause 32, wherein, in order to decode the value of the syntax element, the one or more processors are configured to: decode an image header of an image including the block, the image header including the syntax element representing the scaling factor.

[0220] Clause 34: The apparatus according to Clause 32, wherein, in order to decode the value of the syntax element, the one or more processors are configured to decode at least one of: a slice header comprising a slice of the block, and the slice header comprising the syntax element; an adaptive parameter set (APS) for the slice, and the APS comprising the syntax element; or a block header of the block, and the block header comprising the syntax element.

[0221] Clause 35: The device according to any one of Clauses 32-34, wherein, in order to decode the value of the syntax element, the one or more processors are configured to: decode an index value mapped to the scaling factor in an index table; and use the index table to determine the scaling factor based on the index value.

[0222] Clause 36: A device according to any one of Clauses 32-35, wherein the one or more processors are further configured to: inversely quantize the value by means of an integer value having a predefined bit precision.

[0223] Clause 37: An apparatus according to any one of Clauses 31-36, wherein the scaling factor includes a first scaling factor of a plurality of scaling factors, and wherein the one or more processors are configured to multiply the sample of the filtered block by each of the scaling factors of the plurality of scaling factors.

[0224] Clause 38: The apparatus according to any one of Clauses 31-37, wherein the one or more processors are configured to: add the offset to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

[0225] Clause 39: The apparatus according to any one of Clauses 31-37, wherein, in order to form the refined filtered block, the one or more processors are configured to: determine an offset value; apply a bitwise shift to the offset value to form a shifted offset value; and add the shifted offset value to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

[0226] Clause 40: The device according to any one of Clauses 31-39, wherein, in order to apply the filter, the one or more processors are configured to apply at least one of a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter.

[0227] Clause 41: The apparatus according to any one of Clauses 31-40, wherein the one or more processors are further configured to: determine, before multiplying the sample of the filtered block by the scaling factor, that a syntax element indicating whether to refine the filtered block has a value indicating that the filtered block should be refined.

[0228] Clause 42: The apparatus according to Clause 41, wherein the one or more processors are further configured to decode the value of the syntax element indicating whether the filtered block is refined in at least one of the following: a slice header including a slice of the block, an image header including an image of the block, or an adaptive parameter set (APS) corresponding to the slice including the block.

[0229] Clause 43: An apparatus according to any one of Clauses 31-42, wherein the filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, and wherein the one or more processors are further configured to: apply a second filter to the decoded block to form a second filtered block; and multiply a sample of the second filtered block by a second scaling factor to form a second refined filtered block, wherein, in order to combine the sample of the first refined filtered block with the corresponding sample of the decoded block, the one or more processors are configured to: combine the sample of the first refined filtered block and the sample of the second refined filtered block with the corresponding sample of the decoded block.

[0230] Clause 44: An apparatus according to any one of Clauses 31-42, wherein the filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, and wherein the one or more processors are configured to: apply each of a plurality of filters including the first filter to the decoded block to form a corresponding plurality of filtered blocks including the first filtered block; and multiply the samples of the corresponding plurality of filtered blocks including the first filtered block by a corresponding plurality of scaling factors including the first scaling factor to form a plurality of refined filtered blocks including the first refined filtered block, wherein, in order to combine the samples of the first refined filtered block with the corresponding samples of the decoded block, the one or more processors are configured to: combine the samples of each of the plurality of refined filtered blocks with the corresponding samples of the decoded block.

[0231] Clause 45: A device according to any one of Clauses 31-44, wherein the one or more processors are further configured to encode the current block before decoding the current block.

[0232] Clause 46: The device according to any one of Clauses 31-45 further includes: a display configured to display decoded video data.

[0233] Clause 47: The device pursuant to any one of Clauses 31-46, wherein the device includes one or more of a camera, computer, mobile device, broadcast receiver device or set-top box.

[0234] Clause 48: A computer-readable storage medium having instructions stored thereon, which, when executed, cause a processor to: decode blocks of video data to form decoded blocks; apply a filter to the decoded blocks to form filtered blocks; multiply samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combine samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0235] Clause 49: The computer-readable storage medium according to Clause 48 further includes instructions that cause the processor to perform the following operation: decode the value of the syntax element representing the scaling factor.

[0236] Clause 50: A computer-readable storage medium according to Clause 49, wherein the instructions causing the processor to decode the value of the syntax element include instructions causing the processor to: decode an image header of an image including the block, the image header including the syntax element representing the scaling factor.

[0237] Clause 51: A computer-readable storage medium according to Clause 49, wherein the instructions causing the processor to decode the value of the syntax element include instructions causing the processor to decode at least one of: a slice header comprising a slice of the block, wherein the slice header comprises the syntax element; an adaptive parameter set (APS) for the slice, wherein the APS comprises the syntax element; or a block header comprising the block, wherein the block header comprises the syntax element.

[0238] Clause 52: A computer-readable storage medium according to any one of Clauses 49-51, wherein the instructions causing the processor to decode the value of the syntax element include instructions causing the processor to: decode an index value mapped to the scaling factor in an index table, the method further comprising: using the index table to determine the scaling factor based on the index value.

[0239] Clause 53: A computer-readable storage medium according to any one of Clauses 49-52 further includes instructions that cause the processor to perform: inverse quantization of the value by an integer value having a predefined bit precision.

[0240] Clause 54: A computer-readable storage medium according to any one of Clauses 48-53, wherein the scaling factor includes a first scaling factor of a plurality of scaling factors, and wherein the instruction causing the processor to multiply the sample of the filtered block by the scaling factor includes instructions causing the processor to perform: multiplying the sample of the filtered block by each of the scaling factors of the plurality of scaling factors.

[0241] Clause 55: A computer-readable storage medium according to any one of Clauses 48-54, wherein the instruction causing the processor to multiply the sample of the filtered block by the scaling factor to form the refined filtered block further comprises instructions causing the processor to perform the following operation: adding the offset to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

[0242] Clause 56: A computer-readable storage medium according to any one of Clauses 48-55, wherein the instruction causing the processor to multiply the sample of the filtered block by the scaling factor to form the refined filtered block further comprises instructions causing the processor to: determine an offset value; apply a bitwise shift to the offset value to form a shifted offset value; and add the shifted offset value to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

[0243] Clause 57: A computer-readable storage medium according to any one of Clauses 48-56, wherein the instructions causing the processor to apply the filter include instructions causing the processor to perform at least one of a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter.

[0244] Clause 58: A computer-readable storage medium according to any one of Clauses 48-57 further includes instructions that cause the processor to perform the following operation: before multiplying the sample of the filtered block by the scaling factor, determining that a syntax element indicating whether to refine the filtered block has a value indicating that the filtered block should be refined.

[0245] Clause 59: A computer-readable storage medium according to any one of Clauses 48-58 further includes instructions that cause the processor to decode the value of the syntax element indicating whether the filtered block is refined in at least one of the following: a slice header including a slice of the block, an image header including an image of the block, or an adaptive parameter set (APS) corresponding to the slice including the block.

[0246] Clause 60: A computer-readable storage medium according to any one of Clauses 48-59, wherein the filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, the computer-readable storage medium further comprising instructions causing the processor to: apply a second filter to the decoded block to form a second filtered block; and multiply samples of the second filtered block by a second scaling factor to form a second refined filtered block, wherein the instructions causing the processor to combine the samples of the first refined filtered block with the corresponding samples of the decoded block include instructions causing the processor to: combine the samples of the first refined filtered block and the samples of the second refined filtered block with the corresponding samples of the decoded block.

[0247] Clause 61: A computer-readable storage medium according to any one of Clauses 48-59, wherein the filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, wherein the instruction causing the processor to apply the first filter to the decoded block includes instructions causing the processor to: apply each of a plurality of filters including the first filter to the decoded block to form a corresponding plurality of filtered blocks including the first filtered block; and wherein the processor is caused to apply the first filtered block to the first filtered block. The instruction to multiply the sample by the first scaling factor includes instructions to cause the processor to perform the following operations: multiply the sample of the corresponding plurality of filtered blocks including the first filtered block by the corresponding plurality of scaling factors including the first scaling factor to form a plurality of refined filtered blocks including the first refined filtered block, wherein the instruction to cause the processor to combine the sample of the first refined filtered block with the corresponding sample of the decoded block includes instructions to cause the processor to perform the following operations: combine the sample of each of the plurality of refined filtered blocks with the corresponding sample of the decoded block.

[0248] Clause 62: A computer-readable storage medium according to any one of Clauses 48-61 further includes instructions that cause the processor to encode the current block before decoding it.

[0249] Clause 63: An apparatus for filtering decoded video data, the apparatus comprising: a unit for decoding blocks of video data to form decoded blocks; a unit for applying a filter to the decoded blocks to form filtered blocks; a unit for multiplying samples of the filtered blocks by a scaling factor to form refined filtered blocks; and a unit for combining samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0250] Clause 64: A method for filtering decoded video data, the method comprising: decoding blocks of video data to form decoded blocks; applying a filter to the decoded blocks to form filtered blocks; multiplying samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combining samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0251] Clause 65: The method according to Clause 64 further includes: decoding the value of the syntax element representing the scaling factor.

[0252] Clause 66: The method according to Clause 65, wherein decoding the value of the syntax element comprises: decoding an image header of an image including the block, the image header including the syntax element representing the scaling factor.

[0253] Clause 67: The method according to Clause 65, wherein decoding the value of the syntax element comprises decoding at least one of: a slice header of a slice comprising the block, and the slice header comprising the syntax element; an adaptive parameter set (APS) for the slice, and the APS comprising the syntax element; or a block header of the block, and the block header comprising the syntax element.

[0254] Clause 68: The method according to Clause 65, wherein decoding the value of the syntax element comprises: decoding an index value mapped to the scaling factor in an index table, the method further comprising: using the index table to determine the scaling factor based on the index value.

[0255] Clause 69: The method according to Clause 65 further includes: inverse quantizing the value using an integer value having a predefined bit precision.

[0256] Clause 70: The method according to Clause 64, wherein the scaling factor includes a first scaling factor of a plurality of scaling factors, and wherein multiplying the sample of the filtered block by the scaling factor includes: multiplying the sample of the filtered block by each of the scaling factors of the plurality of scaling factors.

[0257] Clause 71: The method according to Clause 64, wherein multiplying the sample of the filtered block by the scaling factor to form the refined filtered block further comprises: adding the offset to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

[0258] Clause 72: The method according to Clause 64, wherein multiplying the sample of the filtered block by the scaling factor to form the refined filtered block further comprises: determining an offset value; applying a bitwise shift to the offset value to form a shifted offset value; and adding the shifted offset value to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

[0259] Clause 73: The method according to Clause 64, wherein applying the filter includes: applying at least one of a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter.

[0260] Clause 74: The method according to Clause 64 further comprises: determining, before multiplying the sample of the filtered block by the scaling factor, that a syntax element indicating whether to refine the filtered block has a value indicating that the filtered block should be refined.

[0261] Clause 75: The method according to Clause 74 further includes decoding the value of the syntax element indicating whether the filtered block is refined in at least one of the following: a slice header including a slice of the block, an image header including an image of the block, or an adaptive parameter set (APS) corresponding to the slice including the block.

[0262] Clause 76: The method according to Clause 64, wherein the filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, the method further comprising: applying a second filter to the decoded block to form a second filtered block; and multiplying samples of the second filtered block by a second scaling factor to form a second refined filtered block, wherein combining the samples of the first refined filtered block with the corresponding samples of the decoded block comprises: combining the samples of the first refined filtered block and the samples of the second refined filtered block with the corresponding samples of the decoded block.

[0263] Clause 77: The method according to Clause 64, wherein the filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, wherein applying the first filter to the decoded block includes: applying each of a plurality of filters including the first filter to the decoded block to form a corresponding plurality of filtered blocks including the first filtered block; and wherein multiplying the sample of the first filtered block by the first scaling factor includes: multiplying the sample of the corresponding plurality of filtered blocks including the first filtered block by a corresponding plurality of scaling factors including the first scaling factor to form a plurality of refined filtered blocks including the first refined filtered block, wherein combining the sample of the first refined filtered block with the corresponding sample of the decoded block includes: combining the sample of each of the plurality of refined filtered blocks with the corresponding sample of the decoded block.

[0264] Clause 78: The method according to Clause 64 further includes: encoding the current block before decoding the current block.

[0265] Clause 79: An apparatus for filtering decoded video data, the apparatus comprising: a memory configured to store the video data; and one or more processors implemented in a circuit and configured to: decode blocks of the video data to form decoded blocks; apply a filter to the decoded blocks to form filtered blocks; multiply samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combine samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0266] Clause 80: The device according to Clause 79, wherein the one or more processors are further configured to: decode the value of the syntax element representing the scaling factor.

[0267] Clause 81: The apparatus according to Clause 80, wherein, in order to decode the value of the syntax element, the one or more processors are configured to: decode an image header of an image including the block, the image header including the syntax element representing the scaling factor.

[0268] Clause 82: The apparatus according to Clause 80, wherein, in order to decode the value of the syntax element, the one or more processors are configured to decode at least one of: a slice header comprising a slice of the block, and the slice header comprising the syntax element; an adaptive parameter set (APS) for the slice, and the APS comprising the syntax element; or a block header comprising the block, and the block header comprising the syntax element.

[0269] Clause 83: The apparatus according to Clause 80, wherein, in order to decode the value of the syntax element, the one or more processors are configured to: decode an index value mapped to the scaling factor in an index table; and use the index table to determine the scaling factor based on the index value.

[0270] Clause 84: The device according to Clause 80, wherein the one or more processors are further configured to: inversely quantize the value by means of an integer value having a predefined bit precision.

[0271] Clause 85: The apparatus according to Clause 79, wherein the scaling factor includes a first scaling factor of a plurality of scaling factors, and wherein the one or more processors are configured to multiply the sample of the filtered block by each of the scaling factors of the plurality of scaling factors.

[0272] Clause 86: The apparatus according to Clause 79, wherein the one or more processors are configured to: add the offset to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

[0273] Clause 87: The apparatus according to Clause 79, wherein, in order to form the refined filtered block, the one or more processors are configured to: determine an offset value; apply a bitwise shift to the offset value to form a shifted offset value; and add the shifted offset value to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

[0274] Clause 88: The device according to Clause 79, wherein, in order to apply the filter, the one or more processors are configured to apply at least one of a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter.

[0275] Clause 89: The apparatus according to Clause 79, wherein the one or more processors are further configured to: determine, before multiplying the sample of the filtered block by the scaling factor, that a syntax element indicating whether to refine the filtered block has a value indicating that the filtered block should be refined.

[0276] Clause 90: The apparatus according to Clause 89, wherein the one or more processors are further configured to decode the value of the syntax element indicating whether the filtered block is refined in at least one of: a slice header including a slice of the block, an image header including an image of the block, or an adaptive parameter set (APS) corresponding to the slice including the block.

[0277] Clause 91: The apparatus according to Clause 79, wherein the filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, and wherein the one or more processors are further configured to: apply a second filter to the decoded block to form a second filtered block; and multiply a sample of the second filtered block by a second scaling factor to form a second refined filtered block, wherein, in order to combine the sample of the first refined filtered block with the corresponding sample of the decoded block, the one or more processors are configured to: combine the sample of the first refined filtered block and the sample of the second refined filtered block with the corresponding sample of the decoded block.

[0278] Clause 92: The apparatus according to Clause 79, wherein the filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, and wherein the one or more processors are configured to: apply each of a plurality of filters including the first filter to the decoded block to form a corresponding plurality of filtered blocks including the first filtered block; and multiply the samples of the corresponding plurality of filtered blocks including the first filtered block by a corresponding plurality of scaling factors including the first scaling factor to form a plurality of refined filtered blocks including the first refined filtered block, wherein, in order to combine the samples of the first refined filtered block with the corresponding samples of the decoded block, the one or more processors are configured to: combine the samples of each of the plurality of refined filtered blocks with the corresponding samples of the decoded block.

[0279] Clause 93: The apparatus according to Clause 79, wherein the one or more processors are further configured to encode the current block before decoding the current block.

[0280] Clause 94: The device according to Clause 79 further includes: a display configured to display decoded video data.

[0281] Clause 95: The device as described in Clause 79, wherein the device includes one or more of a camera, computer, mobile device, broadcast receiver device or set-top box.

[0282] Clause 96: A computer-readable storage medium having instructions stored thereon, which, when executed, cause a processor to: decode blocks of video data to form decoded blocks; apply a filter to the decoded blocks to form filtered blocks; multiply samples of the filtered blocks by a scaling factor to form refined filtered blocks; and combine samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0283] Clause 97: The computer-readable storage medium according to Clause 96 further includes instructions that cause the processor to decode the value of the syntax element representing the scaling factor.

[0284] Clause 98: A computer-readable storage medium according to Clause 97, wherein the instructions causing the processor to decode the value of the syntax element include instructions causing the processor to: decode an image header of an image including the block, the image header including the syntax element representing the scaling factor.

[0285] Clause 99: A computer-readable storage medium according to Clause 97, wherein the instructions causing the processor to decode the value of the syntax element include instructions causing the processor to decode at least one of: a slice header comprising a slice of the block, wherein the slice header comprises the syntax element; an adaptive parameter set (APS) for the slice, wherein the APS comprises the syntax element; or a block header comprising the block, wherein the block header comprises the syntax element.

[0286] Clause 100: A computer-readable storage medium according to Clause 97, wherein the instructions causing the processor to decode the value of the syntax element include instructions causing the processor to: decode an index value mapped to the scaling factor in an index table, the method further comprising: using the index table to determine the scaling factor based on the index value.

[0287] Clause 101: The computer-readable storage medium according to Clause 97 further includes instructions that cause the processor to perform: inverse quantization of the value by an integer value having a predefined bit precision.

[0288] Clause 102: A computer-readable storage medium according to Clause 96, wherein the scaling factor includes a first scaling factor of a plurality of scaling factors, and wherein the instruction causing the processor to multiply the sample of the filtered block by the scaling factor includes instructions causing the processor to perform: multiply the sample of the filtered block by each of the scaling factors of the plurality of scaling factors.

[0289] Clause 103: A computer-readable storage medium according to Clause 96, wherein the instruction causing the processor to multiply the sample of the filtered block by the scaling factor to form the refined filtered block further includes instructions causing the processor to perform the following operation: adding the offset to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

[0290] Clause 104: A computer-readable storage medium according to Clause 96, wherein the instruction causing the processor to multiply the sample of the filtered block by the scaling factor to form the refined filtered block further includes instructions causing the processor to: determine an offset value; apply a bitwise shift to the offset value to form a shifted offset value; and add the shifted offset value to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

[0291] Clause 105: A computer-readable storage medium according to Clause 96, wherein the instructions causing the processor to apply the filter include instructions causing the processor to perform at least one of a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter.

[0292] Clause 106: The computer-readable storage medium according to Clause 96 further includes instructions that cause the processor to perform the following operation: before multiplying the sample of the filtered block by the scaling factor, determining that a syntax element indicating whether to refine the filtered block has a value indicating that the filtered block should be refined.

[0293] Clause 107: The computer-readable storage medium according to Clause 106 further includes instructions that cause the processor to decode the value of the syntax element indicating whether to refine the filtered block in at least one of the following: a slice header including a slice of the block, an image header including an image of the block, or an adaptive parameter set (APS) corresponding to the slice including the block.

[0294] Clause 108: A computer-readable storage medium according to Clause 96, wherein the filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, the computer-readable storage medium further comprising instructions causing the processor to: apply a second filter to the decoded block to form a second filtered block; and multiply samples of the second filtered block by a second scaling factor to form a second refined filtered block, wherein the instructions causing the processor to combine the samples of the first refined filtered block with the corresponding samples of the decoded block include instructions causing the processor to: combine the samples of the first refined filtered block and the samples of the second refined filtered block with the corresponding samples of the decoded block.

[0295] Clause 109: A computer-readable storage medium according to Clause 96, wherein the filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, wherein the instruction causing the processor to apply the first filter to the decoded block includes instructions causing the processor to perform: applying each of a plurality of filters including the first filter to the decoded block to form a corresponding plurality of filtered blocks including the first filtered block; and wherein the processor is caused to apply the first filtered block to the first filtered block. The instruction to multiply a sample by the first scaling factor includes instructions that cause the processor to perform the following operations: multiply the samples of the corresponding plurality of filtered blocks including the first filtered block by the corresponding plurality of scaling factors including the first scaling factor to form a plurality of refined filtered blocks including the first refined filtered block, wherein the instruction that causes the processor to combine the samples of the first refined filtered block with the corresponding samples of the decoded block includes instructions that cause the processor to perform the following operations: combine the samples of each of the plurality of refined filtered blocks with the corresponding samples of the decoded block.

[0296] Clause 110: The computer-readable storage medium according to Clause 96 further includes instructions that cause the processor to encode the current block before decoding it.

[0297] Clause 111: An apparatus for filtering decoded video data, the apparatus comprising: a unit for decoding blocks of video data to form decoded blocks; a unit for applying a filter to the decoded blocks to form filtered blocks; a unit for multiplying samples of the filtered blocks by a scaling factor to form refined filtered blocks; and a unit for combining samples of the refined filtered blocks with corresponding samples of the decoded blocks.

[0298] It should be recognized that, based on the examples, certain actions or events of any of the techniques described herein may be performed in a different order, and may be added, combined, or omitted entirely (e.g., not all of the described actions or events are necessary for the practice of the technique). Furthermore, in some examples, actions or events may be performed concurrently rather than sequentially, for example, through multi-threaded processing, interrupt handling, or multiple processors.

[0299] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium such as a data storage medium or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to obtain instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0300] For example, rather than limiting, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave). However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer instead to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically magnetically copy data, while optical discs utilize lasers to optically copy data. Combinations of the above items should also be included within the scope of computer-readable media.

[0301] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Accordingly, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, the techniques can be implemented substantially within one or more circuit or logic elements.

[0302] The technologies disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed technologies, but they do not necessarily need to be implemented through different hardware units. Rather, as described above, the various units can be combined in a codec hardware unit, or provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0303] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for filtering decoded video data, the method comprising: Decode blocks of video data to form decoded blocks; The filter is applied to the decoded block to form a filtered block; The samples of the filtered block are multiplied by a scaling factor to form a refined filtered block; as well as The samples of the refined and filtered block are combined with the corresponding samples of the decoded block.

2. The method according to claim 1, further comprising: The value of the syntax element representing the scaling factor is decoded.

3. The method according to claim 2, further comprising: The value of the syntax element is inversely quantized using an integer value with predefined bit precision.

4. The method according to claim 1, wherein, The scaling factor includes a first scaling factor among a plurality of scaling factors, and wherein multiplying the sample of the filtered block by the scaling factor includes multiplying the sample of the filtered block by each of the plurality of scaling factors.

5. The method according to claim 1, wherein, Multiplying the samples of the filtered block by the scaling factor to form the refined filtered block further includes adding the offset to the product of the samples of the filtered block and the scaling factor to form the refined filtered block.

6. The method according to claim 1, wherein, Multiplying the samples of the filtered block by the scaling factor to form the refined filtered block further includes: Determine the offset value; The shifted bits are applied to the offset value to form the shifted offset value; and The refined filtered block is formed by adding the shifted offset value to the product of the sample of the filtered block and the scaling factor.

7. The method according to claim 1, wherein, The application of the filter includes at least one of the following: a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter.

8. The method according to claim 1, further comprising: Before multiplying the sample of the filtered block by the scaling factor, it is determined that the syntax element indicating whether to refine the filtered block has a value indicating that the filtered block should be refined.

9. The method according to claim 1, wherein, The filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor. The method further includes: The second filter is applied to the decoded block to form a second filtered block; and The samples of the second filtered block are multiplied by a second scaling factor to form a second refined filtered block. The process of combining the sample of the first refined and filtered block with the corresponding sample of the decoded block includes: combining the sample of the first refined and filtered block and the sample of the second refined and filtered block with the corresponding sample of the decoded block.

10. The method according to claim 1, wherein, The filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor. Applying the first filter to the decoded block includes: applying each of a plurality of filters including the first filter to the decoded block to form a plurality of corresponding filtered blocks including the first filtered block; and The step of multiplying the samples of the first filtered block by the first scaling factor includes: multiplying the samples of the corresponding plurality of filtered blocks including the first filtered block by the corresponding plurality of scaling factors including the first scaling factor, to form a plurality of refined filtered blocks including the first refined filtered block. The step of combining the sample of the first refined and filtered block with the corresponding sample of the decoded block includes: combining the sample of each of the plurality of refined and filtered blocks with the corresponding sample of the decoded block.

11. An apparatus for filtering decoded video data, the apparatus comprising: A memory configured to store video data; as well as One or more processors, which are implemented in a circuit and configured to: Decode blocks of video data to form decoded blocks; The filter is applied to the decoded block to form a filtered block; The samples of the filtered block are multiplied by a scaling factor to form a refined filtered block; as well as The samples of the refined and filtered block are combined with the corresponding samples of the decoded block.

12. The device according to claim 11, wherein, The one or more processors are further configured to decode the values ​​of the syntax elements representing the scaling factor.

13. The device according to claim 12, wherein, The one or more processors are further configured to inverse quantize the value of the syntax element using an integer value with a predefined bit precision.

14. The device according to claim 11, wherein, The scaling factor includes a first scaling factor among a plurality of scaling factors, and wherein the one or more processors are configured to multiply the sample of the filtered block by each of the plurality of scaling factors.

15. The device according to claim 11, wherein, The one or more processors are configured to add the offset to the product of the sample of the filtered block and the scaling factor to form the refined filtered block.

16. The device according to claim 11, wherein, In order to form the refined and filtered block, the one or more processors are configured to: Determine the offset value; The shifted bits are applied to the offset value to form the shifted offset value; and The refined filtered block is formed by adding the shifted offset value to the product of the sample of the filtered block and the scaling factor.

17. The device according to claim 11, wherein, In order to apply the filter, the one or more processors are configured to apply at least one of a neural network-based filter, a neural network-based loop filter, a neural network-based post-loop filter, an adaptive in-loop filter, or a predefined adaptive in-loop filter.

18. The device according to claim 11, wherein, The one or more processors are further configured to: determine, before multiplying the sample of the filtered block by the scaling factor, that a syntax element indicating whether to refine the filtered block has a value indicating that the filtered block should be refined.

19. The device according to claim 11, wherein, The filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, wherein the one or more processors are further configured to: The second filter is applied to the decoded block to form a second filtered block; and The samples of the second filtered block are multiplied by a second scaling factor to form a second refined filtered block. In order to combine the samples of the first refined and filtered block with the corresponding samples of the decoded block, the one or more processors are configured to combine the samples of the first refined and filtered block and the samples of the second refined and filtered block with the corresponding samples of the decoded block.

20. The device according to claim 11, wherein, The filter includes a first filter, the filtered block includes a first filtered block, the refined filtered block includes a first refined filtered block, and the scaling factor includes a first scaling factor, wherein the one or more processors are configured to: Each of a plurality of filters, including the first filter, is applied to the decoded block to form a corresponding plurality of filtered blocks including the first filtered block; and The samples of the corresponding plurality of filtered blocks, including the first filtered block, are multiplied by the corresponding plurality of scaling factors, including the first scaling factor, to form a plurality of refined filtered blocks, including the first refined filtered block. In order to combine the samples of the first refined and filtered block with the corresponding samples of the decoded block, the one or more processors are configured to combine the samples of each of the plurality of refined and filtered blocks with the corresponding samples of the decoded block.

Images