Unified neural network loop filter

Abstract

A unified neural network loop filter and method implemented by a video encoding device includes applying a neural network (NN) filter to unfiltered samples of a video unit to generate filtered samples, where the NN filter is based on a NN filter model generated using a quality index indicator (QI) input. The method also includes converting between a video media file and a bitstream based on the generated filtered samples.

Description

[0001] Cross-references to related applications

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 171,225, filed April 6, 2021, entitled "Unified Neural Network Designed for Loop Filtering in Video Encoding and Decoding," which is incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to image and video encoding and decoding. Background Technology

[0004] Despite advancements in video compression, digital video still accounts for the largest share of bandwidth usage on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth required for digital video usage is expected to continue to grow. Summary of the Invention

[0005] The disclosed aspects / embodiments provide one or more neural network (NN) filter models that are trained as part of a loop filtering technique or a filtering technique used in a post-processing stage to reduce distortion caused during compression. Furthermore, samples with different characteristics are processed using different NN filter models. Additionally, a uniform NN filter model can be generated by providing at least one indicator as input to the NN filter; this indicator can be a quality level indicator.

[0006] The first aspect relates to a method implemented by an encoding / decoding device. The method includes: applying a neural network (NN) filter to unfiltered samples of a video unit to generate filtered samples, wherein the NN filter is based on an NN filter model generated using a quality level indicator (QI) input. The method also includes converting between a video media file and a bitstream based on the generated filtered samples.

[0007] Alternatively, in any of the foregoing aspects, another embodiment of that aspect provides that the Qi input includes the quantization parameters (QP) of the video unit.

[0008] Alternatively, in any of the foregoing aspects, another embodiment of that aspect provides a Qi input comprising a sequence of quantization parameters (QP), the sequence comprising the video unit.

[0009] Optionally, in any of the foregoing aspects, another embodiment of that aspect provides a Qi input comprising quantization parameters (QP) of a picture or stripe, the picture or stripe comprising the video unit.

[0010] Optionally, in any of the foregoing aspects, another embodiment of that aspect provides that the Qi input includes the output of a quantization process applied to the video unit, wherein the quantization process is based on the quantization parameters (QP) of the video unit, the QP of a sequence of the video unit, or the QP of a picture or stripe of the video unit.

[0011] Alternatively, in any of the foregoing aspects, another embodiment of that aspect provides expanding the quality level indicator into a two-dimensional array having the same size as the video unit, wherein the two-dimensional array is an additional input plane to the NN filter.

[0012] Alternatively, in any of the foregoing aspects, another embodiment of that aspect provides that the video unit is a codec tree unit (CTU) or a codec unit (CU).

[0013] Optionally, in any of the foregoing aspects, another embodiment of this aspect provides that the QI input is fed into multiple fully connected layers of the NN filter to expand the QI input into a one-dimensional vector, wherein the elements of the one-dimensional vector are channel scaling factors for recalibrating the feature maps of the NN filter model.

[0014] Alternatively, in any of the foregoing aspects, another embodiment of that aspect provides the feature map of the NN filter model by f∈R N×W×H Given, where N represents the number of channels, W represents the channel width, and H represents the channel height, wherein the one-dimensional vector provided by the fully connected layer is given by S∈R N It indicates, and among them, according to F i,j,k =f i,j,k ×S i Generate the recalibrated feature map, where 1≤i≤N, 1≤j≤W, 1≤k≤H, and F represents the recalibrated feature map.

[0015] Alternatively, in any of the foregoing aspects, another embodiment of this aspect provides the use of one selected from the group consisting of a sequence parameter set (SPS), a picture parameter set (PPS), a picture header, a strip header, a codec tree unit (CTU), and a codec unit (CU) to signal the quality level indicator to the decoder.

[0016] Optionally, in any of the foregoing aspects, another embodiment of this aspect provides a video unit including a first color component and a second color component, wherein the Qi input indicates the quality of the first color component, and wherein the NN filter is applied to the first color component and the second color component to generate the filtered samples.

[0017] Optionally, in any of the foregoing aspects, another embodiment of that aspect provides a video unit including a first color component and a second color component, wherein the Qi input includes a first Qi indicating the quality of the first color component and a second Qi indicating the quality of the second color component, and wherein the first Qi is applied to the first color component and the second Qi is applied to the second color component to generate the filtered samples.

[0018] Optionally, in any of the foregoing aspects, another embodiment of that aspect provides a video unit including a first color component and a second color component, wherein the QI input indicates the quality of the first color component, and wherein the NN filter is applied only to the first color component to generate the filtered samples.

[0019] Alternatively, in any of the foregoing aspects, another embodiment of that aspect provides a QI indicator for the quality of the area or image including the video unit.

[0020] Alternatively, in any of the foregoing aspects, another embodiment of that aspect provides a QI indicator for the quality of a region or image that does not include the video unit.

[0021] Optionally, in any of the foregoing aspects, another embodiment of that aspect provides a video unit referencing the area or the image.

[0022] Alternatively, in any of the foregoing aspects, another embodiment of that aspect provides a conversion that includes generating the bitstream based on the video media file.

[0023] Alternatively, in any of the foregoing aspects, another embodiment of that aspect provides a conversion that includes parsing the bitstream to obtain the video media file.

[0024] The second aspect relates to an apparatus for encoding and decoding video data, including a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: apply a neural network (NN) filter to unfiltered samples of video units to generate filtered samples, wherein the NN filter is based on an NN filter model generated using a quality level indicator (QI) input; and convert between a video media file and a bitstream based on the generated filtered samples.

[0025] The third aspect relates to a non-transitory computer-readable medium storing a bitstream of video, the bitstream being generated by a method performed by a video processing apparatus. The method performed by the video processing apparatus includes: applying a neural network (NN) filter to unfiltered samples of video cells to generate filtered samples, wherein the NN filter is based on an NN filter model generated using a quality level indicator (QI) input; and generating the bitstream based on the generated filtered samples.

[0026] The fourth aspect relates to a method for storing a video bitstream performed by a video processing apparatus. The method performed by the video processing apparatus includes applying a neural network (NN) filter to unfiltered samples of video cells to generate filtered samples, wherein the NN filter is based on an NN filter model generated using a quality level indicator (QI) input; and generating the bitstream based on the generated filtered samples.

[0027] For clarity, any of the foregoing embodiments may be combined with any one or more other foregoing embodiments to create new embodiments within the scope of this disclosure.

[0028] These and other features will become clearer from the following detailed description taken in conjunction with the accompanying drawings and claims. Attached Figure Description

[0029] For a more complete understanding of this disclosure, please refer to the following brief description in conjunction with the accompanying drawings and detailed description, wherein the same reference numerals represent the same elements.

[0030] Figure 1 This is an example of raster scan strip segmentation of an image.

[0031] Figure 2 This is an example of rectangular strip segmentation in an image.

[0032] Figure 3 These are examples of how an image is divided into slices, tiles, and rectangular strips.

[0033] Figure 4A This is an example of a codec tree block (CTB) that spans the bottom image boundary.

[0034] Figure 4B This is an example of a CTB that spans the right edge of the image.

[0035] Figure 4C This is an example of a CTB that spans the bottom right edge of the image.

[0036] Figure 5 This is an example of an encoder block diagram.

[0037] Figure 6 This is a diagram of the sample points within an 8×8 sample point block.

[0038] Figure 7 These are examples of pixels involved in filter on / off decisions and strong / weak filter selection.

[0039] Figure 8 Four one-dimensional (1-D) orientation schemes for EO sample point classification are shown.

[0040] Figure 9 An example of the filter shape for an adaptive loop filter (GALF) based on geometric transformation is shown.

[0041] Figure 10 An example of relative coordinates for a 5×5 diamond filter support is shown.

[0042] Figure 11 Another example of relative coordinates for 5×5 rhombus filter support is shown.

[0043] Figure 12A This is an example architecture of the proposed CNN filter.

[0044] Figure 12B This is an example of constructing a ResBlock.

[0045] Figure 13 Examples of the process of generating filtered samples based on a neural network filter model with a received quality level indicator as input, based on various examples.

[0046] Figure 14 Examples of quality level indicators that are expanded (spanned) or tiled (tiles) into a two-dimensional array, based on various examples.

[0047] Figure 15 This is an example of feeding quality level indicators into multiple fully connected layers (e.g., fully connected layers of NN filters) to expand them into a one-dimensional vector, based on various examples.

[0048] Figure 16 A block diagram is shown to illustrate an example video processing system.

[0049] Figure 17 This is a block diagram of a video processing device.

[0050] Figure 18 A block diagram of an example video codec system is shown.

[0051] Figure 19 A block diagram illustrating an example of a video encoder.

[0052] Figure 20 A block diagram illustrating an example of a video decoder.

[0053] Figure 21 This is a video data encoding method according to an embodiment of the present disclosure. Detailed Implementation

[0054] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the illustrative embodiments, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but modifications can be made within the full scope of the appended claims and their equivalents.

[0055] The Universal Video Codec (VVC), also known as H.266, is used in some descriptions for ease of understanding only and not to limit the scope of the disclosed technology. Therefore, the techniques described herein are also applicable to other video codec protocols and designs.

[0056] This patent document relates to video encoding and decoding. Specifically, this description relates to loop filters in image / video encoding and decoding. The disclosed examples can be applied individually or in various combinations to video bitstreams encoded and decoded using existing video encoding and decoding standards (e.g., VVC, High Efficiency Video Codec (HEVC)) or pending standards (e.g., AVS3). The disclosed examples can also be applied to future video encoding and decoding standards, future video codecs, or as post-processing methods outside the encoding / decoding process.

[0057] Video codec standards have primarily evolved through the development of well-known standards from the International Telecommunication Union-Telecommunication (ITU-T) and the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). ITU-T developed H.261 and H.263, while ISO / IEC developed Moving Picture Experts Group (MPEG)-1 and MPEG-4 Vision. The two organizations jointly developed the H.262 / MPEG-2 video, H.264 / MPEG-4 Advanced Video Coding (AVC), and H.265 / High-Efficiency Video Coding (HEVC) standards.

[0058] Since H.262, video codec standards have been based on a hybrid video codec architecture, employing temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, the Video Codec Experts Group (VCEG) and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and applied them to reference software called the Joint Exploration Model (JEM).

[0059] In April 2018, a Joint Video Experts Team (JVET) was created between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) to research the VVC standard, aiming for a 50% bitrate reduction compared to HEVC. VVC version 1 was completed in July 2020.

[0060] Color spaces and chromaticity subsampling are discussed. A color space, also known as a color model (or color system), is an abstract mathematical model that simply describes a range of colors as tuples of numbers, typically 3 or 4 values ​​or color components (e.g., red, green, and blue (RGB)). Fundamentally, a color space is an explanation of a coordinate system and its subspaces.

[0061] For video compression, the most commonly used color spaces are YCbCr and RGB. YCbCr, Y'CbCr, or YPb / CbPr / Cr are also written as YC. B C R or Y'C B C R RGB is a color space family used as part of the color image pipeline in video and digital photography systems. Y' is the luminance component, while CB and CR are the blue and red difference chromaticity components. Y' (with an apostrophe) is different from Y, which is luminance, meaning that light intensity is based on gamma-corrected RGB primary color non-linear encoding.

[0062] Chromaticity subsampling is a practice that utilizes the fact that the human visual system is less sensitive to color differences than to brightness, and encodes images by applying lower precision to chromaticity information than to brightness information.

[0063] For 4:4:4 chroma subsampling, each of the three Y'CbCr components has the same sampling rate, therefore there is no chroma subsampling. This scheme is sometimes used in high-end film scanners and film post-production.

[0064] For 4:2:2 chroma subsampling, the two chroma components are sampled at half the luminance sampling rate: the horizontal chroma precision is halved. This reduces the bandwidth of the uncompressed video signal by one-third, but with almost no visual difference.

[0065] For 4:2:0 chroma subsampling, the horizontal sampling is doubled compared to 4:1:1, but the vertical precision is halved because the Cb and Cr channels are sampled only on each alternating line in this scheme. Therefore, the data rate is the same. Cb and Cr are subsampled by a factor of 2 in the horizontal and vertical directions, respectively. There are three variants of the 4:2:0 scheme with different horizontal and vertical addressing.

[0066] In MPEG-2, Cb and Cr are co-addressed horizontally. Cb and Cr are also addressed between pixels in the vertical direction (gap addressing). In Joint Picture Experts Group (JPEG) / JPEG File Exchange Format (JFIF), H.261, and MPEG-1, Cb and Cr are gap-addressed, located between alternating luma samples. In 4:2:0 DV, Cb and Cr are co-addressed horizontally. In the vertical direction, they are co-addressed on alternating lines.

[0067] The definition of a video unit is provided. An image is divided into one or more tile rows and one or more tile columns. A tile is a sequence of codec tree units (CTUs) that cover a rectangular area of ​​the image. A tile is divided into one or more bricks, each brick consisting of multiple CTU rows within the tile. A tile that is not divided into multiple bricks is also called a brick. However, a brick that is a proper subset of a tile is not called a tile. A stripe includes multiple tiles of an image or multiple bricks of a tile.

[0068] Two stripe modes are supported: raster scan stripe mode and rectangular stripe mode. In raster scan stripe mode, the stripe consists of a sequence of slices from a raster scan of the image. In rectangular stripe mode, the stripe consists of multiple tiles from the image, which together form a rectangular area of ​​the image. The tiles within the rectangular stripe are arranged according to the raster scan order of the stripe.

[0069] Figure 1 An example of raster scan strip segmentation for image 100, where the image is divided into twelve slices 102 and three raster scan strips 104. As shown, each slice 102 and strip 104 includes multiple CTUs 106.

[0070] Figure 2 This is an example of rectangular strip segmentation of image 200 according to the VVC specification, where the image is divided into twenty-four slices 202 (six slice columns 203 and four slice rows 205) and nine rectangular strips 204. As shown in the figure, each slice 202 and strip 204 includes multiple CTUs 206.

[0071] Figure 3 This is an example of dividing an image 300 into slices, tiles, and rectangular strips according to the VVC specification. The image is divided into four slices 302 (two slice columns 303 and two slice rows 305), eleven tiles 304 (the top left slice contains one tile, the top right slice contains five tiles, the bottom left slice contains two tiles, and the bottom right slice contains three tiles) and four rectangular strips 306.

[0072] The CTU and Codec Tree Block (CTB) sizes are discussed. In VVC, the CTU size, notified by the syntax element log2_ctu_size_minus2 in the Sequence Parameter Set (SPS) signaling, can be as small as 4×4. The Sequence Parameter Set Raw Byte Sequence Payload (RBSP) syntax is as follows.

[0073]

[0074]

[0075]

[0076] log2_ctu_size_minus2plus2 specifies the size of the luminance codec tree block for each CTU.

[0077] log2_min_luma_coding_block_size_minus2plus2 specifies the minimum luma encoding / decoding block size.

[0078] The variables CtbLog2SizeY, CtbSizeY, MinCbLog2SizeY, MinCbSizeY, MinTbLog2SizeY, MaxTbLog2SizeY, MinTbSizeY, MaxTbSizeY, PicWidthInCtbsY, PicHeightInCtbsY, PicSizeInCtbsY, PicWidthInMinCbsY, PicHeightInMinCbsY, PicSizeInMinCbsY, PicSizeInSamplesY, PicWidthInSamplesC, and PicHeightInSamplesC are derived as follows:

[0079] CtbLog2SizeY=log2_ctu_size_minus2+2 (7-9)

[0080] CtbSizeY = 1 <CtbLog2SizeY (7-10)

[0081] MinCbLog2SizeY=log2_min_luma_coding_block_size_minus2+2 (7-11)

[0082] MinCbSizeY=1<<MinCbLog2SizeY (7-12)

[0083] MinTbLog2SizeY=2 (7-13)

[0084] MaxTbLog2SizeY=6 (7-14)

[0085] MinTbSizeY=1<<MinTbLog2SizeY (7-15)

[0086] MaxTbSizeY=1<<MaxTbLog2SizeY (7-16)

[0087] PicWidthInCtbsY=Ceil(pic_width_in_luma_samples÷CtbSizeY) (7-17)

[0088] PicHeightInCtbsY=Ceil(pic_height_in_luma_samples÷CtbSizeY) (7-18)

[0089] PicSizeInCtbsY=PicWidthInCtbsY*PicHeightInCtbsY (7-19)

[0090] PicWidthInMinCbsY=pic_width_in_luma_samples / MinCbSizeY (7-20)

[0091] PicHeightInMinCbsY=pic_height_in_luma_samples / MinCbSizeY (7-21)

[0092] PicSizeInMinCbsY=PicWidthInMinCbsY*PicHeightInMinCbsY (7-22)

[0093] PicSizeInSamplesY=pic_width_in_luma_samples*pic_height_in_luma_samples (7-23)

[0094] PicWidthInSamplesC = pic_width_in_luma_samples / SubWidthC (7-24)

[0095] PicHeightInSamplesC = pic_height_in_luma_samples / SubHeightC (7-25)

[0096] Figure 4A is an example of a CTB that straddles the bottom picture boundary. Figure 4B is an example of a CTB that straddles the right picture boundary. Figure 4C is an example of a CTB that straddles the bottom-right picture boundary. In Figures 4A-4C there are K = M, L < N; K < M, L = N; K < M, L < N, respectively.

[0097] Reference Figures 4A-4C discussed the CTUs in picture 400. Assume a CTB / largest coding unit (LCU) size indicated by M×N (usually M equals N, as defined in HEVC / VVC), and for a CTB located at the boundary of a picture (or slice or strip or other type, taking the picture boundary as an example), K×L samples are within the picture boundary, where K < M or L < N. For Figures 4A-4C those CTBs 402 depicted in, the CTB size still equals M x N. However, the bottom boundary / right boundary of the CTB is outside picture 400.

[0098] discussed the encoding / decoding process of a typical video encoder / decoder (aka codec). Figure 5 is an example of an encoder block diagram of VVC, which includes three loop filter blocks: deblocking filter (DF), sample adaptive offset (SAO), and adaptive loop filter (ALF). Different from the DF that uses predefined filters, SAO and ALF utilize the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding offsets and applying finite impulse response (FIR) filters respectively, with signaling of the encoded side information of the offsets and filter coefficients. ALF is at the last processing stage of each picture and can be regarded as a tool to try to capture and fix the artifacts established in the previous stage.

[0099] Figure 5This is a schematic diagram of encoder 500. Encoder 500 is suitable for implementing VVC technology. Encoder 500 includes three loop filters: a deblocking filter (DF) 502, a sample adaptive offset (SAO) 504, and an ALF 506. Unlike DF 502, which uses predefined filters, SAO 504 and ALF 506, by means of signaling notification of the offset and filter coefficients' encoding / decoding side information, reduce the mean square error between the original and reconstructed samples by adding an offset and applying an FIR filter, respectively, using the original samples of the current image. ALF 506 is located at the final processing stage of each image and can be considered as a tool to attempt to capture and repair artifacts established in previous stages.

[0100] The encoder 500 also includes an intra-frame prediction component 508 and a motion estimation / compensation (ME / MC) component 510, configured to receive input video. The intra-frame prediction component 508 is configured to perform intra-frame prediction, while the ME / MC component 510 is configured to perform inter-frame prediction using a reference image obtained from a reference image buffer 512. Residual blocks from inter-frame or intra-frame prediction are fed to a transform component 514 and a quantization component 516 to generate quantized residual transform coefficients, which are then fed to an entropy codec component 518. The entropy codec component 518 entropy codes and decodes the prediction results and the quantized transform coefficients and sends them to a video decoder (not shown). Quantized components output from the quantization component 516 can be fed to an inverse quantization component 520, an inverse transform component 522, and a reconstruction (REC) component 524. The REC component 524 is capable of outputting images to a DF 502, a SAO 504, and an ALF 506 for filtering before these images are stored in the reference image buffer 512.

[0101] The input to the DF 502 is the reconstructed samples prior to the loop filter. First, vertical edges in the image are filtered. Then, using the samples modified by the vertical edge filtering process as input, horizontal edges in the image are filtered. Vertical and horizontal edges in the CTB of each CTU are processed separately on a codec unit basis. Vertical edges of the codec block in the codec unit are filtered starting from the left edge of the codec block and proceeding geometrically towards the right edge of the codec block. Horizontal edges of the codec block in the codec unit are filtered starting from the top edge of the codec block and proceeding geometrically towards the bottom edge of the codec block.

[0102] Figure 6 This is a diagram 600 showing sample points 602 within an 8×8 sample block 604. As shown, diagram 600 includes horizontal block boundaries 606 and vertical block boundaries 608 on the 8×8 grid. Furthermore, diagram 600 depicts a non-overlapping 8×8 sample block 610, which can be deblocked in parallel.

[0103] Boundary decision-making is discussed. The filter is applied to 8×8 block boundaries. Furthermore, it must be a transform block boundary or an encoding / decoding sub-block boundary (e.g., due to the use of affine motion prediction, optional temporal motion vector prediction (ATMVP)). For those boundaries that are not such boundaries, the filter is disabled.

[0104] Boundary strength calculation was discussed. For transform block boundaries / encoder / decoder sub-block boundaries, if they are located in an 8×8 grid, the transform block boundaries / encoder / decoder sub-block boundaries can be filtered, and the bS[xD] of that edge... i ][yD j The settings for [xDi][yDj] (where [xDi][yDj] represent coordinates) are defined in Table 1 and Table 2, respectively.

[0105] Table 1. Boundary Strength (when SPS IBC is disabled)

[0106]

[0107]

[0108] Table 2. Boundary Strength (when SPS IBC is enabled)

[0109]

[0110] The deblocking decision for the luminance component is discussed.

[0111] Figure 7 Example 700 shows the pixels involved in the filter on / off decision and strong / weak filter selection. A wider, stronger brightness filter is used only if conditions 1, 2, and 3 are all true. Condition 1 is the "bulk condition." This condition detects whether samples on the P-side and Q-side belong to a bulk, represented by the variables bSidePisLargeBlk and bSideQisLargeBlk, respectively. The definitions of bSidePisLargeBlk and bSideQisLargeBlk are as follows.

[0112] bSidePisLargeBlk = ((edge ​​type is vertical, and p0 belongs to CU with width >= 32) || (edge ​​type is horizontal, and p0 belongs to CU with height >= 32)) ? TRUE:FLASE

[0113] bSideQisLargeBlk = ((edge ​​type is vertical, and q0 belongs to CU with width >= 32) || (edge ​​type is horizontal, and q0 belongs to CU with height >= 32)) ? TRUE:FLASE

[0114] Based on bSidePisLargeBlk and bSideQisLargeBlk, condition 1 is defined as follows.

[0115] Condition 1=(bsidepislageblk||bsidepislageblk)? TRUE:FLASE

[0116] Next, if condition 1 is true, condition 2 will be further checked. First, the following variables are derived.

[0117] In HEVC, first export dp0, dp3, dq0, and dq3.

[0118] If (p-side is greater than or equal to 32)

[0119] dp0=(dp0+Abs(p50-2*p40+p30)+1)>>1

[0120] dp3=(dp3+Abs(p53-2*p43+p33)+1)>>1

[0121] If (q-side is greater than or equal to 32)

[0122] dq0=(dq0+Abs(q50-2*q40+q30)+1)>>1

[0123] dq3=(dq3+Abs(q53-2*q43+q33)+1)>>1

[0124] Condition 2 = (d < β)? TRUE: FALSE

[0125] Where d = dp0 + dq0 + dp3 + dq3.

[0126] If conditions 1 and 2 are valid, then further check whether any block uses sub-blocks.

[0127]

[0128] Finally, if both conditions 1 and 2 are valid, the proposed deblocking method will check condition 3 (large block strong filtering condition), which is defined as follows.

[0129] In condition 3 StrongFilterCondition, export the following variables.

[0130] dpq is derived in the same way as in HEVC.

[0131]

[0132] In HEVC, StrongFilterCondition = (dpq < (β >> 2), sp3 + sq3 < (3 * β >> 5), and Abs(p0 - q0) < (5 * t). C +1)>>1)? TRUE:FALSE.

[0133] A stronger deblocking filter for brightness (designed for larger blocks) is discussed.

[0134] A bilinear filter is used when samples on either side of the boundary belong to a large block. Samples belonging to a large block are defined when the width of the vertical edge is greater than or equal to 32 and the height of the horizontal edge is greater than or equal to 32.

[0135] Bilinear filters are listed below.

[0136] Then, in the above HEVC deblocking, for the block boundary sample points p from i = 0 to Sp-1... i And the block boundary sample points q from j=0 to Sq-1 i Replaced by linear interpolation, p i and q i This is to filter the i-th sample point in a row of vertical edges, or to filter the i-th sample point in a column of horizontal edges, as shown below.

[0137] p i ′=(f i *Middle s,t +(64-f i )*P s +32)>>6), clipped to p i ±tcPD i

[0138] q j ′=(g j *Middle s,t +(64-g j )*Q s +32)>>6), clipped to q j ±tcPD j

[0139] Among them, tcPD i and tcPD j The terminology is the position-related clipping described below, and g is given below. j f i Middle s,t P s and Q s .

[0140] The deblocking control of chroma was discussed.

[0141] A strong chroma filter is used on both sides of the block boundary. Here, the chroma filter is selected when both sides of the chroma edge are greater than or equal to 8 (chroma position), and the decision is based on the following three conditions: The first condition is the decision regarding boundary strength and block size. The proposed filter can be applied when the width or height of the block orthogonal to the block edge in the chroma sampling domain is equal to or greater than 8. The second and third conditions are essentially the same as the HEVC luminance deblocking decision, namely the on / off decision and the strong filtering decision, respectively.

[0142] In the first decision, the boundary strength (bS) of the chroma filter is modified, and the conditions are checked sequentially. If a condition is met, the remaining lower-priority conditions are skipped.

[0143] Chromatic deblocking is performed when bS equals 2, or when a large block boundary is detected when bS equals 1.

[0144] The second and third conditions are essentially the same as the HEVC luminance strong filter decision, as shown below.

[0145] In the second condition: d is derived as in the HEVC luminance deblock. The second condition is TRUE when d is less than β.

[0146] In the third condition, StrongFilterCondition is deduced as follows.

[0147] For example, deriving dpq in HEVC

[0148] sp3 = Abs(p3 - p0), as derived in HEVC.

[0149] sq3 = Abs(q0 - q3), as derived in HEVC.

[0150] In HEVC design, StrongFilterCondition = (dpq < (β >> 2), sp3 + sq3 < ...

[0151] For (β>>3), Abs(p0-q0) is less than (5*t). C +1)>>1).

[0152] Strong deblocking filters for chroma are discussed. The following strong deblocking filters for chroma are defined.

[0153] p2′=(3*p3+2*p2+p1+p0+q0+4)>>3

[0154] p1′=(2*p3+p2+2*p1+p0+q0+q1+4)>>3

[0155] p0′=(p3+p2+p1+2*p0+q0+q1+q2+4)>>3

[0156] The proposed chromaticity filter performs deblocking on a 4×4 chromaticity sample grid.

[0157] Position-dependent limiting (tcPD) is discussed. tcPD is applied to the output samples of the brightness filtering process, which involves modifying the strong and long filters at the boundaries of 7, 5, and 3 samples. Assuming a quantization error distribution, it is proposed to increase the limiting value of samples expected to have higher quantization noise, thus resulting in a larger deviation between the expected reconstructed sample values ​​and the true sample values.

[0158] For each P or Q boundary filtered using an asymmetric filter, a position-related threshold table is selected from two tables (i.e., Tc7 and Tc3 in the table below) that serve as edge information to the decoder, based on the result of the decision process in the boundary strength calculation.

[0159] Tc7={6,5,4,3,2,1,1}; Tc3={6,4,2};

[0160] tcPD=(Sp==3)? Tc3:Tc7;

[0161] tcQD=(Sq==3)? Tc3:Tc7;

[0162] For P or Q boundaries filtered by a short symmetric filter, apply a lower amplitude position correlation threshold.

[0163] Tc3 = {3, 2, 1};

[0164] After defining the threshold, the filtered p' is adjusted based on the tcP and tcQ limiting values. i q' i The sample values ​​are limited.

[0165] p” i =Clip3(p' i +tcP i ,p' i –tcP i ,p' i );

[0166] q” j =Clip3(q' j +tcQ j ,q' j –tcQ j ,q' j );

[0167] Among them, p' i and q'i It is the filtered sample value p” i and q” j It is the output sample value after clipping, tcP i The clipping threshold is derived from the VVCtc parameters, as well as tcPD and tcQD. The Clip3 function is the clipping function defined in VVC.

[0168] Now we discuss sub-block deblocking adjustments. To achieve parallel-friendly deblocking using long filters and sub-block deblocking, the long filter is restricted to modifying a maximum of 5 samples on the side using sub-block deblocking (AFFINE or ATMVP or decoder-side motion vector refinement (DMVR)), as shown in the long filter's brightness control. Furthermore, sub-block deblocking is adjusted such that sub-block boundaries on the 8×8 grid near the codec unit (CU) or implicit TU boundary are restricted to modifying a maximum of two samples on each side.

[0169] The following applies to sub-block boundaries that are not aligned with the CU boundary.

[0170]

[0171] An edge equal to 0 corresponds to the CU boundary, an edge equal to 2 or orthogonalLength-2 corresponds to 8 sample points of the sub-block boundary from the CU boundary, and so on. If implicit partitioning of TU is used, then implicit TU is true.

[0172] Sample Adaptive Offset (SAO) is discussed. The input to SAO is the deblocked reconstructed samples (DB). The concept of SAO is to reduce the average sample distortion of a region by first classifying region samples into multiple categories using a selected classifier, obtaining an offset for each category, and then adding this offset to each sample of that category. The classifier index and the region offset are encoded and decoded in the bitstream. In HEVC and VVC, the region (the unit for SAO parameter signaling notification) is defined as a CTU.

[0173] HEVC employs two SAO types that meet low complexity requirements. These two types are Edge Offset (EO) and Band Offset (BO), which will be discussed in detail below. The index of the SAO type is encoded and decoded (within the range [0, 2]). For EO, ​​sample classification is based on a comparison between the current sample and its neighboring samples, according to a one-dimensional orientation scheme: horizontal, vertical, 135° diagonal, and 45° diagonal.

[0174] Figure 8 Four one-dimensional (1-D) orientation schemes 800 for EO sample point classification are shown: horizontal (EO classification = 0), vertical (EO classification = 1), 135° diagonal (EO classification = 2), and 45° diagonal (EO classification = 3).

[0175] For a given EO classification, each sample point within the CTB is classified into one of five categories. The current sample point value, labeled "c", is compared with its two neighboring sample point values ​​along the selected 1-D pattern. The classification rules for each sample point are summarized in Table 3. Categories 1 and 4 are associated with local valleys and local peaks along the selected 1-D pattern, respectively. Categories 2 and 3 are associated with concave and convex angles along the selected one-dimensional scheme, respectively. If the current sample point does not belong to EO categories 1-4, it belongs to category 0, and SAO is not applied.

[0176] Table 3: Sampling classification rules for edge offset

[0177]

[0178] An adaptive loop filter based on geometric transformation (SAO) is discussed in the Joint Exploration Model (JEM). The input to the DB is the reconstructed samples after DB and SAO. The sample classification and filtering processes are based on the reconstructed samples after DB and SAO.

[0179] In JEM, a geometric transformation-based adaptive loop filter (GALF) with block-based filter adaptation is applied. For the luminance component, one of 25 filters is selected for each 2×2 block based on the direction and effectiveness of the local gradient.

[0180] The shape of the filter was discussed. Figure 9 An example of a GALF filter shape 900 is shown, including a 5×5 rhombus on the left, a 7×7 rhombus in the middle, and a 9×9 rhombus on the right. In JEM, up to three rhombus filter shapes can be selected for the luminance component (e.g., ...). Figure 9 (As shown). The image-level signaling notification index indicates the filter shape used for the luminance component. Each square represents a sample point, and Ci (i is 0–6 (left), 0–12 (middle), 0–20 (right)) represents the coefficient applied to that sample point. For the chrominance component in the image, a 5×5 rhombus is always used.

[0181] Block classification was discussed. Each 2×2 block was divided into one of 25 classes. The classification index c was based on its directionality D and validity. The quantized values ​​are exported as shown below.

[0182]

[0183] To calculate D and First, the gradients in the horizontal, vertical, and two diagonal directions are calculated using the 1-D Laplacian operator.

[0184]

[0185]

[0186]

[0187]

[0188] Indices i and j refer to the coordinates of the top-left sample point in the 2×2 block, and R(i,j) represents the reconstructed sample point at coordinates (i,j).

[0189] The maximum and minimum values ​​of the gradients in the horizontal and vertical directions are then set as follows:

[0190]

[0191] The maximum and minimum values ​​of the gradients in the two diagonal directions are set as follows:

[0192]

[0193] To derive the values ​​of directionality D, these values ​​are compared with each other and with two thresholds t1 and t2:

[0194] Step 1. If and If true, then D is set to 0.

[0195] Step 2. If Continue from step 3; otherwise, continue from step 4.

[0196] Step 3. If D is set to 2; otherwise, D is set to 1.

[0197] Step 4. If D is set to 4; otherwise, D is set to 3.

[0198] The validity value A is calculated as follows:

[0199]

[0200] A is further quantized to the range of 0 to 4, and the quantized value is represented as

[0201] For the two chromaticity components in the image, no classification method is applied; that is, a single set of ALF coefficients is applied for each chromaticity component.

[0202] The geometric transformation of the filter coefficients is discussed.

[0203] Figure 10 The relative coordinates 1000 are shown for the 5×5 diamond filters that support (from left to right) diagonal, vertical flip, and rotation.

[0204] Before filtering each 2×2 block, geometric transformations such as rotation, diagonal flip, and vertical flip are applied to the filter coefficients f(k,l) associated with coordinates (k,l) based on the gradient values ​​calculated for that block. This is equivalent to applying these transformations to samples within the filter's support region. The idea is to make the different blocks applying ALF more similar by aligning their orientations.

[0205] Three geometric transformations are introduced: diagonal transformation, vertical flip, and rotation.

[0206] Diagonal: f D (k,l)=f(l,k),

[0207] Vertical Flip:f V (k,l)=f(k,Kl-1), (9)

[0208] Rotation: f R (k,l)=f(Kl-1,k).

[0209] Where K is the size of the filter, and 0 ≤ k, l ≤ K⁻¹ are the coefficient coordinates, such that position (0, 0) is located at the top left corner and position (K⁻¹, K⁻¹) is located at the bottom right corner. The transform is applied to the filter coefficients f(k, l) based on the gradient values ​​calculated for this block. Table 4 summarizes the relationship between the transform and the four gradients in the four directions.

[0210] Table 4: Mapping between gradients and transformations computed for a block

[0211]

[0212] Signaling notification of filter parameters is discussed. In JEM, GALF filter parameters are signaled for the first CTU, i.e., after the stripe header of the first CTU and before the SAO parameters. A maximum of 25 groups of luminance filter coefficients can be sent. To reduce bit overhead, filter coefficients from different categories can be merged. Furthermore, the GALF coefficients of a reference image are stored and can be reused as GALF coefficients for the current image. The current image can optionally use the GALF coefficients stored for the reference image and bypass GALF coefficient signaling. In this case, only the index of a reference image is signaled, and the stored GALF coefficients of the indicated reference image are inherited by the current image.

[0213] To support GALF temporal prediction, a candidate list of GALF filter sets is maintained. At the beginning of decoding a new sequence, the candidate list is empty. After decoding an image, the corresponding filter set can be added to the candidate list. Once the size of the candidate list reaches the maximum allowed value (i.e., 6 in the current JEM), new filter sets overwrite the oldest sets in decoding order; that is, a first-in-first-out (FIFO) rule is applied to update the candidate list. To avoid duplication, a set is added to the list only if the corresponding image does not use GALF temporal prediction. To support temporal scalability, multiple candidate lists of filter sets exist, and each candidate list is associated with a temporal layer. More specifically, each array assigned by the temporal layer index (TempIdx) can form the filter set of previously decoded images with a lower TempIdx. For example, the k-th array is assigned to be associated with a TempIdx equal to k, and the k-th array contains only the filter set from images with a TempIdx less than or equal to k. After an image is encoded or decoded, the set of filters associated with that image will be used to update the arrays associated with TempIdx that are equal to or higher than TempIdx.

[0214] Temporal prediction of GALF coefficients is used for inter-frame encoding / decoding to minimize signaling overhead. For intra-frame frames, temporal prediction is not available, and a set of 16 fixed filters is assigned to each class. To indicate the use of fixed filters, signaling informs the flag for each class and, if necessary, the index of the selected fixed filter. Even when a fixed filter is selected for a given class, the coefficients of the adaptive filter f(k,l) can still be sent for that class; in this case, the coefficients of the filter applied to the reconstructed image are the sum of the coefficients from both sets.

[0215] The filtering process for the luminance component can be controlled at the CU level. A signaling flag indicates whether GALF is applied to the luminance component of the CU. For the chrominance component, whether GALF is applied is only indicated at the image level.

[0216] The filtering process is discussed. On the decoder side, when GALF is enabled for a block, each sample R(i,j) within the block is filtered, producing sample values ​​R′(i,j) as shown below, where L represents the filter length, f m,n Let f(k,l) represent the filter coefficients, and let f(k,l) represent the decoding filter coefficients.

[0217]

[0218] Figure 11This example illustrates the relative coordinates supported by a 5×5 diamond filter, assuming the current sample point's coordinates (i, j) are (0, 0). Sample points at different coordinates, filled with the same color, are multiplied by the same filter coefficients.

[0219] The geometric transformation-based adaptive loop filter (GALF) for VVC is discussed. In VVC test model 4.0 (VTM4.0), the filtering process of the adaptive loop filter is performed as follows:

[0220] O(x,y)=∑ (i,j) w(i,j).I(x+i,y+j), (11) where I(x+i,y+j) are the input samples, O(x,y) are the filtered output samples (i.e., the filtering result), and w(i,j) represent the filtering coefficients. In fact, in VTM4.0, it uses integer arithmetic to achieve fixed-point precision calculation.

[0221]

[0222] Where L represents the filter length, and w(i,j) are the fixed-point filter coefficients.

[0223] Compared to JEM, the current design of GALF in VVC has the following main changes:

[0224] 1) Adaptive filter shapes have been removed. Only 7×7 filter shapes are allowed for the luminance component, and only 5×5 filter shapes are allowed for the chrominance component.

[0225] 2) Move the signaling notification of ALF parameters from the strip / picture level to the CTU level.

[0226] 3) The category index is calculated at a 4×4 level instead of a 2×2 level. Furthermore, as proposed in JVET-L0147, a subsampling Laplacian calculation method for ALF classification is utilized. More specifically, it is not necessary to calculate the horizontal / vertical / 45-degree diagonal / 135-degree gradient for each sample point within a block. Instead, 1:2 subsampling is used.

[0227] Regarding filtering and reconstruction, the nonlinear ALF in the current VVC is discussed.

[0228] Equation (11) can be reformulated without affecting encoding / decoding efficiency as follows:

[0229] O(x,y)=I(x,y)+∑ (i,j)≠(0,0) w(i,j).(I(x+i,y+j)-I(x,y)), (2)

[0230] Where w(i,j) are the same filter coefficients as in equation (11) [except that w(0,0) is equal to 1 in equation (13), while it is equal to 1-∑ in equation (11). (i,j)≠(0,0) w(i,j)].

[0231] Using the above filter formula in (13), VVC introduces nonlinearity to reduce the impact of large differences between neighboring sample values ​​(I(x+i,y+j)) and the current sample value being filtered (I(x,y)) by using a simple limiting function, thereby making ALF more effective.

[0232] More specifically, the ALF filter is modified as follows:

[0233] O′(x,y)=I(x,y)+∑ (i,j)≠(0,0) w(i,j).K(I(x+i,y+j)-I(x,y),k(i,j)), (14)

[0234] Where K(d,b)=min(b,max(-b,d)) is the limiting function, and k(i,j) is the limiting parameter that depends on the (i,j) filter coefficients. The encoder performs optimization to find the optimal k(i,j).

[0235] In the JVET-N0242 implementation, a limiting parameter k(i,j) is specified for each ALF filter, and a limiting value is signaled for each filter coefficient. This means that a maximum of 12 limiting values ​​can be signaled for each luminance filter in the bitstream, and a maximum of 6 limiting values ​​can be signaled for each chrominance filter.

[0236] To limit signaling notification costs and encoder complexity, only four fixed values ​​are used, which are the same for both INTER and INTRA stripes.

[0237] Because the local differences in luminance are typically greater than those in chrominance, two separate sets of luminance and chrominance filters are applied. The maximum sample value in each set (1024 for a 10-bit depth) is also introduced so that clipping can be disabled when not necessary.

[0238] Table 5 provides a set of limiting values ​​used in the JVET-N0242 test. These four values ​​were selected by roughly dividing the full range of luminance sample values ​​(encoded in 10 bits) and the chrominance range from 4 to 1024 in the logarithmic domain.

[0239] More precisely, the brightness table for the limit value is obtained using the following formula:

[0240]

[0241] Similarly, the colorimetric table for the limiting values ​​is obtained using the following formula:

[0242]

[0243] Table 5: Authorized Limit Values

[0244]

[0245] The selected threshold value is encoded and decoded in the "alf_data" syntax element using the Golomb coding scheme corresponding to the threshold value index in Table 5 above. This coding scheme is the same as that used for the filter index.

[0246] A convolutional neural network-based loop filter for video encoding and decoding is discussed.

[0247] In deep learning, convolutional neural networks (CNNs or ConvNets) are a type of deep neural network most commonly used for analyzing visual images. They have been very successful in image and video recognition / processing, recommender systems, image classification, medical image analysis, and natural language processing.

[0248] CNNs are a regularized version of multilayer perceptrons. Multilayer perceptrons typically mean fully connected networks, where every neuron in one layer is connected to all neurons in the next layer. This "full connectivity" makes them prone to overfitting data. Typical regularization methods involve adding some form of weight magnitude metric to the loss function. CNNs take a different approach to regularization: they utilize hierarchical schemes in the data and assemble more complex methods using smaller and simpler schemes. Therefore, CNNs are at the lower extremes in terms of connectivity and complexity.

[0249] Compared to other image classification / processing algorithms, CNNs use relatively little preprocessing. This means the network learns filters that are hand-designed in traditional algorithms. This independence from existing knowledge and human effort in feature design is a major advantage.

[0250] Deep learning-based image / video compression generally has two meanings: end-to-end compression based entirely on neural networks, and traditional frameworks enhanced by neural networks. End-to-end compression based entirely on neural networks is discussed in Johannes Balle, Valero Laparra, and Eero P. Simoncelli, “End-to-end optimization of nonlinear transform codes for perceptual quality,” 2016, Workshop on Picture Coding (PCS), pp. 1–5, Institute of Electrical and Electronics Engineers (IEEE), and in Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár, “Lossy image compression using compression autocoders,” arXiv prequel arXiv:1703.00395 (2017). The traditional framework enhanced by neural networks is discussed in the following articles: Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong and Wen Gao, “Intra-frame prediction for image coding and decoding based on fully connected networks”, IEEE Transactions on Image Processing 27, 7 (2018), 3236–3247; Yuanying Dai, Dong Liu and Feng Wu, “Post-processing convolutional neural network approach in HEVC intra-frame coding and decoding”, MMM.Springer, 28–39; Rui Song, Dong Liu, Houqiang Li and Feng Wu, “Neural network-based arithmetic coding and decoding of intra-frame prediction modes in HEVC”, VCIP.IEEE, 1–4; and J. Pfaff, P. Helle, D. Maniry, S. Kaltenstadler, W. Samek, H. Schwarz, D. Marpe and T. Wiegand, “Neural network-based intra-frame prediction for video coding and decoding”, Digital Image Processing Applications XLI, Vol. 10752, International Society for Optics and Photonics, 1075213.

[0251] The first type typically employs an autoencoder-like structure, implemented through convolutional neural networks or recurrent neural networks. While relying solely on neural networks for image / video compression avoids any manual optimization or design, the compression efficiency may not be satisfactory. Therefore, research focused on the second type aims to enhance traditional compression frameworks by replacing or strengthening certain modules with neural networks. In this way, they can inherit the advantages of highly optimized traditional frameworks. For example, the fully connected network for intra-frame prediction proposed in HEVC is discussed in Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong, and Wen Gao, “Intra-frame Prediction Based on Fully Connected Networks for Image Encoding and Decoding,” IEEE Transactions on Image Processing 27, 7 (2018), pp. 3236-3247.

[0252] Besides intra-frame prediction, deep learning is also used to enhance other modules. For example, the loop filter in HEVC is replaced by a convolutional neural network, and satisfactory results were achieved in Yuanying Dai, Dong Liu, and Feng Wu, "A Convolutional Neural Network Approach for Post-Processing in HEVC Intra-Frame Encoding and Decoding," MMM. Springer, 28–39. Research in Rui Song, Dong Liu, Houqiang Li, and Feng Wu, "Neural Network-Based Arithmetic Encoding and Decoding for HEVC Intra-Frame Prediction Modes," VCIP. IEEE, 1–4, applies neural networks to improve the arithmetic encoding and decoding engine.

[0253] Loop filtering based on convolutional neural networks is discussed. In lossy image / video compression, the reconstructed frame is an approximation of the original frame, and because the quantization process is irreversible, this leads to distortion in the reconstructed frame. To mitigate this distortion, convolutional neural networks can be trained to learn the mapping from distorted frames to the original frames. In fact, training is necessary before deploying CNN-based loop filtering.

[0254] Training was discussed. The goal of the training process is to find the optimal values ​​for the parameters, including the weights and biases.

[0255] First, an encoder-decoder (e.g., HM, JEM, VTM, etc.) is used to compress the training dataset to generate distorted reconstructed frames. Then, the reconstructed frames are fed into a CNN, and the cost is calculated using the CNN's output and the ground truth frames (original frames). Common cost functions include Sum of Absolute Differences (SAD) and Mean Squared Error (MSE). Next, the gradient of the cost with respect to each parameter is derived using the backpropagation algorithm. The parameter values ​​are updated using the gradients. This process is repeated until the convergence criterion is met. After training is complete, the derived optimal parameters are saved for use in the inference phase.

[0256] The convolution process is discussed. During convolution, the filter moves across the image from left to right and from top to bottom, changing one pixel column for horizontal movement and one pixel row for vertical movement. The amount of movement by which the filter is applied across the input image is called the stride, and it is almost always symmetrical in the height and width dimensions. For height and width movements, the default (multiple) stride in two dimensions is (1, 1).

[0257] Figure 12A This is an example architecture 1200 of the proposed CNN filter, and Figure 12B This is an example of constructing a ResBlock (1250). In most deep convolutional neural networks, ResBlocks are used as basic modules and stacked several times to build the final network. In one example, ResBlocks are obtained by combining convolutional layers, ReLU / PReLU activation functions, and convolutional layers, such as... Figure 12B As shown.

[0258] Inference is discussed. During the inference phase, distorted reconstructed frames are fed into the CNN and processed by the CNN model, whose parameters have been determined during the training phase. The input samples to the CNN can be reconstructed samples before or after DB, or before or after SAO, or before or after ALF.

[0259] Current CNN-based loop filtering suffers from the following problems. First, a separate CNN model is trained for each quality level (e.g., quantization parameter (QP), constant ratio factor (CRF), or bit rate), resulting in a large number of CNN models. Second, when QPs are used as input to the neural network, they are first concatenated into a two-dimensional array of the same size as the block to be filtered, and then used as an additional input plane. Therefore, QP information may not be fully utilized. Third, the characteristics of an encoder-decoder block are not fully explored when constructing a candidate list containing multiple models for a given block.

[0260] This document discloses techniques for addressing one or more of the aforementioned problems. For example, this disclosure provides one or more neural network (NN) filter models trained as part of a loop filtering technique or a filtering technique used in a post-processing stage to reduce distortion caused during compression. Furthermore, samples with different characteristics are processed using different NN filter models. This disclosure also illustrates how to design a unified NN filter model by feeding at least one indicator (which may be related to a quality level (e.g., QP, CRF, or bit rate, and is therefore a quality level indicator (QI)) as input to the NN filter, and how to construct a candidate list of multiple models containing coded blocks by considering the encoding / decoding statistics of the blocks. It should be noted that the concept of a unified NN filter model by feeding the QI as input to the NN filtering process can also be extended to other NN-based encoding / decoding tools, such as NN-based super-resolution or motion compensation designs. In the following example, a neural network-based filtering technique is used as an example.

[0261] The following list of embodiments should be considered as examples to explain general concepts. These embodiments should not be interpreted narrowly. Furthermore, these embodiments can be combined in any way.

[0262] In this disclosure, the NN filter can be any type of NN filter, such as a convolutional neural network (CNN) filter. In the following discussion, the NN filter may also be referred to as a CNN filter.

[0263] In the following discussion, a video unit can be a sequence, picture, strip, slice, tile, sub-picture, CTU / CTB, CTU / CTB line, one or more CU / codec block (CB), one or more CTU / CTB, one or more Virtual Pipeline Data Units (VPDUs), or a sub-region within a picture / strip / slice / tile. A parent video unit represents a unit larger than a video unit. Generally, a parent unit will contain several video units; for example, when the video unit is a CTU, the parent unit can be a strip, a CTU line, multiple CTUs, etc. In some embodiments, a video unit can be a sample / pixel.

[0264] Figure 13 An example of the process for generating filtered samples for an NN filter model based on a received quality level indicator (QI) as input. Figure 13In the process 1300 shown, at least some unfiltered samples are provided as input to the NN filter. In one example, the unfiltered samples are samples (e.g., pixels) of video units that have not yet undergone any filtering or have not undergone sufficient filtering. Therefore, the output of the NN filter can be filtered samples. The output of the NN filter is also based on the NN filter model generated using a Quality Level Indicator (QI) input. The QI input can be related to the quality level of the video unit, such as QP, CRF, or bitrate.

[0265] Figure 14 This is an example of process 1400 where quality level indicators are expanded or concatenated into a two-dimensional array, based on some examples. For instance, in process 1400, the two-dimensional array may have the same size as the video unit to be filtered (e.g., in each of the two dimensions) and serve as an additional input plane for the NN filter.

[0266] Figure 15 This is an example of process 1500 where a quality level indicator (QI) is fed into multiple fully connected layers (e.g., fully connected layers of an NN filter) to expand into a one-dimensional vector. In process 1500, the elements of the resulting one-dimensional vector can be used as channel scaling factors. In process 1500, channel scaling factors are useful for recalibrating the feature maps of the NN filter model (e.g., where N represents the number of channels, W represents the width, and H represents the height), and thus provide recalibrated feature maps in some examples.

[0267] A discussion of model selection is provided.

[0268] Example 1

[0269] 1. The NN filter model can take at least one indicator that may be related to the quality level as input. This indicator is called the quality level indicator (QI).

[0270] a. In one example, the quality level indicator for a video cell is the quantization parameter (QP) associated with the video cell.

[0271] b. In one example, the quality level indicator for a video unit is the QP associated with the sequence to which the video unit belongs (e.g., the initial QP of signaling notification in SPS).

[0272] c. In one example, the quality level indicator for a video cell is the QP associated with the picture or strip to which the video cell belongs.

[0273] d. Alternatively, the QP can be replaced by a quantization step corresponding to the QP described in bullet points a, b, and c above. For example, the QI input can be the output of a quantization process applied to a video unit, where the quantization process is based on the QP of the video unit, the QP of a sequence containing video units, or the QP of a picture or strip containing video units.

[0274] e. In one example, the quality level indicators are first concatenated or expanded into a two-dimensional array with the same size as the video units to be filtered, and then treated as an additional input plane, for example... Figure 14 The splicing process is shown.

[0275] i. In one example, the video unit is CTU / CTB.

[0276] f. In one example, the quality level indicator is fed into several fully connected layers of the NN filter to be expanded into a one-dimensional vector, the elements of which will be used as channel scaling factors to recalibrate the feature maps of the NN filter model.

[0277] i. In one example, the feature map of the NN filter model is represented as f∈R N×W×H Let N, W, and H be the number of channels, width, and height, respectively. Let S ∈ R N Let F represent a one-dimensional vector obtained from the fully connected layer. Then, the above recalibration process can be written as: F i,j,k =f i,j,k ×S i ,1≤i≤N,1≤j≤W,1≤k≤H, where f is the recalibrated feature map, for example, Figure 15 As shown.

[0278] g. In one example, the quality level indicator is used as an additional input plane and a recalibration factor, i.e., aspects e and f can be combined.

[0279] h. In one example, the quality level indicator can be communicated from encoder signaling to the decoder, such as in the SPS, Picture Parameter Set (PPS), Picture header, Strip header, CTU, CU, or at any region level. In some examples, the PPS may include information about entropy encoding / decoding modes, strip groups, motion prediction, and deblocking filters.

[0280] i. In one example, f(QI) instead of QI can be fed into the NN, where f is any function.

[0281] j. In one example, the use of QI can depend on the color components.

[0282] i. For example, a first QI can be used for a first component (e.g., luminance), and a second QI can be used for a second component (e.g., chromaticity, such as Cb or Cr). The first QI can indicate the quality level of the first component, and the second QI can indicate the quality level of the second component.

[0283] ii. For example, the first QI can be used for a first component (e.g., luminance), and the first QI can also be used for a second component (e.g., chromaticity, such as Cb or Cr). The first QI can indicate the quality level of the first component.

[0284] iii. For example, QI can be used for NN filtering of the first component (e.g., luminance), but not for the second component (e.g., chrominance, such as Cb or Cr).

[0285] iv. For example, Qi can be used for NN filtering of all components.

[0286] k. In one example, the QI used to filter the first video unit can indicate the quality of the region or image containing the first video unit.

[0287] l. In one example, the QI used to filter the first video unit can indicate the quality of the region or image that does not contain the first video unit.

[0288] i. For example, QI can indicate the quality of the area or image referenced by the first video unit.

[0289] Example 2

[0290] 2. In the second embodiment, the construction of the filter candidate list may depend on the encoding and decoding statistics of the video unit (e.g., prediction mode, qp, stripe type, etc.).

[0291] a. In one example, the candidate list may depend on the ratio of the first set of samples in the video unit that are encoded and decoded by the codec tool (or codec mode) X.

[0292] i. In one example, X is a prediction mode, such as an intra-frame codec mode.

[0293] 1) Alternatively, X can be any other codec tool, such as any inter-frame prediction tool or transform / quantization codec tool.

[0294] ii. In one example, X is a variety of prediction modes, such as intra-frame / palette / IBC codec modes.

[0295] iii. In one example, if the ratio of intra-frame codec samples is not less than or greater than a threshold, then one of the candidates in the candidate list is the NN filter model corresponding to the candidate for the intra-frame codec.

[0296] iv. In one example, if the ratio of intra-frame codec samples is not less than or greater than a threshold, the first candidate is the NN filter model corresponding to the intra-frame stripe.

[0297] v. In the example above, the threshold can be predefined.

[0298] 1) In one example, the video unit is CTU / CTB.

[0299] 2) In one example, the threshold is set to 0.5.

[0300] 3) Alternatively, the threshold can be derived on the fly, for example, based on decoding information such as sequence or image or strip-level QP.

[0301] 4) Alternatively, the threshold can be exported instantly, for example, based on the color components.

[0302] b. In one example, the method described above in Example 2 may be applied conditionally.

[0303] i. In one example, the above methods(s) may be applied only to the luminance component.

[0304] ii. In one example, the above methods(s) can be applied to multiple color components.

[0305] 1) In one example, the encoding and decoding statistics of the luma block can be used to determine the candidate list of luma; and the encoding and decoding statistics of the chroma block can be used to determine the candidate list of chroma.

[0306] 2) In another example, the encoding and decoding statistics of the luma block can be used to determine a candidate list of luminance and chrominance.

[0307] iii. In one example, the above methods(s) can be applied to certain time-domain layers.

[0308] 1) In one example, they can be applied to time-domain layers with an ID greater than K (e.g., K=3).

[0309] 2) In one example, they can be applied to time-domain layers with ID less than K (e.g., K=4).

[0310] iv. In one example, the above methods(s) can be applied to certain image / strip types, such as P or B stripes.

[0311] c. The first set of samples can be located in the same picture containing the video unit to be filtered by the NN.

[0312] i. Alternatively, the first set of samples can be located in a different image than the video unit to be filtered by the NN. For example, the first set of samples can be located in the reference image of the video unit.

[0313] ii. Alternatively, the first set of samples may be in multiple images that are encoded / decoded prior to the image containing the video unit to be filtered.

[0314] Figure 16 This is a block diagram illustrating an example video processing system 1600, in which various techniques disclosed herein can be implemented. Various implementations may include some or all of the components of the video processing system 1600. The video processing system 1600 may include an input 1602 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8-bit or 10-bit multi-component pixel values, or may be received in a compressed or encoded format. Input 1602 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, Passive Optical Network (PON), and wireless interfaces such as Wi-Fi or cellular interfaces.

[0315] Video processing system 1600 may include codec component 1604, which implements the various codec or encoding methods described herein. Codec component 1604 can reduce the average bit rate of the video from input 1602 to the output of codec component 1604 to produce a codec representation of the video. Therefore, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of codec component 1604 can be stored or transmitted via connected communication (as shown in component 1606). The stored or transmitted bitstream (or codec) representation of the video received at input 1602 can be used by component 1608 to generate pixel values ​​or to send displayable video to display interface 1610. The process of generating a user-viewable video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “codec” operations or tools, it should be understood that codec tools or operations are used at the encoder, and the corresponding decoding tools or operations for the reverse codec results will be performed by the decoder.

[0316] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), PCI, IDE, etc. The techniques described herein can be implemented in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0317] Figure 17This is a block diagram of a video processing apparatus 1700. Apparatus 1700 can be used to implement one or more methods described herein. Apparatus 1700 can be implemented in smartphones, tablets, computers, Internet of Things (IoT) receivers, etc. Apparatus 1700 may include one or more processors 1702, one or more memories 1704, and video processing hardware 1706. The processors(multiple) 1702 can be configured to implement one or more methods described in this application. The memories(multiple) 1704 can be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 1706 can be used to implement some of the techniques described in this application using hardware circuitry. In some embodiments, hardware 1706 may be implemented wholly or partially within processor 1702, such as a graphics coprocessor.

[0318] Figure 18 This is a block diagram illustrating an example video codec system 1800 that can utilize the techniques disclosed herein. Figure 18 As shown, the video encoding / decoding system 1800 may include a source device 1810 and a destination device 1820. The source device 1810 generates encoded video data, which can be referred to as a video encoding device. The destination device 1820 can decode the encoded video data generated by the source device 1810 and can be referred to as a video decoding device.

[0319] The source device 1810 may include a video source 1812, a video encoder 1814, and an input / output (I / O) interface 1816.

[0320] Video source 1812 may include sources such as video capture devices, interfaces for receiving video data from video content providers, and / or computer graphics systems for generating video data, or combinations thereof. Video data may include one or more images. Video encoder 1814 encodes the video data from video source 1812 to generate a bitstream. The bitstream may include bit sequences that form a codec representation of the video data. The bitstream may include codec images and associated data. A codec image is a codec representation of an image. Associated data may include sequence parameter sets, image parameter sets, and other syntax structures. I / O interface 1816 may include a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be transmitted directly to destination device 1820 via network 1830 through I / O interface 1816. Encoded video data may also be stored on storage medium / server 1840 for access by destination device 1820.

[0321] The target device 1820 may include an I / O interface 1826, a video decoder 1824, and a display device 1822.

[0322] I / O interface 1826 may include a receiver and / or a modem.

[0323] I / O interface 1826 can acquire encoded video data from source device 1810 or storage medium / server 1840. Video decoder 1824 can decode the encoded video data. Display device 1822 can display the decoded video data to the user. Display device 1822 can be integrated with target device 1820, or it can be external to target device 1820, which is configured to interface with an external display device.

[0324] The video encoder 1814 and the video decoder 1824 can operate according to video compression standards, such as the High Efficiency Video Coding (HEVC) standard, the Universal Video Coding (VVC) standard, and other current and / or further standards.

[0325] Figure 19 This is a block diagram illustrating an example of a video encoder 1900, which can be... Figure 18 The video encoder 1814 in the video codec system 1800 described herein.

[0326] The video encoder 1900 can be configured to perform any or all of the techniques disclosed herein. Figure 19 In the example, the video encoder 1900 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 1900. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0327] The functional components of the video encoder 1900 may include a segmentation unit 1901, a prediction unit 1902 (which may include a mode selection unit 1903, a motion estimation unit 1904, a motion compensation unit 1905, and an intra-frame prediction unit 1906), a residual generation unit 1907, a transform unit 1908, a quantization unit 1909, an inverse quantization unit 1910, an inverse transform unit 1911, a reconstruction unit 1912, a buffer 1913, and an entropy encoding / decoding unit 1914.

[0328] In other examples, the video encoder 1900 may include more, fewer, or different functional components. In one example, the prediction unit 1902 may include an intra-block copy (IBC) unit. The IBC unit can perform prediction in IBC mode, where at least one reference picture is the picture containing the current video block.

[0329] Furthermore, some components, such as the motion estimation unit 1904 and the motion compensation unit 1905, can be highly integrated, but for interpretable purposes... Figure 19 The example is shown separately.

[0330] Segmentation unit 1901 can segment an image into one or more video blocks. Figure 18 The video encoder 1814 and video decoder 1824 in the program can support various video block sizes.

[0331] The mode selection unit 1903 can select one of the encoding / decoding modes (intra-frame or inter-frame, e.g., based on error results) and provides the resulting intra-frame or inter-frame encoded / decoded block to the residual generation unit 1907 to generate residual block data, and provides it to the reconstruction unit 1912 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 1903 can select a combination of intra-frame and inter-frame prediction (CIIP) modes, where prediction is based on inter-frame prediction signals and intra-frame prediction signals. In the case of inter-frame prediction, the mode selection unit 1903 can also select the resolution of the motion vector (e.g., sub-pixel or integer pixel precision) for the block.

[0332] To perform inter-frame prediction on the current video block, motion estimation unit 1904 can generate motion information for the current video block by comparing one or more reference frames from buffer 1913 with the current video block. Motion compensation unit 1905 can determine the predicted video block for the current video block based on motion information and decoded samples from images other than those associated with the current video block from buffer 1913.

[0333] The motion estimation unit 1904 and the motion compensation unit 1905 can perform different operations on the current video block, for example, depending on whether the current video block is in an I-band, P-band, or B-band.

[0334] In some examples, motion estimation unit 1904 may perform unidirectional prediction for the current video block, and may search reference images in list 0 or list 1 to find a reference video block for the current video block. Motion estimation unit 1904 may then generate a reference index indicating that the reference image in list 0 or list 1 contains the reference video block, and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 1904 may output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 1905 may generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0335] In other examples, motion estimation unit 1904 can perform bidirectional prediction for the current video block. Motion estimation unit 1904 can search for a reference video block for the current video block in the reference images in list 0, and can also search for another reference video block for the current video block in the reference images in list 1. Motion estimation unit 1904 can then generate reference indices indicating the reference images in lists 0 and 1, which contain reference video blocks and motion vectors indicating spatial displacement between the reference video blocks and the current video block. Motion estimation unit 1904 can output the reference index and motion vector of the current video block as motion information for the current video block. Motion compensation unit 1905 can generate a predicted video block for the current video block based on the reference video blocks indicated by the motion information of the current video block.

[0336] In some examples, the motion estimation unit 1904 can output a complete set of motion information for the decoder's decoding process.

[0337] In some examples, the motion estimation unit 1904 may not output the complete set of motion information for the current video. Instead, the motion estimation unit 1904 may signal the motion information of the current video block to another video block. For example, the motion estimation unit 1904 may determine that the motion information of the current video block is sufficiently similar to the motion information of neighboring video blocks.

[0338] In one example, the motion estimation unit 1904 may indicate a value in the syntactic structure associated with the current video block that indicates to the video decoder 1824 that the current video block has the same motion information as another video block.

[0339] In another example, motion estimation unit 1904 can identify another video block and motion vector difference (MVD) within the syntactic structure associated with the current video block. The motion vector difference represents the difference between the motion vector of the current video block and the motion vector of the indicated video block. Video decoder 1824 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0340] As discussed above, the video encoder 1814 can predictively signal motion vectors. Two examples of predictive signaling notification techniques that can be implemented by the video encoder 1814 include advanced motion vector prediction (AMVP) and merge pattern signaling notification.

[0341] The intra-prediction unit 1906 can perform intra-prediction on the current video block. When the intra-prediction unit 1906 performs intra-prediction on the current video block, it can generate prediction data for the current video block based on the decoded samples of other video blocks in the same frame. The prediction data for the current video block may include the predicted video block and various syntax elements.

[0342] The residual generation unit 1907 can generate residual data for the current video block by subtracting (e.g., indicated by a negative sign) the predicted video block of the current video block from the current video block. The residual data for the current video block may include residual video blocks that correspond to different sample components of the samples in the current video block.

[0343] In other examples, residual data for the current video block may not exist, for example in skip mode, and residual generation unit 1907 may not perform subtraction operations.

[0344] The transform processing unit 1908 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video blocks associated with the current video block.

[0345] After the transform processing unit 1908 generates a transform coefficient video block associated with the current video block, the quantization unit 1909 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0346] The dequantization unit 1910 and the inverse transform unit 1911 can apply dequantization and inverse transform to the transform coefficient video block, respectively, to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 1912 can add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 1902 to generate a reconstructed video block associated with the current block and store it in the buffer 1913.

[0347] After the video block is reconstructed in reconstruction unit 1912, a loop filtering operation can be performed to reduce video block artifacts in the video block.

[0348] The entropy encoding unit 1914 can receive data from other functional components of the video encoder 1900. When the entropy encoding unit 1914 receives data, it can perform one or more entropy encoding / decoding operations to generate entropy-encoded data and output a bitstream including the entropy-encoded data.

[0349] Figure 20 This is an example block diagram illustrating a video decoder 2000. A video decoder 2000 can be... Figure 18 The video decoder 1824 in the video codec system 1800 shown in the figure.

[0350] The video decoder 2000 can be configured to perform any or all of the technologies disclosed herein. Figure 20 In the example, the video decoder 2000 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 2000. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0351] exist Figure 20 In the example, the video decoder 2000 includes an entropy decoding unit 2001, a motion compensation unit 2002, an intra-frame prediction unit 2003, an inverse quantization unit 2004, an inverse transform unit 2005, a reconstruction unit 2006, and a buffer 2007. In some examples, the video decoder 2000 may perform functions typically associated with the video encoder 1814 (e.g., Figure 18 The encoding channel (pass) is the opposite of the decoding channel.

[0352] Entropy decoding unit 2001 can retrieve the encoded bitstream. The encoded bitstream may include entropy-coded video data (e.g., encoded video data blocks). Entropy decoding unit 2001 can decode the entropy-coded video data, and based on the entropy-coded video data, motion compensation unit 2002 can determine motion information including motion vectors, motion vector precision, reference image list index, and other motion information. For example, motion compensation unit 2002 can determine this information by executing AMVP and merge modes.

[0353] The motion compensation unit 2002 can generate motion compensation blocks, possibly performing interpolation based on an interpolation filter. The identifier of the interpolation filter used at sub-pixel precision can be included in the syntax element.

[0354] The motion compensation unit 2002 can use the interpolation filter used by the video encoder 1814 during the encoding of the video block to calculate the interpolated values ​​of the sub-integer pixels of the reference block. The motion compensation unit 2002 can determine the interpolation filter used by the video encoder 200 based on the received syntax information and use the interpolation filter to generate the prediction block.

[0355] The motion compensation unit 2002 can use some syntax information to determine the size of the blocks used to encode the frames and / or stripes of the encoded video sequence, segmentation information describing how each macroblock of the picture of the encoded video sequence is segmented, a mode indicating how each segment is encoded, one or more reference frames (and a list of reference frames) for each inter-frame coded block, and other information of the decoded encoded video sequence.

[0356] Intra-prediction unit 2003 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Inverse quantization unit 2004 performs inverse quantization, i.e., dequantization, on the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 2001. Inverse transform unit 2005 applies an inverse transform.

[0357] The reconstruction unit 2006 can add the residual block to the corresponding prediction block generated by the motion compensation unit 2002 or the intra-frame prediction unit 2003 to form a decoded block. If necessary, a deblocking filter can also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in a buffer 2007, which provides a reference block for subsequent motion compensation / intra-frame prediction and also generates decoded video for display device rendering.

[0358] Figure 21 This is a method 2100 for encoding and decoding video data according to an embodiment of the present invention. Method 2100 can be executed by an encoding / decoding device (e.g., an encoder) having a processor and memory. Method 2100 can be implemented to provide a uniform NN filter model, which is generated using at least one indicator (e.g., a quality level indicator) as input to the NN filter. This NN filter model avoids training a separate CNN model for each quality level, thereby avoiding the creation of a large number of such CNN models.

[0359] In block 2102, the encoding / decoding device applies a neural network (NN) filter to the unfiltered samples of the video unit to generate filtered samples. The NN filter is based on an NN filter model generated using a quality level indicator (QI) input. In one embodiment, unfiltered samples are samples (or pixels) that have not undergone any filtering process or have not been sufficiently filtered. For example, unfiltered samples have not been processed by any NN filter. As another example, unfiltered samples have not been processed by an NN filter, an adaptive loop filter (ALF), a deblocking filter (DF), a sample adaptive offset (SAO) filter, or a combination thereof.

[0360] In block 2104, the codec device converts between the video media file and the bitstream based on the generated filtered samples.

[0361] When implemented in an encoder, the conversion involves receiving a media file (e.g., a video unit) and encoding the filtered samples into a bitstream. When implemented in a decoder, the conversion involves receiving a bitstream comprising the filtered samples and decoding the bitstream to obtain the filtered samples.

[0362] In one embodiment, method 2100 may utilize or combine one or more features or processes of other methods disclosed herein.

[0363] In one example, a non-transitory computer-readable medium stores a bitstream of video generated by a method (e.g., all or part of method 2100) performed by a video processing apparatus (e.g., video processing apparatus 1700 described above). For example, a bitstream can be generated by applying an NN filter to unfiltered samples of video cells to generate filtered samples. As described above, the NN filter is based on an NN filter model generated using a quality level indicator (QI) input, and the bitstream is generated based on the filtered samples.

[0364] The following provides a list of preferred solutions for some embodiments.

[0365] The following solutions illustrate example embodiments of the techniques discussed in this disclosure (e.g., Example 1).

[0366] 1. A method implemented by a video codec apparatus, comprising: for a conversion between video units and a bitstream of video, applying parameters of a loop filter to the video units according to rules; and performing the conversion based on a determination, wherein the determination is based on a quality level indicator (QI) input.

[0367] 2. The method according to claim 1, wherein the Qi of the video unit is the quantization parameter of the video unit.

[0368] 3. The method according to claim 1, wherein the Qi of the video unit is the quantization parameter value of the video.

[0369] 4. The method according to claim 1, wherein the QI of the video unit is a quantization step size applied to the video unit.

[0370] 5. The method of claim 1, wherein the conversion is performed by concatenating the Qi indicator inputs into a two-dimensional array of the same size as the video unit and using it as the determined input plane.

[0371] 6. The method according to any one of claims 1-5, wherein the QI indicator input is indicated in the bit stream.

[0372] 7. The method of claim 1, wherein the function for determining the QI indicator is used.

[0373] The following solutions illustrate example embodiments of the techniques discussed in the previous section (e.g., Example 2).

[0374] 8. A video processing method comprising: for conversion between video units and a bitstream of video, constructing a filter candidate list applicable to loop filtering of the video units according to rules; and performing the conversion based on the determination; wherein the rules specify that at least one entry in the list is based on a computational neural network (CNN) model.

[0375] 9. The method of claim 8, wherein the list is constructed in response to encoding / decoding statistics of the video units.

[0376] 10. The method of claim 9, wherein the encoding / decoding statistics include the ratio of a first set of video samples associated with the video unit to the samples of the video unit, the first set of video samples being encoded / decoded using a predetermined video encoding / decoding tool (referred to as tool X).

[0377] 11. The method of claim 10, wherein tool X comprises intra-frame prediction, palette prediction, or intra-frame block copy prediction mode.

[0378] 12. The method according to any one of claims 8-11, wherein the rule further responds to the encoding / decoding conditions of the video unit.

[0379] 13. The method of claim 12, wherein the encoding / decoding condition is that the video unit belongs to the luminance component of the video.

[0380] 14. The method according to claim 12, wherein the encoding / decoding condition is that the video unit belongs to a certain temporal layer of the video.

[0381] 15. The method of claim 10, wherein the first set of video samples is in an image including the video unit.

[0382] 16. The method of claim 10, wherein the first set of video samples are in one or more images different from the image including the video unit.

[0383] 17. The method according to any one of claims 1-16, wherein the video unit comprises an encoding / decoding unit, a transformation unit, a prediction unit, and stripes or sub-pictures.

[0384] 18. The method according to any one of claims 1-16, wherein the video unit is a codec block, a video strip, a video picture, a video strip, or a video subpicture.

[0385] 19. The method according to any one of claims 1-18, wherein the conversion comprises generating the video from the bitstream or generating the bitstream from the video.

[0386] 20. A video decoding apparatus comprising a processor configured to implement the method of one or more of claims 1 to 19.

[0387] 21. A video encoding apparatus comprising a processor configured to implement the method of one or more of claims 1 to 19.

[0388] 22. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 19.

[0389] 23. Methods, apparatus or bit streams generated according to the disclosed methods or systems described in this document.

[0390] The other solutions, examples, embodiments, modules, and functional operations disclosed in this document can be implemented in digital electronic circuits or computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or combinations thereof. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more computer program instruction modules encoded on a computer-readable medium, which are executed or controlled by a data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of substances affecting machine-readable propagation signals, or a combination thereof. The term "data processing apparatus" encompasses all means, devices, and machines that process data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof. The transmitted signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0391] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language (including compiled or interpreted languages) and can be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple harmonizing files (e.g., files storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on a single computer or on multiple computers located in one location or distributed across multiple locations and interconnected via a communication network.

[0392] The processes and logic described in this document can be executed by one or more programmable processors to execute one or more computer programs, thereby performing functions by manipulating input data and generating output. The processes and logic can also be executed by dedicated logic circuits, and can be implemented as dedicated logic circuits, such as FPGAs (field-programmable gate arrays) or ASICs (application-specific integrated circuits).

[0393] For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into special-purpose logic circuitry.

[0394] Although this patent document contains numerous details, these details should not be construed as limiting any invention or the scope of the claims, but rather as a description of features that may be specific to particular embodiments of a particular invention. Certain features described in this patent document in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0395] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all shown operations to achieve the desired effect. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0396] Only some implementation methods and examples are described, and other implementation methods, enhancements and variations may be made based on the content described and shown in this patent document.

[0397] Although this patent document contains numerous details, these details should not be construed as limiting any invention or the scope of the claims, but rather as a description of features that may be specific to particular embodiments of a particular invention. Certain features described in this patent document in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0398] Similarly, although the operations are depicted in a specific order in the figures, this should not be construed as requiring such operations to be performed in the specific order or sequence shown, or requiring all illustrated operations to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0399] Only a few implementations and examples have been described, and other implementations, improvements and modifications may be made based on the descriptions and instructions in this patent document.

Claims

1. A method implemented by a video encoding / decoding device, comprising: A neural network (NN) filter is applied to unfiltered samples of a video unit to generate filtered samples, wherein the NN filter is based on an NN filter model generated using a quality level indicator (QI) input. as well as Based on the generated filtered samples, conversion is performed between the video media file and the bitstream. The Qi input is expanded into a two-dimensional array with the same size as the video unit, wherein the two-dimensional array is an additional input plane for the NN filter.

2. The method according to claim 1, wherein, The Qi input includes the quantization parameter QP of the video unit.

3. The method according to claim 1, wherein, The Qi input includes the quantization parameter QP of the sequence, and the sequence includes the video unit.

4. The method according to claim 1, wherein, The Qi input includes the quantization parameter QP of the image or strip, and the image or strip includes the video unit.

5. The method according to claim 1, wherein, The Qi input includes the output of a quantization process applied to the video unit, wherein the quantization process is based on the quantization parameters QP of the video unit, the QP of a sequence of the video unit, or the QP of a picture or stripe of the video unit.

6. The method according to claim 1, wherein, The video unit is a codec tree unit (CTU) or a codec unit (CU).

7. The method according to claim 1, wherein, The Qi input is fed into multiple fully connected layers of the NN filter to expand the Qi input into a one-dimensional vector, wherein the elements of the one-dimensional vector are channel scaling factors for recalibrating the feature maps of the NN filter model.

8. The method according to claim 7, wherein, The feature map of the NN filter model is composed of Given, where N represents the number of channels, W represents the channel width, and H represents the channel height, wherein the one-dimensional vector provided by the fully connected layer is... It indicates, and among them, according to Generate the recalibrated feature map, where 1 i 1 j 1 k , where F represents the recalibrated feature map.

9. The method of claim 1, further comprising using one selected from the group consisting of a sequence parameter set SPS, a picture parameter set PPS, a picture header, a strip header, a codec tree unit CTU, and a codec unit CU to signal the Qi input to the decoder.

10. The method according to claim 1, wherein, The video unit includes a first color component and a second color component, wherein the Qi input indicates the quality of the first color component, and wherein the NN filter is applied to the first color component and the second color component to generate the filtered samples.

11. The method according to claim 1, wherein, The video unit includes a first color component and a second color component, wherein the Qi input includes a first Qi indicating the quality of the first color component and a second Qi indicating the quality of the second color component, and wherein the first Qi is applied to the first color component and the second Qi is applied to the second color component to generate the filtered samples.

12. The method according to claim 1, wherein, The video unit includes a first color component and a second color component, wherein the Qi input indicates the quality of the first color component, and wherein the NN filter is applied only to the first color component to generate the filtered samples.

13. The method according to claim 1, wherein, The Qi input indication includes the region of the video unit or the quality of the image.

14. The method according to claim 1, wherein, The Qi input indicates the quality of the area or image of the video unit.

15. The method according to claim 14, wherein, The video unit refers to the area or the image.

16. The method according to claim 1, wherein, The conversion includes generating the bitstream based on the video media file.

17. The method according to claim 1, wherein, The conversion includes parsing the bitstream to obtain the video media file.

18. An apparatus for encoding and decoding video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, When the instruction is executed by the processor, the processor: A neural network (NN) filter is applied to unfiltered samples of a video unit to generate filtered samples, wherein the NN filter is based on an NN filter model generated using a quality level indicator (QI) input. as well as Based on the generated filtered samples, conversion is performed between the video media file and the bitstream. The Qi input is expanded into a two-dimensional array with the same size as the video unit, wherein the two-dimensional array is an additional input plane for the NN filter.

19. A non-transitory computer-readable medium storing instructions and a bit stream, wherein, When the instruction is executed by the processor, it causes the processor to: A neural network (NN) filter is applied to unfiltered samples of a video unit to generate filtered samples, wherein the NN filter is based on an NN filter model generated using a quality level indicator (QI) input. as well as The bitstream is generated based on the generated filtered samples. The Qi input is expanded into a two-dimensional array with the same size as the video unit, wherein the two-dimensional array is an additional input plane for the NN filter.

Images