External Attention in Neural Network-Based Video Encoding and Decoding

Abstract

A method implemented by a video codec device includes applying a neural network (NN) filter to unfiltered samples of video units to generate filtered samples. The NN filter is based on an NN filter model configured to acquire attention based on codec parameter inputs. The method also includes performing a conversion between a video media file and a bitstream based on the generated filtered samples.

Description

[0001] Cross-references to related applications

[0002] This application is a divisional application of Chinese Patent Application No. 202210771651.5, filed on June 30, 2022, which claims priority and benefit to U.S. Provisional Patent Application No. 63 / 216,923, filed on June 30, 2021. The entire disclosure of the aforementioned application is incorporated herein by reference as part of the disclosure of this application. Technical Field

[0003] This disclosure generally relates to image and video encoding and decoding. Background Technology

[0004] Digital video accounts for the largest share of bandwidth usage on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video is expected to continue to grow. Summary of the Invention

[0005] The disclosed aspects / embodiments provide one or more neural network (NN) filter models trained as encoding / decoding tools to improve the efficiency of video encoding / decoding. NN-based encoding / decoding tools can be used to replace or enhance one or more modules implemented by a video encoder / decoder (also referred to as a codec). For example, NN models can be trained to provide additional intra-frame prediction modes, additional inter-frame prediction modes, transform kernels, and / or loop filters. Furthermore, NN models can be generated or designed as attention mechanisms using external information such as prediction, partitioning, and quantization parameters (QP).

[0006] The first aspect relates to a method implemented by an encoding / decoding device. The method includes applying a neural network (NN) filter to unfiltered samples of a video unit to generate filtered samples, wherein the NN filter is based on an NN filter model configured to acquire attention based on encoding / decoding parameter inputs. The method also includes performing a conversion between a video media file and a bitstream based on the generated filtered samples.

[0007] Optionally, in any of the foregoing aspects, another implementation of the aspect provides that the encoding / decoding parameter input includes one or more selected from the group consisting of: a segmentation scheme for the video unit; a prediction mode for the video unit; a quantization parameter associated with the video unit; and a boundary strength parameter for the boundary of the video unit.

[0008] Alternatively, in any of the foregoing aspects, another implementation of this aspect provides the extraction of features from the encoding / decoding parameter input using a convolutional layer with NN filters; and the use of the extracted features as attention in the NN filter model.

[0009] Alternatively, in any of the foregoing aspects, another implementation of this aspect provides that the intermediate feature map of the NN filter model is recalibrated by attention, and the attention is obtained by concatenating the encoding / decoding parameter input with the intermediate feature map to provide a concatenation result, and feeding the concatenation result into the convolutional layer of the NN filter.

[0010] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides that the attention is obtained using a two-layer convolutional neural network, and the attention is a single-channel feature map having the same spatial resolution as the intermediate feature map of the NN filter model that will be recalibrated by the attention.

[0011] Alternatively, in any of the foregoing aspects, another implementation of this aspect provides using attention to recalibrate the intermediate feature maps of the NN filter model, wherein the intermediate feature maps of the NN filter model are given as G, where Where N is the number of channels, W is the channel width, and H is the channel height, and the attention obtained is given as A, where .

[0012] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides This represents the recalibrated intermediate feature map, and the attention applied includes: based on To provide recalibrated intermediate feature maps, where 1 i , of which 1 j And one of them k .

[0013] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides This represents the recalibrated intermediate feature map, and the attention applied includes: based on To provide recalibrated intermediate feature maps, where 1 i , of which 1 j , of which 1 k , and among them, This represents the mapping function applied to each element of the attention.

[0014] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides a mapping function. This includes the sigmoid function or the hyperbolic tangent function.

[0015] Alternatively, in any of the foregoing aspects, another implementation of this aspect provides the use of different A or B values ​​for different channels of the intermediate feature map. .

[0016] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides This represents the recalibrated intermediate feature map, and the attention applied includes: based on + To provide recalibrated intermediate feature maps, where 1 i , of which 1 j , of which 1 k , and among them, This represents the mapping function applied to each element of the attention.

[0017] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides a mapping function. This includes the sigmoid function or the hyperbolic tangent function.

[0018] Alternatively, in any of the foregoing aspects, another implementation of this aspect provides the use of different A or B values ​​for different channels of the intermediate feature map. .

[0019] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides that attention is applied to a specified layer within the NN filter model.

[0020] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides an NN filter model that includes residual blocks, and attention is applied only to the feature maps from the last layer of each residual block.

[0021] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides that the NN filter is selected from one or more of the group consisting of: an adaptive loop filter, a deblocking filter, and a sample adaptive offset filter.

[0022] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides a conversion that includes generating a bitstream from a video media file.

[0023] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides a conversion that includes parsing a bitstream to obtain a video media file.

[0024] The second aspect relates to an apparatus for encoding and decoding video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: apply a neural network (NN) filter to unfiltered samples of video units to generate filtered samples, wherein the NN filter is based on an NN filter model configured to acquire attention based on encoding and decoding parameter inputs; and convert between a video media file and a bitstream based on the generated filtered samples.

[0025] The third aspect relates to a non-transitory computer-readable medium for storing a video bitstream generated by a method performed by a video processing apparatus. The method performed by the video processing apparatus includes applying a neural network (NN) filter to unfiltered samples of video cells to generate filtered samples, wherein the NN filter is based on an NN filter model configured to acquire attention based on encoding / decoding parameter inputs; and generating a bitstream based on the generated filtered samples.

[0026] The fourth aspect relates to a method for storing a video bitstream, performed by a video processing apparatus. The method performed by the video processing apparatus includes applying a neural network (NN) filter to unfiltered samples of video units to generate filtered samples, wherein the NN filter is based on an NN filter model configured to acquire attention based on encoding / decoding parameter inputs; and generating a bitstream based on the generated filtered samples.

[0027] For clarity, any of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create new embodiments within the scope of this disclosure.

[0028] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. Attached Figure Description

[0029] To gain a more complete understanding of this disclosure, reference is now made to the following brief description in conjunction with the accompanying drawings and detailed description, wherein like reference numerals denote like parts.

[0030] Figure 1 This is an example of raster scan strip segmentation of an image.

[0031] Figure 2 This is an example of rectangular strip segmentation in an image.

[0032] Figure 3These are examples of images divided into slices, bricks, and rectangular strips.

[0033] Figure 4A This is an example of a codec tree block (CTB) that spans the bottom image boundary.

[0034] Figure 4B This is an example of a CTB that spans the right-hand image boundary.

[0035] Figure 4C This is an example of a CTB that spans the bottom right image boundary.

[0036] Figure 5 This is an example of an encoder block diagram.

[0037] Figure 6 This is a diagram of the sample points within an 8×8 sample point block.

[0038] Figure 7 These are examples of pixels involved in filter on / off decisions and strong / weak filter selection.

[0039] Figure 8 Four one-dimensional (1-D) orientation patterns for EO sample point classification are shown.

[0040] Figure 9 An example of the filter shape for an adaptive loop filter (GALF) based on geometric transformation is shown.

[0041] Figure 10 An example of relative coordinates for 5×5 rhombus filter support is shown.

[0042] Figure 11 Another example of relative coordinates for 5×5 rhombus filter support is shown.

[0043] Figure 12A This is an example architecture of the proposed CNN filter.

[0044] Figure 12B This is an example of constructing a residual block (ResBlock).

[0045] Figure 13 This is an example of a process for generating filtered samples based on various examples of neural network filter models that use received encoding / decoding parameters as input.

[0046] Figure 14 These are examples of applying attention, obtained using external information such as encoding / decoding parameters, to the feature maps of a neural network filter model to provide recalibrated feature maps, based on various examples.

[0047] Figure 15A It is a schematic block diagram of the architecture of neural network filtering methods based on various examples, and Figure 15B It is based on various examples Figure 15A A schematic diagram of the attention residual blocks used in the architecture.

[0048] Figure 16 This is a block diagram illustrating an example video processing system.

[0049] Figure 17 This is a block diagram of a video processing device.

[0050] Figure 18 This is a block diagram illustrating an example of a video encoding / decoding system.

[0051] Figure 19 This is a block diagram illustrating an example of a video encoder.

[0052] Figure 20 This is a block diagram illustrating an example of a video decoder.

[0053] Figure 21 This is a method for encoding and decoding video data according to embodiments of the present disclosure. Detailed Implementation

[0054] It should be understood from the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the illustrative embodiments, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but modifications can be made within the scope of the appended claims and their equivalents.

[0055] The use of the term "Variety Video Codec (VVD)" (also known as H.266) in some descriptions is for ease of understanding only and not to limit the scope of the disclosed techniques. Thus, the techniques described herein are also applicable to other video codec protocols and designs.

[0056] This patent document relates to video encoding and decoding. Specifically, this specification relates to loop filters in image / video encoding and decoding. The disclosed examples can be applied individually or in various combinations to video bitstreams encoded and decoded using existing video encoding and decoding standards such as VVC, HEVC, or standards yet to be finalized (e.g., AVS3). The disclosed examples can also be applied to future video encoding and decoding standards, future video codecs, or as post-processing methods outside the encoding / decoding process.

[0057] Video codec standards have primarily evolved through the development of well-known standards from the International Telecommunication Union-Telecommunication (ITU-T) and the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). ITU-T developed H.261 and H.263, while ISO / IEC developed Moving Picture Experts Group (MPEG)-1 and MPEG-4 Vision. The two organizations jointly developed the H.262 / MPEG-2 video, H.264 / MPEG-4 Advanced Video Coding (AVC), and H.265 / High-Efficiency Video Coding (HEVC) standards.

[0058] Since H.262, video codec standards have been based on a hybrid video codec architecture, employing temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, the Video Codec Experts Group (VCEG) and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and applied them to reference software called the Joint Exploration Model (JEM).

[0059] In April 2018, a Joint Video Experts Team (JVET) was created between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) to work on the VVC standard, aiming for a 50% bitrate reduction compared to HEVC. VVC version 1 was completed in July 2020.

[0060] Color spaces and chromaticity subsampling are discussed. A color space, also known as a color model (or color system), is an abstract mathematical model that simply describes a range of colors as tuples of numbers, typically 3 or 4 values ​​or color components (e.g., red, green, and blue (RGB)). Fundamentally, a color space is an explanation of a coordinate system and its subspaces.

[0061] For video compression, the most commonly used color spaces are YCbCr and RGB. YCbCr, Y'CbCr, or YPb / CbPr / Cr are also written as YC. B C R or Y'C B C R is a color space family used as part of the color image pipeline in video and digital photography systems. Y' is the luminance component, C... B and C RThese are the blue and red chromaticity components. Y' (with an apostrophe) is different from Y, which is luminance. This means that the light intensity is non-linearly encoded based on gamma-corrected RGB primary colors.

[0062] Chromaticity subsampling is a practice that utilizes the fact that the human visual system is less sensitive to color differences than to brightness, and encodes images by applying a lower resolution to chromaticity information than to brightness information.

[0063] For 4:4:4 chroma subsampling, each of the three Y'CbCr components has the same sampling rate, therefore there is no chroma subsampling. This scheme is sometimes used in high-end film scanners and film post-production.

[0064] For 4:2:2 chroma subsampling, the two chroma components are sampled at half the luminance sampling rate: the horizontal chroma resolution is halved. This reduces the bandwidth of the uncompressed video signal by one-third, but with almost no visual difference.

[0065] For 4:2:0 chroma subsampling, the horizontal sampling is doubled compared to 4:1:1, but the vertical resolution is halved because the Cb and Cr channels are sampled only on each alternating line in this scheme. Therefore, the data rate is the same. Cb and Cr are subsampled by a factor of 2 in the horizontal and vertical directions, respectively. There are three variants of the 4:2:0 scheme with different horizontal and vertical sitings.

[0066] In MPEG-2, Cb and Cr are co-addressed horizontally. Cb and Cr are also addressed between pixels in the vertical direction (gap addressing). In Joint Picture Experts Group (JPEG) / JPEG File Exchange Format (JFIF), H.261, and MPEG-1, Cb and Cr are gap-addressed, located between alternating luma samples. In 4:2:0 DV, Cb and Cr are co-addressed horizontally. In the vertical direction, they are co-addressed on alternating lines.

[0067] The definition of a video unit is provided. An image is divided into one or more tile rows and one or more tile columns. A tile is a sequence of codec tree units (CTUs) covering a rectangular area of ​​the image. A tile is divided into one or more bricks, each brick consisting of multiple CTU rows within the tile. A tile that is not divided into multiple bricks is also called a brick. However, a brick that is a proper subset of a tile is not called a tile. A stripe includes multiple tiles of an image or multiple bricks of a tile.

[0068] Two stripe modes are supported: raster scan stripe mode and rectangular stripe mode. In raster scan stripe mode, the stripe consists of a sequence of slices from a raster scan of the image. In rectangular stripe mode, the stripe consists of multiple tiles from the image, which together form a rectangular area of ​​the image. The tiles within the rectangular stripe are arranged according to the raster scan order of the stripe.

[0069] Figure 1 This is an example of raster scan strip segmentation of image 100, where the image is divided into twelve slices 102 and three raster scan strips 104. As shown, each slice 102 and strip 104 includes multiple CTUs 106.

[0070] Figure 2 This is an example of rectangular strip segmentation of image 200 according to the VVC specification, where the image is divided into twenty-four slices 202 (six slice columns 203 and four slice rows 205) and nine rectangular strips 204. As shown in the figure, each slice 202 and rectangular strip 204 contains multiple CTUs 206.

[0071] Figure 3 This is an example of dividing an image 300 into slices, tiles, and rectangular strips according to the VVC specification. The image is divided into four slices 302 (two slice columns 303 and two slice rows 305), eleven tiles 304 (the top left slice contains one tile, the top right slice contains five tiles, the bottom left slice contains two tiles, and the bottom right slice contains three tiles) and four rectangular strips 306.

[0072] The CTU and Codec Tree Block (CTB) sizes are discussed. In VVC, the CTU size, as small as 4×4, can be specified in the Sequence Parameter Set (SPS) signaling notification via the syntax element log2_ctu_size_minus2. The Sequence Parameter Set Raw Byte Sequence Payload (RBSP) syntax is as follows.

[0073]

[0074]

[0075]

[0076] log2_ctu_size_minus2 plus 2 specifies the size of the luminance codec tree block for each CTU.

[0077] log2_min_luma_coding_block_size_minus2 plus 2 specifies the minimum luma encoding / decoding block size.

[0078] The variables CtbLog2SizeY, CtbSizeY, MinCbLog2SizeY, MinCbSizeY, MinTbLog2SizeY, MaxTbLog2SizeY, MinTbSizeY, MaxTbSizeY, PicWidthInCtbsY, PicHeightInCtbsY, PicSizeInCtbsY, PicWidthInMinCbsY, PicHeightInMinCbsY, PicSizeInMinCbsY, PicSizeInSamplesY, PicWidthInSamplesC, and PicHeightInSamplesC are derived as follows:

[0079] CtbLog2SizeY = log2_ctu_size_minus2 + 2 (7-9)

[0080] CtbSizeY = 1 << CtbLog2SizeY (7-10)

[0081] MinCbLog2SizeY = log2_min_luma_coding_block_size_minus2 + 2 (7-11)

[0082] MinCbSizeY = 1 << MinCbLog2SizeY (7-12)

[0083] MinTbLog2SizeY = 2 (7-13)

[0084] MaxTbLog2SizeY = 6 (7-14)

[0085] MinTbSizeY = 1 << MinTbLog2SizeY (7-15)

[0086] MaxTbSizeY = 1 << MaxTbLog2SizeY (7-16)

[0087] PicWidthInCtbsY = Ceil( pic_width_in_luma_samples÷ CtbSizeY) (7-17)

[0088] PicHeightInCtbsY = Ceil(pic_height_in_luma_samples÷CtbSizeY ) (7-18)

[0089] PicSizeInCtbsY = PicWidthInCtbsY PicHeightInCtbsY (7-19)

[0090] PicWidthInMinCbsY = pic_width_in_luma_samples / MinCbSizeY (7-20)

[0091] PicHeightInMinCbsY = pic_height_in_luma_samples / MinCbSizeY (7-21)

[0092] PicSizeInMinCbsY = PicWidthInMinCbsY PicHeightInMinCbsY (7-22)

[0093] PicSizeInSamplesY=pic_width_in_luma_samples pic_height_in_luma_samples (7-23)

[0094] PicWidthInSamplesC = pic_width_in_luma_samples / SubWidthC (7-24)

[0095] PicHeightInSamplesC = pic_height_in_luma_samples / SubHeightC (7-25)

[0096] Figure 4A is an example of CTBs straddling the bottom picture border. Figure 4B is an example of CTBs straddling the right picture border. Figure 4C is an example of CTBs straddling the bottom-right picture border. In Figures 4A-4C there are K = M, L < N; K < M, L = N; K < M, L < N, respectively.

[0097] Refer to Figures 4A-4C discusses the CTUs in Picture 400. Assume that the CTB / largest coding unit (LCU) size is indicated by M×N (usually M equals N, as defined in HEVC / VVC), and for CTBs located at the picture (or slice or strip or other type, taking the picture border as an example) border, K×L samples are within the picture border, where K < M or L < N. For Figures 4A-4CThe CTBs depicted in the image 402 still have a size equal to M x N; however, the bottom / right boundary of the CTB is outside the image 400.

[0098] The encoding and decoding process of a typical video encoder / decoder (also known as a codec) is discussed. Figure 5 This is an example of a VVC encoder block diagram, containing three in-loop filtering blocks: Deblocking Filter (DF), Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF). Unlike DF, which uses predefined filters, SAO and ALF utilize the original samples of the current image by adding an offset and applying a Finite Impulse Response (FIR) filter, respectively, to reduce the mean square error between the original and reconstructed samples, thanks to signaling notification of the offset and filter coefficients' encoding / decoding side information. ALF is located at the final processing stage of each image and can be viewed as a tool to attempt to capture and repair artifacts established in previous stages.

[0099] Figure 5 This is a schematic diagram of encoder 500. Encoder 500 is suitable for implementing VVC technology. Encoder 500 includes three in-loop filters: a deblocking filter (DF) 502, a sample adaptive offset (SAO) 504, and an ALF 506. Unlike DF 502, which uses predefined filters, SAO 504 and ALF 506 utilize the original samples of the current image by adding an offset and applying an FIR filter, respectively, to reduce the mean square error between the original and reconstructed samples, thanks to the encoding / decoding side information of the offset and filter coefficients notified by signaling. ALF 506 is located at the final processing stage of each image and can be considered as a tool to attempt to capture and repair artifacts established in previous stages.

[0100] The encoder 500 also includes an intra-frame prediction component 508 and a motion estimation / compensation (ME / MC) component 510, configured to receive input video. The intra-frame prediction component 508 is configured to perform intra-frame prediction, while the ME / MC component 510 is configured to perform inter-frame prediction using a reference image obtained from a reference image buffer 512. Residual blocks from inter-frame or intra-frame prediction are fed to a transform component 514 and a quantization component 516 to generate quantized residual transform coefficients, which are then fed to an entropy codec component 518. The entropy codec component 518 entropy codes and decodes the prediction results and the quantized transform coefficients and sends them to a video decoder (not shown). Quantized components output from the quantization component 516 can be fed to an inverse quantization component 520, an inverse transform component 522, and a reconstruction (REC) component 524. The REC component 524 is capable of outputting images to a DF 502, a SAO 504, and an ALF 506 for filtering before these images are stored in the reference image buffer 512.

[0101] The input to the DF 502 is the reconstructed samples prior to the in-loop filter. First, vertical edges in the image are filtered. Then, using the samples modified through the vertical edge filtering process as input, horizontal edges in the image are filtered. Vertical and horizontal edges in the CTB of each CTU are processed separately on a codec unit basis. Vertical edges of the codec block in the codec unit are filtered starting from the left edge of the codec block and proceeding geometrically towards the right edge of the codec block. Horizontal edges of the codec block in the codec unit are filtered starting from the top edge of the codec block and proceeding geometrically towards the bottom edge of the codec block.

[0102] Figure 6 This is a diagram 600 showing sample point 602 within an 8×8 sample point block 604. As shown, diagram 600 includes horizontal block boundaries 606 and vertical block boundaries 608 on the 8×8 grid. Furthermore, diagram 600 depicts a non-overlapping block 610 of 8×8 sample points, which can be deblocked in parallel.

[0103] Boundary determination was discussed. The filter was applied to 8×8 block boundaries. Furthermore, it must be a transform block boundary or an encoding / decoding sub-block boundary (e.g., due to the use of affine motion prediction, optional temporal motion vector prediction (ATMVP)). For those boundaries that are not such boundaries, the filter was disabled.

[0104] Boundary strength calculation was discussed. For transform block boundaries / encoder / decoder sub-block boundaries, if they are located in an 8×8 grid, the transform block boundaries / encoder / decoder sub-block boundaries can be filtered, and the bS[xD] of that edge... i ][ yD j ] (where [xD i][ yD j The settings for (representing coordinates) are defined in Table 1 and Table 2, respectively.

[0105] Table 1. Boundary Strength (when SPS IBC is disabled)

[0106]

[0107] Table 2. Boundary Strength (when SPS IBC is enabled)

[0108]

[0109] The deblocking decision for the luminance component was discussed.

[0110] Figure 7 Example 700 shows the pixels involved in the filter on / off decision and strong / weak filter selection. A wider, stronger brightness filter is used only if conditions 1, 2, and 3 are all true. Condition 1 is the "bulk condition." This condition detects whether samples on the P-side and Q-side belong to a bulk, represented by the variables bSidePisLargeBlk and bSideQisLargeBlk, respectively. The definitions of bSidePisLargeBlk and bSideQisLargeBlk are as follows.

[0111] bSidePisLargeBlk = ((edge ​​type is vertical, and p0 belongs to CU with width >= 32) | | (edge ​​type is horizontal, and p0 belongs to CU with height >= 32)) ? TRUE: FLASE

[0112] bSideQisLargeBlk = ((edge ​​type is vertical, and q0 belongs to CU with width >= 32) | |(edge ​​type is horizontal, and q0 belongs to CU with height >= 32))? TRUE: FLASE

[0113] Based on bSidePisLargeBlk and bSideQisLargeBlk, condition 1 is defined as follows.

[0114] Condition 1 =(bsidepislageblk | | bsidepislageblk)? TRUE: FLASE

[0115] Next, if condition 1 is true, condition 2 will be further examined. First, the following variables are derived.

[0116] First, derive dp0, dp3, dq0, and dq3 in HEVC.

[0117] If (p-side is greater than or equal to 32)

[0118] dp0 = (dp0 + Abs(p50 - 2)) p40 + p30 + 1) >> 1

[0119] dp3 = (dp3 + Abs(p53 - 2)) p43 + p33 + 1) >> 1

[0120] If (q-side is greater than or equal to 32)

[0121] dq0 = (dq0 + Abs(q50 - 2)) q40 + q30 + 1) >> 1

[0122] dq3 = (dq3 + Abs(q53 - 2)) q43 + q33 + 1) >> 1

[0123] Condition 2 = (d < β) ? TRUE: FALSE

[0124] Where d = dp0 + dq0 + dp3 + dq3.

[0125] If conditions 1 and 2 are valid, then further check whether any block uses sub-blocks.

[0126] If (bSidePisLargeBlk)

[0127] {

[0128] If (mode block P == SUBBLOCKMODE)

[0129] Sp = 5

[0130] otherwise

[0131] Sp = 7

[0132] }

[0133] otherwise

[0134] Sp = 3

[0135] If (bSideQisLargeBlk)

[0136] {

[0137] If (mode block Q == SUBBLOCKMODE)

[0138] Sq = 5

[0139] otherwise

[0140] Sq = 7

[0141] }

[0142] otherwise

[0143] Sq = 3

[0144] Finally, if both conditions 1 and 2 are valid, the proposed deblocking method will check condition 3 (large block strong filtering condition), which is defined as follows.

[0145] In condition 3 StrongFilterCondition, the following variables are derived.

[0146] dpq is derived as in HEVC.

[0147] sp3 = Abs( p3 - p0 ), as derived in HEVC.

[0148] If (p-side is greater than or equal to 32)

[0149] If (Sp==5)

[0150] sp3 = ( sp3+ Abs( p5 - p3) + 1) >> 1

[0151] otherwise

[0152] sp3 = ( sp3+ Abs( p7 - p3) + 1) >> 1

[0153] sq3 = Abs( q0 - q3 ), as derived in HEVC.

[0154] If (q-side is greater than or equal to 32)

[0155] If (Sq==5)

[0156] sq3 = ( sq3+ Abs( q5 - q3) + 1) >> 1

[0157] otherwise

[0158] sq3 = ( sq3+ Abs( q7 - q3) + 1) >> 1

[0159] In HEVC, for example, StrongFilterCondition = (dpq < (β >> 2), sp3 + sq3 < (3 β>> 5), and Abs(p0 -q0) is less than (5). t C + 1 ) >> 1)? TRUE : FALSE.

[0160] A stronger deblocking filter for brightness (designed for larger blocks) was discussed.

[0161] A bilinear filter is used when samples on either side of the boundary belong to a large block. Samples belonging to a large block are defined when the width of the vertical edge is greater than or equal to 32 and the height of the horizontal edge is greater than or equal to 32.

[0162] Bilinear filters are listed below.

[0163] Then, in the above HEVC deblocking, for the block boundary sample points p from i=0 to Sp-1 i And block boundary sample points q from j=0 to Sq-1 j Replaced by linear interpolation, p i and q j It is the i-th sample in a row used for vertical edge filtering, or the j-th sample in a column used for horizontal edge filtering, as shown below.

[0164]

[0165]

[0166] in, and The item is a position-related clipping as described below, and is given below. , , , and .

[0167] The deblocking control of chroma was discussed.

[0168] A strong chroma filter is used on both sides of the block boundary. Here, the chroma filter is selected when both sides of the chroma edge are greater than or equal to 8 (chroma position), and the following decision with three conditions is satisfied: The first is a decision for boundary strength and large block size. The proposed filter can be applied when the width or height of the block orthogonal to the block edge in the chroma sampling domain is equal to or greater than 8. The second and third decisions for HEVC luminance deblocking are essentially the same, namely the on / off decision and the strong filtering decision, respectively.

[0169] In the first decision, the boundary strength (bS) of the chroma filter is modified, and the conditions are checked sequentially. If the condition is met, the remaining lower priority conditions are skipped.

[0170] Chromatic deblocking is performed when bS equals 2, or when a large block boundary is detected when bS equals 1.

[0171] The second and third conditions are essentially the same as those determined by HEVC luminance strong filtering, as shown below.

[0172] In the second condition: d is derived as shown in the HEVC luminance deblock. The second condition is true when d is less than β.

[0173] In the third condition, StrongFilterCondition is deduced as follows.

[0174] dpq is derived as described in HEVC.

[0175] sp3 = Abs( p3 - p0 ), as derived in HEVC.

[0176] sq3 = Abs( q0 - q3 ), as derived in HEVC.

[0177] In HEVC design, StrongFilterCondition = (dpq < (β >> 2), sp3 + sq3 < (β >> 3), Abs(p0 - q0) < (5 t C + 1 ) >> 1).

[0178] Strong deblocking filters for chroma are discussed. The following strong deblocking filters for chroma are defined.

[0179] p2′= (3 p3+2 p2+p1+p0+q0+4) >> 3

[0180] p1′= (2 p3+p2+2 p1+p0+q0+q1+4) >> 3

[0181] p0′= (p3+p2+p1+2 p0+q0+q1+q2+4) >> 3

[0182] The proposed chromaticity filter performs deblocking on a 4×4 chromaticity sample grid.

[0183] Position-related clipping (tcPD) is discussed. tcPD is applied to the output samples of the brightness filtering process, which involves modifying the strong and long filters at the boundaries of 7, 5, and 3 samples. Assuming a quantization error distribution, it is proposed to increase the clipping value of samples expected to have higher quantization noise, thus resulting in a larger deviation between the expected reconstructed sample values ​​and the true sample values.

[0184] For each P or Q boundary filtered using an asymmetric filter, depending on the result of the decision-making process in the boundary strength calculation, a position-related threshold table is selected from two tables (i.e., Tc7 and Tc3, tabulated below) provided to the decoder as edge information.

[0185] Tc7 = {6, 5, 4, 3, 2, 1, 1}; Tc3 = {6, 4, 2};

[0186] tcPD = (Sp == 3) ? Tc3 : Tc7;

[0187] tcQD = (Sq == 3) ? Tc3 : Tc7;

[0188] For P or Q boundaries filtered by a short symmetric filter, apply a lower amplitude position correlation threshold.

[0189] Tc3 = { 3, 2, 1};

[0190] After defining the threshold, the filtered filter is pruned based on the tcP and tcQ values. and The sample values ​​are clipped.

[0191]

[0192] in and These are the filtered sample values. and These are the output sample values ​​after cropping. and The clipping threshold is derived from the VVC tc parameters, tcPD, and tcQD. The Clip3 function is the clipping function defined in VVC.

[0193] Now we discuss sub-block deblocking adjustments. To achieve parallel-friendly deblocking using long filters and sub-block deblocking, the long filter is restricted to modifying a maximum of 5 samples on the side using sub-block deblocking (AFFINE or ATMVP or decoder-side motion vector refinement (DMVR)), as shown in the long filter's brightness control. Furthermore, sub-block deblocking is adjusted such that sub-block boundaries on the 8×8 grid near the codec unit (CU) or implicit TU boundary are restricted to modifying a maximum of two samples on each side.

[0194] The following applies to sub-block boundaries that are not aligned with the CU boundary.

[0195]

[0196] An edge of 0 corresponds to the CU boundary, an edge of 2 or orthogonalLength - 2 corresponds to 8 sample points of the sub-block boundary from the CU boundary, and so on. If implicit partitioning of TU is used, then implicit TU is true.

[0197] Sample Adaptive Offset (SAO) is discussed. The input to SAO is the reconstructed samples after deblocking (DB). The concept of SAO is to reduce the average sample distortion of a region by first classifying region samples into multiple categories using a selected classifier, obtaining an offset for each category, and then adding this offset to each sample of that category. The classifier index and the region offset are encoded and decoded in the bitstream. In HEVC and VVC, the region (the unit for SAO parameter signaling notification) is defined as a CTU.

[0198] HEVC employs two SAO types that meet low complexity requirements. These two types are Edge Offset (EO) and Band Offset (BO), which will be discussed in detail below. The index of the SAO type is encoded and decoded (within the range [0, 2]). For EO, ​​sample classification is based on a comparison between the current sample and its neighboring samples, according to a one-dimensional orientation pattern: horizontal, vertical, 135° diagonal, and 45° diagonal.

[0199] Figure 8 The four one-dimensional (1-D) orientation patterns 800 for EO sample point classification are shown: horizontal (EO classification = 0), vertical (EO classification = 1), 135° diagonal (EO classification = 2), and 45° diagonal (EO classification = 3).

[0200] For a given EO classification, each sample point within the CTB is classified into one of five categories. The current sample point value, labeled "c", is compared with its two neighbors along the selected 1-D pattern. The classification rules for each sample point are summarized in Table 3. Categories 1 and 4 are associated with local valleys and local peaks along the selected 1-D pattern, respectively. Categories 2 and 3 are associated with concave and convex corners along the selected 1-D pattern, respectively. If the current sample point does not belong to EO categories 1-4, it belongs to category 0, and SAO does not apply.

[0201] Table 3: Sampling classification rules for edge offset

[0202]

[0203] An adaptive loop filter based on geometric transformation (SAO) is discussed in the Joint Exploration Model (JEM). The input to the DB is the reconstructed samples after DB and SAO. The sample classification and filtering processes are based on the reconstructed samples after DB and SAO.

[0204] In JEM, a geometric transformation-based adaptive loop filter (GALF) with block-based filter adaptation is applied. For the luminance component, one of 25 filters is selected for each 2×2 block based on the direction and activity of the local gradient.

[0205] The filter shape was discussed. Figure 9 Examples of GALF filter shapes 900 are shown, including a 5×5 rhombus on the left, a 7×7 rhombus in the middle, and a 9×9 rhombus on the right. In JEM, up to three rhombus filter shapes can be selected for the luminance component (e.g., ...). Figure 9 (As shown). The image-level signaling notification index indicates the filter shape used for the luminance component. Each square represents a sample point, and Ci (i = 0~6 (left), 0~12 (middle), 0~20 (right)) represents the coefficient applied to that sample point. For the chrominance component in the image, a 5×5 rhombus is always used.

[0206] Block classification was discussed. Each 2×2 block was divided into one of 25 classes. The classification index C was based on its directionality D and validity. The quantization value is derived as follows.

[0207]

[0208] To calculate D and First, the gradients in the horizontal, vertical, and two diagonal directions are calculated using the 1-D Laplacian operator.

[0209]

[0210] Indices i and j refer to the coordinates of the top-left sample point in the 2×2 block, and This represents the reconstructed sample point at coordinates (i, j).

[0211] Then, the maximum and minimum values ​​of the gradients in the horizontal and vertical directions are set as follows:

[0212]

[0213] The maximum and minimum values ​​of the gradients in the two diagonal directions are set as follows:

[0214]

[0215] To derive the value of the directionality D, these values ​​are compared with each other and with two thresholds t1 and t2:

[0216] Step 1. If and If true, then Set as .

[0217] Step 2. If If yes, continue from step 3; otherwise, continue from step 4.

[0218] Step 3. If ,but Set as ;otherwise Set as .

[0219] Step 4. If ,but Set as ;otherwise It was set to 3.

[0220] Validity value The calculation is as follows:

[0221]

[0222] It is further quantized to the range of 0 to 4, and the quantized value is represented as

[0223] For the two chromaticity components in the image, no classification method is applied; that is, a single set of ALF coefficients is applied for each chromaticity component.

[0224] The geometric transformation of the filter coefficients is discussed.

[0225] Figure 10The relative coordinates 1000 for 5×5 rhombus filter support are shown—diagonal, vertical flip, and rotation (from left to right).

[0226] Before filtering each 2×2 block, the filter coefficients associated with the coordinates (k, l) are adjusted based on the gradient values ​​calculated for that block. Apply geometric transformations such as rotation, diagonal flip, and vertical flip. This is equivalent to applying these transformations to samples within the filter's support region. The idea is to make the different blocks applying ALF more similar by aligning their orientations.

[0227] Three geometric transformations are introduced: diagonal, vertical flip, and rotation.

[0228]

[0229] Where K is the size of the filter, and The coefficient coordinates are such that position (0, 0) is in the upper left corner and position (K-1, K-1) is in the lower right corner. Based on the gradient values ​​calculated for this block, the transform is applied to the filter coefficients f(k, l). Table 4 summarizes the relationship between the transform and the four gradients in the four directions.

[0230] Table 4: Mapping between gradients and transformations computed for a block

[0231]

[0232] Signaling notification of filter parameters is discussed. In JEM, GALF filter parameters are signaled for the first CTU, i.e., after the stripe header of the first CTU and before the SAO parameters. Up to 25 groups of luminance filter coefficients can be signaled. To reduce bit overhead, filter coefficients from different categories can be merged. Furthermore, the GALF coefficients of a reference image are stored and can be reused as GALF coefficients for the current image. The current image can optionally use the GALF coefficients stored for the reference image and bypass GALF coefficient signaling. In this case, only the index of one reference image is signaled, and the current image inherits the stored GALF coefficients of the indicated reference image.

[0233] To support GALF temporal prediction, a candidate list of GALF filter sets is maintained. At the beginning of decoding a new sequence, the candidate list is empty. After decoding an image, the corresponding filter set can be added to the candidate list. Once the size of the candidate list reaches the maximum allowed value (i.e., 6 in the current JEM), new filter sets overwrite the oldest sets in decoding order; that is, a first-in-first-out (FIFO) rule is applied to update the candidate list. To avoid duplication, a set is added to the list only if the corresponding image does not use GALF temporal prediction. To support temporal scalability, multiple candidate lists of filter sets exist, and each candidate list is associated with a temporal layer. More specifically, each array assigned by the temporal layer index (TempIdx) can form the filter set of previously decoded images with a lower TempIdx. For example, the k-th array is assigned to be associated with a TempIdx equal to k, and the k-th array contains only filter sets from images with TempIdx less than or equal to k. After an image is encoded or decoded, the set of filters associated with that image will be used to update the arrays associated with TempIdx that is equal to or higher than TempIdx.

[0234] Temporal prediction of GALF coefficients is used for inter-frame encoding / decoding frames to minimize signaling overhead. For intra-frame frames, temporal prediction is not available, and a set of 16 fixed filters is assigned to each class. To indicate the use of fixed filters, signaling is used to notify a flag for each class, and, if necessary, the index of the selected fixed filter. Even when a fixed filter is selected for a given class, adaptive filters... The coefficients can still be sent for that category, in which case the coefficients of the filter to be applied to the reconstructed image are the sum of the two sets of coefficients.

[0235] The filtering process for the luminance component can be controlled at the CU level. Signaling indicates whether GALF is applied to the luminance component of the CU. For the chrominance component, whether GALF is applied is only indicated at the image level.

[0236] The filtering process is discussed. On the decoder side, when GALF is enabled for a block, each sample within the block... The filtered values ​​produce the sample values ​​shown below. Where L represents the filter length, Represents the filter coefficients. This represents the decoded filter coefficients.

[0237] (10)

[0238] Figure 11This example illustrates the relative coordinates supported by a 5×5 diamond filter, assuming the current sample point's coordinates (i, j) are (0, 0). Sample points at different coordinates, filled with the same color, are multiplied by the same filter coefficients.

[0239] The geometric transformation-based adaptive loop filter (GALF) in VVC is discussed. In VVC test model 4.0 (VTM4.0), the filtering process of the adaptive loop filter is performed as follows:

[0240] (11)

[0241] Among them, sample points These are the input samples. These are the filtered output samples (i.e., the filtering result), and This represents the filter coefficients. In fact, in VTM 4.0, it is implemented using integer arithmetic for fixed-point precision calculations.

[0242] (12)

[0243] Where L represents the filter length, and where, These are the filter coefficients with fixed-point precision.

[0244] Compared to the design in JEM, the current design of GALF in VVC has the following main changes:

[0245] 1) Adaptive filter shapes have been removed. Only 7×7 filter shapes are allowed for the luminance component, and only 5×5 filter shapes are allowed for the chrominance component.

[0246] 2) Move the signaling notification of ALF parameters from the strip / picture level to the CTU level.

[0247] 3) The category index is calculated at a 4×4 level instead of 2×2. Furthermore, as proposed in JVET-L0147, a subsampling Laplacian calculation method for ALF classification is utilized. More specifically, it is not necessary to calculate the horizontal / vertical / 45-degree diagonal / 135-degree gradient for each sample point within a block. Instead, 1:2 subsampling is used.

[0248] Regarding filtering and reconstruction, the nonlinear ALF in the current VVC is discussed.

[0249] Equation (11) can be reformulated without affecting encoding / decoding efficiency as follows:

[0250] (13)

[0251] in These are the same filter coefficients as in equation (11) [except for] It is equal to 1 in equation (13), and equal to 1 in equation (11). ].

[0252] Using the above filter formula (13), VVC introduces nonlinearity to reduce the value at neighboring sample points by using a simple clipping function. Compared with the filtered current sample value ( The effect of large differences makes ALF more effective.

[0253] More specifically, the ALF filter is modified as follows:

[0254] (14)

[0255] in, It is a clipping function, and It depends on The pruning parameters for the filter coefficients. The encoder performs optimization to find the optimal... .

[0256] In the JVET-N0242 implementation, a pruning parameter is specified for each ALF filter. Furthermore, a pruning value is signaled for each filter coefficient. This means that up to 12 pruning values ​​can be signaled for each luminance filter in the bitstream, and up to 6 pruning values ​​can be signaled for each chroma filter.

[0257] To limit signaling notification costs and encoder complexity, only four fixed values ​​are used, which are the same for both INTER and INTRA stripes.

[0258] Because the variance of local differences in luminance is typically higher than that in chrominance, two distinct sets of luminance and chrominance filters are applied. The maximum sample value in each set is also introduced (here, 1024 for a 10-bit depth) so that clipping can be disabled when not necessary.

[0259] Table 5 provides a set of clipped values ​​used in the JVET-N0242 test. These four values ​​were selected by roughly dividing the full range of luminance sample values ​​(encoded in 10 bits) and the chrominance range from 4 to 1024 in the logarithmic domain.

[0260] More accurately, the brightness table of the clipping values ​​is obtained using the following formula:

[0261] AlfClip L Where M=2 10 And N=4 (15)

[0262] Similarly, the color table of clipping values ​​is obtained using the following formula:

[0263] AlfClip C Where M=2 10 N=4 and A=4 (16)

[0264] Table 5: Authorized Clipping Values

[0265]

[0266] The selected clipping values, corresponding to the clipping value indices in Table 5 above, are encoded and decoded using the Golomb encoding scheme in the "alf_data" syntax element. This encoding scheme is the same as that used for the filter indices.

[0267] A convolutional neural network-based loop filter for video encoding and decoding is discussed.

[0268] In deep learning, convolutional neural networks (CNNs or ConvNets) are a type of deep neural network most commonly used for analyzing visual images. They have been very successful in image and video recognition / processing, recommender systems, image classification, medical image analysis, and natural language processing.

[0269] CNNs are a regularized version of multilayer perceptrons. Multilayer perceptrons typically mean fully connected networks, meaning that every neuron in one layer is connected to all neurons in the next layer. This "full connectivity" makes them prone to overfitting data. Typical regularization methods involve adding some form of weight magnitude metric to the loss function. CNNs take a different approach to regularization: they leverage hierarchical patterns in the data and assemble more complex patterns using smaller and simpler ones. Therefore, CNNs are at the lower extremes in terms of connectivity and complexity.

[0270] Compared to other image classification / processing algorithms, CNNs use relatively little preprocessing. This means the network learns filters that are hand-designed in traditional algorithms. This independence from existing knowledge and human effort in feature design is a major advantage.

[0271] Deep learning-based image / video compression generally has two meanings: purely neural network-based end-to-end compression, and traditional frameworks enhanced by neural networks. Purely neural network-based end-to-end compression is discussed in Johannes Ballé, Valero Laparra, and Eero P. Simoncelli, “End-to-end optimization of nonlinear transform codes for perceptual quality,” 2016, Workshop on Picture Coding (PCS), pp. 1–5, Institute of Electrical and Electronics Engineers (IEEE), and in Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár, “Lossy image compression with compressive autoencoders,” arXiv prequel arXiv: 1703.00395 (2017).In Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong, and Wen Gao, “Fully Connected Network-Based Intra Prediction for Image Coding,” IEEE Transactions on Image Processing 27, 7 (2018), 3236–3247; Yuanying Dai, Dong Liu, and Feng Wu, “A convolutional neural network approach for post-processing in HEVC intra coding,” MMM. Springer, 28–39; Rui Song, Dong Liu, Houqiang Li, and Feng Wu, “Neural network-based arithmetic coding of intra prediction modes in HEVC,” VCIP. IEEE, 1–4; and J. Pfaff, P. Helle, D. Maniry, S. Kaltenstadler, W. Samek, H. Schwarz, D. Marpe and T. Wiegand, “Neural network based intra prediction for video coding,” Digital Image Processing Applications XLI, Vol. 10752, International Society for Optics and Photonics, 1075213, discuss the traditional framework enhanced by neural networks.

[0272] The first type typically employs an autoencoder-like structure, implemented through convolutional neural networks or recurrent neural networks. While relying solely on neural networks for image / video compression avoids any manual optimization or design, the compression efficiency may not be satisfactory. Therefore, research on the second type focuses on enhancing traditional compression frameworks by replacing or strengthening certain modules with neural networks. In this way, they can inherit the advantages of highly optimized traditional frameworks. For example, the fully connected network for intra-prediction proposed in HEVC is discussed in Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong, and Wen Gao, “Fully Connected Network-Based Intra Prediction for Image Coding,” IEEE Transactions on Image Processing 27, 7 (2018), pp. 3236-3247.

[0273] Besides intra-frame prediction, deep learning is also used to enhance other modules. For example, the intra-loop filters in HEVC have been replaced by convolutional neural networks, and satisfactory results have been achieved in Yuanying Dai, Dong Liu, and Feng Wu, “A convolutional neural network approach for post-processing in HEVC intra coding”, MMM. Springer, 28–39. Research in RuiSong, Dong Liu, Houqiang Li, and Feng Wu, “Neural network-based arithmetic coding of intra prediction modes in HEVC”, VCIP. IEEE, 1–4, applies neural networks to improve the arithmetic codec engine.

[0274] Intra-loop filtering based on convolutional neural networks (CNNs) is discussed. In lossy image / video compression, the reconstructed frame is an approximation of the original frame, and because the quantization process is irreversible, this leads to distortion in the reconstructed frame. To mitigate this distortion, a CNN can be trained to learn the mapping from the distorted frame to the original frame. In fact, training is necessary before deploying CNN-based intra-loop filtering.

[0275] Training was discussed. The goal of the training process is to find the optimal values ​​for the parameters, including the weights and biases.

[0276] First, an encoder / decoder (e.g., HM, JEM, VTM, etc.) is used to compress the training dataset to generate distorted reconstructed frames. Then, the reconstructed frames are fed into a CNN, and the cost is computed using the CNN's output and the ground truth frames (original frames). Common cost functions include Sum of Absolute Differences (SAD) and Mean Squared Error (MSE). Next, the gradient of the cost with respect to each parameter is derived using backpropagation. The parameter values ​​are updated using the gradients. This process is repeated until a convergence criterion is met. After training is complete, the derived optimal parameters are saved for use in the inference phase.

[0277] The convolution process is discussed. During convolution, the filter moves across the image from left to right and from top to bottom, changing a column of pixels horizontally and a row of pixels vertically. The amount of movement by which the filter is applied across the input image is called the stride, and it is almost always symmetrical in the height and width dimensions. For height and width movements, the default stride (multiple) in two dimensions is (1, 1).

[0278] Figure 12A This is an example architecture 1200 of the proposed CNN filter, and Figure 12B This is an example of constructing a ResBlock (1250). In most deep convolutional neural networks, ResBlocks are used as basic modules and stacked several times to build the final network. In one example, the ResBlock is obtained by combining convolutional layers, ReLU / PReLU activation functions, and convolutional layers, such as... Figure 12B As shown.

[0279] Inference is discussed. During the inference phase, distorted reconstructed frames are fed into the CNN and processed by the CNN model, whose parameters have been determined during the training phase. The input samples to the CNN can be reconstructed samples before or after DB, or before or after SAO, or before or after ALF.

[0280] Current CNN-based loop filtering has certain problems. For example, the NN model does not use or otherwise utilizes external information (e.g., information from the video codec, such as codec parameters and / or codec syntax, which are information beyond what the NN model itself generates) as attention or as an attention mechanism. Therefore, information including various codec parameters (such as predictions (e.g., prediction patterns, motion vectors, etc.), partitioning (e.g., segmentation information), etc.) may not be fully utilized, such as for recalibrating feature maps.

[0281] This document discloses techniques for addressing one or more of the aforementioned problems. For example, this disclosure provides one or more neural network (NN) filter models trained as encoding / decoding tools to improve the efficiency of video encoding / decoding. NN-based encoding / decoding tools can be used to replace or otherwise enhance modules included in a video codec. For example, NN models can be used as additional intra-frame prediction modes, inter-frame prediction modes, transform kernels, or loop filters. This disclosure also illustrates how to design NN models using external information such as prediction, partitioning, QP, etc. (as further described below) as attention or attention mechanisms, which will be further described below. It should be noted that NN models can be used as any encoding / decoding tool, such as NN-based intra-frame / inter-frame prediction, NN-based super-resolution, NN-based motion compensation, NN-based reference generation, NN-based fractional pixel interpolation, NN-based intra-loop / post-filtering, etc.

[0282] The following list of embodiments should be considered as examples for explaining general concepts. These embodiments should not be interpreted narrowly. Furthermore, these embodiments can be combined in any way.

[0283] In this disclosure, the NN model can include any kind of NN architecture, such as a convolutional neural network (CNN) or a fully connected NN, or a combination of a CNN and a fully connected NN. In the discussion below, the NN model may also be referred to as a CNN model.

[0284] In the following discussion, a video unit can be a sequence, picture, strip, slice, tile, sub-picture, CTU / CTB, CTU / CTB line, one or more CU / codec block (CB), one or more CTU / CTB, one or more Virtual Pipeline Data Units (VPDUs), or a sub-region within a picture / strip / slice / tile. A parent video unit represents a unit larger than a video unit. Typically, a parent unit will contain several video units; for example, when the video unit is a CTU, the parent unit can be a strip, a CTU line, multiple CTUs, etc. In some embodiments, a video unit can be a sample / pixel.

[0285] Figure 13This is an example of a process 1300 for generating filtered samples based on an NN filter model that receives encoding / decoding parameters (e.g., extrinsic information about the NN filter model) as input. The NN filter model has an attention mechanism based on the encoding / decoding parameter input. The attention mechanism is configured to generate or acquire attention, which is useful for mimicking cognitive attention and actually enhances certain parts of the data input to the NN filter while reducing other parts of the data input to the NN filter. For example, attention can be acquired by processing extrinsic information about the NN filter model (such as one or more encoding / decoding parameters). The attention is configured to be applied to one or more feature maps, such as by weighting the feature maps according to the attention.

[0286] exist Figure 13 In the process 1300 shown, at least some unfiltered samples are provided as input to the NN filter. In the example, the unfiltered samples are samples (e.g., pixels) of video units that have not undergone any filtering or have not undergone a sufficient amount of filtering. Therefore, the output of the NN filter can be filtered samples. The output of the NN filter is also based on an NN filter model generated using codec parameter inputs. The codec parameter inputs may be related to the reconstruction of the video unit, the partitioning or segmentation scheme of the video unit, the prediction mode of the video unit, the quantization parameter (QP) associated with the video unit, and / or the boundary strength parameter of the boundary of the video unit.

[0287] For example, convolutional layers can be used to extract features from encoding / decoding parameters (e.g., external information), or from both encoding / decoding parameters and internal information, such as features extracted within an NN filter model. At least one of the extracted features is used as attention obtained in the NN filter model.

[0288] Figure 14 This is an example of a process 1400 that applies attention, obtained using external information (such as encoding / decoding parameters), to feature maps of an NN filter model to provide recalibrated feature maps. The feature maps of an NN filter model are the result of applying filters to an input image (or feature maps provided as the output of prior layers). For example, in each layer of an NN filter model, the feature map is the output of that layer. Therefore, feature maps produced by layers within an NN filter model can be referred to as intermediate feature maps, while feature maps produced by the final layer of the NN filter model can be referred to as final feature maps. For example, the final output of an NN filter model can be the final feature map.

[0289] exist Figure 14 In the example, the operation performed by the (multiple) convolutional layers of the NN filter model is represented as External information (e.g., encoding / decoding parameters) is denoted as E, and internal information (e.g., feature maps extracted within the NN filter model) is denoted as I. The obtained attention is denoted as A. Procedure 1400 applies the attention A to the feature map G to generate a recalibrated feature map. .

[0290] Therefore, operation Attention A is applied to external information (e.g., encoding / decoding parameters) or the concatenation of external and internal information to extract or otherwise obtain attention. Attention A is then applied to feature map G to generate a recalibrated feature map. For example, applying attention A to feature map G effectively reweights feature map G (e.g., applying different weights to different elements of feature map G), and the recalibrated feature map... This is the reweighted feature map obtained by applying attention A to feature map G. In process 1400, N is the number of channels, W is the channel width, and H is the channel height.

[0291] exist Figure 14 In the NN filter model, the intermediate feature map is represented by G, where The attention gained is denoted as A, where This indicates the attention received. In this article, The symbol G represents G as Given the elements in the set, where A is... The elements in the given set.

[0292] For example, R is the domain to which a particular element in the attention or feature map belongs. R can be a domain that includes any real number or any integer real number. In some examples, the feature map is a two-dimensional or three-dimensional array. Therefore, the feature map G (such as an intermediate feature map G, where...) The expression indicates that each element of the feature map G belongs to R, and that there are N x W x H elements in the feature map G. In this example, the values ​​of N, W, and H indicate that the feature map G is a three-dimensional array with dimensions along each dimension having values ​​of N, W, and H respectively. In another example, such as attention A, where... Each element of attention A belongs to R, and attention A has W x H elements. In this example, the values ​​of W and H indicate that attention A is a two-dimensional array with dimensions having values ​​of W and H along each dimension.

[0293] In one example, the recalibrated feature map is generated based on the following:

[0294] , 1 i , 1 j , and 1 k .

[0295] In another example, the recalibrated feature map is generated based on the following:

[0296] , 1 i , 1 j , 1 k ,and This represents the mapping function applied to each element of the attention.

[0297] In yet another example, the recalibrated feature map is generated based on the following:

[0298] + , 1 i , 1 j , 1 k ,and This represents the mapping function applied to each element of the attention.

[0299] As mentioned above, residual blocks serve as the basic building blocks of neural network models, and multiple residual blocks can be stacked to construct the final network. For example, residual blocks can be obtained by combining convolutional layers, ReLU / PreLU activation functions, and other convolutional layers, such as... Figure 12B As shown. Therefore, in at least some examples, attention is applied only to the last layer of the residual block (e.g., the feature map output of the last layer), and not to every layer of the residual block. Also... Figure 15B An example of applying attention in this way is shown in the image, which will be discussed further below.

[0300] In another example, attention is applied only to a specified layer of the NN filter model (e.g., the feature map output by the specified layer), rather than to every layer of the NN filter model. The specified layer can be determined by the topology of the NN filter model.

[0301] A discussion of model selection is provided.

[0302] Example 1

[0303] 1. NN filter models can use external information such as reconstruction parameters, partitioning or segmentation parameters, prediction parameters, boundary strength parameters, QP, etc. (generally, encoding and decoding parameters related to the video unit being filtered) as input to obtain attention. For example, NN filter models have attention mechanisms based on encoding and decoding parameter inputs.

[0304] a. In one example, the convolutional layers of the NN filter model are used to extract features from external information or both external and internal information. Internal information refers to features extracted within the network, while external information refers to other information not obtainable from the network's internal features, such as encoding / decoding parameters related to the video unit being filtered. At least one of these extracted features will be used as attention. As described above, Figure 14 Illustrations of attention mechanisms based on some examples are provided, where the operations performed by (multiple) convolutional layers are represented as follows: External information is represented by E, internal information by I, and the attention received is represented by A.

[0305] i. In one example, E is one of the reconstruction parameters, partitioning or segmentation parameters, prediction parameters, boundary strength parameters, QP, etc.

[0306] ii. In one example, E can be any combination of reconstruction parameters, partitioning or segmentation parameters, prediction parameters, boundary strength parameters, QP, etc.

[0307] iii. In one example, I is the (multiple) intermediate feature maps of the NN model that will be recalibrated through the acquired attention.

[0308] iv. In one example, A = (E).

[0309] v. In one example, A = (E, I), where E and I are first concatenated and then fed into the convolutional layer.

[0310] vi. In one example, A = (E), where E is the concatenation of the reconstructed image and the segmented image. It is a two-layer convolutional neural network, where A is a single-channel feature map with the same spatial resolution as the feature map that A will be applied to.

[0311] b. In one example, the acquired attention is used to recalibrate the intermediate feature maps. The intermediate feature maps of the NN model are denoted as G, where , where N, W, and H are the number of channels, width, and height, respectively. The attention gained is denoted as A, where This indicates the attention received. In this context, The symbol indicating G is... Given the elements in the set, and A is... The elements in the given set.

[0312] i. In one example, the process of applying attention can be written as:

[0313] , 1 i , 1 j , 1 k ,in This is the recalibrated feature map.

[0314] ii. In one example, the process of applying attention can be written as:

[0315] , 1 i , 1 j , 1 k ,in It is the recalibrated feature map. This represents the mapping function applied to each element of the attention. The mapping function can be a sigmoid function, a hyperbolic tangent (e.g., tanh) function, etc. For example, the sigmoid function is a function with a special "S" shape or sigmoid curve. The hyperbolic tangent function is an example of a sigmoid function.

[0316] 1. In one example, different A and / or different f can be used for different channels of the feature map.

[0317] iii. In one example, the process of applying attention can be written as:

[0318] + , 1 i , 1 j , 1 k ,in It is the recalibrated feature map. This represents the mapping function applied to each element of the attention. The mapping function can be a sigmoid function, a hyperbolic tangent (e.g., tanh) function, etc.

[0319] 1. In one example, different A and / or different f can be used for different channels of the feature map.

[0320] iv. In one example, attention operations can be applied to a specific layer within the network.

[0321] 1. In one example, when the network contains residual blocks, the attention operation is applied only to the feature map from the last layer of each residual block.

[0322] Example 2

[0323] 2. In the second embodiment, the external attention mechanism described herein can be applied in various ways. For example, the proposed external attention mechanism can be used in any NN model for video encoding and decoding, such as NN-based intra / inter-frame prediction, NN-based super-resolution, NN-based motion compensation, NN-based reference generation, NN-based fractional pixel interpolation, NN-based intra / post-loop filtering, etc.

[0324] a. In one example, the external attention mechanism is used for in-loop filtering based on the neural network.

[0325] A first exemplary embodiment was discussed.

[0326] This first exemplary embodiment proposes an in-loop filtering method based on a convolutional neural network, incorporating adaptive model selection (DAM). The proposed deep in-loop filter utilizing DAM is an evolution of the previous paper JVET-V0100, introducing a new network architecture into the VTM-11.0+NewMCTF codebase. Compared to VTM-11.0+NewMCTF, the proposed method demonstrates reduced BD rates for {Y, Cb, Cr} under AI, RA, and LDB configurations.

[0327] In this first exemplary embodiment, a convolutional neural network (CNN) is used to construct in-loop filters to reduce distortion occurring during compression. The network structure is a revision of the network structure discussed in Yue Li, Li Zhang, and Kai Zhang's "Deep in-loop filter with adaptive model selection" (JVET-V0100). Similar to JVET-V0100, residual blocks are used as basic modules and are stacked several times to construct the final network. As a further development of JVET-V0100, an external attention mechanism is introduced into this paper, resulting in increased representational power at a similar model size. Furthermore, to handle different types of content, the individual networks are trained for different types of stripes and quality levels.

[0328] First exemplary embodiment and Figure 12A and Figure 12B The illustrated embodiment is relevant. To improve the architecture, an external attention mechanism is now introduced. Figure 15A and Figure 15B . Figure 15A This is a schematic block diagram of the architecture 1500 based on various examples of NN filtering methods, and Figure 15B This is shown based on various examples. Figure 15A A schematic block diagram of the construction of the attention residual block 1550 used in the architecture 1500.

[0329] Except for attention residual block 1550, the rest of architecture 1500 is identical to that in JVET-V0100. The computation process in attention module 1550 can be written as:

[0330] F_out = F_in f (Rec, Pred) + F_in

[0331] Where F_in and F_out represent the input and output of attention module 1550, respectively. Rec and Pred represent reconstruction and prediction, respectively. In this example, f consists of two convolutional layers, with an activation function applied after the first convolutional layer. The objective of f is to generate a spatial attention map based on external information and then recalibrate the feature map F_in.

[0332] In the technique of the first exemplary embodiment proposed, each strip or block can determine whether a CNN-based filter is applied. When it is determined that a CNN-based filter is applied to a strip / block, it can be further determined which model from a candidate model list comprising three models can be selected. For this purpose, each model is trained with QPs in {17, 22, 27, 32, 37, 42}. Given a sequence of test QPs (denoted as q), the candidate model list includes models trained with QPs equal to {q, q-5, q-10}. The selection process is based on the rate-distortion cost on the encoder side. If necessary, signaling instructions for on / off control and model indexes are communicated in the bitstream. Further details regarding model selection are provided below.

[0333] The granularity of filter determination and model selection depends on the resolution and QP. In the proposed scheme, given higher resolution and a larger QP, determination and selection will be performed over a larger region.

[0334] The list of candidate models differs at different temporal levels. For lower temporal levels, if the ratio of samples encoded and decoded intra-frame is not less than a threshold, the third candidate is replaced by the intra-strip NN filter model. For higher temporal levels, the third candidate (i.e., the model corresponding to q-10) is replaced by the model corresponding to q+5.

[0335] For full-frame configurations, model selection is disabled, but on / off control is still retained.

[0336] Open-source machine learning frameworks (such as PyTorch) are useful for performing inference of the CNN filters proposed in VTM. Table 1-1 provides network information during the inference phase, as suggested in “JVET common test conditions and evaluation procedures for neural network-based video coding technology” (JVET-V2016) by S. Liu, A. Segall, E. Alshina, and R.-L. Liao.

[0337] Table 1-1. Network information tested in the inference phase for NN-based video encoding / decoding tools

[0338]

[0339] PyTorch can also be used as a training platform. The DIV2K dataset (from https: / / data.vision.ee.ethz.ch / cvl / DIV2K / , R. Timofte, E. Agustsson, S. Gu, J. Wu, A. Ignatov, LV Gool) and the BVI-DVC dataset (from “BVI-DVC: A Training Database for Deep Video Compression” by Ma, Di, Fan Zhang, and David R. Bull, arXiv pre-publication arXiv:2003.13552 (2020)) were used to train I-strip and B-strip CNN filters, respectively. Different CNN models were trained to adapt to different QP points. Table 1-2 provides network information during the training phase, as recommended in JVET-V2016 by S. Liu, A. Segall, E. Alshina, and R.-L. Liao, “JVET common test conditions and evaluation procedures for neural network-based video coding technology”.

[0340] Table 1-2. Network information for testing NN-based video encoding / decoding tools during the training phase.

[0341]

[0342] Based on the common test conditions recommended in "JVET common test conditions and evaluation procedures for neural network-based video coding technology" (JVET-V2016) by S. Liu, A. Segall, E. Alshina, and R.-L. Liao, the proposed CNN-based in-loop filtering method was tested on VTM-11.0+NewMCTF (from https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM / - / tags / VTM-11.0). NewMCTF patches are from https: / / vcgit.hhi.fraunhofer.de / jvet-ahg-nnvc / nnvc-ctc / - / tree / master / Software%20Patches.

[0343] Deblocking filtering and SAO were disabled when ALF (and CCALF) were placed after the proposed CNN-based filtering. Tables 1-3 to 1-5 show the test results. In the AI ​​configuration, the proposed method reduced the BD rates of Y, Cb, and Cr by an average of 9.12%, 22.39%, and 22.60%, respectively. In the RA configuration, the proposed method reduced the BD rates of Y, Cb, and Cr by an average of 12.32%, 27.48%, and 27.22%, respectively. In the LDB configuration, the proposed method reduced the BD rates of Y, Cb, and Cr by an average of %, %, and %, respectively.

[0344] Table 1-3. Performance (RA) of the proposed method on VTM11.0+NewMCTF

[0345]

[0346] Table 1-4. Performance (LDB) of the proposed method on VTM11.0+NewMCTF

[0347]

[0348] Table 1-5. Performance of the proposed method on VTM11.0 + NewMCTF (AI)

[0349]

[0350] A first exemplary embodiment proposes a CNN-based in-loop filtering method. The proposed CNN-based filtering method demonstrates useful encoding / decoding gains.

[0351] Turn now Figure 16 This is a block diagram illustrating an example video processing system 1600 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of the video processing system 1600. The video processing system 1600 may include an input 1602 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8 or 10-bit multi-component pixel values, or it may be in a compressed or encoded format. The input 1602 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, Passive Optical Network (PON), and wireless interfaces such as Wi-Fi or cellular interfaces.

[0352] Video processing system 1600 may include codec component 1604 which can implement the various coding or encoding methods described in this document. Codec component 1604 can reduce the average bit rate of the video from input 1602 to the output of codec component 1604 to produce a codec representation of the video. Codec techniques are therefore sometimes referred to as video compression or video transcoding techniques. The output of codec component 1604 may be stored or transmitted via a communication connection such as represented by component 1606. The bitstream (or codec) representation of the video received at input 1602, whether stored or communicated, can be used by component 1608 to generate pixel values ​​or transmit as displayable video to display interface 1610. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “codec” operations or tools, it will be understood that codec tools or operations are used at the encoder, and the corresponding decoding tools or operations that inversely represent the codec results will be performed by the decoder.

[0353] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), Peripheral Component Interconnect (PCI), and Integrated Drive Electronic Devices (IDE) interfaces. The technologies described in this document can be found in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0354] Figure 17This is a block diagram of a video processing apparatus 1700. Apparatus 1700 can be used to implement one or more methods described herein. Apparatus 1700 can be embodied in a smartphone, tablet, computer, Internet of Things (IoT) receiver, etc. Apparatus 1700 may include one or more processors 1702, one or more memories 1704, and video processing hardware 1706 (also known as video processing circuitry). The processors (multiple) 1702 can be configured to implement one or more methods described in this document. The memories (multiple memories) 1704 can be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 1706 can be used to implement some of the techniques described in this document in a hardware circuitry system. In some embodiments, hardware 1706 may be partially or entirely located within processor 1702 (e.g., a graphics processor).

[0355] Figure 18 This is a block diagram illustrating an example video codec system 1800 that can utilize the techniques disclosed herein. Figure 18 As shown, the video encoding / decoding system 1800 may include a source device 1810 and a target device 1820. The source device 1810 generates encoded video data, and this source device 1810 may be referred to as a video encoding device. The target device 1820 can decode the encoded video data generated by the source device 1810, and the target device 1820 may be referred to as a video decoding device.

[0356] The source device 1810 may include a video source 1812, a video encoder 1814, and an input / output (I / O) interface 1816.

[0357] Video source 1812 may include sources such as video capture devices, interfaces for receiving video data from video content providers, and / or computer graphics systems for generating video data, or combinations of these sources. Video data may include one or more pictures. Video encoder 1814 encodes the video data from video source 1812 to generate a bitstream. The bitstream may include a sequence of bits forming a codec representation of the video data. The bitstream may include codec pictures and related data. A codec picture is a codec representation of a picture. Related data may include sequence parameter sets, picture parameter sets, and other syntax structures. I / O interface 1816 may include a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be transmitted directly to target device 1820 via network 1830 through I / O interface 1816. Encoded video data may also be stored on storage medium / server 1840 for access by target device 1820.

[0358] The target device 1820 may include an I / O interface 1826, a video decoder 1824, and a display device 1822.

[0359] I / O interface 1826 may include a receiver and / or a modem. I / O interface 1826 may acquire encoded video data from source device 1810 or storage medium / server 1840. Video decoder 1824 may decode the encoded video data. Display device 1822 may display the decoded video data to a user. Display device 1822 may be integrated with target device 1820 or may be external to target device 1820, which may be configured to interface with an external display device.

[0360] The video encoder 1814 and video decoder 1824 can operate according to video compression standards, such as the High Efficiency Video Codec (HEVC) standard, the Universal Video Codec (VVC) standard, and other current and / or additional standards.

[0361] Figure 19 This is a block diagram illustrating an example of a video encoder 1900, which can be... Figure 18 The video encoder 1814 in the video encoding and decoding system 1800 shown.

[0362] The video encoder 1900 can be configured to perform any or all of the techniques disclosed herein. Figure 19 In the example, the video encoder 1900 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 1900. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0363] The functional components of the video encoder 1900 may include a segmentation unit 1901, a prediction unit 1902 (which may include a mode selection unit 1903, a motion estimation unit 1904, a motion compensation unit 1905, and an intra-frame prediction unit 1906), a residual generation unit 1907, a transform unit 1908, a quantization unit 1909, an inverse quantization unit 1910, an inverse transform unit 1911, a reconstruction unit 1912, a buffer 1913, and an entropy coding unit 1914.

[0364] In other examples, the video encoder 1900 may include more, fewer, or different functional components. In one example, the prediction unit 1902 may include an intra-block copy (IBC) unit. The IBC unit can perform prediction in IBC mode, where at least one reference picture is the picture containing the current video block.

[0365] Furthermore, some components, such as the motion estimation unit 1904 and the motion compensation unit 1905, can be highly integrated, but for illustrative purposes, in Figure 19 The examples are shown separately.

[0366] Segmentation unit 1901 can segment an image into one or more video blocks. Figure 18 The video encoder 1814 and video decoder 1824 can support various video block sizes.

[0367] The mode selection unit 1903 can select one of the encoding / decoding modes (e.g., intra-frame or inter-frame) based on the error result, and provide the resulting intra-frame or inter-frame codec block to the residual generation unit 1907 to generate residual block data, and to the reconstruction unit 1912 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 1903 can select a combination of intra-frame and inter-frame prediction modes (CIIP), where prediction is based on inter-frame prediction signals and intra-frame prediction signals. In the case of inter-frame prediction, the mode selection unit 1903 can also select the resolution of the block's motion vector (e.g., sub-pixel or integer pixel precision).

[0368] To perform inter-frame prediction on the current video block, motion estimation unit 1904 can generate motion information for the current video block by comparing one or more reference frames from buffer 1913 with the current video block. Motion compensation unit 1905 can determine the predicted video block for the current video block based on the motion information and decoded samples of images from buffer 1913 other than the image associated with the current video block.

[0369] The motion estimation unit 1904 and motion compensation unit 1905 can perform different operations on the current video block, depending on whether the current video block is in an I-band, P-band, or B-band. I-bands (or I-frames) are the least compressible but do not require other video frames for decoding. S-bands (or P-frames) can be decompressed using data from previous frames and are more compressible than I-frames. B-bands (or B-frames) can use previous and forward frames for data reference to achieve the highest data compression.

[0370] In some examples, motion estimation unit 1904 can perform unidirectional prediction on the current video block, and can search for reference images in list 0 or list 1 for reference video blocks of the current video block. Motion estimation unit 1904 can then generate a reference index indicating the reference image in list 0 or list 1, which contains the reference video block and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 1904 can output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 1905 can generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0371] In other examples, motion estimation unit 1904 can perform bidirectional prediction on the current video block. Motion estimation unit 1904 can search for a reference video block for the current video block in the reference images in list 0, and can also search for another reference video block for the current video block in list 1. Motion estimation unit 1904 can then generate a reference index indicating the reference images in lists 0 and 1 containing the reference video blocks, and a motion vector indicating the spatial displacement between the reference video blocks and the current video block. Motion estimation unit 1904 can output the reference index and motion vector of the current video block as motion information for the current video block. Motion compensation unit 1905 can generate a predicted video block for the current video block based on the reference video blocks indicated by the motion information of the current video block.

[0372] In some examples, the motion estimation unit 1904 can output a complete set of motion information for use in the decoder's decoding process.

[0373] In some examples, the motion estimation unit 1904 may not output the complete set of motion information for the current video. Instead, the motion estimation unit 1904 may refer to motion information signaling from another video block to inform the motion information of the current video block. For example, the motion estimation unit 1904 may determine that the motion information of the current video block is sufficiently similar to the motion information of neighboring video blocks.

[0374] In one example, the motion estimation unit 1904 may indicate a value in the syntax structure associated with the current video block that indicates to the video decoder 1024 that the current video block has the same motion information as another video block.

[0375] In another example, motion estimation unit 1904 can identify another video block and motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. Video decoder 1824 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0376] As discussed above, the video encoder 1814 can predictively signal motion vectors. Two examples of predictive signaling notification techniques that can be implemented by the video encoder 1814 include Advanced Motion Vector Prediction (AMVP) and Merge Pattern Signaling Notification.

[0377] The intra-prediction unit 1906 can perform intra-prediction on the current video block. When the intra-prediction unit 1906 performs intra-prediction on the current video block, it can generate prediction data for the current video block based on the decoded samples of other video blocks in the same frame. The prediction data for the current video block may include the predicted video block and various syntax elements.

[0378] The residual generation unit 1907 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) multiple predicted video blocks from the current video block. The residual data for the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0379] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 1907 may not perform the subtraction operation.

[0380] Transform unit 1908 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0381] After the transform unit 1908 generates a transform coefficient video block associated with the current video block, the quantization unit 1909 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0382] The inverse quantization unit 1910 and the inverse transform unit 1911 can apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 1912 can add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 1902 to produce a reconstructed video block associated with the current block, which is stored in the buffer 1913.

[0383] After the video block is reconstructed by the reconstruction unit 1912, a loop filtering operation can be performed to reduce video blocking artifacts in the video block.

[0384] The entropy coding unit 1914 can receive data from other functional components of the video encoder 1900. When the entropy coding unit 1914 receives data, it can perform one or more entropy coding operations to generate entropy-coded data and output a bit stream including the entropy-coded data.

[0385] Figure 20 This is a block diagram illustrating an example of a video decoder 2000. The video decoder 2000 can be... Figure 18 The video decoder 1824 in the video codec system 1800 shown.

[0386] The video decoder 2000 can be configured to perform any or all of the technologies disclosed herein. Figure 20In the example, the video decoder 2000 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 2000. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0387] exist Figure 20 In the example, the video decoder 2000 includes an entropy decoding unit 2001, a motion compensation unit 2002, an intra-frame prediction unit 2003, an inverse quantization unit 2004, an inverse transform unit 2005, a reconstruction unit 2006, and a buffer 2007. In some examples, the video decoder 2000 can perform functions typically associated with the video encoder 1814 (…). Figure 18 The encoding process described is the opposite of the decoding process.

[0388] Entropy decoding unit 2001 can retrieve the encoded bitstream. The encoded bitstream may include entropy-coded video data (e.g., encoded blocks of video data). Entropy decoding unit 2001 can decode the entropy-coded video data, and based on the entropy-coded video data, motion compensation unit 2002 can determine motion information including motion vectors, motion vector precision, reference image list index, and other motion information. Motion compensation unit 2002 can determine such information, for example, by executing AMVP and Merge mode signaling notifications.

[0389] The motion compensation unit 2002 can generate motion compensation blocks, possibly by performing interpolation based on an interpolation filter. The identifier of the interpolation filter to be used at sub-pixel precision can be included in the syntax element.

[0390] The motion compensation unit 2002 can use an interpolation filter, such as that used by the video encoder 1814 during the encoding of a video block, to calculate the interpolation of sub-integer pixels of the reference block. The motion compensation unit 2002 can determine the interpolation filter used by the video encoder 1814 based on the received syntax information, and use the interpolation filter to generate the prediction block.

[0391] The motion compensation unit 2002 may use some syntax information to determine the size of the blocks used to encode (multiple) frames and / or (multiple) stripes of the encoded video sequence, segmentation information describing how each macroblock of the picture of the encoded video sequence is segmented, a pattern indicating how each segment is encoded, one or more reference frames (and a list of reference frames) for each inter-frame coded block, and other information used to decode the encoded video sequence.

[0392] Intra-prediction unit 2003 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Inverse quantization unit 2004 performs inverse quantization, i.e., dequantization, on the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 2001. Inverse transform unit 2005 applies an inverse transform.

[0393] The reconstruction unit 2006 can add the residual block to the corresponding prediction block generated by the motion compensation unit 2002 or the intra-frame prediction unit 2003 to form a decoded block. If necessary, a deblocking filter can also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in the buffer 2007 to provide a reference block for subsequent motion compensation / intra-frame prediction, and also generates the decoded video for presentation on the display device.

[0394] Figure 21 This is a method 2100 for encoding and decoding video data according to embodiments of the present disclosure. Method 2100 can be performed by an encoding / decoding device (e.g., an encoder) having a processor and memory. Method 2100 can be implemented to provide a neural network (NN) filter model that uses external information (e.g., encoding / decoding parameters) as an attention mechanism. Such an NN filter model allows the use of this external information to recalibrate one or more feature maps generated by the NN filter.

[0395] In block 2102, the codec device applies a neural network (NN) filter to the unfiltered samples of the video unit to generate filtered samples. The NN filter is based on an NN filter model with an attention mechanism based on the input of the codec parameters. In an embodiment, unfiltered samples are samples (or pixels) that have not undergone any filtering process or have not been sufficiently filtered. For example, unfiltered samples have not been processed by any NN filter. As another example, unfiltered samples have not been processed by an NN filter, an adaptive loop filter (ALF), a deblocking filter (DF), a sample adaptive offset (SAO) filter, or a combination thereof.

[0396] In box 2104, the encoding / decoding device converts between the video media file and the bitstream based on the generated filtered samples.

[0397] When implemented in an encoder, the conversion involves receiving a media file (e.g., a video unit) and encoding the filtered samples into a bitstream. When implemented in a decoder, the conversion involves receiving a bitstream including the filtered samples and decoding the bitstream to obtain the filtered samples.

[0398] In an embodiment, method 2100 may utilize or combine one or more features or processes of other methods disclosed herein.

[0399] In one example, a non-transitory computer-readable medium stores a bitstream of video generated by methods (such as all or part of method 2100) performed by a video processing apparatus (e.g., video processing apparatus 1700 as described above). For example, a bitstream can be generated by applying an NN filter to unfiltered samples of video units. As described above, the NN filter is based on an NN filter model generated using a quality level indicator (QI) input, and the bitstream is generated based on the filtered samples.

[0400] The following is a list of preferred solutions for some embodiments.

[0401] The following solutions illustrate example embodiments of the techniques discussed in this disclosure (e.g., Example 1).

[0402] 1. A video processing method comprising: performing a rule-based conversion between a video comprising video blocks and a bitstream of the video, wherein the conversion comprises filtering at least some samples of the video blocks using a neural network (NN) filter, the neural network (NN) filter using one or more NN models, and wherein the rule specifies that the one or more NN models are equipped with an attention mechanism based on external information of the video.

[0403] 2. According to the method of Solution 1, the rule stipulates that the convolutional layers of the NN are used to extract features from external information, and the extracted features E are used as attention A in one or more NN models.

[0404] 3. According to the method of Solution 2, the rule specifies that internal information I is used to determine A.

[0405] 4. According to the method of Solution 3, where I includes intermediate feature maps of one or more NN models.

[0406] 5. According to the method of any one of solutions 2-4, where the rule stipulates that A is obtained only from E and not from I.

[0407] 6. According to the method of any one of solutions 2-4, wherein the rule stipulates that A is obtained by connecting E and I in series.

[0408] 7. According to the method of any one of solutions 2-6, wherein the rule specifies that A is obtained using a two-layer convolutional neural network.

[0409] 8. The method according to any one of solutions 2-7, wherein the transformation includes: using attention A to recalibrate the intermediate feature maps of the video.

[0410] 9. According to the method of Solution 8, wherein the intermediate feature maps of the one or more NN models are represented as Where N, W, and H are the number of channels, width, and height, respectively, and the attention A is represented as... .

[0411] 10. According to the method of Solution 9, where attention is applied: 1 i 1 j 1 k ,in This is the recalibrated feature map.

[0412] 11. The method of claim 10, wherein the process of applying attention is: 1 i 1 j 1 k ,in This is the recalibrated feature map. This represents the mapping function applied to each element of the attention.

[0413] 12. According to the method of solution 11, where the function is a sigmoid function or a tanh function.

[0414] 13. According to the method of solutions 11-12, different A and / or different f are used for different channels of the feature map.

[0415] 14. According to the method of Solution 10, where attention is applied: + 1 i 1 j 1 k ,in This is the recalibrated feature map. This represents the mapping function applied to each element of the attention.

[0416] 15. According to the method of solution 14, different A and / or different f can be used for feature maps of different channels.

[0417] 16. According to the method of solutions 10-15, the attention operation can be applied to a specified layer inside the NN.

[0418] 17. The method of any one of solutions 10-16, wherein the rule stipulates that when the NN contains residual blocks, the attention operation is applied only to the feature map from the last layer of each residual block.

[0419] 18. According to the method of solution 14, where the function is a sigmoid function or a tanh function.

[0420] 19. The method of any one of solutions 1-18, wherein the external information includes a partitioning scheme for video blocks.

[0421] 20. The method according to Solution 1-18, wherein the external information includes the prediction pattern of the video block.

[0422] 21. The method of any one of solutions 1-18, wherein the external information includes quantization parameters associated with the video block.

[0423] 22. The method of any one of solutions 1-21, wherein the external information includes boundary strength parameters of the boundaries of the video block.

[0424] 23. The method according to any one of solutions 1-22, wherein the conversion includes generating a bitstream from the video.

[0425] 24. The method according to any one of solutions 1-22, wherein the conversion includes generating video from a bitstream.

[0426] 25. A video decoding apparatus, comprising a processor configured to implement one or more of the methods described in solutions 1 to 24.

[0427] 26. A video encoding apparatus comprising a processor configured to implement one or more of the methods described in solutions 1 to 24.

[0428] 27. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to implement the method of any one of solutions 1 to 24.

[0429] 28. A computer-readable medium having a bitstream stored thereon, the bitstream being generated by any one of solutions 1 to 24.

[0430] 29. A method for generating a bitstream, comprising generating a bitstream using one or more of solutions 1-24 and writing the bitstream to a computer-readable medium.

[0431] 30. The methods, apparatus or systems described in this document.

[0432] The disclosed and other solutions, examples, embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in this document and their structural equivalents), or in a combination of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for use by a data processing apparatus to operate or control the operation of the data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances affecting machine-readable propagation signals, or a combination of one or more of them. The term "data processing apparatus" includes all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an operating environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. Propagation signals are artificially generated signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device.

[0433] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or code sections). Computer programs can be deployed to run on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communications network.

[0434] The processes and logic described in this document can be executed by one or more programmable processors running one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic can also be executed by dedicated logic circuits, and the devices can be implemented as dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits).

[0435] Processors suitable for running computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from, transfer data to, or receive data from and transfer data to such mass storage devices. However, a computer does not require such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and optical disc read-only memory (CD-ROM) and digital versatile disc read-only memory (DVD-ROM) disks. Processors and memory can be supplemented by dedicated logic circuits or incorporated into dedicated logic circuits.

[0436] While this patent document contains numerous details, these details should not be construed as limiting any subject matter or potentially claimed scope, but rather as descriptions of features specific to particular embodiments of a particular art. Certain features described in this patent document within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be excluded from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0437] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring the operations to be performed in the specific order shown or in a sequential manner, or as performing all shown operations to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0438] Only some implementation methods and examples are described, and other implementation methods, enhancements and variations can be made based on the content described and shown in this patent document.

[0439] While this patent document contains numerous details, these details should not be construed as limiting any subject matter or potentially claimed scope, but rather as descriptions of features specific to particular embodiments of a particular art. Certain features described in this patent document within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be excluded from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0440] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring the operations to be performed in the specific order shown or in a sequential manner, or as performing all shown operations to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0441] Only some implementation methods and examples are described, and other implementation methods, enhancements and variations can be made based on the content described and shown in this patent document.

Claims

1. A method implemented by a video encoding / decoding device, comprising: A neural network (NN) filter is applied to unfiltered samples of a video unit to generate filtered samples, wherein the NN filter is based on an NN filter model configured to acquire attention based on encoding / decoding parameter inputs; and The conversion between video media files and bitstreams is performed based on the generated filtered samples.

2. The method according to claim 1, wherein, The encoding / decoding parameter inputs include one or more selected from the group consisting of: a segmentation scheme for the video unit; a prediction mode for the video unit; quantization parameters associated with the video unit; and boundary strength parameters for the boundaries of the video unit.

3. The method according to claim 1, further comprising: The convolutional layer using the NN filter extracts features from the encoding / decoding parameter input; as well as The extracted features are used as attention in the NN filter model.

4. The method according to claim 3, wherein, The intermediate feature map of the NN filter model is recalibrated by the attention, wherein the attention is obtained by concatenating the encoding / decoding parameter input with the intermediate feature map to provide a concatenation result, and feeding the concatenation result into the convolutional layer of the NN filter.

5. The method according to claim 3, wherein, The attention is obtained using a two-layer convolutional neural network, wherein the attention is a single-channel feature map having the same spatial resolution as the intermediate feature map of the NN filter model that will be recalibrated by the attention.

6. The method of claim 3, further comprising using the attention to recalibrate intermediate feature maps of the NN filter model, wherein, The intermediate feature map of the NN filter model is given as G, where Where N is the number of channels, W is the channel width, and H is the channel height, and the attention obtained is given as A, where .

7. The method according to claim 6, wherein, This represents the recalibrated intermediate feature map, wherein applying the attention includes: according to To provide recalibrated intermediate feature maps, where 1 i , of which 1 j And one of them k .

8. The method according to claim 6, wherein, This represents the recalibrated intermediate feature map, wherein applying the attention includes: according to To provide recalibrated intermediate feature maps, where 1 i , of which 1 j , of which 1 k , and among them, This represents the mapping function applied to each element of the attention.

9. The method according to claim 8, wherein, The mapping function This includes the sigmoid function or the hyperbolic tangent function.

10. The method according to claim 8, wherein, Different A values ​​are used for different channels of the intermediate feature map. .

11. The method according to claim 6, wherein, This represents the recalibrated intermediate feature map, wherein applying the attention includes: according to + To provide recalibrated intermediate feature maps, where 1 i , of which 1 j , of which 1 k , and among them, This represents the mapping function applied to each element of the attention.

12. The method according to claim 11, wherein, The mapping function This includes the sigmoid function or the hyperbolic tangent function.

13. The method according to claim 11, wherein, Different A values ​​are used for different channels of the intermediate feature map. .

14. The method according to any one of claims 6-13, wherein, The attention is applied to a specified layer within the NN filter model.

15. The method according to claim 14, wherein, The NN filter model comprises residual blocks, wherein the attention is applied only to the feature map from the last layer of each residual block.

16. The method according to any one of claims 1-13, wherein, The NN filter is selected from one or more of the following: adaptive loop filter, deblocking filter, and sample adaptive offset filter.

17. The method according to any one of claims 1-13, wherein, The conversion includes generating the bitstream based on the video media file.

18. The method according to any one of claims 1-13, wherein, The conversion includes parsing the bitstream to obtain the video media file.

19. An apparatus for encoding and decoding video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, When executed by a processor, the instructions cause the processor to perform the method according to any one of claims 1 to 18.

20. A non-transitory computer-readable medium having a computer program and a bit stream stored thereon, wherein, When the computer program is executed by the video processing device, it generates the bitstream using the method described in any one of claims 1 to 18.

21. A non-transitory computer-readable storage medium for storing instructions, said instructions causing a processor to perform the method of any one of claims 1-18.

22. A method for storing a video bitstream, comprising: The bitstream is generated by performing the method according to any one of claims 1-18; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

Images