Reducing the complexity of video encoding and decoding

Abstract

A method for encoding or decoding video is provided. The method includes obtaining values ​​of components of reconstructed samples. The method includes obtaining first additional input data. The method includes providing the obtained values ​​of the components of the reconstructed samples to a first group of one or more convolutional layers, thereby generating outputs of the first group of convolutional layers. The method further includes providing the obtained first additional input data to a second group of one or more convolutional layers, thereby generating outputs of the second group of convolutional layers. The method includes encoding or decoding the video based on outputs of the first group of convolutional layers and outputs of the second group of convolutional layers. The first group of convolutional layers and the second group of convolutional layers differ in number of channels, kernel filter size, and / or number of convolutional layers.

Description

[Technical Field]

[0001] FIELD This disclosure relates to reducing the complexity of video encoding and decoding. [Background technology]

[0002] Video is the dominant form of data traffic in today's networks and is expected to continue to increase its share, as disclosed in P. Cervall et al., Ericsson Mobility Report, https: / / www.ericsson.com / en / mobility-report, November 2019. One way to reduce data traffic from video is compression. In compression, source video is encoded into a bitstream, which can then be stored and transmitted to an end user. Using a decoder, the end user can extract the video data and display the video data on a screen.

[0003] However, because an encoder may not know what kind of device the encoded bitstream will be sent to, the encoder must compress the video into a standardized format. This way, all devices that support the chosen standard can successfully decode the video. The compression can be lossless, i.e., the decoded video will be identical to the source video given to the encoder, or lossy, where some degradation of the content is acceptable. Whether the compression is lossless or lossy has a significant impact on the bit rate, i.e., how high the compression ratio, because factors such as noise can make lossless compression prohibitively expensive.

[0004] A video sequence contains a sequence of pictures. The color space typically used in video sequences is YCbCr, where Y is the luma (brightness) component and Cb and Cr are the chroma components. Sometimes the Cb and Cr components are referred to as U and V. Other color spaces are also used, such as ICtCp (also known as IPT) (where I is the luma component and Ct and Cp are the chroma components), constant-luminance YCbCr (where Y is the luma component and Cb and Cr are the chroma components), RGB (where R, G, and B correspond to the blue, green, and blue components, respectively), and YCoCg (where Y is the luma component and Co and Cg are the chroma components).

[0005] The order in which pictures are arranged in a video sequence is called the "display order." Each picture is assigned a Picture Order Count (POC) value to indicate its display order. In this disclosure, the terms "image," "picture," or "frame" are used interchangeably.

[0006] Video compression is used to compress a video sequence into a sequence of coded pictures. In many existing video codecs, a picture is divided into blocks of different sizes. A block is a two-dimensional array of samples. The blocks serve as the basis for coding. A video decoder then decodes the coded picture into a picture containing the sample values.

[0007] Video standards are typically developed by international organizations because they represent different companies and research institutes with different areas of expertise and interests. The most widely adopted video compression standard today is H.264 / AVC (Advanced Video Coding), jointly developed by the ITU-T (International Telecommunication Union-Telecommunication) and ISO. The first version of H.264 / AVC was finalized in 2003, with several updates in later years. A successor to H.264 / AVC, also developed by the ITU-T (International Telecommunication Union-Telecommunication) and the International Organization for Standardization (ISO), known as H.265 / HEVC (High-Efficiency Video Coding), was finalized in 2013. MPEG and ITU-T created the successor to HEVC within the Joint Video Exploratory Team (JVET). The name of this video codec is Versatile Video Coding (VVC), and version 1 of the VVC specification was published as Rec. ITU-T H.266|ISO / IEC (International Electrotechnical Commission) 23090-3, "Versatile Video Coding", 2020.

[0008] The VVC video coding standard is a block-based video codec that utilizes both temporal and spatial prediction. Spatial prediction is achieved using intra (I) prediction from within the current picture. Temporal prediction is achieved using unidirectional (P) or bidirectional inter (B) prediction at the block level from a previously decoded reference picture. At the encoder, the difference between the original and predicted pixel data, called the residual, is transformed into the frequency domain, quantized, and then entropy coded before being transmitted along with the necessary prediction parameters, such as the prediction mode and motion vectors (which may also be entropy coded). A decoder performs entropy decoding, inverse quantization, and an inverse transform to obtain the residual, which is then added to the intra or inter prediction to reconstruct the picture.

[0009] The VVC video coding standard uses a block structure called a quadtree plus binary tree plus ternary tree block structure (QTBT+TT), in which each picture is first partitioned into square blocks called coding tree units (CTUs). All CTUs are of the same size, and the partitioning of a picture into CTUs is done without any syntax to control the partitioning.

[0010] Each CTU is further partitioned into coding units (CUs), which can have either a square or rectangular shape. The CTUs are first partitioned by a quadtree structure, and then the CTUs can be further partitioned into equal-sized partitions either vertically or horizontally in a binary structure to form coding units (CUs). Thus, blocks can have either a square or rectangular shape. The depth of the quadtree and binary tree can be set by the encoder in the bitstream. The ternary tree (TT) section adds the possibility of dividing a CU into three partitions instead of two equal-sized partitions. This increases the possibility of using a block structure that better matches the content structure of the picture, such as approximately following important edges in the picture.

[0011] Intra-coded blocks are I-blocks. Unidirectionally predicted blocks are P-blocks, and bidirectionally predicted blocks are B-blocks. For some blocks, the encoder may determine that the prediction is close enough to the original that it is not necessary to code the residual. The encoder then signals to the decoder that transform coding for that block should be bypassed, i.e., skipped. Such blocks are called skipped blocks.

[0012] At the 20th JVET Conference, it was decided to establish an Exploratory Experiment (EE) on neural network-based (NN-based) video coding. The EE continued at the 21st and 22nd JVET Conferences with two EE tests: NN-based filtering and NN-based super-resolution. At the 23rd JVET Conference, it was decided that the tests would continue in three categories: enhancement filters, super-resolution methods, and intra-prediction. In the enhancement filter category, two settings were considered: (i) the proposed filter used as an in-loop filter and (ii) the proposed filter used as a post-processing filter.

[0013] VVC includes three in-loop filtering processes that are not based on neural networks: a deblocking filter, a sample adaptive offset (SAO) filter, and an adaptive loop filter (ALF). The deblocking filter is used to remove block artifacts by smoothing discontinuities in the horizontal and vertical directions across block boundaries. The deblocking filter uses a block boundary strength (BS) parameter to determine the filtering strength. The BS parameter can have values ​​of 0, 1, and 2, with larger values ​​indicating stronger filtering. The output of the deblocking filter is further processed by the SAO, and then the SAO output is processed by the ALF operation. The output of the ALF is then placed in a display picture buffer (DPB), which is used for predicting pictures to be coded (or decoded) later. Because the deblocking filter, SAO filter, and ALF affect the pictures in the DPB used for prediction, they are classified as in-loop filters, also known as loop filters. It may happen that the decoder further filters the image but does not send the filtered output to the DPB, but only to the display. In contrast to loop filters, such filters do not affect future predictions and are therefore classified as post-processing filters, also known as post-filters.

[0014] EE1-1.6: Combined Test of EE1-1.2 and EE1-1.4, Y. Li, K. Zhang, L. Zhang, H. Wang, J. Chen, K. Reuze, A.M. Kotra, M. Karczewicz, JVET-X0066, October 2021, and EE1-1.2: Test on Deep In-Loop Filter with Adaptive Parameter Selection and Residual Scaling, Y. Li, K. Zhang, L. Zhang, H. Wang, K. Reuze, A.M. Kotra, M. Karczewicz, JVET-Y0143, January 2022, are two consecutive contributions that describe NN-based in-loop filtering. Both contributions use the same NN model for filtering. The NN-based in-loop filter is placed before the SAO and ALF, and the deblocking filter is turned off. The purpose of using the NN-based filter is to improve the quality of the reconstructed samples. Here, it helps that the NN model is nonlinear. The deblocking filter, SAO, and ALF all contain nonlinear elements, such as conditions, and are therefore not strictly linear, but all three are based on linear filters. In contrast, a sufficiently large NN model can, in principle, learn any nonlinear mapping and is therefore capable of representing a broader class of functions compared to deblocking, SAO, and ALF.

[0015] In JVET-X0066 and JVET-Y0143, there are four NN models, i.e., four NN-based in-loop filters: one for luma intra samples, one for chroma intra samples, one for luma inter samples, and one for chroma inter samples. The use of NN filtering can be controlled at the block (CTU) level or the picture level. The encoder can decide whether to use NN filtering for each block or each picture.

[0016] The NN-based in-loop filter presented in JVET-X0066 and JVET-AB0053 substantially increases the compression efficiency of the codec; that is, it substantially reduces the bit rate without reducing the objective quality measured by the MSE-based PSNR (Peak Signal-to-Noise Ratio). The increase in compression efficiency, or simply "gain," is often measured as the Bjontegaard delta rate (BDR) relative to the anchor. As an example, a BDR of -1% means that the same PSNR can be achieved with 1% fewer bits. As reported in JVET-Y0143, in the random access (RA) setting, the BDR gain for the luma component (Y) is -9.80%, and in the all-intra (AI) setting, the BDR gain for the luma component is -7.39%. The complexity of the NN model used for compression is often measured by the number of per-pixel multiply-accumulate (MAC) operations. The high gain of a NN model is directly related to its high complexity. The Luma Intra model described in JVET-Y0143 has a complexity of 430kMAC / pixel, i.e., 430,000 multiply-accumulate operations per pixel. There are other measures of complexity, such as total model size in terms of stored parameters. Summary of the Invention

[0017] Currently, there are several challenges. For example, the high complexity of NN filters is a major challenge for practical hardware implementation, so it is highly desirable to maintain the performance of NN filters while reducing their complexity (i.e., optimizing the tradeoff between complexity and performance). However, the structure of the NN filter described above may not be optimal in terms of the tradeoff between complexity and performance.

[0018] In one example, the NN filter (neural network loop filter) described above is configured to generate an improved output picture based on several inputs, but before these inputs reach the "trunk" or "body" of the NN filter (where most of the NN filter's calculations occur), these inputs are processed separately and then combined. These separate processing and combinations correspond to a significant portion of the NN filter's overall process. Therefore, there is a need to reduce this portion of the NN filter's process.

[0019] Thus, according to a first aspect of the present disclosure, a method for encoding or decoding video is provided. The method includes obtaining values ​​of components of reconstructed samples, obtaining first additional input data, and providing the obtained values ​​of the components of the reconstructed samples to a first group of one or more convolutional layers to generate outputs of the first group of convolutional layers. The method further includes providing the obtained first additional input data to a second group of one or more convolutional layers to generate outputs of the second group of convolutional layers, and encoding or decoding the video based on the outputs of the first group of convolutional layers and the outputs of the second group of convolutional layers. The first group of convolutional layers and the second group of convolutional layers differ in terms of the number of channels, the size of the kernel filter, and / or the number of convolutional layers.

[0020] According to a second aspect of the present disclosure, there is provided a computer program comprising instructions which, when executed by a processing circuit, cause the processing circuit to perform a method according to the first aspect.

[0021] According to a third aspect of the present disclosure, there is provided a carrier containing the computer program of the above embodiments, the carrier being one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0022] In another aspect, there is provided an apparatus for encoding or decoding video, the apparatus being configured to perform the method according to the first aspect.

[0023] Some embodiments of the present disclosure provide ways to substantially maintain or improve the performance of NN filters while reducing the complexity of the NN model.

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various embodiments. [Brief explanation of the drawings]

[0025] [Figure 1A] FIG. 1 illustrates a system, according to some embodiments. [Figure 1B] FIG. 1 illustrates a system, according to some embodiments. [Figure 1C] FIG. 1 illustrates a system, according to some embodiments. [Figure 2] 1 is a schematic block diagram of an encoder according to some embodiments. [Figure 3] FIG. 2 is a schematic block diagram of a decoder according to some embodiments. [Figure 4] FIG. 2 is a schematic block diagram of a portion of a NN filter, according to some embodiments. [Figure 5] FIG. 2 is a schematic block diagram of a portion of a NN filter, according to some embodiments. [Figure 6] FIG. 1 illustrates the computation operations performed by a convolutional layer. [Figure 7] FIG. 1 illustrates a process, according to some embodiments. [Figure 8] FIG. 1 illustrates an apparatus, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0026] The following terminology is used in the description of the embodiments below.

[0027] Neural network: a generic term for an entity with one or more layers of simple processing units called neurons or nodes, which have activation functions and interact with each other through weighted connections and biases, collectively creating a tool in the context of nonlinear transformations.

[0028] Neural network architecture, network architecture, or architecture for short: The layout of a neural network, describing the arrangement of nodes and their connections, usually in the form of several interconnected layers. It may also specify the dimensionality of the input(s) and output(s), as well as activation functions for the nodes.

[0029] Neural network weights, or weights for short: weight values ​​assigned to connections between nodes in a neural network.

[0030] Neural network model, or model for short: A transformation in the form of one or more trained neural networks. A neural network model may be specified along with the neural network architecture, activation functions, biases, and / or weights.

[0031] Filter: A transforming entity. A neural network model is one implementation of a filter. The term NN filter may be used as a contraction of neural network-based filter or neural network filter.

[0032] Neural network training, or training for short: The process of finding values ​​for the weights and biases for a neural network. Typically, a training data set is used to train a neural network, and the goal of training is to minimize a specified error. The amount of training data must be large enough to avoid overtraining. Training a neural network is usually a time-consuming task and generally involves several iterations over the training data, each iteration being called an epoch.

[0033] 1A illustrates a system 100 according to some embodiments. The system 100 comprises a first entity 102, a second entity 104, and a network 110. The first entity 102 is configured to transmit a video stream (also known as a "video bitstream," "bitstream," or "encoded video") 106 towards the second entity 104.

[0034] The first entity 102 may be any computing device (e.g., a network node such as a server) capable of encoding video using an encoder 112 and transmitting the encoded video over a network 110 toward a second entity 104. The second entity 104 may be any computing device (e.g., a network node) capable of receiving the encoded video and decoding the encoded video using a decoder 114. Each of the first entity 102 and the second entity 104 may be a single physical entity or a combination of multiple physical entities. The multiple physical entities may be located in the same location or distributed in a cloud.

[0035] 1B , the first entity 102 is a video streaming server 132, and the second entity 104 is a user equipment (UE) 134. The UE 134 may be a desktop, a laptop computer, a tablet, a mobile phone, or any other computing device. The video streaming server 132 may transmit a video bitstream 136 (e.g., YouTube™ video streaming) to the video streaming client 134. Upon receiving the video bitstream 136, the UE 134 may decode the received video bitstream 136, thereby generating and displaying a video for video streaming.

[0036] 1C , the first entity 102 and the second entity 104 are a first UE 152 and a second UE 154. For example, the first UE 152 may be a video conference session host or a video chat initiator, and the second UE 154 may be a video conference session responder or a video chat responder. In the embodiment shown in FIG. 1C , the first UE 152 may transmit a video bitstream 156 for a video conference (e.g., Zoom™, Skype™, MS Teams™, etc.) or a video chat (e.g., Facetime™) to the second UE 154. Upon receiving the video bitstream 156, the UE 154 may decode the received video bitstream 156 and thereby generate and display video for the video conference session or the video chat.

[0037] 2 shows a schematic block diagram of an encoder 112 according to some embodiments. The encoder 112 is configured to encode blocks of sample values ​​(hereinafter "blocks") in video frames of a source video 202. In the encoder 112, a current block (e.g., a block contained in a video frame of the source video 202) is predicted by a motion estimator 250 from an already provided block in the same frame or in a previous frame by performing motion estimation. The result of the motion estimation is a motion vector or displacement vector associated with a reference block in the case of inter prediction. The motion vector is utilized by the motion compensator 250 to output an inter prediction of the block.

[0038] An intra predictor 249 calculates an intra prediction of the current block. The output from the motion estimator / compensator 250 and the output from the intra predictor 249 are input to a selector 251, which selects either intra or inter prediction for the current block. The output from the selector 251 is input to an error calculator in the form of an adder 241, which also receives the sample values ​​of the current block. The adder 241 calculates and outputs a residual error as the difference in sample values ​​between the block and its prediction. The error is transformed in a transformer 242, such as by a discrete cosine transform, and quantized by a quantizer 243, followed by coding in an encoder 244, such as by an entropy encoder. In inter coding, the estimated motion vectors are provided to the encoder 244 to generate a coded representation of the current block.

[0039] The transformed and quantized residual error for the current block is also provided to an inverse quantizer 245 and an inverse transformer 246 to recover the original residual error. This error is added by an adder 247 to the block prediction output from the motion compensator 250 or intra predictor 249 to create a reconstructed sample block 280 that can be used in predicting and coding the next block. The reconstructed sample block 280 is then processed by an embodiment of a neural network filter 230 (also known as a "neural network loop filter" or "NNLF") to perform filtering to eliminate blocking artifacts. The output from the neural network filter 230, i.e., output data 290, is then temporarily stored in a frame buffer 248, where it is available to the intra predictor 249 and the motion estimator / compensator 250.

[0040] In some embodiments, the encoder 112 may include an SAO unit 270 and / or an ALF 272. The SAO unit 270 and the ALF 272 may be configured to receive output data 290 from the NN filter 230, perform additional filtering on the output data 290, and provide the filtered output data to the buffer 248.

[0041] 2, the NN filter 230 is disposed between the SAO unit 270 and the adder 247, although in other embodiments, the NN filter 230 may replace the SAO unit 270 and / or the ALF 272. Alternatively, in other embodiments, the NN filter 230 may be disposed between the buffer 248 and the motion compensator 250. Furthermore, in some embodiments, a deblocking filter (not shown) may be disposed between the NN filter 230 and the adder 247 such that the reconstructed sample block 280 undergoes a deblocking process and is then provided to the NN filter 230.

[0042] 3 is a schematic block diagram of a decoder 114 according to some embodiments. The decoder 114 includes a decoder 361, such as an entropy decoder, for decoding the coded representation of the block to obtain a set of quantized and transformed residual errors. These residual errors are dequantized in an inverse quantizer 362 and inverse transformed by an inverse transformer 363 to obtain a set of residual errors. These residual errors are added to the sample values ​​of a reference block in an adder 364. The reference block is determined by a motion estimator / compensator 367 or an intra predictor 366, depending on whether inter or intra prediction is performed.

[0043] A selector 368 is thereby interconnected to the adder 364, the motion estimator / compensator 367 and the intra predictor 366. The resulting decoded block 380 output from the adder 364 is input to an embodiment of a NN filter unit 330 for filtering any blocking artifacts. A filtered block 390 is output from the NN filter 330 and is further preferably temporarily provided to the frame buffer 365 where it can be used as a reference block for subsequent blocks to be decoded.

[0044] A frame buffer (e.g., a decoded picture buffer (DPB)) 365 is thereby connected to the motion estimator / compensator 367 to make stored blocks of samples available to the motion estimator / compensator 367. The output from the adder 364 is also preferably input to the intra predictor 366 to be used as an unfiltered reference block.

[0045] In some embodiments, the decoder 114 may include an SAO unit 380 and / or an ALF 372. The SAO unit 380 and the ALF 382 may be configured to receive output data 390 from the NN filter 330, perform additional filtering on the output data 390, and provide the filtered output data to a buffer 365.

[0046] 3, the NN filter 330 is disposed between the SAO unit 380 and the adder 364, but in other embodiments, the NN filter 330 may replace the SAO unit 380 and / or the ALF 382. Alternatively, in other embodiments, the NN filter 330 may be disposed between the buffer 365 and the motion compensator 367. Furthermore, in some embodiments, a deblocking filter (not shown) may be disposed between the NN filter 330 and the adder 364 such that the reconstructed sample block 380 undergoes a deblocking process and is then provided to the NN filter 330.

[0047] 4 is a schematic block diagram of a portion of the NN filter 230 / 330 for filtering intra-luma samples according to some embodiments. In this disclosure, a luma (or chroma) intra sample is a luma (or chroma) component of an intra-predicted sample. Similarly, a luma (or chroma) inter sample is a luma (or chroma) component of an inter-predicted sample.

[0048] 4, the NN filter 230 / 330 may have four inputs: (1) the value of the luma component of the reconstructed sample 280 / 380 (“rec”), (2) the value of the luma component of the predicted sample 295 / 395 (“pred”), (3) block boundary strength (BBS) information (“bs”) indicating the strength of filtering applied to the boundaries of the luma components of the samples, and (4) a quantization parameter (“qp”). In some embodiments, additional input(s) (e.g., partition information indicating how the luma components of the samples are partitioned) may be further used as inputs for the NN filter 230 / 330.

[0049] Each of the four inputs may separately pass through a convolutional layer (labeled "conv3x3" in FIG. 4) and a parametric rectified linear unit (PReLU) layer (labeled "PReLU"). The four outputs from the four PReLU layers may then be concatenated via a concatenating unit (labeled "concat" in FIG. 4) and fused together to generate the data (a.k.a., "signal") "y." The convolutional layer "conv3x3" is a convolutional layer with a kernel size of 3x3, and the convolutional layer "conv1x1" is a convolutional layer with a kernel size of 1x1. The PReLU may create an activation layer.

[0050] In some embodiments, qp may be a scalar value. In such embodiments, the NN filter 230 / 330 may also include a dimensional operation unit (labeled "Unsqueeze expand" in FIG. 4), which may be configured to expand qp so that the expanded qp has the same size as the other inputs (i.e., rec, pred, and bs). However, in other embodiments, qp may be a matrix, whose size may be the same as the size of the other inputs (e.g., rec, pred, and bs). For example, different samples inside a CTU may be associated with different qp values. In such embodiments, a dimensional operation unit is not required.

[0051] In some embodiments, the NN filter 230 / 330 may also include a downsampler (labeled "2↓" in FIG. 4), which is configured to perform downsampling by a factor of two.

[0052] 4, data "y" may be provided to a group of N successive attention residual (hereinafter "AR") blocks 402. In some embodiments, the N successive AR blocks 402 may have the same structure, while in other embodiments, the N successive AR blocks 402 may have different structures. N may be any integer greater than or equal to 2. For example, N may be equal to 8.

[0053] 4, a first AR block 402 included in a group may be configured to receive data “y” and generate first output data “z0.” A second AR block 402 disposed immediately after the first AR block 402 may be configured to receive the first output data “z0” and generate second output data “z1.”

[0054] If the group includes only two AR blocks 402 (i.e., the first AR block and the second AR block described above), the second output data “z1” may be provided to the final processing unit 550 (shown in FIG. 5) of the NN filter 230 / 330.

[0055] On the other hand, if a group includes three or more AR blocks, each AR block 402 included in the group, except for the first and last AR blocks, may be configured to receive output data from the previous AR block 402 and provide the output data to the next AR block. The last AR block 402 may be configured to receive output data from the previous AR block and provide the output data to the final processing unit 550 of the NN filter 230 / 330. In Figure 4, N-1 corresponds to the number of AR blocks included in the NN filter 230 / 330.

[0056] In some embodiments, some or all of the AR blocks 402 may include a spatial attention block 412 configured to generate an attention mask f. The attention mask f may have one channel, and its size may be the same as the data "y." Taking the first AR block 402 as an example, the spatial attention block included in the first AR block 402 may be configured to multiply the attention mask f by the residual data "r" to obtain data "rf." The data "rf" may be combined with the residual data "r" and then combined with the data "y," thereby generating first output data "z0."

[0057] As shown in FIG. 5, the output Z of the group of AR blocks 402 N-1 may be processed by a convolutional layer 502, a PReLU 504, another convolutional layer 506, pixel shuffling (or actually, sample shuffling) 508, and final scaling 510, thereby producing filtered output data (“output”) 290 / 390.

[0058] Referring again to FIG. 4, the NN filter 230 / 330 may include four separate first layers 452-458. As shown in FIG. 4, each of the four first layers 452-458 may include a 3×3 convolutional layer and a PReLU. Each of the four first layers 452-458 may be configured to receive different input data. For example, layers 452, 454, 456, and 458 may be configured to receive input data “rec,” “pred,” “bs,” and “qp,” respectively.

[0059] The 3x3 convolutional layer included in each of the first layers 452-458 may include 96 channels (i.e., 96 3x3 kernel filters), such that for a single input (comprising multiple pixel values) provided to each of the first layers 452-458, there will be 96 outputs (i.e., one output corresponding to each of the 96 channels). Thus, the first layers 452-458 may be configured to generate 4x96 outputs (a.k.a., output channels) for four inputs (i.e., "rec," "pred," "bs," and "qp"). These 4x96 outputs may be processed through a concatenate unit, a fuse unit, and a transition unit and thus combined into y, corresponding to the 96 outputs.

[0060] Generally, a crude way to calculate the number of multiply-accumulate (MAC) operations required for a convolutional neural network layer is to multiply the number of inputs (a.k.a. input channels) by the number of filter coefficients by the number of outputs (a.k.a. output channels).

[0061] For example, assume that the input "rec" is an image with nine pixel values, as shown in FIG. 6. Note that this number is provided merely for ease of explanation and does not limit the embodiments of the present disclosure in any way. For each pixel value contained in the input "rec", 96 different kernel filters 602 are applied. For example, if the kernel filters 602 of the convolutional layer contained in the first layer 452 are applied to pixel values ​​a of the input "rec", 22 When applied to a pixel value a 22 The filtered value corresponding to a 11 ×w 11 +a 12 ×w 12 +a 13 ×w 13 +a 21 ×w 21 +a 22 ×w 22 +a 23 ×w 23 +a 31 ×w 31 +a 32 ×w 32 +a 33 ×w 33 It can be calculated as follows:

[0062] Therefore, for each pixel value, the filter coefficient w 11 , w 12 , ...), nine multiplication operations are performed 96 times (# of kernel filters corresponding to # of output channels). Thus, now the number of MAC operations performed for each pixel for the first layer 452 (which generally indicates the computational complexity of the first layer 452) is roughly 1×(3×3)×96=864 MAC operations. If all of the layers 452-458 have the same # of filtering channels (i.e., # of kernel filters), the total number of MAC operations required for the four layers 452-458 is 864×4=3456.

[0063] 4, the outputs of the first layers 452-458 are provided to a second layer 462 of the NN filter 230 / 330. The first operation of the second layer 462 (performed by a fusing unit) is a 1×1 convolution, followed by the second operation of the second layer 462 (performed by a transitioning unit) which is a reduced-scale 3×3 convolution.

[0064] For the first operation of the second layer 462, the number of MAC operations may be calculated as follows: For the first operation of the second layer 462, all outputs of the first layers 452-458 are provided to the second layer 462, so the number of inputs is 96×4=384. Also, for the first operation of the second layer 462, the number of filter coefficients is 1 (because a 1×1 kernel filter is used in the fusion unit), and the number of outputs from the 1×1 kernel filter is 96. Therefore, the number of MAC operations performed for the first operation of the second layer 462 is 384×1×1×96=36864.

[0065] Similarly, the number of MAC operations for the second operation of the second layer 462 may be calculated as follows: For the second operation of the second layer 462, the number of inputs (i.e., the number of outputs from the first operation of the second layer 462) is 96, the number of filter coefficients is 9 (because a 3×3 kernel filter is used in a 3×3 convolutional layer), and the number of outputs is 96. Therefore, the number of MAC operations performed for the second operation of the second layer 462 is TIFF2025542460000002.tif8170 MAC operations, where: Multiplying twice by TIFF2025542460000003.tif8170 means that only one out of every four output samples needs to be calculated due to the output being downsampled in the x and y directions.

[0066] In summary, if 96 filtering channels are used for each of the four inputs, the total number of MAC operations performed to generate the output y in FIG. 4 is 3456+36864+20736=61056 MAC operations.

[0067] In the embodiment described above, all inputs (e.g., "rec", "pred", "bs", and "qp") are given the same level of importance. However, different inputs may not be equally important. For example, the input "rec" corresponds to the pixel values ​​of the reconstructed image. Since this input is the input that needs to be improved (which means that this input is the one that receives attention in the filtering process performed by the NN filter 230 / 330), it makes sense to place more emphasis on this input compared to other inputs during the filtering process.

[0068] Thus, according to some embodiments, different levels of importance may be assigned to different inputs such that different numbers of MAC operations are performed for different inputs. More particularly, in some embodiments, more importance may be given to inputs that are more important to the end result of the filtering process by setting a higher number of MAC operations for the important input(s) and a lower number of MAC operations for the less important input(s).

[0069] In some embodiments, different inputs are assigned different levels of importance by using different numbers of filtering channels for the different inputs. For example, the number of filtering channels applied to input "rec" (the most important input out of the four inputs) may be increased from 96 to 192, thereby placing more emphasis on input "rec." Meanwhile, the number of filtering channels applied to input "pred" may be reduced from 96 to 48, and the number of filtering channels applied to each of input "bs" and input "qp" may be reduced from 96 to 24.

[0070] Varying the number of filtering channels applied to different inputs may result in a reduction in the number of MAC operations performed for NN filtering. More specifically, in the above example, the number of MAC operations for input “rec” would be 1×(3×3)×192=1728, the number of MAC operations for input “pred” would be 1×(3×3)×48=432, and the number of MAC operations for each of inputs “bs” and “qp” would be 1×(3×3)×24=216. Thus, in this example, the total number of MAC operations performed by the head of the NN filter 230 / 330 (i.e., the upper portion of the NN filter 230 / 330 shown in FIG. 4) is 1728+432+216+216=2592. The following table shows the difference between the total number of MAC operations performed when the same number of filtering channels are used for different inputs and when different numbers of filtering channels are used for different inputs. TIFF2025542460000004.tif28170

[0071] A change in the number of filtering channels applied to different inputs may result in a larger change for the next stage, i.e., the second layer 462 of the NN filter 230 / 330. For instance, in the above example, the number of inputs (i.e., the number of input channels) for the second layer 462 of the NN filter 230 / 330 is 192+48+24+24=288 (compared to (96×4)=384). Thus, the number of MAC operations performed by the 1×1 convolutional layer included in the fusion unit of the second layer 462 is 288×1×1×96=27,648 MAC operations (compared to 384×1×1×96). Also, in the above example, the number of inputs (i.e., the number of input channels) for the 3×3 convolutional layer included in the transition unit of the second layer 462 of the NN filter 230 / 330 is 96, and therefore the number of MAC operations performed by the 3×3 convolutional layer is 96×3×3×96=20736 MAC operations. TIFF2025542460000005.tif83170

[0072] Thus, the total number of MAC operations performed by the head portion of the NN filter 230 / 330 was reduced from 61,056 to 50,976 MAC operations, i.e., a 16.5% reduction. The entire network (including body and tail) involves approximately 500k MAC operations, so the head change is equivalent to a reduction of approximately (61,056-50,976) / 500,000=2% in the overall kMACs / pixel for the network. At the same time, increased attention to critical reconstruction inputs leads to a slight increase in compression efficiency by -0.03%.

[0073] In some embodiments, instead of changing the number of filtering channels applied to each of the inputs “rec”, “pred”, “bs”, and “qp” to 192, 24, 12, and 12, the number of filtering channels may be changed to 192, 24, 12, and 12 for the inputs “rec”, “pred”, “bs”, and “qp”, respectively. Based on the method for computing MAC operations described above, changing the number of filtering channels used in the head portion of the NN filter 230 / 330 gives a 4% reduction in the overall kMAC / pixel for the entire model.

[0074] In some embodiments, in addition to changing the number of filtering channels as described above, the number of channels between the fusion layer and the transition layer may be reduced from 96 to 48. This leads to a total reduction of kMAC / pixel for the model of 9% and a BD rate penalty of only +0.03%.

[0075] Note that in the above-described embodiments, the NN filter 230 / 330 is an in-loop filter, which means that its output can be used to predict future pictures in a video. However, a similar or equivalent neural network can instead be used as a post-filter to filter pictures after they have been used for prediction.

[0076] In some embodiments, in addition to or instead of changing the number of filtering channels, a different set of inputs may be provided to the NN filter 230 / 330. More specifically, in some embodiments, instead of the four inputs (“rec,” “pred,” “bs,” and “qp”), additional inputs may be provided, such as block size, motion vector, and / or prediction mode. In other embodiments, any one or more additional inputs may be provided instead of some of the inputs “pred,” “bs,” and “qp.” In the embodiments described above, each of these additional or alternative inputs may also have a reduced number of channels compared to the reconstructed picture input. For example, the first layer of the NN filter 230 / 330 may receive five inputs, i.e., the four inputs shown in FIG. 4 and a fifth input identifying the block type (I / P / B, intra-coded / uni-predictive / bi-predictive), where 192 filtering channels are used for the input “rec,” 24 filtering channels are used for the input “pred,” and 12 filtering channels are used for the remaining three inputs.

[0077] In some embodiments, in addition to changing the number of filtering channels applied to different inputs, the size of the kernel filter used for filtering may also be changed. For example, instead of using a 3×3 kernel filter for input “rec”, a first size of kernel filter (e.g., 5×5) may be used to filter input “rec”, while a second size of kernel filter (3×3) is used to filter other inputs, where the first size is larger than the second size.

[0078] In some embodiments, the first layer 452 for filtering the input “rec” may use kernel filters having different sizes to filter the input “rec.” For example, if there are 196 filtering channels in the first layer 452, the first 96 filtering channels among the 196 filtering channels may use kernel filters each having a first size (e.g., 5×5), while the remaining filtering channels among the 196 filtering channels may use kernel filters each having a second size (3×3), where the first size and the second size are different. In some embodiments, the first size is larger than the second size.

[0079] In some embodiments, more emphasis may be placed on a certain input (e.g., input “rec”) by including two or more convolutional layers in the first layer (e.g., 452). For example, in FIG. 4, the first layer 452 includes a single 3×3 convolutional layer followed by a single PreLU, but in some embodiments, the first layer 452 includes multiple 3×3 convolutional layers and multiple PreLUs. More specifically, in such an example, the first layer 452 may include a first 3×3 convolutional layer followed by a first PreLU, followed by a second 3×3 convolutional layer followed by a second PreLU.

[0080] In some embodiments, more emphasis may be placed on a certain input (e.g., the input “rec”) by increasing the number of bits for the kernel filter(s). For example, a first number of bits (e.g., 14) may be used to assign filter weights (e.g., W , shown in FIG. 6 ) to the kernel filter of a 3×3 convolutional layer in the first layer 452. 11 , W 12 , ...), and a second number of bits (e.g., 7) may be used to indicate the filter weights of the kernel filters of the 3x3 convolutional layers of the first layers 454, 456, and / or 458, where the first number is greater than the second number.

[0081] In some embodiments, the input “rec” corresponds to pixel values ​​of a reconstructed luma picture or pixel values ​​of a reconstructed chroma picture. However, in other embodiments, the input “rec” corresponds to pixel values ​​of both a reconstructed luma picture and a reconstructed chroma picture. In such embodiments, different emphasis may be placed on pixel values ​​of a reconstructed luma picture and pixel values ​​of a reconstructed chroma picture. For example, if the NN filter 230 / 330 is used to predict chroma samples, more emphasis may be placed on pixel values ​​of the reconstructed chroma samples. The techniques described above (e.g., increasing the number of filtering channels) may be used to place more emphasis on pixel values ​​of the reconstructed chroma samples.

[0082] In summary, some embodiments described above involve increasing the number of filtering channels, the size of the kernel filter, and / or the number of bits used to indicate the filter weights of the kernel filter for a particular input (e.g., input “rec”), and / or using additional convolutional layers while decreasing or maintaining the number of filtering channels, the size of the kernel filter, the number of bits used to indicate the filter weights of the kernel filter, and / or the number of convolutional layers for other inputs. Through these selective adjustments, more emphasis can be placed on more important inputs, and thus the accuracy of the filtering operation of the NN filter 230 / 330 can be increased or maintained while reducing the complexity of the filtering operation.

[0083] FIG. 7 illustrates a process 700 for encoding or decoding video according to some embodiments. FIG. 7 begins with step s702. Step s702 includes obtaining values ​​of components of reconstructed samples. Step s704 includes obtaining first additional input data. Step s706 includes providing the obtained values ​​of the components of the reconstructed samples to a first group of one or more convolutional layers, thereby generating outputs of the first group of convolutional layers. Step s708 includes providing the obtained first additional input data to a second group of one or more convolutional layers, thereby generating outputs of the second group of convolutional layers. Step s710 includes encoding or decoding video based on outputs of the first group of convolutional layers and outputs of the second group of convolutional layers. The first group of convolutional layers and the second group of convolutional layers differ in terms of the number of channels, the size of the kernel filter, and / or the number of convolutional layers.

[0084] In some embodiments, process 700 includes combining outputs of the first group of convolutional layers and outputs of the second group of convolutional layers, thereby generating a combined output; and providing the combined output to a third group of one or more convolutional layers, thereby generating an output of the third group of convolutional layers, wherein video is encoded or decoded based on the outputs of the third group of convolutional layers.

[0085] In some embodiments, the first additional input data is any one of values ​​of components of the sample to be predicted, partitioning information indicating how the components of the sample are partitioned, block boundary strength information indicating the strength of filtering applied to the boundaries of the components of the sample, values ​​of quantization parameters, block size information indicating block sizes, motion vector information indicating motion vectors, prediction mode information indicating prediction modes, and / or block type information indicating block types.

[0086] In some embodiments, the number of channels included in the first group of convolutional layers is more than twice the number of channels included in the second group of convolutional layers.

[0087] In some embodiments, the number of channels included in the first group of convolutional layers is 192, and the number of channels included in the second group of convolutional layers is 48, 24, or 12.

[0088] In some embodiments, process 700 includes obtaining second additional input data and providing the second additional input data to a fourth group of one or more convolutional layers, thereby generating an output of the fourth group of convolutional layers, wherein the video is encoded or decoded based on the output of the fourth group of convolutional layers, wherein the first group of convolutional layers and the fourth group of convolutional layers differ in number of channels, kernel filter size, and / or number of convolutional layers, and wherein the second group of convolutional layers and the fourth group of convolutional layers differ in number of channels, kernel filter size, and / or number of convolutional layers.

[0089] In some embodiments, the number of channels included in the second group of convolutional layers is 48 or 24, and the number of channels included in the fourth group of convolutional layers is 24 or 12.

[0090] In some embodiments, the size of the kernel filters for the first group of convolutional layers is different from the size of the kernel filters for the second group of convolutional layers.

[0091] In some embodiments, the kernel filters for the first group of convolutional layers have a size of 5x5, and the kernel filters for the second group of convolutional layers have a size of 3x3.

[0092] In some embodiments, a convolutional layer in a first group of convolutional layers comprises a set of channels, a first subset of channels in the set of channels has a kernel filter having a first size, and a second subset of channels in the set of channels has a kernel filter having a second size, the first size and the second size being different.

[0093] In some embodiments, the kernel filter size for the first subset of channels is 5x5 and the kernel filter size for the second subset of channels is 3x3.

[0094] In some embodiments, the first group of convolutional layers comprises a first number of convolutional layers and the second group of convolutional layers comprises a second number of convolutional layers, the first number being greater than the second number.

[0095] In some embodiments, the first number is two and the second number is one.

[0096] In some embodiments, the first group of convolutional layers includes a first convolutional layer and a second convolutional layer, the first convolutional layer having a kernel filter with a first size and the second convolutional layer having a kernel filter with a second size, the first size and the second size being different.

[0097] In some embodiments, the first size is 5x5 and the second size is 3x3.

[0098] In some embodiments, a first group of convolutional layers has a kernel filter having a first plurality of filter values, and a second group of convolutional layers has a kernel filter having a second plurality of filter values, wherein the number of bits in each filter value included in the first plurality of filter values ​​is greater than the number of bits in each filter value included in the second plurality of filter values.

[0099] In some embodiments, the number of bits for each filter value included in the first plurality of filter values ​​is 14 bits, and the number of bits for each filter value included in the second plurality of filter values ​​is 7 bits.

[0100] In some embodiments, the reconstructed sample is one of a reconstructed luma sample and a reconstructed chroma sample. Process 700 includes obtaining a value of a component of another of the reconstructed luma sample and the reconstructed chroma sample, providing the obtained value of the component of the other of the reconstructed luma sample and the reconstructed chroma sample to a fifth group of one or more convolutional layers, and encoding or decoding video based on an output of the fifth group of convolutional layers, wherein the first group of convolutional layers and the fifth group of convolutional layers differ in number of channels, size of a kernel filter, and / or number of convolutional layers.

[0101] 8 is a block diagram of an apparatus 800 for implementing the encoder 112, the decoder 114, or a component included in the encoder 112 or the decoder 114 (e.g., the NN filter 280 or 330) according to some embodiments. When the apparatus 800 implements a decoder, the apparatus 800 may be referred to as a "decoding apparatus 800," and when the apparatus 800 implements an encoder, the apparatus 800 may be referred to as an "encoding apparatus 800." As shown in FIG. 8 , device 800 includes a processing circuit (PC) 802 that may include one or more processors (P) 855 (e.g., a general-purpose microprocessor and / or one or more other processors, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.), which may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., device 800 may be a distributed computing device), and at least one network interface 848, such that device 800 is The PC 802 may comprise at least one network interface 848, and a storage unit (a.k.a., a "data storage system") 808, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments in which the PC 802 includes a programmable processor, a computer program product (CPP) 841 may be provided. The CPP 841 includes a computer-readable medium (CRM) 842, which stores a computer program (CP) 843, which includes computer-readable instructions (CRI) 844.CRM 842 may be a non-transitory computer-readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a memory device (e.g., a random access memory, a flash memory), or the like. In some embodiments, CRI 844 of computer program 843, when executed by PC 802, configures CRI to cause device 800 to perform steps described herein (e.g., steps described herein with reference to flowcharts). In other embodiments, device 800 may be configured to perform steps described herein without the need for code. That is, for example, PC 802 may simply consist of one or more ASICs. Thus, features of the embodiments described herein may be implemented in hardware and / or software.

[0102] Overview of the embodiment A1. A method (700) for encoding or decoding video, the method comprising: obtaining component values ​​of the reconstructed samples (s702); obtaining first additional input data (s704); providing the obtained values ​​of the components of the reconstructed samples to a first group of one or more convolutional layers (s706), thereby generating outputs of the first group of convolutional layers; providing the obtained first additional input data to a second group of one or more convolutional layers, thereby generating an output of the second group of convolutional layers (s708); encoding or decoding the video based on the outputs of the first group of convolutional layers and the outputs of the second group of convolutional layers (s710); Including, the first group of convolutional layers and the second group of convolutional layers differ in terms of the number of channels, the size of the kernel filter, and / or the number of convolutional layers; Method (700). A1-1. The method is combining the outputs of the first group of convolutional layers and the outputs of the second group of convolutional layers, thereby generating a combined output; providing the combined output to a third group of one or more convolutional layers, thereby generating an output of the third group of convolutional layers; and Including, The video is encoded or decoded based on the output of the third group of convolutional layers. The method of embodiment A1. A2. A method as described in embodiment A1 or A1-1, wherein the first additional input data is any one of the following: a value of a component of the sample to be predicted, partition information indicating how the component of the sample is partitioned, block boundary strength information indicating the strength of the filtering applied to the boundary of the component of the sample, a value of a quantization parameter, block size information indicating the block size, motion vector information indicating the motion vector, prediction mode information indicating the prediction mode, and / or block type information indicating the block type. A3. The method of embodiment A1 or A1-1, wherein the number of channels included in the first group of convolutional layers is more than twice the number of channels included in the second group of convolutional layers. A4. The number of channels included in the first group of the convolutional layer is 192; The number of channels in the second group of convolutional layers is 48, 24, or 12. The method of embodiment A3. A5. The method is obtaining second additional input data; providing second additional input data to a third group of one or more convolutional layers, thereby generating outputs of the third group of convolutional layers; Including, The video is encoded or decoded based on the output of the third group of convolutional layers; and the first group of convolutional layers and the third group of convolutional layers differ in terms of the number of channels, the size of the kernel filter, and / or the number of convolutional layers; the second group of convolutional layers and the third group of convolutional layers differ with respect to the number of channels, the size of the kernel filter, and / or the number of convolutional layers; The method of any one of embodiments A1 to A4. A6. (When embodiment A5 depends on embodiment A4) The number of channels included in the second group of convolutional layers is 48 or 24; The number of channels included in the third group of convolutional layers is 24 or 12. The method of embodiment A5. A7. The method of any one of embodiments A1 to A6, wherein the size of the kernel filter for the first group of convolutional layers is different from the size of the kernel filter for the second group of convolutional layers. A8. The kernel filter size for the first group of convolutional layers is 5x5, The kernel filter size for the second group of convolutional layers is 3x3, The method of embodiment A7. A9. a convolutional layer in the first group of convolutional layers comprises a set of channels; a first subset of channels in the set of channels has a kernel filter having a first size; a second subset of channels in the set of channels has a kernel filter having a second size; The first size and the second size are different. The method of any one of embodiments A1 to A8. A10. the kernel filter for the first subset of channels has size 5x5; The kernel filter for the second subset of channels is of size 3x3. The method of embodiment A9. A11. the first group of convolutional layers comprises a first number of convolutional layers; the second group of convolutional layers comprises a second number of convolutional layers; The first number is greater than the second number, The method of any one of embodiments A1 to A10. A12. The method of embodiment A11, wherein the first number is 2 and the second number is 1. A13. the first group of convolutional layers includes a first convolutional layer and a second convolutional layer; the first convolutional layer has a kernel filter having a first size; the second convolutional layer has a kernel filter having a second size; The first size and the second size are different. The method of embodiment A11 or A12. A14. The first size is 5x5, The second size is 3x3, The method of embodiment A13. A15. a first group of convolutional layers having a kernel filter with a first plurality of filter values; a second group of convolutional layers having a kernel filter with a second plurality of filter values; the number of bits in each filter value included in the first plurality of filter values ​​is greater than the number of bits in each filter value included in the second plurality of filter values; The method of any one of embodiments A1 to A14. A16. the number of bits of each filter value included in the first plurality of filter values ​​is 14 bits; the number of bits of each filter value included in the second plurality of filter values ​​is 7; The method of embodiment A15. A17. the reconstructed sample is one of a reconstructed luma sample and a reconstructed chroma sample, The method is obtaining a component value of another one of the reconstructed luma sample and the reconstructed chroma sample; providing the obtained values ​​of the other one of the reconstructed luma samples and the reconstructed chroma samples to a third group of one or more convolutional layers; and encoding or decoding the video based on the output of the third group of convolutional layers; Including, the first group of convolutional layers and the third group of convolutional layers differ in terms of the number of channels, the size of the kernel filter, and / or the number of convolutional layers; The method of any one of embodiments A1 to A16. B1. A computer program (800) comprising instructions (844) that, when executed by a processing circuit (802), cause the processing circuit to perform a method according to any one of embodiments A1 to A17. B2. A carrier containing the computer program of embodiment B1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium. C1. An apparatus (800) for encoding or decoding video, the apparatus comprising: obtaining component values ​​of the reconstructed samples (s702); obtaining first additional input data (s704); providing the obtained values ​​of the components of the reconstructed samples to a first group of one or more convolutional layers (s706), thereby generating outputs of the first group of convolutional layers; providing the obtained first additional input data to a second group of one or more convolutional layers, thereby generating an output of the second group of convolutional layers (s708); encoding or decoding the video based on the outputs of the first group of convolutional layers and the outputs of the second group of convolutional layers (s710); It is set to do the first group of convolutional layers and the second group of convolutional layers differ in terms of the number of channels, the size of the kernel filter, and / or the number of convolutional layers; Equipment (800). C2. The device of embodiment C1, wherein the device is further configured to perform a method according to any one of embodiments A2 to A17. D1. An apparatus (800) comprising: A processing circuit (802); Memory (841) and and wherein the memory includes instructions executable by the processing circuitry, such that the apparatus is operable to perform the method of any one of embodiments A1 to A17.

[0103] conclusion

[0104] While various embodiments have been described herein, it should be understood that these embodiments have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, unless otherwise indicated herein or clearly contradicted by context, any combination of the above-described elements in all possible variations thereof is encompassed by the present disclosure.

[0105] Additionally, while the processes described above and illustrated in the figures have been shown as a sequence of steps, this has been done for purposes of illustration only, and it is therefore contemplated that some steps may be added, some steps may be omitted, the order of steps may be rearranged, and some steps may be performed in parallel.

Claims

1. A method (700) for encoding or decoding video, said method comprising: Obtaining component values ​​of the reconstructed samples (s702); Obtaining first additional input data (s704); providing the obtained values ​​of the components of the reconstructed samples to a first group of one or more convolutional layers (s706), thereby generating outputs of the first group of convolutional layers; providing the obtained first additional input data to a second group of one or more convolutional layers, thereby generating outputs of the second group of convolutional layers (s708); encoding or decoding the video based on the outputs of the first group of convolutional layers and the outputs of the second group of convolutional layers (s710); Including, the first group of convolutional layers and the second group of convolutional layers differ in number of channels, size of kernel filters, and / or number of convolutional layers; Method (700).

2. combining the outputs of the first group of convolutional layers with the outputs of the second group of convolutional layers, thereby generating a combined output; providing the combined output to a third group of one or more convolutional layers, thereby generating an output of the third group of convolutional layers; and further comprising the video is encoded or decoded based on the output of the third group of convolutional layers. The method of claim 1.

3. 3. The method of claim 1, wherein the first additional input data is any one of values ​​of components of a sample to be predicted, partition information indicating how the components of the sample are partitioned, block boundary strength information indicating the strength of filtering applied to boundaries of the components of the sample, values ​​of a quantization parameter, block size information indicating block sizes, motion vector information indicating motion vectors, prediction mode information indicating prediction modes, and / or block type information indicating block types.

4. 3. The method of claim 1, wherein the number of channels included in the first group of convolutional layers is more than twice the number of channels included in the second group of convolutional layers.

5. the number of channels included in the first group of convolutional layers is 192; the number of channels included in the second group of convolutional layers is 48, 24, or 12; The method of claim 4.

6. The method comprises: obtaining second additional input data; providing the second additional input data to a fourth group of one or more convolutional layers, thereby generating outputs of the fourth group of convolutional layers; and further comprising the video is encoded or decoded based on the output of the fourth group of convolutional layers; the first group of convolutional layers and the fourth group of convolutional layers differ in number of channels, size of kernel filters, and / or number of convolutional layers; the second group of convolutional layers and the fourth group of convolutional layers differ in number of channels, size of kernel filters, and / or number of convolutional layers; 6. The method according to any one of claims 1 to 5.

7. When dependent on claim 5, the number of channels included in the second group of convolutional layers is 48 or 24; the number of channels included in the fourth group of convolutional layers is 24 or 12; The method of claim 6.

8. 8. The method of claim 1, wherein a size of a kernel filter for the first group of convolutional layers is different from a size of a kernel filter for the second group of convolutional layers.

9. the size of the kernel filter for a first group of convolutional layers is 5×5; the size of the kernel filter for the second group of convolutional layers is 3x3; The method of claim 8.

10. a convolutional layer in the first group of convolutional layers comprising a set of channels; a first subset of channels in the set of channels having a kernel filter with a first size; a second subset of channels included in the set of channels has a kernel filter having a second size; The first size and the second size are different.

10. The method according to any one of claims 1 to 9.

11. the size of the kernel filter for the first subset of channels is 5×5; the size of the kernel filter for the second subset of channels is 3x3; The method of claim 10.

12. the first group of convolutional layers comprises a first number of convolutional layers; the second group of convolutional layers comprises a second number of convolutional layers; the first number is greater than the second number; 12. The method according to any one of claims 1 to 11.

13. 13. The method of claim 12, wherein the first number is two and the second number is one.

14. the first group of convolutional layers includes a first convolutional layer and a second convolutional layer; the first convolutional layer has a kernel filter having a first size; the second convolutional layer has a kernel filter having a second size; The first size and the second size are different.

14. The method of claim 12 or 13.

15. the first size is 5×5; the second size is 3x3; 15. The method of claim 14.

16. the first group of convolutional layers includes a kernel filter having a first plurality of filter values; the second group of convolutional layers includes a kernel filter having a second plurality of filter values; the number of bits in each filter value included in the first plurality of filter values ​​is greater than the number of bits in each filter value included in the second plurality of filter values; 16. The method of any one of claims 1 to 15.

17. the number of bits in each filter value included in the first plurality of filter values ​​is 14 bits; the number of bits for each filter value included in the second plurality of filter values ​​is 7 bits; 17. The method of claim 16.

18. the reconstructed samples are one of reconstructed luma samples and reconstructed chroma samples; The method comprises: obtaining a component value of another one of the reconstructed luma sample and the reconstructed chroma sample; providing the obtained values ​​of the components of the other one of the reconstructed luma samples and the reconstructed chroma samples to a fifth group of one or more convolutional layers; and encoding or decoding the video based on outputs of the fifth group of convolutional layers; and Including, the first group of convolutional layers and the fifth group of convolutional layers differ in number of channels, size of kernel filters, and / or number of convolutional layers; 18. The method of any one of embodiments 1 to 17.

19. A computer program (800) comprising instructions (844) that, when executed by a processing circuit (802), cause the processing circuit to perform the method of any one of claims 1 to 18.

20. 20. A carrier containing the computer program of claim 19, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

21. An apparatus (800) for encoding or decoding video, said apparatus (800) being configured to perform the method of any one of claims 1 to 18.

Images