In-loop neural network for video coding
By adopting the multi-channel training and spatial partition training methods of neural networks in the video encoding and decoding system, the problem of high complexity of loop filters in the prior art is solved, and low-complexity video encoding and decoding gain and image quality improvement are achieved.
Patent Information
- Application Number
- CN202380016170.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-07-22
- Filing Date
- 2023-01-12
- Publication Date
- 2024-12-06
AI Technical Summary
In the existing video encoding and decoding technology, loop filters are complex and difficult to effectively improve image quality and reduce calculation costs.
Using multi-channel training and spatial partitioning training methods based on neural networks, by deploying neural networks in encoders and decoders, using multi-channel training process and spatial partitioning to divide pixels in frames, and using different parameter sets to process different regions. pixels for low complexity codec gain.
It realizes a significant improvement in the image quality of the video codec while maintaining low complexity, reduces calculation costs, and improves the encoding and codec efficiency.
Smart Images

Figure CN119096543A_ABST
Abstract
Description
[0001]
Cross reference
[0002] This disclosure is part of a non-provisional application claiming priority to U.S. Provisional Patent Application No. 63 / 299,058, filed on January 13, 2022, “Multi-pass training for in-loop neural network filtering” and U.S. Provisional Patent Application No. 63 / 369,085, filed on July 22, 2022, “Spatially divided training for in-loop neural network filtering”. The contents of the above two applications are incorporated into this case. [Technical field]
[0003] The present disclosure relates to video encoding and decoding. More specifically, the present disclosure relates to applying Neural Networks (NNs) to target signals in video encoding and decoding systems. [Background technology]
[0004] A neural network, also called an "artificial" neural network (ANN), is an information processing system that has certain performance characteristics in common with biological neural networks. Neural network systems consist of many simple and highly interconnected processing elements that process information through their dynamic responses to external inputs. The processing elements can be thought of as neurons in the human brain, where each perceptron accepts multiple inputs and computes a weighted sum of the inputs. In the field of neural networks, perceptrons are considered to be mathematical models of biological neurons. Furthermore, these interconnected processing elements are often organized in layers. For recognition applications, the external inputs may correspond to patterns presented to the network, which communicates with one or more intermediate layers (also called "hidden layers"), where the actual processing is done through a system of weighted "connections".
[0005] It is desirable to develop a low-complexity neural network-based in-loop filter to enhance the performance of conventional codecs. [Summary of the invention]
[0006] Aspects of the present disclosure provide a method for video decoding. The method includes receiving a video frame reconstructed based on data received from a bitstream. The method also includes extracting a first syntax element from the bitstream indicating whether a spatial partition for partitioning the video frame is valid. The method also includes, in response to the first syntax element indicating that the spatial partition for partitioning the video frame is active, determining a configuration of the spatial partition for partitioning the video frame, determining multiple parameter sets for a neural network, and applying the neural network to the video frame. The video frame is spatially partitioned based on the determined configuration of the spatial partition for partitioning the video frame into multiple parts, and the neural network is applied to the multiple parts according to the determined multiple parameter sets.
[0007] Aspects of the present disclosure provide an apparatus for video decoding. The apparatus includes a circuit configured to receive a reconstructed video frame, the video frame being reconstructed based on data received from a bitstream. The circuit is also configured to extract a first syntax element from the bitstream indicating whether a spatial partition for dividing the video frame is active. The circuit is also configured to determine a configuration of the spatial partition for dividing the video frame in response to the first syntax element indicating that the spatial partition for dividing the video frame is active, determine multiple parameter sets for a neural network, and apply the neural network to the video frame. The video frame is spatially partitioned into multiple parts based on the determined configuration, and the neural network is applied to one of the multiple parts according to each of the multiple parameter sets determined.
[0008] Aspects of the present disclosure provide another method for video encoding. The method includes receiving data representing a video frame. The method also includes determining a configuration of spatial partitions for segmenting the video frame. The method also includes determining multiple parameter sets for a neural network. In addition, the method includes applying the neural network to the video frame. The video frame is spatially partitioned into multiple portions based on the determined configuration, and the neural network is applied to the multiple portions according to the determined multiple parameter sets. In addition, the method includes communicating multiple syntax elements associated with the spatial partitions for segmenting the video frame.
[0009] Please note that this overview section does not specifically describe every embodiment and / or incremental novel aspect of the present disclosure or claimed invention. Instead, this overview only provides a preliminary discussion of different embodiments and corresponding novel points. For additional details and / or possible viewpoints of the present invention and embodiments, the reader is referred to the detailed description section of the present disclosure and the corresponding drawings discussed further below.
Brief Description of the Drawings
[0010] Various embodiments of the present disclosure, presented as examples, will be described in detail with reference to the following drawings, in which like numerals refer to like elements, and in which:
[0011] Figure 1A block diagram of a video encoder based on the Versatile Video Codec (VVC) standard or the High Efficiency Video Codec (HEVC) standard (with an adaptive loop filter (ALF) added);
[0012] Figure 2 A block diagram of a video decoder based on the VVC standard or the HEVC standard (with ALF added);
[0013] Figure 3 A video frame containing a complex spatial variance distribution is shown;
[0014] Figures 4A-4F A plurality of exemplary spatial partitions implemented on a video frame according to an embodiment of the present disclosure are shown;
[0015] Figure 5 A flowchart of a process for implementing a NN-based loop filter in a video encoder according to an embodiment of the present disclosure is shown;
[0016] Figure 6 A flowchart of a process for implementing a NN-based loop filter in a video decoder according to an embodiment of the present disclosure is shown;
[0017] Figure 7 A block diagram illustrating a multi-pass training process of a neural network according to an embodiment of the present disclosure, wherein a single parameter set is used in multiple passes;
[0018] Figure 8 A block diagram showing a multi-channel training process of a neural network according to an embodiment of the present disclosure, wherein different parameter sets are used in multiple channels; and
[0019] Fig. 9 A block diagram of a multi-channel training process of a neural network according to an embodiment of the present disclosure is shown, wherein partially different parameter sets are used in multiple channels. [Specific implementation method]
[0020] The following disclosure provides different embodiments or examples for implementing different features of the subject matter provided. Specific examples of components and arrangements are described below to simplify the disclosure. Of course, these are only examples and are not restrictive.
[0021] Artificial neural networks can use different architectures to specify which variables are involved in the network and their topological relationships. For example, the variables involved in a neural network may be the weights of the connections between neurons, as well as the activity of neurons. A feedforward network is a neural network topology where the nodes of each layer are fed to the next level, and there are connections between nodes in the same layer. Most ANNs contain some form of "learning rule" that modifies the weights of the connections based on the input pattern presented. In a sense, ANNs learn by example like their other biological counterparts. Backpropagation neural networks are a more advanced type of neural network that allow for reverse error propagation of weight adjustments. As a result, backpropagation neural networks are able to improve performance by minimizing the error fed back to the neural network.
[0022] A neural network can be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), or other NN variants. Deep multilayer neural networks or deep neural networks (DNNs) correspond to neural networks with multiple levels of interconnected nodes, allowing them to compactly represent highly nonlinear and highly varying functions. However, the computational complexity of DNNs grows rapidly with the number of nodes associated with a large number of layers.
[0023] CNN is a type of feedforward artificial neural network most commonly used to analyze visual images. A recurrent neural network (RNN) is a type of artificial neural network in which the connections between nodes form a directed graph along a sequence. Unlike feedforward neural networks, RNNs can use their internal state (memory) to process input sequences. RNNs may have loops in them to allow information to persist. RNNs allow operations on sequences of vectors, such as sequences in input, output, or both.
[0024] The High Efficiency Video Codec (HEVC) standard was developed under a joint video project of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG) standardization bodies, specifically in a partnership called the Joint Collaborative Team on Video Codecs (JCT-VC).
[0025] In HEVC, a slice is divided into multiple codec tree units (CTUs). CTUs are further divided into multiple codec units (CUs) to accommodate various local features. HEVC supports multiple intra prediction modes. For intra-coded CUs, the selected intra prediction mode is communicated. In addition to the concept of codec units, the concept of prediction units (PUs) is also introduced in HEVC. Once the splitting of the CU hierarchical tree is completed, each leaf CU is further split into one or more prediction units (PUs) according to the prediction type and PU partition. After prediction, the residual associated with the CU is divided into transform blocks, named transform units (TUs) for the transform process.
[0026] The HEVC standard specifies two loop filters, a deblocking filter (DF) for reducing block artifacts and a sample adaptive offset (SAO) for attenuating ringing artifacts and correcting local average intensity changes. Due to the high bit rate overhead, the final version of HEVC did not use adaptive loop filtering (ALF).
[0027] Compared to previous video codec standards such as HEVC, the Versatile Video Codec (VVC) standard developed by the Joint Video Experts Group (JVET) is designed to achieve significantly improved compression capabilities and is highly versatile to be effectively used in a wider range of applications. In VVC, pictures are divided into codec tree units (CTUs), which represent basic codec processing units and are similarly designated in HEVC. A CTU consists of one or three coding tree blocks (CTBs), depending on whether the video signal is monochrome or contains three color components.
[0028] In VVC, four different loop filters are specified: DF, SAO, ALF and Cross-Component Adaptive Loop Filtering (CC-ALF), which are used to further correct the signal based on linear filtering and adaptive clipping.
[0029] Figure 1A block diagram of a video encoder is shown, which can be implemented based on the VVC standard, the HEVC standard (with ALF added), or any other video codec standard. When the inter-frame mode is used, the intra / inter-frame prediction unit 110 generates an inter-frame prediction based on motion estimation (ME) / motion compensation (MC). When the intra-frame mode is used, the intra / inter-frame prediction unit 110 generates an inter-frame prediction. The intra / inter-frame prediction data (i.e., the intra / inter-frame prediction signal) is provided to the subtractor 115, which is subtracted from the intra / inter-frame prediction signal from the signal associated with the input frame to form a prediction error (also called a residual). The process of generating intra / inter-frame prediction data is referred to as a prediction process in the present disclosure. The prediction error (i.e., the residual) is then processed by transform (T) and quantization (Q) (T+Q, 120). The transformed and quantized residual is then encoded and decoded by the entropy coding unit 125 to be included in the video bit stream corresponding to the compressed video data.
[0030] The bitstream associated with the transform coefficients is then packaged together with auxiliary information such as motion, codec mode, and other information associated with the image area. The auxiliary information can also be compressed by entropy coding to reduce the required bandwidth. Since the reconstructed frame can be used as a reference frame for inter-frame prediction, one or more reference frames must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (IQ) and inverse transform (IT) (IQ+IT, 130) to recover the residual. The reconstructed residual is then added back to the intra / inter prediction data at the reconstruction unit (REC) 135 to reconstruct the video data. The process of adding the reconstructed residual to the intra / inter prediction signal is referred to as the reconstruction process in the present disclosure. The output frame of the reconstruction process is referred to as the reconstructed frame.
[0031] In order to reduce artifacts in the reconstructed frame, loop filters are used, including but not limited to 130. In the present disclosure, DF, SAO and ALF are all labeled as filtering processes. The filtered reconstructed frame at the output of all filtering processes is referred to as a decoded frame in the present disclosure. The decoded frame is stored in a frame buffer 155 and used to predict other frames.
[0032] Figure 2A block diagram of a video decoder is shown, which can be implemented based on the VVC standard, the HEVC standard (with ALF added), or any other video codec standard. Since the encoder contains a local decoder for reconstructing video data, many decoder components have been used in the encoder in addition to the entropy decoder. On the decoder side, the entropy decoding unit 226 is used to recover the coded symbols or syntax from the bitstream. The process of generating a reconstructed residual from an input bitstream is referred to as a residual decoding process in the present disclosure. The prediction process for generating intra / inter prediction data is also applied to the decoder side, however, the intra / inter prediction unit 211 is different from the intra / inter prediction unit 110 on the encoder side because inter prediction only requires motion compensation to be performed using motion information derived from the bitstream. In addition, the adder 215 is used to add the reconstructed residual to the intra / inter prediction data.
[0033] In general, embodiments of the present disclosure relate to using neural networks to improve the image quality of video codecs. The neural network is deployed as a filtering process at the encoder side and the decoder side. The parameters of the neural network are learned at the encoder and transmitted to the decoder in a bitstream along with various information about how to apply the neural network at the decoder side based on the transmitted parameters.
[0034] The neural network is run at the same loop position in the decoder as in the encoder. This position can be chosen to be at the output of the reconstruction process or at the output of one of the filtering processes. Figure 1 and Figure 2 As shown in the video codec, the neural network can be applied to the reconstructed signal from the reconstruction unit 135 / 235, or the filtered reconstructed signal from the DF 140 / 240, SAO 145 / 245, ALF 150 / 250 or any other type of loop filter. The specific location of the neural network can be predefined or can be signaled from the encoder to the decoder.
[0035] Notice, Figure 1 and Figure 2 The sequence of filters DF, SAO, and ALF shown in is not limiting. Although three types of filters are illustrated here, it does not limit the scope of the present disclosure, as fewer or more filters may be included.
[0036] Two types of variance are considered when designing filtering tools with neural networks: temporal variance and spatial variance. It is observed that temporal variance is small in random access segments (RAS); therefore, training a neural network on 128 frames can achieve almost the same codec gain as training 8 neural networks on every 16 frames.
[0037] In contrast, the spatial variance is usually large within a single frame. Figure 3 A typical video frame with a complex spatial variance distribution is shown. The upper half of the image has various textured regions, such as the sky, buildings, trees, and people, while the content in the lower half is relatively homogeneous. This results in the neural network having to learn different reconstruction error statistics in order to predict the error for each pixel of the image.
[0038] In order to take into account the different predictors required in different spatial regions, it is beneficial to divide the pixels in the frame into multiple parts and train different neural network parameters for each part. Since each part has a specific set of parameters that define the predictors dedicated to the pixels within that specific part, this parameter set is well suited to the reconstruction error statistics of a relatively small part. With this approach, large codec gains can be achieved with a lightweight neural network with lower complexity and less computational cost.
[0039] Figures 4A-4F A number of possible modes for dividing pixels in a frame into a number of portions according to an embodiment of the present disclosure are illustrated. Figures 4A-4C Three fixed partitioning patterns are shown, namely horizontal partitioning (4A), vertical partitioning (4B), and quadrant partitioning (4C). Figures 4D-4F A non-limiting example of block partitioning is shown in FIG. Those skilled in the art will appreciate that other partitioning schemes are also feasible without departing from the scope of the present disclosure.
[0040] In one embodiment, the partitioning pattern used in the codec may be predefined. Alternatively, for a frame (e.g., an I frame), the encoder may select one from a set of available partitioning patterns and notify the decoder, for example, what partitioning pattern is selected for the current frame.
[0041] Figure 5 A flow chart of a process 500 for implementing a NN-based loop filter in a video encoder according to an embodiment of the present disclosure is shown. At step 510, data representing a video frame is obtained. For example, the frame is an I frame. As described above, the data may be obtained at the output of the reconstruction unit REC 135 or at the output of any filter (including but not limited to DF 140, SAO 145, and ALF 150).
[0042] Assuming that the encoder has decided to activate the spatial partitioning mode, it is determined at step 520 what spatial partitioning configuration will be used to divide the frame. As described above, the spatial partitioning can be predefined; alternatively, the encoder can adaptively select different spatial partitioning for different frames. Furthermore, the spatial partitioning can be shared by all frames in a sequence of frames. For example, in the case of an I-frame, the encoder can select one of horizontal partitioning, vertical partitioning, and quadrant partitioning, or define a specific block-based partitioning in order to divide the frame into the desired number of parts. If the frame is a B-frame or a P-frame, the encoder simply reuses the spatial partitioning determined for the I-frame.
[0043] At step 530, the parameter sets for the neural network are determined. That is, for each portion of the frame, the encoder decides which parameter sets to use to construct the neural network. For example, the left portion of the frame may correspond to a portion with parameter set θ l The neural network developed with the parameter set θr is applied to the right part of the frame. l and θ r can be completely different. Alternatively, some layers, filters, weights and / or biases of the neural network can have some common parameters. Similarly, a new set of parameters can be determined for an I frame, and if the frame is a P frame or a B frame, the parameter set is the parameter set previously determined for the I frame. Figure 7-9 Describe in detail the training process used to learn the parameters of a neural network.
[0044] Based on the spatial partitions determined at step 520 and the set of neural network parameters determined at step 530, a neural network is applied to portions of the frame at step 540. Because each portion is processed by the neural network using a set of parameters tailored for that particular portion, the neural network can fit the corresponding error statistics with a small number of operations per pixel.
[0045] At step 550, the encoder generates and transmits various syntax elements (flags) to the decoder to indicate how the neural network is deployed on the decoder side. For example, one syntax element may indicate whether the spatial partitioning mode is active or inactive, another syntax element may indicate the position of the neural network in the loop, etc.
[0046] Other syntax elements may indicate the spatial partitioning scheme, the parameter set of the neural network, and the correspondence between multiple parts and multiple parameter sets. The codec may use any combination of one or more fixed partitioning patterns and / or one or more block-wise partitioning patterns. In this case, for a certain frame, the encoder may send one or more syntax elements to indicate which partitioning pattern is valid. Likewise, the spatial partitioning scheme may be predefined rather than signaled by a syntax element.
[0047] Optionally, additional syntax elements may be used to indicate whether and how parameters are shared between two or more parts. In addition, when a neural network is referenced by Fig. 9 When training a multi-pass process described above, a set of syntax elements can be used to indicate how to derive a parameter set for a current frame by replacing some parameters of a previously transmitted parameter set. The above syntax elements can be transmitted at the frame level, for example. Non-limiting examples of syntax elements are given in Tables 1 and 2 below.
[0048] Figure 6 A flow chart of a process for implementing a NN-based loop filter in a video decoder according to an embodiment of the present disclosure is shown. Process 600 begins at step 610 by obtaining a video frame reconstructed based on data received from a bitstream. The video frame may be a reconstructed frame (output from REC 235) or a filtered reconstructed frame (output from DF 240, SAO 245, or ALF 250).
[0049] At step 620, syntax elements are extracted from the bitstream. For example, one of the syntax elements may indicate whether the spatial partitioning mode is valid. Other syntax elements may indicate the spatial partitions that divide the frame, neural network parameters, how to develop the neural network with these parameters, etc. As described above, some information may be predefined or reused. For example, for P-frames or B-frames, previously determined spatial partitions and parameter sets may be reused, so syntax elements are not required for these frames.
[0050] Based on the parsed syntax elements (and optionally predefined information and / or reuse information), a spatial partitioning configuration is determined at step 630 to divide the frame into a plurality of portions, and a plurality of neural network parameter sets are determined at step 640. At step 650, a neural network is developed with one of the plurality of parameter sets and applied to each of the plurality of portions of the frame.
[0051] Table 1 lists a set of syntax elements defined in a non-limiting example of the present disclosure. These syntax elements can be transmitted at the frame level to inform the decoder of various information, including but not limited to whether the spatial partitioning mode is activated, which one of a set of spatial partitioning candidates is selected, whether new neural network parameters are available, how to share neural network parameters among parts, and which parameter set to apply for a specific part.
[0052] Table 1: Syntax elements for an example of a messaging space partition configuration and associated parameter sets
[0053]
[0054] In Table 1, the presence of syntax elements with higher numbers (indicated by "#") may be conditional on syntax elements with lower numbers. Syntax element #1 indicates whether the spatial partitioning mode is activated. If the spatial partitioning mode is activated, #1 may be followed by two Boolean syntax elements #2 and #3. Syntax element #2 indicates whether a new spatial partitioning configuration is transmitted and valid starting from this frame. Syntax element #3 indicates whether a new network parameter set is transmitted and valid starting from this frame. Note that after an I frame, syntax elements #2 and #3 may not be required because there is no new partitioning configuration and no new parameter set to be transmitted.
[0055] If syntax element #2 is set, syntax element #4 indicates the configuration of the spatial partitioning, i.e., which spatial partitioning pattern is used. The spatial partitioning pattern can be a fixed spatial partitioning, where the frame is divided into two halves (upper / lower or left / right) or four quadrants of equal size. Otherwise, the spatial partitioning pattern refers to a block-based partitioning, where each part is associated with one of the parameter sets. If syntax element #4 indicates a fixed partitioning, syntax element #5 indicates which partitioning is used. From the partitioning, the number of required parameter sets P can be inferred. On the other hand, if syntax element #4 indicates a block-based partitioning, syntax element #6 indicates the number of available parameter sets P, one of which is selected per part. Furthermore, syntax element #7 then contains a series of integers, one per part, referencing a parameter set, so that the maximum value of each integer is given by P-1.
[0056] If syntax element #3 is set, a new set of neural network parameters is transmitted and is valid starting from the current frame. The parameter sets associated with different parts can be completely different, but this is not required. That is, parameter sets can be partially shared between parts at the layer level, filter level or filter element level.
[0057] For example, a neural network has a 5-layer structure; under horizontal partitioning, the frame is divided into two halves. The neural network for the upper half can share the same layer 1 and the same layer 5 with the neural network for the lower half, while layers 2-4 of the two halves are different. In this case, the sharing specification on how to share the neural network parameter set can be indicated by one or more syntax elements.
[0058] For example, syntax element #8 indicates whether each layer l of the neural network is shared between parameter sets. If layer l is not shared, then each parameter set p has a parameter group If l is shared, syntax element #9 indicates the total number of parameter groups G for layer l. l Each parameter group Need to be associated with a parameter set. This information is signaled in syntax element #10. For each layer and each parameter set, an integer is signaled indicating which Gl parameter groups to construct parameter sets. Note that if G l =1, that is, there is only one parameter set, and there is no need to signal #10 for layer k.
[0059] According to the information transmitted, the decoder transforms the neural network with the parameter set θ p Assemble them and apply the neural network to the relevant parts of the frame.
[0060] Note that the set of syntax elements listed in Table 1 is not restrictive. For example, in one embodiment, only some fixed partitions are supported and block partitioning is not allowed; therefore, syntax elements with one or more different types, value ranges, and meanings can be defined from #2 and #3. In another embodiment, for syntax elements #8, #9, and #10, whether parameters in a layer are shared is predetermined and does not require signaling. In yet another embodiment, for syntax element #7, this selection and other syntax elements can be signaled at the CTU level with one CTU. In yet another embodiment, spatial partitioning is predefined and does not require signaling.
[0061] As mentioned above, a training process is required on the encoder side to derive the parameters of the neural network. When training a neural network-based filter during or after encoding a series of frames, only decoded frames without the noise suppressing influence of the neural network are used as training data. If the neural network is run in post-loop mode, the training data is exactly matched to the test data (e.g., data to be processed or decoded frames). However, when used as a loop filtering tool, the neural network will change the frame f a , which is then used as (for example) the subsequent coded frame f b Since the neural network is not available during the encoding process to generate the training data, the frame f b Different frames than those used during training lead to different error statistics. In order to take into account the re-application of the trained neural network to its own output during the loop operation, a multi-pass training procedure is proposed.
[0062] Figure 7 A block diagram of an exemplary multi-channel training process of a neural network according to an embodiment of the present disclosure is shown, wherein a single set of neural network parameters is used in multiple channels. Figure 7In the first pass, the reconstructed data (represented by reconstructed Y / Cb / Cr) is taken as input and combined with auxiliary inputs such as motion vectors, residuals, and / or position information. For example, the position information may inform the neural network of the location of the pixel that the neural network is processing. The output of the first neural network is added to the reconstructed Y / Cb / Cr to produce output O1. Using output O1 and the auxiliary inputs, another pass of the neural network is calculated using the same parameters as the first pass. Once again, a second output O2 is generated by adding the output of the second neural network to the reconstructed Y / Cb / Cr. This process can continue for any number of passes, creating a new output O in the nth pass. n .
[0063] The error between the output and the original signal Y / Cb / Cr (ground truth) can be calculated for n outputs O1, O2, ..., O n To update the neural network parameters using the gradient descent algorithm, the final loss can be calculated as where the weights wn can be chosen arbitrarily. When the final loss converges, the learned neural network parameters can be quantized and signaled to the decoder, where the neural network is applied in the loop to the reconstructed Y / Cb / Cr. Note that as mentioned before, the filtered reconstructed data can be used instead of the reconstructed Y / Cb / Cr, for example, the data output from any of DO, SAO, and ALF.
[0064] exist Figure 7 In the illustrated embodiment, the multi-channel training process simulates the output of a neural network being improved one or more times by the same neural network in succession. Other embodiments of the present disclosure may simulate the output of a neural network being improved by one or more different or partially different neural networks, such as Figure 8-9 shown.
[0065] Figure 8 A block diagram of an exemplary multi-channel training process of a neural network according to an embodiment of the present disclosure is shown, where different parameter sets are used in multiple channels. In each channel, the neural network has a separate set of parameters, so there will be N sets of parameters for N channels. In a non-limiting example, only the first parameter set is signaled to the decoder and the other parameter sets are discarded. Alternatively, if the first n (n≤N) neural networks are to be used in series, the first n parameter sets may be signaled.
[0066] Figure 8The illustrated embodiment simulates the loop application of multiple neural networks trained on consecutive frames. For example, a set of neural network parameters θ1 is trained and used for encoding and decoding of a first set of frames; thereafter, another set of parameters θ2 is trained and used for encoding and decoding of a second set of frames. In this case, the first set of parameters θ1 can be trained while taking into account that when content is referenced in subsequent frames, its output may be reprocessed by a neural network with a different second set of parameters θ2.
[0067] Fig. 9 A block diagram of an exemplary multi-channel training process of a neural network according to an embodiment of the present disclosure is shown, wherein partially different parameter sets are used in multiple channels. In this embodiment, the neural network parameter sets θ1, θ2, ..., θ n Only some of the parameters of each neural network (in Fig. 9 Some of the parameters (referred to as "shared NN parameters" in the example) are the same, while others (referred to as "NN parameters θ1" and "NN parameters θ2") are specific to a single neural network. The distinction between common parameters and individual parameters can be layer-level, filter-level, or element-level. Using this mechanism, only individual parts of the parameters have to be communicated for subsequently trained neural networks, thereby reducing rate overhead.
[0068] To inform the decoder which part of the neural network is being replaced, appropriate syntax elements can be inserted into the frame header as shown in Table 2 below.
[0069] Table 2: Syntax elements for example of message parameter replacement
[0070]
[0071] For example, syntax element #1 is a Boolean value used to signal whether a new set of network parameters is included in the frame header. If this is the case, syntax element #2 will be present to indicate whether a new complete set of parameters is signaled (syntax element #2 is set to 0) or only a partial set. In the case of a partial set, syntax element #2 indicates which network parameter set was used as a basis and then some parts of it were replaced, where syntax element #2 is an index in the list of previously received network parameter sets (including network parameter sets created by partially replacing a base network parameter set). The index starts with 1, indicating the most recently received network parameter set.
[0072] If syntax element #2 signals a replacement, then syntax element #3 indicates the type of replacement. If syntax element #3 is set to 0, it ends the replacement signaling. Otherwise, it indicates that a layer (value: 1), filter (value: 2), weight (value: 3), or bias (value: 4) is being replaced. Syntax element #4 specifies which layer of the neural network the replacement refers to. If syntax element #3 represents a filter, weight, or bias, then syntax element #5 will indicate the corresponding filter, which is either replaced in its entirety or the weight or bias therein is replaced. If syntax element #3 represents a weight, then syntax element #6 is present to indicate which weight is to be replaced.
[0073] By extracting this information, it is now possible to infer the data type and number of entries to be extracted from the entropy codec such as CABAC, VLC, etc. The data type depends on whether weights or biases are being read, and what data type the previously signaled network used to transmit the parameters. These data types can be integers up to 32 bits or floating point numbers up to 32 bits. After the parameters are decoded, another syntax element #3 is read. If it is equal to zero, the parameters for the new network are complete, otherwise the process proceeds as described until a syntax element #3 equal to 0 is read after reading the parameters.
[0074] In the present disclosure, specific syntax and semantics have been used to illustrate examples to implement embodiments of the present invention. Without departing from the spirit of the present invention, a person skilled in the art can implement the present invention by replacing the syntax and semantics with equivalent syntax and semantics.
[0075] Aspects of the disclosure are further described as follows.
[0076] I. Multi-channel training of loop neural network filters
[0077] Recent results (see, e.g., Refs. 1 and 2) have demonstrated that small neural networks (NNs) requiring only a few hundred operations per pixel can achieve codec gains if trained as a post-loop filter on a limited set of at most a few hundred frames and signaled to the decoder using quantized or raw floating point parameters.
[0078] Reference 1: JPKlopp, L.-G.Chen and S.-Y.Chien, Utilizing Low ComplexityCNNs to Lift Non-Local Redundancies in Video Coding, IEEE Transactions onImage Processing, 2020.
[0079] Reference 2: JPKlopp, K.-C.Liu, S.-Y.Chien and L.-G.Chen, Online-trainedUpsampler for Deep Low Complexity Video Compression, in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021.
[0080] In those works, the NN is used as a post-loop filter after the decoding loop, i.e., the output of the NN is not used as a reference for another frame. This limits the impact of the NN on noise reduction and codec gain, since the processed content is not reused. Applying the post-loop training process to the loop encoding process creates a mismatch between training and test data, since the NN has to process data created by referencing its output (e.g., motion compensation). To mitigate this drawback, we propose a different neural network in-loop training technique.
[0081] The proposed method uses convolutional neural network (CNN) as an image restoration method in video codec systems. Figure 1 and Figure 2 As shown in , we can apply CNN to the ALF output picture to generate the final decoded picture. Alternatively, CNN can be applied directly after SAO, DF or REC, with or without other recovery methods in a video codec system, such as Figure 1 and Figure 2 shown.
[0082] In order to allow for the re-application of a trained CNN to its own content during in-loop operation, multi-pass training is proposed. When training a CNN during or after encoding a sequence of frames, only decoded frames without the influence of CNN noise suppression are used as training data. When operating in post-loop mode, this training data is perfectly matched to the test data. However, in in-loop mode, the CNN will change the frame f a , which is then used as the subsequent coded frame f b Therefore, frame f b will be different from the frames used during training, since the CNN was not available during the encoding pass that generated the training data.
[0083] In one embodiment, a single set of parameters is used to continuously process the outputs, such as Figure 7As shown. The first execution of the neural network takes the "reconstructed Y / Cb / Cr" input from the decoder and combines it with "auxiliary inputs" such as motion vectors, residuals, or position information. The output of the neural network is added to the "reconstructed Y / Cb / Cr" to produce output O1. This output is used together with the auxiliary input to calculate another channel of the neural network using the same parameters as before. The output of the second channel is added to the "reconstructed Y / Cb / Cr" to produce output O2. This process can continue for any number of channels, creating a new output O in the nth channel. n For each output O n , we can calculate the loss by calculating the error between the output and the original Y / Cb / Cr To update the neural network parameters using gradient descent, a final loss is calculated as The weight w n Once trained, the neural network parameters can be quantized and passed to the decoder where the neural network is applied in-loop to reconstructed Y / Cb / Cr.
[0084] In another embodiment, it is proposed to use a neural network with a separate set of parameters for each channel, such as Figure 8 As shown. For N channels, there will be N sets of parameters. This simulates the in-loop application of multiple neural networks trained on consecutive frames. In in-loop processing, one set of parameters is trained taking into account that its output may be reprocessed by another set when the content is referenced in subsequent frames. Only the first or first n ≤ N sets of parameters are sent to the decoder.
[0085] In another proposed example of this training scheme, the neural network parameter set θ n Only some of them are different, such as Fig. 9As shown. Some parameters of each neural network are shared, other parameters are specific to a single neural network. The distinction between shared parameters and individual parameters can be layer-level, filter-level or element-level. Using this mechanism, only individual parts of the parameters have to be signaled for subsequently trained neural networks, thus reducing rate overhead. In order to inform the decoder which part of the neural network is being replaced, appropriate flags are inserted into the frame header, as shown in Table A: Flags for signaling parameter replacement. Flag #1 is a Boolean value that is signaled if a new set of network parameters is contained in the frame header. If this is the case, flag #2 will be present to indicate whether a new complete set of parameters is signaled (flag #2 set to 0) or only a partial set. In the case of a partial set, flag #2 indicates which network is used as a basis and then some parts of it are replaced, where flag #2 is an index in the list of previously received networks (including networks created by partially replacing the base network). The index starts with 1, indicating the most recently received network. If flag #2 indicates a replacement, flag #3 indicates the type of replacement. If flag #3 is set to 0, it ends the replacement signaling. Otherwise it indicates that a layer (value: 1), filter (value: 2), weight (value: 3), or bias (value: 4) is being replaced. Flag #4 specifies which layer of the neural network the replacement refers to. If flag #3 indicates filters, weights, or biases, then flag #5 will indicate the corresponding filter, which is either completely replaced or has weights or biases replaced. If flag #3 indicates weights, then flag #6 appears to indicate which weight is being replaced. By extracting this information, it is now possible to infer the data type and the number of entries to be extracted from the entropy codec (such as CABAC, VLC, or other). The data type depends on whether weights or biases are being read, and what data type the previously signaled network used to transmit parameters. These data types can be integers up to 32 bits or floating point numbers up to 32 bits. After the parameters are decoded, another flag #3 is read. If it is equal to zero, the new network parameters are complete, otherwise the process proceeds as described until a flag #3 equal to 0 is read after reading the parameters.
[0086] Table A: Flags used for message parameter substitution
[0087]
[0088] II. Spatial Partition Training of In-Loop Neural Network Filters
[0089] Recent results (see, e.g., Refs. 1 and 2) have demonstrated that small neural networks (NNs) requiring only a few hundred operations per pixel can achieve codec gains if trained as a post-loop filter on a limited set of at most a few hundred frames and signaled to the decoder using quantized or raw floating point parameters.
[0090] In those works, the NN is used as a post-loop filter after the decoding loop, i.e., the output of the NN is not used as a reference for another frame. This limits the impact of the NN on noise reduction and codec gain, since the processed content is not reused. Applying the post-loop training process to the loop encoding process creates a mismatch between training and test data, since the NN has to process data created by referencing its output (e.g., motion compensation). To mitigate this drawback, we propose a different neural network in-loop training technique.
[0091] The proposed method uses convolutional neural network (CNN) as an image restoration method in video codec systems. Figure 1 and Figure 2 As shown in , we can apply CNN to the ALF output picture to generate the final decoded picture. Alternatively, CNN can be applied directly after SAO, DF or REC, with or without other recovery methods in a video codec system, such as Figure 1 and Figure 2 shown.
[0092] Different regions of a frame sequence usually have different contents. This may cause the CNN to have to learn different reconstruction error statistics to predict the error of each pixel. In order to consider the different predictors required for different spatial regions of the frame sequence, spatial partitioning training is proposed. Spatial partitioning training divides the pixels in a frame into different groups. Each group has a parameter set θ p , which defines the predictors used for the pixels in the group. The parameter sets can, but do not have to be, distinct. Parameters organized in filters, layers, or groups can be shared between parameter sets.
[0093] The spatial partitioning can be based on a fixed partitioning pattern, such as horizontal or vertical partitioning into two half-frames or block-wise, where the parameter set used can be different for each block.
[0094] Table B lists the flags used to signal to the decoder whether spatial partitioning is valid, as well as the configuration of spatial partitions and the (possibly shared) parameter sets associated with those spatial partitions.
[0095] Table B: Communication space partition configuration and related parameter level symbols
[0096]
[0097] These flags are signaled at the frame level. The presence of a flag with a higher number (indicated by a "#") may be conditional on flags with lower numbers. The first flag indicates whether spatial partitioning is active. If this is the case, it is followed by two Boolean flags, the first indicating whether a new spatial partitioning configuration has been transmitted and is valid from this frame onwards. The second indicates whether a new set of network parameters has been transmitted and is valid from this frame onwards.
[0098] If flag #2 is set, flag #4 indicates which kind of spatial partitioning is used. This can be a fixed spatial partitioning, where the frame is divided into two halves (upper / lower or left / right) or into four quadrants of equal size. Otherwise, it refers to a block-wise partitioning, where each block is associated with one of the parameter sets. If #4 indicates a fixed partitioning, #5 indicates which kind of partitioning is used. From the partitioning, the number P of required parameter sets can be inferred. On the other hand, if #4 indicates a partitioning by block, then #6 indicates the number P of available parameter sets, where one is selected per block. Furthermore, #7 then contains a series of integers, one per part, referencing a parameter set, so that the maximum value of each integer is given by P-1.
[0099] If flag #3 is set, flag #8 indicates whether each layer l is shared between parameter sets. If layer l is not shared, then each parameter set p has a parameter group If l is shared, then #9 represents the total number of parameter groups G of layer l l Each parameter group needs to be associated with a parameter set. This information is signaled in flag #10. For each layer and each parameter set, an integer is signaled indicating which G1 parameter group was referenced to construct the parameter set. Note that if G l =1, that is, there is only one parameter set, then for layer l, signaling #10 is not required.
[0100] Note that after an I-frame, flags #2 and #3 are not required because there is no partition configuration and no parameter set available.
[0101] Using the transmitted information, the decoder assembles the parameter set θ p , which determines the functionality of the CNN. The CNN is then applied to the restored image, such as described in Refs. 3–5, where the set of parameters is chosen based on the pixels being reconstructed.
[0102] Reference 3: C.-Y. Chen, T.-D. Chuang, Y.-W. Huang and JPKlopp, Method and Apparatus of Neural Networks with Grouping for Video Coding, U.S. Patent Application No. 16 / 963,566, 2021.2.25.
[0103] Reference 4: Y.-L. Hsiao, Y.-C. Su, JPKlopp, C.-Y. Chen, T.-D. Chuang, C.-W. Hsu and Y.-W. Huang, Method and Apparatus of Neural Network for Video Coding, U.S. Patent Application No. 17 / 047,244,2021.7.3.
[0104] Reference 5: Y.-C. Su, JPKlopp, C.-Y. Chen, T.-D. Chuang, and Y.-W. Huang, Method and Apparatus of Neural Network for Video Coding, U.S. Patent Application No. 16 / 646,624, 2020.8.6.
[0105] The above description is an example. It is not necessary to apply all parts of the above method together. For example, in one embodiment, for flag #2, only some fixed partitions are supported, and block partitioning is not allowed. In another embodiment, for syntax #8, #9 and #10, whether parameters in a layer are shared or not is predetermined and does not need to be signaled. In another embodiment, for syntax #7, the selection is signaled at the CTU level together with other syntax elements in a CTU.
[0106] Any of the aforementioned proposed methods may be implemented in an encoder and / or decoder. For example, any of the proposed methods may be implemented in a loop filtering process of an encoder and / or decoder. Alternatively, any of the proposed methods may be implemented as a circuit coupled to a loop filtering process of an encoder and / or decoder to provide information required for the loop filtering process.
[0107] Those with ordinary knowledge in the field will also understand that many changes can be made to the operation of the above-mentioned technology while still achieving the same goal of the present disclosure. Such changes are intended to be included in the scope of the present disclosure. Therefore, the foregoing description of the embodiments of the present disclosure is not intended to be limiting. On the contrary, any limitation to the embodiments of the present disclosure is proposed in the appended claims.
Claims
1. A video decoding method, comprising: receiving a video frame reconstructed based on data received from a bitstream; extracting a first syntax element from the bitstream, the first syntax element indicating whether a spatial partition for segmenting the video frame is valid; and In response to the first syntax element indicating that the spatial partition for segmenting the video frame is valid: determining a configuration of the spatial partitions for segmenting the video frame, determine multiple parameter sets for the neural network, and Apply the neural network to the video frame, where The video frame is spatially partitioned based on the determined configuration of the spatial partitions for dividing the video frame into a plurality of portions, and the neural network is applied to the plurality of portions according to the determined plurality of parameter sets.
2. The method according to claim 1, wherein the step of determining the configuration further comprises: extracting from the bitstream a second syntax element indicating whether a new configuration is available, When the second syntax element indicates that no new configuration is available, using the configuration determined for the previous video frame as the determined configuration, and When the second syntax element indicates that a new configuration is available, the new configuration is obtained from the bitstream, and the obtained configuration is used as the determined configuration.
3. The method according to claim 2, wherein the step of acquiring the new configuration further comprises: extracting from the bitstream a first set of one or more further syntax elements indicating a particular configuration, and The particular configuration is identified as the new configuration.
4. The method according to claim 3, wherein the specific configuration is: Horizontal partitioning, dividing the video frame into upper and lower parts, Vertical partitioning, dividing the video frame into left and right parts, Quadrant partitioning, which divides the video frame into upper left, upper right, lower left, and lower right parts, or Block partitioning divides the video frame into a specific number of parts, which is neither 2 nor 4.
5. The method according to claim 1, wherein the step of determining the configuration further comprises using a predefined configuration as the determined configuration, and the predefined configuration is: Horizontal partitioning, dividing the video frame into upper and lower parts, Vertical partitioning, dividing the video frame into left and right parts, Quadrant partitioning, which divides the video frame into upper left, upper right, lower left, and lower right parts, or Block partitioning divides the video frame into a specific number of parts, which is neither 2 nor 4.
6. The method according to claim 1, wherein the step of determining the plurality of parameter sets further comprises: extracting from the bitstream a third syntax element indicating whether a new plurality of parameter sets are available, When the third syntax element indicates that no new parameter sets are available, using a plurality of parameter sets determined for a previous video frame as the determined plurality of parameter sets, and When the third syntax element indicates that new multiple parameter sets are available, the new multiple parameter sets are obtained from the bitstream, and the obtained multiple parameter sets are used as the determined multiple parameter sets.
7. The method according to claim 6, wherein the step of obtaining the new plurality of parameter sets further comprises: extracting from the bitstream a second set of one or more further syntax elements indicating the particular plurality of parameter sets, and The particular plurality of parameter sets is identified as the new plurality of parameter sets.
8. The method according to claim 6, wherein the step of obtaining the new plurality of parameter sets further comprises: extracting from the bitstream a second set of one or more further syntax elements indicating a particular previous video frame and a replacement specification defining replacements for some of the plurality of parameter sets determined for the particular previous video frame, and The new plurality of parameter sets are generated based on the plurality of parameter sets determined for the particular previous video frame and the replacement specification.
9. The method according to claim 8, wherein: The replacement is at the layer level, filter level or filter element level of the neural network.
10. The method according to claim 6, wherein the step of obtaining the new plurality of parameter sets further comprises: extracting from the bitstream a second set of one or more further syntax elements indicating the particular plurality of parameter sets, extracting from the bitstream a third set of one or more further syntax elements indicating a sharing specification defining that some of the particular plurality of parameter sets are shared between two or more of the plurality of portions, and According to the sharing specification, the new plurality of parameter sets are generated based on the specific plurality of parameter sets.
11. The method of claim 10, wherein some parameter sets of the particular plurality of parameter sets are shared between the two or more of the plurality of parts at a layer level, a filter level or a filter element level of the neural network.
12. The method according to claim 6, wherein the step of obtaining the new plurality of parameter sets further comprises: extracting from the bitstream a second set of one or more further syntax elements indicating the particular plurality of parameter sets, and The new plurality of parameter sets are generated based on the particular plurality of parameter sets according to a predefined sharing specification defining that some of the particular plurality of parameter sets are shared between two or more of the plurality of portions.
13. The method of claim 1, wherein the step of determining the plurality of parameter sets further comprises: extracting from the bitstream a fourth set of one or more further syntax elements indicating a corresponding specification defining a correspondence between one of the plurality of parts and each of the determined plurality of parameter sets, The steps of applying the neural network also include: Based on the corresponding specification, the neural network having the determined one of the plurality of parameter sets is applied to a corresponding one of the plurality of portions.
14. The method of claim 1, wherein the video frame is received from an output of a reconstruction unit, an adaptive loop filter, a sample adaptive offset filter, or a deblocking filter.
15. An apparatus for video decoding, comprising a circuit configured to perform the following operations: receiving a video frame reconstructed based on data received from a bitstream; extracting a first syntax element from the bitstream, the first syntax element indicating whether a spatial partition for segmenting the video frame is valid; and In response to the first syntax element indicating that the spatial partition for segmenting the video frame is valid: determining a configuration of the spatial partitions for segmenting the video frame, determine multiple parameter sets for the neural network, and The neural network is applied to the video frame, wherein the video frame is spatially divided into a plurality of portions based on the determined configuration, and the neural network is applied to the plurality of portions according to the determined plurality of parameter sets.
16. A video encoding method, comprising: receiving data representing a video frame; determining a configuration of spatial partitions for segmenting the video frame; Determine multiple parameter sets for the neural network; as well as applying the neural network to the video frame, wherein the video frame is spatially partitioned into a plurality of portions based on the determined configuration, and the neural network is applied to the plurality of portions according to the determined plurality of parameter sets; as well as Syntax elements associated with the spatial partitions used to partition the video frame are signaled.
17. The method according to claim 16, further comprising: The neural network is trained by cascading N training stages to learn each of the multiple parameter sets, where N ≥ 2, Each training stage includes the neural network to be trained. Given 2≤n≤N, the input of the nth training stage is derived based on the output of the (n-1)th training stage, The data representing the reconstructed video frames are used as training data input to the first training stage, The data of the original video frame representing the reconstructed video frame is used as the ground truth, and The total loss is calculated as the weighted sum of the losses over the N training stages.
18. The method according to claim 16, wherein: The neural networks of the N training stages are developed with the same parameter set, or N parameter sets are dedicated to each training stage.
19. The method according to claim 17, wherein: The neural networks in the N training stages are developed using parameter sets that are partially shared between the N training stages at the layer level, filter level, or filter element level.
20. The method of claim 17, wherein the data representing the reconstructed video frame is obtained from an output of a reconstruction unit, an adaptive loop filter, a sample adaptive offset filter, or a deblocking filter.
Citation Information
Cited By
Encoding method and decoding method, encoder and decoder, and storage medium
WO2026152887A1
