Content-adaptive overfitting techniques for video coding using neural networks

Abstract

Methods and systems are described for content adaptive overfitting (CAO) techniques in video coding using neural networks (NN). Using a two-pass encoding of the same NN model under evaluation, a selection is made between a baseline model and a CAO model based on a loss metric dependent on bit rate overhead to transmit updated CAO weights and improvements in quality. The updated NN weights are transmitted using a k-th order Golomb encoder. Example high-level syntax to support CAO for a neural network-based loop filter is described as well.

Description

CONTENT-ADAPTIVE OVERFITTING TECHNIQUES FOR VIDEO CODING USING NEURAL NETWORKSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from Indian Provisional Patent Application No. 202411074474, filed on October 1, 2024, and Indian Provisional Patent Application No. 202511023418 filed on March 17, 2025, each of which is incorporated by reference herein in its entirety.TECHNOLOGY

[0002] The present document relates generally to images and video coding. More particularly, embodiments of the present invention relate to content- adaptive overfitting (CAO) techniques for video coding using neural networks.BACKGROUND

[0003] In 2020, the MPEG group in the International Standardization Organization (ISO), jointly with the International Telecommunications Union (ITU), released the first version of the Versatile Video Coding Standard (VVC), also known as H.266. More recently, the same group has been working on the development of the next generation coding standard that provides improved coding performance over existing video coding technologies. As part of this investigation, new coding techniques are also examined.

[0004] As appreciated by the inventors here, improved techniques for video coding techniques using deep learning and neural networks are desired, and they are described herein.The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more approaches should not be assumed to have been recognized in any prior art on the basis of this section, unless otherwise indicated.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] An embodiment of the present invention is illustrated by way of example, and not in way by limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0006] FIG. 1 depicts an example processing pipeline using a neural-network loop filter (NNLF) with content- adaptive overfitting (CAO);

[0007] FIG. 2A-FIG. 2D depict an example of an encoding CAO processing pipeline according to an embodiment of this invention;

[0008] FIG. 3 depicts an example of a decoding CAO processing pipeline according to an embodiment of this invention; and

[0009] FIG. 4 depicts an alternative example process for CAO training according to an embodiment of this invention.DESCRIPTION OF EXAMPLE EMBODIMENTS

[0010] Example embodiments that relate to applying content-adaptive overfitting techniques in video coding using deep learning are described herein. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments of the present invention. It will be apparent, however, that the various embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating embodiments of the present invention.SUMMARY

[0011] The present disclosure provides a method of encoding video sequences using a neural-network architecture and a method of decoding a coded video bitstream using a neural-network architecture, according to the independent claims. Also provided are a tangible computer-readable storage medium having stored thereon computer-executable instructions for executing these methods, and an apparatus comprising a processor and configured to perform these methods. Advantageous features are provided in dependent claims.

[0012] More specifically, example embodiments described herein relate to applying video coding techniques using deep learning and content-adaptive overfitting techniques. In anembodiment, using a two-pass encoding of the same NN model under evaluation, a selection is made between a baseline model and a CAO model based on a loss metric dependent on bit rate overhead to transmit updated CAO weights, and improvements in video quality (e.g., minimizing the mean square error (MSE) of the video content to be overfitted). The updated NN weights are transmitted using a k-th order Golomb encoder. Example high-level syntax to support CAO for a neural network-based loop filter is described as well.CONTENT- ADAPTIVE OVERFITTING (CAO) IN VIDEO CODING

[0013] In video coding, Content Adaptive Overfitting (CAO) is used to enhance the coding efficiency of neural networks-based architectures, such as a neural network loop filter (NNLF). CAO utilizes a neural network model which is initially trained on general data, but which is then further refined through training on the specific type of data it will process, for example, in video coding, a specific group of pictures of the input video to be encoded.

[0014] FIG. 1 illustrates an example pipeline of a video architecture with a neural- networks (NN) Loop filter (NNLF), where the NNLF may employ CAO. Part of the pipeline may also include a deblocking filter, a sample adaptive offset (SAO) filter, and an adaptive loop filter (ALF). An example of NNLF is described in Refs. [1-2].

[0015] Note that while embodiments presented herein provide example application of the proposed CAO techniques on NNLFs, CAO-based techniques as presented herein may be applied to any NN-based video coding architecture, such as NN-based Intra coding, NN- based Inter-coding, and the like.Earlier CAO implementation related to NNLF

[0016] An early version of CAO was first introduced for a NNLF in Ref. [3] and Ref. [7]. As described in Ref. [3], the content-adaptive low-operation point (LOP) NNLF filter is obtained by overfitting multiplier parameters that are applied after a convolutional block, like: cr((VI / * x + b) ■ m), (1) where W denotes a kernel of a base model, * denotes the convolution operator, x is the input, b is the bias, m is the multiplier, and <r is the activation function. The base model represents the original model, before CAO modules are added. The values of W and b are generatedduring the training of the base model. Thus, during CAO training only m is updated. Examples of activation functions include PReLU, leakyReLu, and the like.

[0017] These multiplier layers are added after each convolutional (CONV) block in each of the luma and chroma backbones. The generated weight-updates are coded with the MPEG- NNR standard (Refs. [5-6]) and are signaled to a decoder with a NN filter update (NNFU) message using adaptation-parameter signaling (APS). It is noted that such design introduces new NN blocks, thus a new NN architecture.

[0018] The content adaptation is done per random-access (RA) segment independently, and the weight-update is signaled within the first B-slice of such segment. To lower the overhead of the weight-update that is coded with MPEG-NNR, only a subset of multiplier layers is overfitted. The multiplier layers to overfit are selected by an Ll-norm. Specifically, first, all multiplier layers are overfitted for 10 iterations; then, assuming that K is the number of trainable multiplier layers, the top 0.8 • K and 0.2 • K multiplier layers with the highest Ll-norm in luma and chroma backbones, respectively, will be overfitted. K is determined by bitrate overhead percentage, using the following steps:1. In an offline stage, an optimal class-wise ratio rcis determined, which indicates an acceptable bitrate increase.2. Let bmbe an empirically determined average bitrate of an NNR (neural network data and related metadata) bitstream representing one multiplier layer overfitted on one segment. Letbe the bitrate of the i-th anchor-coded video sequence. The number of multiplier layers, K, to overfit in each segment is given byK = rc* 7i / bm. (2)Thus, the number K of trainable multiplier layers is dependent on the specific video sequence.

[0019] The floating-point overfitted models, obtained after decoding the NNR weight updates, are quantized to int!6 (16-bit integers) using the same quantizers as the base models (for convolutional kernels and bias terms). Additionally, the multiplier layers use a fixed quantizer of 13.

[0020] Table 1 and Table 2 show in Italics the proposed modification to allow NNLF updates to be signaled using the adaptation parameter set, by introducing a new NN_UPDATE_APS flag.Table 1 Example of APS signaling syntax for NN updates (Ref.[3])aps_params_type specifies the type of APS parameters carried in the APS as specified in Table 2. The value of aps_params_type shall be in the range of 0 to 2, inclusive, in bitstreams conforming to this version of this Specification. Other values of aps_params_type are reserved for future use by ITU-T I ISO / IEC. Decoders conforming to this version of this Specification shall ignore APS NAL units with reserved values of aps_params_type.Table 2. APS parameters type codes and types of APS parameters (Ref. [3])

[0021] All APS NAL units with a particular value of aps_params_type, regardless of the nuh_layer_id values and whether they are prefix or suffix APS NAL units, share the same value space for aps_adaptation_parameter_set_id. APS NAL units with different values of aps_params_type use separate values spaces for aps_adaptation_parameter_set_id.aps_adaptation_parameter_set_id provides an identifier for the APS for reference by other syntax elements.When aps_params_type is equal to ALF_APS or SCALING_APS, the value of aps_adaptation_parameter_set_id shall be in the range of 0 to 7, inclusive.When aps_params_type is equal to LMCS_APS, the value of aps_adaptation_parameter_set_id shall be in the range of 0 to 3, inclusive.When aps_params_type is equal to NN_UPDATE_APS, the value of aps_adaptation_parameter_set_id shall be in the range of 0 to 3, inclusive.

[0022] Table 3 provides an example of the proposed in Ref. [3] neural network filter update syntax.Table 3. Example CAO parameters signaling (Ref.[3])nnfu_bitstream_size_minusl plus 1 specifies the size in bytes of the nnfu_payload. nnfu_payload_byte[ i ] contains the i-th byte of a bitstream conforming to ISO / IEC 15938- 17. The byte sequence nnfu_payload_byte[ i ] for all present values of i shall be a complete bitstream that conforms to ISO / IEC 15938-17.

[0023] During inference, in an encoder, each RA segment is overfitted independently and once the overfitted model is ready to be deployed, the encoding of the RA segment is also done independently from other segments. Adjacent RA segments have overlapping I-frames and these are filtered using the original LOP2 filter.

[0024] After an RA segment of a sequence-QP pair has been encoded using the respective overfitted LOP2 filter, a decision is performed to determine whether this segment or the one encoded with the anchor configuration (i.e., with the base model) is better in terms of delta PSNR. Therefore, a final merged bitstream may include some RA segments filtered with the original (base) LOP2 filter and some RA segments filtered with the overfitted LOP2 filter.

[0025] In a decoder, when the decoder parses an NNFU APS, the reconstruction of the overfitted LOP2 filter is triggered. The process relies on the original (base) LOP2 filter and the decoded NNR bitstream carried within the NNFU APS.Proposed Improvements to the CAO implementation

[0026] The next sections describe example embodiments according to the invention, an example encoding pipeline, which includes training strategy, high-level syntax (HLS) design, a decoding pipeline for CAO, and some differences between the earlier CAO implementation and the proposed enhancements.

[0027] In an embodiment, initially, the neural network filter model is trained using broad (general purpose) datasets. For example, to train NNLF models, JVET uses DIV2K images, and the BVI-DVC and TVD video datasets. This results in a generalized model referred herein as the “base” or “baseline” model. Following this, only specific layers / parameters of the neural network model are overfitted (retrained) to optimize the coding efficiency for a group of pictures (GOP) of a specific video content to be encoded. The GOP encompasses all the frames starting from an Intra Random Access Point (IRAP), such as an Instantaneous Decode Refresh (IDR) I-frame or a Clean Random Access (CRA) I-frame, or a Broken Link Access (BLA) picture, until the start of the next IRAP point or the end of the coded video sequence. In the case of low latency applications that use Gradual Decoder Refresh (GDR) for Intra refresh, the same CAO parameters must be used for all the frames encompassing a GDR period. This specialization involves locking (or freezing) all layers except the ones being fine-tuned (overfitted), using frames from the specified content. It is noted, that unlike Ref. [3], the proposed approach does not introduce any new NN blocks with their own parameters. It uses the same NN blocks as the base NN model.Training procedure for overfitting

[0028] In experiments, given the NNLF LOP (Refs. [2,4]), in an embodiment, it was decided to overfit convolutional layers from one or more backbone layers of the LOP NNLF using the approach described in the next section. Any layer with trainable parameters can be overfitted, for example: pReLU, the dense layers, and the like.

[0029] It is not essential to adjust every parameter within a layer, i.e., only a few trainable parameters of a layer can be targeted for fine-tuning. In this case a few parameters of a particular layer would be fine-tuned and the rest of them would be locked / frozen to have the same values as the base model. It is also to be noted that any number of trainable parameters of a model can be chosen to be fine-tuned.

[0030] In a generalized overfitting algorithm, which works for any neural network architecture, the overfitting is done by jointly optimizing distortion and bit rate using a QP- dependent LaGrange multiplier, without restricting the trainable parameters to specific layers. For example, the CAO training loss function may be adjusted to include both distortion and rate using the equation below:CAO loss = CAOMSE+ * 6r, (3) where X is the slope of the rate distortion curve and 5r is the signaling overhead to code the adjusted NN parameters (e.g., the nnlf_cao_data() in the NN_APS parameters of Table 4 below). In the above equation, X is computed using the slope derivation process explained in the next section and illustrated in FIG. 2C, and rate overhead is computed using the sum of all the Exponential Golomb (EGK) coding bits of the modified NN layers, as explained in the next section. Given a loss function, the CAO model is selected when the CAO loss < BaseMSE., where BaseMSE denotes the distortion (e.g., the mean square error) for the base model as computed for the CAO training dataset, e.g., derived from a single GOP of video to be transmitted.

[0031] This loss function considers the bitrate overhead, penalizing the model for updating more layers during back propagation. This approach ensures that weight updates occur in a way that improves the MSE beyond the overhead needed to transmit the updated weights. In other words, only the layers that contribute to a better MSE with reasonable rate overhead are overfitted during backpropagation.Encoder Pipeline

[0032] FIG. 2A to FIG. 2D depict an example process flow for an encoding CAO process (200), according to an embodiment. As depicted in FIG. 2A, a two-pass encoding process is being used. First (205), the content for which the target NN architecture (e.g., a NNLF) needs to be overfitted is encoded using the base model. The input features used for training the NN model are saved during the first pass encoding. This is used as input for CAO model training (220). The details of CAO model training (220) are depicted in FIG. 2B, with an alternative process depicted in FIG. 4. In step (210), the inputs to the NNLF model (such as, Recon, Pred, Boundary strength, QP Base, and the like (see Ref. [1])) are selected and saved into the file. As an example, in step (215), two frames are selected from the selected input in (210) and the data is prepared for CAO training. For example, in a GOP with 64 frames which would have 6 temporal IDs, only 12 frames are picked. These frames are divided into CTUsof 128x128 pixels. Each CTU is then padded with 8 neighboring pixels on all sides, resulting in a final size of 144x144 pixels. All input data is segmented into CTUs and grouped with their corresponding CTUs, then stored in a single file. This file is then used for training.

[0033] During CAO training, as depicted in FIG. 2B, first (225), the CAO model’s weights are initialized with that of the base model. It is to be noted that the architecture of the base model and the CAO model are always identical. The BaseMSE is computed for the CAO training dataset before the beginning of the training (215). In step (230), all the remaining neural network parameters, except the ones that are chosen for fine tuning, are frozen before the start of training. Next (235), the model is trained for a user-defined number of iterations to fine tune the chosen set of parameters and optimize the Rate-Distortion (RD) loss of the target GOP. An example for such training is depicted in FIG. 4, to be explained in more detail later. After CAO training (220), the updated NN parameters need to be transmitted to the decoder. In an embodiment, k-th order exponential Golomb (EGK) encoding is used to signal the difference between the parameters of the base model and the CAO model. The syntax of the signaling of the parameters is explained in a later section.|00034| Before continuing, it is worth discussing an alternative method for CAO training, as depicted in FIG. 4. This example process provides a generalization of the steps in (220) by offering two alternatives for CAO training and parameter selection: a) as discussed earlier in FIG. 2B, using in-place modification of convolutional weights (path 407), or b) inserting layer post-multipliers (path 409), which represents a generalization of the previous method in Ref. [3]).

[0035] As depicted in FIG 4, this generalized approach can overfit any neural network architecture with trainable parameters. In this algorithm, the parameters are either fine-tuned in-place or a post-multiplier layer is added after the overfitted layer. The post-multiplier might be chosen whenever the in-place parameter overfitting method doesn’t yield good tradeoff between MSE and bitrate overhead. When the post-multiplier is used, a scaling vector (M) is added after the convolutional layer. Also, the bias layer (5) of this convolutional layer may be overfilled to become B' . In this scenario, one multiplication factor will be used for each filter (output channel) in a given layer. For example, if a regular convolutional layer of kernel size 3x3 has 80 filters with 16 input channels, there will be only 80 multipliers for this layer. Therefore, only 160 parameters need to be signaled with 80 scaling factors and 80 modified bias values. In this example, the post- multiplier method provides significant bitrate savings compared to in-place overfitting which has 3x3x80x16parameters to be coded, i.e., 11520 convolutional weight parameters and 80 bias parameters. Details of this generalized algorithm are described next.

[0036] First (405), the inputs to the layer selection algorithm are passed. The inputs include:1. an index z, which is initialized to 0, to keep track of the number of layers to be overfilled;2. Lossbest is initialized to BaseMSE, where BaseMSE is computed for the CAO training dataset before the beginning of the training (see step (215));3. A list of input layers that need to be overfitted jointly. For example, Input = [[layerl], [layer2, layer3]] signifies that layers 2 and 3 will be overfitted together if selected. The decision on how to overfit layers 2 and 3, whether using post-multipliers or in-place training, is also done jointly. Not all input list layers are overfilled. Each list of layers goes through the decision loop to determine if it will be overfitted or not.

[0037] As depicted in FIG. 4, there are two paths: one for in-place training (407) and one for using a post-multiplier (409). Starting with in-place training (407), layers of Inputfi] are unfrozen, that is, their parameters can be readjusted, and their weights are trained in-place for X epochs (e.g., X =2). As used herein the term “unfreezing a layer” denotes that weights and biases in a layer may be readjusted. Similarly, “freezing a layer” denotes that parameters of a layer can’t be further adjusted. The loss (Loss^ ) is calculated using MSE and bitrate overhead as per equation (3). Lossh is the loss generated when the layers are unfrozen and trained for X epochs. MSELi and Srh are MSE and bitrate overhead respectively obtained during the training. Simultaneously, in another path (409), without unfreezing the layers, multipliers are inserted after each layer in the list and trained for X epochs. In this case, LossMi is the loss obtained while training. MSEMiare MSE and bitrate overhead respectively obtained when post-multipliers are inserted in the layers CAO during training. Next (410), the loss from each path is compared to Lossbest (originally set to BaseMSE). If both losses are worse than Lossbest , then the layer is frozen, and weights of that layer are reset to that of the base model (415). Any post-multipliers are also removed. If one of the approaches shows better loss than Lossbest, then LossLi is compared with LossMi (420). In-place update is chosen if Loss < LossMi, and the Lossbest is updated to Lossh; otherwise, the latter approach of inserting the post-multipliers is chosen and the Lossbest is updated with LossMi. This process continues for the entire input list. After deciding on overfilling approaches for all layers, in step 430, one trains the updated model for a user-defined number of epochs. Theinserted post-multipliers (if any) are fused back to the respective layers to maintain the same architecture as that of the base model. The fusion and signaling of the multipliers are explained in the CAO signaling section and the decoder pipeline section.

[0038] In some embodiments to simplify training, one may select to apply only one of the two training approaches (that is 407 or 409) in a particular layer to be overfitted.

[0039] Returning to FIG. 2A, after the CAO model training (220), in step 240, as illustrated in FIG. 2C, one selects either the base model or the CAO model for the second pass encoding based on the loss function of equation (3). To illustrate this, it helps to understand the second pass encoding (250), depicted in FIG. 2D. In the second pass encoding (250), 8(param), i.e., the difference between the fine-tuned or overfitted layers’ parameters and the respective base model parameters is encoded using k-th order exponential Golomb (EGK) coding (246). Each overfitted layer has an independent order for this coding. The 8 (param) is computed using one of the following seven approaches. The first four approaches are used by layers with no post-multiplier parameters and the last three approaches are used by the layers using a post-multiplier. More specifically:1. Under this scheme, the absolute difference between parameters of the overfitted layers and the respective base model parameters is computed and signaled.2. Under this scheme, a global mean is computed by taking the mean of all the parameters of all the overfitted layers. The global mean will be signaled in the beginning, before sending any overfitted parameters. When this approach is selected (i.e., it consumes the least number of bits), then the difference between the overfitted parameters and the global mean is computed and signaled as 8 (param). During encoding of the CAO parameters, it is checked across all the CAO layers whether any of the layers uses this approach. If none of the layers use this approach, then a parameter cao_global_mean_used_flag is set to 0.3. Under this scheme, a local mean is computed by taking the mean of all the parameters of the present overfitted layer for which signaling needs to be done. The local mean will be signaled if this approach is selected (i.e., it consumes least number of bits of the four approaches). After signaling the local mean, the differences between the overfitted parameters and the local mean is computed and signaled as 8 (param). While calculating the bitrate overhead (8r) for this approach, unlike other three approaches, the number of bits required to signal the local mean is added to equation (5) as [log2(local_mean)J .4. Under this scheme, the difference between parameters of the present overfitted layers and the closest previous overfitted layer with the same dimension is computed and signaledas 8(param). For example, consider the layers [layer(l), layer(2), layer(N)] as the CAO layers, with layer(N) utilizing this approach. The dimension of layer(N-l) is determined and verified to see if it matches that of layer(N). If there is no match, the dimension is then compared with layer(N-2), and so on. The layer whose dimensionality matches in terms of kernel size, number of input channels, and number of output channels will be selected to determine 6 (param).5. This scheme is similar to scheme 1. The absolute difference between multipliers and their initial value i.e.,= c, e.g., c = 1, will be computed and signaled, where Afij denotes the j -th multiplier parameter of the i-th layer. The difference (Mij - c) will be determined and coded into the bitstream in this approach.6. Under this scheme, a local mean is computed for all the multiplier parameters for the given layer with a post-multiplier. Similarly to approach 3, the local mean of the multipliers will be signaled first when this approach is selected. The final ATj values are coded and transmitted as their difference with respect to the local mean of multiplier values.7. This is similar to approach 4. The closest previous layer with a post-multiplier and with the same number of multiplier parameters as the current layer is used to compute the 8 (param).

[0040] In an embodiment, one of the first four approaches is selected for layers overfitted without post multipliers, otherwise one of the last three approaches is selected.

[0041] When adding a CAO post-multiplier, parameters may include the scaling factor (M) and the modified bias of the convolutional layer. Assume the layer is overfitted using a post-multiplier, which is added after a convolutional layer which has input x with output y. The equation to determine the output of the layer before modifying the bias and adding the multiplier layer is: y = W * x + B.After adding the post-multiplier, given an updated bias B' , the generated output y' is given by: y' = (14 / * x + B') * M,This can be modified as follows: y' = (14 / * M) * x + (B' * M).Given the equation above, the new bias will be (B' * M) and the convolutional weights will be (IV * M). In the equations above W, M, B, and B' are vectors. All the wy 6 Wj of a filter will be multiplied by Mj where j is the index of the filter of the convolutional layer. For the convolutional weights, signaling the 5(param) for each ivy will increase the signaling overhead significantly. Since all the vy G Wj have the same scaling factor, M it is signaled using the post-multiplier-specific approaches 5-7 discussed earlier, which would result in signaling of only j parameters instead of i*j parameters for the convolutional weights.

[0042] In an embodiment, for the in-place training (407), one may select to adjust the biases only without adjusting the weights or select to adjust the weights only without adjusting the biases. Thus, biases and weights could be adjusted independently. When postmultipliers are used for a convolutional layer with weights (W) and biases (B), the biases can be optionally modified in-place while having the same multiplication factor M) as that of weights. In such a scenario, the updated bias values are signaled before signaling the postmultipliers using one of the first four approaches. It is recommended to code the postmultiplier parameters only once to minimize the bitrate overhead. The encoder handles this by signaling the post-multiplier immediately after coding the updates of the bias layer using the same layer ID as that of the bias layer. In this case the same layer ID is used twice: first, to signal the in-place updates for the bias values using schemes 1-4, and second, the postmultiplier values using schemes 5-7.

[0043] The EGK order 'k' is chosen to minimize the total bits used for EGK encoding, for example, as determined by the following equation:(4)

[0044] Equation (4) represents the overhead bits as will be needed based on the proposed syntax of Table 7. Hence, if a different encoding scheme is selected, equation (4) will need to be adjusted to provide an accurate count of the overhead bits. For each overfitted layer, kmm is calculated separately. The 8 (param) for a given layer is then encoded using exponential Golomb coding with the order equal to kmin. sign(8 (param)) denotes the sign bit for 8 (param) in the above equation, which would be 0 if 8 (param) is equal to 0, else it would be 1.

[0045] During the second pass encoding (240), the CAO model would be chosen over base model if the sum of its MSE validation loss (from training) and the bitrate overhead (Sr) are smaller than the MSE of the base model. This process is illustrated in FIG. 2C. Toaccount for bitrate overhead, the slope (X) is used, which is calculated using the MSE calculated from PSNR and bitrate differences of the content for the current point and the nearest point in the rate-distortion curve. For example, if a BD-Rate curve is generated for QP values {22, 27, 32, 37} during the first pass encoding, X for QP =37 CAO model is generated using the MSE for normalized samples whose range is 0 to 1 (which, for example, may be calculated from computed peak signal to noise ratio (PSNR) as 1 MSE = — P -S —NR •>10 io and the bitrate difference between using QP =37 and QP = 32. BitRateOverhead (8r) is the total bits that will be used to encode the exponential Golomb code symbols of the modified parameters of the overfitted layers as per Table 6 and Table 7.

[0046] In an embodiment, the bitrate overhead includes the following parameters:- Number of CAO layers.- The layer IDs of all the CAO layers- Exponential Golomb coding order ‘k’- Both the CAO model and the base model, whose weights are originally in floating-point format, are quantized to integer weights (such as 16-bit integer or 8-bit integer) for fixed point inference. This conversion uses a selected quantization factor (Q factor), which scales weights by 2 ; (i.e., weight x 2Qi). Here, Qi represents the chosen Q factor for the i-th layer of the model. The Q factor for each layer may vary, depending on the weights of that layer, to prevent overflow while preserving accuracy during inference. Bitrate overhead includes the overhead to code the difference between the quantization parameter of CAO and base layer ( Ql= QCAO - QBase) including sign bits.- EGK bits for the modified CAO parameter, including sign bits for all the CAO layers.

[0047] An example of calculating the bitrate overhead is provided in equation (5):

[0048] In equation (5), ki denotes the kmm used for the i-th layer, cao_nn_layer_id[i] holds the ID of the i-th layer among the CAO NN layers in the model, delta_abs_q_factor[i] denotes the absolute difference in the quantization parameter between base model and CAO model of NN layer ID cao_nn_layer_id[i], delta_abs_q_factor[i] and .s / y / ?(delta_c|_l'actor|i ])specify the absolute difference value for the variable deltaQFactorf i ] and its sign in the quantization parameter between the base model and the CAO model in the cao_nn_layer_id[ i ]-th NN layer.

[0049] CAOMSE and B seMSE are the MSE validation losses of the overfitted model and the base model respectively achieved during training (overfitting) on the CAO training dataset (215). Bitrate overhead is accounted with k * 6r for the CAO model. CAO is chosen over the base model if CAOMSE along with accounted bitrate overhead is better than BasewsE. The model selected here would be used in the second pass encoding. In the case of CRA or BLA based GOP structures, the next GOP's IRAP will serve as a reference picture for several leading B-frames of the current GOP. If both the next and current GOP choose the base model, then the second pass encoding is skipped for the current GOP.

[0050] The decision of using the CAO model or the base model can be done after the second- pass encoding as well. In this case, the CAO parameters will be signaled at the first frame after the GOP’s IDR or CRA I- frame. When the signaling is done at every IDR or CRA I-frame, the decision of switching CAO on / off needs to be done before the second pass encoding. This is because for the random-access encoding configuration, when the adjacent GOPs are encoded independently, the second GOP’s first I-frame and the first GOP’s last I frame will be actually the same frame. This will enforce the usage of the same model between two GOPs for this particular I- frame. But when the base model is used for the IDR or CRA I-frames, this problem doesn’t exist. If the bitstream from the 1stpass, which used base model, can be chosen for one of the GOPs, and the bitstream from the 2ndpass, which used the CAO model, can be chosen for another GOP. In this approach, similar to the prior art, the CAO model will not be used for the IDR or CRA I- frames and instead the base model will be used. The CAO model will be used for all other frames during the second pass encoding unconditionally. After the second pass encoding, based on the PSNR and bitrate of the first pass and second pass encoding, one of them will be selected which will be used by the decoder. Here the BD-Rate for the GOP across QPs is calculated for the first pass vs second pass encoding, and the one that has better BD-Rate is chosen and will be used by the decoder.CAO Signaling

[0051] The encoder uses the overfitted CAO model to encode frames within a selected GOP. The difference between the base model and the CAO model is signaled in the bitstream at every IDR or CRA I-frame. The same parameters are used by all the frames that belong tothe same GOP. The decoder then reads this delta and applies it onto the base model. In an embodiment, the CAO updated parameters are signaled as follows:

[0052] As in Table 1 and Table 2, Table 4 and Table 5 provide examples of APS signaling using the new NNLF_CAO_APS parameter to denote updated CAO signaling for a NNLF.Table 4. Example APS signaling syntax for CAO updatesaps_params_type specifies the type of APS parameters carried in the APS as specified in Table 5. The value of aps_params_type shall be in the range of 0 to 2, inclusive, in bitstreams conforming to this version of this Specification. Other values of aps_params_type are reserved for future use by ITU T I ISO / IEC. Decoders conforming to this version of this Specification shall ignore APS NAL units with reserved values of aps_params_type.Table 5. Example APS parameters type codes and types of APS parameters

[0053] All APS NAL units with a particular value of aps_params_type, regardless of the num_layer_id values and whether they are prefix or suffix APS NAL units, share the same value space for aps_adaptation_parameter_set_id. APS NAL units with different values of aps_params_type use separate values spaces for aps_adaptation_parameter_set_id. aps_adaptation_parameter_set_id provides an identifier for the APS for reference by other syntax elements.

[0054] When aps_params_type is equal to ALF_APS or SCALING_APS, the value of aps_adaptation_parameter_set_id shall be in the range of 0 to 7, inclusive.When aps_params_type is equal to LMCS_APS, the value of aps_adaptation_parameter_set_id shall be in the range of 0 to 3, inclusive.When aps_params_type is equal to NNLF_CAO_APS, the value of aps_adaptation_parameter_set_id shall be in the range of 0 to 3, inclusive.

[0055] Unlike Table 3, which proposes signaling using the MPEG-NNR specification, in an embodiment, it is proposed to code the difference of CAO parameters and base model parameters directly using EGK (k-th order exponential Golomb) coding. The advantage of this proposal is that it does not require the encoder or the decoder to implement the MPEG- NNR standard. Tn addition, the overhead of CAO is relatively low. The coding efficiency benefit of using the more complex NNR standard (for example, using deep CABAC features and the like) is likely not very high. An example of the proposed implementation is shown in Table 6.Table 6. Example CAO signaling syntaxnnlf_ _cao_singal_flag equal to 1 specifies that a CAO (content adaptive overfitting) parameter set is signalled. nnlf_cao_singal_flag equal to 0 specifies that a CAO parameter set is not signalled. num_cao_nn_layers_minusl plus 1 specifies the number of NN layers in the model whose parameters are different than the base model. cao_global_mean_used_flag equal to 1 specifies that at least one of the CAO layers uses global mean across the CAO parameters to code the 6(param). cao_global_mean specifies the mean of all the CAO parameters of the CAO model, i.e. mean taken across parameters of cao_nn_layer_id[0] to cao_nn_layer_id[num_cao_nn_layers_minus 1 ] sign_cao_global_mean specifies the sign of cao_global_mean. If sign_cao_global_mean is equal to 0, cao_global_mean is a positive value otherwise cao_global_mean is a negative value.cao_nn_layer_id[ i ] specifies the index of the i-th NN layer whose parameters differ from the base model. delta_coding_type specifies the method with which the 8 (param) is encoded and signaled. There are the seven approaches that are described in the encoder pipeline section. 0 to 3 will be used for coding parameters with layers with no post-multipliers, and 4 to 6 will determine the coding approach of the post-multiplier parameters. The details of the approaches are as follows:local_mean specifies the mean of the parameters of cao_nn_layer_id[ i ] -th NN layer. This will be signaled only if encoding of the 8 (param) is done with respect to local mean. This will be signaled when delta_coding_type is 2 or 5 for layers without or with post-multipliers respectively. k_golomb_val[ i ] specifies the order value k used in k-th order exponential Golomb code (EGK) for the CAO parameters in the cao_nn_layer_id[ i ]-th NN layer. delta_abs_q_factor[ i ] specifies the absolute difference for the variable deltaQFactor[ i ] in the quantization parameter between base model and CAO model in the cao_nn_layer_id[ i ]- th NN layer.delta_sign_q_factor_flag[ i ] specifies the sign of deltaQFactor[ i ]. If delta_sign_q_factor_flag[ i ] is equal to 0, deltaQFactor[ i ] is a positive value, otherwise, deltaQFactorf i ] is a negative value. delta_cao_param[ i ][ j ] contains the difference in the trainable parameters between base model and CAO model for cao_layer_id| i ]-th NN layer. The sign value is transmitted only when delta_cao_param[ i ][ j ] is nonzero.ExpGolomb( v, k ) specifies a parsing process of decoding v using k-th order Exponential- Golomb (EGK) code. The parsing process is specified in a later section.

[0056] Table 7 provides an example of the parsing process ExpGolomb( v, k ). The input to this process includes bits from the raw byte sequence payload (RBSP) and the order k. The output of this process is syntax element value v.Table 7 k-th Exponential Golomb (EGK) coding syntaxNote: in k-th order exponential Golomb coding, the delta CAO parameter x is encoded in two parts:1. encode [.v / 2kj using order-0 exp-Golomb code. In the above function, it is coded with zeros followed by ‘ 1 ’ .2. encode x mod 2k in binary. In the above function, it is coded with fixed code of length k.

[0057] In an embodiment, parameter “ph_nnlf_aps_id” is used to signal the APS ID for NNLF CAO. An example syntax is provided in Table 8.Table 8. Example of a CAO picture header syntaxph_nnlf_cao_enabled_flag equal to 1 specifies that NNLF CAO is enabled for the current picture. ph_nnlf_cao_enabled_flag equal to 0 specifies that NNLF CAO is disabled for the current picture. When not present, the value of ph_nnlf_cao_enabled_flag is inferred to be equal to 0. ph_nnlf_cao_aps_id specifies the aps_adaptation_parameter_set_id of the NNLF CAO APS that the slices in the current picture refers to.

[0058] An alternative method to code CAO parameters is to use NNR ISO / IEC 15938-17.Table 9 provides such an example based on Table 6.Table 9. Example syntax to signal CAO parameters using EGK or ISO / IEC 15938-17nnlf_ _cao_mode_id identifies the coding method of NNLF CAO parameters as specified in the following Table.nnlf cao bitstream size minusl plus 1 specified the NNLF CAO bitstream size in units of byte. nnlf_ _cao_payload_byte[ i ] specified the ith payload byte.

[0059] In another embodiment, in high level syntax, it is proposed to add a general constraint flag to indicate if NNLF is used in the bitstream, and / or additionally if NNLF CAO is used in the bitstream. Such an example is depicted in Table 10.Table 10. Example high-level syntax for NNLF CAOgci_no_nnlf_constraint_flag equal to 1 specifies that sps_nnlf_enabled_flag for all pictures in OlsInScope shall be equal to 0. gci_no_nnlf_constraint_flag equal to 0 does not impose such a constraint. gci_no_nnlf_cao_constraint_flag equal to 1 specifies that sps_nnlf_cao_enabled_flag for all pictures in OlsInScope shall be equal to 0. gci_no_nnlf_constraint_flag equal to 0 does not impose such a constraint.

[0060] If the signalling is in sequence parameter set (SPS), then an example is provided in Table 11.Table 11. Example SPS signaling for NNLF CAOsps_nnlf_enabled_flag equal to 1 specifies that NNLF is enabled for the coded layer video sequence (CLVS). sps_nnlf_enabled_flag equal to 0 specifies that NNLF is disabled for the CL VS. sps_nnlf_cao_enabled_flag equal to 1 specifies that NNLF CAO (content adaptive optimization) is enabled for the CLVS. sps_nnlf_cao_enabled_flag equal to 0 specifies that NNLF CAO is disabled for the CLVS.Decoder pipeline

[0061] An example of the decoding pipeline is shown in FIG.3. First (305), the CAO delta parameters (6(param)) are extracted from the bitstream, then (310), they are applied to the base model to create the updated CAO model, and finally (315), the NN network runs using this updated model. There are seven approaches with which the CAO delta parameters (8 (param)) are computed and encoded in the bitstream. These seven approaches were explained in the encoder pipeline section and were also described during the definition of delta_coding_type in Table 6. The delta_coding_type which determines the approach used to code CAO parameters for each layer is also signaled in the bitstream. delta_coding_type values 0 to 3 specify applying CAO without any post-multipliers and values 4 to 7 specify the usage of post-multipliers. The decoder operation that needs to be done for each of theseapproaches are described below:1. delta_coding_type=0 specifies that this layer is not using a post-multiplier and 6(param) denotes the absolute difference between the base model and the CAO model parameters. After reading the 8 (param) from the bitstream, it is added to the base layer’s parameters to obtain the CAO layer parameters.2. delta_coding_type=l specifies that this layer is not using a post- multiplier and the8 (param) is the difference between the global mean of all the CAO parameters and the current layer’s CAO parameters. After reading the 8(param) from the bitstream, it is added to the global mean (cao_global_mean) signaled in the bitstream to obtain the CAO layer parameters.3. delta_coding_type=2 specifies that this layer is not using a post- multiplier and the8 (param) is the difference between the current CAO layer’s local mean and the current CAO layer’s parameters. After reading the 8(param) from the bitstream, it is added to the local mean (local_mean) to obtain the CAO layer parameters.4. delta_coding_type=3 specifies that this layer is not using a post-multiplier and the8 (param) is the difference with the first preceding CAO layer having the same dimension i.e, same kernel size, number of input and output channels. After reading 8 (param) from the bitstream, the first preceding CAO layer with exact same dimension is searched for and the 6 (param) is added to that CAO layer’s parameters.5. delta_coding_type=4 specifies that this is a layer with a post-multiplier and the 8 (param)is the actual multiplier parameter and c is a constant (e.g., c = 1).is obtained using 8(param) + c.6. delta_coding_type=5 specifies that this is a layer with a post-multiplier and the 8 (param) is the difference between the current multiplier parameter and the mean of the current layer’s multiplier parameter.is obtained by 8 (param) + local_mean.7. delta_coding_type=6 specifies that this is a layer with a post-multiplier and the 6 (param) is the difference with the first preceding layer with a CAO post-multiplier having the same number of parameters. After reading 8 (param) from the bitstream AZij is obtained by searching for the first preceding multiplier layer with the same number of parameters and adding 8(param)to that set’s multiplier parameters.

[0062] When the 8(param) is a post-multiplier which is specified by delta_coding_type values greater than 3, the multipliers (M) need to be fused back to the actual layer to maintain the same architecture as that of the base model. The base model will have the convolutionallayer which takes x as input and provides y as output. As explained earlier, given M, the modified output is: y' = (W * M) * x + (B' * M), where (W * M) denotes the updated weights and (B' * M) denotes the updated biases. When the post-multipliers are used for a convolutional layer, they are signaled in the bitstream with the bias layer’s layer ID to reduce the need of coding post-multipliers (M) twice. Therefore, the cao_nn_layer_id[ i ] value for the post-multiplier is same as that of the bias updates. The bias updates B'are coded first using syntax and semantics of delta_coding_type in 0,3 J. The post-multiplier values are then coded using syntax and semantics of delta_coding_type in [4,6].

[0063] The encoder inserts the CAO parameters into the bitstream at IRAP frames. Upon decoding, the system reads these parameters and generates a new model by combining the base model with the extracted CAO parameters. An IRAP frame from the subsequent GOP may appear between frames of the current GOP. To manage such situations, two CAO model copies are maintained simultaneously. The CAO model is linked to the GOP sequence, and all frames within that GOP will utilize their respective CAO model during inference. When an IRAP frame is encountered, a new CAO model is created using the base model and CAO parameters from the bitstream. If two CAO models are already in place, the oldest model, i.e., the model linked to the oldest GOP whose frames are already decoded, is replaced with the newly created one. When CAO is turned off for the GOP, " nnlf_cao_signal_flag " will be set to zero, making the CAO model identical to the base model.

[0064] In the use case when the decision between considering CAO ON / OFF is done after the second pass encoding, the encoder inserts the CAO parameters into the hitstream to the next frame after the IRAP frames, while the IRAP frames will be using the base model during decoding. During this case having multiple copies of CAO model is not necessary.

[0065] Compared to Ref. [3], the proposed embodiments differ in a number of ways, including, but not limited, to the following:1. Ref. [3] uses NNR compression and signals the compressed parameters using fixed length coding. Embodiments of this invention apply exponential Golomb coding to signal the parameters, thus eliminating any dependency on NNR compression during both encoding and decoding.2. Ref. [3] adds a scalar multiplier layer m after the convolutional layer. This would have the same multiplier for convolutional weights and bias. Also, this would create extra layers and increase the number of multiplications (kMac / pixel) or would require fusing the multipliers back into the convolutional weights and bias. For example, for a kernel 3x3, all the 3x3 parameters would be multiplied by the same factor, whereas embodiments of this invention allow to overfit each parameter providing more flexibility in overfitting along with having the post-multipliers. When a post- multiplier is used it will be fused back to the convolutional layer maintaining the same architecture. Furthermore, convolutional weights and biases can be overfitted independently of each other.3. Ref. [3] uses the base model’s parameters and not the CAO model’s parameters for the I- frames. In embodiments of this invention, the CAO model’s parameters may be used for I-frames as well, which would result in better reference I-frames. In embodiments of this invention, if the decision of turning CAO on / off needs to be done after the second pass encoding, then the base model’s parameters are used for the I- frames, which is the same as in Ref. [3].4. Ref. [3] encodes the first pass with a base model and a second pass with the CAO model. A selection between the two models is based on PSNR values alone, that is, the CAO model is chosen if the PSNR of the second pass is better than the PSNR of the first pass by 0.02dB. In embodiments of this invention, CAO selection is based either during the CAO training, according to a loss factor that accounts for both quality improvement and bit-rate overhead, or after the second-pass encoding based on the BD-Rate of the first pass vs the BD-Rate of the second pass, without relying on a PSNR difference.5. In Ref. [3], the implementation has adaptive layer selection for overfitting during training. All the layers are trained for 10 epochs and then the layer whose weights have been changed the most are chosen for overfitting. Furthermore, the number of layers chosen to overfit is based on the bitrate of the bitstream during the first-pass encoding. Embodiments of this invention allow for improved flexibility in adaptively selecting layers for CAO.

[0066] For LOP, preliminary experimental results have shown at least 0.73% BD-rate improvement in luma PSNR for the proposed CAO compared to the baseline model.ReferencesEach one of the references listed herein is incorporated by reference in its entirety. The terms JVT, JCT-VC, and JVET refer to Joint Video Experts Teams of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29.1 1 ] T. Shao, et al. , “Optimization Techniques for Loop Filtering Using Neural Networks / 1Indian Provisional Patent Application, Ser. No. 202411050190, filed on 1 July 2024, (D23131).[2] T. Shao, et al. , “EEl-related: Further complexity reduction on the joint EE1-0” (LOP.2) unified filter,’" JVET-A F0071 (v3), JVET 32ndmeeting, Hannover, Germany, 13-20 Oct. 2023.[3] M. Santamaria, et al., “EE1-1.4: Content-adaptive loop-filter”, JVET-AH0096 (v2), JVET 34thmeeting, Rennes, France, 17-24 April 2024.[4] D. Rusanovskyy, et al, “AHG11 : Status of the joint EE1-0 (LOP.2) unified filter training,” JVET-AF0043. JVET 32ndmeeting, Hannover, Germany, 13-20 Oct. 2023.[5] ISO / IEC 15938-17 (2020(E)), “Information technology - Multimedia content description interface - Part 17: Compression of neural networks for multimedia content description and analysis, EDIS, 2021.[6] “Test Model of Incremental Compression of Neural Networks for Multimedia Content Description and Analysis (INCTM),” N0179, MPEG Jan. 2022 meeting, output document, WG 4, February 2022.[7] R. Yang, et al., “AHG11: Content- adaptive neural network loop-filter”, JVET-AE0093, JVET 31-st meeting, Geneva, Switzerland, 11-19 July 2023.EXAMPLE COMPUTER SYSTEM IMPLEMENTATION

[0067] Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or apparatus that includes one or more of such systems, devices or components. The computer and / or IC may perform, control, or execute instructions relating to applying content- adaptive overfitting techniques in video coding, suchas those described herein. The computer and / or IC may compute any of a variety of parameters or values that relate to applying content-adaptive overfitting techniques in video coding as described herein. The image and video embodiments may be implemented in hardware, software, firmware and various combinations thereof.

[0068] Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors in a display, an encoder, a set top box, a transcoder, or the like may implement methods related to applying content-adaptive overfitting techniques in video coding as described above by executing software instructions in a program memory accessible to the processors. Embodiments of the invention may also be provided in the form of a program product. The program product may comprise any non- transitory and tangible medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of non-transitory and tangible forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0069] Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a "means") should be interpreted as including as equivalents of that component any component which performs the function of the described component (e.g., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated example embodiments of the invention.

[0070] Various aspects of the present disclosure may be appreciated from the following Enumerated Example Embodiments (EEEs):EEE1. A method of encoding video sequences using a neural-network architecture, the method comprising: accessing a base neural network (NN) model of an architecture to be used in video coding; accessing a sequence of video pictures to be encoded;identifying (210) a list of NN parameters in the base NN model to be adjusted by content- adaptive overfitting (CAO) to form a CAO NN model; generating a CAO training data set (215) based on the sequence of video pictures; applying the CAO training set to the base NN model to generate a base quality metric; applying the CAO training data set to the CAO NN model to generate a CAO NN model with adjusted NN parameters (220) and a cost metric representing a cost of using the CAO NN model over the base NN model; selecting the CAO NN model (240) if the cost metric is smaller than the base quality metric, else selecting the base NN model; generating an output bitstream (250) by applying the selected NN model to the sequence of video pictures; and if the selected NN model is the CAO NN model, then encoding the adjusted NN parameters of the CAO NN model and transmitting them together with the output bitstream.EEE2. The method of EEE1, wherein the CAO training set is generated using a subset of pictures in a group of pictures (GOP) of the sequence of video pictures.EEE3. The method of EEE1 or EEE2, wherein generating a CAO NN model with adjusted NN parameters further comprises: initializing the CAO NN model parameters to be adjusted by values of corresponding parameters in the base NN model; and applying the CAO training data set to the CAO NN for one or more training iterations to generate the CAO NN model.EEE4. The method of any preceding EEE, wherein the cost metric represents coding gain or loss adjusted by bit rate overhead when using the CAO NN model over the base NN model.EEE5. The method of EEE4, wherein the cost metric comprises computing loss = C OMSE+ A * dr, wherein CAOMSEdenotes a mean-square error when using the CAO NN model, X denotes a slope of a rate distortion curve generated during encoding, and 5r is a signaling overhead to code and transmit the adjusted NN parameters of the CAO NN model.EEE6. The method of any preceding EEE, wherein the adjusted NN parameters of the CAO NN model are encoded using k-th order Exponential Golomb (EGK) coding.EEE7. The method of EEE6, wherein an EGK order k is selected to minimize bits used for EGK encoding.EEE8. The method of EEE7, wherein computing an EGK order kmin comprises: sign (8(param)) ,wherein 6 (param) denotes absolute differences between the adjusted NN parameters and corresponding base model parameters, sign(8 (param)) denotes a sign bit for S(param), which is 0 if 8 (param) is equal to 0, and 1 otherwise.EEE9. The method of EEE5, wherein the signaling overhead is computed based on one or more of: a parameter indicating how many layers of the CAO NN model are being adjusted, and for each layer being adjusted: its layer ID, its EGK order k, differential Q factor values with respect to corresponding base NN model, and number of bits required to encode differential values for the adjusted NN parameters in the CAO NN model.EEE10. The method of EEE6, where coding the adjusted NN parameters further comprises: for each adjusted layer of the CAO NN model: generating differences between corresponding parameter values of the CAO NN model and the base NN model; generating an optimal ‘k’ order value for encoding the differences in the adjusted layer; and encoding the differences using an optimal ‘k’ order EGK coding.EEE11. The method of EEE 10, further comprising generating metadata representing the encoded differences, wherein the metadata comprise one or more of: a first flag (nnlf_cao_signal_flag) identifying whether CAO NN parameters are signaled;if CAO NN parameters are signaled, then: a parameter (num_cao_nn_layers_minusl) specifying how many NN layers in the CAO NN model are different from the base NN model, and for each such layer “i”: an index (cao_nn_layer_id[i]) to the NN layer to be adjusted; an EGK k order (k_golomb_val[i]) being used in the NN layer to be adjusted; differences in Q factor parameters between layers of the CAO and base NN models; andEGK coded values of differences in the adjusted NN parameters between adjusted layers in the CAO and base NN models.EEE12. The method of any preceding EEE, wherein generating a CAO NN model with adjusted NN parameters comprises: receiving a list of layers in the base NN model to be adjusted in the CAO NN model; for an input layer in the base NN model to be adjusted: unfreezing NN parameters in the input layer to be adjusted for an in-place CAO NN model; applying the CAO training data set to the in-place CAO NN model to generate a first CAO NN model; generating a first cost metric representing a cost of using the first CAO NN model; inserting a post-multiplier in the input layer to generate a post-multiplier CAO NN model; applying the CAO training data set to the post-multiplier CAO NN model to generate a second CAO NN model; generating a second cost metric representing a cost of using the second CAO NN model; and selecting among the base NN model, the first CAO NN model, and the second CAO NN model based on a best quality metric, the first cost metric, and the second cost metric.EEE13. The method of EEE 12, wherein selecting among the base NN model, the first CAO NN model, and the second CAO NN model comprises:if neither of the first cost metric or the second cost metric is smaller than the best quality metric, then freeze the input layer, reset the NN parameters of the input layer as in the base NN model, and remove the post-multiplier in the input layer; else if the first cost metric is smaller than the second cost metric; removing the post-multiplier in the input layer; selecting the first CAO NN model for the input layer; and updating the best quality metric with the first cost metric; else freezing the input layer; resetting the NN parameters of the input layer as in the base NN model; selecting the second CAO NN model for the input layer; and updating the best quality metric with the second cost metric.EEE14. The method of EEE12 wherein the best quality metric is initialized with the base quality metric.EEE15. The method of EEE 12, wherein if the second CAO NN model is selected for the input layer, after training, weights (W) and biases (B) of the input layer are adjusted as1 " = IV * M, B' = B * M , wherein IV' and B' denote adjusted weights and biases in the input layer and M denotes post-multiplier multiplication parameters.EEE16. The method of EEE15, wherein, biases in the input layer can be further adjusted before applying a post multiplier, and given M and updated biases (Br) values, the B' and the M values are encoded separately for the input layer.EEE17. The method of any preceding EEE, wherein encoding the adjusted NN parameters of the CAO NN model comprises generating CAO metadata for an adjusted layer of the CAO NN model according to a CAO metadata format, wherein the CAO metadata formats comprise: a) absolute difference values between corresponding parameter values of the CAO NN model and the base NN model; b) difference values between parameter values of the CAO NN model and a global mean, wherein the global mean is computed as an average of all adjusted parameters in the CAO NN model;c) difference values between parameter values of the CAO NN model and a local mean, wherein the local mean is computed using an average of adjusted NN parameters in the adjusted layer; d) difference values between parameter values of the adjusted layer of the CAO NN model and a previously adjusted CAO layer with same dimension as the adjusted layer; e) difference values between multiplication factors in a post-multiplier and a constant; f) difference values between multiplication factors in a post-multiplier and a local mean, wherein the local mean is computed using an average of post-multipliers in the adjusted layer; and g) difference values between multiplication factors in a post-multiplier for the adjusted layer and multiplication factors in a post-multiplier in a previously adjusted layer with same dimension as the adjusted layer.EEE18. The method of EEE17, wherein the CAO metadata format is selected to minimize CAO metadata overhead.EEE19. The method of any preceding EEE, wherein high-level syntax in the output bitstream specifies whether the adjusted NN parameters of the CAO NN model are encoded using k-th order Exponential Golomb (EGK) coding or MPEG-NNR (ISO / IEC 15938-17) encoding.EEE20. A method of decoding a coded video bitstream using a neural-network architecture, the method comprising: accessing a base neural network (NN) model of an architecture to be used in video decoding; accessing a coded video bitstream; detecting whether the coded video bitstream was coded using content-adaptive overfitting (CAO); if no CAO was used, generating an output bitstream by decoding the coded video bitstream using the base NN model, else: extracting metadata from the coded video bitstream to generate CAO delta parameters representing differential NN parameters of a CAO NN model over the base NN model; adjusting NN parameters of the base NN model using the CAO delta parameters to generate the CAO NN model; andgenerating the output bitstream by decoding the coded video bitstream using the CAO NN model.EEE21. The method of EEE20, wherein the metadata comprise one or more of: a first flag (nnlf_cao_signal_flag) identifying whether CAO NN parameters are signaled; if CAO NN parameters are signaled, then: parameters for determining whether a global CAO mean is available and its value; a parameter (num_cao_nn_layers_minusl) specifying how many NN layers in the CAO NN model are different from the base NN model, and for each such layer “i”: an index (cao_nn_layer_id[i]) to the NN layer to be adjusted; a delta coding type (delta_coding_type) specifying how adjusted CAO NN parameters are transmitted; an EGK k order (k_golomb_val|i|) being used in the NN layer to be adjusted; differences in quantization factor (Q factor) parameters between layers of the CAO and base NN models; andEGK coded values of differences in the adjusted NN parameters between adjusted layers in the CAO and base NN models.EEE22. The method of EEE21, wherein the delta coding type comprises one of: a) absolute difference values between corresponding parameter values of the CAO NN model and the base NN model; b) difference values between parameter values of the CAO NN model and a global mean, wherein the global mean is computed as an average of all adjusted parameters in the CAO NN model; c) difference values between parameter values of the CAO NN model and a local mean, wherein the local mean is computed using an average of the adjusted NN parameters in the NN layer to be adjusted; d) difference values between parameter values of the NN layer to be adjusted of the CAO NN model and a previously adjusted CAO layer with same dimension as the NN layer to be adjusted; e) difference values between multiplication factors in a post-multiplier and a constant;f) difference values between multiplication factors in a post-multiplier and a local mean, wherein the local mean is computed using an average of post-multipliers in the NN layer to be adjusted; and g) difference values between multiplication factors in a post-multiplier for the NN layer to be adjusted and multiplication factors in a post-multiplier in a previously adjusted layer with same dimension as the NN layer to be adjusted.EEE23. The method of EEE22, wherein if the delta coding type is related to transmitting multiplication factors (Af) for a post-multiplier in a NN layer to be adjusted, then weights (W) and biases (B) of the NN layer to be adjusted are adjusted as:VI / ' = VI / * M, B' = B * M , wherein W' and B' denote adjusted weights and biases.EEE24. The method of EEE23, wherein, if the NN layer to be adjusted includes both adjusted biases (B') and a post multiplier M), then weights and biases of the NN layer to be adjusted are W' = W * M, and B" = B' * M, wherein VI / ' and B" denote the adjusted weights and biases.EEE25. A tangible computer-readable storage medium having stored thereon computerexecutable instructions for executing with one or more processors a method in accordance with any one of the methods recited in EEE1 to EEE24.EEE26. An apparatus comprising a processor and configured to perform any one of the methods recited in EEE1 to EEE24.EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

[0071] Example embodiments that relate to applying content- adaptive overfitting techniques in video coding are thus described. In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and what is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims.Hence, no limitation, element, property, feature, advantage or attribute that is not expresslyrecited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims

CLAIMSWhat is claimed is:

1. A method of encoding video sequences using a neural-network architecture, the method comprising: accessing a base neural network (NN) model of an architecture to be used in video coding; accessing a sequence of video pictures to be encoded; identifying (210) a list of NN parameters in the base NN model to be adjusted by content-adaptive overfitting (CAO) to form a CAO NN model; generating a CAO training data set (215) based on the sequence of video pictures; applying the CAO training set to the base NN model to generate a base quality metric; applying the CAO training data set to the CAO NN model to generate a CAO NN model with adjusted NN parameters (220) and a cost metric representing a cost of using the CAO NN model over the base NN model; selecting the CAO NN model (240) if the cost metric is smaller than the base quality metric, else selecting the base NN model; generating an output bitstream (250) by applying the selected NN model to the sequence of video pictures; and if the selected NN model is the CAO NN model, then encoding the adjusted NN parameters of the CAO NN model and transmitting them together with the output bitstream.

2. The method of claim 1, wherein the CAO training set is generated using a subset of pictures in a group of pictures (GOP) of the sequence of video pictures.

3. The method of claim 1 or 2, wherein generating a CAO NN model with adjusted NN parameters further comprises: initializing the CAO NN model parameters to be adjusted by values of corresponding parameters in the base NN model; andapplying the CAO training data set to the CAO NN for one or more training iterations to generate the CAO NN model.

4. The method of any preceding claim, wherein the cost metric represents coding gain or loss adjusted by bit rate overhead when using the CAO NN model over the base NN model.

5. The method of claim 4, wherein the cost metric comprises computing loss = CAOMSE+ A. * 8r, wherein CAOMSEdenotes a mean-square error when using the CAO NN model, X denotes a slope of a rate distortion curve generated during encoding, and 6r is a signaling overhead to code and transmit the adjusted NN parameters of the CAO NN model.

6. The method of any preceding claim, wherein the adjusted NN parameters of the CAO NN model are encoded using k-th order Exponential Golomb (EGK) coding.

7. The method of claim 6, wherein an EGK order k is selected to minimize bits used for EGK encoding.

8. The method of claim 7, wherein computing an EGK order kmm comprises: kmin = arg Signwherein 8 (param) denotes absolute differences between the adjusted NN parameters and corresponding base model parameters, sign(8(param)) denotes a sign bit for 6 (param), which is 0 if 6 (param) is equal to 0, and 1 otherwise.

9. The method of claim 5, wherein the signaling overhead is computed based on one or more of: a parameter indicating how many layers of the CAO NN model are being adjusted, and for each layer being adjusted: its layer ID, its EGK order k, differential Q factor values with respect tocorresponding base NN model, and number of bits required to encode differential values for the adjusted NN parameters in the CAO NN model.

10. The method of claim 6, where coding the adjusted NN parameters further comprises: for each adjusted layer of the CAO NN model: generating differences between corresponding parameter values of the CAO NN model and the base NN model; generating an optimal ‘k’ order value for encoding the differences in the adjusted layer; and encoding the differences using an optimal ‘k’ order EGK coding.

11. The method of claim 10, further comprising generating metadata representing the encoded differences, wherein the metadata comprise one or more of: a first flag (nnlf_cao_signal_flag) identifying whether CAO NN parameters are signaled; if CAO NN parameters are signaled, then: a parameter (num_cao_nn_layers_minusl) specifying how many NN layers in the CAO NN model are different from the base NN model, and for each such layer “i”: an index (cao_nn_layer_id[i]j to the NN layer to be adjusted; an EGK k order (k_golomb_val[i]) being used in the NN layer to be adjusted; differences in Q factor parameters between layers of the CAO and base NN models; andEGK coded values of differences in the adjusted NN parameters between adjusted layers in the CAO and base NN models.

12. The method of any preceding claim, wherein generating a CAO NN model with adjusted NN parameters comprises: receiving a list of layers in the base NN model to be adjusted in the CAO NN model; for an input layer in the base NN model to be adjusted:unfreezing NN parameters in the input layer to be adjusted for an in- place CAO NN model; applying the CAO training data set to the in-place CAO NN model to generate a first CAO NN model; generating a first cost metric representing a cost of using the first CAO NN model; inserting a post-multiplier in the input layer to generate a post- multiplier CAO NN model; applying the CAO training data set to the post-multiplier CAO NN model to generate a second CAO NN model; generating a second cost metric representing a cost of using the second CAO NN model; and selecting among the base NN model, the first CAO NN model, and the second CAO NN model based on a best quality metric, the first cost metric, and the second cost metric.

13. The method of claim 12, wherein selecting among the base NN model, the first CAO NN model, and the second CAO NN model comprises: if neither of the first cost metric or the second cost metric is smaller than the best quality metric, then freeze the input layer, reset the NN parameters of the input layer as in the base NN model, and remove the post-multiplier in the input layer; else if the first cost metric is smaller than the second cost metric; removing the post-multiplier in the input layer; selecting the first CAO NN model for the input layer; and updating the best quality metric with the first cost metric; else freezing the input layer; resetting the NN parameters of the input layer as in the base NN model; selecting the second CAO NN model for the input layer; and updating the best quality metric with the second cost metric.

14. The method of claim 12 wherein the best quality metric is initialized with the base quality metric.

15. The method of claim 12, wherein if the second CAO NN model is selected for the input layer, after training, weights (W) and biases (B) of the input layer are adjusted asW' = W * M, B' = B * M , wherein W' and B' denote adjusted weights and biases in the input layer and M denotes post-multiplier multiplication parameters.

16. The method of claim 15, wherein, biases in the input layer can be further adjusted before applying a post multiplier, and given M and updated biases (B') values, the B' and the M values are encoded separately for the input layer.

17. The method of any preceding claim, wherein encoding the adjusted NN parameters of the CAO NN model comprises generating CAO metadata for an adjusted layer of the CAO NN model according to a CAO metadata format, wherein the CAO metadata formats comprise: a) absolute difference values between corresponding parameter values of the CAO NN model and the base NN model; b) difference values between parameter values of the CAO NN model and a global mean, wherein the global mean is computed as an average of all adjusted parameters in the CAO NN model; c) difference values between parameter values of the CAO NN model and a local mean, wherein the local mean is computed using an average of adjusted NN parameters in the adjusted layer; d) difference values between parameter values of the adjusted layer of the CAO NN model and a previously adjusted CAO layer with same dimension as the adjusted layer; e) difference values between multiplication factors in a post-multiplier and a constant; f) difference values between multiplication factors in a post-multiplier and a local mean, wherein the local mean is computed using an average of post-multipliers in the adjusted layer; and g) difference values between multiplication factors in a post-multiplier for the adjusted layer and multiplication factors in a post-multiplier in a previously adjusted layer with same dimension as the adjusted layer.

18. The method of claim 17, wherein the CAO metadata format is selected to minimize CAO metadata overhead.

19. The method of any preceding claim, wherein high-level syntax in the output bitstream specifies whether the adjusted NN parameters of the CAO NN model are encoded using k-th order Exponential Golomb (EGK) coding or MPEG-NNR (ISO / IEC 15938-17) encoding.

20. A method of decoding a coded video bitstream using a neural-network architecture, the method comprising: accessing a base neural network (NN) model of an architecture to be used in video decoding; accessing a coded video bitstream; detecting whether the coded video bitstream was coded using content-adaptive overfitting (CAO); if no CAO was used, generating an output bitstream by decoding the coded video bitstream using the base NN model, else: extracting metadata from the coded video bitstream to generate CAO delta parameters representing differential NN parameters of a CAO NN model over the base NN model; adjusting NN parameters of the base NN model using the CAO delta parameters to generate the CAO NN model; and generating the output bitstream by decoding the coded video bitstream using the CAO NN model.

21. The method of claim 20, wherein the metadata comprise one or more of: a first flag (nnlf_cao_signal_flag) identifying whether CAO NN parameters are signaled; if CAO NN parameters are signaled, then: parameters for determining whether a global CAO mean is available and its value; a parameter (num_cao_nn_layers_minusl) specifying how many NN layers in the CAO NN model are different from the base NN model, and for each such layer “i”:an index (cao_nn_layer_id[i]) to the NN layer to be adjusted; a delta coding type (delta_coding_type) specifying how adjusted CAO NN parameters are transmitted; an EGK k order (k_golomb_val[i]) being used in the NN layer to be adjusted; differences in quantization factor (Q factor) parameters between layers of the CAO and base NN models; andEGK coded values of differences in the adjusted NN parameters between adjusted layers in the CAO and base NN models.

22. The method of claim 21, wherein the delta coding type comprises one of: a) absolute difference values between corresponding parameter values of the CAO NN model and the base NN model; b) difference values between parameter values of the CAO NN model and a global mean, wherein the global mean is computed as an average of all adjusted parameters in the CAO NN model; c) difference values between parameter values of the CAO NN model and a local mean, wherein the local mean is computed using an average of the adjusted NN parameters in the NN layer to be adjusted; d) difference values between parameter values of the NN layer to be adjusted of the CAO NN model and a previously adjusted CAO layer with same dimension as the NN layer to be adjusted; e) difference values between multiplication factors in a post-multiplier and a constant; f) difference values between multiplication factors in a post-multiplier and a local mean, wherein the local mean is computed using an average of post-multipliers in the NN layer to be adjusted; and g) difference values between multiplication factors in a post-multiplier for the NN layer to be adjusted and multiplication factors in a post-multiplier in a previously adjusted layer with same dimension as the NN layer to be adjusted.

23. The method of claim 22, wherein if the delta coding type is related to transmitting multiplication factors (M) for a post-multiplier in a NN layer to be adjusted, then weights (W) and biases (B) of the NN layer to be adjusted are adjusted as:W' = W * M, B' = B * M , wherein W' and B' denote adjusted weights and biases.

24. The method of claim 23, wherein, if the NN layer to be adjusted includes both adjusted biases (B') and a post multiplier (Af), then weights and biases of the NN layer to be adjusted are W' = W * M, and B" = B' * M, wherein W' and B" denote the adjusted weights and biases.

25. A tangible computer-readable storage medium having stored thereon computerexecutable instructions for executing with one or more processors a method in accordance with any one of the methods recited in claims 1-24.

26. An apparatus comprising a processor and configured to perform any one of the methods recited in claims 1-24.

Images