Method and apparatus for hybrid training of neural networks for video coding

Abstract

A method and apparatus for hybrid training of a neural network for video coding is provided, the method including: an offline training phase, training a neural network offline using a plurality of first data sets to obtain an offline-trained network; and an online training phase, refining a plurality of neural network layers using a plurality of second data sets, where the plurality of neural network layers may include at least one neural network layer within the offline-trained network or a simple neural network connected to the offline-trained network.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 156,350, entitled "Methods and Apparatus for Hybrid Training of Neural Networks for Video Coding," filed March 3, 2021, which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates to video coding, and more particularly, but not exclusively, to methods and apparatus for hybrid training of neural networks for video coding. [Background technology]

[0003] Various video coding techniques may be used to compress video data. Video coding is performed according to one or more video coding standards. For example, video coding standards include Versatile Video Coding (VVC), Joint Exploration Test Model (JEM), High-Efficiency Video Coding (H.265 / HEVC), Advanced Video Coding (H.264 / AVC), Moving Picture Expert Group (MPEG) coding, etc. Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit redundancy present in a video image or sequence. An important goal of video coding techniques is to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality.

[0004] The first version of the HEVC standard was finalized in October 2013 and provides approximately 50% bitrate savings or equivalent perceptual quality compared to the previous generation video coding standard, H.264 / MPEG AVC. While the HEVC standard offers significant coding improvements over its predecessor, there is evidence that even better coding efficiency can be achieved by adding coding tools beyond HEVC. Based on this, both VCEG and MPEG have begun work on exploring new coding techniques for future video coding standards. A Joint Video Exploration Team (JVET) was formed by ITU-T VECG and ISO / IEC MPEG in October 2015 to initiate a critical review of advanced technologies that could enable significant improvements in coding efficiency. JVET maintains a single reference software, called the Joint Exploration Model (JEM), by integrating several additional coding tools in addition to the HEVC test model (HM).

[0005] A joint Call for Proposal (CfP) for video compression capabilities beyond HEVC was issued by ITU-T and ISO / IEC. 23 CfP responses were received and evaluated at the 10th JVET meeting. This demonstrated a compression efficiency gain of approximately 40% for HEVC. Based on these evaluation results, JVET launched a new project to develop a new generation video coding standard named Versatile Video Coding (VVC). A reference software code base called the VVC Test Model (VTM) was established to demonstrate a reference product for the VVC standard. Summary of the Invention

[0006] This disclosure provides example techniques related to improving video coding efficiency using hybrid training for neural networks for video coding.

[0007] According to a first aspect of the present disclosure, for video encoding, No. 1 A method for hybrid training of a neural network is provided, comprising: in an offline training phase, training the neural network offline using a plurality of first data sets to obtain an offline-trained network; and in an online training phase, refining a plurality of neural network layers using a plurality of second data sets, the plurality of neural network layers being either the offline-trained network or a hybrid network connected to the offline-trained network. No. 2 The neural network may include at least one neural network layer.

[0008] According to a second aspect of the present disclosure, there is provided an apparatus for hybrid training of a neural network for video coding, the apparatus comprising: one or more processors; and a memory configured to store instructions executable by the one or more processors, the one or more processors configured, upon execution of the instructions, to perform a method according to the first aspect.

[0009] According to a third aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform a method according to the first aspect. According to a fourth aspect of the present disclosure, there is provided a video encoding method, the method comprising: receiving, at a decoder, a bitstream including coded video information and online trained parameters; and obtaining decoded video information by decoding using the bitstream, wherein the online trained parameters in the bitstream are generated by the method according to the first aspect. [Brief explanation of the drawings]

[0010] A more particular description of examples of the present disclosure will be made by reference to certain examples that are illustrated in the accompanying drawings, which depict only some examples and are therefore to be considered non-limiting in scope, each example being described and explained with additional specificity and detail through the use of the accompanying drawings.

[0011] [Figure 1] FIG. 1 is a block diagram illustrating a block-based video encoder according to some implementations of this disclosure.

[0012] [Figure 2] FIG. 1 is a block diagram illustrating a block-based video decoder according to some implementations of this disclosure.

[0013] [Figure 3A] 1 is a diagram illustrating a quadtree splitting mode according to some implementations of the present disclosure.

[0014] [Figure 3B] 1 is a diagram illustrating a vertical binary tree splitting mode according to some implementations of the present disclosure.

[0015] [Figure 3C] 1 is a diagram illustrating a horizontal binary tree splitting mode according to some implementations of the present disclosure.

[0016] [Figure 3D] 1 is a diagram illustrating a vertical ternary tree splitting mode according to some implementations of the present disclosure.

[0017] [Figure 3E] 1 is a diagram illustrating a horizontal ternary tree splitting mode according to some implementations of the present disclosure.

[0018] [Figure 4] FIG. 1 illustrates a simple FC-NN consisting of an input layer, an output layer, and multiple hidden layers according to some implementations of the present disclosure.

[0019] [Figure 5A] FIG. 1 illustrates an FC-NN including two hidden layers according to some implementations of the present disclosure.

[0020] [Figure 5B]FIG. 10 illustrates an example of a CNN in which the dimensions of the second hidden layer are [W, H, depth] according to some implementations of the present disclosure.

[0021] [Figure 6] 10A-10C illustrate examples of applying a spatial filter to an input image according to some implementations of the present disclosure.

[0022] [Figure 7A] FIG. 1 illustrates a single image super-resolution (ResNet) including residual blocks that are elements of the ResNet that are element-wise summed with its inputs by identity connections, according to some implementations of the present disclosure.

[0023] [Figure 7B] FIG. 1 illustrates an example of a ResNet with staking residual modules according to some implementations of the present disclosure.

[0024] [Figure 8] FIG. 1 illustrates a hybrid training framework in accordance with some implementations of the present disclosure.

[0025] [Figure 9A] FIG. 1 illustrates an example of an offline training network and online training for a simple neural network connected to the offline training network, according to some implementations of the present disclosure.

[0026] [Figure 9B] FIG. 1 illustrates an example of online training for an offline-trained network according to some implementations of the present disclosure.

[0027] [Figure 9C] FIG. 1 illustrates an example of an offline training network and online training for a simple neural network connected to the offline training network, according to some implementations of the present disclosure.

[0028] [Figure 9D] FIG. 1 illustrates an example of a residual block included in an offline training network according to some implementations of the present disclosure.

[0029] [Figure 10] FIG. 1 is a block diagram illustrating an apparatus for hybrid training of neural networks for video coding according to some implementations of the present disclosure.

[0030] [Figure 11] 1 is a flowchart illustrating a process for hybrid training of a neural network for video coding according to some implementations of the present disclosure.

[0031] [Figure 12] 1 is a flowchart illustrating a process for hybrid training of a neural network for video coding according to some implementations of the present disclosure.

[0032] [Figure 13] 1 is a flowchart illustrating a process for hybrid training of a neural network for video coding according to some implementations of the present disclosure.

[0033] [Figure 14] 1 is a flowchart illustrating a process for hybrid training of a neural network for video coding according to some implementations of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0034] Reference will now be made in detail to specific implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in an understanding of the subject matter presented herein. However, it will be apparent to those skilled in the art that various modifications may be used. For example, it will be apparent to those skilled in the art that the subject matter presented herein may be implemented in many types of electronic devices with digital video capabilities.

[0035] References throughout this specification to "one embodiment," "an embodiment," "example," "some embodiments," "some examples," or similar terms mean that a particular feature, structure, or characteristic described is included in at least one embodiment or example. Furthermore, a feature, structure, element, or characteristic described with respect to one or some embodiments is also applicable to other embodiments unless expressly stated otherwise.

[0036] Throughout this disclosure, the terms "first," "second," "third," etc. are all used to refer to related elements, e.g., devices, components, compositions, steps, etc., and do not imply a spatial or temporal order unless expressly specified otherwise. For example, a "first device" and a "second device" may refer to two separately formed devices or two parts, components, or operational states of the same device, and may be arbitrarily named.

[0037] The terms "module," "sub-module," "electrical circuit," "sub-electrical circuit," "circuit," "sub-circuit," "unit," or "sub-unit" may include memory (shared, dedicated, or group) that stores code or instructions that can be executed by one or more processors. A module may include one or more electrical circuits that may or may not contain stored code or instructions. A module or electrical circuit may include one or more components that are directly or indirectly connected. These components may or may not be physically connected to each other or located near each other.

[0038] As used herein, the terms "if" or "when" may be understood as "upon" or "depending on," depending on the context. These terms, even if recited in a claim, may not imply that the associated limitation or characteristic is conditional or optional. For example, a method may include the steps: i) function or operation X' is performed when or if condition X exists; and ii) function or operation Y' is performed when or if condition Y exists. The method may be performed with both the ability to perform function or operation X' and the ability to perform function or operation Y'. Thus, both functions X' and Y' may be performed at different times during multiple executions of the method.

[0039] A unit or module may be implemented purely by software, purely by hardware, or by a combination of hardware and software. In a purely software implementation, for example, a unit or module may include functionally related coding blocks or software components that are directly or indirectly connected to each other to perform specific functions.

[0040] Like HEVC, VVC is built on a block-based hybrid video coding framework. Figure 1 is a block diagram illustrating a block-based video encoder according to some implementations of this disclosure. In encoder 100, an input video signal is processed in blocks called coding units (CUs). In VTM-1.0, a CU could be up to 128x128 pixels. However, unlike HEVC, which divides blocks based only on quaternary trees, VVC divides one coding tree unit (CTU) into CUs to adapt to various local features based on quaternary / biary / tertial trees. Additionally, the concept of multiple division unit types in HEVC is eliminated; that is, VVC does not have a division into CUs, prediction units (PUs), and transform units (TUs). Instead, each CU is always used as a basic unit for both prediction and transformation without further division. In a multi-type tree structure, one CTU is first divided by a quaternary tree structure. Then, each quaternary tree leaf node can be further divided by biary and tertial tree structures.

[0041] 3A to 3E are diagrams illustrating multi-type tree partitioning modes according to some implementations of the present disclosure, each showing five partitioning types, including 4-way (FIG. 3A), vertical 2-way (FIG. 3B), horizontal 2-way (FIG. 3C), vertical 3-way (FIG. 3D), and horizontal 3-way (FIG. 3E).

[0042] For each given video block, spatial prediction and / or temporal prediction may be performed. Spatial prediction (or "intra prediction") predicts the current video block using pixels from samples (called reference samples) of nearby blocks already coded in the same video picture / slice. Spatial prediction reduces spatial redundancy inherent in video signals. Temporal prediction (also called "inter prediction" or "motion compensated prediction") predicts the current video block using reconstructed pixels from already coded video pictures. Temporal prediction reduces temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically conveyed by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. If multiple reference pictures are supported, a reference picture index is additionally transmitted, which is used to identify which reference picture in the reference picture store the temporal prediction signal comes from.

[0043] After spatial prediction and / or temporal prediction, an intra / inter mode decision circuit 121 in encoder 100 selects the best prediction mode based on, for example, a rate-distortion optimization method. Next, block predictor 120 is subtracted from the current video block, and the resulting prediction residual is decorrelated using transform circuit 102 and quantization circuit 104. The resulting quantized residual coefficients are inverse quantized by inverse quantization circuit 116 and inverse transformed by inverse transform circuit to form a reconstructed residual, which is then added back to the prediction block to form a reconstructed signal for the CU. Furthermore, an in-loop filter 115, such as a deblocking filter, a sample adaptive offset (SAO), and / or an adaptive in-loop filter (ALF), may be applied to the reconstructed CU, which is then placed in reference picture storage in picture buffer 117 and used to encode further video blocks. To form the output video bitstream 114, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit 106 for further compression and packing to form the bitstream.

[0044] For example, deblocking filters are available in the latest versions of VVC, as well as AVC and HEVC. In HEVC, an additional in-loop filter called SAO is specified to further improve coding efficiency. In the latest versions of the VVC standard, yet another in-loop filter called ALF is being actively investigated and is likely to be included in the final standard.

[0045] These in-loop filter operations are optional. Implementing these operations helps improve coding efficiency and visual quality. They may also be turned off, if determined by the encoder 100, to save computational effort.

[0046] It should be noted that intra prediction is typically based on unfiltered reconstructed pixels, while inter prediction is based on filtered reconstructed pixels if these filter options are turned on by the encoder 100 .

[0047] Figure 2 is a block diagram illustrating a block-based video decoder 200 that may be used with numerous video coding standards. This decoder 200 is similar to the reconstruction-related section residing in the encoder 100 of Figure 1. In the decoder 200, an input video bitstream 201 is first decoded through entropy decoding 202 to derive quantized coefficient levels and prediction-related information. The quantized coefficient levels are then processed through inverse quantization 204 and an inverse transform 206 to obtain a reconstructed prediction residual. A block predictor mechanism implemented in an intra / inter mode selector 212 is configured to perform either intra prediction 208 or motion compensation 210 based on the decoded prediction information. A set of unfiltered reconstructed pixels is obtained by adding, using an adder 214, the reconstructed prediction residual from the inverse transform 206 and the prediction output generated by the block predictor mechanism.

[0048] The reconstructed blocks are further passed through an in-loop filter 209 and then stored in a picture buffer 213, which acts as a reference picture store. The reconstructed video in the picture buffer 213 is sent to drive a display device and is also used to predict future video blocks. When the in-loop filter 209 is turned on, a filtering operation is performed on these reconstructed pixels to derive the final reconstructed video output 222.

[0049] When a neural network is used in a video coding system, the neural network model may be trained offline or online. This disclosure describes a method and apparatus for a hybrid training framework in which both online and offline training methods are used to improve video coding efficiency. For ease of explanation, this disclosure takes an in-loop filter as an example, but the proposed idea may be similarly extended to other functional parts of a video coding system, such as prediction mode decision or transformation in FIG. 1. The proposed hybrid training method can be applied to traditional block-based video coding systems or end-to-end neural network-based video coding systems.

[0050] Neural network techniques, such as Fully Connected Neural Networks (FC-NNs), Convolutional Neural Networks (CNNs), and Long Short-Term Memory networks (LSTMs), have already achieved remarkable success in many research areas, including computer vision and video understanding.

[0051] Fully connected neural network (FC-NN) 4 illustrates a simple FC-NN consisting of an input layer, an output layer, and multiple hidden layers according to some implementations of the present disclosure. k (x k-1 ,W k ,B k ) is generated by the following formula:

number

number

number

number

number

number

[0052] According to the universal approximation assumption and equation (4), if any continuous function g(x) and some ε>0 is applied,

number

[0053] Convolutional Neural Networks (CNNs) FIG. 5A illustrates an FC-NN with two hidden layers according to some implementations of the present disclosure. CNNs, a common neural network configuration for image or video applications, are very similar to the FC-NN shown in FIG. 5A, including weight and bias matrices. CNNs can be viewed as a 3D version of neural networks. FIG. 5B illustrates an example of a CNN with the dimensions of the second hidden layer [W, H, depth] according to some implementations of the present disclosure. In FIG. 5B, neurons are arranged in a three-dimensional structure (width, height, and depth) to form a CNN, and the second hidden layer is visualized. In this example, the input layer maintains the input image or video frame, so its width and height are the same as the input data. For application with image or video applications, each neuron in the CNN is a spatial filter element whose depth is expanded to match its input. For example, if the input image has three color components, the depth is 3.

[0054] 6 illustrates an example of applying a spatial filter with an input image according to some implementations of the present disclosure. As shown in FIG. 6, the dimensions of the basic elements of the CNN are [Filter width , Filter height , Input depth , Output depth ] and is set to [5, 5, 3, 4] in this example. Each spatial filter performs a 2D spatial convolution on the input image using 5x5x3 weights. The input image is a 64x64x3 image. In this case, four convolution results are output. Therefore, after padding the borders with an additional 2 pixels, the dimensions of the filtered result are [64+4, 64+4, 4].

[0055] Residual Network (ResNet) In image classification, as the depth of a neural network increases, accuracy saturates and rapidly declines. More specifically, adding more layers to a deep neural network causes the gradient to gradually vanish along the deep network, eventually approaching zero, resulting in a high training error. In this case, ResNet, which is composed of residual blocks, solves the accuracy degradation problem by introducing identity connections.

[0056] 7A illustrates a ResNet including a residual block, which is an element of the ResNet that is element-wise summed with its input by an identity connection, according to some implementations of the present disclosure. As shown in FIG. 7A, the basic module of a ResNet consists of a residual block and an identity connection. According to the universal approximation assumption, when an input x is applied, a weighted layer with the activation function of the residual block approximates the hidden function F(x) rather than the output H(x) = F(x) + x.

[0057] By stacking nonlinear multi-layer neural networks, the residual block searches for features that represent local characteristics of the input image. As shown in Figure 7A, identity connection has been proven to construct trainable deep learning networks by omitting one or more nonlinear weighted layers without introducing any additional parameters or computational complexity. By omitting the weighted layers, the differential output of the residual layer can be expressed as follows:

number

[0058] Therefore, even if the difference term ∂H(x) / ∂x gradually approaches zero, the identity term can still inherit the input and pass it on to the next layer, instead of the gradient remaining zero and further preventing information propagation. If a neuron cannot propagate information to the next neuron, it appears dead and becomes an untrainable element in the neural network. After addition, another nonlinear activation function can be applied as well. FIG. 7B shows an example of a ResNet with residual module staking according to some implementations of the present disclosure. As shown in FIG. 7B, the residual features are fused with the identity features before being propagated to the next module.

[0059] For better generalization, the offline-trained neural network may include multiple layers and is repeatedly updated during the training phase. A relatively large number of layers may increase the size of the parameters, which may result in excessive processing delay and memory consumption during the inference phase. In addition, although the offline-trained neural network model is learned by a given dataset, when the model is applied to other parts, it may not reflect the actual data features, such as illumination, texture, object deformation, etc.

[0060] On the other hand, online training neural networks are well adapted to real application scenarios, but are constrained by smaller parameter sizes, which may limit the utilization of the nonlinear representation capabilities of neural networks.

[0061] For video coding systems, the ground truth, e.g., uncompressed video data, is always accessible at the encoder side, making online training feasible. To pursue combined advantages, we disclose a hybrid training framework that includes both online and offline training.

[0062] Hybrid Training Framework This disclosure proposes a hybrid training framework for improving the coding efficiency of neural networks used for video coding systems. The proposed hybrid training framework may include any combination of the following features:

[0063] In some examples, one or more neural networks with the same or different network configurations may be first trained offline. The datasets selected for offline training may include different data features, such as still images with various background textures, videos with little object motion, etc., for the purpose of better generalization.

[0064] The offline-trained model is then refined by online training, e.g., trained online to adapt to specific video content. During this online training phase, only a small number of layers need to be updated. For layers that are not updated, for example, if the training algorithm is based on gradient descent, the gradients of those layers may be forced to zero. In certain neural network frameworks, such as PyTorch, layers that are not updated may be set to untrainable.

[0065] In some other instances, during the online training phase, the offline trained model may be treated as a feature extractor, with or without further refinement, and a new simple model with very few layers or weights may be generated. neural The network may be connected after offline training models. In this disclosure, the offline trained neural network may also be referred to as the first neural network, and the simple neural network may also be referred to as the second neural network.

[0066] The same or different learning algorithms may be used in different training stages. In one or more examples, in an offline training stage where convergence speed is not critical and generalization is more important, it may be preferable to use a learning algorithm such as Stochastic Gradient Descent (SGD). On the other hand, in an online training stage where both training speed and fast adaptation to training data are important, it may be preferable to use a fast learning algorithm such as Adaptive Moment Estimation (ADAM).

[0067] In one or more examples, switching between different algorithms may occur during the same training phase, for example, during an offline training phase, a fast algorithm such as ADAM is used for the first epoch, and then the learning algorithm is switched to SGD if the loss function produces a loss below a predetermined threshold.

[0068] In some cases, when a learning algorithm is switched during the same training phase or between different training phases, the preferred starting value of the learning rate parameter may also be switched at the same time. A particular value of the learning rate may be associated with the loss function or the algorithm itself.

[0069] In online training-based model refinement, two factors related to the time range include temporal frequency and temporal coverage. The temporal frequency determines when the online training-based model refinement is performed.

[0070] In one or more examples, online training may occur periodically, for example, every 2 seconds, every 1 second, or every N video frames, where N may be a positive integer. In some examples, online training may be triggered based on an event.

[0071] In one or more examples, online training may be triggered by a video encoder's behavior, such as the generation of Instantaneous Decoder Refresh (IDR) coded pictures. Each time a new IDR frame is generated, subsequent motion-predicted video frames may contain entirely different levels of artifacts, for example, due to changes in reference pictures.

[0072] In one or more examples, online training may be triggered by a video encoder configuration, such as an intra-period value or a frame rate. The encoder configuration may be dynamically changed for various reasons, such as bitrate control, error concealment, etc. The encoder configuration change may be performed by a given profile or a lower protocol layer of the operating system running on the device.

[0073] In one or more examples, online training may be triggered by video content, such as scene detection. When an object, an object background, or an object and an object background in a video frame changes, the prediction, partition granularity, or the prediction and partition granularity in a video frame may also change simultaneously. These content changes are the main cause of video coding bits, coding distortion, or video coding bits and coding distortion.

[0074] In some examples, online training may be initiated in a combined periodic and event-triggered manner, for example, online training may first be performed periodically and then omitted in response to a trigger event.

[0075] Furthermore, temporal coverage determines the training data used for model updating. Convergence speed is important in online training. Training data not only affects processing delay but also training convergence. For example, video data with high diversity makes training convergence difficult.

[0076] In one or more examples, coding information, such as temporal layer information, may be used as a criterion for selecting training data. For example, within each group of pictures, one picture from each temporal layer may be selected for online training.

[0077] In one or more examples, video content may be used as a criterion for selecting training data. For example, within each online training period, adjacent video frames may be highly temporally redundant, meaning that objects and backgrounds are highly similar. In this case, scene information or regions of interest, including both objects and background textures, may be extracted at the encoder side, and these extracted small patches, instead of full pictures, may be used as online training data.

[0078] In some examples, information other than video frames is also used, e.g., as training data for multi-frame-based training. For example, motion information between each selected video frame and a fixed frame, e.g., a reference picture, may be used to train a motion prediction frame. In this case, an optical flow map may be generated as a representation of the motion information.

[0079] In the online training stage, each selected video frame may be divided or cropped into small patches before being used as online training data. The small patches may be clustered according to certain criteria. In some examples, only patches classified as the same or different clusters are selected as online training data.

[0080] In some examples, each small patch may be represented by a scale-invariant feature descriptor, e.g., a histogram of oriented gradients or a template, e.g., a feature vector based on a set of words. Different patches are clustered by their representational similarity.

[0081] In some examples, the patches are clustered by different levels of distortion.

[0082] During the online training phase, video frames or patches with low quality, e.g., quantization parameters exceeding a certain threshold, may have blurred object boundaries or texture details. Therefore, these data may cause serious training disruptions that make training convergence difficult and should not be selected as training data.

[0083] In video coding, an offline-trained neural network model is expected to be used on a variety of different video sequences after training, and therefore requires sufficient generalization, while an online-trained neural network model only needs to fit video data within a small time space, e.g., fit video frames within an intra-period, and therefore requires good overfitting. Due to the different requirements, the model training process may be considered in terms of a trade-off between generalization and overfitting.

[0084] In one or more examples, the trade-off between generalization and overfitting can be controlled by training the network on a sufficient amount of data. This has been discussed previously. For example, offline models are trained on larger and more diverse datasets, while online models or model refinement are trained within constrained time, space, and quality bounds.

[0085] In one or more examples, the balance of the trade-off between generalization and overfitting can be controlled by changing the complexity of the network. In one example, the complexity of the network can be changed by adjusting the network structure. Specifically, the number of trainable model parameters or weights can be flexibly configured. For example, an online training network may set only a few layers to be trainable, while an offline training network may set many or all layers to be trainable. This method may be referred to as structural stabilization.

[0086] In another example, network complexity can be varied by adjusting the dynamic range of the network parameters. Parameters with a small dynamic range are less susceptible to statistical fluctuations in the input training data and therefore have better generalization ability. However, when overfitting is desired, such as in online models, parameters with a larger dynamic range are desirable.

[0087] In some examples, data types, such as integers or floating-points, are used to control the dynamic range of network parameters or weights. In some other examples, regularization methods are used to control the dynamic range of network parameters or weights. One typical regularization method is to impose a penalty, such as a loss function, on the model during the training process based on the magnitude of the parameter or weight value. This approach is called weight regularization or weight reduction.

[0088] 8 illustrates a hybrid training framework according to some implementations of the present disclosure. As shown in FIG. 8, one or more models are first trained offline using a large amount of image or video data with diverse data features, and then the models are iteratively refined through online training using a small amount of data. The goal of online training is to reduce the complexity of the network so that the network can better overfit the small amount of data within each refinement period. The network complexity can be reduced by limiting the number of trainable layers, parameters, or weights in the network, or by setting a suitable dynamic range for such trainable parameters or weights.

[0089] 9A to 9D illustrate an offline training network and / or a simple network connected to the offline training network. neural In addition to the network, an example of how online training can be performed is shown in Figures 9A to 9D. As shown in Figure 9A, a new simple network is connected to the offline training network. neural The network may be trained in an online training phase, or a subset of layers in the offline training network may be refined as shown in Figure 9B, or both a new simple network and a subset of layers in the offline training network may be trained in the online training phase as shown in Figure 9C.

[0090] Sending online trained parameters or weights The output after each online training is limited in size of the network parameters or weights. To reduce the signaling overhead, the parameters or weights need to be compressed before being written to the bitstream.

[0091] In one or more examples, parameter pruning and quantization can still be performed, even though the updated parameters may be large and susceptible to variations in the input data due to overfitting. However, to maintain sensitivity, manual setting of the sensitivity is beneficial, which is feasible because the number of layers is typically small.

[0092] In one or more examples, instead of directly communicating the updated parameters, methods such as mean shift or weight sharing may be used as a pre-processing step. The differences before and after the parameter update can be derived and constructed as a difference map, which can further reduce the bits for communicating the updated parameters.

[0093] Interaction with other existing in-loop filters When a network trained for video encoding and decoding is used as an in-loop filter, the updated parameters after online training can be used to guide the training of other in-loop filters. For example, in the case of an adaptive in-loop filter (ALF), the derived filter coefficients can be expanded or adjusted to approximate the updated parameters. Once the ALF filter coefficients are refined, there is no need to propagate the updated neural network parameters, which not only saves bits but also reduces the processing delay due to the use of neural network model filtering.

[0094] 10 is a block diagram illustrating an apparatus for hybrid training of neural networks for video coding according to some implementations of the present disclosure. The apparatus 1000 may be a terminal such as a mobile phone, a tablet computer, a digital broadcast terminal, a tablet device, or a personal digital assistant.

[0095] As shown in FIG. 10, the device 1000 may include one or more of a processing component 1002, a memory 1004, a power supply component 1006, a multimedia component 1008, an audio component 1010, an input / output (I / O) interface 1012, a sensor component 1014, and a communication component 1016.

[0096] The processing component 1002 typically controls the overall operation of the device 1000, including operations related to display, telephony, data communication, camera operation, and audio recording. The processing component 1002 may include one or more processors 1020 that execute instructions to complete all or some of the steps of the methods described above. Additionally, the processing component 1002 may include one or more modules that facilitate interaction between the processing component 1002 and other components. For example, the processing component 1002 may include a multimedia module that facilitates interaction between the multimedia component 1008 and the processing component 1002.

[0097] Memory 1004 is configured to store various types of data to support the operation of device 1000. Examples of such data include instructions for any applications or methods operating on device 1000, contact data, phone book data, messages, photos, videos, etc. Memory 1004 may be implemented by any type of volatile or non-volatile storage device or combination thereof, or memory 1004 may be static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or compact disk.

[0098] Power supply component 1006 provides power to the different components of device 1000. Power supply component 1006 may include a power supply management system, one or more power supplies, and other components related to generating, managing, and distributing power for device 1000.

[0099] The multimedia component 1008 includes a screen that provides an output interface between the device 1000 and a user. In some examples, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen that receives input signals from a user. The touch panel may include one or more touch sensors that detect touches, slides, and gestures on the touch panel. The touch sensors can detect not only the boundaries of a touch or slide operation but also the duration and pressure associated with the touch or slide operation. In some examples, the multimedia component 1008 may include a front camera and / or a rear camera. When the device 1000 is in an operating mode, such as a photo mode or a video mode, the front camera and / or the rear camera can receive external multimedia data.

[0100] The audio component 1010 is configured to output and / or input audio signals. For example, the audio component 1010 includes a microphone (MIC). When the device 1000 is in an operation mode such as a call mode, a recording mode, or a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in the memory 1004 or transmitted via the communication component 1016. In some examples, the audio component 1010 may further include a speaker that outputs the audio signal.

[0101] The I / O interface 1012 forms an interface between the processing component 1002 and a peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a start button, and a lock button.

[0102] The sensor component 1014 includes one or more sensors for providing status assessment of different aspects of the device 1000. For example, the sensor component 1014 can detect the on / off state of the device 1000 and the relative position of components. For example, the components can be the display and keypad of the device 1000. The sensor component 1014 can also detect changes in the position of the device 1000 or its components, the presence or absence of a user's contact with the device 1000, the orientation or acceleration / deceleration of the device 1000, and temperature changes of the device 1000. The sensor component 1014 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor component 1014 may further include an optical sensor, such as a CMOS or CCD image sensor used in imaging applications. In some examples, the sensor component 1014 may further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0103] The communication component 1016 is configured to facilitate wired or wireless communication between the device 1000 and other devices. The device 1000 can access a wireless network based on a communication standard such as WiFi, 4G, or a combination thereof. In one example, the communication component 1016 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In one example, the communication component 1016 may further include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra-Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0104] In one example, the apparatus 1000 may be implemented by one or more of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic elements to perform the aforementioned methods. The non-transitory computer-readable storage medium may be, for example, a hard disk drive (HDD), a solid-state drive (SSD), a flash memory, a hybrid drive or a solid-state hybrid drive (SSHD), a read-only memory (ROM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, etc.

[0105] FIG. 11 is a flowchart illustrating a process for hybrid training of a neural network for video coding according to some implementations of the present disclosure.

[0106] In step 1101, the processor 1020 obtains an offline training network by training a neural network offline using a plurality of first data sets in an offline training phase. As shown in Figure 8, the plurality of first data sets may include large offline data, such as images and videos. A neural network that is trained offline is sometimes called a first neural network.

[0107] In some examples, the multiple first datasets may include datasets with different features for better generalization purposes, e.g., still images with different background textures, videos with little object movement.

[0108] In step 1102, the processor 1020 refines a plurality of neural network layers using a plurality of second data sets in an online training phase.

[0109] In some examples, the multiple neural network layers may include at least one neural network layer within an offline training network or a simple neural network connected to an offline training network, as shown in FIGS. 9A-9C. Simple neural networks are sometimes called second-order neural networks. As shown in Figure 9A, the offline training network or model includes multiple residual blocks (RBs). Each RB may have the same structure as that shown in Figure 9D. As shown in Figure 9D, the RB includes two convolutional layers and a rectified linear unit (Relu) layer sandwiched between the two convolutional layers. A simple RB connected to the offline training network neural The network includes, but is not limited to, two convolutional layers. Online training is simple. neural Only two convolutional layers in the network (Figure 9A), only the last RB in the offline-trained network (Figure 9B), or the last RB in the offline-trained network and a simple neural This may be implemented in two convolutional layers in the network (Figure 9C). The number of neural network layers of the second neural network is less than the number of neural network layers of the first neural network.

[0110] In some examples, the processor 1020 iteratively refines multiple neural network layers using multiple second data sets.

[0111] In some examples, the offline training network may include more than the number of neural network layers that are refined, as shown in Figures 9A-9C.

[0112] In some examples, the simple neural network may include fewer neural network layers than the offline training network.

[0113] In some examples, the processor 1020 may further implement different learning algorithms in the same or different training phases, or apply different starting values ​​of learning rate parameters in the learning algorithms in the same or different training phases. For example, both the SGD algorithm and the ADAM algorithm may be implemented in an offline training phase or an online training phase. In some examples, the SGD algorithm may be implemented in an offline training phase, and the ADAM algorithm may be implemented in an online training phase.

[0114] In some examples, the processor 1020 may switch from the ADAM algorithm to the SGD algorithm in response to determining, during the offline training phase, that the loss produced by the loss function is below a predetermined threshold.

[0115] In some examples, the multiple neural network layers may be refined periodically and / or in response to a trigger event. Further, the trigger event may include one of generating an IDR-encoded picture, a change in video encoder configuration, or a change in video content, where the video content may include an object and / or a background of the object within a video frame.

[0116] In some examples, the plurality of second data sets may include encoding information, video content, and / or information other than video frames, where the encoding information may include temporal layer information, the video content may include a plurality of patches extracted from the video frames based on regions of interest, and the information other than video frames may include motion information.

[0117] In some examples, the multiple patches may be selected from patches belonging to the same cluster or patches belonging to different clusters, where different patches cropped from a video frame correspond to different clusters based on representational similarity or level of distortion.

[0118] In some examples, the plurality of second data sets may include low-quality video frames or patches. For example, the low-quality video frames or patches may have a quality below a predetermined threshold. In one example, the predetermined threshold corresponds to a quantization parameter (QP) equal to 32.

[0119] In some examples, the first plurality of data sets may contain more data than the second plurality of data sets, which may be a small amount of data, for example, a small number of video frames, as shown in FIG.

[0120] In some examples, the processor 1020 iteratively refines multiple neural network layers by controlling the trainable structure of the network parameters by configuring a subset of the network parameters to be trainable.

[0121] In some examples, the processor 1020 iteratively refines multiple neural network layers by controlling the data types of the network parameters.

[0122] In some examples, the processor 1020 iteratively refines the multiple neural network layers by using weight regularization to control network parameters of the multiple neural network layers.

[0123] FIG. 12 is a flowchart illustrating a process for hybrid training of a neural network for video coding according to some implementations of the present disclosure.

[0124] In step 1201, the processor 1020 obtains a plurality of refined network parameters by iteratively refining a plurality of neural network layers.

[0125] In step 1202, the processor 1020 obtains a plurality of processed network parameters based on the plurality of refined network parameters.

[0126] In some examples, the processor 1020 obtains a plurality of processed network parameters based on the plurality of refined network parameters by pruning or quantizing the plurality of refined network parameters, or by processing the plurality of refined network parameters using mean shift or weight sharing.

[0127] In step 1203, the processor 1020 communicates the plurality of processed network parameters.

[0128] FIG. 13 is a flowchart illustrating a process for hybrid training of a neural network for video coding according to some implementations of the present disclosure.

[0129] In step 1301, the processor 1020 obtains a plurality of refined network parameters by iteratively refining a plurality of neural network layers.

[0130] In step 1302, the processor 1020 obtains the differences between the original network parameters of the plurality of neural network layers and the plurality of refined network parameters.

[0131] In step 1303, the processor 1020 constructs a difference map based on the differences. In some examples, instead of directly or indirectly communicating the processed network parameters, the encoder may transmit the difference map to the decoder.

[0132] FIG. 14 is a flowchart illustrating a process for hybrid training of a neural network for video coding according to some implementations of the present disclosure.

[0133] In step 1401, the processor 1020 obtains a plurality of refined network parameters by iteratively refining a plurality of neural network layers.

[0134] In step 1402, the processor 1020 expands or adjusts the filter coefficients of the encoder's ALF to approximate the plurality of refined network parameters.

[0135] In some other examples, a non-transitory computer-readable storage medium 1004 is provided having instructions stored thereon that, when executed by one or more processors 1020, cause the processors to perform the methods described in Figures 11-14 and any of the methods described above. In one example, these instructions may be executed by processor 1020 of apparatus 1000 to perform the above-described method in an encoding step to encode video information (e.g., video blocks indicating video frames and / or associated one or more syntax elements) and the online trained parameters or weights into a bitstream or data stream, and to transmit this bitstream or data stream (e.g., to video decoder 200 of FIG. 2). Alternatively, for example, a non-transitory computer-readable storage medium may store the coded video information (e.g., video blocks indicating coded video frames and / or associated one or more syntax elements) and the bitstream or data stream having the online trained parameters or weights generated by an encoder (e.g., video encoder 100 of FIG. 1) in an encoding step using the above-described method, for use in decoding video data by a decoder (e.g., video decoder 200 of FIG. 2). The non-transitory computer-readable storage medium may be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. In some examples, a video coding method is provided that includes receiving, at a decoder, a bitstream having coded video information and online trained parameters, and obtaining decoded video information by decoding using the bitstream, where the online trained parameters in the bitstream are generated by the above-described method for hybrid training of a neural network.

[0136] The description of the present disclosure has been presented for purposes of illustration, but is not intended to be exhaustive or limiting of the disclosure. Certain modifications, variations and alternative implementations will be apparent to one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings.

[0137] These examples have been chosen and described to explain the principles of the disclosure and to enable others skilled in the art to understand the disclosure with respect to various implementations and to best utilize the underlying principles and various implementations with various modifications suited to the particular applications intended. Accordingly, it should be understood that the scope of the disclosure is not limited to the specific implementations disclosed, and that modifications and other implementations are intended to be included within the scope of the present disclosure.

Claims

1. In an offline training phase, a first neural network is trained offline using a plurality of first data sets to obtain an offline training network; and refining a plurality of neural network layers using a plurality of second data sets during an online training phase, the plurality of neural network layers including at least one neural network layer within the offline training network or a second neural network connected to the offline training network; iteratively refining the plurality of neural network layers to obtain a plurality of refined network parameters; pruning or quantizing the refined network parameters; or processing the refined network parameters using mean shift or weight sharing; obtaining a plurality of processed network parameters based on the plurality of refined network parameters by one of the operations: communicating the plurality of processed network parameters; obtaining differences between original network parameters of the plurality of neural network layers and the plurality of refined network parameters; constructing a difference map based on the differences. A method for hybrid training of neural networks for video coding.

2. 2. The method of claim 1 , wherein refining the plurality of neural network layers using the plurality of second data sets comprises iteratively refining the plurality of neural network layers using the plurality of second data sets.

3. The method of claim 1 , wherein the offline training network includes more neural network layers than the plurality of neural network layers being refined.

4. the offline training network includes a plurality of residual blocks; the output of each of the plurality of residual blocks is factorially summed with the input of the offline training network by an identity connection; The method of claim 1 , wherein the plurality of neural network layers that are refined includes a final residual block in the offline training network.

5. 2. The method of claim 1, wherein the second neural network includes at least one convolutional layer, the number of neural network layers of the second neural network is less than the number of neural network layers of the offline training network, and the plurality of neural network layers to be refined includes the at least one convolutional layer in the second neural network.

6. implementing different learning algorithms in the same or different training stages, which may be online or offline training stages; or Applying different starting values ​​of the learning rate parameter in the learning algorithm, either at the same training stage or at different training stages; The method of claim 1 further comprising:

7. Implementing different learning algorithms at different training stages implementing a stochastic gradient descent (SGD) algorithm in the offline training phase and an adaptive moment estimation (ADAM) algorithm in the online training phase; Implementing different learning algorithms in the same training phase and switching from an adaptive estimation of moments (ADAM) algorithm to a stochastic gradient descent (SGD) algorithm in response to determining, during the offline training stage, that a loss produced by a loss function is below a predetermined threshold. The method of claim 6.

8. the plurality of neural network layers are refined periodically and / or in response to a trigger event; The trigger event is Video encoder behavior, Changing the video encoder configuration, or a change in video content that includes an object or object background within a video frame; The method of claim 1 , wherein the event includes one of:

9. the plurality of second data sets include at least one of coding information, video content, and information other than video frames; the encoded information includes temporal layer information; the video content includes a plurality of patches extracted from a video frame based on a region of interest; and The method of claim 1 , wherein the non-video frame information includes motion information.

10. 10. The method of claim 9, wherein the plurality of patches are selected from patches belonging to the same cluster or patches belonging to different clusters, and different patches cropped from a video frame correspond to different clusters based on expression similarity or distortion level.

11. The method of claim 1 , wherein the amount of data in the first plurality of data sets is greater than the amount of data in the second plurality of data sets.

12. Iteratively refining the plurality of neural network layers comprises: controlling the trainable structure of the network parameters by configuring a subset of the network parameters to be trainable; controlling the dynamic range of a network parameter by controlling the data type of said network parameter; Controlling network parameters of the multiple neural network layers using weight regularization The method of claim 2 , comprising at least one of:

13. Expanding or adjusting filter coefficients of an adaptive in-loop filter (ALF) of an encoder to approximate the plurality of refined network parameters; The method of claim 1 further comprising:

14. one or more processors; a memory configured to store instructions executable by the one or more processors; Equipped with 14. An apparatus for hybrid training of neural networks for video coding, wherein the one or more processors, upon execution of the instructions, are configured to perform the method of any one of claims 1 to 13.

15. 14. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform the method of any one of claims 1 to 13.

16. 14. A computer program stored on a computer-readable storage medium comprising instructions which, when executed by a processor, perform the method of any one of claims 1 to 13.

17. 1. A method for storing a bitstream having coded video information and online trained parameters, comprising: generating the bitstream by performing a method for hybrid training of neural networks for video coding according to any one of claims 1 to 13; storing the bitstream; method.

18. A method for transmitting a bitstream having coded video information and online trained parameters, comprising: generating the bitstream by performing a method for hybrid training of neural networks for video coding according to any one of claims 1 to 13; transmitting the bitstream; method.

Images