Video Enhancement Method, Apparatus, Device, and Computer-Readable Medium

Abstract

A method and apparatus for video enhancement using meta-learning for neural network-based loop filtering are provided. The method includes: receiving reconstructed video data; receiving one or more quality factors associated with the reconstructed video data; and determining a neural network-based loop filter, the neural network-based loop filter including neural network-based loop filter parameters and a plurality of layers, wherein the neural network-based loop filter parameters include shared parameters and adaptive parameters; and using the neural network-based loop filter to generate artifact-reduced enhanced video data based on the one or more quality factors and the reconstructed video data.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 176,787, filed on Apr. 19, 2021, the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0003] This application relates to the field of video processing, and particularly to a video enhancement method, a video enhancement device, a video enhancement apparatus, and a non - transitory computer - readable medium. Background Art

[0004] Video coding standards such as H.264 / Advanced Video Coding (H.264 / AVC), High - Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC) share similar (recursive) block - based hybrid prediction and / or transformation frameworks. In such standards, in order to optimize the overall efficiency, individual coding tools such as intra - frame / inter - frame prediction, integer transformation, and context - adaptive entropy coding are all strongly human - designed. These individual coding tools utilize spatio - temporal pixel neighborhoods to construct prediction signals for obtaining corresponding residuals for subsequent transformation, quantization, and entropy coding. On the other hand, neural networks extract different levels of spatio - temporal stimuli by analyzing spatio - temporal information from the receptive fields of adjacent pixels, essentially exploring highly non - linear and non - local spatio - temporal correlations. There is a need to use highly non - linear and non - local spatio - temporal correlations to explore improved compression quality.

[0005] Lossy video compression methods often suffer from artifacts in the compressed video, which seriously degrades the Quality of Experience (QoE). The allowable amount of distortion usually depends on the application, but generally, the higher the compression ratio, the greater the distortion. Compression quality can be affected by many factors. For example, the quantization parameter (QP) determines the quantization step size. The larger the QP value, the larger the quantization step size and the greater the distortion. In order to adapt to different requests of users, video coding methods need to have the ability to compress videos with different compression qualities.

[0006] Although previous methods involving deep neural networks (DNNs) have shown promising performance by enhancing the video quality of compressed videos, adapting to different QP settings is a challenge for neural network-based (NN) quality enhancement methods. For example, in previous methods, each QP value was treated as a separate task, and an NN model instance was trained and deployed for each QP value. In practice, different input channels have different QP values. For example, the chrominance and luminance components have different QP values. In such cases, previous methods require a combined number of NN model instances. When adding more different types of quality settings, the number of combined NN models becomes very large. In addition, model instances trained for a specific quality factor (QF) setting are generally not applicable to other settings. Although an entire video sequence typically has the same setting for certain QF parameters, different frames may require different QF parameters to achieve the best enhancement effect. Therefore, there is a need for methods, systems, and devices that provide flexible quality control through arbitrarily smooth QF parameter settings. Summary of the Invention

[0007] According to an embodiment, there is provided a method for video enhancement using neural network-based loop filtering with meta-learning. The method may be executed by at least one processor, and the method may include: receiving reconstructed video data; receiving one or more quality factors associated with the reconstructed video data; and using a neural network-based loop filter to generate artifact-reduced enhanced video data based on the one or more quality factors and the reconstructed video data. Wherein, the neural network-based loop filter includes neural network-based loop filter parameters, and the neural network-based loop filter parameters include shared parameters and adaptive parameters.

[0008] According to an embodiment, a video enhancement device for performing neural network-based loop filtering using meta-learning is provided. The device may include: at least one memory configured to store program code; and at least one processor configured to read the program code and operate according to the instructions of the program code. The program code may include: a first receiving code configured to cause at least one processor to receive reconstructed video data; a second receiving code configured to cause at least one processor to receive one or more quality factors associated with the reconstructed video data; and a first generating code configured to cause at least one processor to use a neural network-based loop filter to generate enhanced video data with reduced artifacts based on the one or more quality factors and the reconstructed video data. The neural network-based loop filter includes neural network-based loop filter parameters, and the neural network-based loop filter parameters include shared parameters and adaptive parameters.

[0009] According to an embodiment, a non-transitory computer-readable medium storing instructions may be provided. When executed by at least one processor, the instructions cause the at least one processor to: receive reconstructed video data; receive one or more quality factors associated with the reconstructed video data; and generate enhanced video data with reduced artifacts based on the one or more quality factors and the reconstructed video data using a neural network-based loop filter. The neural network-based loop filter includes neural network-based loop filter parameters, and the neural network-based loop filter parameters include shared parameters and adaptive parameters.

[0010] According to an embodiment, a video enhancement device is provided. The device includes: a first receiving unit configured to receive reconstructed video data; a second receiving unit configured to receive one or more quality factors associated with the reconstructed video data; a determining unit configured to determine a neural network-based loop filter, the neural network-based loop filter including neural network-based loop filter parameters and a plurality of layers, wherein the neural network-based loop filter parameters include shared parameters and adaptive parameters; and a generating unit configured to generate enhanced video data with reduced artifacts based on the one or more quality factors and the reconstructed video data using the neural network-based loop filter.

[0011] Compared with traditional loop filtering or NN-based loop filtering methods, the technical solution of this application can achieve the following beneficial technical effects: using only one QANNLF model example to implement multiple QF settings; using only one meta-NNLF model example to provide arbitrarily smooth QF settings, thereby achieving flexible quality control; providing a flexible and general framework that can accommodate various basic NNLF methods and model architectures; providing a flexible and general framework that can accommodate different types of inputs (such as frame-level or block-level, single-image or multi-image, single-channel or multi-channel) and different types of QF parameters (such as any combination of QP values for different input channels, CTU partitions, deblocking filter boundary strengths, etc.); and providing a flexible and general framework that includes various types of meta-learning algorithms and training optimization methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram of an environment that can implement the methods, apparatuses, and systems described herein according to an embodiment.

[0013] Figure 2 is Figure 1 a block diagram of example components of one or more devices.

[0014] Figure 3A and Figure 3B is a block diagram of a meta neural network loop filter (meta-NNLF) architecture for video enhancement using meta-learning according to an embodiment.

[0015] Figure 4 is a block diagram of an apparatus for a meta-NNLF model for video enhancement using meta-learning according to an embodiment.

[0016] Figure 5 is a block diagram of a training apparatus for a meta-NNLF for video enhancement using meta-learning according to an embodiment.

[0017] Figure 6A and Figure 6B are flowcharts showing a process of video enhancement using meta-NNLP according to an embodiment. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure describe methods, systems, and devices for quality-adaptive neural network-based loop filtering (QANNLF) for processing decoded video to reduce one or more types of artifacts such as noise, blur, blocking artifacts, etc. In an embodiment, a method and / or process of meta neural network-based loop filtering (meta-NNLF) is described to adaptively calculate quality-adaptive weight parameters of a neural network-based loop filtering (NNLF) model based on a current decoded video and QF of the decoded video (such as Coding Tree Unit (CTU) partitioning, QP, deblocking filter boundary strength, CU intra prediction mode, etc.). According to embodiments of the present disclosure, with any smooth QF setting (including visible settings during training and invisible settings in actual applications), only one Meta-NNLF model instance can achieve effective artifact reduction of decoded video.

[0019] Figure 1 FIG. is a diagram of an environment 100 that can implement the methods, apparatuses, and systems described herein according to an embodiment.

[0020] As Figure 1 shown, the environment 100 can include a user device 110, a platform 120, and a network 130. The devices of the environment 100 can be interconnected via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection.

[0021] The user device 110 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with the platform 120. For example, the user device 110 can include a computing device (such as a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (such as a smart phone, a wireless phone, etc.), a wearable device (such as a pair of smart glasses or a smart watch), or a similar device. In some embodiments, the user device 110 can receive information from the platform 120 and / or send information to the platform 120.

[0022] Platform 120 may include one or more devices as described elsewhere herein. In some implementations, platform 120 may include a cloud server or a group of cloud servers. In some implementations, platform 120 may be designed to be modular such that software components can be swapped in or out. In this way, platform 120 can be reconfigured easily and / or quickly for different uses.

[0023] In some implementations, as shown, platform 120 may be hosted in a cloud computing environment 122. It is noted that while the implementations described herein describe platform 120 as being hosted in cloud computing environment 122, in some implementations, platform 120 may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0024] Cloud computing environment 122 may include an environment that hosts platform 120. Cloud computing environment 122 may provide services such as computing, software, data access, storage, etc., which do not require an end user (e.g., user device 110) to know the physical location and configuration of the system and / or device that hosts platform 120. As shown, cloud computing environment 122 may include a set of computing resources 124 (collectively referred to as "computing resources 124" and individually referred to as "computing resource 124").

[0025] Computing resources 124 may include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, computing resources 124 may host platform 120. Cloud resources may include: computing instances executed in computing resources 124, storage devices provided in computing resources 124, data transfer devices provided by computing resources 124, etc. In some implementations, computing resources 124 may communicate with other computing resources 124 via a wired connection, a wireless connection, or a combination of wired and wireless connections.

[0026] As further shown in Figure 1 Computing resources 124 may include a set of cloud resources, such as one or more applications ("Application, APP") 124-1, one or more virtual machines ("Virtual Machine, VM") 124-2, virtualized storage ("Virtualized Storage, VS") 124-3, one or more hypervisors ("Hypervisor, HYP") 124-4, etc.

[0027] The applications 124-1 may include one or more software applications that may be provided to or accessed by the user device 110 and / or the platform 120. The applications 124-1 may eliminate the need to install and execute software applications on the user device 110. For example, the applications 124-1 may include software associated with the platform 120 and / or any other software that may be provided via the cloud computing environment 122. In some implementations, one application 124-1 may send information to or receive information from one or more other applications 124-1 via the virtual machine 124-2.

[0028] The virtual machine 124-2 may include a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. The virtual machine 124-2 may be a system virtual machine or a process virtual machine, depending on the use and degree of correspondence of the virtual machine 124-2 to any real machine. A system virtual machine may provide a complete system platform that supports the execution of a complete operating system ("OS"). A process virtual machine may execute a single program and may support a single process. In some implementations, the virtual machine 124-2 may execute on behalf of a user (e.g., a user device 110) and may manage the infrastructure of the cloud computing environment 122, such as data management, synchronization, or long-duration data transfer.

[0029] Virtualized storage 124-3 may include one or more storage systems and / or one or more devices that use virtualization technology within the storage system or device of the computing resource 124. In some implementations, within the context of the storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage so that the storage system can be accessed without considering the physical storage or heterogeneous structure. Separation may allow administrators of the storage system flexibility in managing storage for end users. File virtualization may eliminate the dependency between data accessed at the file level and the location where the file is physically stored. This may enable optimization of storage usage, server consolidation, and / or performance of non-disruptive file migration.

[0030] Hypervisor 124-4 can provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to execute simultaneously on a host computer such as computing resource 124. Hypervisor 124-4 can present a virtual operating platform to the guest operating systems and can manage the execution of the guest operating systems. Multiple instances of various operating systems can share virtualized hardware resources.

[0031] Network 130 may include one or more wired networks and / or wireless networks. For example, network 130 may include a cellular network (e.g., a fifth generation (5G) network, a Long-Term Evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a Public Land Mobile Network (PLMN), a Local Area Network (LAN), a Wide Area Network (WAN), a Metropolitan Area Network (MAN), a telephone network (e.g., a Public Switched Telephone Network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0032] Figure 1 The number and arrangement of the devices and networks shown are provided as examples. In fact, in addition to Figure 1 the devices and / or networks shown, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks in a different arrangement. Additionally, Figure 1 two or more of the devices shown may be implemented within a single device, or Figure 1 a single device shown may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) in environment 100 may perform one or more functions described as being performed by another set of devices in environment 100.

[0033] Figure 2 is Figure 1 a block diagram of example components of one or more of the devices.

[0034] Device 200 may correspond to user device 110 and / or platform 120. As Figure 2 shown, device 200 may include a bus 210, a processor 220, a memory 230, a storage component 240, an input component 250, an output component 260, and a communication interface 270.

[0035] The bus 210 may include components that permit communication among the components of the device 200. The processor 220 may be implemented in hardware, firmware, or a combination of hardware and software. The processor 220 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 220 may include one or more processors that can be programmed to perform functions. The memory 230 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 220.

[0036] The storage component 240 stores information and / or software related to the operation and use of the device 200. For example, the storage component 240 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid state disk), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge memory, a magnetic tape, and / or another type of non-transitory computer-readable medium, as well as a corresponding drive.

[0037] The input component 250 may include components that permit the device 200 to receive information, such as via a user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and / or a microphone). Additionally or alternatively, the input component 250 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). The output component 260 may include components that provide output information from the device 200 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0038] The communication interface 270 may include transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 270 may allow the device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0039] Device 200 can perform one or more processes described herein. Device 200 can perform these processes in response to processor 220 executing software instructions stored by a non-transitory computer-readable medium (e.g., memory 230 and / or storage component 240). Computer-readable media can be defined herein as a non-transitory memory device. The memory device can include memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

[0040] The software instructions may be read into the memory 230 and / or storage component 240 from another computer-readable medium or from another device via the communication interface 270. The software instructions stored in the memory 230 and / or storage component 240, when executed, may cause the processor 220 to perform one or more of the processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with the software instructions to perform one or more of the processes described herein. Thus, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0041] Figure 2 The number and arrangement of components shown are provided as examples. Figure 2 Device 200 may include additional components, fewer components, different components, or differently arranged components than those shown. Additionally or alternatively, one or more components of device 200 may perform one or more functions described as being performed by another group of components of device 200.

[0042] ===================================================

[0043] A video enhancement method and apparatus based on neural network-based loop filtering using meta-learning will now be described in detail.

[0044] The present disclosure presents the QANNLF mechanism to support flexible quality control through arbitrary smooth settings of the QF parameter for video compression. According to an embodiment, a meta-learning mechanism can be used to adaptively calculate quality-adaptive weight parameters of an underlying NNLF model based on a current decoded video and the QF parameter, such that a single meta-NNLF model instance can enhance the decoded video through different and arbitrary smooth QF settings.

[0045] Embodiments of the present disclosure relate to enhancing a decoded video through arbitrary smooth QF settings (including visible settings during training and invisible settings during actual applications) to achieve effective artifact reduction of the decoded video.

[0046] Generally, a video compression framework can be described as follows. Given an input video including multiple image inputs x1,... x T where each input image x t can have a size (h, w, c), can be an entire frame or a microblock (e.g., CTU) in an image frame, where h, w, and c are the height, width, and number of channels respectively. Each image frame can be a color image ((c = 3), grayscale image (c = 1), rgb + depth image (c = 4), etc.). To encode video data, in a first motion estimation step, the input image can be further divided into spatial blocks, each block is iteratively divided into smaller blocks, and a set of motion vectors m i between the current input x and a set of previously reconstructed inputs t is calculated. The subscript t represents the current t-th encoding cycle, which may not match the timestamp of the image input. In addition, can include reconstructed inputs from multiple previous encoding cycles, such that the time difference between the inputs in can vary arbitrarily. Then, in a second motion compensation step, a predicted input t can be obtained by copying corresponding pixels of a previous based on the motion vector m Then, the residual r t between the original input x and the predicted input t can be obtained. Then a quantization step can be performed, where the residual r t can be quantized. According to an embodiment, a transform is performed before quantizing the residual r t , such as DCT where the DCT coefficients of r t are quantized. The result of quantization can be the quantized Then the motion vector m t and the quantized Encoded as a bitstream and sent to the decoder. On the decoder side, the quantized can be dequantized to obtain the residual r t , and then this residual is added back to the prediction input to obtain the reconstructed input Without limitation, any method or process can be used for dequantization, such as the inverse transform of IDCT with dequantization coefficients. Additionally but not limitedly, any video compression method or coding standard can be used.

[0047] In previous methods, one or more enhancement modules can be selected to process the reconstructed to enhance the reconstructed input The one or more enhancement modules include a deblocking filter (DF), sample - adaptive offset (SAO), adaptive loop filter (ALF), cross - component adaptive loop filter (CCALF), etc.

[0048] Embodiments of the present disclosure aim to further improve the visual quality of the reconstructed input According to embodiments of the present disclosure, a QANNLF mechanism can be provided to enhance the visual quality of the reconstructed input of a video coding system The goal is to reduce artifacts such as noise, blur, and blocking effects in more specifically, the meta - NNLF method can be used to calculate using only one model instance This only one model instance can adapt to multiple and arbitrary smooth QF settings.

[0049] According to embodiments of the present disclosure, the proposed QANNLF and meta - NNLF can be used in combination with one or more of the aforementioned additional components (i.e., DF, SAO, ALF, CCALF, etc.) to improve the visual quality of the reconstructed input For example, the reconstructed input can first pass through the DF, then the Meta - NNLF, then the SAO and ALF. The order of these components can be changed arbitrarily. In one embodiment, the proposed method can be used alone as an alternative to all other components to enhance the visual quality of the reconstructed input to obtain an enhanced

[0050] The implementation of the proposed meta-NNLF mechanism can operate at the frame, tile, slice, CTU, and / or CU levels in combination with one or more of the above additional components (such as DF, SAO, ALF, CCALF, etc.). In other words, the reconstruction input when fed into the meta-NNLF can be an entire frame, but in some embodiments, the reconstruction input when fed into the meta-NNLF can also be blocks, such as tiles, slices, CTUs, or CUs.

[0051] Figure 3A And Figure 3B are block diagrams of meta-NNLF architectures 300A and 300B for video enhancement using meta-learning according to embodiments.

[0052] As Figure 3A shown, the meta-NNLF architecture 300A may include a shared NNLF NN 305, an adaptive NNLF NN 310.

[0053] As Figure 3B shown, the meta-NNLF architecture 300B may include shared NNLF layers 325 and 330 and adaptive NNLF layers 335 and 340.

[0054] In the present disclosure, the model parameters of the underlying NNLF model are divided into two parts θ s , θ a , representing the shared NNLF parameter (SNNLFP) and the adaptive NNLF parameter (Adaptive NNLF Parameter, ANNLFP), respectively. Figure 3A And Figure 3B show two embodiments of the NNLF network architecture.

[0055] In Figure 3A , the shared NNLF NN with SNNLFP θ s and the adaptive NNLF NN with ANNLFP θ a are separate individual NN modules, and these individual modules are connected to each other in sequence for network forward calculation. Here, Figure 3A shows the sequence of connecting these individual NN modules. Other sequences can be used here.

[0056] In Figure 3B , the parameters are split within the NN layer. Let θ s (i), θ a(i) represents SNNLFP and ANNLFP of the i-th layer of the NNLF model. The network will calculate the inference output based on the corresponding inputs for SNNLFP and ANNLFP, respectively, and combine these outputs (for example, by addition, concatenation, multiplication, etc.) before sending them to the next layer.

[0057] Figure 3A The implementation method can be seen as Figure 3B The following situation: Shared NNLF NN 325θ s The layers in (i) can be empty, adaptive NNLF NN 340θ a The layer in (i) can be empty. Therefore, in other embodiments, Figure 3A and Figure 3B The network structure is combined.

[0058] Figure 4 is a block diagram of an apparatus 400 for meta-NNLF for video enhancement using meta-learning during a testing phase, according to an embodiment.

[0059] Figure 4 The overall workflow of the testing phase or inference phase of meta-NNLF is shown.

[0060] Let the size be (h,w,c,d) Reconstruction input represents the input of the Meta-NNLF system, where h, w, c, and d are the height, width, number of channels, and number of frames, respectively. Therefore, The d-1 (d-1≥0) number of adjacent frames can be Used together as input To help generate enhanced These multiple adjacent frames typically include a set of previous frames Each of these It can be the frame decoded at time l or enhanced frame Let Λ t represents the QF setting, each λ l With each Associated to provide corresponding QF information, λ t Can be the current decoded frame QF settings. QF settings can include various quality control factors, such as QP value, CU intra prediction mode, CTU partition, deblocking filter boundary strength, CU motion vector, etc.

[0061] Let θ s (i) and θ a(i) represent the SNNLFP and ANNLFP of the i-th layer of the Meta-NNLF model 400 respectively. This is a general notation because for layers that can be fully shared, θ a (i) is empty. For layers that can be fully adaptive, θ s (i) can be empty. In other words, this notation can be used for Figure 3A and Figure 3B two implementation manners.

[0062] An example implementation manner of the inference workflow of the meta-NNLF model 400 for the i-th layer is provided.

[0063] Given the input and given the QF setting Λ t , the Meta-NNLF method calculates the enhanced Let f(i) and f(i + 1) represent the input and output tensors of the i-th layer of the Meta-NNLF model 400. Based on the current input f(i) and θ s (i), the SNNLFP inference unit 412 calculates the shared feature g(i) based on the shared inference function G i (f(i), θ s (i)), and this shared inference function can be modeled by performing forward calculation using the SEP in the i-th layer. Based on f(i), g(i), θ a (i) and Λ t , the ANNLFP prediction unit 414 calculates the estimated ANNLFP of the i-th layer The ANNLFP prediction unit 414 can be, for example, an NN including a convolutional layer and a fully connected layer, which predicts the updated a based on the original ANNLFP θ t the current input, and the QF setting Λ In some implementation manners, the current input f(i) can be used as the input of the ANNLFP prediction unit 414. In some other implementation manners, the shared feature g(i) can be used instead of the current input f(i). In other implementation manners, the SNNLFP loss can be calculated based on the shared feature g(i), and the gradient of the loss can be used as the input of the ANNLFP prediction unit 414. Based on the estimated ANNLFP and the shared feature g(i), the ANNLFP inference unit 416 calculates the output tensor f(i + 1) based on the ANNLFP inference function , and this inference function can be modeled by performing forward calculation using the estimated AEP in the i-th layer.

[0064] Please note that Figure 4 the workflow described in a(i) is an empty layer that can be fully shared, and the modules related to ANNLFP and f(i + 1) = g(i) can be omitted. For θ s (i) is an empty layer that can be fully adaptive, and the modules related to SNNLFP and g(i) = f(i) can be omitted.

[0065] Assume that the meta-NNLF model 400 has a total of N layers, and the output of the last layer can be enhanced

[0066] Note that the meta-NNLF framework allows any smooth QF setting for flexible quality control. In other words, the above processing workflow will be able to enhance the quality of the decoded frames through any smooth QF setting, which can be included in the training phase or not included in the training phase.

[0067] In one case, when the ANNLFP prediction unit 414 performs predictions through a predefined set of QF settings with or without considering the input f(i), the meta-NNLF model simplifies to a multi-QF NNLF model, which uses one NNLF model instance to adapt to the enhancement of multiple predefined QF settings. Of course, other simplified specific cases can also be covered here.

[0068] Figure 5 is a block diagram of a training device 500 for meta-NNLF that uses meta-learning for video enhancement during the training phase according to an embodiment.

[0069] As Figure 5 shown, the training device 500 may include a task sampler 510, an inner-loop loss generator 520, an inner-loop update unit 530, a meta-loss generator 540, a meta-update unit 550, and a weight update unit 560.

[0070] The training process aims to learn the SNNLFP θ of the Meta-NNLF model 400 s (i) and ANNLFP θ a (i), i = 1, …, N and the ANNLFP prediction NN (model parameters are represented as Φ).

[0071] In an embodiment, the Model-Agnostic Meta-Learning (MAML) mechanism can be used for training purposes. Figure 5 gives an example workflow of the meta-training framework. Other meta-training algorithms can be used here.

[0072] For training, there can be a set of training data where each corresponds to the training QF setting, and there are K training QF settings in total (therefore there are K training datasets). For training, there can be q qp Different training QP values, q CTU different training CTU partitions, etc., and there can be a finite number K = q qp ×q CTU ×… different training QF settings. Therefore, each training dataset There can be associated with each of these QF settings. Additionally, there can be a set of validation data Each of these There are a total of P validation QF settings corresponding to the validation QF settings. The validation QF settings can include values different from those in the training set. The validation QF settings can also have the same values as those in the training set.

[0073] The overall training goal is to learn a meta-NNLF model that can be widely applied to all values of the QF setting (both trained and unseen in the future). It is assumed that the NNLF tasks with the QF setting can be drawn from the task distribution P(Λ). To achieve the above training goal, the loss of the learned meta-NNLF model can be minimized in all training datasets for all training QF settings.

[0074] The MAML training process can have an outer loop and an inner loop for gradient-based parameter updates. For each outer loop iteration, the task sampler 510 first samples a set of K′ training QF settings (K′≤K). Then, for each sampled training QF setting Λ i , the task sampler 510 is trained from the dataset Training dataset Furthermore, the task sampler 510 samples a set of P′ (P′≤P) verification QF settings, and for each sampled verification QF setting Λ j , from the validation dataset Validation dataset Then for each sampled data Can be based on the current parameter Θ s 、Θ a and Φ perform meta-NNLF forward calculation, and then the inner loop loss generator 520 calculates the accumulated inner loop loss

[0075]

[0076] Loss Function Can include real images With enhanced output The distortion loss between: and some other regularization losses (e.g., auxiliary losses that differentiate the intermediate network outputs for different QF factors). Any distortion metric such as MSE, MAE, SSIM, etc. can be used as

[0077] Then, based on the inner loop loss Given a step size α si and α ai As Λ i The inner loop update unit 530 calculates the parameter updates specific to the update task.

[0078] as well as

[0079]

[0080] Cumulative inner ring loss Gradient and gradient They can be used to calculate the adaptive parameters and An updated version of .

[0081] The meta-loss generator 540 then computes the extrinsic meta-objective or loss for all sampled validation quality control parameters:

[0082] as well as

[0083]

[0084] in Can be used as parameter Φ is calculated based on the meta-NNLF forward pass for the decoded frame Calculated loss, QF is set to Λ j Given a step size β aj and β sj As Λ j The meta-updating unit 550 updates the model parameters to:

[0085] as well as

[0086]

[0087] In some implementations, Θ may not be updated in the inner loop. s , that is, α si =0, Not updating helps stabilize the training process.

[0088] As for the parameters Φ of the ANNLFP prediction NN, the weight updating unit 560 updates them in a conventional training manner. That is, based on the training and validation data Based on the current θ s ,θ a , Φ, all samples can be calculated Loss and all samples of And the gradients of all these losses can be accumulated (e.g., added) to perform parameter updates on Φ via conventional backpropagation.

[0089] The embodiments of the present disclosure are not limited to the above-mentioned optimization algorithms or loss functions for updating these model parameters. Any optimization algorithm or loss function known in the art for updating these model parameters can be used.

[0090] For the case where the ANNLFP prediction unit 414 of the Meta-NNLF model performs prediction only on a predefined set of training QF settings, the validation QF settings will be the same as the training QF settings. The same MAML training process can be used to train the simplified meta-NNLF model described above (i.e., a multi-QF setting NNLF model that uses one model instance to accommodate the compression effects of multiple predefined bit rates).

[0091] Embodiments of the present disclosure allow adapting to multiple QF settings using only one QANNLF model instance using meta-learning. In addition, embodiments of the present disclosure enable adapting to different types of inputs (e.g., frame level or block level, single image or multiple images, single channel or multiple channels) and different types of QF parameters (e.g., any combination of QP values of different input channels, CTU partitions, deblocking filter boundary strengths, etc.) using only one instance of the meta-NNLF model.

[0092] Figure 6A is a flow chart of a method 600A for video enhancement based on neural network-based loop filtering using meta-learning, according to an embodiment.

[0093] like Figure 6A As shown, at operation 610 , method 600A may include: receiving reconstructed video data; and receiving one or more quality factors associated with the reconstructed video data.

[0094] In some embodiments, the reconstructed video data may include multiple reconstructed input frames, and the methods described herein may be applied to a current frame among the multiple reconstructed input frames. In some embodiments, the reconstructed input frame may be further decomposed and used as input to the meta-NNLF model.

[0095] In some embodiments, the one or more quality factors associated with the reconstructed video data may include at least one of: coding tree unit partition, quantization parameter, deblocking filter boundary strength, coding unit motion vector, and coding unit prediction mode.

[0096] In some embodiments, reconstructed video data can be generated based on a bitstream including decoded quantized video data and motion vector data. For example, generating the reconstructed video data may include receiving a video data stream including quantized video data and motion vector data. Generating the reconstructed video data may then include dequantizing the quantized data stream using an inverse transform to obtain a recovered residual; and generating the reconstructed video data based on the recovered residual and the motion vector data.

[0097] Method 600A may also include generating enhanced video data with reduced artifacts based on one or more quality factors and the reconstructed video data using a neural network-based loop filter, wherein the neural network-based loop filter includes neural network-based loop filter parameters, and the neural network-based loop filter parameters include shared parameters and adaptive parameters. As an example, the Meta-NNLF model 400 can be used on multiple layers to generate enhanced video data with reduced artifacts based on the one or more quality factors and the reconstructed video data. In some embodiments, generating the enhanced video data using the neural network-based loop filter may include operations 615 to 630. In some preferred embodiments, generating the enhanced video data using the neural network-based loop filter may include repeating operations 615 to 630 for multiple layers of the meta-NNLF model, such as the meta-NNLF model 400.

[0098] Thus, for each of the plurality of layers in the neural network-based loop filtering using meta-learning, at 615, the method may include: generating shared features based on outputs from a previous layer using a first shared neural network loop filter having first shared parameters. As an example, the SNNLFP inference unit 412 may use a first shared neural network loop filter having first shared parameters. The first shared neural network loop filter generates the shared features g(i) of the i-th layer based on the output from the i-1-th layer.

[0099] At 620, the method may include: using a prediction neural network to calculate an estimated adaptive parameter based on the output from the previous layer, the shared features, the first adaptive parameter from the first adaptive neural network loop filter, and one or more quality factors. As an example, the ANNLFP prediction unit 414 may use a prediction neural network to calculate an estimated adaptive parameter based on the output from the i-1 layer, the shared features g(i), the first adaptive parameter from the first adaptive neural network loop filter, and the one or more quality factors. The first adaptive parameter and one or more quality factors Λ are used to calculate the estimated adaptive parameter

[0100] In some embodiments, training the prediction network may include generating a first loss for training data corresponding to the one or more quality factors and a second loss for validation data corresponding to the one or more quality factors based on the one or more quality factors, the first shared parameter, the first adaptive parameter, and the prediction parameters of the prediction neural network, and then updating the prediction parameters based on the gradient of the generated first loss and the generated second loss.

[0101] At 625, the method may include generating an output of the current layer based on the shared features and the estimated adaptive parameters. As an example, the ANNLFP inference unit 416 may generate an output of the current layer based on the shared features g(i) and the estimated adaptive parameters To generate the output f(i+1).

[0102] At 630 , the method may include generating enhanced video data based on an output of a last layer of a neural network-based loop filter.

[0103] In some embodiments, method 600A may include enhancement post-processing, including applying at least one of a deblocking filter, an adaptive loop filter, a sample adaptive offset, and a cross-component adaptive loop filter to the enhanced video data. As an example, the enhancement post-processing may be performed after operation 630.

[0104] In some embodiments, method 600A may include pre-enhancement processing, including applying at least one of a deblocking filter, an adaptive loop filter, a sample adaptive offset, and a cross-component adaptive loop filter to the enhancement video data before generating the enhancement video data. As an example, the pre-enhancement processing may be performed before operation 620.

[0105] Figure 6B is a flow chart of a method 600B for adaptive neural image compression with rate control via meta-learning, according to an embodiment.

[0106] like Figure 6B As shown, at 650 , method 600B may include: receiving reconstructed video data; and receiving one or more quality factors associated with the reconstructed video data.

[0107] In some embodiments, the reconstructed video data may include a plurality of reconstructed input frames, and the methods described herein may be applied to a current frame among the plurality of reconstructed input frames. In some embodiments, the reconstructed input frames may be further decomposed and used as inputs to the meta-NNLF model.

[0108] In some embodiments, one or more quality factors associated with the reconstructed video data may include at least one of a coding tree unit partition, a quantization parameter, a deblocking filter boundary strength, a coding unit motion vector, and a coding unit prediction mode.

[0109] In some embodiments, the reconstructed video data may be generated from a bitstream including decoded quantized video data and motion vector data. As an example, generating the reconstructed video data may include: receiving a video data stream including quantized video data and motion vector data. Then, generating the reconstructed video data may include: dequantizing the quantized data stream using an inverse transform to obtain a recovered residual; and generating the reconstructed video data based on the recovered residual and the motion vector data.

[0110] Method 600B may further include: using a neural-network-based loop filter to generate artifact-reduced enhanced video data based on one or more quality factors and the reconstructed video data, wherein the neural-network-based loop filter includes neural-network-based loop filter parameters, and the neural-network-based loop filter parameters include shared parameters and adaptive parameters. As an example, the Meta-NNLF model 400 may be used over multiple layers to generate artifact-reduced enhanced video data based on one or more quality factors and the reconstructed video data. In some embodiments, generating the enhanced video data using the neural-network-based loop filter may include operations 655 to 670. In some preferred embodiments, generating the enhanced video data using the neural-network-based loop filter may include: repeating operations 655 to 670 for multiple layers of a meta-NNLF model such as the meta-NNLF model 400.

[0111] At 655, the method may include: using a first shared neural-network loop filter having a first shared parameter to generate a first shared output based on a previous combined output from a previous layer. At 660, the method may include: using a first adaptive neural-network loop filter having a first adaptive parameter to generate a first adaptive output based on a previous combined output from a previous layer. At 665, the method may include: combining the first shared output and the first adaptive output to generate a first combined output. At 670, the method may further include: generating the enhanced video data based on the combined output of the last layer of the neural-network-based loop filter.

[0112] The proposed methods can be used alone or in any combination in any order. Additionally, each of the methods (or embodiments), encoders, and decoders can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0113] In some implementations, Figure 6A and Figure 6B one or more processing blocks of can be executed by platform 120. In some implementations, Figure 6A and Figure 6B one or more processing blocks of can be executed by a device or a set of devices (such as user device 110) that is independent of platform 120 or includes platform 120.

[0114] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the exact forms disclosed. Modifications and variations are possible in light of the above disclosure, or may be obtained from practice of the implementations.

[0115] As used herein, the term "component" is intended to be broadly construed as hardware, firmware, or a combination of hardware and software.

[0116] It will be apparent that the systems and / or methods described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual special control hardware or software code used to implement these systems and / or methods does not limit the implementations. Accordingly, the operations and behavior of the systems and / or methods are described herein without reference to specific software code, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0117] Even if combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or not disclosed in the specification. Although each of the dependent claims listed below may directly refer to only one claim, the disclosure of possible implementations may include each dependent claim combined with every other claim in the claim group.

[0118] No element, act, or instruction used herein shall be construed as critical or essential unless expressly described as such. Additionally, as used herein, terms without a quantifier modifier ("a" and "an") are intended to include one or more items and may be used interchangeably with "one or more." Further, as used herein, the term "group" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with "one or more." Where only one item is intended, the term "one" or similar language is used. Additionally, as used herein, the terms "having," "comprising," "containing," etc. are intended to be open-ended terms. Further, unless expressly stated otherwise, the phrase "based on" is intended to mean "at least partially based on."

Claims

1. A method for video enhancement of loop filtering based on a neural network using meta - learning, characterized in that, The method is executed by at least one processor, and the method includes: Receiving reconstructed video data; Receiving one or more quality factors associated with the reconstructed video data; Determining a neural network-based loop filter, the neural network-based loop filter including neural network-based loop filter parameters and multiple layers, wherein the neural network-based loop filter parameters include shared parameters and adaptive parameters; and Using the neural network-based loop filter to generate artifact-reduced enhanced video data based on the one or more quality factors and the reconstructed video data; Using the neural network-based loop filter to generate the enhanced video data includes: For each of the multiple layers in the neural network-based loop filter: Using a first shared neural network loop filter with first shared parameters to generate shared features based on the output from the previous layer; Using a prediction neural network to calculate estimated adaptive parameters based on the output from the previous layer, the shared features, first adaptive parameters from a first adaptive neural network loop filter, and the one or more quality factors; and Generating the output of the current layer based on the shared features and the estimated adaptive parameters; and Generating the enhanced video data according to the output of the last layer of the neural network-based loop filter.

2. The method according to claim 1, characterized in that Training the prediction neural network includes: Generating a first loss of training data corresponding to the one or more quality factors and a second loss of validation data corresponding to the one or more quality factors based on the one or more quality factors, the first shared parameters, the first adaptive parameters, and prediction parameters of the prediction neural network; and Updating the prediction parameters based on the gradients of the generated first loss and the generated second loss.

3. The method according to claim 1, characterized in that One or more quality factors associated with the reconstructed video data include at least one of the following parameters: coding tree unit partition, quantization parameter, deblocking filter boundary strength, coding unit motion vector, and coding unit prediction mode.

4. The method according to claim 1, wherein Training the neural network-based loop filter includes: Generating an inner loop loss of training data corresponding to the one or more quality factors based on the one or more quality factors, the first shared parameters, and the first adaptive parameters; First updating the first shared parameters and the first adaptive parameters based on the gradient of the generated inner loop loss; Generating a meta-loss of validation data corresponding to the one or more quality factors based on the one or more quality factors, the first shared parameters after the first update, and the first adaptive parameters after the first update; and Second updating the first shared parameters after the first update and the first adaptive parameters after the first update based on the gradient of the generated meta-loss.

5. The method according to any one of claims 1 to 4, characterized in that Generating the received reconstructed video data includes: Receiving a video data stream including quantized video data and motion vector data; Dequantizing the quantized video data stream using an inverse transform to obtain a recovered residual; and Generate the reconstructed video data based on the recovered residuals and the motion vector data.

6. The method according to any one of claims 1-4, characterized in that The method further includes enhancement post-processing, which includes applying at least one of the following to the enhanced video data: a deblocking filter, an adaptive loop filter, sample adaptive offset, and a cross-component adaptive loop filter.

7. The method according to any one of claims 1 to 4, characterized in that, Before generating the enhanced video data, the method includes pre-enhancement processing, which includes applying at least one of the following to the enhanced video data: a deblocking filter, an adaptive loop filter, sample adaptive offset, and a cross-component adaptive loop filter.

8. The method according to any one of claims 1 to 4, characterized in that The reconstructed video data includes a plurality of reconstructed input frames; and wherein the method is applied to a current frame among the plurality of reconstructed input frames.

9. The method according to any one of claims 1 to 4, characterized in that, Generating the enhanced video data using the neural network-based loop filter includes: For each of the plurality of layers in the neural network-based loop filter: Using a first shared neural network loop filter with first shared parameters to generate a first shared output based on the previous combined output from the previous layer; Using a first adaptive neural network loop filter with first adaptive parameters to generate a first adaptive output based on the previous combined output from the previous layer; and Combining the first shared output and the first adaptive output to generate a first combined output; and Generating the enhanced video data according to the combined output of the last layer of the neural network-based loop filter.

10. The method according to claim 9, wherein The combination includes at least one of addition, concatenation, or multiplication.

11. A video enhancement device, characterized in that, The apparatus includes: At least one memory configured to store program code; and At least one processor configured to read the program code to execute the method according to any one of claims 1 to 10.

12. A non-transitory computer-readable medium storing instructions, characterized in that, The instructions, when executed by at least one processor, cause the at least one processor to execute the method according to any one of claims 1 to 10.

13. A video enhancement device, characterized in that, The device includes: A first receiving unit configured to receive reconstructed video data; A second receiving unit configured to receive one or more quality factors associated with the reconstructed video data; A determining unit configured to determine a neural network-based loop filter, the neural network-based loop filter including neural network-based loop filter parameters and a plurality of layers, wherein the neural network-based loop filter parameters include shared parameters and adaptive parameters; and A generating unit configured to use the neural network-based loop filter to generate artifact-reduced enhanced video data based on the one or more quality factors and the reconstructed video data; Generating the enhanced video data using the neural network-based loop filter includes: For each of the plurality of layers in the neural network-based loop filter: Using a first shared neural network loop filter with first shared parameters to generate shared features based on the output from the previous layer; Using a prediction neural network, calculate an estimated adaptive parameter based on the output from the previous layer, the shared feature, a first adaptive parameter from a first adaptive neural network loop filter, and the one or more quality factors; and Generate an output of the current layer based on the shared feature and the estimated adaptive parameter; and Generate the enhanced video data based on the output of the last layer of the neural network-based loop filter.

Images