Video processing method and apparatus

Abstract

This invention discloses a video processing method and apparatus. According to one embodiment, the video processing method includes the following steps: receiving a video comprising multiple temporal portions; receiving first model parameters corresponding to a first neural network used for overall processing of the video; receiving residuals between the first model parameters and multiple second model parameters corresponding to multiple second neural networks used to process the multiple temporal portions respectively; and performing at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on the video based on the residuals.

Description

Technical Field

[0001] The following embodiments relate to a video processing method and apparatus. Background Technology

[0002] Convolutional Neural Networks (CNNs) for Super-Resolution (SR) have demonstrated good performance and high fidelity in image restoration. In the field of computer vision, much research has been conducted on CNN-based SR.

[0003] Through SR, video is efficiently encoded and transmitted at low spatial resolution, and at the receiving end, a CNN-based SR method can be used to restore the video at high spatial resolution with high fidelity.

[0004] Furthermore, content-adaptive learning and transfer of neural network model parameters can improve SR performance to achieve higher recovery fidelity. In this case, efficient transfer of neural network parameters is crucial. Summary of the Invention

[0005] Technical problems to be solved

[0006] The following embodiments can provide a video processing technology.

[0007] Technical methods for solving problems

[0008] A video processing method according to an embodiment includes the following steps: receiving a video comprising multiple temporal portions; receiving first model parameters corresponding to a first neural network for overall processing of the video; receiving residuals between the first model parameters and multiple second model parameters corresponding to multiple second neural networks for processing the multiple temporal portions respectively; and performing at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on the video based on the residuals.

[0009] The first neural network and at least one of the plurality of second neural networks may include a light-weight residual dense block containing at least one convolutional layer.

[0010] The first neural network and at least one of the plurality of second neural networks may include a plurality of lightweight residual dense blocks configured in a cascade structure.

[0011] The lightweight residual dense block may include: a depth-wise separable convolution layer; or a 1×1×d convolution layer.

[0012] The depth-separable convolutional layer may include: a depthwise convolutional layer; a pointwise convolutional layer; and a non-linear activation function.

[0013] The execution steps may include the following steps: restoring the plurality of second model parameters by adding the residual to the first model parameters; and performing at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on the plurality of time portions based on the plurality of second model parameters.

[0014] The video processing method may further include the following steps: receiving spatial region segmentation information and category information of the segmented spatial regions on frames included in the plurality of time segments.

[0015] The execution steps may include the following steps: dividing the frame into multiple spatial regions based on the spatial region segmentation information and the category information; and performing at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on the video based on the multiple spatial regions and the category information.

[0016] A video processing apparatus according to one embodiment includes: a receiver that receives a video comprising a plurality of temporal portions and receives first model parameters corresponding to a first neural network for processing the video as a whole, and receives residuals between the first model parameters and a plurality of second model parameters corresponding to a plurality of second neural networks for processing the plurality of temporal portions respectively; and a processor that performs at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on the video based on the residuals.

[0017] The first neural network and at least one of the plurality of second neural networks may include a light-weight residual dense block containing at least one convolutional layer.

[0018] The first neural network and at least one of the plurality of second neural networks may include a plurality of lightweight residual dense blocks configured in a cascade structure.

[0019] The lightweight residual dense block may include: a depth-wise separable convolution layer; or a 1×1×d convolution layer.

[0020] The depth-separable convolutional layer may include: a depthwise convolutional layer; a pointwise convolutional layer; and a non-linear activation function.

[0021] The processor can segment the frames included in the video into multiple regions, recover the multiple second model parameters by adding the residuals to the first model parameters, and perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on the multiple time segments based on the multiple second model parameters.

[0022] The receiver can also receive spatial region segmentation information and category information of the segmented spatial regions on frames included in the plurality of time segments.

[0023] The processor can segment the frame into multiple spatial regions based on the spatial region segmentation information and the category information, and perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on the video based on the multiple spatial regions and the category information.

[0024] According to another embodiment of the video processing method, which performs at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal, the method includes the following steps: receiving a video comprising multiple temporal portions and model information related to a neural network for processing the video; obtaining first model parameters corresponding to a first neural network for processing the video as a whole based on the model information; receiving multiple second model parameters corresponding to multiple second neural networks for processing the multiple temporal portions respectively based on the model information; calculating residuals between the first model parameters and the multiple second model parameters; and transmitting the video and the residuals.

[0025] The video processing method may further include the following steps: generating spatial region segmentation information by segmenting frames included in the plurality of time segments; and generating category information of the segmented spatial regions based on the signal characteristics of the segmented spatial regions.

[0026] The transmission step may further include the following steps: compressing and transmitting at least one of the video, the residual, the spatial region segmentation information, and the category information.

[0027] The video processing method may further include the following step: learning the plurality of second neural networks to minimize the residuals.

[0028] A video processing apparatus according to another embodiment is used to perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal, comprising: a receiver that receives a video including multiple temporal portions and model information related to a neural network for processing the video; and a processor that acquires first model parameters corresponding to a first neural network for processing the video as a whole based on the model information, receives multiple second model parameters corresponding to multiple second neural networks for processing the multiple temporal portions respectively based on the model information, calculates residuals between the first model parameters and the multiple second model parameters, and transmits the video and the residuals.

[0029] The processor can generate spatial region segmentation information by segmenting frames included in the plurality of time segments, and generate category information of the segmented spatial regions based on the signal characteristics of the segmented spatial regions.

[0030] The processor can compress and transmit at least one of the video, the residual, the spatial region segmentation information, and the category information.

[0031] The processor can learn the multiple second neural networks to minimize the residuals. Attached Figure Description

[0032] Figure 1a An exemplary block diagram of a video processing system according to one embodiment is shown.

[0033] Figure 1b To display Figure 1a An exemplary block diagram of a video processing device at the transmitting end.

[0034] Figure 1c To display Figure 1a An exemplary block diagram of a video processing device at the receiving end.

[0035] Figure 2 Show Figure 1b An example of the operation of the video processing device at the transmitting end.

[0036] Figure 3 Show Figure 1b Another example of the operation of the video processing device at the transmitting end.

[0037] Figure 4 Shown by Figure 1a Examples of neural network structures used in the video processing devices at the transmitting and receiving ends.

[0038] Figure 5 Show Figure 4 An example of the structure of a lightweight residual dense block (LwRDB).

[0039] Figure 6 Showing according to Figure 1a The peak signal-to-noise ratio (PSNR) of the video processing method.

[0040] Figure 7 The PSNR and bit size reduction rate are shown based on the hyperparameters.

[0041] Figure 8 Show Figure 1b The sequence of operations of the video processing device at the transmitting end.

[0042] Figure 9 Show Figure 1c The sequence of operations of the video processing device at the receiving end. Detailed Implementation

[0043] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, various modifications can be made to the embodiments, and the scope of the present invention is not limited or restricted by the embodiments. All variations, equivalents, or substitutions of the embodiments are included within the scope of the claims.

[0044] The terminology used in these embodiments is for illustrative purposes only and is not intended to limit the scope of the embodiments. Unless otherwise specified in the text, singular expressions include plural meanings. In this specification, terms such as "comprising" or "having" are used to indicate the presence of the features, numbers, steps, operations, constituent elements, accessories, or combinations thereof described in the specification, and do not exclude the presence of one or more other features, numbers, steps, operations, constituent elements, accessories, or combinations thereof, or additional functions.

[0045] Unless otherwise defined, all terms used herein, including technical or scientific terms, shall have the ordinary meaning as understood by one of ordinary skill in the art. Terms that are commonly used and are identical to their dictionary definitions shall be understood to have a meaning consistent with the general content of the relevant art, and shall not be overly idealized or interpreted as having a formal meaning unless expressly stated in this application.

[0046] Furthermore, in the description with reference to the accompanying drawings, identical constituent elements are assigned the same reference numerals, regardless of the drawing references, and repeated descriptions of these elements are omitted. In describing embodiments, detailed descriptions of relevant well-known technologies are omitted when it is determined that such detailed descriptions would unnecessarily obscure the embodiments.

[0047] Furthermore, when describing the constituent elements of the embodiments, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish one constituent element from other constituent elements and are not used to limit the nature or order of the corresponding constituent elements. For example, a first constituent element may be referred to as a second constituent element, and similarly, a second constituent element may be referred to as a first constituent element. In addition, it should be understood that when the specification describes a constituent element as "connected," "joined," or "contacting" another constituent element, a third constituent element may be "connected," "joined," or "contacting" between the first and second constituent elements, although the first constituent element may be directly connected, joined, or contacting the second constituent element.

[0048] When a constituent element has a common function with a constituent element in one embodiment, the same name is used to describe that constituent element in other embodiments. Unless otherwise stated, the description of one embodiment is applicable to other embodiments, and detailed descriptions of repetitive content are omitted.

[0049] Figure 1 is an exemplary block diagram showing a video processing system according to an embodiment. Figure 1b To illustrate an exemplary block diagram of the video processing apparatus at the transmitting end of Figure 1, Figure 1c An exemplary block diagram of the video processing apparatus at the receiving end of Figure 1 is shown.

[0050] Reference Figures 1a to 1c The video processing system 10 can process images or videos. Images include images of objects formed by the refraction or reflection of light, which may mean using lines or colors to represent the shape of an object. Videos may include frames. Furthermore, a frame may include multiple pixels.

[0051] A video can include multiple temporal portions. A temporal portion can correspond to a set of frames within a specific temporal duration, that is, it can correspond to a video segment (each video segment is typically a part of a specific time period in a longer recording). In other words, a video can include scenes with multiple temporal durations according to a timeline.

[0052] The video processing system 10 can perform at least one of the following on video, frames included in the video, or images: super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

[0053] Super-resolution can refer to video processing that transforms an input image with low spatial resolution into an input image with high spatial resolution. Video processing system 10 can perform super-resolution on video, frames included in the video, or images using neural networks.

[0054] Neural networks (or artificial neural networks) can include statistical learning algorithms that simulate biological neurons in machine learning and cognitive science. A neural network can be a general model that enables artificial neurons (nodes) forming a network through synaptic connections to learn and change the strength of synaptic connections, thereby solving problems.

[0055] Neural networks can include deep neural networks.Neural networks can include Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Perceptrons, Feed Forward Networks (FFs), Radial Basis Networks (RBFs), Deep Feed Forward Networks (DFFs), Long Short Term Memory Networks (LSTMs), Gated Recurrent Units (GRUs), Auto Encoders (AEs), Variational Auto Encoders (VAEs), Denoising Auto Encoders (DAEs), Sparse Auto Encoders (SAEs), Markov Chains (MCs), Hopfield Networks (HNs), Boltzmann Machines (BMs), Restricted Boltzmann Machines (RBMs), and Deep Belief Networks (DBNs). Networks include Deep Convolutional Networks (DCN), Deconvolutional Networks (DN), Deep Convolutional Inverse Graphics Networks (DCIGN), Generative Adversarial Networks (GAN), Liquid State Machines (LSM), Extreme Learning Machines (ELM), Echo State Networks (ESN), Deep Residual Networks (DRN), Differentiable Neural Computers (DNC), Neural Turing Machines (NTM), Capsule Networks (CN), Kohonen Networks (KN), and Attention Networks (AN).

[0056] The video processing system 10 may include a video processing device 100 (hereinafter referred to as video processing device 100) at the transmitting end and a video processing device 500 (hereinafter referred to as video processing device 500) at the receiving end.

[0057] The video processing apparatus 100 can process the video based on video and model information, and send the processing results to the video processing apparatus 500. The processing results may include the residuals of a neural network model used to perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal based on video and model information, spatial region segmentation information of the frames included in the video, and category information of the segmented spatial regions.

[0058] Model information can include information about the type and structure of the neural network, as well as the model parameters of the pre-trained neural network.

[0059] In other words, the video processing apparatus 100 can generate the residual of the neural network model, spatial region segmentation information of the frames included in the video, and category information of the segmented spatial regions based on the video and model information, and send them to the video processing apparatus 500. Furthermore, the video processing apparatus 100 can compress the video and send it to the video processing apparatus 500.

[0060] The video processing apparatus 500 can process the video based on the processing results received from the video processing apparatus 100. The video processing apparatus 500 can perform at least one of the following on the video (or compressed video): super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal, based on the residual of the neural network model, spatial region segmentation information of the frames included in the video, and category information of the segmented spatial regions.

[0061] The video processing apparatus 100 can learn a neural network. The video processing apparatus 100 can compress information from the learned neural network (e.g., model parameters of the neural network) and send it to the video processing apparatus 500. Information about the neural network may include model parameters, structure, or weights of the neural network. For example, model parameters may include filter parameters of a convolutional neural network.

[0062] The video processing apparatus 100 can send information about a neural network model to any video processing apparatus 500. The video processing apparatus 500 can perform at least one of the following based on the information from the neural network model: super-resolution, reverse or inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

[0063] The video processing apparatus 100 can compress images or videos. Furthermore, the video processing apparatus 100 can send the compressed images or videos to the video processing apparatus 500. The video processing apparatus 500 can perform at least one of the following on the images or videos based on pre-stored neural network information and / or received neural network information: super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

[0064] The video processing apparatus 100 can be located at the transmitting end. The video processing apparatus 100 can save bandwidth by transmitting video with reduced spatial and / or temporal resolution at the transmitting end, and perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on the reduced resolution video using information from a neural network pre-stored at the receiving end and / or information from the received neural network.

[0065] like Figure 1b and Figure 1c As shown, video processing apparatus 100 may include receiver 110 and processor 130, and video processing apparatus 500 may include receiver 510 and processor 530. Video processing apparatus 100 may also include memory 150, and video processing apparatus 500 may also include memory 550.

[0066] Receiver 100 can receive images, videos, or frames included in a video, as well as model information for processing the video. For example, receiver 110 can receive video including multiple temporal portions and model information related to a neural network used for processing the video. Receiver 110 can output the received images and / or video to processor 130 and memory 150.

[0067] Receiver 510 can receive video (or compressed video) comprising multiple time segments. Receiver 510 can receive residuals between first model parameters corresponding to first neural network used to process the entire video and multiple second model parameters corresponding to multiple second neural networks used to process the first model parameters and the multiple time segments respectively.

[0068] Receiver 510 can receive spatial region segmentation information of frames included in multiple time segments and category information of the segmented spatial regions.

[0069] The receiver 510 can output the received video, first model parameters, residual, spatial region segmentation information and category information to the processor 530 and the memory 550.

[0070] Processors 130 and 530 can process data stored in memory 150 and memory 550, respectively. Processors 130 and 530 can execute computer-readable code (e.g., software) stored in memory 150 and memory 550, as well as instructions triggered by processors 130 and 530.

[0071] "Processor 130 and processor 530" can be data processing devices implemented in hardware, wherein the hardware includes circuitry having a physical structure that performs desired operations. For example, the desired operations may include codes or instructions contained in a program.

[0072] For example, a data processing device implemented as hardware may include at least one of a microprocessor, a central processing unit (CPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), and a field-programmable gate array (FPGA).

[0073] The processor 130 can obtain first model parameters corresponding to a first neural network used to process the entire received video based on model information. In other words, the first neural network can refer to a neural network that learns for the entire scene.

[0074] Furthermore, the processor 130 can obtain first model parameters from the first neural network to process all general videos, not just the received videos. In other words, the first neural network can refer to a neural network learned for general video scenarios.

[0075] The processor 130 can acquire multiple second model parameters based on model information, corresponding to multiple second neural networks used to process multiple temporal segments respectively. The number of multiple temporal segments can be the same as the number of video segments included within a certain temporal duration of the video (each video segment is typically a part of a specific time period of a longer recording). Therefore, the number of second neural networks can be equal to or less than the number of video segments included in the video. For example, in the case of less, two or more video segments can correspond to one and the same second neural network model.

[0076] The processor 130 can learn a first neural network and multiple second neural networks. The processor 130 can learn the first neural network and multiple second neural networks based on multiple segmented spatial regions and the residuals between the first model parameters and multiple second model parameters, enabling it to perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on images and / or videos.

[0077] The processor 130 can use any loss function to learn a first neural network and multiple second neural networks.

[0078] For example, processor 130 can learn multiple second neural networks to minimize the residuals between the parameters of the first model and the parameters of the multiple second models. (Refer to...) Figure 2 Describe the loss function in detail.

[0079] At least one of the first neural network and a plurality of second neural networks may include a lightweight residual dense block (LwRDB) containing at least one convolutional layer.

[0080] For example, the first neural network and at least one of the plurality of second neural networks may include a plurality of lightweight residual dense blocks configured in a cascade structure.

[0081] The loss function of the first neural network can be determined based on the video, on which at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal is performed. For example, the loss function of the first neural network can be the L1 loss between the ground truth frame and the frame on which at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal has been performed.

[0082] The loss function of the second neural network may include an L1 loss function between frames and ground truth frames that perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal for the corresponding time portion of the video, and a loss function for minimizing the difference between the parameters of the first neural network based on the pre-learned whole video and the parameters of multiple second models.

[0083] That is, the second model parameters of each second neural network can be learned to minimize the value of the model residual to be transmitted, while optimally performing at least one of the following processing methods: super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal. (Refer to...) Figure 2 Describe in detail the loss functions of the first neural network and the second neural network.

[0084] Lightweight residual dense blocks can include depth-wise separable convolutional layers or 1×1×d convolutional layers. Here, d can represent the number of convolutional channels (or feature maps) in the depth direction. Depth-wise separable convolutional layers can include depthwise convolutional layers, pointwise convolutional layers, and non-linear activation functions.

[0085] For example, nonlinear activation functions may include the Rectified Linear Unit (ReLU), Leaky ReLU, Parametric ReLU, Softmax, Sigmoid, Tanh, Exponential Linear Unit (ELU), and SoftPlus function.

[0086] The processor 130 can calculate the residuals between the first model parameters and a plurality of second model parameters, and send the video and the residuals. For example, the processor 130 can send the video and the residuals to the video processing device 500.

[0087] Processor 130 can generate spatial region segmentation information by segmenting frames included in multiple time segments. The spatial region segmentation information may include information related to the shape of the segmented frames. For example, the spatial region segmentation information may include the width, location, or number of segmented regions.

[0088] Processor 130 can generate category information for the segmented spatial regions based on their signal characteristics. For example, processor 130 can segment multiple regions based on the image complexity of multiple segmented spatial regions and determine the categories of the segmented spatial regions. (See also...) Figure 2 Describe in detail the process of generating category information.

[0089] Processor 130 can compress and transmit at least one of video, residual, spatial region segmentation information, and category information. For example, processor 130 can compress the residual between a first model parameter and a plurality of second model parameters, and can send the compressed residual. Processor 130 can send the compressed residual to video processing apparatus 500.

[0090] The processor 130 can compress and transmit spatial region segmentation information about multiple spatial regions of a segmented frame, as well as category information about the categories of the multiple spatial regions. For example, the processor 130 can compress the spatial region segmentation information and the category information and send them to the video processing apparatus 500.

[0091] The processor 530 can perform at least one of the following on images and / or videos based on the residuals between the first model parameters and multiple second model parameters: super-resolution, reverse or inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

[0092] Processor 530 can recover multiple second model parameters by adding residuals to the first model parameters. Processor 530 can perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on multiple time segments based on the multiple second model parameters.

[0093] Processor 530 can segment the frame into multiple spatial regions based on spatial region segmentation information and category information. Processor 530 can perform at least one of the following on the video based on the multiple spatial regions and the category information: super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

[0094] Memory 150 and memory 550 may store instructions (or programs) executable by processor 130 and processor 530. For example, the instructions may include instructions for performing operations of processor 130 and processor 530 and / or operations of each element of processor 130 and processor 530.

[0095] Memory 150 and memory 550 can be implemented as volatile memory devices or non-volatile memory devices.

[0096] Volatile memory devices can be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM).

[0097] Non-volatile memory devices can be implemented as electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic random access memory (MRAM), spin-transfer torque (STT)-MRAM, conductive bridging RAM (CBRAM), ferroelectric random access memory (FeRAM), phase change RAM (PRAM), resistive random access memory (RRAM), nanotube RRAM, polymer RAM (PoRAM), nano floating gate memory (NFGM), holographic memory, molecular electronic memory device, or insulator resistance change memory.

[0098] Figure 2 An example of the operation of the video processing device at the transmitting end of Figure 1 is shown.

[0099] Reference Figure 2 The receiver 110 can receive video comprising multiple time segments (or multiple video clips) (110-1, 110-2, ... 110-n). The processor 130 can process the received multiple time segments (110-1, 110-2, ... 110-n).

[0100] Specifically, the processor 130 can use the received video to learn a first neural network (or, an all-scene (AS) model) 210 and multiple second neural networks (e.g., a one-scene (OS) model) (230-1, 230-2, ..., 230-n).

[0101] Processor 130 can use the entire video to learn the first neural network 210. In other words, processor 130 can update the model parameters of the neural network to minimize the loss function corresponding to the entire video, so that client 250 can perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on the entire video that receives the model parameters of the first neural network 210.

[0102] The processor 130 can compress the model parameters of the neural network by learning the model parameters based on the neural network, and minimize the residual of the filter parameters and the video degradation together.

[0103] The residuals of the filter parameters can be transmitted to the video processing device 500 only for the video segment under consideration, which may be advantageous for image restoration applications in receivers (e.g., user terminals) with low complexity and limited computing and storage resources. Client 250 may include the video processing device 500.

[0104] Processor 130 can transmit at least one updated neural network for each temporal portion of the video. Processor 130 can use the parameter residuals between two neural network models to avoid the overhead of neural network model parameters for each temporal portion.

[0105] The first neural network 210, which learns the entire video, can be transmitted before the video is transmitted, and the second neural network model (230-1, 230-2, ..., 230-n), which learns the temporal portions, can be transmitted before the respective temporal portions of the video are transmitted.

[0106] Through this process, processor 130 can learn neural networks to effectively perform neural network-based image restoration (super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal, etc.). Processor 130 can transfer parameter residuals between two neural network models via client 250 to perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

[0107] At this point, the processor 130 can learn a neural network model to overfit individual time segments, and can learn another neural network model to overfit the entire video.

[0108] The processor 130 can use multiple time segments (or multiple scenes) (110-1, 110-2, ..., 110-n) to learn the second neural network (230-1, 230-2, ..., 230-n).

[0109] For example, processor 130 can use the first time portion 110-1 to learn the second neural network 230-1 (or OS-model 1), and can use the second time portion 110-2 to learn the second neural network (or OS-model 2). Similarly, processor 130 can use the nth time portion 100-n to learn the second neural network (or OS-model n). Here, n can be an integer greater than or equal to 1.

[0110] In this way, processor 130 can learn to overfit the first neural network 210 to the entire video. Processor 130 can learn multiple second neural networks (230-1, 230-2, ..., 230-n) to overfit to multiple time segments (110-1, 110-2, ..., 110-n) included in the video.

[0111] A second neural network (230-1, 230-2, ..., 230-n) can be learned to minimize the residuals (or weight residuals) between the first neural network 210 and the second neural network (230-1, 230-2, ..., 230-n).

[0112] Processor 130 can calculate the residuals between the first neural network 210 and multiple second neural networks (230-1, 230-2, ..., 230-n). For example... Figure 2 As shown, the processor 130 can calculate the residual by the difference between the model parameters of the second neural network 230-n (or OS model n) and the model parameters of the first neural network 210 (AS model).

[0113] The processor 130 can compress the information of the learned first neural network 210, the learned second neural network (230-1, 230-2, ..., 230-n), and the received video, respectively, and then send them to the video processing device 500 or the client 250.

[0114] Processor 130 can compress information from the learned first neural network 210. For example, processor 130 can use binary masking and K-means clustering to compress the model parameters (or filter weights) of the learned first neural network 210. Processor 130 can perform binary masking on zero weights and K-means clustering on non-zero weights.

[0115] Processor 130 can send the model parameters of the compressed first neural network 210 to video processing device 500 or client 250. Processor 130 can compress the received video and send it to client 250. Processor 130 can send an encoded video bitstream. For example, processor 130 can perform High Efficiency Video Coding (HEVC) compression on the video and send it to the receiving end or client 250.

[0116] The processor 130 can perform compression on the residuals between the first neural network 210 and multiple second neural networks (230-1, 230-2, ..., 230-n) and send the compressed data to the video processing device 500 or the client 250. As described above, for the residuals, the processor 130 can perform binary masking on zero-weight data and K-means clustering on non-zero-weight data.

[0117] The processor 530 of the video processing apparatus 500 can reconstruct the received compression model parameters. The received compression model parameters may be the residuals (or weighted residuals) corresponding to each time portion before the start of each corresponding time portion.

[0118] The processor 530 can reconstruct the model parameters of the second neural network (230-1, 230-2, ..., 230-n) by adding residuals (or weight residuals) corresponding to each time portion of the decompressed process to the model parameters (or weights) of the first neural network received for the first time.

[0119] The processor 530 can use each of the second neural networks corresponding to the time portion included in the video to perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

[0120] The following describes in detail the method by which the processor 130 learns the first neural network 210 and multiple second neural networks (230-1, 230-2, ..., 230-n).

[0121] Processor 130 can perform neural network training in a content-adaptive manner to overfit to the entire video or a temporal portion of the video. Afterward, processor 130 can send the learned model parameters of the neural network.

[0122] As an example of a learning dataset, processor 130 can use a learning sequence (or video) of 320 frames (including 10 different scenes). Each scene can include 32 frames. The learning sequence has 1920×1080@24fps as ground truth and can generate a low-resolution version by 2× downsampling. Encoding and decoding can then be performed by HM 16.17 reference software.

[0123] The total loss function used to learn a neural network can include the super-resolution (SR) loss and the weight residual (WR) cost. The SR loss can be represented by Equation 1.

[0124] [Mathematical Expression 1]

[0125]

[0126] Where y can represent a truth frame. It can represent the frame on which super-resolution is performed. In other words, the SR loss can represent the L1 loss function between the ground truth frame and the frame on which super-resolution is performed.

[0127] As another example of mathematical formula 1, the loss can be a loss corresponding to at least one of other image restoration processes such as inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal, rather than SR loss.

[0128] The WR cost can be used to minimize the reference weights W pre-learned for the entire video sequence. ref Compared with the learning weight W for the time component train The residuals between the weights (or model parameters).

[0129] The cost of WR can be represented by mathematical formula 2.

[0130] [Mathematical Expression 2]

[0131] C WR=L1(W ref W train )

[0132] That is, the WR cost can represent the L1 loss function between the reference weights and the learned weights. The first neural network (or, AS model) 210 may include pre-learned reference weights for the entire sequence (or, the entire video).

[0133] When learning the first neural network 210, the total loss can be a loss corresponding to at least one of super-resolution (SR), inverse tone mapping / tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal. When learning the second neural network (or, OS model) (230-1, 230-2, ..., 230-n) for multiple time parts, the total loss can include SR loss and WR cost, as shown in Equation 3.

[0134] [Mathematical Expression 3]

[0135] L total =L SR +λC WR

[0136] Here, λ can represent a hyperparameter and can be determined empirically. For example, for a smaller λ value, SR performance may be enhanced, but weight compression performance may decrease due to the increased cost of WR. Conversely, for a larger λ value, SR performance may decrease, but weight compression performance may be enhanced due to the reduced cost of WR. For example, λ could be 0.003.

[0137] The processor 130 can learn the first neural network 210 for all video frames (e.g., 320 frames). At this point, random cropping can be used in part to overfit the video data without using additional augmentation methods.

[0138] During the learning process, the input patch size can be 60×60, and the batch size can be 2. The processor 130 can learn the first neural network 210 for up to 64,000 iterations, at which point the learning rate can be 0.001.

[0139] The processor 130 can use each frame (e.g., 32 frames) of multiple time segments (or scenes) (110-1, 110-2, ..., 110-n) to learn multiple second neural networks (230-1, 230-2, ..., 230-n). At this time, the model parameters of the learned first neural network can be used as reference weights.

[0140] At this point, the first neural network 210 and multiple second neural networks (230-1, 230-2, ..., 230-n) can have the same structure and can be learned using different data. When learning the second neural networks (230-1, 230-2, ..., 230-n), the input patch size can be 400×400, and the initial model parameters of the second neural networks (230-1, 230-2, ..., 230-n) can be set to the model parameters of the pre-learned first neural network 210. Furthermore, the learning conditions for the first neural network 210 and the second neural networks (230-1, 230-2, ..., 230-n) can be the same.

[0141] Figure 3 Another example of the operation of the video processing device at the transmitting end of Figure 1 is shown.

[0142] Reference Figure 3 The processor 130 can segment the frames included in the video into multiple spatial regions and perform super-resolution on the video based on the multiple spatial regions. The processor 130 can segment the frames into multiple spatial regions and allow the use of neural networks suitable for multiple spatial regions to process the images corresponding to the multiple spatial regions, thereby enhancing the image restoration performance of the client 250.

[0143] Client 250 can not only use information from neural networks that update the time portions of the received video separately, but also use segmentation information of the spatial region of each frame for each time portion and the received neural network model corresponding to each segmented region to enhance image restoration performance.

[0144] A single neural network needs to be large enough to achieve adequate performance for improved quality and to learn properly from a large number of training samples. Because large-scale neural networks may require complex operations and large amounts of storage to process inputs and generate outputs, they can be difficult to use on user terminals with limited operational capabilities.

[0145] Processor 130 can be used in Figure 2 The content adaptation method described herein will segment the frames included in the temporal portion into appropriate spatial regions (or multiple regions), and additionally use multiple neural networks to process the multiple regions separately to perform super-resolution more efficiently.

[0146] At this point, the processor 130 can formally define signaling that identifies which neural network will be applied to the spatial segmentation region of each frame in each temporal segment of the video in service.

[0147] Furthermore, the processor 130 can specifically define the configuration representation of the neural network applied to the segmented spatial region in each frame as a combination of the aforementioned signaling information.

[0148] For example, a neural network performing super-resolution can increase the resolution by a specific factor (e.g., 2 times). The transmitting end can then reduce the video (or content) to the reciprocal of the same factor (1 / 2 times) and send it to the video processing device 500.

[0149] If the video processing device 500 already has a neural network, the processor 130 can send the type of the neural network or the actual weights and topology of the neural network.

[0150] Neural networks can be specifically learned or fine-tuned for the content to be transmitted (e.g., video). When neural networks are learned separately for the temporal portions of the video, each neural network can be sent to the receiving end before or during the transmission of the respective video segments (temporal portions).

[0151] Not only for the temporal portion of the video, but also for the spatial regions of the frames, the processor 530 can use corresponding neural networks to perform image restoration (at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal).

[0152] Processor 130 can determine the categories of multiple spatial regions based on features of multiple spatial regions in a segmented frame. These multiple spatial regions can represent regions of an image or regions of a frame. For a group of images with detailed categories based on the features of multiple regions, processor 130 can use multiple learned neural networks to enhance video processing performance.

[0153] Processor 530 can selectively apply appropriate neural networks to various image regions included in a video frame by means of feature categories of various spatial regions that have been specifically learned based on multiple neural networks, thereby enhancing video processing performance.

[0154] The processor 130 can send information related to which neural network is applied to which spatial region. In this way, the video processing apparatus 500 can be effectively applied to image restoration and quality enhancement (IRQE) applications in user terminals with limited operational capabilities.

[0155] The video processing apparatus 100 and the video processing apparatus 500 can provide signaling between the sending end and the receiving end to negotiate the model (e.g., a neural network model) or model update (topology, architecture, and associated weights) to be used with the video by the receiving end (or client) and the sending end (or server).

[0156] When the video to be processed or the neural network model corresponding to its time portion does not exist in the video processing device 500, the processor 130 of the video processing device 100 may send information about the corresponding neural network model to the video processing device 500.

[0157] When fine-tuning a model for a specific video, the processor 130 of the video processing device 100 can send the model to be applied to the entire video to the video processing device 500. When fine-tuning a model for different time segments of a video, the processor 130 of the video processing device 100 can send the model to be applied to a specific time segment and the video corresponding to that time segment to the video processing device 500.

[0158] If the video processing device 500 has a learned global neural network (GNN) for the entire specific video, the video processing device 100 can send the residual between the model parameters (or weight parameters) of the GNN and the model parameters of the local neural network (LNN) to the video processing device 500.

[0159] Processor 130 can provide a display of the optimization target. For example, processor 130 can use a Uniform Resource Identifier (URI) that indicates an appropriate vocabulary.

[0160] The video processing device 100 can send information to the video processing device 500 regarding which encoder and which encoding parameters to select for video compression. The video processing device 100 can also send to the video processing device 500 one or more values ​​indicating the target error between the raw data and the decoded data.

[0161] If the neural network is fine-tuned for different categories of image texture characteristics, and the video processing device 500 already has a corresponding neural network, the video processing device 100 can send the ID of the neural network model to be applied to the spatial region (or segmented region) of the frame along with the region segmentation information for IRQE application to the video processing device 500.

[0162] If the neural network is fine-tuned for different image texture feature categories, and the video processing device 500 does not have a corresponding neural network model, then the neural network model can be sent to the video processing device 500.

[0163] The video processing device 100 can send video (or content) and a low-quality, fine-tuned neural network together to the device. In this case, four scenarios can be considered.

[0164] Case A could be a scenario where the neural network is fine-tuned for the entire video. The video processing device 100 can send the neural network to the video processing device 500 before or during video streaming. The video and the associated neural network can be paired using unique IDs.

[0165] Case B could be a situation where multiple neural networks are optimized for different time segments of the video. In this case, the video can be segmented temporally or by other methods. The video processing device 100 can send each neural network or the differences between each neural network to the video processing device 500 before or during the streaming of each segment of the video. Even in this case, each different segment of the video and its associated neural network can be paired.

[0166] Case C could be a situation where multiple neural networks are optimized for different feature categories of the video (or image). The video processing apparatus 100 can send the neural network model of the segmented region (or image region) and the region segmentation information of the image to the video processing apparatus 500 before or during image streaming.

[0167] Case D could be a situation where multiple neural network models are already available at the receiving end for the target IRQE application. The video processing device 100 can determine the neural network model for each region of the image to be optimally applied to the IRQE, and send the ID of the neural network model to be applied to the segmented region (or image region) and the region segmentation information of the image to the video processing device 500.

[0168] When using a single neural network model for all content, the processor 130 of the video processing device 100 can send or update the neural network only once a year until a new version of the neural network becomes available. For example, the processor 130 can send or update the neural network to the video processing device 500 once a month.

[0169] When fine-tuning the neural network model for each specific video (or content), the processor 130 of the video processing device 100 can send or update the neural network model for each new video. For example, the processor 130 can send or update the neural network model to the video processing device 500 multiple times per day.

[0170] When fine-tuning the neural network for specific time segments of a video, the processor 130 of the video processing apparatus 100 can send the neural network model for all time segments of the video to the video processing apparatus 500. In this case, the processor 130 can send or update the neural network for all time segments of the video. For example, the processor 130 can send or update the corresponding neural network to the video processing apparatus 500 every 1 to 5 minutes (corresponding to each time segment).

[0171] When multiple neural networks are fine-tuned for different video feature categories and the fine-tuned neural networks are available at the receiving end, the processor 130 of the video processing device 100 can send each image (or frame) of the time portion of the video to be transmitted and / or the ID of the neural network for each time portion and / or just the image segmentation information to the video processing device 500.

[0172] The size of a neural network can range from several kilobytes to several hundred megabytes. The bandwidth of the distribution channel may depend on the distribution channel. For example, the bandwidth could be second-generation (2G, 2 nd -generation), 3G, 4G and 5G network bandwidth.

[0173] When using a single model for all videos, acceptable latency can be any small amount, ranging from seconds to minutes. For neural networks that are fine-tuned for each piece of content, the maximum acceptable latency can be approximately 1 second.

[0174] For a neural network that fine-tunes for each time segment of a video, the maximum acceptable latency can be very small, such as a fraction of the sampling period. For example, the latency of a video stream could be 3 milliseconds (1 / 10 of the sampling period).

[0175] As described above, by segmenting the frames included in the video and processing the images, the video processing apparatus 100 can provide benefits for content retrieval and negotiation interfaces through standardized signaling of neural network representations in terms of inter-operability.

[0176] Furthermore, the video processing device 100 can provide benefits to the media consuming device through a standardized neural network representation suitable for artificial intelligence (AI) capabilities.

[0177] Figure 4 An example of the structure of the neural network used by the video processing device at the transmitting end and the video processing device at the receiving end of Figure 1 is shown. Figure 5 Show Figure 4 An example of the structure of a lightweight residual dense block (LwRDB).

[0178] Reference Figure 4 and Figure 5 The processor 130 of the video processing device 100 and the processor 500 of the video processing device 500 can use the first neural network 210 and the second neural network (230-1, 230-2, ..., 230-n) to process video.

[0179] At least one of the first neural network 210 and the second neural network (230-1, 230-2, ..., 230-n) may include Figure 4 The neural network 400 in the structure shown. Neural network 400 can be referred to as a replicated light-weight residual dense-super-resolution (RLRD-SR) network.

[0180] The neural network 400 may include a convolutional layer 410, a depthwise convolutional layer 420, and a pointwise convolutional layer 430. The neural network 400 may include a combination of lightweight residual dense blocks 440, a convolutional layer 450, a depthwise convolutional layer 460, and a pointwise convolutional layer 470.

[0181] In addition, the neural network 400 may include a transposed convolutional layer 480, a convolutional layer 490-1, and a convolutional layer 490-2.

[0182] The combination 440 of lightweight residual dense blocks can include one or more lightweight residual dense blocks (LwRDBs) 441. Multiple lightweight residual dense blocks 441 can be connected in a cascaded structure. The processor 130 can use a neural network configured with replicated lightweight residual dense blocks 441 to perform video processing.

[0183] A combination 441 of multiple lightweight residual dense blocks can share parameters and can be connected in a cascaded structure. By using N repeating lightweight residual dense blocks 441 with shared parameters, the processor 130 can improve super-resolution performance without increasing the number of model parameters.

[0184] Each of the lightweight residual dense blocks 441 may include three convolutional layers. The lightweight residual dense block 441 may include depth-wise separable convolutional layers to reduce the learning parameters of the first two layers, and may include a 1×1 convolutional layer at the end.

[0185] Since the processor 130 only sends residual weights, it can improve weight compression efficiency by minimizing the weights sent during the learning period.

[0186] Figure 6 The peak signal-to-noise ratio (PSNR) according to the video processing method in Figure 1 is shown. Figure 7 The PSNR and bit size reduction rate are shown based on the hyperparameters.

[0187] Reference Figure 6 and Figure 7 The performance of the video processing device 100 can be measured experimentally. It can be used with 64-bit Windows 10. TM The OS was tested on a PC platform with an Intel i7-8700K 3.7GHz CPU, 64GB RAM, Python 3.6, PyTorch 0.4.0, CUDA v9.0, and CUDNN 7.0.

[0188] Three super-resolution models can be used to compare performance. The three super-resolution models used may include the scene-to-scene corresponding OS model, the scene-to-scene AS model, and bicubic interpolation for video upscaling.

[0189] Table 1 shows the PSNR and bpp (bits per pixel) values ​​for 10 scenes in the video when λ is set to 0.003, the centroid number of K-means clustering is set to 1000 for the AS-model and 300 for the OS-model.

[0190] [Table 1]

[0191]

[0192]

[0193] bicubic interpolation bpp(bpp) bic The bit size (in bits) of all frames in each scene encoded at 1200kbps can be obtained by using HM16.17. 视频 The value of bicubic interpolation (bpp) is obtained by dividing by the height (H), width (W), and number of frames (n) of the up-scaling video. In other words, the bpp of bicubic interpolation can be calculated using mathematical formula 4.

[0194] [Mathematical Expression 4]

[0195]

[0196] To calculate bpp(bpp) of the AS model AS Further consideration can be given to the bits of the parameters learned for the initial transmission. AS The bit size of the AS-model can be divided into the number of scenes (S) to obtain the size allocated to a scene. The bit size of the AS-model can be calculated using mathematical formula 5.

[0197] [Mathematical Expression 5]

[0198]

[0199] To calculate bpp(bpp) of the OS model OS The residual parameters (bits) that can be learned for transmission can be considered. OS-残差 The bpp of the OS-model can be calculated using mathematical formula 6.

[0200] [Mathematical Expression 6]

[0201]

[0202] Figure 6 The curve can represent the PSNR-bpp of bicubic interpolation, the AS-model, and the OS-model. From Figure 6 As can be seen, the OS-model has a PSNR that is about 3 dB higher and a PSNR that is about 0.7 dB higher than that of bicubic interpolation.

[0203] Figure 7 The curve can represent the PSNR and weight compression performance for different λ values. A larger λ value can achieve higher learning weight compression performance, but the PSNR performance may decrease. Therefore, it may be necessary to consider both weight compression performance and PSNR performance to determine the λ value.

[0204] Figure 8 The sequence of operations of the video processing device at the transmitting end of Figure 1 is shown.

[0205] Reference Figure 8In step 810, the receiver 110 of the video processing device 100 may receive video including multiple time segments and / or model information related to the neural network used to process the video.

[0206] In step 830, the processor 130 may obtain first model parameters corresponding to the first neural network used to process the entire video based on the model information.

[0207] In step 850, the processor 130 can obtain multiple model parameters corresponding to multiple second neural networks used to process multiple time segments respectively, based on model information.

[0208] In step 870, the processor 130 may calculate the residuals between the first model parameters and a plurality of second model parameters.

[0209] In step 890, processor 130 can transmit video and residuals.

[0210] Processor 130 can generate spatial region segmentation information by segmenting frames included in multiple time segments. Processor 130 can generate category information for the segmented spatial regions based on the signal characteristics of the segmented spatial regions.

[0211] The processor 130 can compress and transmit at least one of video, residuals between the first neural network and multiple second neural networks, spatial region segmentation information, and category information.

[0212] The processor 130 can learn multiple second neural networks to minimize the residuals.

[0213] At least one of the first neural network and the plurality of second neural networks may include a lightweight residual dense block comprising at least one convolutional layer. At least one of the first neural network and the plurality of second neural networks may include a plurality of lightweight residual dense blocks configured in a cascade structure.

[0214] Lightweight residual dense blocks can include depth-wise separable convolutional layers or 1×1×d convolutional layers. Depth-wise separable convolutional layers can include depthwise convolutional layers, pointwise convolutional layers, and nonlinear activation functions.

[0215] Figure 9 The sequence of operations of the video processing device at the receiving end of Figure 1 is shown.

[0216] Reference Figure 9In step 910, receiver 510 may receive video comprising multiple time segments. In step 930, receiver 510 may receive first model parameters corresponding to a first neural network used to process the entire video.

[0217] In step 950, receiver 510 may receive the residuals between the first model parameters and the multiple second model parameters corresponding to the multiple second neural networks used to process multiple time segments respectively.

[0218] Receiver 510 can receive spatial region segmentation information and category information of the segmented spatial regions associated with frames included in multiple time segments. The spatial region segmentation information and category information are the same as described above.

[0219] In step 970, the processor 530 may perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on the video based on the residual.

[0220] Processor 530 can recover multiple second model parameters by adding residuals to the first model parameters. Processor 530 can perform at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on multiple time segments based on the multiple second model parameters.

[0221] The processor 530 can segment a frame into multiple spatial regions based on spatial region segmentation information and category information.

[0222] The processor 530 can perform at least one of the following on the video based on multiple spatial regions and category information: super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

[0223] At least one of the first neural network and the plurality of second neural networks may include a lightweight residual dense block comprising at least one convolutional layer. At least one of the first neural network and the plurality of second neural networks may include a plurality of lightweight residual dense blocks configured in a cascade structure.

[0224] Lightweight residual dense blocks can include depth-wise separable convolutional layers or 1×1×d convolutional layers. Depth-wise separable convolutional layers can include depthwise convolutional layers, pointwise convolutional layers, and Rectified Linear Units (ReLU).

[0225] The method according to the embodiments is embodied in the form of program instructions executable by various computer means and recorded in a computer read / write medium. The computer read / write medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the medium may be instructions specifically designed and configured to implement the embodiments, or instructions that can be used by a person skilled in the art of computer software based on commonly known methods. The computer read / write recording medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media similar to CD-ROMs and DVDs; magneto-optical media similar to floppy disks; and hardware devices specifically configured to store and execute program instructions, such as read-only memory (ROM), random access memory (RAM), and flash memory. Examples of program instructions include not only machine language code generated by a compiler, but also high-level language code executable by a computer using an interpreter or similar means. To perform the operations of the embodiments, the hardware device may be configured to implement the operations in one or more software modules, or vice versa.

[0226] Software can include computer programs, code, instructions, or a combination of more than one of these, enabling a processing device to operate in a desired manner, or to command the processing device individually or collectively. To interpret or provide commands or data to the processing device, the software and / or data can be permanently or temporarily embodied in any type of device, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave. Software is distributed across a network-connected computer system and can be stored or executed in a distributed manner. Software and data can be stored on more than one computer read / write storage medium.

[0227] In summary, embodiments have been described with reference to the limited accompanying drawings. Those skilled in the art can make various modifications and variations to the above description. For example, the described techniques may be performed in a different order than the described methods, and / or the described constituent elements may be combined or arranged in a different form than the described methods, or the same effect may be obtained by replacing or substituting other constituent elements or equivalents.

[0228] Therefore, all other embodiments, other implementations, and equivalents of the claims fall within the scope of the patent claims.

Claims

1. A video processing method, characterized in that, Includes the following steps: Receives video that includes multiple time segments; Receive first model parameters corresponding to a first neural network used for overall processing of the video; Receive residuals between first model parameters and multiple second model parameters corresponding to multiple second neural networks used to process the multiple time segments respectively, wherein each of the multiple second neural networks is used to process a different one of the multiple time segments; and Based on the residual, perform at least one of the following on the video: super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

2. The video processing method according to claim 1, characterized in that, The first neural network and at least one of the plurality of second neural networks include a lightweight residual dense block containing at least one convolutional layer.

3. The video processing method according to claim 1, characterized in that, The first neural network and at least one of the plurality of second neural networks include a plurality of lightweight residual dense blocks configured in a cascaded structure.

4. The video processing method according to claim 2, characterized in that, The lightweight residual dense block includes: Depth-separable convolutional layers; or 1×1×d convolutional layer.

5. The video processing method according to claim 4, characterized in that, The depth-separable convolutional layer includes: Depthwise convolutional layers; Pointwise convolutional layers; and Nonlinear activation function.

6. The video processing method according to claim 1, characterized in that, The process of performing at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal includes the following steps: The plurality of second model parameters are recovered by adding the residuals to the first model parameters; and Based on the multiple second model parameters, at least one of the following is performed on the multiple time segments: super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

7. The video processing method according to claim 1, characterized in that, It also includes the following steps: Receive spatial region segmentation information of frames included in the plurality of time segments and category information of the segmented spatial regions.

8. The video processing method according to claim 7, characterized in that, The process of performing at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal includes the following steps: The frame is segmented into multiple spatial regions based on the spatial region segmentation information and the category information; and Based on the multiple spatial regions and the category information, perform at least one of the following on the video: super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

9. A video processing apparatus, characterized in that, include: A receiver receives video comprising multiple time segments, and receives first model parameters corresponding to a first neural network for overall processing of the video, and receives residuals between the first model parameters and multiple second model parameters corresponding to multiple second neural networks for processing the multiple time segments respectively, wherein each of the multiple second neural networks is used to process a different one of the multiple time segments; and A processor that performs at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal on the video based on the residual.

10. The video processing apparatus according to claim 9, characterized in that, The first neural network and at least one of the plurality of second neural networks include a lightweight residual dense block containing at least one convolutional layer.

11. The video processing apparatus according to claim 9, characterized in that, The first neural network and at least one of the plurality of second neural networks include a plurality of lightweight residual dense blocks configured in a cascaded structure.

12. The video processing apparatus according to claim 10, characterized in that, The lightweight residual dense block includes: Depth-separable convolutional layers; or 1×1×d convolutional layer.

13. The video processing apparatus according to claim 12, characterized in that, The depth-separable convolutional layer includes: Depthwise convolutional layers; Pointwise convolutional layers; and Nonlinear activation function.

14. The video processing apparatus according to claim 9, characterized in that, The processor, The frames included in the video will be divided into multiple regions. The plurality of second model parameters are recovered by adding the residuals to the first model parameters. Based on the multiple second model parameters, at least one of the following is performed on the multiple time segments: super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

15. The video processing apparatus according to claim 9, characterized in that, The receiver also receives spatial region segmentation information of frames included in the plurality of time segments and category information of the segmented spatial regions.

16. The video processing apparatus according to claim 15, characterized in that, The processor, The frame is divided into multiple spatial regions based on the spatial region segmentation information and the category information. Based on the multiple spatial regions and the category information, perform at least one of the following on the video: super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal.

17. A video processing method, comprising performing at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal. Its features are, Includes the following steps: Receive video including multiple time segments and model information related to the neural network used to process the video; Obtain first model parameters corresponding to a first neural network used to perform overall processing of the video based on the model information; Receive multiple second model parameters corresponding to multiple second neural networks for processing the multiple time segments respectively based on the model information, wherein each of the multiple second neural networks is used to process a different one of the multiple time segments; Calculate the residuals between the first model parameters and the plurality of second model parameters; and Transmit the video and the residual.

18. The video processing method according to claim 17, characterized in that, It also includes the following steps: Spatial region segmentation information is generated by segmenting frames included in the plurality of time segments; as well as The category information of the segmented spatial regions is generated based on the signal characteristics of the segmented spatial regions.

19. The video processing method according to claim 18, characterized in that, The transmission of the video and the residual includes the following steps: Compress and transmit at least one of the video, the residual, the spatial region segmentation information, and the category information.

20. The video processing method according to claim 17, characterized in that, It also includes the following steps: The multiple second neural networks are learned to minimize the residuals.

21. A video processing apparatus for performing at least one of super-resolution, inverse tone mapping, tone mapping, frame interpolation, motion deblurring, denoising, and compression artifact removal, characterized in that, include: A receiver that receives video comprising multiple time segments and model information related to a neural network used to process the video; and The processor acquires first model parameters corresponding to a first neural network used to process the video as a whole based on the model information, and receives multiple second model parameters corresponding to multiple second neural networks used to process the multiple time segments respectively based on the model information, wherein each of the multiple second neural networks is used to process a different one of the multiple time segments, calculates the residual between the first model parameters and the multiple second model parameters, and transmits the video and the residual.

22. The video processing apparatus according to claim 21, characterized in that, The processor, Spatial region segmentation information is generated by segmenting frames included in the plurality of time segments. The category information of the segmented spatial regions is generated based on the signal characteristics of the segmented spatial regions.

23. The video processing apparatus according to claim 22, characterized in that, The processor compresses and transmits at least one of the video, the residual, the spatial region segmentation information, and the category information.

24. The video processing apparatus according to claim 21, characterized in that, The processor learns the multiple second neural networks to minimize the residuals.

Images