System and method for distortion removal over multiple quality levels

Through the initial distortion removal model and adjustment model trained by multi-quality level, combined with convolutional neural network and activation function, the problems of fixed quality level and non-scalability in the existing technology are solved, and the distortion removal of contents of multiple quality levels is achieved. and artifacts improve the flexibility and efficiency of the codec.

CN119094758APending Publication Date: 2024-12-06DISNEY ENTERPRISES INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411019534.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-10-22
Filing Date
2019-04-25
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing deep learning methods are fixed to a single specific degradation/distortion quality level in lossy image and video codec, unable to adapt to varying quality levels between different frames, and are often targeted at specific encoding and decoding algorithms, resulting in unscalability and Multi-model training requirements.

Method used

Initial distortion removal models trained with multiple different quality levels, and the adjusted distortion removal models are generated by training content, adapting to multiple distortion quality levels in the target content, using convolutional neural networks and activation functions, combining course learning and progressive Parameter training to generate the corrected target content.

Benefits of technology

It realizes flexible removal of distortion and artifacts for multiple untrained quality levels, improves content quality, enhances the flexibility and efficiency of codecs, and avoids the need to train multiple models for each quality level.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094758A_ABST
    Figure CN119094758A_ABST
Patent Text Reader

Abstract

The invention discloses a system and method for distortion removal over multiple quality levels. In one embodiment, a method may include receiving training content. The training content may include original content, reconstructed content, and a training distortion quality level corresponding to the reconstructed content. Reconstructed content may be derived from the distorted original content. The method may also include training a distortion quality level corresponding to the reconstructed content. The method may also include receiving an initial distortion removal model. The method may include generating an adjusted distortion removal model by training an initial distortion removal model using the training content. The method may also include storing the adjusted distortion removal model.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of Chinese Patent Application No. 201910336790.3, filed on April 25, 2019, entitled “System and method for distortion removal at multiple quality levels”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Patent Application No. 62 / 664,868, filed on April 30, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present disclosure relates generally to distortion and artifact removal. Summary of the invention

[0005] Embodiments of the present disclosure are directed to systems and methods trained on multiple different training quality levels that remove distortion from media content of varying quality levels that may overlap the multiple different training quality levels.

[0006] In one embodiment, a computer-implemented method includes: receiving training content. The training content may include: original content; reconstructed content derived from distorted original content; and training distortion quality levels corresponding to the reconstructed content. The computer-implemented method may also include receiving an initial distortion removal model; generating an adjusted distortion removal model by training the initial distortion removal model using the training content; and storing the adjusted distortion removal model.

[0007] In an embodiment, the method may further include: receiving target content. The target content may have one or more target distortion quality levels. The method may further include applying the adjusted distortion removal model to the target content to generate corrected target content.

[0008] In an embodiment, the initial distortion removal model and the adjusted distortion removal model comprise one or more user-defined output branches based on one or more distortion quality levels.

[0009] The computer-implemented method of claim 3, wherein the initial distortion removal model comprises two branches, wherein a first set of training content corresponding to the first branch passes through a first set of convolutional layers, and a second set of training content corresponding to the second branch passes through the first set of convolutional layers and a second set of convolutional layers.

[0010] In an embodiment, training an initial distortion removal model using training content includes: applying a first set of training content to a first set of convolutional layers; applying at least one of the first set of training content to a second set of convolutional layers; applying the second set of training content to the first and second sets of convolutional layers; and training the first branch and the second branch with equal weights for the first branch and the second branch when the second branch is adjusted.

[0011] In an embodiment, the training distortion quality level comprises at least one different individual value compared to one or more target distortion quality levels.

[0012] In an embodiment, the initial distortion removal model and the adjusted distortion removal model include convolutional neural networks.

[0013] In an embodiment, the initial distortion removal model and the adjusted distortion removal model comprise an activation function.

[0014] In an embodiment, a training conditioned distortion model is trained to remove distortion from target content.

[0015] In one embodiment, a computer-implemented method includes receiving target content. The target content may have multiple distortion quality levels. The computer-implemented method may also include receiving an adjusted distortion removal model. The adjusted distortion removal model may be adjusted by training an initial distortion removal model using training content. The training content includes original content and corresponding reconstructed content, each reconstructed content having a given distortion quality level. The computer-implemented method may also include applying the adjusted distortion removal model to the target content to generate corrected target content.

[0016] In an embodiment, the adjusted distortion removal model comprises one or more user-defined output branches based on a plurality of distortion quality levels.

[0017] In an embodiment, the target content is corrected based on a given distortion quality level for the given target content, the given distortion quality level corresponding to one of the one or more user-defined output branches of the adjusted distortion removal model.

[0018] In an embodiment, the adjusted distortion removal model includes two branches, wherein a first set of target content corresponding to the first branch passes through a first set of convolutional layers, and a second set of target content corresponding to the second branch passes through a first set of convolutional layers and a second set of convolutional layers.

[0019] In an embodiment, the initial distortion removal model and the adjusted distortion removal model include convolutional neural networks.

[0020] In an embodiment, the content includes one or more of images and videos.

[0021] In an embodiment, the content includes one or more of standard content, high definition (HD) content, ultra HD (UHD) content, 4k UHD content, and 8k UHD content.

[0022] In one embodiment, a system for removing artifacts may include an electronic storage device; one or more physical computer processors configured by machine-readable instructions are used to: obtain target media content. The target media content has multiple distortion quality levels. The one or more physical computer processors can also be configured by the machine-readable instructions to: obtain an adjusted distortion removal model from the electronic storage device. The adjusted distortion removal model has been adjusted by training an initial distortion removal model using training media content. The training media content includes original media content and corresponding reconstructed media content, each reconstructed media content having a given distortion quality level. The one or more physical computer processors can also be configured by the machine-readable instructions to: apply the adjusted distortion removal model to the target media content to generate corrected target media content using the one or more physical computer processors.

[0023] In an embodiment, the adjusted distortion removal model comprises one or more user-defined output branches based on one or more distortion quality levels.

[0024] In an embodiment, the target content is corrected based on a given distortion quality level of the given target content, the given target content corresponding to one of the one or more branches of the adjusted distortion removal model.

[0025] In an embodiment, the initial distortion removal model and the adjusted distortion removal model include convolutional neural networks.

[0026] Other features and aspects of the disclosed method will become apparent from the following detailed description in conjunction with the accompanying drawings, which illustrate features according to embodiments of the present disclosure by way of example. The Summary is not intended to limit the scope of the disclosure claimed, which is limited only by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0028] Other aspects of the present disclosure will become more readily apparent upon review of the following detailed description of various disclosed embodiments in conjunction with the accompanying drawings.

[0029] Figure 1 An example system for correcting target content having multiple untrained distortion quality levels in accordance with various embodiments is described.

[0030] Figure 2 An example distortion removal model in accordance with an embodiment of the disclosed technology is described.

[0031] Figure 3 An example course learning system according to an embodiment of the present disclosure is described.

[0032] Figure 4 An operational flow diagram illustrating an example process for training a distortion removal model using training content and applying the distortion removal model to target content is illustrated in accordance with one embodiment.

[0033] Figure 5A An image with JPEG compression at quality level 20 is illustrated in accordance with an embodiment of the disclosed technology.

[0034] Figure 5B An image with JPEG compression at quality level 20 is illustrated in accordance with an embodiment of the disclosed technology.

[0035] Figure 5C An original image according to an embodiment of the disclosed technology is illustrated.

[0036] Figure 5D An original image according to an embodiment of the disclosed technology is illustrated.

[0037] Figure 5E An image is illustrated to which a progressive distortion removal model has been applied in accordance with an embodiment of the disclosed technology.

[0038] Fig. 5F An image is illustrated to which a progressive distortion removal model has been applied in accordance with an embodiment of the disclosed technology.

[0039] Fig. 6A A WebP original image according to an embodiment of the disclosed technology is illustrated.

[0040] Figure 6B A WebP original image according to an embodiment of the disclosed technology is illustrated.

[0041] Figure 6C An image of WebP compression with quality level 20 and in-loop filter turned off is illustrated in accordance with an embodiment of the disclosed technology.

[0042] Fig.6D An image of WebP compression with quality level 20 and in-loop filter turned off is illustrated in accordance with an embodiment of the disclosed technology.

[0043] Fig. 6E An image of WebP compression with quality level 20 and in-loop filter turned on is illustrated in accordance with an embodiment of the disclosed technology.

[0044] Fig. 6F An image of WebP compression with quality level 20 and in-loop filter turned on according to an embodiment of the disclosed technology is illustrated.

[0045] Figure 6G The embodiments of the disclosed technology have been described in Fig. 6E and Fig. 6F Apply the progressive distortion removal model on the filtered WebP image.

[0046] Figure 6H The embodiments of the disclosed technology have been described in Fig. 6E and Fig. 6F Apply the progressive distortion removal model on the filtered WebP image.

[0047] Fig. 7A An H.264 compressed frame with a constant rate factor (CRF) of 40 and a closed in-loop filter is illustrated in accordance with an embodiment of the disclosed technology.

[0048] Figure 7B An H.264 compressed frame with a constant rate factor (CRF) of 40 and a closed in-loop filter is illustrated in accordance with an embodiment of the disclosed technology.

[0049] Figure 7C An original frame according to an embodiment of the disclosed technology is illustrated.

[0050] Fig.7D An original frame according to an embodiment of the disclosed technology is illustrated.

[0051] Fig. 7E An H.264 compressed frame with a constant rate factor (CRF) 40 and an in-loop filter turned on is illustrated in accordance with an embodiment of the disclosed technology.

[0052] Figure 7F An H.264 compressed frame with a constant rate factor (CRF) 40 and an in-loop filter turned on is illustrated in accordance with an embodiment of the disclosed technology.

[0053] Figure 7G The embodiments of the disclosed technology have been described in Fig. 7A and Figure 7B Image of an unfiltered H.264 frame with the progressive distortion removal model applied on it.

[0054] Figure 7H The embodiments of the disclosed technology have been described in Fig. 7A and Figure 7B Image of an unfiltered H.264 frame with the progressive distortion removal model applied on it.

[0055] Fig. 8A An H.264 compressed frame with a constant rate factor (CRF) of 35 and a closed in-loop filter is illustrated in accordance with an embodiment of the disclosed technology.

[0056] Figure 8B An H.264 compressed frame with a constant rate factor (CRF) of 35 and a closed in-loop filter is illustrated in accordance with an embodiment of the disclosed technology.

[0057] Figure 8C An original frame according to an embodiment of the disclosed technology is illustrated.

[0058] Fig.8D An original frame according to an embodiment of the disclosed technology is illustrated.

[0059] Fig. 8E A compressed frame of H.264 with a constant rate factor (CRF) of 35 and an in-loop filter turned on is illustrated in accordance with an embodiment of the disclosed technology.

[0060] Figure 8F An H.264 compressed frame with a constant rate factor (CRF) of 35 and an in-loop filter turned on is illustrated in accordance with an embodiment of the disclosed technology.

[0061] Figure 8G The embodiments of the disclosed technology have been described in Fig. 8A and Figure 8B Image of an unfiltered H.264 frame with the progressive distortion removal model applied on it.

[0062] Figure 8H The embodiments of the disclosed technology have been described in Fig. 8A and Figure 8B Image of an unfiltered H.264 frame with the progressive distortion removal model applied on it.

[0063] Fig. 9 An unfiltered image, an image filtered using an H.265 filter, and an image filtered using a progressive distortion removal model are illustrated. According to an embodiment of the disclosed technology, the H.265 compressed frame has a constant rate factor (CRF) of 35 and a closed in-loop filter.

[0064] Fig.10 Example computing components that can be used to implement features of various embodiments of the present disclosure are described.

[0065] The accompanying drawings are described in more detail in the following description and examples, which are provided for illustrative purposes only and depict only typical or example embodiments of the present disclosure. The accompanying drawings are not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. It should also be understood that the present disclosure can be practiced through modification or alteration, and the present disclosure may be limited only by the claims and their equivalents. DETAILED DESCRIPTION

[0066] Some current uses of lossy image and video codecs may contain one or more filters designed to reduce distortion and compression artifacts. In video codecs, such filtering is a critical in-loop processing stage because the reconstructed image is used as a reference for future compression decisions in time. While deep learning-based image processing methods can provide better filtering results because they are able to learn more complex mappings between the distorted reconstructed image and the original image, existing deep learning methods are typically fixed to a single specific degradation / distortion quality level, which limits the scalability of such methods. In lossy video coding, the degradation / distortion quality level can vary from frame to frame, adjusted by the target output bitrate, and therefore, a single degradation / distortion quality level approach is not feasible. In addition, existing methods typically consider distortion removal for one specific encoding and decoding algorithm. Therefore, most networks need to be retrained for other codecs and quality levels.

[0067] Therefore, various embodiments of the present disclosure relate to a method or system for training with multiple different quality levels and receiving target content of various untrained quality levels and generating corrected content. The content may include various media content, such as videos and images that may not be encoded. The content may contain various resolutions, such as standard, high definition (HD), ultra HD (UHD), 4kUHD, 8k UHD and / or other resolutions. In an embodiment, the training content may be used to train an initial distortion removal model. The training content may include compressed and / or distorted content, each piece of content having a quality level, and corresponding original content. In some embodiments, the training content may include multiple different quality levels. After training the initial distortion removal model, an adjusted distortion removal model may be generated. In some embodiments, target content having a different quality level from the training content may be applied to the adjusted distortion removal model. The adjusted distortion removal model may correct and / or optimize the target content to remove distortion and / or artifacts in the target content; thereby generating corrected target content. The adjusted distortion removal model may replace or enhance existing filters to achieve quality improvement of compressed and / or distorted content and increase flexibility to accept different content having different quality levels than the quality level for which the initial distortion removal model was trained.

[0068] Before describing the technology in detail, it may be useful to describe an example environment in which the presently disclosed technology may be implemented. Figure 1 One such example environment 100 is illustrated.

[0069] Environment 100 can be used in conjunction with embodiments implementing the disclosed systems, methods, and devices. As an example, Figure 1 Various components described below in can be used to remove distortion from reconstructed content. Reconstructed content can be derived from content that has been compressed, modified, changed and / or otherwise distorted. For example, the original content can be compressed based on a given quality level and sent to / from the electronic device 102 via the server system 106. Therefore, artifacts and distortions may be present in the compressed content. The content may include video, images and / or other visual media content. The server system 106 may include a distortion removal model 114 that is capable of receiving target content with multiple untrained quality levels, as will be described herein. The distortion removal model 114 can generate corrected content to be sent to or from the electronic device 102.

[0070] The distortion removal model 114 may be stored in the storage device 110. In an embodiment, the initial distortion removal model may be trained on one or more training quality levels in the server system 106 to generate an adjusted distortion removal model, as will be described herein. The training content may include one or more training quality levels. The one or more training quality levels may include a plurality of different quality levels. The adjusted distortion removal model may be applied to target content having one or more target quality levels to generate corrected content. In some embodiments, the one or more target quality levels may have different values ​​than the training quality levels.

[0071] The electronic device 102 may include various electronic computing devices, such as smart phones, laptops, notebook computers, computers, wearable devices, televisions, virtual reality devices, augmented reality devices, displays, connected home devices, Internet of Things (IoT) devices, smart speakers, and / or other devices. The electronic device 102 may present content to a user and / or receive a request to send content to another user. In some embodiments, the electronic device 102 may apply the adjusted distortion removal model to the target content. In an embodiment, the electronic device 102 may store the distortion removal model 114.

[0072] like Figure 1As shown, environment 100 may include one or more of electronic device 102 and server system 106. Electronic device 102 may be coupled to server system 106 via communication medium 104. As will be described in detail herein, electronic device 102 and / or server system 106 may exchange communication signals via communication medium 104, including content, metadata, quality levels, user input, and / or other information.

[0073] In various embodiments, the communication medium 104 may be based on one or more wireless communication protocols (such as Wi-Fi, Bluetooth@, ZigBee, 802.11 protocols, infrared (IR), radio frequency (RF), 2G, 3G, 4G, 5G, etc.) and / or wired protocols and media. In some cases, the communication medium 104 may be implemented as a single medium.

[0074] As described above, the communication medium 104 can be used to connect or communicatively couple the electronic devices 102 and / or the server system 106 to each other, or to a network, and the communication medium 104 can be implemented in various forms. For example, the communication medium 104 can include an Internet connection (such as a local area network (LAN), a wide area network (WAN), a fiber optic network, the Internet on a power line, a hard-wired connection (e.g., a bus), etc.) or any other type of network connection. The communication medium 104 can be implemented using any combination of routers, cables, modems, switches, optical fibers, wires, radio (e.g., microwave / RF links), etc. When reading this disclosure, it should be understood that other ways can be used to implement the communication medium 104 for communication purposes.

[0075] Likewise, it should be appreciated that similar communication media may be used to connect or communicatively couple the server 108, storage 110, processor 112, and / or distortion removal model 114 to one another, in addition to other elements of the environment 100. In example embodiments, the communication medium 104 may be or include a wired or wireless wide area network (e.g., cellular, fiber optic, and / or circuit-switched connections, etc.) for the electronic device 102 and / or the server system 106, which may be relatively geographically diverse; and in some cases, aspects of the communication medium 104 may involve a wired or wireless local area network (e.g., Wi-Fi, Bluetooth, unlicensed wireless connections, USB, HDMI, standard AV, etc.), which may be used to communicatively couple aspects of the environment 100 that may be relatively geographically close.

[0076] The server system 106 may provide, receive, collect or monitor information to / from the electronic device 102, such as content, metadata, quality ratings, user input, security and encryption information, etc. The server system 106 may be configured to receive or send such information via the communication medium 104. The information may be stored in the storage device 110 and may be processed using the processor 112. For example, the processor 112 may include an analysis engine capable of performing analysis on information that has been collected, received, etc. by the server system 106 from the electronic device 102. The processor 112 may include a distortion removal model 114 that is capable of receiving target content, analyzing the target content, and otherwise processing the content and generating corrected content that has been collected, received, etc. by the server system 106 based on a request from or from the electronic device 102. In an embodiment, the server 108, the storage device 110, and the processor 112 may be implemented as a distributed computing network, a relational database, etc.

[0077] The server 108 may include, for example, an Internet server, a router, a desktop or laptop computer, a smart phone, a tablet computer, a processor, a module, etc., and may be implemented in various forms, including, for example, an integrated circuit or a collection thereof, a printed circuit board or a collection thereof, or in a separate housing / package / rack or multiple thereof. The server 108 may update information stored on the electronic device 102. The server 108 may send / receive information to / from the electronic device 102 in real time or irregularly. In addition, the server 108 may implement cloud computing capabilities for the electronic device 102. After studying this disclosure, those skilled in the art will appreciate that the environment 100 may include multiple electronic devices 102, a communication medium 104, a server system 106, a server 108, a storage device 110, a processor 112, and / or a distortion removal model component 114.

[0078] Figure 2An example distortion removal model according to an embodiment of the disclosed technology is illustrated. As illustrated, the distortion removal model 114 can be a progressive convolutional network branching at different depths. As described above, in some embodiments, the system 100 can be used to train the distortion removal model 114. In an embodiment, the system 100 can retrieve or obtain the conditioned distortion removal model from the storage device 110. The architecture of the distortion removal model 114 can include multiple layers of 2DConv 3×3×64 204 and leaky rectified linear units (leaky ReLU) 206. As illustrated, there are jump connections from the input to all outputs and convolutional layers in between. The number of input channels for a given layer can be based on the output of the previous layer. In an embodiment, the number of layers used for each quality set or output set index 210 and the number of quality sets can be the same for all compressions, distortions, etc. used for training. In one example, there are three different total quality sets, the first total quality set starts with 10 layers, the second total quality set adds 5 more layers, and the last adds 3 more layers, excluding the first and last convolutions.

[0079] As illustrated, 2DConv 3×3×64 204 may include a 2D convolution with a 3×3 kernel size and 64 output channels or, for example, feature maps. Each feature map may be used to identify, determine, and / or analyze whether an inspected portion of the target content includes distortion or artifacts. Each feature map may have a corresponding weight matrix to determine the impact of the inspected portion of the target content on generating corrected content. Leaky ReLU 206 may be a non-saturated activation function. At the deepest output, in one example, there are a total of twenty convolutional layers. The output set index 210 may be an output branch based on a given quality level. It should be appreciated that this is an embodiment of a distortion removal model and that different numbers of input channels, layers, activation functions, feature maps, output channels, and other components may be used. It should be appreciated that other activation functions may be used in conjunction with different convolutional networks.

[0080] Having described the general architecture, an example training technique is described in more detail. The training content may include original content, reconstructed content, and corresponding quality levels of the reconstructed content. Various techniques may be used to train the distortion removal model 114. In one example, the original content is represented as I in the following equation, and the reconstructed output content of the distortion method has a distortion quality level Q, as I' in the following equation: cQ In one example, a convolutional network f is trained to reconstruct the content I given its distorted counterpart I'. This is based on solving an optimization problem of the form

[0081] arg f min∑ I∈Xl(I,f(I')) (1)

[0082] Where l is the loss metric and X is the set of original content. For the convolutional network f, additional variables can be introduced that allow the network to branch at different depths. For less distorted I' (depending on Q), branching starts earlier than for more distorted I'. Therefore, a progressive parameter is added. The new parameter depends on the distortion quality level Q that defines the decision function s(·). For example, for JPEG, the function can be defined as:

[0083]

[0084] In an embodiment, the distortion quality level Q can be determined in various ways and can be user-defined. For example, in still image compression (JPEG, etc.), the compression quality level Z is usually the input of the compression process, and Q can be derived from Z, Q=g(Z), if Q is not Z itself. Similarly, some video compression methods also attempt to produce consistent output quality by providing a compression quality level Z as an input to the encoding process. The constant rate factor (CRF) of x264 and x265 is such an example. Similarly, the distortion quality level can be Q=g(Z). In some compression methods, the distortion of the compressed output content I' varies frame by frame according to the bit rate target, buffer modeling, and group of pictures (GOP) frame type. In this case, it may be necessary to dynamically determine the distortion quality level Q of I'. In these cases, Q=g(I'). Ultimately, the following problems are solved:

[0085]

[0086] where Q(C) represents all quality levels of the distortion method C.

[0087] Another technique for training the distortion removal model 114 may be to segment the training content into patches (e.g., 128×128) that align with the distortion block size used in the distortion method. If the distortion method uses multiple sizes, the largest possible block size may be used. As a result, the performance of the distortion removal model 114 may be improved by training on smaller content patches while still preserving the location of blocking artifacts. Alternatively, instead of training all patches, a user-selected fixed amount of patches per content may be randomly sampled and the patches may be aligned according to the blocks used in the distortion method.

[0088] Another technique could be curriculum learning. Curriculum learning can involve gradually adding more complex data or content to the distortion removal model to aid in training. Figure 3, train the first quality set with a specific distortion quality level Q1 until the loss is almost converged. Then, the second quality set can be activated with content from distortion quality level Q2, but the output of set index 2 is not used (see Figure 2 ), we can use a linear combination of the outputs of set one and set two:

[0089] OSI i =α*OSI i +(1-α)OSI i-1

[0090] OSI i Refers to output set index i, and for α value 302, the second output set can be slowly mixed. First, when training Q2 content, content can be input from the previous distortion quality level Q1 with a probability of 50%. If there is more than one previous level, the 50% probability can be evenly divided. For example, if there are two previous levels, for the third output set index, the 50% probability of using content from Q1 and Q2 can be evenly divided so that the probability of using content from Q1 is 25% and using content from Q2 is 25%. This allows training to focus on the current content set while mixing in training on previous content. When α is approximately 1, the next output can be introduced using the same process described above. After mixing the last output, the distortion removal model can be trained on all outputs with equal probability.

[0091] For each distortion method, the distortion removal model can be trained independently. For example, just to give a sense of scale, 3*10 -4 Learning rate. It should be understood that users can define different learning rates. The training content used for training is different for various content (e.g., images and videos). During training, content from one or more quality levels can be used depending on the input distortion method. For example, for JPEG and WebP, the quality levels can be Q1=40, Q2=30, Q3=20, with a valid range of 1 to 100, which refers to the compression factor. BPG, H.265 (HEVC), and H.264 use different metrics, whose values ​​range from 0 to 51. In BPG, the metric is called a quantization parameter (QP), and for H.265 and H.264, the metric is a constant rate factor (CRF). CRF is a more complex version of QP that can modify QP at the frame level. It should be understood that the present disclosure can use other distortion methods.

[0092] Now that example training techniques have been disclosed, example applications are disclosed herein. Target content (e.g., reconstructed content) may be received by a distortion removal model 114 at input 202, which has, for example, 3 channels, R, G, and B. In some embodiments, a quality level corresponding to the target content may be received. The quality level may be a different quality level from the trained adjusted distortion removal model. For example, the adjusted distortion removal model may have been trained on quality levels 20, 40, and 70. The target content distortion quality level may be in the range of 12 to 85. Nevertheless, the adjusted distortion removal model may be able to correct all target content.

[0093] The target content may be sent through an initial convolutional layer (e.g., 2DConv 3×3×64 204 and leaky ReLU 206). The target content may be sent through additional convolutional layers based on the quality level of the given target content. Thus, the target content may be corrected and may have the same size as the target content.

[0094] Figure 4 An operational flow chart illustrating an example process for training an initial distortion removal model using training content and applying the distortion removal model to target content according to one embodiment is illustrated. The operations of the various methods described herein are not necessarily limited to the order described or shown in the figures, and those skilled in the art will appreciate upon studying the present disclosure that variations in the order of the operations described herein are within the spirit and scope of the present disclosure.

[0095] In some cases, the operations and sub-operations of the flowchart may be performed by one or more of the components, elements, devices, components, and circuits of the system 100. This may include one or more of the following: Figure 1 and Fig. 9 Reference is made to server system 106, server 108, processor 112, storage device 110, and / or computing component 1000, and to the server system 106, server 108, processor 112, storage device 110, and / or computing component 1000, as well as any of the foregoing descriptions and at least with respect to the foregoing descriptions. Figure 1 and Fig. 9 In the examples of this type, the description of the flowchart may refer to the corresponding components, elements, etc., but whether or not an explicit reference is made, when studying the present disclosure, it will be understood when the corresponding components, elements, etc. can be used. In addition, it should be understood that these references do not necessarily limit the described methods to the specific components, elements, etc. referred to. Therefore, it should be understood that the aspects and features described above in conjunction with (sub)components, elements, devices, circuits, etc. (including variations thereof) can be applied to the various operations described in conjunction with the flowcharts without departing from the scope of the present disclosure.

[0096] At operation 402, training content may be received. The training content may include original content, reconstructed content based on distortion of the original content, and a distortion quality level of the reconstructed content. For example, the original content may be an original image. The reconstructed content may be a JPEG image. The corresponding distortion quality level may be about 40.

[0097] At operation 404, an initial distortion removal model may be received. The initial distortion removal model may include a progressive convolutional neural network including multiple branches based on distortion quality levels. Branches may be user defined based on a given distortion method. For example, different distortion methods may use different metrics, such as JPEG, BPG, H.264, and H.265. It should be appreciated that user defined branches may be used for multiple distortion methods.

[0098] The initial distortion removal model may include multiple feature maps in convolutional layers to inspect, identify, determine and analyze any distortions or artifacts in the reconstructed content, as well as activation functions.

[0099] At operation 406, an adjusted distortion removal model may be generated. The adjusted distortion removal model may be adjusted by training the initial distortion removal model using the training content. Continuing with the above example, using the original image as the final target and the distortion quality level as a parameter, the reconstructed image is trained to remove distortion and artifacts and generate a corrected image as described above. In some embodiments, a curriculum learning method may be used to improve the adjusted distortion removal model.

[0100] At operation 408, target content may be received. The target content may include reconstructed content and a corresponding distortion quality level. The distortion quality level of the target content may be different from the distortion quality level of the training content.

[0101] At operation 410, the adjusted distortion removal model can be applied to the target content. Thus, corrected content is generated by the adjusted distortion removal model. For example, target content below a distortion quality level of 20 can pass through a first set of convolutional layers to reach a first branch of the adjusted distortion removal model, while target content above a distortion quality level of 20 can pass through a first set of convolutional layers and a second set of convolutional layers to reach a second branch of the adjusted distortion removal model. In some embodiments, the corrected target content is presented to a user.

[0102] 5-8 illustrate the application of the techniques disclosed herein to various distortion methods. Figure 5A-5F illustrates facial images from the Kodak PhotoCD dataset with various effects applied. Figure 5A and Figure 5B , depicts a JPEG compressed image with lossy quality level 30. Figure 5C and Figure 5D The original image is illustrated. Figure 5E and Fig. 5F The use of the progressive distortion removal model disclosed in this article is described. As described, due to the removal of compression artifacts, Figure 5E and Fig. 5F Appears to be low pixel.

[0103] Figures 6A-6F Illustrated is an image of a lighthouse from the Kodak PhotoCD dataset. Fig. 6A and Figure 6B , depicts the original image. Figure 6C and Fig.6D Illustrated is a WebP compressed image with lossy quality level 30 and in-loop filter turned off. Fig. 6E and Fig. 6F Illustrated is a WebP compressed image with lossy quality level 30 and in-loop filter turned on. Figure 6G and Figure 6H Explained in Fig. 6E and Fig. 6F The progressive distortion removal model disclosed in this paper is used on the filtered WebP image. Fig. 6E and Fig. 6F In comparison, some blurriness / color fuzziness is mitigated due to the removal of compression artifacts.

[0104] Figure 7A-7H Illustrated is a frame with mountains from the Xiph.org video test media dataset. Fig. 7A and Figure 7B , depicts a frame compressed by H.264 with a constant rate factor (CRF) of 50 and a closed in-loop filter. Figure 7C and Fig.7D The original frame is illustrated. Fig. 7E and Figure 7F An H.264 compressed frame with a constant rate factor (CRF) of 50 and an in-loop filter turned on is illustrated. Figure 7G and Figure 7H Explained in Fig. 7A and Figure 7B The progressive distortion removal model disclosed in this paper is used on the unfiltered H.264 frames of Fig. 7A and Figure 7B In comparison, Figure 7G and Figure 7H In the image above, blurriness and graininess are less noticeable, especially in the rightmost image.

[0105] Figure 8A-8B Illustrated frame of Stockholm from the Xiph.org video test media dataset. Fig. 8A and Figure 8B, depicts a frame compressed by H.264 with a constant rate factor (CRF) of 56 and a closed in-loop filter. Figure 8C and Fig.8D The original frame is illustrated. Fig. 8E and Figure 8F A frame of H.264 compression with a constant rate factor (CRF) of 56 and an in-loop filter turned on is illustrated. Figure 8G and Figure 8H Explained in Fig. 8A and Figure 8B The progressive distortion removal model disclosed in this paper is used on the unfiltered H.264 frames of Fig. 8A and Figure 8B In comparison, Figure 8G and Figure 8H In the image processing, blurriness, edges and pixelation are reduced.

[0106] Fig. 9 An original image and two filtered images are illustrated. The top image illustrates the original frame. The middle image illustrates the effect of the H.265 compressed frame. The bottom image illustrates the effect of using the progressive distortion removal model disclosed herein on the original image. Blurriness, edges, and pixelation are reduced compared to the middle image.

[0107] Existing codecs can be improved by replacing existing distortion removal techniques and not changing the distortion itself. In an embodiment, the currently disclosed technology can be implemented as a stand-alone program. In addition to improving quality compared to standard filters, the network is also applied in real time. Therefore, there is no impact on the running time of existing decoders. The currently disclosed technology has the advantage of being able to handle multiple and untrained distortion quality levels in a better way, because the distortion quality level is incorporated into the network architecture and is treated as an additional parameter during training and testing. The method disclosed in the present invention does not need to train multiple models for each different distortion quality level, but processes multiple distortion quality levels in a single training model without training all distortion quality levels. Therefore, this method can improve performance, as well as save model training time, thereby improving efficiency and flexibility. It should be understood that one or more of the techniques described above can be used alone or in combination to train an initial distortion removal model.

[0108] As used herein, the term component can describe a given functional unit that can be performed according to one or more embodiments of the technology disclosed herein. As used herein, any form of hardware, software, or a combination thereof can be used to implement the component. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms can be implemented to form the component. In an embodiment, the various components described herein can be implemented as discrete components, or the functions and features described can be partially or completely shared between one or more components. In other words, after reading this specification, it is obvious to those of ordinary skill in the art that the various features and functions described herein can be implemented in any given application, and can be implemented in one or more separate or shared components in various combinations and arrangements. As used herein, the term engine can describe a batch of components configured to perform one or more specific tasks. Although various features or functional elements can be described or declared separately as separate components or engines, it will be understood by those of ordinary skill in the art that these features and functions can be shared between one or more common software and hardware elements, and such descriptions shall not require or imply the use of separate hardware or software components to implement these features or functions.

[0109] Where an engine, component, or component of a technique is implemented in whole or in part using software, in one embodiment, these software elements may be implemented to operate with a computing or processing component capable of performing the functions described with respect thereto. Fig.10 One such example computing component is shown in . Various embodiments are described based on this example computing component 1000. After reading this description, it will be apparent to a person skilled in the relevant art how to implement the technology using other computing components or architectures.

[0110] Reference now Fig.10 , computing component 1000 may represent computing or processing capabilities such as found within desktops, laptops, and notebook computers; handheld computing devices (PDAs, smart phones, cell phones, PDAs, etc.); mainframes, supercomputers, workstations, or servers; or any other type of special-purpose or general-purpose computing device that may be desirable or appropriate for a given application or environment. Computing component 1000 may also represent computing capabilities embedded within or otherwise available to a given device. For example, computing components may be found in other electronic devices such as digital cameras, navigation systems, cell phones, portable computing devices, modems, routers, WAPs, terminals, and other electronic devices that may include some form of processing capabilities.

[0111] The computing component 1000 may include, for example, one or more processors, controllers, control components, or other processing devices, such as a processor 1004. The processor 1004 may be implemented using a general or special processing engine, such as a microprocessor, controller, or other control logic. In the illustrated example, the processor 1004 is connected to the bus 1002, but any communication medium may be used to facilitate interaction with other components of the computing component 1000 or for external communication.

[0112] The computing component 1000 may also include one or more memory components, referred to herein as main memory 1008. For example, preferably, a random access memory (RAM) or other dynamic memory may be used to store information and instructions to be executed by the processor 1004. The main memory 1008 may also be used to store temporary variables or other intermediate information during the execution of instructions by the processor 1004. The computing component 1000 may also include a read-only memory ("ROM") or other static storage device coupled to the bus 1002 for storing static information and instructions for the processor 1004.

[0113] The computing component 1000 may also include one or more forms of information storage devices 1010, which may include, for example, a media drive 1012 and a storage unit interface 1020. The media drive 1012 may include a drive or other mechanism to support fixed or removable storage media 1014. For example, a hard drive, a floppy disk drive, a tape drive, an optical drive, a CD or DVD drive (R or RW), or other removable or fixed media drive may be provided. Thus, the storage media 1014 may include, for example, a hard disk, a floppy disk, a tape, a cassette, an optical disk, a CD or DVD, or other fixed or removable media that is read, written, or accessed by a media drive. As these examples illustrate, the storage media 1014 may include a computer-usable storage medium having computer software or data stored therein.

[0114] In alternative embodiments, information storage mechanism 1010 may include other similar means for allowing computer programs or other instructions or data to be loaded into computing component 1000. Such means may include, for example, fixed or removable storage units 1022 and interfaces 1020. Examples of such storage units 1022 and interfaces 1020 may include program cartridges and cartridge interfaces, removable memory (e.g., flash memory or other removable memory components) and memory slots, PCMCIA slots and cards, and other fixed or removable storage units 1022 and interfaces 1020 that allow software and data to be transferred from storage unit 1022 to computing component 1000.

[0115] The computing component 1000 may also include a communication interface 1024. The communication interface 1024 may be used to allow software and data to be transmitted between the computing component 1000 and an external device. Examples of the communication interface 1024 may include a modem or soft modem, a network interface (such as Ethernet, a network interface card, WiMedia, IEEE 802.XX or other interface), a communication port (e.g., a USB port, an IR port, an RS232 port, a Bluetooth@ interface or other port) or other communication interface. Software and data transmitted via the communication interface 1024 may generally be transmitted on signals, which may be electronic, electromagnetic (including optical) or other signals that can be exchanged by a given communication interface 1024. These signals may be provided to the communication interface 1024 via a channel 1028. The channel 1028 may transmit signals and may be implemented using a wired or wireless communication medium. Some examples of channels may include telephone lines, cellular links, RF links, optical links, network interfaces, local or wide area networks, and other wired or wireless communication channels.

[0116] In this document, the terms "computer program medium" and "computer-usable medium" are generally used to refer to media / media, such as memory 1008, storage unit 1020, media 1014, and channel 1028. These and other various forms of computer program media / media or computer-usable media / media may involve transmitting one or more sequences of one or more instructions to a processing device for execution. These instructions embodied on the media / media are generally referred to as "computer program code" or "computer program products" (which may be grouped in the form of a computer program or in other groupings). When executed, such instructions may enable computing component 1000 to perform features or functions of the disclosed technology as discussed herein.

[0117] Although various embodiments of the disclosed technology have been described above, it should be understood that they are presented only in an exemplary manner, rather than by way of limitation. Similarly, various figures may depict example architectures or other configurations for the disclosed technology, which are completed to help understand the features and functions that may be included in the disclosed technology. The disclosed technology is not limited to the illustrated example architectures or configurations, but various alternative architectures and configurations may be used to implement the desired features. In fact, it will be apparent to those skilled in the art how to implement alternative functions, logical or physical parts (partitions) and configurations to implement the desired features of the technology disclosed herein. In addition, multiple different component names other than those depicted herein may be applied to various parts (partitions). Additionally, with respect to flow charts, operational descriptions, and method claims, the order in which the steps are presented herein should not be mandatory to implement various embodiments to perform the functions in the same order, unless otherwise indicated by the context.

[0118] Although the disclosed technology is described above according to various exemplary embodiments and implementations, it should be understood that the various features, aspects, and functions described in one or more individual embodiments are not limited in their applicability by the specific embodiments in which they are described, but can be applied alone or in various combinations to one or more other embodiments of the disclosed technology, whether or not these embodiments are described and whether or not these features are presented as part of the described embodiments. Therefore, the breadth and scope of the technology disclosed herein should not be limited by any of the above exemplary embodiments.

[0119] Unless expressly stated otherwise, the terms and phrases used in this document and their variations should be interpreted as open ended and not limiting. As an example of the foregoing: the term "including" should be interpreted as meaning "including but not limited to" and the like; the term "example" is used to provide an illustrative example of the item in question, rather than an exhaustive or limiting list thereof; the term "a / an / an" should be interpreted as meaning "at least one", "one or more" and the like; adjectives such as "conventional", "traditional", "normal", "standard", "known" and terms of similar meaning should not be interpreted as limiting the items described to a given time period or to items available at a given time, but should be read to include conventional, traditional, normal or standard technologies that may be available or known now or in the future. Similarly, where the technology referred to in this document will be obvious or known to a person of ordinary skill in the art, such technology includes the technology that is obvious or known to a person of ordinary skill in the art at any time now or in the future.

[0120] In some cases, the presence of expansive words and phrases such as "one or more," "at least," "but not limited to," or other similar phrases should not be understood to mean that a narrower case is intended or required in the possible absence of such expansive phrases. The use of the term "component" does not imply that the components or functionality described or claimed as part of the component are all configured in a common package. In fact, any or all of the various components of the component, whether control logic or other components, may be combined in a single package or maintained separately, and may be further distributed in multiple groups or packages or across multiple locations.

[0121] Additionally, the various embodiments described herein are described in terms of exemplary block diagrams, flow charts, and other illustrations. As will become apparent to one of ordinary skill in the art after reading this document, the illustrated embodiments and their various alternatives may be implemented without limiting the illustrated examples. For example, the block diagrams and their accompanying descriptions should not be interpreted as mandating a specific architecture or configuration.

Claims

1. A computer-implemented method comprising: Accept training content, the training content includes: original content, reconstructed content, and a plurality of training distortion quality levels corresponding to the reconstructed content, wherein the reconstructed content is derived from the distorted original content; receiving an initial distortion removal model; An adjusted distortion removal model is generated by training the initial distortion removal model using the training content.

2. The computer-implemented method of claim 1 , further comprising: receiving target content, wherein the target content has one or more target distortion quality levels, wherein the plurality of training distortion quality levels includes at least one different value compared to the one or more target distortion quality levels; and The adjusted distortion removal model is applied to the target content to generate corrected target content. 3 . The computer-implemented method of claim 2 , wherein the adjusted distortion removal model comprises one or more user-defined output branches based on the one or more target distortion quality levels.

4. The computer-implemented method of claim 1 , wherein: The initial distortion removal model includes a first branch and a second branch, wherein the training content corresponding to the first group of the first branch passes through a first group of convolutional layers, and the training content corresponding to the second group of the second branch passes through the first group of convolutional layers and a second group of convolutional layers.

5. The computer-implemented method of claim 4, wherein: Using the training content to train the initial distortion removal model includes: Applying the training content of the first group to the first group of convolutional layers; Applying at least one training content of the first group of training contents to the second group of convolutional layers; Applying the training content of the second group to the first group of convolutional layers and the second group of convolutional layers; and When the second branch is adjusted, the first branch and the second branch are trained with equal weights applied to the first branch and the second branch.

6. The computer-implemented method of claim 1, wherein the initial distortion removal model or the adjusted distortion removal model comprises a convolutional neural network.

7. The computer-implemented method of claim 1, wherein the initial distortion removal model or the adjusted distortion removal model comprises an activation function.

8. The computer-implemented method of claim 1, wherein the adjusted distortion removal model is trained to remove distortion from the target content.

9. The computer-implemented method of claim 1, wherein the target content comprises one or more of an image or a video.

10. A computer-implemented method comprising: receiving target content, wherein the target content has a plurality of target distortion quality levels; receiving an adjusted distortion removal model, the adjusted distortion removal model having been adjusted by training an initial distortion removal model using training content, wherein the training content comprises original content, reconstructed content derived from the distorted original content, and a plurality of training distortion quality levels corresponding to the reconstructed content; as well as The adjusted distortion removal model is applied to the target content to generate corrected target content.