Neural network assisted removal of video compression artifacts

Abstract

The data compression system may include a neural compression artifact removal module (NCARM) configured to receive compressible data and output data with compression artifacts removed. A lossy compression module is configured to at least one of receive and transmit data to the neural compression artifact removal module, and a restoration module is configured to at least one of receive and transmit data to the neural compression artifact removal module. In some embodiments, the neural compression artifact removal module transmits data to the lossy compression module. Alternatively, the neural compression artifact removal module may receive data from the restoration module and / or may receive data from the lossy compression module. Many lossy data compression schemes, including commonly available audio and video compression methods, may benefit from artifact removal.

Description

[Technical field]

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 289,454, entitled "Neural Network Assisted Removal of Video Compression Artifacts," filed on December 14, 2021, the entirety of which is incorporated herein by reference.

[0002] The present disclosure relates to a system that utilizes neural networks to remove lossy compression artifacts to reduce bandwidth requirements and improve image quality. [Background technology]

[0003] Data compression involves encoding information using fewer bits than in its original representation. Typically, data compression is performed in two separate steps: encoding and decoding. During the encoding step, an input stream is transformed into a coded representation according to a compression scheme. During the decoding step, an inverse transform is applied to restore the coded representation to, or nearly restore, the original input stream. A special case of data compression is transcoding, where data compressed by a first compression scheme is decoded and then re-encoded using an encoder from a second compression scheme.

[0004] Data compression can be either lossless or lossy. Lossless compression reduces the number of bits of information by identifying and removing statistical redundancies. During lossless compression, no information is actually lost and all bits from the original representation can be restored during the decoding (reconstruction) process. In contrast, lossy compression does not retain all bits of the original representation during encoding, but instead removes bits that are not useful or important according to some metric. This process can significantly reduce the overall number of bits, at the expense of quality degradation. Unfortunately, lossy compression results in compression artifacts. Examples of compression artifacts include blocking artifacts, cosine or wavelet transform artifacts, quantization artifacts, aliasing artifacts, etc.

[0005] Digital image or video cameras typically require a digital image processing pipeline to convert the signals received by the image sensor into a usable image through the use of image processing algorithms and filters. Due to the large amount of digital information involved, data encoding, decoding, and transcoding using lossy compression schemes are often used to support connections with streaming devices. Summary of the Invention [Problem to be solved by the invention]

[0006] To reduce bandwidth requirements and improve image quality, a system and method is needed that removes lossy compression artifacts. [Means for solving the problem]

[0007] In order to solve the above problems, the present invention provides a data compression system and a camera data compression system as claimed in the claims of the present application.

[0008] In the following figures, reference is made in describing non-limiting and non-exhaustive embodiments of the present disclosure, like reference characters refer to like parts throughout the various figures unless otherwise specified. [Brief description of the drawings]

[0009] [Figure 1A] FIG. 1 illustrates neural network assisted compression artifact removal before lossy compression and decompression. [Figure 1B] FIG. 1 illustrates neural network assisted compression artifact removal after lossy compression and decompression. [Figure 1C] FIG. 1 illustrates neural network assisted compression artifact removal after lossy compression and before decompression. [Figure 2A] FIG. 1 illustrates a system and process for neural network assisted compression artifact removal and adjustment. [Figure 2B] FIG. 1 illustrates a lossy compression system and process for neural network assisted encoding and conditioning. [Figure 2C] FIG. 1 illustrates a lossy compression system and process for neural network assisted decoding and conditioning. [Figure 2D] FIG. 1 illustrates a lossy compression system and process for modifying high quality data streams using neural network assisted decoding and conditioning. [Figure 2E] FIG. 1 illustrates a system and process for neural network assisted decoding followed by neural network assisted compression artifact removal. [Figure 2F] FIG. 1 illustrates a decoder system and process that supports neural network assisted compression artifact removal. [Figure 2G] FIG. 1 illustrates an encoder system and process including neural network assisted compression artifact removal. [Figure 2H]FIG. 1 illustrates an encoder system and process including neural network assisted compression artifact removal followed by encoding. [Figure 2I] FIG. 1 illustrates a transcoding system and process including neural network assisted compression artifact removal and integrated decoding followed by encoding. [Figure 2J] FIG. 1 illustrates a transcoding system and process for neural network assisted compression artifact removal and integrated encoding after decoding. [Figure 2K] FIG. 1 illustrates a transcoding system and process for neural network assisted compression artifact removal and separate encoding after decoding. [Figure 3A] FIG. 1 illustrates a multiple camera system supported by a transcoding system and process for neural network assisted compression artifact removal. [Figure 3B] FIG. 1 illustrates a multiple camera system where each camera is supported by a transcoding system and process for neural network assisted compression artifact removal. [Figure 3C] FIG. 1 illustrates a cloud system including neural network assisted compression artifact removal, encoding and decoding, and cloud storage. [Figure 3D] FIG. 1 illustrates a multiple user device system with each user device supported by a decoding system and process for neural network assisted compression artifact removal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In some embodiments described below, methods, processing schemes, and systems for improving neural network processing are described. As disclosed in more detail, the improved neural network processing includes a data compression system that can include a neural compression artifact removal module (NCARM) configured to receive compressible data and output data with compression artifacts removed. The lossy compression module can be configured to at least one of receive and transmit data to the NCARM, and the decompression module can be configured to at least one of receive and transmit data to the NCARM. In some embodiments, the NCARM transmits data to the lossy compression module. Alternatively, the NCARM can receive data from the decompression module and / or can receive data from the lossy compression module. As will be appreciated, any lossy compressible data may be processed by the described system, but in some embodiments, the data is at least one of audio and video.

[0011] In another embodiment, the data compression system may include a neural compression artifact removal module (NCARM) forming part of at least one of the transcoder and the lossy compression module, the NCARM configured to receive compressible data and output data with compression artifacts removed, the data encoder and decoder are each coupled to the NCARM, and the NCARM adjustment module is configured to at least one of receive and transmit data to and from the NCARM.

[0012] In another embodiment, the data compression system may include a neural compression artifact removal module (NCARM) forming part of at least one of the transcoder, the lossy compression module, the encoding module, and the decoding module, the NCARM configured to receive compressible data and output data with compression artifacts removed. The data encoder and the decoder are each coupled to the NCARM. The NCARM neural network forms part of one of the data encoder and the decoder.

[0013] In yet another embodiment, the camera data compression system can include a camera and a Neural Compression Artifact Removal Module (NCARM) coupled to the camera. The NCARM is configured to receive the compressible data and output data with compression artifacts removed. In some embodiments, the NCARM is operable on the camera, while in other embodiments, the NCARM is operable on a VMS system or cloud that receives the compressible data from the camera.

[0014] 1A is a diagram illustrating neural network assisted compression artifact removal (NCARM) before lossy compression and decompression. In this embodiment, a system 100A receives a data input, which is first processed using a module 110A that provides neural network mediated compression artifact removal. The processed data is provided to a module 120A that provides lossy compression of the data. After storage, transmission, or streaming, a decompression module 130A can decode the compressed data, for example to allow further processing or playback by a user.

[0015] In one embodiment, the data can include, but is not limited to, a wide range of video, audio, streaming, sensor, or control data. Lossy compression is often used in some such applications due to its large input complexity and due to the high degree of redundancy in the data stream. Data processing according to this disclosure can benefit image quality, reduced file size and bandwidth requirements, and provide improved downstream machine or artificial intelligence (AI) application performance. In fact, lossy data compression reduces file size at the expense of some signal loss. In addition to signal loss, the compression process often also introduces undesirable artifacts. The use of neural network techniques also allows partial recovery of lost signals while removing compression artifacts. The described systems and methods can improve signal fidelity and further reduce file size or bandwidth requirements, thereby allowing aggressive compression without compromising the original signal.

[0016] As will be appreciated, a wide variety of compression schemes can be utilized in the systems described in this disclosure. For example, both intraframe (which utilizes single-frame image compression) and interframe video compression systems (which compress the content of a current frame using one or more previous and / or subsequent frames) can benefit from neural network mediated artifact removal. Common compression schemes include, but are not limited to, Motion JPEG (M-JPEG), MPEG-1 (CD, VCD), MPEG-2 (DVD), MPEG-4, and H.264 based compression (encoding) and decompression (decoding) schemes.

[0017] In one embodiment, the module providing Neural Network Assisted Compression Artifact Removal (NCARM) services is a neural network tuned (trained) to remove input complexity and compression artifacts while maintaining data fidelity. In the case of images and video, the network receives an image or a sequence of images as input and outputs an improved image or sequence of images. Depending on the system architecture, the NCARM processing module can be applied at the encoding stage, at the decoding stage, or at the transcoding stage. Furthermore, the module can be standalone or integrated with a decoder or encoder. Typically, the processing sub-module receives as input an uncoded data stream and removes data complexity from the signal so that 1) there are fewer artifacts and the encoding process is more effective, or 2) any artifacts resulting from the compression scheme are removed.

[0018] In some embodiments, tuning of the module providing neural network assisted compression artifact removal (NCARM) can be performed using manual or automatic parameters. This can be achieved through training of the neural network of the processing submodule, where some loss function is minimized or maximized. In some embodiments, the tuning module receives both the high fidelity signal as well as the encoded and decoded degraded signal. The tuning submodule adjusts the parameters of the processing submodule such that most of the original high fidelity signal is restored. In the absence of a "pair" of high fidelity / degraded signals, a reference high quality stream can be utilized. In this case, the tuning submodule does not attempt to restore the degraded stream to an identical copy of the high fidelity stream, but ensures that the statistical distribution of the degraded stream matches that of the high fidelity stream, such as by methods such as generative adversarial training.

[0019] As will be appreciated, various implementations of neural networks can be utilized. For example, neural networks can include fully convolutional, recurrent, generative adversarial, or deep convolutional networks. Convolutional neural networks are particularly useful for image processing applications described herein. Images can be pre-processed with convolutional pixel operations or fed into a trained convolutional neural network, preferably with minimal modifications. Processing can proceed through one or more convolutional layers, pooling layers, fully connected layers, and ends with an output suitable for encoding or decoding. In operation, one or more convolutional layers apply a convolutional operation to the input and convey the result to the next layer. After convolution, a local or global pooling layer combines the output to one or a small number of nodes in the next layer. Iterative convolutions or paired convolution / pooling iterations are possible. After neural network processing is complete, the output may be passed between neural networks to other local neural networks, or additionally or alternatively to neural network-based cloud-based processing for additional neural network-based modifications.

[0020] One neural network embodiment of particular utility is a fully convolutional and recurrent neural network. A fully convolutional and recurrent neural network consists of convolutional layers without the fully connected layers that are usually found at the end of the network. Advantageously, a fully convolutional neural network is image size agnostic and can accept images of any size as input for training or spot image correction. Recurrent behavior is provided by feeding at least a portion of the output back into the convolutional layer or to other connected neural networks.

[0021] Various neural networks can identify and improve data compression for many types of artifacts. For example, noise introduced due to camera sensors and scene lighting is a common artifact. This noise is not caused by the encoding process but contributes to signal complexity (file size / bandwidth) and quality. The introduced noise can be separated into two cases: 1) low compression where the noise is moderately represented in the compressed image artifacts and can be identified as "graininess"; 2) high compression where the noise is poorly represented in the compressed image and can be identified as irregular vertical or horizontal lines or grain patterns when viewed closely.

[0022] Many artifacts are represented by the use of quantization and basis functions. Quantization noise exists because compression schemes often represent data as a combination of basis functions (wavelets, discrete cosine, etc.). In limits, they can perfectly represent the image. However, many compression schemes reduce or eliminate high frequency components, which have little effect on human perception. When aggressively compressing a signal, high frequency components appear as small patches of horizontal, vertical, or grainy patterns. These are basis functions that approximate the original signal with some errors. Fortunately, by using neural networks and the described NCARM system and method, such artifact errors can be corrected.

[0023] Because many types of compression schemes aim to reuse as much information as possible, another type of artifact, known as blocking artifacts, occurs. One method takes a "patch" from the current or nearby frame and references it in multiple other regions. Since the patch never perfectly represents the other regions, some errors must be compensated for. In aggressive compression, this error compensation is traded off against file size. After compression and decompression, the resulting current image data contains small square patches in the image whose boundaries do not perfectly match their neighbors. Again, by utilizing neural networks and the described NCARM system and method, such artifact errors are corrected.

[0024] After compression and decompression, another type of artifact known as aliasing also occurs. Aliasing is the result of limited spatial sampling periods due to jagged or moire patterns in the applied signal. By utilizing neural networks and the described NCARM system and method, such artifact errors are corrected.

[0025] Artifacts can be identified automatically using machine intelligence techniques, or alternatively or additionally, by trained operators. The NCARM module is trained to identify and remove these artifacts by ensuring that they are well represented in the dataset. A team of data labelers can build a database of each artifact, which can be directly fed as training data to the NCARM module, or used to train an automatic "artifact classifier" algorithm that automates the labeling of newly acquired data. Additionally, for training purposes, these artifacts can be "pushed" into the training data by deliberately using aggressive compression on some material (by modifying the input data or compression parameters so that the desired artifacts predominate).

[0026] FIG. 1B illustrates neural network assisted compression artifact removal after lossy compression and decompression. This embodiment is considered a variation of the embodiment described in FIG. 1A, differing in the order of data processing by each module for neural network mediated compression artifact removal, lossy compression, and decompression. In this embodiment, system 100B receives a data input that is first processed using module 120B providing lossy compression of data. The lossy compressed data is then provided to decompression module 130B, which is capable of decoding the compressed data. The decompressed data is provided to module 110B providing neural network mediated compression artifact removal.

[0027] FIG. 1C illustrates neural network assisted compression artifact removal after lossy compression and before decompression. This embodiment is considered a variation of the embodiment described in FIG. 1A, differing in the order of data processing by each module for neural network mediated compression artifact removal, lossy compression, and decompression. In this embodiment, system 100C receives a data input that is first processed using module 120C providing lossy compression of data. The lossy compressed data is then provided to module 110C providing neural network mediated compression artifact removal. This data is provided to decompression module 130C, which is able to decode the compressed de-artifacted data.

[0028] 2A illustrates a system and process 200A for neural network assisted compression artifact removal and adjustment. As shown, data in the form of a high quality data stream 210A is provided to both an encoder 212A and a NCARM adjustment module 220A. The data stream encoded by the encoder 212A is provided to a NCARM transcoder 230. The NCARM transcoder 230 decodes the data using parameters specified by the training and NCARM adjustment module 220A, processes the data using a module providing neural network mediated compression artifact removal 216A, and encodes the data using an encoder 218A. The encoded data is provided to a decoder 240A and converted to a decoded stream 242A, which is also provided back to the NCARM adjustment module 220A to help improve long term system performance.

[0029] 2B illustrates a lossy compression system and process 200B for neural network assisted encoding and conditioning. As shown, data in the form of a high quality data stream 210B is provided to both a lossy compression module 218B and a NCARM conditioning module 220B. Within the lossy compression module 218B, the high quality data stream 210B is first provided to a NCARM encoder 216B and then to a decoder 214B. The NCARM encoder 216B encodes the data utilizing parameters specified by the training and NCARM conditioning module 220B. The encoded data in the lossy compression module 218B is converted to a decoded stream 242B, which is also fed back to the NCARM conditioning module 220B to help improve long term system performance.

[0030] 2C illustrates a lossy compression system and process 200C for neural network assisted decoding and conditioning. As shown, data in the form of a high quality data stream 210C is provided to both a lossy compression module 218C and a NCARM conditioning module 220C. Within the lossy compression module 218C, the high quality data stream 210C is first provided to an encoder 212C and then to a NCARM decoder 214C. The NCARM decoder 214C decodes the data utilizing parameters specified by the training and NCARM conditioning module 220C. The data encoded in the lossy compression module 218C is converted to a decoded stream 242C, which is also fed back to the NCARM conditioning module 220C to help improve long term system performance.

[0031] FIG. 2D illustrates a lossy compression system and process 200D for modifying a high quality data stream using neural network assisted decoding and tuning. As shown, data in the form of a high quality data stream 210D is provided to a lossy compression module 218D. Additionally, data in the form of a high quality reference data stream 211D is provided to an NCARM tuning module 220D. Within the lossy compression module 218D, the high quality data stream 210D is first provided to an encoder 212D and then to an NCARM decoder 214D. The NCARM decoder 214D decodes the data using parameters specified by the training and NCARM tuning module 220D. The data encoded in the lossy compression module 218D is converted to a decoded stream 242D, which is also fed back to the NCARM tuning module 220D to help improve long term system performance.

[0032] 2E illustrates a system and process 200E for neural network assisted decoding followed by neural network assisted compression artifact removal. As shown, data in the form of an encoded data stream 210E is provided to an NCARM decoding module 230E. Within the NCARM decoding module 230E, the encoded data stream 210E is first provided to a decoder 212E and then to an NCARMNN module 216E. Data from the NCARM decoding module 230E is converted to a decoded stream 242E.

[0033] 2F illustrates a decoder system and process 200F that supports neural network assisted compression artifact removal. As shown, data in the form of an encoded data stream 210F is provided to an NCARM decoding module 230F. Within the NCARM decoding module 230F, the encoded data stream 210F is first provided to a decoder 212F that supports an internal NCARMNN module 216F. Data from the NCARM decoding module 230F is converted to a decoded stream 242F.

[0034] 2G illustrates an encoder system and process 200G including neural network assisted compression artifact removal. As shown, data in the form of an uncompressed data stream 211G is provided to an NCARM encoding module 230G. Within the NCARM encoding module 230G, the uncompressed data stream 211G is first provided to an encoder 212G that supports an internal NCARMNN module 216G. Data from the NCARM encoding module 230G is converted to an encoded stream 242G.

[0035] 2H illustrates an encoder system and process 200H including neural network assisted compression artifact removal followed by encoding. As shown, data in the form of an uncompressed data stream 211H is provided to an NCARM encoding module 230H. Within the NCARM encoding module 230H, the uncompressed data stream 211H is first provided to an internal NCARMNN module 216H and then provided to an encoder 212H. Data from the NCARM encoding module 230H is converted to an encoded stream 242H.

[0036] 2I illustrates a transcoding system and process 200I including neural network assisted compression artifact removal followed by encoding and integrated decoding. As shown, data in the form of an encoded data stream 210I is provided to an NCARM transcoding module 230I. Within the NCARM transcoding module 230I, the encoded data stream 210I is first provided to a decoder 212I having an internal NCARMNN module 216I and then provided to an encoder 212I. Data from the NCARM transcoding module 230I is converted to an encoded stream 242I.

[0037] 2J illustrates a transcoding system and process 200J for post-decoding neural network assisted compression artifact removal and integrated encoding. As shown, data in the form of an encoded data stream 210J is provided to an NCARM transcode module 230J. Within the NCARM transcode module 230J, the encoded data stream 210J is first provided to a decoder 212J and then to an encoder 212J having an internal NCARMNN module 216J. Data from the NCARM transcode module 230J is converted to an encoded stream 242J.

[0038] 2K is a diagram illustrating a transcoding system and process 200K for neural network assisted compression artifact removal and separate encoding after decoding. As shown, data in the form of encoded data stream 210K is provided to NCARM transcode module 230K. Within NCARM transcode module 230K, encoded data stream 210K is first provided to decoder 212K, then to internal NCARMNN module 216K, and then to encoder 212K. Data from NCARM transcode module 230K is converted to encoded stream 242K.

[0039] FIG. 3A illustrates a multiple camera system 300A supported by a transcoding system and process for neural network assisted compression artifact removal. As shown, multiple edge cameras (1, 2, ...N) provide video and any other data to a cloud / VMS system 352A. This data is processed by an encoder module associated with each edge camera. Cloud / VMS system 352A includes a NCARM transcode module 316A and an associated storage 350A that receives data from NCARM transcode module 316A. As one alternative, data from NCARM transcode module 316A is provided as real-time streaming viewing to one or more end user devices. As another alternative, data from storage 350A is provided as archived data viewing to one or more end user devices.

[0040] FIG. 3B illustrates a multiple camera system 300B where each camera is supported by a transcoding system and process for neural network assisted compression artifact removal. As shown, multiple edge cameras (1, 2, ...N) provide video and any other data to a cloud / VMS system 352B. This data is processed by a NCARM encoder module associated with each edge camera. The cloud / VMS system 352B includes an attached storage 350B. As one alternative, data from the NCARM module from the edge cameras is provided as real-time streaming viewing to one or more end user devices. As another alternative, data from the storage 350B provided by the NCARM module from the edge cameras is provided as archived data viewing to one or more end user devices.

[0041] 3C is a diagram illustrating a cloud system 300C including neural network assisted compression artifact removal, encoding and decoding, and cloud storage. As shown, audio, video, or other compressible data contained in or provided to the cloud / VMS system 352C is communicated to a NCARM transcoding module 316C and connected storage 350C that receives data from the NCARM transcoding module 316C. This allows for efficient conversion of lossy data between various audio or video compression schemes.

[0042] 3D illustrates a multiple user device system 300D, with each user device supported by a decoding system and process for neural network assisted compression artifact removal. As shown, data stored or provided to cloud / VMS system 352D is maintained in storage 350D. Audio, video, or other compressible data from storage 350D is also provided to one or more end user devices that include NCARM decoding capabilities.

[0043] As can be appreciated from the above, a wide range of still or video cameras can benefit from the use of the neural network assisted image or video processing systems and methods described in this disclosure. Camera types can include, but are not limited to, conventional digital single-lens reflex (DSLR) cameras with still or video capabilities, smartphones, tablet cameras, or laptop cameras, dedicated video cameras, webcams, or security cameras. In some embodiments, specialized cameras can be used, such as infrared cameras, thermal imaging cameras, millimeter wave imaging systems, x-ray or other radiation imagers, etc. Embodiments can also include cameras with sensors capable of detecting infrared, ultraviolet, or other wavelengths, enabling hyperspectral imaging.

[0044] A camera can be a standalone, portable, or fixed system. Typically, a camera includes a processor, memory, an image sensor, a communication interface, a camera optical actuator system, and memory storage. The processor controls the overall operation of the camera, such as the operation of the camera optical sensor system, and the available communication interfaces. The camera optical sensor system controls the operation of the camera, such as exposure control of the images captured by the image sensor. The camera optical sensor system can include a fixed lens system or an adjustable lens system (e.g., zoom and autofocus capabilities). The camera can support memory storage systems, such as removable memory cards, wired USB, wireless data transfer systems, and the like.

[0045] In some embodiments, the neural network processing can occur after transfer of the audio, video, or other compressible data to a remote computing resource, including a dedicated neural network processing system, laptop, PC, server, or cloud, hi other embodiments, the neural network processing can occur in-camera using optimization software, neural processing chips, dedicated ASICs, custom integrated circuits, or programmable FPGA systems.

[0046] As can be appreciated from the above, the camera system and method described herein may interface with devices such as servers, desktop computers, laptops, tablets, or smartphones locally or through a connection to either a wired or wireless connectivity subsystem. Data and control signals may be received, generated, or transmitted between various external data sources such as wireless networks, personal area networks, cellular networks, the Internet, or cloud-mediated data sources. Additionally, local data of user-specified preferences or protocols may be stored by the source of local data (e.g., hard drive, solid state drive, flash memory, or any other suitable memory including dynamic memory such as SRAM or DRAM). In certain embodiments, multiple communication systems may be provided. For example, a separate 4G cellular connection may be used in addition to a direct Wi-Fi connection (802.11b / g / n).

[0047] Also, embodiments connecting to a remote server may be implemented in a cloud computing environment. Cloud computing can be defined as a model that enables ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that are rapidly provisioned through virtualization, released with minimal management effort or interaction with a service provider, and then scaled appropriately. The cloud model can be configured with a variety of characteristics (e.g., on-demand self-service, broad network access, resource pooling, rapid scalability, metered services, etc.), service models (e.g., Software as a Service ("SaaS"), Platform as a Service ("PaaS"), Infrastructure as a Service ("IaaS"), and deployment models (e.g., private cloud, community cloud, public cloud, hybrid cloud, etc.).

[0048] Throughout this specification, the phrases "in one embodiment," "embodiment," "an example," or "example" mean that a particular feature, structure, or characteristic described in connection with an embodiment or example is included in at least one embodiment of the present disclosure. Thus, the appearance of the phrases "in one embodiment," "in an embodiment," "an example," or "example" in various places throughout this specification do not necessarily all refer to the same embodiment or example. Furthermore, particular features, structures, databases, or characteristics may be combined in any suitable combinations and / or subcombinations in one or more embodiments or examples. Additionally, the drawings accompanying this specification are for illustrative purposes for persons skilled in the art, and the drawings are not necessarily drawn to scale.

[0049] The flowcharts and block diagrams in the accompanying drawings are intended to illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of code that includes one or more executable instructions for implementing a specified logical function. It should be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or a combination of dedicated hardware and computer instructions. Furthermore, these computer program instructions may be stored in a computer-readable medium that can instruct a computer or other programmable data processing apparatus to function in a specific manner, i.e., the instructions stored in the computer-readable medium realize a product that includes instruction means for implementing the function / operation specified in the block or blocks of the flowcharts and / or block diagrams.

[0050] Embodiments according to the present disclosure may be embodied as an apparatus, a method, or a computer program product. Thus, the present disclosure may be realized in the form of an embodiment that is entirely comprised of hardware, an embodiment that is entirely comprised of software (including firmware, resident software, microcode, etc.), or an embodiment that combines software and hardware aspects, collectively referred to herein as a "circuit," "module," or "system." Furthermore, embodiments of the present disclosure may also be realized in the form of a computer program product embodied in any tangible medium having computer usable program code embodied in the medium.

[0051] Any combination of one or more computer usable or computer readable media may be utilized. For example, the computer readable medium may include one or more of a portable computer diskette, a hard disk, a random access memory (RAM) device, a read-only memory (ROM) device, an erasable programmable read-only memory (EPROM or flash memory) device, a portable compact disc read-only memory (CD-ROM), an optical storage device, and a magnetic storage device. Computer program code for carrying out the operations of the present disclosure may be written in any combination of one or more programming languages. Such code may be compiled from source code into a computer readable assembly language or machine code suitable for the device or computer on which the code is to be executed.

[0052] Based on the teachings presented in the above description and the associated drawings, one skilled in the art can conceive of numerous modifications and other embodiments of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and these modifications and other embodiments are clearly within the scope of the appended claims. Also, other embodiments of the present invention can be practiced in the absence of elements / steps not specifically disclosed herein. [Explanation of symbols]

[0053] 100A~C System 200A SYSTEM AND PROCESS FOR NEURAL NETWORK AIDED COMPRESSION ARTIFACT REMOVAL AND ADJUSTMENT 200B Lossy compression system and process for neural network assisted encoding and conditioning 200C Lossy compression system and process for neural network assisted decoding and conditioning 200D Lossy compression system and process for modifying high quality data streams using neural network assisted decoding and conditioning 200E SYSTEM AND PROCESS FOR NEURAL NETWORK AIDED DECODING SUBSEQUENTLY NEURAL NETWORK AIDED COMPRESSION ARTIFACT REMOVAL 200F Decoder system and process supporting neural network assisted compression artifact removal 200G Encoder System and Process Including Neural Network Assisted Compression Artifact Removal 200H Encoder system and process including neural network assisted compression artifact removal followed by encoding 200I Transcoding system and process including neural network assisted compression artifact removal followed by encoding and integrated decoding 200J Transcoding system and process for post-decoding neural network assisted compression artifact removal and integrated encoding Transcoding system and process for neural network assisted compression artifact removal and separate encoding after 200K decoding 300A Multi-camera system supported by transcoding system and process for neural network assisted compression artifact removal 300B Multi-camera system with each camera supported by a transcoding system and process for neural network assisted compression artifact removal 300C Cloud system including neural network assisted compression artifact removal, encoding and decoding, and cloud storage Multiple user device system, each user device supported by a neural network assisted decoding system and process for compression artifact removal

Claims

1. a neural compression artifact removal module (NCARM) configured to receive compressible data and output data with compression artifacts removed; a lossy compression module configured to at least one of receive and transmit data to and from the neural compression artifact removal module; a decompression module configured to at least one of receive and transmit data to and from the neural compression artifact removal module; 1. A data compression system comprising:

2. 2. The data compression system of claim 1, wherein the neural compression artifact removal module transmits data to the lossy compression module.

3. 2. The data compression system of claim 1, wherein the neural compression artifact removal module receives data from the decompression module.

4. 2. The data compression system of claim 1, wherein the neural compression artifact removal module receives data from the lossy compression module.

5. 2. The data compression system of claim 1, wherein the data is at least one of audio and video.

6. a neural compression artifact removal module (NCARM) forming part of at least one of the transcoder and lossy compression modules and configured to receive compressible data and output data with compression artifacts removed; a data encoder and a decoder respectively coupled to the neural compression artifact removal module; an NCARM adjustment module configured to at least one of receive and transmit data to and from said neural compression artifact removal module; 1. A data compression system comprising:

7. 7. The data compression system of claim 6, wherein the lossy compression module receives high quality streaming data.

8. 7. The data compression system of claim 6, wherein said NCARM adjustment module receives high quality reference streaming data.

9. 7. The data compression system of claim 6, wherein the lossy compression module outputs decoded streaming data.

10. 7. The data compression system of claim 6, wherein the data is at least one of audio and video.

11. a neural compression artifact removal module (NCARM) forming part of at least one of the transcoder, the lossy compression module, the encoding module, and the decoding module, configured to receive compressible data and to output data with compression artifacts removed; a data encoder and a decoder respectively coupled to the neural compression artifact removal module; Including, A data compression system in which an NCARM neural network forms part of one of said data encoders and decoders.

12. 12. The data compression system of claim 11, wherein the lossy compression module receives encoded streaming data.

13. 12. The data compression system of claim 11, wherein the NCARM adjustment module receives uncompressed streaming data.

14. 12. The data compression system of claim 11, wherein the lossy compression module outputs at least one of encoded streaming data and decoded streaming data.

15. 12. The data compression system of claim 11, wherein the data is at least one of audio and video.

16. A camera and a Neural Compression Artifact Removal Module (NCARM) coupled to the camera and configured to receive compressible data and output compression artifact-removed data; 1. A camera data compression system comprising:

17. 20. The camera data compression system of claim 16, wherein the neural compression artifact removal module is operable on-camera.

18. 20. The camera data compression system of claim 16, wherein the neural compression artifact removal module is operable on a cloud / VMS system that receives compressible data from the camera.

Images