Spatial-frequency transform-based image correction using inter-channel correlation information

Abstract

The present disclosure relates to image modification, such as image correction, where the processing is based at least in part on a neural network. In particular, the image modification includes multi-channel processing where a primary channel is processed separately and a secondary channel is processed based on the processed primary channel. The primary channel is processed based on a first spatial frequency transformation to obtain a transformed primary channel, and the secondary channel is processed based on a second spatial frequency transformation to obtain a transformed secondary channel. The transformed primary channel is processed by a first neural network to obtain a modified transformed primary channel, and the transformed secondary channel is processed by a second neural network based on the transformed primary channel to obtain a modified transformed secondary channel.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present disclosure relate generally to the field of databased encoding and decoding on neural network architectures. In particular, some embodiments relate to methods and apparatus for such encoding and decoding of images and / or videos from bitstreams using multiple processing layers, in particular for image enhancement. [Background technology]

[0002] Hybrid image and video codecs have been used for decades to compress image and video data. In such codecs, signals are typically encoded block-by-block by predicting the block and further coding only the difference between the original block and its prediction. In particular, such coding may include transforming, quantizing, and generating a bitstream, and typically includes some entropy coding. Typically, the three components of a hybrid coding method, namely transforming, quantizing, and entropy coding, are optimized separately. Modern video compression standards such as High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC) also use a transformed representation to code the residual signal after prediction.

[0003] In recent years, neural network architectures have been applied to image and / or video coding. In general, these neural network (NN)-based approaches can be applied to image and video coding in a variety of different ways. For example, some end-to-end optimized image or video coding frameworks have been discussed. Furthermore, deep learning has been used to determine or optimize some parts of the end-to-end coding framework, such as prediction parameter selection or compression. Moreover, some neural network-based approaches have also been discussed for use in hybrid image and video coding frameworks, for example, for implementation as trained deep learning models for intra- or inter-prediction in image or video coding.

[0004] The end-to-end optimized image or video coding applications discussed above have in common that they generate some feature map data to be communicated between an encoder and a decoder.

[0005] A neural network is a machine learning model that uses one or more layers of nonlinear units that can predict an output for a received input based on it. Some neural networks include one or more hidden layers in addition to an output layer. As an output of each hidden layer, a corresponding feature map can be provided. Such corresponding feature map of each hidden layer can be used as an input to a subsequent layer in the network, i.e., a subsequent hidden layer or an output layer. Each layer of the network generates an output from a received input according to the current value of a respective parameter set. In a neural network that is split between different devices, for example, between an encoder and a decoder, or between a device and a cloud, the feature map at the output of the location of the split (e.g., a first device) is compressed and transmitted to the remaining layers of the neural network (e.g., to a second device).

[0006] Further improvements in encoding and decoding using trained network architectures may be desirable. Summary of the Invention

[0007] The present invention relates to a method and apparatus for modifying, eg, enhancing, an image or video. [Means for solving the problem]

[0008] These and other objects are achieved by the subject matter of the independent claims. Further implementation forms are evident from the dependent claims, the description and the drawings.

[0009] Particular embodiments are outlined in the accompanying independent claims, while further embodiments are defined in the dependent claims.

[0010] In particular, embodiments of the present invention provide an approach for modifying an image based on a neural network system that processes multiple image channels: a primary channel is processed separately; one or more secondary channels are processed taking into account information from the primary channel; prior to processing by the neural network system, the first channel and the secondary channels undergo a spatial frequency transformation; prior to processing by the neural network system, it may be selected which image channel is the primary channel.

[0011] According to a first aspect, the present disclosure relates to a method of modifying an image region represented by two or more image channels, the method comprising: processing a primary channel of the two or more image channels based on a first spatial frequency transformation to obtain a transformed primary channel; and processing a secondary channel of the two or more image channels, different from the primary channel, based on a second spatial frequency transformation to obtain a transformed secondary channel.

[0012] The two or more image channels may include color channels and / or feature channels. The color channels and feature channels reflect image characteristics. Each type of channel may provide information not present in the other channels, so that collaborative processing may improve them relative to the primary channel. For example, the two or more image channels may be YUV channels and the primary channel may be the Y channel. The image region may be a patch of a given size corresponding to a portion of an image or portions of multiple images, or the image region may be an image or multiple images. The images may be still images or frames of a video sequence.

[0013] Furthermore, the method according to the first aspect includes the steps of processing the transformed primary channel by a first neural network to obtain a modified transformed primary channel, and processing the transformed secondary channel based on the transformed primary channel (used as auxiliary information) by a second neural network to obtain a modified transformed secondary channel. The first and second networks may be different from each other and may operate independently of each other.

[0014] The method according to the first aspect further includes processing the modified transformed primary channel based on the first inverse spatial frequency transform to obtain a modified primary channel, and processing the modified transformed secondary channel based on the second inverse spatial frequency transform to obtain a modified secondary channel. Based on the modified primary channel and the modified secondary channel, a modified image region is obtained.

[0015] The first spatial frequency transform and the second spatial frequency transform provide information in the spatial frequency domain, but are not limited thereto. It should be noted that the transformation by the spatial frequency transform can be considered as a kind of "preconditioning" in which the signal is converted into a more redundant format before being further processed. A more redundant signal is easier for a neural network to process.

[0016] In order to improve the quality of the modified image area on the primary channel, information on the primary channel is used as auxiliary information for processing the secondary channel. The auxiliary information is given in the spatial frequency transformed domain and is conveniently added to the spatial frequency transformed secondary channel before input to the second neural network. The transformed primary channel can be processed by the first neural network independently from the second neural network. Thus, the coefficients of one of the neural networks can be changed / optimized without affecting the output of the other network. Thereby, the overall conditioning / optimization of the neural network can be performed fairly quickly. Furthermore, if the first neural network and the second neural network are implemented as convolutional networks, different kernels may be used for them. In particular, according to the method of the first aspect, it is not necessary to process all the channels by exactly the same network at one stage in order to obtain the modified channel. Thereby, the time consumption of the overall processing can be reduced compared to the state of the art.

[0017] According to one implementation, each of the first neural network and the second neural network is or includes a convolutional neural network (CNN). CNNs have proven to be superior to other networks, such as multiple perceptrons, in many image processing applications and are known for their relatively robust and fast processing. Each of the convolutional neural networks may include at least one residual network component that enables residual learning with reduced memory demands. One or more of the convolutional neural networks may use a scaling layer represented by one or more scaling values. The scaling layer may be adapted to signal one or more scaling values ​​accordingly.

[0018] In one possible implementation, one or both of the first spatial frequency transform and the second spatial frequency transform are selected from the group consisting of energy compaction transforms including wavelet transform, discrete Fourier transform, fast Fourier transform, and discrete cosine transform. The spatial frequency transform can be selected according to the actual application, which may require different transforms to achieve the desired quality of the modified image region. The first spatial frequency transform and the second spatial frequency transform may be the same (one of wavelet transform, discrete Fourier transform, fast Fourier transform, energy compaction transform, and discrete cosine transform).

[0019] Depending on the actual application and in terms of processing speed and memory demand, the choice of wavelet transform may be appropriate. In this case, one or both of the first spatial frequency transform and the second spatial frequency transform may be a wavelet transform selected from the group consisting of discrete wavelet transform (DWT) and stationary wavelet transform. If DWT is to be used, Haar (for simplicity) or Daubechies (for accuracy) wavelets may be selected.

[0020] In another possible implementation of the method of the first aspect, the primary channel is selected (rather than fixedly predetermined) from two or more image channels. By processing the image patch by patch or multi-image by multi-image, regions of the image or video sequence can be processed differently, and in particular the selection of the primary channel can be changed appropriately. As the content in the image and / or in the video sequence can change, it can be advantageous for image correction / correction to adapt the primary channel accordingly.

[0021] According to another implementation, the secondary channel can also be selected from two or more image channels. Providing this additional selection option increases processing flexibility.

[0022] If at least one secondary channel is selected, then exactly one primary channel may be selected. If a flag in the encoded stream indicates that other channels of the two or more channels should not be processed, then there may be no need to label the selected channel as a primary or secondary channel.

[0023] According to another implementation, the primary and secondary channels can be selected from two or more image channels based on the output of a classifier operating on the basis of another neural network. The use of a classifier makes it possible to train or design such a classifier in order to appropriately select the image channel that will be the primary channel so that the quality of the image modification (such as image correction) can be improved.

[0024] In principle, the method according to the first aspect is suitable for processing primary and secondary channels of the same size and different sizes. If the primary and secondary channels are of the same size, according to one implementation, the processing of the transformed secondary channel based on the transformed primary channel includes a step of concatenating a second three-dimensional tensor representing the transformed secondary channel with a first three-dimensional tensor representing the transformed primary channel. The concatenation is performed along the first non-spatial dimension of the tensor. The spatial dimensions of the tensor are the height and width dimensions of the image region. The first non-spatial dimension results from the spatial frequency transformation. For example, if a discrete wavelet transform is used for the spatial frequency transformation, the first non-spatial dimension of the tensor is given by the spatial low frequency subband LL and the spatial high frequency subbands HL (vertical features), LH (horizontal features) and HH (diagonal features).

[0025] Such concatenation for use of the auxiliary information in the image correction process can be performed relatively fast and memory efficient.

[0026] The size of the primary channel can be larger than the size of the secondary channel (hence the resolution is better). In this case, according to one implementation, the transformed primary channel is processed based on at least one additional first spatial frequency transformation to obtain an auxiliary transformed primary channel of the same size as the transformed secondary channel in the height direction and width direction of the image area (the first spatial frequency transformation and the additional first spatial frequency transformation form a cascaded spatial frequency transformation). In this case, the processing of the transformed secondary channel is based on the auxiliary transformed primary channel. On the other hand, if the size of the secondary channel is larger than the size of the primary channel, according to one implementation, the transformed secondary channel is processed based on at least one additional second spatial frequency transformation to obtain an auxiliary transformed secondary channel of the same size as the transformed primary channel in the height direction and width direction of the image area (the second spatial frequency transformation and the additional second spatial frequency transformation form a cascaded spatial frequency transformation). In this case, the processing of the transformed secondary channel includes processing the auxiliary transformed secondary channel based on the transformed primary channel.

[0027] In either case, the cascade transform allows the processing of channels of different sizes without significantly increasing the processor load and processing time. Moreover, the concatenation for using auxiliary information in the image correction process can also be used for processing channels of different sizes by the cascade transform. Thus, the processing of the transformed secondary channel based on the transformed primary channel according to the method of the first aspect according to one implementation form includes a step of concatenating the second three-dimensional tensor representing the transformed secondary channel with the first three-dimensional tensor representing the auxiliary transformed primary channel when the size of the primary channel is larger than the size of the secondary channel, and a step of concatenating the second three-dimensional tensor representing the auxiliary transformed secondary channel with the first three-dimensional tensor representing the transformed primary channel when the size of the secondary channel is larger than the size of the primary channel.

[0028] In many applications, the primary channel has a larger size (if any), but this is not necessarily the case: for example, in the case of a combination of a low-resolution grayscale camera and a high-resolution noisy color camera, the low-resolution channel of the low-resolution grayscale camera may be selected as the primary channel having a correspondingly smaller size than the secondary channel provided by the high-resolution noisy color camera.

[0029] It should be noted that in general, the overall processing may be facilitated by restricting the image region to the shape of a square region in the height and width dimensions of the image region. According to one embodiment, the steps of dividing the image into image regions containing the image region and padding the image regions resulting from the division that are not square in the height and width dimensions of the image region so that they are square in the height and width dimensions of the image region are performed. Alternatively, the steps of dividing the image into image regions containing the image region and padding the image are performed such that the image is divided only into image regions that are square in the height and width dimensions of the image region, if the image cannot be divided only into image regions that are square in the height and width dimensions of the image region, such that the image is divided only into image regions that are all square in the height and width dimensions of the image region containing the image region.

[0030] Furthermore, it should be noted that before undergoing their respective spatial frequency transformations, the primary and secondary channels may undergo pixel shifting as described in the detailed description below. Pixel shifting may further increase processing efficiency. In this case, the inverse post-modification channel obtained by the inverse spatial frequency transformation undergoes pixel unshifting.

[0031] In some example implementations, the method further includes selecting a minimum size of the image region based on a number of hidden layers of the neural network, where the minimum size is at least 2*((kernel_size-1) / 2*n_layers)+1, where kernel_size is the size of the kernel of the neural network, which is a convolutional neural network, and n_layers is the number of layers of the neural network.

[0032] Such a lower bound on the choice of patch size may allow to fully exploit the information in the processed image without adding redundancy by padding etc., depending on the design of the neural network.

[0033] According to one embodiment, (which may be combined with any previous or later described embodiments and examples), the method further comprises the step of rearranging pixels of each of at least two image channels of an image region into a plurality S of sub-regions, each of the sub-regions of an image channel of the at least two image channels comprising a subset of samples of said image channel, and wherein for all image channels a horizontal dimension of the sub-regions is the same and equal to an integer multiple mh of the greatest common divisor of the horizontal dimensions of the images, and wherein for all image channels a vertical dimension of the sub-regions is the same and equal to an integer multiple mv of the greatest common divisor of the vertical dimensions of the images.

[0034] By such rearrangement, the neural network can be used to process images whose image channels have different dimensions / resolutions.

[0035] In particular, the S subregions of an image region are disjoint with S=mh*mv and have horizontal dimension dimh and vertical dimension dimv, the subregions contain samples of the image region on positions {kh*mh+offh,kv*mv+offv}, where kh∈[0,dimh-1] and kv∈[0,dimv-1], and each combination of offh and offv specifies a respective subregion with offk∈[1,mh] and offv∈[1,mv].

[0036] The above patch size determination makes it possible to utilize images and effectively adapt the patch size for each channel to the dimensions of the images, even when the channels have different resolutions and / or dimensions (vertical and / or horizontal).

[0037] As already mentioned, the first and second neural networks can be operated independently of each other. According to one implementation, the weights (and activation functions) of one of the first and second neural networks are determined and used independently of the weights of the other of the first and second neural networks. The individual adaptation of one of the networks does not affect the configuration of the other network.

[0038] According to a second aspect, there is provided a method for encoding an image or a video sequence or images, comprising the steps of obtaining an original image region, encoding the obtained image region into a bitstream, and applying a modification of the obtained image region by reconstructing the encoded image region as described above.

[0039] The use of image modifications in image or video coding allows for an improvement of the quality of the decoded image. This may be quality in the sense of the distortion that may be reduced. However, in some applications there may be some special effects that may be desired and the modifications may lead to their improvement (without necessarily reducing the distortion with respect to the original picture).

[0040] For example, the encoding may include a step of including an indication of the selected primary channel in the bitstream, which allows for a possibly better reconstruction at the decoder side, i.e. better in terms of distortion relative to the original (undistorted) image.

[0041] According to one example implementation, the method further includes including an adaptation of one or more weights of at least one of the first neural network weights and the second neural network weights in the bitstream.

[0042] According to an exemplary implementation, the method further includes acquiring a plurality of image regions, applying the method for modifying the acquired image regions to the image regions of the acquired plurality of image regions individually, and including in the bitstream of each of the plurality of image regions at least one of an indication indicating that the method for modifying the acquired image region should not be applied to the image region, an adaptation of one or more weights of at least one of the first and second neural networks, or an indication of a selected primary channel of the region. The region-based processing facilitates adaptation to image or video content.

[0043] When applying the method for modifying the acquired image region, the selection of the primary and secondary channels may be made based on the reconstructed image region without reference to the acquired image region input to the encoding step. This avoids additional overhead (rate requirements). According to a third aspect, there is provided a method for decoding an image or a video sequence or an image from a bitstream, comprising the steps of reconstructing the reconstruction from the bitstream and applying the method for modifying the image region as described above.

[0044] Application of image or video modifications at the decoder side may improve the quality of the decoded image.

[0045] In some embodiments, a method for decoding an image or video sequence includes parsing a bitstream to obtain at least one of an indication indicating that a method for modifying an acquired image region should not be applied to the image region, an indication of a selected primary channel for the region, and an adaptation of one or more weights of at least one of a first neural network and a second neural network, reconstructing the image region from the bitstream, and if the indication indicates a selected primary channel, modifying the reconstructed image region using the indicated primary channel as the selected primary channel.

[0046] Reconstruction based on side information may provide better performance in terms of quality, as described above for the corresponding encoding methods. The modifications may be applied as in-loop filters or as post-processing filters in the encoder and / or decoder.

[0047] According to one example implementation, the method further includes, if an adaptation of the weights of at least one of the first neural network and the second neural network is present in the bitstream, modifying the weights of the respective neural network accordingly.

[0048] According to a fourth aspect, there is provided a computer program product comprising program code stored on a non-transitory medium, the program, when executed on one or more processors, performing a method according to any one of the aspects and implementations described above.

[0049] According to a fifth aspect, there is provided an apparatus for modifying an image region represented by two or more image channels, the apparatus comprising a circuit configured to perform steps according to the method according to any one of the aspects and implementations described above. The apparatus provides technical means for implementing the operations in the method defined according to the first aspect. This functionality may be implemented by hardware or by hardware executing corresponding software.

[0050] According to a sixth aspect, an apparatus for modifying an image region represented by two or more image channels is provided, the apparatus including a first spatial frequency transformation unit configured to process a primary channel of the two or more image channels to obtain a transformed primary channel, and a second spatial frequency transformation unit configured to process a secondary channel of the two or more image channels different from the primary channel to obtain a transformed secondary channel. The apparatus further includes a first neural network configured to process the transformed primary channel to obtain a modified transformed primary channel, and a second neural network configured to process the transformed secondary channel based on the transformed primary channel to obtain a modified transformed secondary channel. The apparatus further includes a first inverse spatial frequency transformation unit configured to process the modified transformed primary channel to obtain a modified primary channel, a second inverse spatial frequency transformation unit configured to process the modified transformed secondary channel to obtain a modified secondary channel, and a combining unit configured to obtain a modified image region based on the modified primary channel and the modified secondary channel.

[0051] Further features and implementations of the method according to the first aspect of the present disclosure correspond to the respective possible features and implementations of the device according to the sixth aspect of the present disclosure. The advantages of the device according to the sixth aspect may be the same as the advantages of the corresponding implementations of the method according to the first aspect.

[0052] According to a seventh aspect, there is provided an encoder for encoding an image or a video sequence or an image, the encoder comprising: an input module for obtaining an original image region, a compression module for encoding the obtained image region into a bitstream, a reconstruction module for reconstructing the encoded image region, and one of the above-mentioned devices for modifying the reconstructed image region according to the fifth and sixth aspects.

[0053] According to an eighth aspect, there is provided a decoder for decoding an image or a video sequence or an image from a bitstream, the decoder comprising: a reconstruction module for reconstructing an image region from the bitstream; and an apparatus for modifying the reconstructed image region according to the fifth and sixth aspects, respectively.

[0054] According to a ninth aspect, the present disclosure relates to a video stream decoding device, including a processor and a memory, the memory storing instructions for causing the processor to perform a method according to the first aspect and implementations thereof.

[0055] According to a tenth aspect, the present disclosure relates to a video stream encoding device, including a processor and a memory, the memory storing instructions for causing the processor to perform a method according to the first aspect and implementations thereof.

[0056] According to an eleventh aspect, a computer-readable storage medium is proposed having stored thereon instructions which, when executed, cause one or more processors to encode video data, the instructions causing the one or more processors to perform a method according to the first aspect or the second aspect, or any possible embodiment of the first aspect and its implementations.

[0057] The above-described apparatus may be embodied on an integrated chip.

[0058] Any of the above described embodiments and example implementations may be combined with each other where deemed appropriate.

[0059] In the following, embodiments of the invention are explained in more detail with reference to the accompanying figures and drawings. [Brief description of the drawings]

[0060] [Figure 1] FIG. 2 is a schematic diagram illustrating channels processed by layers of a neural network. [Diagram 2] FIG. 1 is a schematic diagram illustrating an autoencoder type neural network. [Figure 3A] FIG. 1 is a schematic diagram illustrating an example network architecture in which the encoder and decoder sides include hyper-prior models. [Figure 3B] FIG. 1 is a schematic diagram illustrating a general network architecture in which the encoder side includes a hyper-prior model. [Figure 3C] FIG. 1 is a schematic diagram illustrating a general network architecture in which the decoder side includes a hyper-prior model. [Figure 4] FIG. 1 is a schematic diagram illustrating an example network architecture in which the encoder and decoder sides include hyper-prior models. [Diagram 5] FIG. 1 is a block diagram illustrating the structure of a cloud-based solution for machine-based tasks, such as machine vision tasks. [Figure 6] FIG. 1 is a block diagram illustrating an end-to-end video compression framework based on neural networks. [Figure 7] FIG. 1 is a schematic diagram of the collaborative processing of three color channels by a convolutional neural network. [Figure 8] Schematic diagram of n-channel cooperative processing by a convolutional neural network. [Figure 9] FIG. 1 is a schematic diagram of n-channel DWT-based collaborative processing by neural networks according to one embodiment. [Figure 10] FIG. 1 is a schematic diagram of DWT-based collaborative processing of n channels of different sizes by a neural network according to one embodiment. [Figure 11] 1 illustrates a neural network involved in n-channel DWT-based collaborative processing according to one embodiment. [Figure 12] FIG. 4 is a flow diagram illustrating an example method for modifying an image region. [Figure 13] 1 illustrates an exemplary apparatus for modifying an image region. [Figure 14] FIG. 1 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present disclosure. [Figure 15] 2 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present disclosure. [Figure 16] 1 is a block diagram illustrating an example of an encoding device or a decoding device. [Figure 17] 1 is a block diagram illustrating another example of an encoding device or a decoding device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0061] Like reference numbers and designations in different drawings may indicate like elements.

[0062] In the following description, reference is made to the accompanying drawings, which form a part of this disclosure and show, by way of example, specific aspects of the disclosed embodiments or specific aspects in which the disclosed embodiments may be used. It is understood that the disclosed embodiments may be used in other ways and may include structural or logical changes not shown in the drawings. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0063] For example, it is understood that disclosure related to a described method may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, when one or more particular method steps are described, the corresponding device may include one or more units, e.g., functional units, for performing the described one or more method steps (e.g., one unit performing one or more steps, or multiple units performing one or more of the multiple steps, respectively), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, when a particular apparatus is described based on one or more units, e.g., functional units, the corresponding method may include one step performing the function of one or more units (e.g., one step performing the function of one or more units, or multiple steps performing the function of one or more of the multiple units), even if such one or more steps are not explicitly described or illustrated in the drawings. Furthermore, it is understood that the features of various exemplary embodiments and / or aspects described herein may be combined with each other, unless otherwise specified.

[0064] The following provides an overview of some of the terminology used and the framework within which embodiments of the present disclosure may be used.

[0065] Artificial Neural Networks An artificial neural network (ANN), or connectionist system, is a computing system loosely inspired by the biological neural circuits that make up animal brains. Such systems "learn" to do a task by considering examples, generally without being programmed with task-specific rules. For example, in image recognition, an ANN may learn to identify images that contain cats by analyzing example images that have been manually labeled as "cat" or "not cat" and using the results to identify cats in other images. The ANN does this without any prior knowledge of cats, e.g., that cats have fur, tails, whiskers, and cat-like faces. Instead, the ANN automatically generates discriminative characteristics from the examples that the ANN processes.

[0066] ANNs are based on a collection of connected units or nodes called artificial neurons that loosely model neurons in the biological brain. Each connection can send signals to other neurons, similar to synapses in the biological brain. An artificial neuron that receives a signal can then process the signal and signal to neurons connected to it.

[0067] In an ANN implementation, the "signals" at the connections are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of its inputs. The connections are called edges. Neurons and edges typically each have weights that adjust as learning progresses. The weights increase or decrease the strength of the signal at the connection. Neurons may have thresholds such that a signal is sent only if the aggregate signal exceeds that threshold. Neurons are typically aggregated into layers. Different layers may perform different transformations on their inputs. A signal progresses from the first layer (input layer) to the last layer (output layer), possibly after traversing the layer multiple times.

[0068] The original goal of the ANN approach was to solve problems in the same way that the human brain solves them. Over time, the focus shifted to performing specific tasks, leading to a departure from biology. ANNs have been used for a variety of tasks including computer vision, speech recognition, machine translation, social network filtering, board and video games, medical diagnosis, and even in activities traditionally considered to be exclusively human, such as painting.

[0069] The name "convolutional neural network" (CNN) indicates that the network uses a mathematical operation called convolution, which is a special kind of linear operation. Convolutional networks are neural networks that use convolution instead of the common matrix multiplication in at least one of their layers.

[0070] FIG. 1 illustrates a schematic of the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of an input layer and an output layer, as well as multiple hidden layers. The input layer is the layer to which the input (such as a portion of an input image 11 as shown in FIG. 1) is provided for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve with multiplication or other dot products. The output of the layer is one or more feature maps (illustrated by empty solid rectangles), sometimes called channels. There may be resampling (such as subsampling) involved in the operation of some or all of the layers. As a result, the feature maps may be smaller, as illustrated in FIG. 1. Note that convolutions with strides may also reduce the size (resampling) of the input feature maps. The activation function in a CNN is typically a ReLU (rectified linear unit) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, which are called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. The layer is colloquially called a convolutional layer, but this is only by convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important to the index in the matrix, in that it affects how the weights are determined at that particular index point.

[0071] When programming a CNN to process images, the input is a tensor with dimensions (number of images) × (image width) × (image height) × (image depth), as shown in Figure 1. It should be understood that the image depth can be composed by the channels of the image. After passing through the convolutional layers, the image is abstracted into a feature map with dimensions (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolutional layers in a neural network should have the following attributes: A convolution kernel defined by width and height (hyperparameters); The number of input and output channels (hyperparameters); The depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0072] In the past, traditional multi-layer perceptron (MLP) models have been used for image recognition. However, due to the full connectivity between nodes, MLP models suffer from high dimensionality and do not scale well with high-resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is too high to be efficiently and conveniently processed with full connectivity. Also, such network architectures do not take into account the spatial structure of the data, treating input pixels that are far away from each other the same as pixels that are close to each other. This ignores the locality of reference in image data, both computationally and semantically. Thus, for purposes such as image recognition, where spatially local input patterns dominate, full connectivity of neurons is wasteful.

[0073] Convolutional neural networks are biologically inspired variants of multi-layer perceptrons specifically designed to emulate the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by exploiting the strong spatially local correlations present in natural images. The convolutional layer is the core building block of CNNs. The parameters of the layer consist of a set of learnable filters (kernels, as mentioned above) that have small receptive fields but extend across the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the filter's entries and the input, generating a two-dimensional activation map for that filter. As a result, the network learns filters that activate when it detects some particular type of feature at some spatial location in the input.

[0074] Stacking the activation maps of all filters along the depth dimension forms the full output volume of the convolutional layer. Thus, every entry in the output volume can also be interpreted as the output of neurons that look at a small region in the input and share parameters with neurons in the same activation map. A feature map, or activation map, is the output activation of a given filter. Feature map and activation have the same meaning. It is called an activation map in some papers because it is a mapping that corresponds to the activation of different parts of the image, and also a mapping of where a particular type of feature can be found in the image, so it is also called a feature map. High activation means that a particular feature has been found.

[0075] Another important concept in CNNs is pooling, which is a form of nonlinear downsampling. There are several nonlinear functions to implement pooling, of which max pooling is the most common. Max pooling divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such subregion.

[0076] Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers serve to progressively reduce the spatial size of the representation, reducing the number of parameters in the network, the memory footprint, and the amount of computation, and thus also controlling overfitting. In CNN architectures, it is common to periodically insert pooling layers between successive convolutional layers. The pooling operation provides another form of translation invariance.

[0077] A pooling layer operates independently on each depth slice of the input and resizes it spatially. The most common form is a pooling layer with a filter of size 2 × 2, applied with a stride of 2 for each depth slice of the input by 2 along both the width and height, discarding 75% of the activations. In this case, all max operations are over 4 numbers. The depth dimension remains unchanged. In addition to max pooling, the pooling unit can also use other functions such as average pooling and L2 norm pooling. Average pooling was often used in the past, but has recently fallen out of favor in comparison to max pooling, which often works better in practice. For aggressive reduction of the size of the representation, the recent trend is to use smaller filters or to discard the pooling layer altogether. "Region of interest" pooling (also known as ROI pooling) is a variant of max pooling, where the output size is fixed and the input rectangle is a parameter. Pooling is a key component of convolutional neural networks for object detection based on the Faster R-CNN architecture.

[0078] The ReLU above is short for rectified linear unit and applies a non-saturating activation function. ReLU effectively removes negative values ​​from the activation map by setting them to zero. ReLU increases the non-linearity of the decision function and the entire network without affecting the receptive fields of the convolutional layers. Other functions, such as saturated hyperbolic tangent and sigmoid functions, are also used to increase the non-linearity. ReLU is often preferred over other functions because it trains neural networks several times faster without significantly compromising generalization accuracy.

[0079] After several convolutional and max-pooling layers, higher-level inference in neural networks is performed via fully connected layers, whose neurons are connected to all activations of the previous layer, as in regular (non-convolutional) artificial neural networks. Their activations can thus be computed as affine transformations, matrix multiplications followed by bias offsets (vector addition of a learned or fixed bias term).

[0080] The "loss layer" (which contains the computation of the loss function) specifies how training penalizes deviations between predicted (output) labels and true labels, and is usually the final layer of a neural network. Various loss functions suitable for different tasks may be used: softmax loss is used to predict a single class out of K mutually exclusive classes; sigmoid cross entropy loss is used to predict K independent probability values ​​in [0,1]; Euclidean loss is used to regress to real-valued labels.

[0081] In summary, FIG. 1 shows the data flow in a typical convolutional neural network. First, an input image is passed through a convolutional layer and abstracted into a feature map containing multiple channels corresponding to several filters in the set of learnable filters of this layer. The feature map is then subsampled, for example using a pooling layer, which reduces the dimensionality of each channel in the feature map. The data then reaches another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input and output channels are hyperparameters of the layer. To establish the connectivity of the network, these parameters need to be synchronized between two connected layers, such that the number of input channels of the current layer is equal to the number of output channels of the previous layer. For the first layer, which processes input data, e.g., an image, the number of input channels is usually equal to the number of channels of the data representation, e.g., three channels for an RGB or YUV representation of an image or video, or one channel for a grayscale image or video representation. The channels obtained by one or more convolutional layers (and possibly a resampling layer(s)) may be passed to an output layer. Such an output layer may be convolutional or resampled in some implementations. In an exemplary and non-limiting implementation, the output layer is a fully connected layer.

[0082] Autoencoders and Unsupervised Learning An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. Its schematic diagram is shown in Figure 2. An autoencoder includes an encoder side 210 where an input x is input to the input layer of an encoder sub-network 220, and a decoder side 250 where an output x' is output from a decoder sub-network 260. The goal of an autoencoder is to learn a representation (encoding) 230 of a dataset x, usually for dimensionality reduction, by training the networks 220, 260 to ignore signal "noise". Together with the reduction (encoder) side sub-network 220, the reconstruction (decoder) side sub-network 260 is trained, and the autoencoder tries to generate a representation x' from the reduced encoding 230 that is as close as possible to its original input x, hence its name. In the simplest case, given one hidden layer, the encoder stage of the autoencoder takes the input x and maps it to h, h = σ(Wx+b) It is.

[0083] This image h is typically called the code 230, latent variable, or latent representation. Here, σ is an element-wise activation function such as the sigmoid function or the rectified linear unit. W is a weight matrix and b is a bias vector. The weights and biases are typically initialized randomly and then updated iteratively during training via backpropagation. The decoder stage of the autoencoder then maps h to a reconstruction x' of the same shape as x: x'=σ'(W'h'+b') where σ', W', and b' of the decoder may be independent of the corresponding σ, W, and b of the encoder.

[0084] Variational autoencoder models make strong assumptions about the distribution of the latent variables. They use a variational approach to latent representation learning, which results in an additional loss component and a specific estimator for the training algorithm called the Stochastic Gradient Variational Bayesian (SGVB) estimator. The data is then fed into a directed graphical model p θ(x|h), and the encoder estimates the posterior distribution p θ Approximation q for (h|x) φ Suppose we are learning (h|x), where φ and θ represent the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors in a VAE typically matches the probability distribution of the training data much more closely than standard autoencoders. The objective function of a VAE has the form L(φ,θ,x)=D KL (q φ (h|x)||p θ (h))-E qφ(h│x) (log p θ (x|h)

[0085] In the formula, D KL stands for the Kullback-Leibler divergence. The prior distribution for the latent variables is usually a centrally isotropic multivariate Gaussian distribution p θ It is set so that (h) = N(0,Ι). In general, the shapes of the variational and likelihood distributions are chosen so that they are factored Gaussians: q φ (h|x)=N(ρ(x),ω 2 (x)I) p φ (x|h)=N(μ(h),σ 2 (h)I) where ρ(x) and ω 2 (x) is the encoder output, μ(h) and σ 2 (h) is the decoder output.

[0086] Recent advances in the field of artificial neural networks, especially convolutional neural networks, have enabled researchers to apply neural network-based techniques to the task of image and video compression. For example, end-to-end optimized image compression using networks based on variational autoencoders has been proposed.

[0087] Accordingly, data compression is considered a fundamental and well-studied problem in engineering, and is generally formulated with the goal of designing a code for a given discrete data ensemble with minimum entropy. The solution relies heavily on knowledge of the probabilistic structure of the data, and thus the problem is closely related to probabilistic source modeling. However, since all practical codes must have finite entropy, continuous-valued data (such as a vector of image pixel intensities) must be quantized into a finite set of discrete values, which introduces errors.

[0088] In this context, known as the lossy compression problem, one must trade off two competing costs: the entropy of the discretized representation (rate) and the error resulting from quantization (distortion). Different compression applications, such as data storage and transmission over limited-capacity channels, require different rate-distortion tradeoffs.

[0089] Joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in a high-dimensional space is intractable. For this reason, most existing image compression methods work by linearly transforming the data vector into an appropriate continuous-valued representation, quantizing its elements independently, and then encoding the resulting discrete representation using a reversible entropy code. This scheme is called transform coding because of the central role of the transform.

[0090] For example, JPEG uses a discrete cosine transform on blocks of pixels, and JPEG2000 uses a multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform coding method, transform, quantization, and entropy code, are optimized separately (often by manual parameter tuning). Modern video compression standards such as HEVC, VVC, and EVC also use transformed representations to code the residual signal after prediction. Several transforms are used for that purpose, such as the discrete cosine transform and discrete sine transform (DCT, DST), as well as the low-frequency non-separable manually optimized transform (LFNST).

[0091] Variational Image Compression The variable autoencoder (VAE) framework can be regarded as a nonlinear transform coding model. The transform process can be mainly divided into four parts. This is illustrated in FIG. 3A, which shows the VAE framework.

[0092] The transformation process can be divided into four main parts, and FIG. 3A illustrates the VAE framework. In FIG. 3A, the encoder 101 maps the input image x to a latent representation (represented by y) via a function y=f(x). This latent representation may be referred to below as a part of the "latent space" or a point in the "latent space". The function f() is a transformation function that transforms the input signal x into a more compressible representation y. The quantizer 102 converts the latent representation y into a quantization function Q,

number

number

number

[0093] A latent space can be understood as a representation of compressed data where similar data points are close to each other in the latent space. Latent spaces are useful for learning data features and finding simpler representations of data for analysis. Quantized Latent Representation

number

number

number

number

number

number

[0094] In FIG. 3A, component AE105 is an arithmetic encoding module that generates a quantized latent representation

number

number

number

[0095] Arithmetic decoding (AD) 106 is the process of undoing the binarization process, where binary numbers are converted back to sample values. Arithmetic decoding is provided by arithmetic decoding module 106.

[0096] It should be noted that the present disclosure is not limited to this particular framework. Furthermore, the present disclosure is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0097] In FIG. 3A there are two sub-networks connected to each other. A sub-network in this context is a logical division between parts of the total network. For example, in FIG. 3A modules 101, 102, 104, 105 and 106 are called the "encoder / decoder" sub-network. The "encoder / decoder" sub-network is responsible for encoding (generating) and decoding (parsing) the first bitstream "Bitstream 1". The second network in FIG. 3A includes modules 103, 108, 109, 110 and 107 and is called the "hyperencoder / decoder" sub-network. The second sub-network is responsible for generating the second bitstream "Bitstream 2". The purpose of the two sub-networks is different.

[0098] The first sub-network is responsible for: Transformation 101 of an input image x into its latent representation y (where it is easier to compress x); · Quantize the latent representation y

number

number

number

[0099] The purpose of the second sub-network is to obtain statistical properties of the samples of "Bitstream 1" (e.g., mean, variance and correlation between samples of Bitstream 1) so that the compression of Bitstream 1 by the first sub-network is more efficient. The second sub-network generates a second bitstream "Bitstream 2" that includes said information (e.g., mean, variance and correlation between samples of Bitstream 1).

[0100] The second network is the quantized latent representation

number

number

number

number

number

number

number

number

number

number

number

number

number

[0101] 3A illustrates an example of a VAE (Variational Autoencoder), the details of which may vary in different implementations. For example, in a particular implementation, there may be additional components to more efficiently obtain statistical characteristics of samples of bitstream 1. In one such implementation, there may be a context modeler that targets extracting cross-correlation information of bitstream 1. The statistical information provided by the second sub-network may be used by the AE (Arithmetic Encoder) 105 and AD (Arithmetic Decoder) 106 components.

[0102] 3A shows the encoder and the decoder in a single diagram. As will be apparent to those skilled in the art, the encoder and the decoder may, and very often are, integrated into different devices.

[0103] FIG. 3B shows an encoder, and FIG. 3C shows the decoder component of the VAE framework in isolation. As input, the encoder receives a picture according to some embodiments. The input picture may include one or more channels, such as color channels or other types of channels, e.g., depth channels, motion information channels, etc. The outputs of the encoder (as shown in FIG. 3B) are Bitstream 1 and Bitstream 2. Bitstream 1 is the output of a first sub-network of the encoder, and Bitstream 2 is the output of a second sub-network of the encoder.

[0104] Similarly, in FIG. 3C, two bitstreams, Bitstream 1 and Bitstream 2, are received as input, and the reconstructed (decoded) image is

number

[0105] Specifically, as seen in Figure 3B, the encoder includes an encoder 121 that converts an input x into a signal y that is then provided to a quantizer 322. The quantizer 122 provides information to an arithmetic encoding module 125 and to a hyper-encoder 123. The hyper-encoder 123 provides the bitstream 2 already mentioned above to a hyper-decoder 147, which provides this information to the arithmetic encoding module 105 (125).

[0106] The output of the arithmetic encoding module is bitstream 1. Bitstream 1 and bitstream 2 are the output of the encoding of the signal, which are then provided (sent) to the decoding process. Unit 101 (121) is called "encoder", but it is also possible to call the complete sub-network described in FIG. 3B "encoder". An encoder generally means a unit (module) that converts an input into an encoded (e.g., compressed) output. As can be seen from FIG. 3B, unit 121 can in fact be considered as the core of the entire sub-network, since it performs the conversion of the input x into y, which is a compressed version of x. The compression in encoder 121 can be achieved, for example, by applying a neural network, or in general any processing network with one or more layers. In such a network, the compression can be performed by cascading processes, including downsampling, which reduces the size and / or number of channels of the input. Thus, the encoder can be called, for example, a neural network (NN)-based encoder, etc.

[0107] The remaining parts in the figure (quantization unit, hyperencoder, hyperdecoder, arithmetic encoder / decoder) are all parts that improve the efficiency of the encoding process or are responsible for converting the compressed output y into a series of bits (bitstream). Quantization may be provided to further compress the output of the NN encoder 121 by lossy compression. The AE 125, in combination with the hyperencoder 123 and hyperdecoder 127 used to configure the AE 125, may perform binarization that may further compress the quantized signal by lossless compression. Therefore, the entire sub-network of FIG. 3B may be called an "encoder".

[0108] Most deep learning (DL) based image / video compression systems reduce the dimensionality of a signal before converting it into binary numbers (bits). For example, in the VAE framework, an encoder, which is a nonlinear transform, maps an input image x to y, where y has a smaller width and height than x. Since y has a smaller width and height, and thus a smaller size, the dimensionality (size) of the signal is reduced, and thus it is easier to compress the signal y. Note that in general, an encoder does not necessarily need to reduce the size of both (or generally all) dimensions. Rather, some exemplary implementations may provide an encoder that reduces the size of only one dimension (or generally a subset of dimensions).

[0109] In "Density Modeling of Images Using a Generalized Normalization Transformation," J. Balle, L. Valero Laparra, and EP Simoncelli (2015), presented at the 4th Int. Conf. for Learning Representations, 2016, In: arXiv e-prints (hereafter "Balle"), the authors propose a framework for end-to-end optimization of image compression models based on nonlinear transformations. They optimize for the mean squared error (MSE) but use a more flexible transformation built from a cascade of linear convolutions and nonlinearities. Specifically, they use a generalized divisive normalization (GDN) combined nonlinearity that has proven effective in Gaussianizing image densities, inspired by models of neurons in the biological visual system. This cascade transformation is followed by a uniform scalar quantization (i.e., each element is rounded to the nearest integer), effectively implementing a parametric form of vector quantization on the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0110] Such an example of a VAE framework is shown in Figure 4, which utilizes six downsampling layers marked 401-406. The network architecture includes a hyper-prior model. On the left (g a ,g s ) shows the image autoencoder architecture, and on the right (h a ,h s ) corresponds to an autoencoder that implements a hyper-prior. The factorized prior model is then transformed into an analytic and synthetic transformation g a and g s The same architecture is used for both. Q stands for quantization, and AE and AD stand for arithmetic encoder and decoder, respectively. The encoder applies g a , resulting in responses y (latent representations) with spatially varying standard deviations. a contains multiple convolutional layers with subsampling and generalized decomposition normalization (GDN) as the activation function.

[0111] The response is h a to summarize the distribution of standard deviations in z. z is then quantized, compressed, and transmitted as side information. The encoder then

number

number

number

number

number

number

number

[0112] Layers that include downsampling are indicated with a downward arrow in the layer description. The layer description "Conv N,k1,2↓" means that the layer is a convolutional layer, has N channels, and the size of the convolution kernel is k1×k1. For example, k1 may be equal to 5 and k2 may be equal to 3. As above, 2↓ means that downsampling by a factor of two is performed in this layer. Downsampling by a factor of two results in one of the dimensions of the input signal being reduced by half at the output. In FIG. 4, 2↓ indicates that both the width and height of the input image are reduced by a factor of two. Since there are six downsampling layers, if the width and height of the input image 414 (also represented by x) are given by w and h, the output signal z^ 413 has a width and height equal to w / 64 and h / 64, respectively. The modules represented by AE and AD are arithmetic encoders and arithmetic decoders, which are described with reference to FIG. 3A to FIG. 3C. The arithmetic encoder and decoder are specific implementations of entropy coding. AE and AD can be replaced by other means of entropy coding. In information theory, entropy encoding is a lossless data compression scheme used to convert the value of a symbol into a binary representation, and is a reversible process. Also, "Q" in the figure corresponds to the quantization operation also described above in connection with FIG. 4 and further explained in the "Quantization" section above. Also, the quantization operation and corresponding quantization unit as part of the component 413 or 415 are not necessarily present and / or can be replaced with another unit.

[0113] Also shown in Figure 4 is a decoder including upsampling layers 407-412. A further layer 420, implemented as a convolutional layer but which does not provide upsampling to the received input, is provided between upsampling layers 411 and 410 in the processing order of the input. A corresponding convolutional layer 430 is also shown for the decoder. Such a layer does not change the size of the input, but may be provided in a NN to perform operations on the input that change certain characteristics. However, it is not necessary that such a layer be provided.

[0114] Viewed in the processing order of the bitstream 2 through the decoder, the upsampling layers are shown in reverse order, i.e., from upsampling layer 412 to upsampling layer 407, each successive upsampling layer here indicated with ↑, providing upsampling with an upsampling ratio of 2. Of course, not all upsampling layers necessarily have the same upsampling ratio, other upsampling ratios such as 3, 4, 8, etc. may be used. Layers 407-412 are implemented as convolutional layers (conv). In particular, they may be intended to provide, on the input, an inverse operation of the encoder's operation, and thus the upsampling layers may apply a deconvolution operation to the received input such that its size is increased by a factor corresponding to the upsampling ratio. However, the present disclosure is not generally limited to deconvolution, and upsampling may be performed in any other way, such as by bilinear interpolation between two neighboring samples, or by nearest neighbor sample copying, etc.

[0115] In the first sub-network, some convolutional layers (401-403) are followed by Generalized Divided Normalization (GDN) on the encoder side and Inverse GDN (IGDN) on the decoder side. In the second sub-network, the activation function applied is ReLu. It should be noted that the present disclosure is not limited to such implementations and in general, other activation functions may be used instead of GDN or ReLu.

[0116] Cloud solutions for machine tasks Machine-oriented video coding (VCM) is another computer science direction that is popular today. The main idea behind this approach is to transmit a coded representation of image or video information intended for further processing by computer vision (CV) algorithms like object segmentation, detection and recognition. In contrast to traditional image and video coding, which is intended for human perception, the quality characteristic is not the reconstruction quality but the performance of a computer vision task, e.g. object detection accuracy. This is illustrated in Figure 5.

[0117] Machine-directed video coding, also called collaborative intelligence, is a relatively new paradigm for efficient deployment of deep neural networks across a mobile cloud infrastructure. By partitioning the network between the mobile side 510 and the cloud side 590 (e.g., cloud servers), it is possible to distribute the computational workload so that the overall energy and / or latency of the system is minimized. In general, collaborative intelligence is a paradigm in which the processing of a neural network is distributed between two or more different computational nodes, e.g., between devices, but generally between any functionally defined nodes. Here, the term "node" does not refer to the neural network nodes described above. Rather, a (computational) node here refers (physically or at least logically) to a separate device / module that implements a part of a neural network. Such devices may be different servers, different end-user devices, a mix of servers and / or user devices, and / or a cloud and / or a processor, etc. In other words, the computational nodes may be considered as nodes that belong to the same neural network and communicate with each other to convey coding data within / for the neural network. For example, one or more layers may be executed on a first device (such as a device on the mobile side 510) and one or more layers may be executed on another device (such as a cloud server on the cloud side 590) to be able to perform complex calculations. However, the distribution may also be finer and a single layer may be executed on multiple devices. In this disclosure, the term "multiple" refers to two or more. In some existing solutions, part of the neural network function is executed on a device (such as a user device or an edge device) or multiple such devices, and then the output (feature map) is passed to the cloud. The cloud is a collection of processing or computing systems located outside the device running part of the neural network. The concept of collaborative intelligence has also been extended to model training.In this case, data flows in both directions: from cloud to mobile during backpropagation in training, and during the forward pass in training, as well as from mobile to cloud (illustrated in Figure 5) in inference.

[0118] Some works presented semantic image compression by encoding deep features and then reconstructing the input image from them. Compression based on uniform quantization was presented, followed by context-based adaptive arithmetic coding (CABAC) from H.264. In some scenarios, it may be more efficient to transmit the output of the hidden layer (deep feature map) 550 from the mobile part 510 to the cloud 590, rather than sending the compressed natural image data to the cloud and using the reconstructed image to perform object detection. Thus, it may be advantageous to compress the data (features) generated by the mobile side 510, which may include a quantization layer 520 for this purpose. Correspondingly, the cloud side 590 may include an inverse quantization layer 560. Efficient compression of feature maps is beneficial for image and video compression and reconstruction for both human perception and machine vision. Entropy coding methods, e.g., arithmetic coding, are common approaches for compression of deep features (i.e., feature maps).

[0119] Today, video content contributes to over 80% of Internet traffic, and the percentage is expected to further increase. Therefore, it is important to build an efficient video compression system to generate higher quality frames for a given bandwidth budget. In addition, most video-related computer vision tasks, such as video object detection and video object tracking, are sensitive to the quality of the compressed video, and efficient video compression can benefit other computer vision tasks. Meanwhile, the techniques of video compression also help in action recognition and model compression. However, in the past decades, video compression algorithms, as mentioned above, rely on hand-crafted modules, such as block-based motion estimation and discrete cosine transform (DCT), to reduce redundancy in video sequences. Although each module is well designed, the entire compression system is not optimized end-to-end. It is desirable to further improve the video compression performance by optimizing the entire compression system together.

[0120] End-to-end image or video compression DNN-based image compression methods can take advantage of large-scale end-to-end training and highly nonlinear transformations that are not used in traditional approaches. However, it is not trivial to directly apply these techniques to build end-to-end learning systems for video compression. First, learning how to generate and compress tuned motion information for video compression remains an open problem. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences.

[0121] A straightforward solution is to represent the motion information using learning-based optical flow. However, current learning-based optical flow approaches aim to generate a flow field as accurate as possible. Precise optical flow is often suboptimal for a particular video task. Moreover, the amount of data for optical flow increases significantly compared to the motion information in traditional compression systems, and directly applying existing compression approaches to compress the optical flow values ​​would significantly increase the number of bits required to store the motion information. Second, it is unclear how to build a DNN-based video compression system by minimizing a rate-distortion-based objective for both the residual and motion information. Rate-distortion optimization (RDO) aims to achieve higher quality (i.e., less distortion) of the reconstructed frames given the number of bits (or bit rate) for compression. RDO is critical for video compression performance. To take advantage of the power of end-to-end training for learning-based compression systems, an RDO strategy is needed to optimize the entire system.

[0122] In "DVC: An End-to-end Deep Video Compression Framework," by Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pages 11006-11015, the authors propose an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual coding.

[0123] Such an encoder is shown in Fig. 6. In particular, Fig. 6 shows the overall structure of the end-to-end trainable video compression framework. To compress the motion information, we use the optical flow v tThe corresponding representation m that is suitable for better compression. t A CNN is specified to convert the optical flow to a vector. Specifically, an autoencoder type network is used to compress the optical flow.

[0124] In general, video compression can degrade the perceived quality of the image, and image correction filters may commonly be used to improve the output quality of the compressed video.

[0125] One type of image correction filter improves the quality of multi-channel images by exploiting similarities between the channels. The performance of a multi-channel image correction algorithm depends on several parameters of the input multi-channel image (e.g., number of channels, quality of the channels) and also varies across the image data of each channel.

[0126] A variety of image correction algorithms exist. Only a few of them utilize inter-channel correlation information for image correction. In this disclosure, we focus on multi-channel image correction filters that use neural networks, such as convolutional neural networks. In neural network-based correction filters, the network is trained on two sets of images, one representing the original (target, desired) quality and the other representing the expected range and type of distortion. Such networks can be trained to improve images corrupted by, for example, sensor noise, or images corrupted by video compression or by other kinds of distortion. Typically, different (individual, separate) training is required for each distortion type. More general networks (e.g., handling a wider range and type of distortion) have lower average performance. Here, performance refers to the quality of the reconstruction, which can be measured, for example, by an objective criterion such as PSNR, or by some metric that also takes into account human vision.

[0127] In some embodiments of the present application, a deep convolutional neural network (CNN) is trained to reduce compression artifacts and improve the visual quality of the image while maintaining a high compression ratio. In particular, according to one embodiment, a method is provided for retouching an input image region, where retouching refers to any retouching, such as retouching typically obtained by filtering or other image correction approaches. The type of retouching may depend on the particular application.

[0128] One network that gives good results for a wide range of distortions without needing to be trained for each specific example is known from Cui, Kai and Steinbach, Eckehard (2018), "Decoder Side Image Quality Enhancement exploiting Inter-channel Correlation in a 3-stage CNN," Submission to CLIC 2018, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018. In it, a three-stage convolutional neural network (CNN) based approach is proposed that can exploit inter-channel correlation to improve image quality on the decoder side.

[0129] FIG. 7 illustrates such a three-channel CNN framework. A CNN is a neural network that uses convolution instead of common matrix multiplication in at least one of its layers. A convolutional layer convolves the input and passes the result to the next layer. They have several beneficial characteristics, especially for image / video processing. The CNN of FIG. 7 is described and applied stages are explained, including known configurations, as well as possible configurations and alternatives that may facilitate the application of CNN in some embodiments of the present disclosure.

[0130] The input image is stored in RGB (Red, Green, Blue) format (color space). The input image may be a still image or an image that is a frame of a video sequence (moving image).

[0131] The circled numbers in Figure 7 represent stages of processing. In stage 1, a patch is selected from the input image. In a particular example, the patch has a predefined size, such as a size of 240x240 samples (pixels). The patch size may be fixed or predetermined. For example, the patch size may be configurable by a user or application, or may be a parameter that may be conveyed in the bitstream and set accordingly. The selection of the patch size may be made depending on the image size and / or the amount of detail in the image.

[0132] Here, the term "patch" refers to a portion of an image that is processed by filtering, and the processed portion is then pasted back into the location of the patch. The patch may be regular, such as a rectangle or a square. However, the disclosure is not limited thereto, and the patch may have any shape, such as a shape that follows the shape of a detected / recognized object to be filtered. In some embodiments, the entire image is filtered (corrected) patch by patch. In other embodiments, only selected patches (e.g., corresponding to an object) may be filtered, and the remaining part of the image is not filtered or is filtered by another approach. Filtering refers to any kind of correction.

[0133] The selection may be the result of a sequential or parallel process in which all patches are filtered. In such cases, the selection may be made in a predefined order, such as from left to right or from top to bottom. However, the selection may also be made by a user or an application and only a part of the image may be considered. The patches may be contiguous or distributed.

[0134] When the entire image is divided into patch regions, padding may be applied when an integer multiple of the patch dimension (vertical or horizontal) does not match the image size. Padding may involve mirroring the available image portion across the axis formed by the image boundary (horizontal or vertical) to achieve a size that fits an integer number of patches. More specifically, padding is done such that the vertical dimension (number of samples after padding) is an integer multiple of the vertical patch dimension. Furthermore, the horizontal dimension (number of samples after padding) is an integer multiple of the horizontal patch dimension.

[0135] Image correction can then be performed by selecting and processing each of the patches sequentially or in parallel. The patches do not have to overlap as proposed above. However, the patches may also overlap, which may improve quality and reduce possible boundary effects between separately processed patches.

[0136] In stage 2, the pixels of the patch are rearranged to make it easier to process. The rearrangement may include a so-called pixel shift. The pixel shift rearranges the pixels in each channel, which converts the channels with dimensions N×N×1 into a 3D array with dimensions N / 2×N / 2×4 (where the symbol "×" stands for "times", i.e. multiplication). This is done by subsampling the channels, where a single value is taken from each non-overlapping block of 2×2 values. The first layer of the 3D array is created from the top-left values, the second layer from the top-right, the third layer from the bottom-left values, and the fourth layer from the bottom-right values. At the end of the pixel shift operation, the three-channel RGB patch with size N×N pixels becomes a stack (3D array) with dimensions N / 2×N / 2×4. This pixel shift is done mainly for computational reasons, i.e. because it is easier for a processor (e.g., a Graphical Processing Unit, GPU) to process a narrow and deep stack than a wide and shallow stack. Note that this disclosure is not limited to subsampling by 4. In general, more or less subsampling may be performed resulting in a corresponding stack depth of more or less of the resulting subsampled image.

[0137] In stage 3 of FIG. 7, the green channel is processed in single channel mode. In this example of FIG. 7, it is assumed that the green channel is the primary channel and the remaining channels are secondary channels. It is assumed that the distortion of each channel is strongly correlated with the color of the channel. The green channel is assumed to be the channel with the least distortion. This may be a reasonable assumption for RGB images that, on average, suffer from sensor noise or compression, since RGB images are captured using an RGGB sensor pattern (Bayer pattern), which captures more green samples than red and blue samples. However, according to the present disclosure, as discussed below, the green channel is not necessarily the best and it may be advantageous to make channel selection. Furthermore, channels other than the color channels may be used and may provide higher quality information.

[0138] In Figure 7, the primary (i.e., green) channel is processed only in stage 3. The red channel is processed along with the improved green channel (stage 4). The blue channel is processed along with the improved green channel (stage 5). The improved red, green, and blue channels are then stacked (combined) together (stage 6) and processed without or with side information.

[0139] Overall, the framework has four NN stages: stage 3, stage 4, stage 5, and stage 7 in Figure 7. Stages 3 and 7 input a single 3D array of values ​​and output a 3D array that has the same size as the input. Stages 4 and 5 input two 3D arrays, one primary and one auxiliary. The output is of the same size as the primary output and is intended to be a processed (e.g., corrected) version of the primary input; the auxiliary input is only used to assist in processing and is not output.

[0140] In stage 4, the final hidden layer is processed by N convolution kernels 3x3x64, outputting a stack of size Z. In stage 5, the input is added to the processed output as the network is trained to approximate the difference between the input and the desired output.

[0141] In step 5 of Figure 7, the blue channel is processed in conjunction with the corrected green channel, the processing being similar to that described with reference to the red channel and Figure 7.

[0142] It should be noted that in this disclosure, the terms "channel" or "image channel" do not necessarily refer to color channels. Other (tensor) channels, such as depth channels or other feature channels, may also be corrected using the embodiments described herein.

[0143] In stage 6 of Fig. 7, all processed channels (red, green, blue) are combined into a Z=12 (4 subsampled images per color channel) image of size 120x120, where combining means stacking together as a common input to the next stage 7. In stage 7, the combined channels are processed together by a network (S).

[0144] In step 8, the pixels are rearranged back to form the processed patch. This may include removing the padding by cropping out the mirrored portion. Note that the above padding by mirroring is only one possible way to do the padding. The present disclosure is not limited to such specific examples.

[0145] In step 9, the processed patches are inserted back into the original image. In other words, the original image is updated by the correction patches. During the training procedure, the original image and a set of distorted images are used as inputs and all the convolution kernels of all four networks are selected. The goal of the networks is to take a distorted image and produce a close match to the original image.

[0146] CNN-based multi-channel correction filters as described above operate in a rigid, non-adaptive manner, i.e., the processing parameters are set up during the design (or training) of the filter and are applied in the same way regardless of the content of the image passing through the filter. However, the optimal choice of the primary channel may differ from image to image or even for parts of the same image. For the quality of the image correction, it is advantageous if the primary (leading) channel is the one with the highest quality, which means the lowest distortion. This is because the primary channel also participates in the correction of the remaining channels. The inventors realized, and this was confirmed by experiments, that better performance may be achieved by carefully selecting the primary channel, such as per patch or per image. Secondly, the correction performance (of all correction filters) varies with the image quality of the input. The correction filter works optimally for a certain range of distortion strengths and does not bring much improvement for some very high or very low distortion levels. This is because high quality inputs can hardly be improved further and low quality inputs are too distorted to be reliably improved. As a result, for some inputs, it is beneficial to skip the correction process altogether. Third, the image correction described above (see FIG. 7) is designed for RGB input, where the resolution of all three channels is the same. However, in some video standard formats (e.g., YUV 4:2:0 or YUV 4:2:2, etc.), different channels may have different resolutions and pixel counts. Note that the image correction described herein is not necessarily applied during encoding or decoding. The image correction may be used for pre-processing. For example, the image correction may be used to correct images or videos in a raw format, such as a Bayer pattern-based format. In other words, the image correction may be applied to any image or video.

[0147] The above problem is addressed in WO 2021 / 249684 A1, which provides an image correction filter that can adapt to different input formats and different contents, multi-channel image formats with any number of channels (e.g., n channels as illustrated in FIG. 8) and different numbers of pixels in each channel, and to which a content analysis module is added to adjust the filter to process the input channels in a specific order or to skip the processing altogether. In particular, the primary channel can be selected, for example, based on content analysis. The configuration shown in FIG. 8 allows for processing n channels. After the selection 810 of a patch of the image to be processed, the application of a pixel shift 820 to the selected patch and the selection of the primary channel, the primary channel is processed by network 1, and the modified modified primary channel so obtained is used as auxiliary information when processing the secondary channel. The output m'1 of network 1 is a combination of the secondary channels m2 to m n 3D array m 2c From m nc are processed by Network 2 through Network N, respectively. The outputs of Networks 1 through N are concatenated 850 to obtain m'1+m'2+...+m' n , are processed by network M. The output of network M is pixel unshifted 860 to obtain the corrected multi-channel image region.

[0148] Unlike the prior art, this disclosure provides spatial frequency transform-based image modification (e.g., correction) using inter-channel correlation information. Spatial frequency transforms provide information about spatial frequency, but not necessarily only about spatial frequency (e.g., wavelet transforms also provide information about position).

[0149] Suitable spatial-frequency transforms that may be used include energy compaction transforms including the wavelet transform (discrete wavelet transform, DWT, or stationary wavelet transform), the discrete Fourier transform, the fast Fourier transform, and the discrete cosine transform.

[0150] A particular embodiment is illustrated in Figure 9. This and other embodiments represent adjustable correction filters that can adapt to different input formats and different content, and can process multi-channel image formats with any number of channels and different numbers of pixels in each channel. The channels representing the image regions to be processed can be any feature channels deemed appropriate, e.g. color channels, and may have any size (depth), e.g. different sizes.

[0151] For a provided (distorted) input image, patches (or image regions) to be processed are selected 910. Note that the selection of patches does not necessarily imply that there is any intelligence in selecting the patches. Rather, the selection may be done according to a sequential process (e.g., in a loop) or for more than one patch (or even all patches) at once, e.g., when parallel processing is applied. The selection step simply corresponds to determining which patches should be processed. A selection step is a set of k...2 patches (in height and width dimensions) with integers k, l, and p. p ×l…2 p For image size, the image is of size 2 p ×2 pThe image is divided into k×l square patches, each having a square size of k×l. If the image cannot be divided into k×l square patches, the image is padded before the division. Alternatively, the image is divided into patches, and if the patches resulting from the division are not square in the width and height dimensions, the patches are padded to obtain square patches. Depending on the type of image correction, there may be several criteria that are advantageously observed when selecting the patch size (size of the image region to be corrected). In particular, there may be some minimum size given by the type of processing applied during image correction (see WO 2021 / 249684 A1 for details).

[0152] The input image may be a still image or part of a video sequence. p ×2 p A patch with is split into (distorted) channels, e.g., color channels as RGB or YUV (luminance and chroma) channels (components). One of the channels is selected as the primary channel (e.g., luma for YUV channels). The other channels are secondary channels that are processed based on information about the primary channel. The channels are of size 2 p ×2 pwhere the integer p does not have to be the same for each of the channels, i.e., different sample resolutions may be provided for different channels. In the embodiment illustrated in FIG. 9, p is the same for all channels, for example, to process a YUV444 image (patch). If the channel is a YUV channel, the Y channel may be selected as the primary channel. The selection of the primary channel may be performed according to the teachings provided in WO 2021 / 249684(A1). In particular, the primary channel may be selected based on an analysis, for example a content analysis of the patch or the selected patch. For example, a classifier based on a neural network or a convolutional neural network may be used for the selection of the primary channel (and the secondary channel). The classifier may also be implemented by several algorithms, such as algorithms that determine the level of detail and / or the strength and / or direction of edges, the distribution of gradients, the movement characteristics (in the case of video) or other image features. Based on a comparison of such features of the respective image channels, the primary channel may then be selected. For example, the image channel that contains the most edges (corresponding to the most details) or the sharpest edges may be selected as the primary channel.

[0153] The selection can be made on the encoder side and signaled to the decoder side, can be pre-determined, or can be made independently on the encoder and decoder side using image analysis methods performed on the patch. In the latter two cases, the selection of the primary channel is not signaled. Furthermore, it should be noted that the selection of the primary channel for a patch can be different for some or each of the patches into which the image is divided. Alternatively, a fixed, predetermined primary channel may be used.

[0154] All channels are processed 920, 930, 940 with a two-dimensional discrete wavelet transform (DWT). In what follows, the DWT is used as an example for using spatial frequency transforms. Other examples as mentioned above may also be used. Any type of wavelet deemed suitable may be used for the DWT. For example, the Haar wavelet or the Daubechies wavelet may be deemed suitable choices. The application of the DWT 920 to the pixel values ​​of the pixels of the primary channels is performed using a wavelet of size 2 p-1 ×2 p-1 The DWT transform 930, 940 results in a DWT transform of size 2×4 for the primary channel. The third dimension (of size 4) of the three-dimensional array output by the DWT is given by the spatial low frequency subband LL and the spatial high frequency subbands HL (vertical feature), LH (horizontal feature) and HH (diagonal feature). All subbands may be arranged in a single matrix (layer) so that the output of the DWT transform is still a single layer, or may be represented by individual layers. Correspondingly, the application 930, 940 of the DWT to the pixel values ​​of the pixels of the secondary channel results in a DWT transform of size 2×4 for the primary channel. p-1 ×2 p-1 A ×4 DWT transform results in the secondary channel.

[0155] The DWT transformed primary channel is processed independently from the DWT transformed secondary channel by a first network, Network 1. Furthermore, the DWT transformed primary channel is concatenated (along the third dimension) 950, 960 with each of the DWT transformed secondary channels. Thus, the DWT transformed primary channel is used as auxiliary information for the processing of the (DWT transformed) secondary channels. The arrays resulting from the concatenation process are input to Networks 2 to N, respectively, which are networks of networks operating independently of each other. Network 1 is a network of networks of the same size (2 p-1 ×2 p-1 Networks 2 through N are designed to have inputs with more (tensor) layers than outputs (2 × 4) due to the concatenation process. p-1 ×2 p-1x8 inputs and 2 p-1 ×2 p-1 x4 output).

[0156] The output of network 1 undergoes the inverse DWT 970, and the outputs of networks, network 2 and network N also undergo the inverse DWT 980, 990. After applying the inverse DWT 970, 980, 990, the size of p ×2 p Correction of primary channel and size 2 p ×2 p A corrected secondary channel is obtained and corrected, e.g., an image of size 2 p ×2 p The next patch may be selected and processed as described above. All the processed patches may be assembled to build a reconstructed image, and any padding (if necessary) is removed.

[0157] The above mentioned filters include N DWT transforms, N neural networks, and N inverse DWT transforms. For example, they are trained to receive a distorted image (e.g., by compression) and output an approximation of the original image. Using the processed (e.g., restored) primary channel to help process (e.g., restore) the secondary channel allows the filters to exploit correlation between channels and achieve good reconstruction with a relatively small number of processing steps. The use of DWT transforms helps to decorrelate the input data (into high and low spatial frequency components), which helps the network reach good performance with fewer parameters and makes training easier compared to the state of the art. During the training process, each network learns to take a distorted input (e.g., by compression) and output the best approximation of the original (undistorted) image. This is done by training the network on pairs of distorted (source) and original (target) patches. One common approach is to train on multiple sets of distorted images, thus obtaining multiple sets of neural network coefficients, each set best suited for a particular type of content (e.g., computer games, video conferencing, etc.) or a particular level of distortion (e.g., high compression, medium compression, etc.). During compression, the encoder may analyze the content and select the best set of trained coefficients. This selection may also vary from patch to patch and channel to channel, e.g., network 1 may use a set of coefficients for "highly compressed computer screen content" and network 2 may use a set for "medium compressed video conferencing screen content," with processing by network N turned off and operating in a "pass-through" mode. Also, any of the networks (network 1 to network N) may be independently turned off and skip processing altogether.

[0158] Furthermore, the use of the DWT results in a rearrangement of the array elements input to the DWT, i.e., a quarter of the spatial dimensions and four layers along the third dimension. In the subsequent neural network, the DWT transform convolution is applied to the first two dimensions, which allows the neural network processing units to process the data efficiently in parallel.

[0159] In fact, compared to the state of the art, the overall process can be accelerated by more than five times depending on the actual application.

[0160] Moreover, in the art, all channels are combined and then processed together (see network M in FIG. 8). A set of networks is trained, each optimized for a different content. It may happen that the optimal processing setup for a given image involves turning off the processing of the primary channel and a different setup for each of the secondary channels. In the art, it was necessary to run all networks multiple times and assemble the output (e.g., take original for channel 1, version 4 for channel 2, etc.). According to the present disclosure, channel processing can be turned on / off independently, and the selection / training of weighting coefficients can advantageously be done independently for each of the networks / channels. Optimal parameter selection usually requires different processing for each of the channels, e.g., the three YUV channels. In the art, it is necessary to run the filter three times (for the three channels), using the optimal parameters of one of the components each time, and then combine the desired components to form the output. According to the present disclosure, the filter only needs to be run once, regardless of the filter selection. The coefficients of one of the neural networks can be changed without affecting the output of the other neural network.

[0161] As already mentioned, channels of different sizes corresponding to different channel resolutions can also be handled. Figure 10 shows an embodiment where the size of the primary channel is four times the size of each of the secondary channels, k...2 with integers k, l and p. p ×l…2 p The size of the image is size 2 p ×2 p The image is divided into k×l square patches, each having size 2. If the image cannot be divided into k×l square patches, the image is padded before division. p ×2 p A patch of is selected for processing 1010 and split into (distorted) channels, e.g., color channels as RGB or YUV (luminance and chroma) channels. One of the channels is selected or predetermined as the primary channel. The other channels are secondary channels that are processed based on information about the primary channel. The primary channel is a pixel of size 2 p ×2 p and each of the secondary channels has size 2 p-1 ×2 p-1 For example, the processed path of the image has a YUV420 format.

[0162] All channels are processed 1020, 1030, 1040 using a two-dimensional DWT. Any type of wavelet deemed appropriate may be used for the DWT. For example, the Haar wavelet or the Daubechies wavelet may be deemed suitable choices. The application of the DWT 1020 to the pixel values ​​of the pixels of the primary channels is performed using a two-dimensional wavelet with a third dimension given by the spatial low frequency subband LL and the spatial high frequency subbands HL, LH, HH. p-1 ×2 p-1 Correspondingly, application 1030, 1040 of the DWT to the pixel values ​​of the pixels of the secondary channel results in a DWT transformation of size 2 p-2 ×2 p-2 A ×4 DWT transform results in a secondary channel of size 2 p-1×2 p-1 The ×4 DWT transform primary channel is processed by network 1, which is designed to have input and output of the same size. p-1 ×2 p-1 The output of network 1 of size ×4 is a p-1 ×2 p-1 The 1080 undergoes inverse DWT to obtain the correction / correction primary channel.

[0163] In order to use the DWT-transformed primary channel as side information for the processing of the (DWT-transformed) secondary channel, the first two dimensions must be the same. Therefore, a size of 2 p-1 ×2 p-1 The primary channel of the ×4 DWT transform is of size 2 p-2 ×2 p-2 The three-dimensional array thus obtained is then subjected to a further (cascaded) DWT 1050 to obtain an auxiliary DWT transformed primary channel of size 16×16. The three-dimensional array thus obtained is concatenated 1060, 1070 with the array of DWT transformed secondary channels, and the concatenated array thus obtained is fed to the networks, Networks Network 2 to Network N, respectively. The networks, Networks Network 2 to Network N, are designed to have a larger input in the third dimension, and an output having a size in the third dimension of exactly 4 (due to the DWT / inverse DWT), and the output of the networks, Networks Network 2 to Network N, is of size 2×16. p-2 ×2 p-2 The inverse DWTS 1090 and 1100 are then subjected to the respective inverse DWTS 1090 and 1100 to obtain the corrected / compensated secondary channel of size 2 based on the combination of the corrected / compensated primary and secondary channels. p ×2 p A corrected (substantially undistorted) patch of . The next patch may be selected and processed as described above. All the processed patches may be assembled to build a reconstructed image, and any padding (if necessary) is removed.

[0164] The configuration illustrated in Figure 10 offers the same advantages as the configuration described with reference to Figure 9. In the embodiment described with reference to Figures 9 and 10, the DWT is used, but alternatively, any other spatial frequency transform deemed appropriate may be used. In particular, a transform may be chosen that does not result in a downsampling by a factor of two in the height and width dimensions of the image region.

[0165] The neural network used in the embodiments described with reference to Figures 9 and 10 may be a convolutional neural network. It is further noted that the embodiments described with reference to Figures 9 and 10 may be modified to be configured to perform the above-mentioned pixel shifting and pixel unshifting operations before the DWT and after the inverse DWT (or any other spatial frequency transform used), respectively.

[0166] Figure 11 illustrates a particular example of a convolutional neural network that may be used in the embodiments described with reference to Figures 9 and 10. Except for the number of input tensor layers, the topology of all of the networks illustrated in Figures 9 and 10 may be the same or similar to one another.

[0167] In the case of processing a DWT transformed primary channel, the number of input tensor layers is four. In the case of processing a DWT transformed secondary channel, the input of the involved neural network is larger to accommodate both the DWT transformed secondary channel itself and the DWT transformed primary channel concatenated in the third dimension. Furthermore, if one of the channels (e.g., the primary channel) is larger and a cascaded DWT is used to equalize the first two dimensions, each of the two concatenated channels may have a third dimension larger than four. For example, one channel may be two dimensions larger than the other. n If the number of input tensor layers in the secondary channel is 4…(2 n+1). When processing YUV420 content, the size of the primary channel is 4 times the size of the secondary channel (2 times in each direction), N=2, and there are 20 input tensor layers. The number of output tensor layers is 4 (as required by the following IDWT block) and is controlled by the size of the output convolution block ("Convolution 2" in Fig. 11). When the number of input and output tensor layers differs from each other, the "select m" block shown in Fig. 11 extracts only the first m tensor layers from the p(>m) input tensor layers (thus omitting information coming from the auxiliary input) and sends them to the summation block at the output.

[0168] Several cascaded residual blocks (ResBlocks) are placed between the input convolution block Convolution1 and the output convolution block Convolution2 and can be configured for residual learning during the training phase and during neural network inference. According to one embodiment, 8 ResBlocks with 48 features each are used, but other combinations are possible. Rectified Linear Unit (ReLU) layers are used, for example, to implement the activation function. Batch normalization layers and scaling layers, well known in the context of CNNs, may be used to facilitate the training process. According to some embodiments, such a scaling layer may apply a linear transformation to a preceding layer, for example, by multiplying each element of the preceding layer by a scaling factor. This may be achieved by a single scaling factor or by layers with multiple identical weights. According to some embodiments, scaling may be selectively applied to some elements of a preceding layer, for example, using a scaling layer, where only selected elements have weights that achieve the desired scaling value.

[0169] According to some embodiments, such a scaling layer may be represented by one or more scaling values ​​used as multipliers for the preceding block elements. Different arrangements of the scaling layers in the convolutional neural network layer configuration are contemplated. According to one configuration, the scaling layer may be placed after the output convolution block ("Convolution 2" in FIG. 11) as shown in FIG. 11 above. According to another configuration, the scaling layer may also be placed before the output of each ResBlock, as shown in FIG. 11 below. The scaling layer may be adapted to control the ratio of processed to unprocessed inputs to be summed before output. According to some embodiments, one or more scaling values ​​of the scaling layer may have default values ​​obtained during the training process. According to some embodiments, the encoder may also use different values ​​for a given image, image component, or image region. In such cases, the different values ​​may be provided by signaling one or more scale values ​​from the encoder to the decoder. In some embodiments, a configuration in which a single scaling element is used for all elements of the preceding layer may be preferred. In these embodiments, signaling a single scale value per ResBlock allows for great flexibility in controlling the output of the neural network.

[0170] In particular, as illustrated in FIG. 12, a method is provided for modifying an image region represented by two or more image channels. Here, the term "modify" refers to any modification, such as image filtering, image correction, etc. In principle, the image can be a patch of a given size corresponding to a part of an image or parts of multiple images, or it can be an image or multiple images. The two or more channels can be color channels, or other channels such as depth channels or multispectral image channels, or any other feature channels. One of the channels is a primary channel and another one of the channels is a secondary channel. There may be multiple secondary channels. The method may include selecting one of the two or more image channels as a primary channel and selecting at least another one of the two or more image channels as a secondary channel. It is noted that the primary channel can also be considered as a leading channel (according to some embodiments). The secondary channel can also be considered as a response / reaction channel (according to some embodiments). For example, two or more secondary channels can be selected.

[0171] The method 1200 of modifying an image region represented by two or more image channels illustrated in FIG. 12 includes a step S1210 of processing a primary channel of the two or more image channels based on a first spatial frequency transformation to obtain a transformed primary channel. Similarly, a secondary channel of the two or more image channels different from the primary channel is processed S1220 based on a second spatial frequency transformation to obtain a transformed secondary channel. Of course, multiple secondary channels can be processed by multiple second spatial frequency transformations (and multiple second neural networks, see below). The first spatial frequency transformation and the second spatial frequency transformation can be selected from the group consisting of energy compaction transformations including wavelet transformation (DWT or stationary wavelet transformation), discrete Fourier transform, fast Fourier transform, and discrete cosine transform. According to an example, the first spatial frequency transformation and the second spatial frequency transformation can be the same type of spatial transformation of the group.

[0172] Furthermore, the method 1200 includes a step S1230 of processing the transformed primary channel by a first neural network to obtain a modified transformed primary channel, and a step S1240 of processing the transformed secondary channel based on the transformed primary channel (used as auxiliary information) by a second neural network to obtain a modified transformed secondary channel. The first and second neural networks may be different from each other. If the primary and secondary channels have different sizes (according to different resolutions of image regions in different channels), a cascaded spatial frequency transform may be used on the larger of the channels to adjust the height and width dimensions of the transformed channels to each other (see the description with reference to FIG. 10 above).

[0173] The output of the first neural network, i.e. the modified transformed primary channel, is processed S1250 based on a first inverse spatial frequency transform (corresponding to the first spatial frequency transform) to obtain a modified primary channel. Similarly, the modified transformed secondary channel is processed S1260 based on a second inverse spatial frequency transform to obtain a modified secondary channel. Subsequently, a modified image region can be obtained S1270 based on the modified primary channel and the modified secondary channel. This procedure can be repeated for another selected image region. After all image regions of the image to be processed have been processed by the method 1200 illustrated in FIG. 12, a modified version of the entire image can be obtained.

[0174] Although operations are shown in a particular order in the figures, this should not be understood as requiring that such operations be performed in the particular order or sequence shown, or that all of the illustrated operations be performed, to achieve desired results. In certain situations, multitasking or parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0175] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the appended figures do not necessarily require the particular order or sequence depicted to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0176] The method 1200 illustrated in Figure 12 may be implemented in an apparatus 1300 configured to modify an image region illustrated in Figure 13, which may be configured to perform the steps of the method 1200 illustrated in Figure 12. The apparatus 1300 may be configured by an encoder (e.g., the encoder 20 illustrated in Figures 14 and 15) or a decoder (e.g., the decoder 20 illustrated in Figures 14 and 15), or may be included by the video coding device 8000 illustrated in Figure 17 or the apparatus 9000 illustrated in Figure 18.

[0177] The apparatus 1300 for modifying an image region represented by two or more image channels illustrated in FIG. 13 includes a first spatial frequency conversion unit 1310 configured to process a primary channel of the two or more image channels to obtain a transformed primary channel, and a second spatial frequency conversion unit 1320 configured to process a secondary channel of the two or more image channels different from the primary channel to obtain a transformed secondary channel.

[0178] Further, the apparatus 1300 includes a first neural network (NN) 1330 configured to process the transformed primary channel to obtain a modified transformed primary channel, and a second neural network (NN) 1340 configured to process the transformed secondary channel based on the transformed primary channel to obtain a modified transformed secondary channel. A first inverse spatial frequency transform unit 1350 configured to process the modified transformed primary channel to obtain a modified primary channel is also included by the apparatus 1300. Similarly, a second inverse spatial frequency transform section 1360 configured to process the modified transformed secondary channel to obtain a modified secondary channel is also included by the apparatus 1300.

[0179] Furthermore, the apparatus 1300 includes a combining unit 1370 configured to obtain a modified image region based on the modified primary channel and the modified secondary channel.

[0180] Some exemplary implementations in hardware and software A corresponding system in which the above-mentioned processes may be arranged, particularly with an encoder-decoder processing chain, is illustrated in FIG. 14. FIG. 14 is a schematic block diagram illustrating an example coding system, e.g., video, image, audio, and / or other coding system (or short coding system), that may utilize the techniques of the present application. The video encoder 20 (or short encoder 20) and the video decoder 30 (or short decoder 30) of the video coding system 10 represent examples of devices that may be configured to perform the techniques according to various examples described in the present application. For example, the video coding and decoding may be distributed and may use neural networks that may apply the above-mentioned bitstream parsing and / or bitstream generation to communicate feature maps between distributed computational nodes (two or more).

[0181] As shown in FIG. 14, coding system 10 includes a source device 12 configured to provide encoded picture data 21 to a destination device 14 for decoding, for example, encoded picture data 13.

[0182] The source device 12 includes an encoder 20, and may additionally, i.e. optionally, include a picture source 16, a pre-processor (or pre-processing unit) 18, for example a picture pre-processor 18, and a communications interface or unit 22.

[0183] Picture source 16 may include or be any kind of picture capture device, such as a camera for capturing real-world pictures, and / or any kind of picture generation device, such as a computer graphics processor for generating computer-animated pictures, or any kind of other device for obtaining and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). Picture source may also be any kind of memory or storage that stores any of the aforementioned pictures.

[0184] In distinction to the pre-processor 18 and the processing performed by the pre-processing unit 18 , the picture or picture data 17 may also be referred to as a raw picture or raw picture data 17 .

[0185] The pre-processor 18 is configured to receive (raw) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It will be understood that the pre-processing unit 18 may be an optional component. It is noted that the pre-processing may also use a neural network (as shown in any of Figures 1 to 7) that uses presence indicator signaling.

[0186] Video encoder 20 is configured to receive pre-processed picture data 19 and to provide encoded picture data 21 .

[0187] Communications interface 22 of source device 12 may be configured to receive encoded picture data 21 and transmit the encoded picture data 21 (or any further processed version thereof) via communications channel 13 to another device, such as destination device 14 or any other device, for storage or direct reconstruction.

[0188] The destination device 14 includes a decoder 30 (e.g., a video decoder 30) and may additionally, i.e. optionally, include a communications interface or unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0189] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof), for example from the source device 12 or directly from any other source, for example a storage device, for example an encoded picture data storage device, and to provide the encoded picture data 21 to a decoder 30.

[0190] The communications interface 22 and the communications interface 28 may be configured to transmit or receive the encoded picture data 21 or the encoded data 13 via a direct communications link between the source device 12 and the destination device 14, e.g., a direct wired or wireless connection, or via any type of network, e.g., a wired or wireless network or any combination thereof, any type of private and public network, or any combination thereof.

[0191] The communications interface 22 may be configured, for example, to package the encoded picture data 21 in a suitable format, e.g., packets, and / or process the encoded picture data using any type of transmission encoding or processing for transmission over a communications link or network.

[0192] Communications interface 28, forming a counterpart of communications interface 22, may for example be configured to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or depackaging to obtain encoded picture data 21.

[0193] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow for communication channel 13 in FIG. 14 pointing from source device 12 to destination device 14, or as bidirectional communication interfaces, and may be configured, e.g., to send and receive messages, e.g., to set up connections, and to ascertain and exchange any other information related to the communication link and / or data transmission, e.g., encoded picture data transmission. Decoder 30 is configured to receive encoded picture data 21 and provide decoded picture data 31 or decoded pictures 31.

[0194] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), e.g., the decoded picture 31, to obtain post-processed picture data 33, e.g., the post-processed picture 33. The post-processing performed by the post-processing unit 32 may include color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing to prepare the decoded picture data 31 for display, e.g., by a display device 34.

[0195] A display device 34 of destination device 14 is configured to receive the post-processed picture data 33, for example for displaying the picture to a user or viewer. The display device 34 may be or include any type of display for presenting the reconstructed picture, for example an integrated or external display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any type of other display.

[0196] 14 illustrates source device 12 and destination device 14 as separate devices, an embodiment of the devices may also include both or both functionality, source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0197] As will be apparent to a person skilled in the art based on the description, the presence and (exact) division of functions or functions of different units within source device 12 and / or destination device 14 as shown in FIG. 14 may vary depending on the actual device and application.

[0198] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30) or both the encoder 20 and the decoder 30 may be implemented via processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding, or any combination thereof. The encoder 20 may be implemented via processing circuitry 46 to embody various modules, including a neural network or portions thereof. The decoder 30 may be implemented via processing circuitry 46 to embody any coding system or subsystem described herein. The processing circuitry may be configured to perform various operations, as described below. If the technology is implemented in part in software, the device may store instructions for the software in a suitable non-transitory computer-readable storage medium and may execute the instructions in hardware to perform the technology of the present disclosure using one or more processors. Either the video encoder 20 and the video decoder 30 may be integrated as part of a combined encoder / decoder (codec) in a single device, for example, as shown in FIG. 15.

[0199] Source device 12 and destination device 14 may include any of a wide range of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video gaming console, a video streaming device (such as a content service server or a content delivery server), a broadcast receiving device, a broadcast transmitting device, and the like, and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0200] In some cases, the video coding system 10 illustrated in FIG. 14 is only an example, and the techniques of the present application may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily include data communication between encoding and decoding devices. In other examples, data may be retrieved from local memory or streamed over a network. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but simply encode data in memory and / or retrieve data from memory and decode it.

[0201] 16 is a schematic diagram of a video coding device 8000 according to one embodiment of the present disclosure. The video coding device 8000 is suitable for implementing embodiments of the disclosure described herein. In one embodiment, the video coding device 8000 may be a decoder, such as the video decoder 30 of FIG. 14, or an encoder, such as the video encoder 20 of FIG. 14.

[0202] The video coding device 8000 includes a receive port 8010 (or input port 8010) and a receiver unit (Rx) 8020 for receiving data, a processor, logic unit, or central processing unit (CPU) 8030 for processing the data, a transmitter unit (Tx) 8040 and a transmit port 8050 (or output port 8050) for transmitting data, and a memory 8060 for storing data. The video coding device 8000 may also include optical-electrical (OE) and electrical-optical (EO) components coupled to the receive port 8010, the receiver unit 8020, the transmitter unit 8040, and the transmit port 8050 for transmitting or receiving optical or electrical signals.

[0203] The processor 8030 is implemented by hardware and software. The processor 8030 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGA, ASIC, and DSP. The processor 8030 communicates with the receiving port 8010, the receiver unit 8020, the transmitter unit 8040, the transmitting port 8050, and the memory 8060. The processor 8030 includes a neural network-based codec 8070. The neural network-based codec 8070 implements the above-described embodiments of the disclosure. For example, the neural network-based codec 8070 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the neural network-based codec 8070 provides a substantial improvement to the functionality of the video coding device 8000 and provides the transformation of the video coding device 8000 into different states. Alternatively, the neural network based codec 8070 is implemented as instructions stored in memory 8060 and executed by the processor 8030.

[0204] The memory 8060 may include one or more disks, tape drives, and solid state drives, and may be used as an overflow data storage device to store programs when such programs are selected for execution and to store instructions and data read during program execution. The memory 8060 may be, for example, volatile and / or non-volatile, and may be read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0205] FIG. 17 is a simplified block diagram of an apparatus that may be used as either or both of source device 12 and destination device 14 from FIG. 14, according to an example embodiment.

[0206] The processor 9002 in the device 9000 may be a central processing unit. Alternatively, the processor 9002 may be any other type of device, or multiple devices, now existing or later developed, capable of manipulating or processing information. Although implementations of the disclosure may be implemented using a single processor as shown, e.g., processor 9002, advantages in speed and efficiency may be achieved using multiple processors.

[0207] The memory 9004 in the apparatus 9000 may be a read-only memory (ROM) device or a random access memory (RAM) device in one implementation. Any other suitable type of storage device may be used as the memory 9004. The memory 9004 may include code and data 9006 that is accessed by the processor 9002 using the bus 9012. The memory 9004 may further include an operating system 9008 and application programs 9010, which include at least one program that enables the processor 9002 to perform the methods described herein. For example, the application programs 9010 may include applications 1 through N, which further include a video coding application that performs the methods described herein.

[0208] The apparatus 9000 may also include one or more output devices, such as a display 9018. The display 9018, in one example, may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 9018 may be coupled to the processor 9002 via the bus 9012.

[0209] Although shown here as a single bus, the bus 9012 of the device 9000 may be comprised of multiple buses. Additionally, the secondary storage may be directly coupled to other components of the device 9000 or may be accessed over a network, and may include a single integrated unit such as a memory card, or multiple units such as multiple memory cards. Thus, the device 9000 may be implemented in a wide variety of configurations. [Explanation of symbols]

[0210] 11 Part of the input image 210 Encoder side 220 Encoder Sub-Network 230 Representation (encoding) of dataset x 250 Decoder side 260 Decoder Sub-Network 101 Encoder 102 Quantizer 103 HyperEncoder 104 Decoder 105 Arithmetic Encoder 106 Arithmetic Decoder 107 Hyper Decoder 108 Quantizer 109 Arithmetic Encoder 110 Arithmetic Decoder 121 Encoder 122 Quantizer 123 HyperEncoder 125 Arithmetic Encoding Module 127 Hyper Decoder 144 Decoder 147 Hyper Decoder 401 Downsampling Layer 402 Downsampling Layer 403 Downsampling Layer 404 Downsampling Layer 405 Downsampling Layer 406 Downsampling Layer 407 Upsampling Layer 408 upsampling layers 409 Upsampling Layer 410 Upsampling Layer 411 Upsampling Layer 412 upsampling layers 413 Components 414 Input images 415 Components 420 More Layers 430 corresponding convolutional layers 510 Mobile side 520 Quantization layer 550 hidden layers (deep feature maps) 560 Inverse quantization layer 590 Cloud side 810 Selecting a patch 820 pixel shift 830 Connect 840 Connect 850 Connect 860 pixel unshift 910 Select a patch 920 Discrete Wavelet Transform (DWT) 930 DWT 940 DWT 950 Connect 960 Connect 970 Reverse DWT 980 Reverse DWT 990 Reverse DWT 1010 Select a patch 1020 DWT 1030 DWT 1040 DWT 1050 DWT 1060 Connect 1070 Connect 1080 Reverse DWT 1090 Reverse DWT 1100 Reverse DWT 1200 How to rectify image regions represented by two or more image channels 1300 Apparatus for modifying an image region represented in two or more image channels 1310 1st spatial frequency conversion unit 1320 Second spatial frequency conversion unit 1330 The First Neural Network 1340 Second Neural Network 1350 First inverse spatial frequency transformation unit 1360 Second Inverse Spatial Frequency Transformation Unit 1370 Combined Unit 10. Video Coding System 12 Source Device 13 Communication Channels 14 Destination Device 16 Picture Source 17 Picture Data 18 Preprocessors 19 Preprocessed Picture Data 20 Encoder / Video Encoder 21 Encoded Picture Data 22 Communication Interface 28 Communication Interface 30 Decoder / Video Decoder 31 Decoded Picture Data 32 Post Processor 33 Post-Processed Picture Data 34 Display Devices 40 Video Coding System 41 Imaging device(s) 42 Antenna 43 (one or more) processors 44 (one or more) memory stores 45 Display Devices 46 Processing Circuit 8000 Video Coding Device 8010 inbound port 8020 Receiver Unit (Rx) 8030 processor 8040 Transmitter Unit (Tx) 8050 outbound port 8060 Memory 8070 Neural Network Based Codec 9000 equipment 9002 (one or more) processors 9004 Memory 9006 Data 9008 Operating Systems 9010 Application Program 9012 Bus 9018 Display

Claims

1. 1. A method of modifying an image region represented by two or more image channels, the method comprising: processing a primary channel of the two or more image channels based on a first spatial frequency transform to obtain a transformed primary channel (S1210); processing a secondary channel of the two or more image channels that is different from the primary channel based on a second spatial frequency transform to obtain a transformed secondary channel (S1220); processing the transformed primary channel by a first neural network to obtain a modified transformed primary channel (S1230); processing (S1240) the transformed secondary channel based on the transformed primary channel by a second neural network to obtain a modified transformed secondary channel; processing (S1250) the modified transformed primary channel based on a first inverse spatial frequency transform to obtain a modified primary channel; processing (S1260) the modified transformed secondary channel based on a second inverse spatial frequency transform to obtain a modified secondary channel; obtaining a modified image region based on the modified primary channel and the modified secondary channel (S1270); A method comprising:

2. 2. The method of claim 1, wherein one or both of the first spatial frequency transform and the second spatial frequency transform are selected from the group consisting of energy compaction transforms including a wavelet transform, a discrete Fourier transform, a fast Fourier transform, and a discrete cosine transform.

3. The method of claim 2 , wherein both the first spatial frequency transform and the second spatial frequency transform are one of a wavelet transform, a discrete Fourier transform, a fast Fourier transform, an energy compaction transform, and a discrete cosine transform.

4. 4. The method of claim 2 or 3, wherein one or both of the first spatial frequency transform and the second spatial frequency transform is a wavelet transform selected from the group consisting of a discrete wavelet transform and a stationary wavelet transform.

5. The method of claim 1 , further comprising the step of selecting the primary channel from the two or more image channels.

6. The method of claim 5 , further comprising the step of selecting the secondary channel from the two or more image channels.

7. The method of claim 6 , wherein the primary channel and the secondary channel are selected from the two or more image channels based on the output of a classifier operating based on another neural network.

8. 8. A method according to claim 1, wherein the processing (S1240) of the transformed secondary channel based on the transformed primary channel comprises concatenating (950, 960) a second three-dimensional tensor representing the transformed secondary channel with a first three-dimensional tensor representing the transformed primary channel.

9. The method of claim 1 , wherein the size of the primary channel is different from the size of the secondary channel.

10. a) if the size of the primary channel is greater than the size of the secondary channel; processing the transformed primary channel based on at least one additional first spatial frequency transform to obtain an auxiliary transformed primary channel having the same size in height and width directions of the image region as the transformed secondary channel (S1230); The processing (S1240) of the converted secondary channel is based on the auxiliary converted primary channel, b) if the size of the secondary channel is greater than the size of the primary channel; processing the transformed secondary channel based on at least one additional second spatial frequency transform to obtain an auxiliary transformed secondary channel having the same size in the height and width directions of the image region as the transformed primary channel (S1240); said processing (S1240) of said transformed secondary channel includes processing said auxiliary transformed secondary channel based on said transformed primary channel; The method of claim 9.

11. The processing (S1240) of the transformed secondary channel based on the transformed primary channel comprises: a) if the size of the primary channel is greater than the size of the secondary channel; Concatenating (1060, 1070) a second 3D tensor representing the transformed secondary channel with a first 3D tensor representing the auxiliary transformed primary channel; b) if the size of the secondary channel is greater than the size of the primary channel; concatenating a second three-dimensional tensor representing the auxiliary transformed secondary channel with a first three-dimensional tensor representing the transformed primary channel; The method of claim 10, comprising:

12. 12. The method of claim 1, wherein the image region is a square region in the height and width dimensions of the image region.

13. 13. The method of claim 1, further comprising the steps of dividing the image into image regions comprising the image region, and padding image regions resulting from the division that are not square in the height and width dimensions of the image region such that the image regions are square in the height and width dimensions of the image region.

14. Segmenting the image into image regions including said image region; if the image cannot be divided only into image regions that are square in the height and width dimensions of the image region, padding the image such that the image is divided only into image regions that are all square in the height and width dimensions of the image region containing the image region; 13. The method of any one of claims 1 to 12, further comprising:

15. 15. The method of claim 1, wherein the first neural network and the second neural network operate independently of each other.

16. 16. The method of claim 15, wherein weights of one of the first and second neural networks are determined and used independently from weights of the other of the first and second neural networks.

17. 17. The method of claim 1, wherein each of the first neural network and the second neural network is or includes a convolutional neural network.

18. 20. The method of claim 17, wherein each of the convolutional neural networks includes at least one residual network component.

19. 19. The method of claim 17 or 18, wherein one or more of the convolutional neural networks employ a scaling layer represented by one or more scaling values.

20. The method of claim 19 , wherein the scaling layer is adapted to signal the one or more scaling values.

21. 21. The method of claim 1, wherein the two or more image channels include a color channel and / or a feature channel.

22. The image area includes: a patch of a given size corresponding to a portion of an image or portions of multiple images, or An image or multiple images 22. The method according to any one of claims 1 to 21, wherein

23. obtaining an original image region; encoding the captured image region into a bitstream; applying the method according to any one of claims 1 to 22 to modify the acquired image area by reconstructing the encoded image area; 23. A method for encoding an image or a video sequence or an image, comprising:

24. 24. A method for encoding an image or a video sequence of images as claimed in claim 23, comprising the step of including an indication of the selected primary channel in the bitstream.

25. 25. A method for encoding an image or a video sequence of images as claimed in claim 23 or 24, comprising the step of including in the bitstream an adaptation of one or more weights of at least one of the first and second neural networks.

26. acquiring a plurality of image regions; applying the method for modifying the captured image regions individually to the image regions of the captured image regions; in the bitstream for each of the plurality of image regions, an indication that the method for modifying the captured image region should not be applied to the image region; an indication of the selected primary channel for the region; Adapting the one or more weights of at least one of the first neural network and the second neural network. and including at least one of:

26. A method for encoding an image or a video sequence of images according to any one of claims 23 to 25, comprising:

27. 1. A method for decoding an image or a video sequence or an image from a bitstream, comprising the steps of: reconstructing an image region from the bitstream; applying the method according to any one of claims 1 to 22 to modify said image region; A method comprising:

28. 1. A method for decoding an image or a video sequence or an image from a bitstream, comprising the steps of: an indication that the method for modifying the captured image region should not be applied to the image region; an indication of a primary channel for said region; Adapting one or more weights of at least one of the first neural network and the second neural network. parsing the bitstream to obtain at least one of: reconstructing an image region from the bitstream; if the indication indicates a selected primary channel, modifying the reconstructed image region according to the method of any one of claims 1 to 22 with the indicated primary channel as the selected primary channel; A method comprising:

29. if an adaptation of weights of at least one of the first and second neural networks is present in the bitstream, modifying the weights of the respective neural network accordingly.

30. A method for decoding an image or a video sequence or an image from a bitstream according to claim 28, further comprising:

30. A method for decoding an image or a video sequence according to claim 27 or 28, wherein the modification of the image region is applied by an in-loop filter or a post-processing filter.

31. 31. A computer program product comprising program code stored on a non-transitory medium, the program being adapted, when executed on one or more processors, to perform the method of any one of claims 1 to 30.

32. 31. An apparatus for modifying an image region represented by two or more image channels, the apparatus comprising circuitry configured to perform steps according to the method of any one of claims 1 to 30.

33. An apparatus (1300) for modifying an image region represented by two or more image channels, comprising: a first spatial-frequency transformation unit (1310) configured to process a primary channel of the two or more image channels to obtain a transformed primary channel; a second spatial-frequency transformation unit (1320) configured to process a secondary channel of the two or more image channels different from the primary channel to obtain a transformed secondary channel; a first neural network (1330) configured to process the transformed primary channel to obtain a modified transformed primary channel; a second neural network (1340) configured to process the transformed secondary channel based on the transformed primary channel to obtain a modified transformed secondary channel; a first inverse spatial-frequency transformation unit (1350) configured to process the modified transformed primary channel to obtain a modified primary channel; a second inverse spatial-frequency transformation unit (1360) configured to process the modified transformed secondary channel to obtain a modified secondary channel; a combining unit (1370) configured to obtain a modified image region based on the modified primary channel and the modified secondary channel; The apparatus (1300).

34. An encoder for encoding an image or a video sequence or an image, said encoder comprising: an input module for obtaining an original image region; a compression module for encoding the captured image regions into a bitstream; a reconstruction module for reconstructing the encoded image region; An apparatus (1300) for modifying a reconstructed image region according to claim 32 or 33; Including, an encoder.

35. 1. A decoder for decoding an image or a video sequence or an image from a bitstream, the decoder comprising: a reconstruction module for reconstructing image regions from the bitstream; An apparatus (1300) for modifying a reconstructed image region according to claim 32 or 33; A decoder.

Images