Global skip connection based cnn filter for image and video coding

Abstract

The invention relates to image processing, in particular to modifying an image using a neural network or the like. The processing is performed to generate a corrected image from an input image. The input image is then modified by combining it with the corrected image. The processing using the neural network comprises at least one stage of image down-sampling and filtering the down-sampled image; at least one stage of image up-sampling. One advantage of this approach is that it improves the efficiency of the neural network, which can speed up the learning and improve the performance. Embodiments of the invention provide methods and apparatus for processing using a trained neural network, and methods and apparatus for training such a neural network for image modification.

Description

[0001] This application is a divisional application. The original application has the application number 202180035432.4 and the original application date is April 20, 2021. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] The embodiments of the present invention generally relate to the field of image processing, and more specifically, to neural network-based filtering for image and video encoding. Background Technology

[0003] Video decoding (video encoding and decoding) is widely used in digital video applications such as broadcast digital television (TV), video transmission based on the Internet and mobile networks, real-time conversational applications such as video chat and video conferencing, DVD and Blu-ray discs, video content capture and editing systems, and portable cameras for security applications.

[0004] Even with shorter videos, a large amount of video data is required for description, which can be challenging when data needs to be sent over communication networks with limited bandwidth or transmitted in other ways. Therefore, video data is typically compressed before transmission over modern telecommunications networks. The size of the video can also be a problem when storing it on storage devices due to potentially limited memory resources. Video compression devices typically encode video data using software and / or hardware at the source side before transmission or storage, reducing the amount of data required to represent the digital video image. Video decompression devices then receive the compressed data at the destination side. Given limited network resources and the growing demand for higher video quality, there is a need to improve compression and decompression techniques that can increase compression ratios with minimal sacrifice in image quality.

[0005] Generally, image compression can be lossless or lossy. In lossless image compression, the original image can be perfectly reconstructed from the compressed image. However, the compression ratio is quite low. In contrast, lossy image compression can achieve a high compression ratio, but the downside is that it cannot perfectly reconstruct the original image. Especially when used at low bitrates, lossy image compression introduces visible spatial compression artifacts. Summary of the Invention

[0006] The present invention relates to methods and apparatus for image modification (e.g., image enhancement or other types of modification).

[0007] This invention is defined by the scope of the independent claims. Several advantageous embodiments are provided in the dependent claims.

[0008] In particular, embodiments of the present invention provide an efficient image modification method by using machine learning features.

[0009] According to one aspect, a method for modifying an input image is provided, wherein the method includes: generating a corrected image by processing the input image using a neural network, wherein the processing using the neural network includes at least one stage of image downsampling and filtering the downsampled image; at least one stage of image upsampling; and modifying the input image by combining the input image with the corrected image.

[0010] This method provides an efficient processing approach where only the corrected image is learned and generated, rather than the entire image, in order to modify the input image.

[0011] In one exemplary implementation, the corrected image and the input image have the same vertical and horizontal dimensions. The corrected image is a difference image, and the combination is performed by adding the difference image to the input image.

[0012] Providing a difference image of the same size as the input image enables low-complexity combination and processing.

[0013] For example, the neural network is based on a U-net. To build the neural network, the U-net is modified by introducing skip connections adapted to connect the input image to the output image.

[0014] U-net has a structure that is advantageous for image processing. The use of U-net also allows for at least partial utilization of some available implementations in certain processing stages, or further modification of these implementations, potentially resulting in simpler implementations.

[0015] In one embodiment, the neural network is parameterized according to the value of a parameter representing the amount or type of distortion of the input image.

[0016] Parameterizing a neural network by the type or amount of distortion may help train the network specifically for different types and amounts of distortion, thus providing more accurate results.

[0017] According to one embodiment, the image downsampling is performed by applying strided convolution and / or padding convolution.

[0018] Applying the strided convolution can reduce complexity, while using padded convolution may help maintain image size throughout the processing.

[0019] In one exemplary implementation, the activation function of the neural network is a leaky rectified linear unit (ReLU) activation function. A leaky ReLU is close to the sigmoid function and allows for improved learning.

[0020] According to one aspect, a method for reconstructing an encoded image from a bitstream is provided. The method includes: decoding the encoded image from the bitstream and applying the method described above for modifying an input image, wherein the input image is the decoded image. This corresponds to the application of processing as a post-filter, such as reducing compression artifacts or addressing specific perceptual preferences of the viewer.

[0021] According to one aspect, a method for reconstructing a compressed image of video is provided, comprising: reconstructing an image using image prediction based on a reference image stored in a memory; applying a method for modifying an input image as described above, wherein the input image is the reconstructed image; and storing the modified image in the memory as a reference image. This corresponds to applying the processing as an in-loop filter, for example, reducing compression artifacts during encoding and / or decoding. The improvement is not only at the level of the decoded image, but also due to the in-loop application, the prediction can also be improved.

[0022] According to one aspect, a method for training a neural network to modify a distorted image is provided, wherein the method includes: inputting a distorted image as a target input and a corrected image as a target output into the neural network, wherein the corrected image is obtained based on an original image, wherein processing performed using the neural network includes at least one stage of image downsampling and filtering the downsampled image; and at least one stage of image upsampling, adjusting at least one parameter of the filtering based on the input pair.

[0023] For example, the at least one parameter of the filter is adjusted according to a loss function corresponding to the mean squared error (MSE).

[0024] Alternatively or additionally, the at least one parameter of the filter is adjusted according to a loss function that includes the weighted average of the squared differences of multiple color channels.

[0025] According to one aspect, a computer program is provided, wherein when executed on one or more processors, the one or more processors are caused to perform the steps of the method described above.

[0026] According to one aspect, an apparatus for modifying an input image is provided. The apparatus includes: a processing unit for generating a corrected image by processing the input image using a neural network, wherein the processing using the neural network includes at least one stage of image downsampling and filtering the downsampled image; at least one stage of image upsampling; and a modification unit for modifying the input image by combining the input image with the corrected image.

[0027] According to one aspect, an apparatus is provided for reconstructing an encoded image from a bitstream, wherein the apparatus includes: a decoding unit for decoding the encoded image from the bitstream, the apparatus being configured to modify the decoded image as described above.

[0028] According to one aspect, an apparatus for reconstructing a compressed image of a video is provided, wherein the apparatus (device) comprises: a reconstruction unit for reconstructing an image using image prediction based on a reference image stored in a memory; the apparatus for modifying the decoded image as described above; and a storage unit for storing the modified image as a reference image.

[0029] According to one aspect, an apparatus for training a neural network to modify a distorted image is provided, wherein the apparatus includes: a training input unit for inputting a distorted image as a target input and a corrected image as a target output into the neural network, wherein the corrected image is obtained from an original image; a processing unit for processing using the neural network, wherein the processing using the neural network includes at least one stage of image downsampling and filtering the downsampled image; at least one stage of image upsampling; and an adjustment unit for adjusting at least one parameter of the filtering based on the input pair.

[0030] In addition, a method corresponding to the steps performed by the above-described processing circuit is also provided.

[0031] According to one aspect, a computer product is provided, comprising program code for executing the method described above. The computer product may be provided on a non-transitory medium and includes instructions that, when executed on one or more processors, perform the steps of the method.

[0032] Any of the above devices can be implemented on an integrated chip.

[0033] Any of the above embodiments and exemplary implementations can be combined. Attached Figure Description

[0034] The embodiments of the present invention are described in detail below with reference to the following figures, wherein: Figure 1 This is an exemplary flowchart of a method for modifying an image.

[0035] Figure 2 A schematic diagram of a general machine learning system that follows a U-shape.

[0036] Figure 3 A schematic diagram of a specific exemplary U-net class structure.

[0037] Figure 4 This is a schematic diagram illustrating known applications of U-net in image segmentation.

[0038] Figure 5 This is a schematic diagram of the combination of an input image and a corrected image that produces the modified image.

[0039] Figure 6 This is a schematic diagram of a specific exemplary U-net class structure with a global skip connection.

[0040] Figure 7 This is a schematic diagram of some types of connections used in neural network processing.

[0041] Figure 8 This is a schematic diagram of the step max pooling operation.

[0042] Figure 9A This is a schematic diagram of a non-stride convolution with a stride of 1.

[0043] Figure 9B This is a schematic diagram of strided convolution with a stride of 2.

[0044] Figure 10 This is a schematic diagram of a filled convolution.

[0045] Figure 11 A block diagram of an exemplary device for modifying an image.

[0046] Figure 12 This is a block diagram of an example video encoding system for implementing embodiments of the present invention.

[0047] Figure 13 This is a block diagram of another example of a video decoding system for implementing embodiments of the present invention.

[0048] Figure 14 This is a block diagram of an example of a video encoder used to implement an embodiment of the present invention.

[0049] Figure 15 This is a block diagram of an exemplary structure for implementing a video decoder according to an embodiment of the present invention.

[0050] Figure 16 A block diagram of a decoding device that modifies the applied image as a post-filter.

[0051] Figure 17 A block diagram of a decoding device that modifies the image as a loop filter.

[0052] Figure 18 This is a block diagram of a training device used to train a neural network.

[0053] Figure 19A block diagram of an example encoding or decoding device.

[0054] Figure 20 A block diagram for another example of an encoding or decoding device. Detailed Implementation

[0055] In the following description, reference is made to the accompanying drawings, which form part of this invention, and which illustrate specific aspects of embodiments of the invention or aspects in which embodiments of the invention may be used. It should be understood that embodiments of the invention can be used in other aspects and include structural or logical variations not depicted in the drawings. Therefore, the detailed description below is not intended to be limiting, and the scope of the invention is defined by the appended claims.

[0056] It is understood that the disclosure relating to the described methods also applies to the devices or systems corresponding to performing the methods, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units, for performing the described one or more method steps (e.g., a unit performing the one or more steps, or multiple units each performing one or more of the plurality of steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, for example, if a particular apparatus is described according to one or more units (e.g., functional units), the corresponding method may include a step to perform the function of one or more units (e.g., a step performing the function of the one or more units, or multiple steps each performing the function of one or more of the plurality of units), even if such one or more steps are not explicitly described or illustrated in the figures. Furthermore, it should be understood that, unless otherwise specifically indicated, features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0057] Video decoding generally refers to the processing of a sequence of images that constitute a video or video sequence. In the field of video decoding, the terms "frame" or "picture / image" can be used synonymously. Video decoding (or generally referred to as decoding) consists of two parts: video encoding and video decoding. Video encoding is performed on the source side and typically involves processing (e.g., by compression) the raw video image to reduce the amount of data required to represent the video image (for more efficient storage and / or transmission). Video decoding is performed on the destination side and typically involves the inverse processing relative to the encoder to reconstruct the video image. The “decoding” of video images (or generally referred to as images) in the embodiments should be understood as involving the “encoding” or “decoding” of video images or corresponding video sequences. The encoding and decoding parts are also collectively referred to as codec (CODEC) (encoding and decoding).

[0058] In lossless video decoding, the original video image can be reconstructed, meaning the reconstructed video image has the same quality as the original (assuming no transmission loss or other data loss during storage or transmission). In lossy video decoding, further compression (e.g., through quantization) is performed to reduce the amount of data representing the video image that cannot be fully reconstructed in the decoder; that is, the reconstructed video image has lower or worse quality compared to the original video image.

[0059] Several video decoding standards belong to the "lossy hybrid video codec" group (i.e., combining spatial and temporal prediction in the sample domain with 2D transform decoding for quantization in the transform domain). Each image in a video sequence is typically segmented into a set of non-overlapping blocks, and decoding is usually performed at the block level. In other words, in the encoder, the video is typically processed (i.e., encoded) in blocks (video blocks), for example, by generating prediction blocks through spatial (intra-frame) prediction and temporal (inter-frame) prediction; the prediction blocks are subtracted from the current block (the block currently being processed / to be processed) to obtain residual blocks; the residual blocks are transformed and quantized in the transform domain to reduce the amount of data to be transmitted (compressed), while in the decoder, the inverse processing relative to the encoder is applied to the encoded or compressed blocks to reconstruct the current block for representation. Furthermore, the encoder's processing loop is the same as the decoder's processing loop, such that both will produce the same prediction (e.g., intra-frame and inter-frame prediction) blocks and / or reconstructed blocks for processing (i.e., decoding) of subsequent blocks.

[0060] To date, numerous image compression codecs exist. For ease of description, embodiments of the invention will be described, for example, by reference to currently state-of-the-art image codecs. The most advanced image codec currently available is Better Portable Graphics (BPG), based on intra-frame coding of the video compression standard High Efficiency Video Coding (HEVC, H.265). BPG has been proposed to replace the Joint Photographic Experts Group (JPEG) standard as a more compression-efficient alternative in terms of image quality and file size. It will be understood by those skilled in the art that embodiments of the invention are not limited to these standards.

[0061] However, while lossy image compression can achieve high compression ratios, a drawback of all compression codecs is the presence of visible spatial compression artifacts. Some exemplary compression artifacts in BPG image codecs can be blockiness, blurring, ringing, staircases, or basic patterns. However, many more types of artifacts can occur, and this invention is not limited to the artifacts described above.

[0062] In recent years, neural networks have garnered significant attention, leading to proposals for their application in image processing. Convolutional Neural Networks (CNNs), in particular, have been used for such applications. One possibility is to completely replace the compression pipeline with a neural network. The CNN then learns image compression end-to-end. Several publications have proposed this approach in the literature. While structural compression artifacts are significantly reduced in learned image compression, only recent publications have shown compression ratios comparable to BPG.

[0063] Another possibility for reducing these compression artifacts is to apply filters after compression. Simple in-loop filters already exist in the HEVC compression standard. More complex filters have been proposed in the literature, particularly those based on Convolutional Neural Networks (CNNs). However, the improvement in visual quality has been limited.

[0064] A neural network is a signal processing model that supports machine learning. This model mimics the human brain and consists of multiple interconnected neurons. In neural network implementations, the signal at the connection between two neurons is a number, and the output of each neuron is calculated using a nonlinear function of its weighted input sum. These connections are called edges. Neurons and edges typically have weights that adjust as they are learned. Weights increase or decrease the signal strength at the connection. The nonlinear function of the weighted sum is also called the "activation function" or "neuron transfer function." In some simple implementations, the output can be binary, depending on whether the weighted sum is greater than a certain threshold, corresponding to the step function as a nonlinear activation function. In other implementations, other activation functions, such as sigmoid, can be used. Typically, neurons are clustered in layers. Different layers can perform different transformations on their inputs. The signal travels from the first layer (input layer) to the last layer (output layer), possibly traversing multiple layers. Weights are learned through training, which can be performed through supervised or unsupervised learning. It should be noted that the above model is only a general model. For specific applications, neural networks can have different processing stages, which can correspond to CNN layers and be suitable for the desired input, such as an image.

[0065] CNNs are a subclass of neural networks that use shared weights to reduce the number of trainable parameters. They are most commonly used for visual images.

[0066] In some embodiments of this application, a deep convolutional neural network (CNN) is trained to reduce compression artifacts and enhance the visual quality of the image while maintaining a high compression ratio.

[0067] Specifically, according to one embodiment, a method for modifying an input image is provided. Here, modification refers to any alteration, such as modifications typically made through filtering or other image enhancement methods. The type of modification may depend on the specific application. The method includes the steps of: generating a corrected image by processing the input image using a neural network, wherein the processing using the neural network includes at least one stage of image downsampling and filtering the downsampled image; at least one stage of image upsampling; and modifying the input image by combining the input image with the corrected image. The term "corrected image" here refers to an image other than the input image and is used to modify the input image.

[0068] An example of method 100 according to this embodiment is shown in Figure 1 and Figure 2 As shown in the image.

[0069] Downsampling and filtering 120 can be a shrinking path 299, and upsampling and filtering 130 can be an expanding path 298 of a neural network (also known as a "neural network" or "network"). The shrinking path is a convolutional network that may include repeated applications of convolutions, each followed by an activation function and downsampling of the image.

[0070] The method according to this embodiment can use at least one convolutional stage and at least one activation function stage in both downsampling and upsampling. During the shrinking process, spatial information decreases while feature information increases. The expanding path combines features and spatial information with high-resolution features in the shrinking path through a series of upconvolutions and concatenations.

[0071] For example, the activation function could be the rectified linear unit (ReLU). For all negative numbers, ReLU is 0; for positive numbers, ReLU is a linear function (ramp). However, the invention is not limited to this; different activation functions, such as the sigmoid or step function, can often be used. The shape of the ReLU function is similar to the sigmoid, but less complex.

[0072] Generally, downsampling can be performed in many different ways. For example, every second row / line of an image can be discarded. Alternatively, max pooling can be applied, which replaces x samples with the sample that has the maximum value among the x samples. Another possibility is to replace x samples with a sample that is equal to the average of the x samples. This invention is not limited to any particular method, and other types of downsampling are also possible.

[0073] Combining the input image 110 with the correction image 140 allows for more efficient use of the neural network because the correction image does not necessarily resemble the fully modified image. This may be particularly advantageous when combined with the downsampling and upsampling described above.

[0074] Input image 110 can be a video frame or a still image. Modifications can include reducing compression artifacts, artifacts caused by storage / channel errors, or any other defects in the image or video, and improving the perceptual quality of the image. This can also include reducing defects or improving the quality of digitized image or video frames, such as images or videos recorded or stored at low quality. Improvements can also include colorization of black and white recordings or improving or modifying the colorization of recordings. In fact, any artifacts or unwanted features of images or videos recorded using older or suboptimal equipment can be reduced. For example, modifications can also include super-resolution, artistic or other visual effects, and depth artifacts.

[0075] In an exemplary implementation, the neural network architecture can be based on a U-shaped machine learning structure. An example of such a structure is U-Net. U-Net is a convolutional neural network (CNN) originally developed for biomedical image segmentation but also used in other related technological fields, such as super-resolution. In the following text, the term U-net is used more broadly to refer to a general U-shaped neural network (network) structure.

[0076] Figure 1 An example of a small U-shaped network (U-Net) is shown in the figure. Figure 2 An example with more processing stages is shown. U-Net demonstrates a shrinking path 299 and an expanding path 298, giving it a U-shaped structure. The shrinking path is a convolutional network consisting of repeated stages of convolutions, each stage subsequently downsampling the image using methods such as rectified linear unit (ReLU) activation functions and max-pooling operations. During shrinking, spatial information is reduced while feature information is increased. The expanding path combines features and spatial information with high-resolution features from the shrinking path through a series of up-convolutions and concatenations. In other words, Figure 2The exemplary U-net includes multiple processing stages with downsampling and (at least one) convolution, and multiple stages with upsampling and convolution. The downsampling module reduces the resolution of the image in its input. Therefore, at each stage of the shrinking path, the image resolution becomes smaller. Thus, at each stage, the convolution module analyzes image features from fine to coarse, features that are critical to a specific resolution. Convolution refers to feature analysis, which uses a predefined mask (pattern) for each sample of the image in the convolutional pattern input. Mathematically, the convolution is computed for each sample of the image as the sum of the products of the predefined mask and the samples of the image at said sample location. It should be noted that the convolution exemplified here is merely a preferred choice. Generally, the invention is not limited to this type of convolution. As mentioned above, in general, any type of filtering, including any type of feature extraction mechanism, can be applied. Feature analysis can include, for example, gradient or edge detection, or other features.

[0077] Figure 3 A more detailed example corresponding to a similar U-net structure used in https: / / lmb.informatik.uni-freiburg.de / people / ronneber / u-net is shown. Figure 3 The shrinkage path in the U-Net shown follows the architecture of a convolutional network. Elements of this exemplary U-Net can also be used in embodiments of this application. It shows a repeated application of two 3×3 convolutions (here, unpadded convolutions) to image tiles, each followed by a rectified linear unit (ReLU) 399 and a 2×2 max pooling operation 398 with a stride of 2 for downsampling. Figure 3 Each bounding box in the diagram corresponds to a multi-channel feature map. Specifically, input image blocks of size 572×572 samples are processed. After the first and second convolutions, 64 feature channels are generated. The difference between the location of the input image and the size of the feature channels (570×570 and 568×568) is due to the lack of padding during convolution; therefore, convolutions cannot be correctly computed for samples at image boundaries. The number of feature channels doubles in each downsampling step (see stages 128, 256, 512, and finally 1024 feature channels below).

[0078] Each step in the expansion path involves upsampling the feature map, followed by a 2×2 convolution (upconvolution) 397 that halves the number of feature channels, a concatenation with the corresponding cropped feature map from the contraction path, and two 3×3 convolutions, each followed by ReLU. The cropping 396 is due to the loss of boundary pixels in each of the convolutions mentioned above. In the final layer, each 64-component feature vector is mapped to the desired number of classes using a 1×1 convolution 395. This exemplary network has a total of 23 convolutional layers. To facilitate seamless segmentation of the output segmentation mapping, the input block size can be chosen so that all 2×2 max-pooling operations are applied to layers with uniform x and y sizes.

[0079] U-Net is a convolutional neural network (CNN) originally developed for biomedical image segmentation. In segmentation, the network input is an image, and the network output is a segmentation mask that assigns each pixel a specific class label. An example of this input and output is... Figure 4 As shown, this is from https: / / lmb.informatik.uni-freiburg.de / people / ronneber / u-net. The input is an electron microscope image 401 of a neuronal structure, which is segmented using U-Net to generate a segmentation mask 402. For image filtering using U-Net, such as post-processing and in-loop filtering, where both network input and output are images, a conventional U-Net with the structure described above may not be suitable.

[0080] The network structure described above can be used with some modifications by the methods and apparatus of this invention, which will be described in more detail later. Neural networks typically have two operating modes: a learning mode and an operational mode. Learning can typically be supervised or unsupervised. During supervised learning, the network is presented with a training dataset consisting of pairs of input images (e.g., distorted images) and desired output images (e.g., enhanced images). Supervised learning can be employed to achieve image modification. It should be noted that this invention is not limited to simultaneously supporting both learning and operational modes. Generally, the network does not necessarily have to undergo a learning process. For example, weights can be obtained from other sources, and the network can be directly configured with appropriate weights. In other words, once a neural network is properly trained, the weights can be stored for later use and / or provided for configuring other networks.

[0081] The desired image can be the original image, such as an undisturbed image. The input image can be a perturbed version of the original image. For example, the undisturbed image can be an uncompressed or losslessly compressed image. The perturbed image can be based on the compressed and subsequently decompressed undisturbed image. For example, BPG or HEVC can be used during compression and decompression during training and testing. The CNN can then learn filters to reduce compression artifacts and enhance the image.

[0082] In this example, during the learning phase, the network's parameters are adjusted to make the enhanced image resemble the uncompressed image more than the decompressed image. This can be accomplished with the help of a loss function that compares the original, uncompressed, and enhanced images.

[0083] However, the input image can also be an undisturbed image, while the original image is one that has been manually or otherwise modified. In this configuration, the neural network can modify the images so that they resemble modifications applied to the training image set in a larger set of undisturbed images that are somewhat similar to the training set.

[0084] Accordingly, when the corrected image is combined with the input image, the resulting image can resemble the original image better than the input image. For example, combining the corrected image with the input image can correspond to adding pixel values ​​from both images, i.e., adding them pixel by pixel. However, this is only an example, and further methods for combining the corrected image with the input image can be used, as described later. It should be noted that in this invention, the terms "pixel" and "sample" are used interchangeably.

[0085] According to some embodiments, the process does not directly output an enhanced / modified image. Instead, it outputs a corrected image. In one example, the corrected image may be a difference image. For example, the corrected image may correspond to the difference between the input image and the original image (which may also be referred to as the desired image or target image).

[0086] According to one embodiment, the corrected image and the input image have the same size, i.e., the same horizontal and vertical dimensions and the same resolution, and the corrected image is a difference image, and the combination is performed by adding the difference image to the input image.

[0087] Figure 5A schematic example of combining a difference image with an input image is shown. In this example, 510 is the input image. The neural network creates a corrected image 520. Adding the input image and the corrected image produces an enhanced version 530 of the input image. Advantages may include the fact that the neural network only needs to create the difference image, rather than the complete modified image. For example, in the case of this method being used to improve frames in a video, the difference image can often be simpler than the final modified image. This can mean that the corrected image includes fewer degrees of freedom (information) than the enhanced image. Therefore, the neural network can operate more efficiently.

[0088] However, combining the corrected image with the input image can also correspond to other operations, such as averaging or filtering the input image using the corrected image, or replacing pixels in the input image with pixels from the corrected image. In the configuration of replacing pixels, for example, the combination can select pixels to be replaced based on a threshold. Those pixels in the corrected image represented by values ​​above the threshold can be selected to replace corresponding pixels in the input image. Furthermore, the combination can be based, for example, on a weighted or local combination of the two images. Additionally, the combination of the input image and the corrected image can alternatively or additionally include non-linear operations such as cropping, multiplication, etc.

[0089] In the example described, the corrected image and the input image have the same x and y directions. However, in some embodiments, the corrected image may have different sizes. For example, the corrected image may provide one or more local patches to be combined with the input image. Furthermore, the corrected image may differ from the input image because it has a different resolution. For example, the corrected image may have a higher resolution than the input image. In such a configuration, for example, the corrected image may be used to sharpen features of the input image and / or increase the resolution of the image or video.

[0090] Figure 2 An example of the neural network used in this embodiment is shown. Figure 2 The exemplary network shown illustrates a contraction path and an expansion path, giving it a U-shaped structure. Furthermore, the network combines the corrected image produced by the contraction and expansion paths with an unmodified copy of the input image. During contraction, spatial information is reduced while feature information is increased. The expansion path combines features and spatial information with high-resolution features from the contraction path through a series of up-convolutions and cascades. The contraction path in this example is a convolutional network, which involves repeatedly applying convolutions, each followed by a rectified linear unit (ReLU) activation function and a max-pooling operation to downsample the image as described above. However, different downsampling operations and activation functions can be used, as described later in various embodiments of the invention.

[0091] Figure 6An overview of the technologies that can be used in different exemplary embodiments, which will be described in more detail below, is shown. In particular, Figure 6 It shows the relationship with Figure 2 U-net has a U-shaped neural network structure that is somewhat similar to U-net. However, several techniques are applied to U-net to improve its performance. These techniques can be applied alternatively or in combination. Combinations can be any combination of two or more techniques, or all of them. Figure 6 The single-digit numbers in the circles indicate where these techniques are applied in the network. For example, as mentioned above, the learned downsampling of 598 can be used in the downsampling stage. Each technique corresponding to one of the circled single-digit numbers will be explained in more detail below.

[0092] As mentioned above, the shrinking and expanding paths of the network according to this application can also be found in U-Net. Accordingly, the method according to this application can be considered to be based on a modified U-Net.

[0093] Specifically, in an exemplary implementation, the neural network is based on a U-net. To build the neural network, the U-net is modified by introducing a skip connection 599 to concatenate the input image with the output image. The skip connection 599 can be implemented by storing a copy of the input image in memory unaffected by the neural network. When the neural network creates a corrected image, the copy of the input image can be retrieved and combined with the corrected image created by the neural network.

[0094] Image modifications as described above can be easily applied to existing or future video codecs (encoders and / or decoders). Specifically, image modifications can be used for in-loop filtering purposes on the encoder and decoder sides. Alternatively or additionally, image modifications can be used in post-filters on the decoder side. In-loop filters are filters used in the encoder and decoder after reconstructing the quantized image, with the aim of storing the reconstructed image in a buffer / memory for use in prediction (temporal or spatial). Image modifications can be used here to enhance the image and reduce compression artifacts. Post-filters are filters applied to the decoded image on the decoder side before rendering the image. Post-filters can also be used to reduce compression artifacts and / or make the image visually pleasing, or to provide effects, color correction, etc.

[0095] In image / video post-processing and in-loop filtering, both the network input signal and the neural network output signal are images. It should be noted that image modification can be used for encoding and / or decoding of still images and / or video frames. However, encoding and decoding are not the only applications of this invention. Instead, standalone image modification deployments are possible, for example, in applications that enhance images or videos by adding effects, as described above.

[0096] In encoding / decoding scenarios, the input and output images are largely similar because the CNN only attempts to invert compression artifacts. Therefore, introducing a global skip connection 599 from the input to the output image into the network is particularly advantageous. Figure 7 This example is also shown in the diagram. Therefore, the CNN only needs to learn the difference between the input image (decompressed image) and the original image (uncompressed image), rather than converting the input image to the original image. This simplifies the training of the network. Figure 7 In the example shown, It is the input image for decompression. It is the original, uncompressed image. This is the filtered output image. Under normal conditions with standard connections, a CNN filters the image by applying learned filters. Convert to image .

[0097] However, due to the image and Very similar, simplifying learning by using a global skip connection 599 forwarding. The network now only learns... and The difference between , .

[0098] Alternatively, the network output can be switched from... Rewritten as And adjust the loss function accordingly. Here, Indicates by function The function estimates the corrected image obtained. It is a function that describes the processing of a neural network.

[0099] According to another embodiment, the neural network is parameterized based on the value of a parameter representing the amount or type of distortion of the input image.

[0100] Image compression is always a trade-off between compression ratio and image quality. Generally, the less an image is compressed, the better its quality. Since different compression levels introduce different compression artifacts, instead of training a single CNN to handle all the different compression levels, a specific CNN can be trained for each compression level. This can further improve the visual quality of the filtered image, as a specific network can better adapt to a particular compression level.

[0101] In some implementations, one or more parameters may indicate which compression level and / or which compression technique (codec) to use to compress the image or video. These parameters may indicate which CNN structure should be used to enhance the decompressed image or video. These parameters can be used to determine the structure of the neural network during training and when applying the neural network to the decompressed image. In some implementations, the parameters may be transmitted or stored along with the image data. However, in some implementations, the corresponding parameters can be determined from the decompressed image. In other words, the properties of the decompressed image can be used to determine the structure of the neural network, which is then used to improve the decompressed image. For example, such a parameter may be the quantization step or another quantization parameter that reflects the quantization step. Additionally or alternatively, other encoder parameters such as prediction settings may be used. For example, intra-frame prediction and inter-frame prediction can produce different artifacts. This disclosure is not limited to any particular parameter. Bit depth and the application of specific transforms or filters during encoding and decoding are other examples of parameters that can be used to parameterize or train a neural network.

[0102] Furthermore, in some implementations, a complete set of parameters describing the neural network can be transmitted or stored along with an image, video, or set of images or videos. The parameter set can include all parameters, including weights learned using the corresponding image set.

[0103] In other implementations, the weights of the neural network can be learned using a training dataset. The training data can be any set of images or videos. The same weights can then be used to improve any input data. Specifically, all videos or images compressed using the same compression technique and the same compression rate can be improved using the same neural network, and there is no need to transmit or store separate weights with each video or image. The advantage is that a larger training dataset can be used to train the neural network, and less data overhead is required when transmitting or storing compressed images or videos.

[0104] According to one embodiment, image downsampling is performed by applying strided convolutions and / or padding convolutions.

[0105] The original U-Net uses max pooling to downsample the input image along the shrinking path. Max pooling is a form of non-linear downsampling. It divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such sub-region. Figure 8 An example of max pooling is shown. Given an image x, the size of the pooling mask s, and row indices r and column indices c that identify the pixels of x, the max pooling operation is defined as follows: .

[0106] exist Figure 8The example shown illustrates a model max-pooling operation with a 2×2 filter. In this example, 4×4 pixels are pooled into 2×2 pixels. The same method can also be applied to arrays that are not 4×4. First, in this example, the 4×4 input is separated into 2×2 arrays. From each 2×2 array, the maximum value is determined. Then, the value of each maximum value is used to fill the corresponding field in a new 2×2 pixel array.

[0107] Therefore, max pooling with pooling size s=2 has a resolution one-quarter of x, meaning the width and height of x are halved. This naturally leads to information loss. To limit this loss, the method according to this embodiment uses strided convolution to downsample the image instead of max pooling. The stride defines the size of the kernel as it traverses the image. While its default value is usually 1, a stride of 2 can be used to downsample the image similar to max pooling. However, other strides can also be used. In standard (non-strided) convolution, the stride is 1, while in strided convolution, the stride is greater than 1. This results in learning a downsampled value of 598 on the input image. The difference between standard (non-strided) convolution and strided convolution is as follows: Figure 9A and Figure 9B As shown.

[0108] In particular, Figure 9A The diagram illustrates a convolution with a stride of 1 (non-stretch). A portion of image 920A with a sample size of 4×4 is downsampled to image 910A with a sample size of 2×2. Each sample in downsampled image 910A is obtained from 9 samples of a 3×3 sub-region of image 920A. The sub-regions contributing to the samples in downsampled image 910A overlap, and their centers are located in adjacent samples, i.e., 1 sample apart, corresponding to a stride of 1.

[0109] Figure 9B This shows a convolution with a stride of 2. (Compared to...) Figure 9A Similarly, a portion of image 920B with a 5×5 sample size is downsampled to image 910B with a 2×2 sample size. Each sample in downsampled image 910B is obtained from nine samples of a 3×3 sub-region of image 920A. The sub-regions contributing to the samples in downsampled image 910A overlap, and their centers are 2 samples apart, corresponding to a stride of 2.

[0110] Let again The input image has a depth of , For network weights, The depth after downsampling is typically twice the size of the input image. Then, the convolutional image and the downsampled image Defined as: .

[0111] In other words, the weights w determine the contribution of the corresponding samples from images 920A and 920B to the sampled images 910A and 910B. Therefore, the weights filter the input images while performing downsampling. These weights can be fixed in some implementations. However, the population can also be trained.

[0112] Staggered convolution is used here as downsampling. Nevertheless, filtering is performed after downsampling, such as... Figure 6 As shown. However, it should be noted that in some implementations, at least a portion of the filtering can already be performed via convolution with appropriately set weights (to perform the required feature extraction / filtering). In this implementation, computational power can be saved through this joint downsampling and filtering.

[0113] In addition to strided convolutions, or optionally, padded convolutions can be used. Due to the unpadded convolutions in the original U-Net, the resolution of the network output is lower than the resolution of the network input with a constant boundary width, as referenced above. Figure 2 This behavior may be disadvantageous for post-processing and in-loop filtering (loop filtering) because the input and output images should have the same resolution.

[0114] Therefore, padded convolution can be used instead of non-padded convolution. In padded convolution, 597 extra pixels of a predefined value are added around the boundaries of the input image, increasing the image resolution, which is then reduced to the original resolution after convolution for other purposes. Typically, the values ​​of the extra pixels can all be set to 0. However, different strategies can be chosen to padded the extra pixels. Some of these strategies may include filling the corresponding pixel with the average value of nearby pixels or, for example, the minimum value of nearby pixels. Nearby pixels can be adjacent pixels or pixels within a predetermined radius. Figure 10 The concept of padded convolution is illustrated. The dashed box 1025 represents the additional pixels padded around the original image 1020 to maintain the resolution of the resulting convolutional image 1010. Specifically, as in... Figure 10 As can be seen in the first image of a series of images, when convolution is performed at the positions of diagonal samples, on both sides ( Figure 10 The top left of the first image, Figure 10There are no other samples in the upper right corner of the last image (image 1020). To still allow convolution computation, an additional sample 1025 is added to the position immediately adjacent to image 1020. This addition can be an extrapolation of pixels from image 1020, or a fixed and predefined value, etc. It should be noted that in this simple example, since the convolution mask size is 3×3, only one additional sample is needed on the boundary of image 1020. However, for larger mask sizes, more than one additional sample is needed around image 1020. Padding convolution and strided convolution can be applied in combination.

[0115] However, it should be noted that the present invention is not limited to using padded convolution 597. Instead, padding-free convolution or other techniques for processing boundaries can be applied.

[0116] In one embodiment, the activation function of the neural network is the rectified linear unit (ReLU) activation function.

[0117] The original U-Net uses the rectified linear unit (ReLU) as the nonlinear activation function, defined as follows: .

[0118] In other words, this means that negative values ​​are reduced to 0. Therefore, there are no more gradients among the previously negative values.

[0119] Using standard ReLU during training can be problematic due to the lack of gradient information. A value consistently below 0 will prevent the network from learning.

[0120] However, due to the greater gradient information, using leaky ReLU as the activation function may result in faster learning and better convergence. Leaky ReLU is defined as follows: .

[0121] In other words, values ​​greater than 0 are unaffected, while negative values ​​are scaled. The scaling factor can be a number less than 1 to reduce the absolute magnitude of negative values. Alternatively, any other activation function can be used, such as a soft-addition activation function.

[0122] To improve the visual quality of video frames, this disclosure provides methods and apparatus that can be used as post-processing filters or as in-loop filters.

[0123] As mentioned above, image filtering can be used in different ways for video processing. Both the encoder and decoder attempt to reconstruct an image as close as possible to the original. In doing so, it is advantageous to filter each frame of the video while reconstructing it. The filtered image can then be used to better predict the next frame (cyclic filtering).

[0124] In some embodiments, post-processing of the image may be advantageous. In this case, filters can be applied to each frame after decoding before the frame or image is displayed, saved, or buffered. Prediction of the next frame may still be based on the last frame, which was decoded but not filtered. According to one embodiment, a method for reconstructing an encoded image from a bitstream is provided, wherein the method includes: decoding the encoded image from the bitstream and applying a method for modifying an input image according to any of the above embodiments, wherein the input image is a decoded image.

[0125] In this approach, any video encoding / decoding technique can be used. The frames are then filtered. In other words, filtering can be independent of the encoding / decoding application. This can potentially help improve the visual quality of any compressed video without changing the encoding / decoding method. As mentioned above, the filter can be adapted to the encoding method and / or compression ratio.

[0126] Alternatively, in video coding, frames can be filtered within a loop (loop filter). This may mean filtering these frames before using them to predict other frames. Accordingly, according to one embodiment, a method for reconstructing a compressed image of a video is provided, comprising: predicting a reconstructed image using an image based on a reference image stored in a memory; applying the method for modifying an input image as described above, wherein the input image is the reconstructed image; and storing the modified image in memory as a reference image.

[0127] Using filtered images in the prediction of consecutive frames can improve and / or enhance the accuracy of predictions for specific frames or blocks. This can reduce the amount of data required to store or transmit video without compromising the accuracy of predictions for consecutive blocks or frames.

[0128] The same loop filter used in video decoding can also be used in encoding.

[0129] For neural networks according to embodiments of this application, in order to modify images and videos so that the modified images resemble the target images, efficient training of the network parameters (i.e., the weights of the neural network) is required. Therefore, a method for training a neural network to modify distorted images is provided, wherein the method includes: inputting a distorted image as a target input and a corrected image pair as a target output into the neural network, wherein the corrected image is obtained based on the original image, wherein the processing performed using the neural network includes at least one stage of image downsampling and filtering the downsampled image; and at least one stage of image upsampling, adjusting at least one parameter of the filtering based on the input pair.

[0130] According to this embodiment, supervised learning techniques can be used to optimize network parameters. The goal of learning can be for the network to create a corrected image as the target output from the target input. To achieve this goal, after applying the neural network to the target input, the generated output (corrected image) is added to a copy of the target output. This can be achieved by skipping connection 599. Subsequently, a loss function can be calculated.

[0131] According to one embodiment, at least one parameter of the filter is adjusted based on a loss function 595 corresponding to the mean squared error (MSE).

[0132] The original U-Net was developed for biomedical image segmentation. In segmentation, the network takes an image as input and outputs a segmentation mask. The segmentation mask assigns each image pixel to a specific class label. Cross-entropy, which measures the distance between two probability distributions, is used as the loss function.

[0133] For in-loop filtering, the cross-entropy loss function may not be optimal. To measure the reconstruction quality of lossy image compression, the peak signal-to-noise ratio (PSNR) might be a better metric, which can be defined by the mean squared error (MSE). Given a width... and height The original uncompressed image and the corresponding filtered output image The MSE loss function 595 is defined as follows: .

[0134] However, the present invention is not limited to using MSE as the loss function 595. Other loss functions can be used, some of which may be similar to the MSE loss function 595. Generally, other known alternative functions for image quality assessment can be used. In some embodiments, a loss function optimized for measuring the perceived visual quality of an image or video frame can be used. For example, a weighted loss function can be used, which can be a weighted average for, for example, reducing certain types of defects or residuals. In other embodiments, the loss function can be a weighted average over certain regions of the image. Generally, it may be advantageous to adjust the loss function to suit the type of image modification the neural network is applied to.

[0135] Furthermore, using several output channels (color channels) in the loss function may be advantageous. According to one embodiment, the adjustment of at least one parameter of the filter is based on a loss function that includes the weighted average of the squared differences of multiple color channels 594.

[0136] The network provided in this embodiment has multiple output channels 594, instead of using a single network output to calculate the loss function. For example, in Figure 6 In this context, we are processing the three outputs 1, 2, and 3: this allows for a weighted average of the importance of different parts of the loss function during network training. Let... The input image after decompression. This is the original, uncompressed image. This is the enhanced image after filtering. , and This is a scalar weighted constant. Furthermore, the image is positioned in the RGB color space, with red being... ,green For and blue for Then, the mean squared error loss function for multiple outputs can be calculated as follows: .

[0137] In having brightness chromaticity and of In the YUV color space, the loss function is calculated as follows: .

[0138] It should be noted that the present invention is not limited to these examples. Generally, it is not necessary to perform a weighted average for all color channels. For example, a weighted average can be performed for only two of the three channels.

[0139] Figure 11An example of a device 1100 for modifying an input image according to any of the methods described above is shown. A processing unit 1110 is used to process the input image using a neural network to generate a corrected image. The processing using the neural network includes at least one stage of image downsampling and filtering the downsampled image; at least one stage of image upsampling; and a modification unit 1120 is used to modify the input image by combining the input image with the corrected image.

[0140] Here, the filtering parameters (weights) can be learned (trained) through machine learning. However, as mentioned above, more parameters can be learned, such as the convolution weights used for downsampling.

[0141] As mentioned above, refer to Figure 6 The described techniques can also be applied independently. For example, when applied to... Figure 2 When working with neural networks, strided convolutions also offer benefits, without being associated with... Figure 6 One or more modifications shown are combined. Correspondingly, a method for modifying an input image is provided, comprising: generating an output image by processing the input image using a neural network, wherein the processing using the neural network includes at least one stage of image downsampling and filtering the downsampled image; and at least one stage of image upsampling, wherein the image downsampling is performed by applying strided convolution.

[0142] The advantage of strided convolution is that it reduces complexity. In one exemplary embodiment, the stride of a strided convolution is 2. This value represents a good trade-off between the complexity and quality of downsampling.

[0143] According to one exemplary implementation, the neural network is based on a U-net. In order to build the neural network, the U-net is modified by introducing skip connections to connect the input image to the output image.

[0144] For example, a neural network is parameterized based on the values ​​of parameters representing the amount or type of distortion in the input image. Alternatively or additionally, the activation function of the neural network is a rectified linear unit (ReLU) activation function.

[0145] To further preserve the image size from image boundaries with unavailable pixels, image downsampling is performed by applying padded convolutions.

[0146] In some embodiments, the output image is a corrected image, and the method further includes modifying the input image by combining the input image with the corrected image.

[0147] For example, the corrected image and the input image have the same vertical and horizontal dimensions, and the corrected image is a difference image, and the combination is performed by adding the difference image to the input image.

[0148] According to one embodiment, a method for reconstructing an encoded image from a bitstream is provided, wherein the method includes: decoding the encoded image from the bitstream and applying the method for modifying an input image as described in this invention, wherein the input image is the decoded image.

[0149] According to one aspect, a method for reconstructing a compressed image of a video is provided, comprising: using an image to predict and reconstruct an image based on a reference image stored in a memory; applying a method for modifying an input image as described above, wherein the input image is the reconstructed image; and storing the modified image in the memory as a reference image.

[0150] According to one aspect, a method for training a neural network to modify a distorted image is provided, wherein the method includes: inputting a distorted image as a target input and a target output image pair into the neural network, wherein the target output image is based on an original image, wherein processing performed using the neural network includes at least one stage of image downsampling and filtering the downsampled image; at least one stage of image upsampling, wherein the image downsampling is performed by applying strided convolution; and adjusting at least one parameter of the filtering based on the input pair.

[0151] Specifically, at least one parameter of the filter is adjusted according to the loss function corresponding to the mean squared error (MSE).

[0152] Alternatively or additionally, the at least one parameter of the filter is adjusted according to a loss function that includes the weighted average of the squared differences of multiple color channels.

[0153] According to one aspect, an apparatus for modifying an input image is provided, wherein the apparatus includes: a processing unit for generating an output image by processing the input image using a neural network, wherein the processing using the neural network includes at least one stage of image downsampling and filtering the downsampled image; and at least one stage of image upsampling, wherein the image downsampling is performed by applying strided convolution.

[0154] According to one aspect, an apparatus is provided for reconstructing an encoded image from a bitstream, wherein the apparatus includes: a decoding unit for decoding the encoded image from the bitstream, the apparatus being configured to modify the decoded image as described above.

[0155] According to one aspect, an apparatus for reconstructing a compressed image of a video is provided, wherein the apparatus includes: a reconstruction unit for reconstructing an image using image prediction based on a reference image stored in a memory; the apparatus for modifying the decoded image as described above; and a storage unit for storing the modified image as a reference image.

[0156] According to one aspect, an apparatus for training a neural network to modify a distorted image is provided, wherein the apparatus includes: a training input unit for inputting a distorted image as a target input and an original image as a target output into the neural network; a processing unit for processing using the neural network, wherein the processing using the neural network includes at least one stage of image downsampling and filtering the downsampled image; at least one stage of image upsampling, wherein the downsampling is performed by applying strided convolution; and an adjustment unit for adjusting at least one parameter of the filtering based on the input pair.

[0157] Figure 12 An exemplary system shown for deploying the above-described processing is an encoder-decoder processing chain (decoding system 10). Decoding system 10 includes a video encoder 20 and a video decoder 30, which are based on... Figure 14 and Figure 15 The above image modifications are described in detail and can be implemented at the location of the in-loop filter or the post-filter.

[0158] Figure 12 The present application provides an exemplary decoding system 10 that can utilize the technologies described herein, such as a video decoding system 10 (or simply decoding system 10). The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) of the video decoding system 10 represent examples of devices that can be used to perform the various technologies according to the various examples described herein.

[0159] like Figure 12 As shown, the decoding system 10 includes a source device 12, for example, the source device 12 is used to provide encoded image data 21 to a destination device 14 for decoding encoded image data 13. The source device 12 includes an encoder 20, and optionally may include an image source 16, a preprocessor (or preprocessing unit) 18 such as an image preprocessor 18, a communication interface or communication unit 22.

[0160] Image source 16 may include or may be any type of image capture device, such as a camera for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any other type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images) and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory / storage for storing any of the aforementioned images.

[0161] Unlike the processing performed by the preprocessor 18 and the preprocessing unit 18, the image or image data 17 can also be referred to as the original image or original image data 17. The preprocessor 18 receives the (original) image data 17 and preprocesses it to obtain a preprocessed image 19 or preprocessed image data 19. The preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color correction, or denoising. It should be understood that the preprocessing unit 18 may be an optional component. It should be noted that embodiments of image modification according to the present invention can also be used for preprocessing to enhance or denoise images (video frames).

[0162] Video encoder 20 is used to receive preprocessed image data 19 and provide encoded image data 21 (hereinafter referred to as...) Figure 14 (Further details to follow).

[0163] The communication interface 22 in the source device 12 can be used to: receive encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0164] Destination device 14 includes decoder 30 (e.g., video decoder 30), and optionally may include communication interface or communication unit 28, post-processor 32 (or post-processing unit 32), and display device 34. Communication interface 28 in destination device 14 is used to receive encoded image data 21 (or any other processed version) directly from source device 12 or from any other source device such as a storage device (e.g., an encoded image data storage device), and to provide the encoded image data 21 to decoder 30.

[0165] Communication interfaces 22 and 28 can be used to send or receive encoded image data 21 or encoded data 13 via a direct communication link (e.g., a direct wired or wireless connection) between source device 12 and destination device 14, or via any type of network (e.g., wired or wireless networks or any combination thereof, or any type of private and public network), or any combination thereof.

[0166] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing for transmission on a communication link or communication network.

[0167] For example, the communication interface 28 corresponding to the communication interface 22 can be used to receive the transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21.

[0168] Both communication interface 22 and communication interface 28 can be configured as Figure 12 The communication channel 13, from source device 12 to destination device 14, is a one-way communication interface, or configured as a two-way communication interface, and can be used to send and receive messages, etc., to establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission (e.g., encoded image data transmission).

[0169] Decoder 30 is used to receive encoded image data 21 and provide decoded image data 31 or decoded image 31. As described above, the decoder can perform image modification within the in-loop filter and / or the post-filter.

[0170] The post-processor 32 of the destination device 14 is used to post-process the decoded image data 31 (also referred to as reconstructed image data) (e.g., decoded image 31) to obtain post-processed image data 33 (e.g., post-processed image 33). The post-processing performed by the post-processing unit 32 may include color format conversion (e.g., from YCbCr to RGB), color correction, trimming or resampling, or any other processing to provide the decoded image data 31 for display by a display device 34, etc. It should be noted that the image modifications described in the above embodiments and exemplary implementations can also be used here as post-processing after the decoder 30.

[0171] The display device 34 in the destination device 14 is used to receive the post-processed image data 33 and display the image to a user or viewer. The display device 34 can be or can include any type of display for representing the reconstructed image, such as an integrated or external display or screen. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0172] although Figure 12 Source device 12 and destination device 14 are described as separate devices; however, device embodiments may also include two devices or two functions, namely source device 12 or its corresponding function and destination device 14 or its corresponding function. In these embodiments, source device 12 or its corresponding function and destination device 14 or its corresponding function may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0173] According to the description, Figure 12 The different units or functions that are (accurately) divided in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0174] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30, can be transmitted via, for example, Figure 13 The processing circuitry 46 shown is implemented, for example, by one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video decoding dedicated processors, or any combination thereof. Encoder 20 can be implemented by processing circuitry 46 to embody various modules and / or any other encoder system or subsystem described herein. Decoder 30 can be implemented by processing circuitry 46 to embody the various modules described in conjunction with decoder 30 and / or any other decoder system or subsystem described herein. Processing circuitry 46 can be used to perform various operations, as discussed later. Figure 20As shown, when the technology is partially implemented in software, the device may store the software instructions in a suitable non-transitory computer-readable storage medium (such as memory 44), and may execute the instructions in hardware using one or more processors (in the processing circuitry 46) to perform the technology of the invention. Other processors may be provided to system 40, which may control, for example, display device 45, imaging device 41, and antenna 42, or other devices. The video encoder 20 or video decoder 30 may be integrated into a single device as part of a combined encoder / decoder (codec), such as... Figure 13 As shown.

[0175] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as a laptop or tablet computer, mobile phone, smartphone, tablet / tabletcomputer, camera, desktop computer, set-top box, television, display device, digital media player, video game console, video streaming device (such as a content service server or content distribution server), broadcast receiver device, broadcast transmitter device, etc., and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0176] In some cases, Figure 12 The video decoding system 10 shown is merely exemplary, and the technology provided in this application can be applied to video decoding setups (e.g., video encoding, video decoding, or post-processing) that do not necessarily include any data communication between the encoding and decoding devices. In other examples, data is retrieved from local memory, sent over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding, decoding, and post-processing are performed by devices that do not communicate with each other but simply encode data into memory and / or retrieve data from memory and decode it.

[0177] Figure 14 This is a schematic block diagram of an exemplary video encoder 20 used to implement the technology of this application. Figure 14In the example, the video encoder 20 includes an input terminal 201 (or input interface 201), a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filtering unit 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal 272 (or output interface 272). The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 14 The video encoder 20 shown can also be called a hybrid video encoder or a hybrid video codec-based video encoder. It should be noted that the present invention is not limited to applications in such a hybrid encoder. The essence of image modification is that, regardless of other stages of video encoding and decoding, it can be used to modify images through any type of encoding or decoding.

[0178] The residual calculation unit 204, transform processing unit 206, quantization unit 208, and mode selection unit 260 can form the forward signal path of the encoder 20, while the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, buffer 216, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 can form the backward signal path of the video encoder 20. The backward signal path of the video encoder 20 corresponds to the decoder (see [link to decoder]). Figure 3 The signal path of the video decoder 30 in the video encoder 20. The inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244 and intra-frame prediction unit 254 also constitute the "built-in decoder" of the video encoder 20.

[0179] Encoder 20 can be used to receive image 17 (or image data 17) via input terminal 201, for example, an image in an image sequence forming a video or video sequence. The received image or image data can also be a pre-processed image 19 (or pre-processed image data 19). For simplicity, the following description uses image 17. Image 17 can also be referred to as the current image or the image to be decoded (especially in video decoding when distinguishing the current image from other images, such as those in the same video sequence, i.e., including previously encoded and / or decoded images in the video sequence of the current image).

[0180] A (digital) image is, or can be, a two-dimensional array or matrix of samples with intensity values. Samples in an array can also be called pixels (a short form of image element). The number of samples in the horizontal and vertical directions (or axes) of the array or image defines the image size and / or resolution. To represent color, three color components are typically used; that is, the image can be represented as or comprise three sample arrays. In RBG format or color space, an image comprises corresponding red, green, and blue sample arrays. However, in video decoding, each pixel is typically represented in a luminance and chrominance format or in a color space, for example, YCbCr, comprising a luminance component represented by Y (sometimes also L) and two chrominance components represented by Cb and Cr. The luminance component Y represents luminance or grayscale intensity (e.g., as in a grayscale image), and the two chrominance components Cb and Cr represent chrominance or color information components. Accordingly, an image in YCbCr format comprises a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). An RGB format image can be converted or transformed to YCbCr format, and vice versa. This process is also known as color transformation or conversion. If the image is monochrome, it may consist only of a luminance sample array. Accordingly, for example, an image may be a luminance sample array in monochrome format or a luminance sample array and two corresponding chrominance sample arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0181] The video encoder 20 may include an image segmentation unit ( Figure 14 (Not shown in the image) is used to segment image 17 into multiple (typically non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (in H.264 / AVC), or coding tree blocks (CTBs) or coding tree units (CTUs) (in H.265 / HEVC and VVC). The image segmentation unit can be used to segment each image into corresponding blocks by using the same block size and a corresponding grid with defined block sizes for all images in a video sequence, or by varying the block size between images, subsets of images, or groups of images.

[0182] In other embodiments, the video encoder may be used to directly receive blocks 203 of image 17, such as one, several, or all of the blocks that make up image 17. Image block 203 may also be referred to as the current image block or the image block to be decoded.

[0183] Similar to image 17, image block 203 is also a two-dimensional array or matrix composed of samples with intensity values ​​(sample values), but the size of image block 203 is smaller than that of image 17. That is, for example, block 203 may include, for example, a single sample array (e.g., a luminance array in the case of black and white image 17, or a luminance or chrominance array in the case of a color image) or three sample arrays (e.g., a luminance array and two chrominance arrays in the case of color image 17) or any other number and / or type of array depending on the applied color format. The number of samples in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Therefore, a block can be an M×N (M columns × N rows) sample array, or an M×N transform coefficient array, etc.

[0184] In one embodiment, Figure 14 The video encoder 20 shown is used to encode the image 17 block by block, for example, to perform encoding and prediction for each block 203.

[0185] Figure 14 The illustrated embodiment of the video encoder 20 can also be used for image segmentation and / or encoding using stripes (also known as video stripes), wherein one or more stripes (typically non-overlapping) can be used to segment or encode an image. Each strip may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., tiles (H.265 / HEVC and VVC) or bricks (VVC) for enabling parallel decoding). Applying image segmentation can help reduce processing complexity. In particular, regardless of whether it is used for block-based encoders and / or decoders, processing for modifying images can also be performed based on blocks or tiles or any other type of image portion. This allows for limiting the network size and making it suitable for different image sizes and / or resolutions.

[0186] In one embodiment, Figure 14 The video encoder 20 shown can also be used to segment and / or encode an image using strips / block groups (also known as video block groups) and / or blocks (also known as video blocks), wherein the image can be segmented into one or more strips / block groups (typically non-overlapping) or encoded using one or more strips / block groups (typically non-overlapping), each strip / block group may include one or more blocks (e.g., CTUs) or one or more blocks, wherein each block may be a shape such as a rectangle and may include one or more complete or partial blocks (e.g., CTUs).

[0187] Residual calculation The residual calculation unit 204 is used to calculate the residual block 205 (also called residual 205) based on the image block 203 and the prediction block 265 (which is described in detail later): for example, the residual block 205 in the sample domain is obtained by subtracting the sample value of the prediction block 265 from the sample value of the image block 203 on a sample-by-sample (pixel-by-pixel) basis.

[0188] Transformation The transform processing unit 206 performs discrete cosine transform (DCT) or discrete sine transform (DST) on the sample values ​​of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 can also be called transform residual coefficients, representing the residual block 205 in the transform domain.

[0189] Transform processing unit 206 can be used to apply an integer approximation of DCT / DST (e.g., a transform specified for H.265 / HEVC). This integer approximation is typically scaled by a certain factor compared to orthogonal DCT transforms. Other scaling factors are applied during the transform process to maintain the norm of the residual blocks in the forward and inverse transform processes. These scaling factors are typically selected based on certain constraints, such as the scaling factor being a power of 2 used for shift operations, the bit depth of the transform coefficients, and a trade-off between accuracy and implementation cost. For example, a specific scaling factor is specified for the inverse transform (and the corresponding inverse transform at video decoder 30) via inverse transform processing unit 212, and correspondingly, a scaling factor can be specified for the forward transform in encoder 20 via transform processing unit 206, etc.

[0190] An embodiment of the video encoder 20 (corresponding to the transform processing unit 206) can be used to directly output or encode or compress transform parameters (e.g., one or more types of transforms) after being output by the entropy encoding unit 270, for example, so that the video decoder 30 can receive and use the transform parameters for decoding.

[0191] Quantification Quantization unit 208 is used to quantize the transform coefficients 207 by, for example, scalar quantization or vector quantization, to obtain quantization coefficients 209. Quantization coefficients 209 can also be referred to as quantization transform coefficients 209 or quantization residual coefficients 209.

[0192] The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scaling can be applied to achieve finer or coarser quantization. Smaller quantization steps correspond to finer quantization, while larger quantization steps correspond to coarser quantization. The applicable quantization step size can be represented by the quantization parameter (QP). For example, the quantization parameter can be a set of predefined indices of applicable quantization step sizes. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step size), and a larger quantization parameter can correspond to coarse quantization (larger quantization step size), and vice versa. Quantization can involve division by the quantization step size and a corresponding quantization or dequantization, for example, performed by the dequantization unit 210, or it can involve multiplication by the quantization step size. Embodiments of some standards, such as HEVC, can be used to determine the quantization step size using the quantization parameter. Typically, the quantization step size can be calculated based on the quantization parameter using a fixed-point approximation method that includes division. Quantization and dequantization can introduce additional scaling factors to recover the norm of the residual block. The scaling used in the fixed-point approximation of the equations for the quantization step size and quantization parameters may modify the norm of the residual block. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and signaled to the decoder by the encoder via the bitstream, etc. Quantization is a lossy operation, and the loss increases with the quantization step size.

[0193] In an embodiment, the video encoder 20 (corresponding to the quantization unit 208) can be used to output quantization parameters (QP), for example, directly outputting them or outputting them after being encoded by the entropy encoding unit 270, for example, so that the video decoder 30 can receive and use the quantization parameters for decoding.

[0194] Inverse Quantization The dequantization unit 210 is used to perform dequantization on the quantization coefficients by the quantization unit 208 to obtain the dequantization coefficients 211. For example, it performs the dequantization scheme of the quantization scheme performed by the quantization unit 208 using the same quantization step size as the quantization unit 208. The dequantization coefficients 211 can also be called dequantization residual coefficients 211, corresponding to the transform coefficients 207. However, due to the loss caused by quantization, the dequantization coefficients 211 are usually not exactly the same as the transform coefficients.

[0195] Inverse Transformation The inverse transform processing unit 212 is used to perform the inverse transform of the transform performed by the transform processing unit 206, such as the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), to obtain the reconstructed residual block 213 (or the corresponding dequantization coefficients 213) in the sample domain. The reconstructed residual block 213 can also be referred to as the transform block 213.

[0196] reconstruction Reconstruction unit 214 (e.g., adder or summer 214) is used to add transform block 213 (i.e. reconstruction residual block 213) to prediction block 265 to obtain reconstruction block 215 in the sample domain, for example, by adding the sample values ​​of reconstruction residual block 213 and prediction block 265.

[0197] Filtering The loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain the filtered block 221, or is generally used to filter the reconstructed samples to obtain filtered sample values. The method according to this application can be used for loop filters. Figure 17 The illustration shows an example of a filter provided in this application that can be used as a loop filter, wherein the reconstruction unit corresponds to 214, the storage unit 1730 corresponds to the decoded image buffer 230, and the device 1700 corresponds to the loop filter 220. For example, the loop filter unit is used to smoothly perform pixel transformations or improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. In one example, the loop filter unit 220 may include a deblocking filter, a SAO filter, and an ALF filter. The filtering process may be performed in the order of deblocking filter, SAO filter, and ALF. As another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking filtering. For example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra-sub-partition (ISP) edges. Although the loop filtering unit 220 in... Figure 14 The filter is shown as an in-loop filter, but in other configurations, the loop filter unit 220 can be implemented as a post-loop filter. Filter block 221 can also be referred to as filtered reconstruction block 221.

[0198] In an embodiment of the video encoder 20, the video encoder 20 (correspondingly, the loop filter unit 220) can be used, for example, to directly output or, after encoding by the entropy coding unit 270, output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), so that the decoder 30 can receive and decode using the same or different loop filter parameters. Any one of the above-mentioned combinations of two or more (or all) filters can be implemented as the image modification device 1700.

[0199] Decoding image buffer The decoded picture buffer (DPB) 230 can be a memory that stores reference images or reference image data for the video encoder 20 to encode video data. The DPB 230 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer (DPB) 230 can be used to store one or more filter blocks 221. The decoded picture buffer 230 can also be used to store other previous filter blocks from different images, such as the same current image or a previously reconstructed image, for example, previous reconstructed and filter blocks 221, and can provide complete previous reconstructed (i.e., decoded image and corresponding reference blocks and samples) and / or partially reconstructed current image (and corresponding reference blocks and samples), for example, for inter-frame prediction. If the reconstructed block 215 is not filtered by the loop filtering unit 220, the decoded picture buffer (DPB) 230 can also be used to store one or more unfiltered reconstructed blocks 215, or typically store unfiltered reconstructed samples, or reconstructed blocks or reconstructed samples that have not undergone any other processing.

[0200] Pattern selection (segmentation and prediction) The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, for receiving or obtaining raw image data, such as raw block 203 (current block 203 of current image 17), from the decoded image buffer 230 or other buffers (e.g., line buffers, not shown in the figure), and reconstructed image data, such as the same (current) image and / or one or more previously decoded images filtered and / or unfiltered reconstructed samples or reconstructed blocks. The reconstructed image data is used as reference image data for predictions such as inter-frame prediction or intra-frame prediction to obtain prediction block 265 or prediction value 265.

[0201] The mode selection unit 260 can be used to determine or select the segmentation type for the current block prediction mode (including no segmentation) and prediction mode (e.g., intra-frame or inter-frame prediction mode), and generate the corresponding prediction block 265 to calculate the residual block 205 and reconstruct the reconstructed block 215.

[0202] Video encoder 20 is used to determine or select the best or optimal prediction mode from (e.g., a predetermined) prediction mode set. For example, the prediction mode set may include intra-frame prediction modes and / or inter-frame prediction modes. In this context, terms such as "best," "minimum," and "optimal" do not necessarily mean "best," "minimum," or "optimal" overall, but may also refer to meeting termination or selection criteria, such as values ​​exceeding or falling below a threshold or other constraints. A "suboptimal selection" may be made, but this reduces complexity and processing time.

[0203] Intra-frame prediction The intra-prediction mode set can include various intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in HEVC. For example, several traditional angular intra-prediction modes are adaptively replaced with wide-angle intra-prediction modes for non-square blocks as defined in VVC. Furthermore, to avoid division operations in DC prediction, only the longer side is used to calculate the average value of non-square blocks. Moreover, the intra-prediction results of planar mode can be modified using the position-dependent intraprediction combination (PDPC) method.

[0204] Intra-prediction unit 254 is used to generate intra-prediction block 265 using reconstructed samples of neighboring blocks of the same current image based on intra-prediction modes in the intra-prediction mode set.

[0205] Intra-prediction unit 254 (or typically mode selection unit 260) is also used to output intra-prediction parameters (or typically information on the selected intra-prediction mode of the indicator block) to entropy coding unit 270 in the form of syntax element 266, so as to be included in the encoded image data 21, for example, so that video decoder 30 can receive and use the prediction parameters for decoding.

[0206] Inter-frame prediction The set of (possible) inter-frame prediction modes depends on the available reference image (i.e., at least a partially decoded image stored in the DBP 230 as described above) and other inter-frame prediction parameters, such as whether the entire reference image or only a portion of the reference image (e.g., the search window region around the current block) is used to search for the best-matching reference block, and / or whether pixel interpolation, such as half-pixel, quarter-pixel, and / or 1 / 16-pixel interpolation, is applied.

[0207] In addition to the prediction modes mentioned above, skip mode, direct mode and / or other inter-frame prediction modes can also be used.

[0208] Inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in... Figure 2 (Not shown in the image). The motion estimation unit can be used to receive or acquire image block 203 (current image block 203 of current image 17) and decoded image 231, or at least one or more previously reconstructed blocks, such as one or more other / different previously decoded image blocks 231, for motion estimation. For example, the video sequence may include the current image and the previous decoded image 231, or in other words, the current image and the previous decoded image 231 may be part of an image sequence that makes up the video sequence or an image sequence that forms the video sequence.

[0209] For example, encoder 20 can be used to select a reference block from multiple reference blocks of the same or different images in multiple other images, and provide the offset (spatial offset) between the position (x-coordinate, y-coordinate) of the reference image (or reference image index) and / or the position of the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. This offset is also called the motion vector (MV).

[0210] The motion compensation unit is used to acquire (e.g., receive) inter-frame prediction parameters and perform inter-frame prediction based on or using these parameters to obtain inter-frame prediction blocks 265. Motion compensation performed by the motion compensation unit may involve extracting or generating prediction blocks based on motion / block vectors determined by motion estimation, and may also include interpolation of sub-pixel precision. Interpolation filtering can generate samples of other pixels from samples of known pixels, potentially increasing the number of candidate prediction blocks available for decoding the image blocks. Upon receiving the motion vector corresponding to the PU of the current image block, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0211] The motion compensation unit can also generate syntax elements associated with the blocks and video stripes for the video decoder 30 to use for decoding image blocks of the video stripes. In addition to stripes and corresponding syntax elements, or as a substitute for stripes and corresponding syntax elements, tile groups and / or blocks and corresponding syntax elements can also be received and / or used.

[0212] Entropy decoding Entropy coding unit 270 is used to apply or not apply entropy coding algorithms or schemes (such as variable length coding (VLC), context adaptive VLC (CAVLC) scheme, arithmetic coding scheme, binarization, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to (uncompressed) quantization coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements to obtain encoded image data 21 that can be output through output terminal 272 in the form of encoded bitstream 21, etc., so that video decoder 30, etc., can receive and use these parameters for decoding. The encoded bitstream 21 can be transmitted to video decoder 30, or stored in memory for later transmission or retrieval by video decoder 30.

[0213] Other architectural variations of the video encoder 20 can be used to encode video streams. For example, a non-transform-based encoder 20 can directly quantize the residual signals of certain blocks or frames without the transform processing unit 206. In another implementation, the encoder 20 may include a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0214] Figure 15 An example of a video decoder 30 for implementing the technology of this application is shown. The video decoder 30 is used to receive, for example, encoded image data 21 (e.g., encoded bitstream 21) encoded by encoder 20, to obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the encoded image data, such as data representing image blocks (and / or chunks or blocks) of encoded video strips (and / or chunks) and associated syntax elements.

[0215] exist Figure 15 In the example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320 and a post-processing filter 321, a decoded picture buffer (DPB) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. The inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform functions typically associated with… Figure 14 The video encoder 100 describes the encoding process as the opposite of the decoding process.

[0216] The method according to this application can be used, for example, in loop filter 320 and post-processing filter 321. Figure 17 An example of a device that can be used as a loop filter 320 is shown, as mentioned in the reference encoder above. Figure 11 An example of a device that can be used as a post-processing filter is shown.

[0217] Figure 16 An implementation of the decoder 30 provided in an embodiment of the present invention is shown. The decoder includes: a decoding unit 1810 for decoding an encoded image from a bitstream; and a modification unit 1820 for modifying the decoded image according to the above embodiment.

[0218] Figure 17 An implementation of a device for reconstructing compressed images or video frames is shown. The device includes: a reconstruction unit 1710 for reconstructing an image using image prediction based on a reference image stored in a memory; a device 1100 for modifying the decoded image according to the above embodiment; and a storage unit 1730 for storing the modified image as a reference image. For example, such a device can be used.

[0219] Figure 18 An apparatus 2000 for training a neural network to modify a distorted image is shown, comprising: a training input unit 2010 for inputting a pair of real images as target inputs and corrected images as target outputs into the neural network, wherein the corrected images are obtained from the original images; a processing unit 2020 for processing using the neural network, wherein the processing using the neural network includes at least one stage of image downsampling and filtering the downsampled image; at least one stage of image upsampling; and an adjustment unit 2030 for adjusting at least one parameter 2040 of the filtering based on the input pair.

[0220] Figure 19 This is a schematic diagram of a video decoding device (or generally an image modification device) 400 provided for an embodiment of the present invention. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder (such as...) Figure 14 Video decoder 30) or encoder (such as Figure 15 (Video encoder 20). In the case of the independent implementation of image modification described in the above embodiments, device 400 may be an image modification device 1100 instead of a video encoding device.

[0221] The video decoding device 400 includes an input port 410 (or input port 410) for receiving data and a receiver unit (Rx) 420, a processor, logic unit, or central processing unit (CPU) 430 for processing data (including the pre-processing described in this application), a transmitter unit (Tx) 440 for transmitting data and an output port 450 (or output port 450), and a memory 460 for storing data. The video decoding device 400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 410, receiver unit 420, transmitter unit 440, and output port 450, serving as input or output points for optical or electrical signals.

[0222] Processor 430 is implemented through hardware and software. Processor 430 can be implemented as one or more CPU chips, one or more cores (e.g., a multi-core processor), one or more FPGAs, one or more ASICs, and one or more DSPs. Processor 430 communicates with ingress port 410, receiver unit 420, transmitter unit 440, egress port 450, and memory 460. Processor 430 includes decoding module 470. Decoding module 470 implements the embodiments disclosed above. For example, decoding module 470 performs, processes, prepares, or provides various decoding operations. Therefore, decoding module 470 provides a substantial improvement to the functionality of video decoding device 400 and affects the switching of video decoding device 400 to different states. Alternatively, decoding module 470 can be implemented with instructions stored in memory 460 and executed by processor 430.

[0223] Memory 460 may include one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store programs when they are selected for execution, and to store instructions and data read during program execution. For example, memory 460 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM). The aforementioned memory module may be part of a memory, or in some implementations may be provided as a separate memory.

[0224] Figure 20 A simplified block diagram of a device 800 provided for an exemplary embodiment, the device 800 being used as Figure 5 The device 800 may use either or both of the source device 512 and the destination device 514. Preprocessing 518 may also be performed independently by the device 800.

[0225] The processor 802 in device 800 may be a central processing unit. Alternatively, processor 802 may be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. While the disclosed implementation may be implemented using a single processor, such as processor 802 as shown in the figure, using more than one processor can improve speed and efficiency.

[0226] In one implementation, the memory 804 in device 800 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 804. Memory 804 may include code and data 806 accessed by processor 802 via bus 812. Memory 804 may also include an operating system 808 and an application program 810, which includes at least one program that causes processor 802 to perform the methods described herein. For example, application program 810 may include applications 1 to M, and may also include a video post-processing application, a video decoding application, or a video encoding application that performs the methods described herein.

[0227] The device 800 may also include one or more output devices, such as a display 818. In one example, the display 818 may be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 818 may be coupled to the processor 802 via a bus 812.

[0228] Although bus 812 in device 800 is described herein as a single bus, bus 812 may include multiple buses. Furthermore, auxiliary memory 814 may be directly coupled to other components of device 800 or accessible via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, device 800 can have a wide variety of configurations.

[0229] In summary, this invention relates to image processing, and more particularly to modifying images using processing methods such as neural networks. The processing is performed to generate a corrected image based on an input image. Then, the input image is modified by combining the input image with the corrected image. The processing using the neural network includes at least one stage of image downsampling and filtering the downsampled image; and at least one stage of image upsampling. One advantage of this method is that it improves the efficiency of the neural network, thereby accelerating the learning speed and improving performance. Embodiments of the invention provide methods and apparatus for processing using a trained neural network, and methods and apparatus for training such a neural network for image modification.

Claims

1. A method (100) for modifying an input image (110), characterized in that, include: A corrected image is generated by processing the input image (110) using a neural network; wherein the processing using the neural network includes at least one stage (120) of image downsampling and filtering the downsampled image obtained by the image downsampling process, and at least one stage (130) of image upsampling and filtering the upsampled image obtained by the image upsampling process; wherein image downsampling and filtering are a shrinking path used to reduce spatial information and increase feature information; image upsampling and filtering are an expanding path used to combine the feature information and the spatial information; The input image is modified (140) by combining the input image with the correction image.

2. The method according to claim 1, characterized in that, The corrected image (520) and the input image (510) have the same vertical and horizontal dimensions; The corrected image (520) is a difference image, and the combination is performed by adding the difference image to the input image (510).

3. The method according to claim 1, characterized in that, The corrected image (520) and the input image (510) have different sizes, and the corrected image (520) is used to provide one or more local patches to be combined with the input image (510).

4. The method according to claim 1, characterized in that, The corrected image (520) and the input image (510) have different resolutions, with the corrected image (520) having a higher resolution than the input image (510). The corrected image (520) is used to sharpen the features of the input image (510) and / or increase the resolution of the input image (510).

5. The method according to any one of claims 1 to 4, characterized in that, The neural network is a U-shaped neural network structure. In order to establish the neural network, the U-net is modified by introducing a skip connection (599) to the U-net, which is used to connect the input image to the output image.

6. The method according to any one of claims 1 to 5, characterized in that, The neural network is parameterized according to the values ​​of parameters representing the amount or type of distortion of the input image.

7. The method according to any one of claims 1 to 6, characterized in that, The image downsampling (120) is performed by applying strided convolution and / or padding convolution.

8. The method according to any one of claims 1 to 7, characterized in that, The activation function of the neural network is a leaky modified linear unit activation function.

9. A method for reconstructing an encoded image from a bitstream, characterized in that, The method includes: Decode the encoded image (331) from the bitstream (21); The encoded image (331) is modified by applying the method (100) for modifying an input image according to any one of claims 1 to 8; wherein the input image is the decoded image (331).

10. A method for compressing images for video, characterized in that, include: Based on the reference images (231, 331) stored in the memory (230, 330), the image prediction unit (244, 254, 344, 354) and the reconstruction unit (214, 314) are used to obtain the reconstructed image (215, 315). The reconstructed image (215, 315) is modified by applying the method (100) for modifying an input image according to any one of claims 1 to 8; wherein the input image is the reconstructed image (215, 315). The modified image is stored as a reference image in the memory (230, 330).

11. A method for training a neural network to modify a distorted image, characterized in that, The method includes: The distorted image (2005) as the target input and the corrected image (2002) as the target output are input to the neural network; wherein the corrected image (2002) is obtained from the original image (2010); The processing is performed using the neural network; wherein the processing (2020) using the neural network includes at least one stage of image downsampling and filtering the downsampled image obtained by the image downsampling process, and at least one stage of image upsampling and filtering the upsampled image obtained by the image upsampling process; wherein image downsampling and filtering are a shrinking path used to reduce spatial information and increase feature information; image upsampling and filtering are an expanding path used to combine the feature information and the spatial information; Adjust at least one parameter (2040) of the filter according to the input pair (2002, 2005).

12. The method according to claim 11, characterized in that, At least one parameter of the filter described in (2030) is adjusted according to the loss function corresponding to the mean square error.

13. The method according to claim 11 or 12, characterized in that, At least one parameter of the filter (2030) is adjusted according to a loss function that includes the weighted average of the squared differences of multiple color channels.

14. A computer program (810), characterized in that, When executed on one or more processors (802), the one or more processors (802) are caused to perform the steps of the method according to any one of claims 1 to 13.

15. A device (1100) for modifying an input image, characterized in that, include: The processing unit (1110) is configured to generate a corrected image by processing the input image using a neural network; wherein the processing using the neural network includes at least one stage of image downsampling and filtering the downsampled image obtained by the image downsampling process, and at least one stage of image upsampling and filtering the upsampled image obtained by the image upsampling process; wherein image downsampling and filtering are a shrinking path used to reduce spatial information and increase feature information; image upsampling and filtering are an expanding path used to combine the feature information and the spatial information; The modification unit (1120) is used to modify the input image by combining the input image with the correction image.

16. An apparatus for reconstructing an encoded image from a bitstream, characterized in that, include: Decoding unit (1810) is used to decode the encoded image from the bitstream; The device (1820) is used to modify the decoded image according to claim 15.

17. An apparatus for reconstructing a compressed image of a video, characterized in that, include: The reconstruction unit (1710) is used to obtain a reconstructed image based on a reference image stored in memory using image prediction and image reconstruction methods; The device (1100) is used to modify the reconstructed image according to claim 15; Storage unit (1730) is used to store the modified image as a reference image.

18. An apparatus for training a neural network to modify distorted images, characterized in that, include: A training input unit (2010) is used to input a distorted image (2005) as the target input and a corrected image (2002) as the target output into the neural network; wherein the corrected image (2002) is obtained from the original image; A processing unit (2020) is configured to perform processing using the neural network; wherein the processing using the neural network includes at least one stage of image downsampling and filtering the downsampled image obtained by the image downsampling process, and at least one stage of image upsampling and filtering the upsampled image obtained by the image upsampling process; wherein image downsampling and filtering are a shrinking path used to reduce spatial information and increase feature information; image upsampling and filtering are an expanding path used to combine the feature information and the spatial information; The adjustment unit (2020) is used to adjust at least one parameter (2040) of the filter according to the input pair (2002, 2005).

Images