Parallel processing image and video compression method based on neural network

Abstract

An image decoding method includes: acquiring a reconstruction potential value # imgabs0 # using an arithmetic decoder, and inputting the reconstruction potential value # imgabs0 # into a synthetic neural network; segmenting the output feature picture into a plurality of parts based on the decoding parameters of the one or more positions; respectively inputting each part in the plurality of parts into the next level of the plurality of convolutional layers so as to obtain a feature map of spatial domain segmentation at an output end; and cutting the feature map of the spatial domain segmentation and splicing the feature map into an overall feature map in the spatial domain until the image is reconstructed.

Description

[0001] Cross-references to related patent applications

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 389,788, filed on July 15, 2022, the teachings and disclosures of which are hereby incorporated by reference in their entirety. Technical Field

[0003] This patent application relates to the generation, storage and use of digital audio-visual media information in a file format. Background Art

[0004] Digital video accounts for the largest share of bandwidth used on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, bandwidth demands for digital video usage are likely to continue to grow. Summary of the invention

[0005] The disclosed aspects / embodiments provide techniques related to neural network based image and video compression methods. The present invention addresses the memory shortage problem when the image or video sequence is too large to fit into the memory during decoding, thus causing decoding failure. The present invention provides a slice partitioning scheme that provides the feasibility of successfully decoding from the bitstream regardless of the spatial size, which is particularly beneficial for limited memory budget or high resolution images / videos.

[0006] The first aspect relates to an image processing method, comprising the following steps: obtaining a reconstruction potential value The reconstructed potential value is input into a synthetic neural network; based on parameters of one or more positions, the output feature image is segmented into multiple parts; each of the multiple parts is input into the next level of multiple convolutional layers to obtain a spatially segmented feature map at the output end; and the spatially segmented feature map is cropped and spliced ​​back to the overall feature map in the spatial domain until the image is reconstructed.

[0007] The second aspect relates to an image processing method, comprising the following steps: obtaining a quantized potential value; obtaining a parameter of slice segmentation; and encoding the potential value and the parameter of slice segmentation into a bit stream so that a decoder receiving the bit stream can crop the feature map of the spatial domain segmentation and splice it back to the overall feature map in the spatial domain until the image is reconstructed.

[0008] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the slice division is horizontal or vertical.

[0009] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that: the slice division is only vertical.

[0010] Optionally, in any of the preceding aspects, another embodiment of this aspect provides: the slice segmentation is vertical, and wherein the slice segmentation is performed more than once.

[0011] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that: the slice division is only horizontal.

[0012] Optionally, in any of the preceding aspects, another embodiment of this aspect provides: the slice segmentation is horizontal, and wherein the slice segmentation is performed more than once.

[0013] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that: the slice division is horizontal and vertical.

[0014] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the slice segmentation is performed recursively.

[0015] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the slice segmentation is fixed for all frames.

[0016] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that: the slice segmentation is different for two or more frames within a sequence.

[0017] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: a feature map value in a first part of the multiple parts adjacent to the segmentation boundary is used to fill a second part of the multiple parts.

[0018] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the slice segmentation is inserted into a position of the synthetic neural network.

[0019] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the slice segmentation is inserted into multiple positions of the synthetic neural network.

[0020] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides that: the filling size of each sub-portion is controllable, and wherein the filling sizes of different sub-portions are the same or different.

[0021] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: encoding one or more of the padding size, the number of vertical partitions, and the number of horizontal partitions into the bitstream.

[0022] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the bitstream includes the number of slices used to decode the bitstream.

[0023] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the bitstream includes a first indication and a second indication, wherein a part of the synthetic neural network is segmented using the first indication, and wherein another part of the synthetic neural network is segmented using the second indication.

[0024] Optionally, in any of the preceding aspects, another embodiment of this aspect provides: the bitstream includes a layer identifier, and wherein the layer identifier identifies a layer of the synthetic neural network.

[0025] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the layer identifier indicates a starting layer, and the slice segmentation is performed after the starting layer.

[0026] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the layer identifier indicates a starting layer, and the slice segmentation stops after the starting layer.

[0027] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the layer identifier indicates a starting layer, a specified number of slices are applied after the starting layer, and wherein the specified number is indicated in the bitstream.

[0028] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the layer identifier indicates a starting layer, and a specified number of slices are no longer applied after the starting layer.

[0029] Optionally, in any of the preceding aspects, another embodiment of this aspect provides: the layer identifier comprises an index to a table, and wherein the table comprises information about the association between the index and the layer of the synthetic neural network.

[0030] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the bit stream includes an indicator indicating the size of the slice.

[0031] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the bit stream includes an indicator indicating the position of the slice.

[0032] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the bitstream includes an indicator indicating whether the slice segmentation is applied, and when the slice segmentation is not applied, neural network-based image reconstruction is performed.

[0033] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the bitstream includes an indicator indicating a minimum slice size, and wherein the minimum slice size is used to determine the number of slices.

[0034] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the size of the reconstructed image is also used to determine the number of slices.

[0035] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the size of the output feature map is also used to determine the number of slices.

[0036] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the bitstream includes an indicator indicating the size of the reconstructed image, and wherein the size of the reconstructed image is used to determine the number of slices.

[0037] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that: the number of slices to be used is based on the size of the input feature map.

[0038] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the layer of the synthetic neural network applies slice segmentation based on the size of the input feature map, and wherein before applying the layer, it is determined whether to apply the slice segmentation.

[0039] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the layer of the synthetic neural network applies slice segmentation based on the size of the input feature map, and before applying the layer, determines the number of slices to be used.

[0040] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that: two or more of the plurality of slices are independently decoded.

[0041] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that decoding of the second slice depends on decoding of the first slice.

[0042] Optionally, in any of the preceding aspects, another embodiment of this aspect provides: the bitstream includes an indicator indicating whether two or more of the plurality of slices are independently decoded.

[0043] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: filtering the boundary between two slices.

[0044] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the intermediate tensor size reaches a maximum value before the last deconvolution or pixel rearrangement layer in the synthesis transformation.

[0045] Optionally, in any of the foregoing aspects, another embodiment of this aspect provides: The sharding is performed with overlap to control the maximum memory size of the intermediate latent tensor, where Represents a latent space tensor of size [C,h4,w4] And where C represents the channels, h represents the height of the tensor, and w represents the width of the tensor.

[0046] Optionally, in any of the preceding aspects, another embodiment of this aspect provides: independently signaling slice positions and slice sizes for primary components and secondary components in the bitstream.

[0047] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: each slice is independently subjected to synthesis transformation.

[0048] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: after reconstruction, the slices are spliced ​​to each other, and wherein overlapping areas of the slices are discarded.

[0049] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the bitstream includes one or both of a slice-enabled luma flag and a slice-enabled chroma flag for slices of the primary component and the secondary component.

[0050] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the bitstream includes one or both of a slice-size luminance syntax element and a slice-size chrominance syntax element for the primary component and the secondary component.

[0051] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the bitstream includes one or both of slice-overlapped luminance syntax elements and slice-overlapped chrominance syntax elements for the primary component and the secondary component.

[0052] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: using an arithmetic decoder to obtain the reconstructed potential value.

[0053] A third aspect relates to a video data processing device, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method as disclosed in any one of the embodiments.

[0054] The fourth aspect relates to a non-transitory computer-readable medium, comprising: a computer program product used by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, so that when the computer executable instructions are executed by a processor, the video codec device performs a method as described in any of the disclosed embodiments.

[0055] A fifth aspect relates to a non-transitory computer-readable recording medium storing a bit stream of a video generated by a method executed by a video processing device, wherein the method includes the method as described in any of the disclosed embodiments.

[0056] A sixth aspect relates to a method for storing a bitstream of a video, including the method described in any of the disclosed embodiments.

[0057] A seventh aspect relates to a method, apparatus or system described in this document.

[0058] For clarity, any of the aforementioned embodiments may be combined with any one or more of the other aforementioned embodiments to create new embodiments within the scope of the present disclosure.

[0059]

[0011] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] For a more complete understanding of the present disclosure, reference is now made to the following brief description, taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0061] Figure 1 An example of a typical transform coding scheme is shown.

[0062] Figure 2 An example of quantization potential values ​​when using a super encoder / decoder is shown.

[0063] Figure 3 An example of a network architecture for an autoencoder implementing a hyper-prior model is shown.

[0064] Figure 4 A combined model is shown that jointly optimizes an autoregressive component that, together with a hyper-prior and an underlying autoencoder, estimates the probability distribution of latent values ​​from the causal context (i.e., the context model).

[0065] Figure 5 The encoding process using a super encoder and a super decoder is shown.

[0066] Figure 6 An example decoding process is shown.

[0067] Figure 7 Example implementations of encoding and decoding processes are shown.

[0068] Figure 8 A two-dimensional forward wavelet transform is shown.

[0069] Fig. 9 A possible partitioning of the latent representation after the 2D forward transform is shown.

[0070] Fig. 10A It shows that the spatial size increases significantly after passing through the synthesis network input.

[0071] Fig. 10B It shows that the number of pixels along one edge of the feature map can be 16 times that before the synthesis network.

[0072] Fig.11 The detailed spatial size variation is shown as the feature map moves through the synthesis network.

[0073] Fig.12 Shown is a slice segmentation of the feature map of the synthesis network.

[0074] Fig.13 An example of inserting patch segmentation into a synthesis network is shown.

[0075] Fig.14 is a block diagram illustrating an example video processing system.

[0076] Fig.15 is a block diagram of an example video processing device.

[0077] Fig.16 is a flow chart of an example method of video processing.

[0078] Fig.17 is a block diagram illustrating an example video codec system.

[0079] Fig.18 is a block diagram illustrating an example encoder.

[0080] Fig.19 is a block diagram illustrating an example decoder.

[0081] Fig. 20 is a schematic diagram of an example encoder.

[0082] Fig.21 The present invention relates to an image decoding method according to an embodiment of the present disclosure.

[0083] Fig. 22 It is an image encoding method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0084] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or yet to be developed. The present disclosure should not be limited in any way to the exemplary implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but may be modified within the full scope of the appended claims and their equivalents.

[0085] The section titles used in this document are for ease of understanding and do not limit the applicability of the techniques and embodiments disclosed in each section to only that section. In addition, the techniques described herein are applicable to other video codec protocols and designs.

[0086] 1. Overview

[0087] A neural network-based image and video compression method includes an autoregressive subnetwork and an entropy coding and decoding engine, wherein the entropy coding and decoding is performed independently of the autoregressive subnetwork. 1. Background Technology

[0089] The past decade has witnessed the rapid development of deep learning in various fields, especially in computer vision and image processing. Inspired by the great success of deep learning techniques in computer vision, many researchers have shifted their attention from conventional image / video compression techniques to neural image / video compression techniques. Neural networks were originally invented at the intersection of neuroscience and mathematics. They have shown strong capabilities in the context of nonlinear transformation and classification. Neural network-based image / video compression techniques have made significant progress in the past five years. It is reported that the latest neural network-based image compression algorithms [1] achieve similar rate-distortion (RD) performance as the Versatile Video Codec (VVC) [2], which is the latest video codec standard developed by the Joint Video Experts Team (JVET) with experts from the Moving Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG). With the continuous improvement of neural image compression performance, neural network-based video compression has become an actively developing research field. However, due to the inherent difficulty of the problem, neural network-based video codecs are still in their infancy.

[0090] 2.1 Image / Video Compression

[0091] Image / video compression generally refers to computing techniques that compress images / videos into binary codes for easy storage and transmission. The binary codes may or may not support lossless reconstruction of the original image / video, which is called lossless compression and lossy compression. Since lossless reconstruction is not necessary in most scenarios, most work is devoted to lossy compression. Usually, the performance of image / video compression algorithms is evaluated from two aspects, namely compression rate and reconstruction quality. The compression rate is directly related to the number of binary codes. The smaller the number of binary codes, the better the compression effect; the reconstruction quality is measured by comparing the reconstructed image / video with the original image / video. The higher the similarity, the better the reconstruction quality.

[0092] Image / video compression technology can be divided into two branches, classical video codec methods and neural network-based video compression methods. Classical video codec schemes adopt transform-based solutions, in which researchers exploit statistical dependencies in latent variables (e.g., discrete cosine transform (DCT) or wavelet coefficients) by manually designing entropy codes to simulate dependencies in quantization mechanisms. There are two forms of neural network-based video compression, namely neural network-based codec tools and end-to-end neural network-based video compression. The former is embedded into existing classical video codecs as a codec tool and is only part of the framework, while the latter is an independent framework developed based on neural networks and does not rely on classical video codecs.

[0093] Over the past three decades, a series of classic video codec standards have been developed to adapt to the increasing visual content. The International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) has two expert groups, the Joint Photographic Experts Group (JPEG) and the Moving Picture Experts Group (MPEG), and the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) also has its own Video Codec Experts Group (VCEG), which is responsible for the standardization of image / video codec technologies. Influential video codec standards released by these organizations include Joint Photographic Experts Group (JPEG), JPEG 2000, H.262, H.264 / Advanced Video Codec (AVC), and H.265 / High Efficiency Video Codec (HEVC). After H.265 / HEVC, the Joint Video Experts Group (JVET) composed of MPEG and VCEG has been working on a new video codec standard - Versatile Video Codec (VVC). The first version of VVC was released in July 2020. It is reported that VVC reduces the bit rate by 50% on average at the same visual quality compared to HEVC.

[0094] Neural network-based image / video compression is not a new technology, as many researchers have devoted themselves to neural network-based image encoding and decoding [3]. However, the network architecture is relatively shallow and the performance is not ideal. Benefiting from the support of abundant data and powerful computing resources, neural network-based methods have been better developed in various applications. Currently, neural network-based image / video compression has shown promising improvements, confirming its feasibility. However, the technology is still far from mature and many challenges need to be addressed.

[0095] 2.2 Neural Networks

[0096] Neural networks, also known as artificial neural networks (ANNs), are computational models used in machine learning techniques that typically consist of multiple processing layers, with each layer consisting of multiple simple but nonlinear basic computational units. One advantage of such deep networks is believed to be the ability to process data with multiple levels of abstraction and transform the data into different types of representations. Note that these representations are not manually designed; rather, the deep networks comprising the processing layers are learned from massive amounts of data using general machine learning procedures. Deep learning eliminates the need for handcrafted representations and is therefore believed to be particularly useful for processing natural unstructured data such as acoustic and visual signals, which has been a long-standing problem in the field of artificial intelligence.

[0097] 2.3 Neural Networks for Image Compression

[0098] Existing neural networks for image compression methods can be divided into two categories, namely pixel probability models and autoencoders. The former belongs to predictive encoding and decoding strategies, while the latter is a transformation-based solution. Sometimes, these two methods are combined in the literature.

[0099] 2.3.1 Pixel Probability Model

[0100] According to Shannon's information theory [6], the best approach for lossless codecs can achieve a minimum codec rate of -log2p(x), where p(x) is the probability of symbol x. Many lossless codecs have been proposed in the literature, among which arithmetic codecs are considered to be one of the best approaches [7]. Given a probability distribution p(x), arithmetic codecs ensure that the codec rate is as close to its theoretical limit -log2p(x) as possible without considering round-off errors. Therefore, the remaining question is how to determine the probability, which is very challenging for natural images / videos due to the curse of dimensionality.

[0101] Following the predictive coding strategy, one way to model p(x) is to predict the pixel probability one by one in raster scan order based on previous observations, where x is the image.

[0102] p(x)=o(x1)p(x2|x1)…p(x i |x1,…,x i-1 )…p(x m×n |x1,…,x m×n-1 ) (1)

[0103] Where m and n are the height and width of the image, respectively. The previous observation is also called the context of the current pixel. When the image is large, it may be difficult to estimate the conditional probability. Therefore, a simplified approach is to limit the scope of its context.

[0104] p(x)=p(x1)p(x2|x1)…p(x i |x i-k ,…,x i-1 )…p(x m×n |x m×n-k ,…,x m×n-1 ) (2)

[0105] Where k is a predefined constant that controls the scope of the context.

[0106] It should be noted that the condition may also consider the sample values ​​of other color components. For example, when encoding and decoding RGB color components, the R sample depends on the previously encoded and decoded pixels (including R / G / B samples), and the current G sample can be encoded and decoded based on the previously encoded and decoded pixels and the current R sample, and when encoding and decoding the current B sample, the previously encoded and decoded pixels and the current R and G samples can also be considered.

[0107] Most compression methods directly model the probability distribution in the pixel domain. Some researchers have also tried to model the probability distribution as a conditional probability distribution based on explicit or latent representations. In other words, we can estimate

[0108]

[0109] Where h is an additional condition, and p(x)=p(h)p(x|h), which means that modeling is divided into unconditional modeling and conditional modeling. The additional condition can be image label information or high-level representation.

[0110] 2.3.2 Autoencoder.

[0111] The autoencoder is derived from the well-known work proposed by Hinton and Salakhutdinov

[17] . The method is trained for dimensionality reduction and consists of two parts: encoding and decoding. The encoding part converts the high-dimensional input signal into a low-dimensional representation, which usually has a reduced spatial size but a larger number of channels. The decoding part tries to recover the high-dimensional input from the low-dimensional representation. Autoencoders enable automatic learning of representations without the need for handcrafted features, which is considered one of the most important advantages of neural networks.

[0112] Figure 1 A typical transform coding scheme is shown. a The original image x is transformed to obtain the latent representation y. The latent representation y is quantized and compressed into bits. The number of bits R is used to measure the encoding and decoding rate. Then it is passed through the synthesis network g s Quantizing the Latent Representation Perform inverse transform to obtain the reconstructed image By using the function gp Transform x and Distortion is computed in perceptual space.

[0113] It is intuitive to apply autoencoder networks to lossy image compression. We only need to encode the latent representation learned from the trained neural network. However, adapting autoencoders to image compression is not trivial because the original autoencoders are not optimized for compression, so directly using the trained autoencoders is not efficient. In addition, there are other significant challenges. First, the low-dimensional representation should be quantized before being encoded, but quantization is not differentiable, which is required in backpropagation when training neural networks. Second, the goals in the compression scenario are different because both distortion and rate need to be considered. Estimating the rate is challenging. Third, practical image coding and decoding schemes need to support variable rates, scalability, encoding / decoding speed, and interoperability. To address these challenges, many researchers are actively contributing to this field.

[0114] Prototype autoencoder for image compression such as Figure 1 As shown, it can be regarded as a transform coding strategy. Using the analysis network y = g a (x) transforms the original image x, where y is the latent representation that will be quantized and decoded. The synthesis network will transform the quantized latent representation Perform inverse transform to obtain the reconstructed image The framework is trained using the rate-distortion loss function, which is where D is x and The distortion between them, R is expressed from the quantization is the rate calculated or estimated in , and λ is the Lagrange multiplier. It should be noted that D can be calculated in the pixel domain or the perception domain. All existing research works follow this prototype, and the difference may only be in the network structure or loss function.

[0115] In terms of network structure, recurrent neural networks (RNNs) and convolutional neural networks (CNNs) are the most widely used architectures. In the RNN-related category, Toderici et al.

[18] proposed a general framework for variable rate image compression using RNNs. They used binary quantization to generate codes and did not consider the rate during training. The framework does provide scalable encoding and decoding capabilities, where RNNs with convolutional and deconvolutional layers are reported to perform well. Toderici et al.

[19] subsequently proposed an improved version by upgrading the encoder with a pixel RNN-like neural network to compress binary codes. It is reported that its performance is better than JPEG on the Kodak image dataset when using the multi-scale structural similarity (MS-SSIM) evaluation metric. Johnston et al.

[20] further improved the RNN-based solution by introducing a hidden state activation mechanism. In addition, an SSIM weighted loss function was designed and a spatially adaptive bitrate mechanism was enabled. They achieved better results than Better Portable Graphics (BPG) on the Kodak image dataset when using MS-SSIM as the evaluation metric. Covell et al.

[21] supported spatially adaptive bitrate by training a blocking-tolerant RNN.

[0116] Ballé et al.

[22] proposed a general framework for rate-distortion optimized image compression. Integer codes are generated using multi-level quantization and the rate during training is considered, i.e., the loss is a joint rate-distortion cost, which can be the mean square error (MSE) or other. They add random uniform noise to motivate quantization during training and use the differential entropy of the noise code as a proxy for the rate. They use generalized divisive normalization (GDN) as the network structure, which consists of a linear mapping followed by a nonlinear parameter normalization. The effectiveness of GDN in image coding and decoding was verified in

[23] . Ballé et al.

[24] then proposed an improved version in which they used 3 convolutional layers as the forward transform, each followed by a downsampling layer and a GDN layer. Therefore, they used 3 layers of inverse GDN to simulate the inverse transform, each followed by an upsampling layer and a convolutional layer. In addition, an arithmetic coding method was designed to compress integer codes. It is reported that the performance on the Kodak dataset is better than JPEG and JPEG 2000 in terms of MSE. In addition, Ballé et al.

[25] improved the method by designing a scale hyper-prior in the autoencoder. They used a subnet h a The potential representation y is transformed into z = h a (y), and z will be quantized and sent as side information. Therefore, using subnet h s Implementing the inverse transformation, this subnetwork attempts to quantize the side information Decoding to quantization The standard deviation of is further used during arithmetic coding and decoding of . On the Kodak image set, their method is slightly inferior to BPG in terms of PSNR. D.Minnen et al.

[26] further exploit the structure in the residual space by introducing an autoregressive model to estimate the standard deviation and mean. In the latest work

[27] , Z.Cheng et al. use a Gaussian mixture model to further remove redundancy in the residual. Using peak signal-to-noise ratio (PSNR) as the evaluation metric, the reported performance on the Kodak image set is comparable to VVC

[28] .

[0117] 2.3.3 Super Prior Model

[0118] In the transform codec approach to image compression, the encoder subnetwork (Section 2.3.2) uses a parametric analysis transform The image vector x is transformed into a latent representation y, which is then quantized to form because It is a discrete value, so it can be losslessly compressed using entropy coding and decoding technology (such as arithmetic coding and decoding) and transmitted as a bit sequence.

[0119] from Figure 2 It can be clearly seen from the center-left and center-right images that There is significant spatial correlation between the elements of . Notably, their scales (center-right image) appear to be spatially coupled. In

[25] , an additional set of random variables was introduced to capture spatial correlation and further reduce redundancy. In this case, the image compression network is Figure 3 shown.

[0120] exist Figure 3 In the figure, the left side of the model is the encoder g a and decoder g s (Explained in Section 2.3.2). The right side is used to obtain The additional super encoder h a Network and Super Decoder s In this architecture, the encoder takes the input image x and feeds it to g a , resulting in a response y with a spatially varying standard deviation. The response y is fed to h a , thus summarizing the distribution of the standard deviation of z. z is then quantified compressed and transmitted as side information. The encoder then uses the quantized vector to estimate σ, the spatial distribution of the standard deviation, and use it to compress and transmit the quantized image representation The decoder first recovers the compressed signal Then use h s to obtain σ, which provides the correct probability estimate for successfully recovering Then Type g s to obtain the reconstructed image.

[0121] When super encoders and super decoders are added to image compression networks, the quantization potential is reduced. space redundancy. Figure 2 The rightmost image in the figure corresponds to the quantized potential values ​​when using the super encoder / decoder. Compared to the center-right image, the spatial redundancy is significantly reduced because the samples of the quantized potential values ​​are less correlated.

[0122] exist Figure 2 , an image from the Kodak dataset is shown on the left; a visualization of the latent representation y of the image is shown on the left; the standard deviation σ of the latent values ​​is shown on the right; and the latent values ​​y after being processed by the super-prior (super-encoder and decoder) network are shown on the right.

[0123] Figure 3 The network architecture of the autoencoder implementing the super-prior model is shown. The image autoencoder network is shown on the left, and the right corresponds to the super-prior subnetwork. The analysis and synthesis transformations are denoted as g a and g s Q represents quantization, and AE and AD represent arithmetic encoder and arithmetic decoder respectively. The super prior model consists of two sub-networks, the super encoder (with h a denoted) and super decoder (with h s denoted by ). The hyper-prior model generates a quantitative hyper-potential value It includes information about quantifying potential values The probability distribution information of the sample points. is included in the bitstream and is related to are transmitted together to the receiver (decoder).

[0124] 2.3.4 Context Model

[0125] Although the hyper-prior model improves the quantitative potential value , but additional improvements can be obtained by using autoregressive models to predict quantitative latent values ​​from their causal context (context model).

[0126] The term "autoregressive" means that the output of a process is later used as input to that process. For example, the context model subnetwork generates a sample of potential values, which is later used as input to get the next sample.

[0127] In

[26] , the authors used a joint architecture, where both the super-prior model sub-network (super-encoder and super-decoder) and the context model sub-network were used. The super-prior and context models were combined to learn to quantize the latent value The probability model of is then used for entropy coding and decoding. Figure 4 As shown, the outputs of the context sub-network and the super decoder sub-network are combined by a sub-network called the entropy parameter, which generates the mean μ and scale (or variance) σ parameters for the Gaussian probability model. The Gaussian probability model is then used to encode the samples of the quantized potential values ​​into the bitstream with the help of the arithmetic encoder (AE) module. In the decoder, the Gaussian probability model is used by the arithmetic decoder (AD) module to obtain the quantized potential values ​​from the bitstream

[0128] Figure 4 A combined model is shown that jointly optimizes an autoregressive component that estimates the probability distribution of the latent value according to its causal context (context model) as well as the hyper-prior and the underlying autoencoder. The real-valued latent representation is quantized (Q) to create a quantized latent value and quantify the super potential value It is compressed into a bitstream via an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD).The highlighted areas correspond to the components performed by the receiver (ie, decoder) to restore the image from the compressed bitstream.

[0129] Typically, the latent samples are modeled as Gaussian distributions or Gaussian mixture models (but not limited to these). In

[26] and according to Figure 4 , the context model and the hyper-prior are jointly used to estimate the probability distribution of the potential sample points. Since the Gaussian distribution can be defined by the mean and variance (also known as sigma or scale), the joint model is used to estimate the mean and variance (denoted as μ and σ).

[0130] 2.3.5 Encoding process using joint autoregressive hyper-prior model

[0131] Figure 4 This corresponds to the prior art compression method proposed in

[26] . In this section and the next section, the encoding and decoding processes are described respectively.

[0132] Figure 5 The encoding process according to

[26] is shown.

[0133] exist Figure 5 In the previous section, the encoding process is described. The input image is first processed by the encoder subnetwork. The encoder transforms the input image into a transformed representation called a potential value (denoted by y). y is then input to the quantizer block, denoted by Q, to obtain a quantized potential value Then it is converted into a bit stream (bits1) using an arithmetic coding module (denoted as AE). The arithmetic coding block sequentially converts Each sample point is converted into a bit stream (bits1) one by one.

[0134] The modular super encoder, context, super decoder, and entropy parameter subnetworks are used to estimate the quantized latent value. The potential value y is input to the super encoder, and the super encoder outputs the super potential value (represented by z). Then, the super potential value is quantized The factorization entropy module generates a probability distribution which is used to encode the quantized super potential value into a bitstream. The quantized super potential value includes information about the quantized potential value. information about the probability distribution of .

[0135] The entropy parameter subnetwork generates probability distribution estimates, which are used to encode quantized latent values. The information generated by the entropy parameters usually includes the mean μ and scale (or variance) σ parameters, which are used together to obtain a Gaussian probability distribution. The Gaussian distribution of a random variable x is defined as where the parameter μ is the mean or expectation of the distribution (as well as its median and mode), and the parameter σ is its standard deviation (or variance, or scale). To define a Gaussian distribution, the mean and variance need to be determined. In

[26] , the entropy parameter module is used to estimate the mean and variance values.

[0136] The subnetwork superdecoder generates part of the information used by the entropy parameter subnetwork, and another part of the information is generated by an autoregressive module called the context module. The context module uses the samples that have been encoded by the arithmetic coding (AE) module to generate information about the probability distribution of the samples of the quantized potential value. It is usually a matrix composed of many sample points. Dimensions can be used such as or The index of the sample point is used to indicate the sample point. Encoded one by one by the AE, usually using raster scan order. In raster scan order, the rows of the matrix are processed from top to bottom, where the samples in the row are processed from left to right. In this scenario (where the AE encodes the samples into the bitstream using raster scan order), the context module uses the previously encoded samples in raster scan order to generate the same samples as the The information generated by the context module and the super decoder is combined by the entropy parameter module to generate the information used to quantize the potential value The probability distribution encoded into the bit stream (bits1).

[0137] Finally, the first and second bit streams are transmitted to a decoder as a result of the encoding process.

[0138] Note that other names may be used for the above modules.

[0139] In the above description, Figure 5 All elements in are collectively referred to as encoders. The analysis transformation that transforms the input image into a latent representation is also called an encoder (or autoencoder).

[0140] 2.3.6 Decoding process using the joint autoregressive hyper-prior model

[0141] Figure 6 The decoding process corresponding to

[26] is shown. Figure 6 The decoding process corresponding to

[26] is depicted separately.

[0142] During the decoding process, the decoder first receives the first bitstream (bits1) and the second bitstream (bits2) generated by the corresponding encoder. Bits2 is first decoded by the arithmetic decoding (AD) module using the probability distribution generated by the factorization entropy subnetwork. The factorization entropy module typically generates the probability distribution using a predetermined template, such as a predetermined mean and variance value in the case of a Gaussian distribution. The output of the arithmetic decoding process for bits2 is It is the quantized super potential value. The AD process is restored to the AE process applied in the encoder. The AE and AD processes are lossless, which means that the quantized super potential value generated by the encoder is Can be reconstructed at the decoder without any changes.

[0143] In obtaining After that, it is processed by the super decoder, whose output is fed to the entropy parameter module. The three subnetworks, context, super decoder, and entropy parameters employed in the decoder are the same as those in the encoder. Therefore, the exact same probability distribution is available in the decoder as in the encoder, which is essential for reconstructing the quantized latent values ​​without any loss. is crucial. Therefore, the quantized potential value obtained in the encoder can be obtained in the decoder The same version of .

[0144] After obtaining the probability distribution (e.g., mean and variance parameters) through the entropy parameter subnetwork, the arithmetic decoding module decodes the samples of the quantized potential value one by one from the bitstream bits1. From a practical point of view, the autoregressive model (context model) is inherently serial and therefore cannot be accelerated using techniques such as parallelization.

[0145] Finally, the fully reconstructed quantized potential value Input to the composite transform (in Figure 6The decoder module is used to obtain the reconstructed image.

[0146] In the above description, Figure 6 All elements in are collectively referred to as decoders. The synthetic transform that transforms the quantized latent values ​​into the reconstructed image is also called a decoder (or self-decoder).

[0147] 2.3.7 Wavelet-based Neural Compression Architecture

[0148] Figure 5 The analysis transform in (expressed as an encoder) and Figure 6 The synthetic transform in (denoted as decoder) can be replaced by a wavelet-based transform. Figure 7 An example implementation based on wavelet transform is shown. In the figure, the input image is first converted from RGB color format to YUV color format. This conversion process is optional and may be missing in other implementations. However, if such a conversion is applied to the input image, the inverse conversion (from YUV to RGB) is also applied before generating the output image. In addition, two additional post-processing modules (Post-processing 1 and 2) are shown in the figure. These modules are also optional and may be missing in other implementations. The core of the encoder based on wavelet transform consists of a wavelet-based forward transform, a quantization module, and an entropy encoding and decoding module. After applying these three modules to the input image, a bitstream is generated. The core of the decoding process consists of entropy decoding, a dequantization process, and a wavelet-based inverse transform operation. The decoding process converts the bitstream into an output image. Figure 7 The encoding process and decoding process are described in .

[0149] After applying the wavelet-based forward transform to the input image, the image is divided into its frequency components in the output of the wavelet-based forward transform. The output of the two-dimensional (2D) forward wavelet transform (depicted in the figure as the iWave forward module) can be used Figure 8 The input to the transformation is the image of a castle. In the example, after the transformation, an output with 7 distinct regions is obtained. The number of distinct regions depends on the specific implementation of the transformation and may be different from 7. Potential numbers of regions are 4, 7, 10, 13, etc.

[0150] Fig. 9The possible partitioning of the latent representation after a two-dimensional (2D) forward transform is shown. The latent representation is the sample points (latent samples or quantized latent samples) obtained after a 2D forward transform. The latent samples are divided into the above 7 parts, which are represented as HH1, LH1, HL1, LL2, HL2, LH2 and HH2. HH1 describes that the part includes high-frequency components in the vertical direction, high-frequency components in the horizontal direction, and the partition depth is 1. HL2 describes that the part includes low-frequency components in the vertical direction, high-frequency components in the horizontal direction, and the partition depth is 2.

[0151] After obtaining the latent samples by forward wavelet transform at the encoder, these latent samples are sent to the decoder by using entropy encoding and decoding. At the decoder, entropy decoding is applied to obtain the latent samples, which are then inversely transformed (by using Figure 7 The iWave inverse module in is used to obtain the reconstructed image.

[0152] 2.4 Neural Networks for Video Compression

[0153] Similar to traditional video codec technology, in neural network-based video compression, neural image compression serves as the basis for intra-frame compression, so the development of neural network-based video compression technology is later than that of neural network-based image compression technology, but due to its complexity, more efforts are needed to solve these challenges. Since 2017, some researchers have been working on neural network-based video compression schemes. Compared with image compression, video compression requires effective methods to eliminate inter-frame redundancy. Therefore, inter-frame prediction is a key step in these works. Motion estimation and compensation are widely adopted, but only recently have they been implemented by trained neural networks.

[0154] Research on neural network-based video compression can be divided into two categories according to the target scenario: random access and low latency. In the case of random access, it requires that decoding can start from any point in the sequence, which usually divides the entire sequence into multiple separate segments, and each segment can be decoded independently. In the case of low latency, its purpose is to reduce the decoding time, so that usually only the previous frame in time can be used as a reference frame to decode the subsequent frame.

[0155] 2.4.1 Low Latency

[0156]

[29] first proposed a video compression scheme using a trained neural network. They first divided the video sequence frames into blocks, and each block will choose one of the two available modes, intra-frame codec or inter-frame codec. When intra-frame codec is selected, there is an associated autoencoder to compress the block. When inter-frame codec is selected, motion estimation and compensation are performed using traditional methods, and the trained neural network will be used for residual compression. The output of the autoencoder is directly quantized and encoded by the Huffman method.

[0157] Chen et al.

[31] proposed another neural network-based video codec scheme using PixelMotionCNN. Frames are compressed in temporal order, and each frame is divided into blocks that are compressed in raster scan order. Each frame is first extrapolated using the two previous reconstructed frames. When a block is to be compressed, the extrapolated frame is fed into PixelMotionCNN along with the context of the current block to derive a latent representation. The residual is then compressed using a variable rate image scheme

[34] . The performance of this scheme is comparable to that of H.264.

[0158] Lu et al.

[32] proposed a realistic end-to-end neural network-based video compression framework, in which all modules are implemented using neural networks. The scheme accepts the current frame and the previously reconstructed frame as input, and uses the optical flow derived from a pre-trained neural network as motion information. The motion information is warped with the reference frame, and then the motion compensated frame is generated by the neural network. The residual and motion information are compressed using two separate neural autoencoders. The entire framework is trained using a single rate-distortion loss function. It achieves better performance than H.264.

[0159] Rippel et al.

[33] proposed an advanced video compression scheme based on neural networks. It inherits and extends the traditional neural network video codec scheme with the following main features: 1) only one autoencoder is used to compress motion information and residuals; 2) motion compensation with multiple frames and multiple optical flows; 3) online states are learned and propagated through subsequent frames over time. The scheme has better performance than the HEVC reference software in terms of multi-scale structural similarity (MS-SSIM).

[0160] J.Lin et al.

[36] proposed an extended video compression framework based on end-to-end neural network based on

[32] . In this solution, multiple frames are used as references. Thus, it is possible to provide a more accurate prediction of the current frame by using multiple reference frames and associated motion information. In addition, motion field prediction is deployed to eliminate motion redundancy along the time domain channel. A post-processing network is also introduced in this study to remove reconstruction artifacts in the previous process. In terms of peak signal-to-noise ratio (PSNR) and MS-SSIM, the performance is significantly better than

[32] and H.265.

[0161] Eirikur et al.

[37] proposed scale-space flow to replace the commonly used optical flow by adding a scale parameter based on the framework of

[32] . It is reported that it achieves better performance than H.264.

[0162] Z.Hu et al.

[38] proposed a multi-resolution representation for optical flow based on

[32] . Specifically, the motion estimation network generates multiple optical flows with different resolutions and lets the network learn which one to choose under the loss function. Compared with

[32] , the performance is slightly improved and better than H.265.

[0163] 2.4.2 Random Access

[0164] Wu et al.

[30] proposed a neural network-based video compression scheme with frame interpolation. The key frames are first compressed using a neural image compressor, and the remaining frames are compressed in a hierarchical order. They perform motion compensation in the perceptual domain, i.e., feature maps are derived at multiple spatial scales of the original frame and the motion is used to warp these feature maps, which are then used in the image compressor. The method is reported to be comparable to H.264.

[0165] Djelouah et al.

[41] proposed an interpolation-based video compression method, in which the interpolation model combines motion information compression and image synthesis, and the same autoencoder is used for both the image and the residual.

[0166] Amirhossein et al.

[35] proposed a neural network-based video compression method based on a variational autoencoder with a deterministic encoder. Specifically, the model includes an autoencoder and an autoregressive prior. Unlike previous methods, this method accepts a group of pictures (GOP) as input and incorporates a three-dimensional (3D) autoregressive prior by considering temporal correlations when encoding and decoding the latent representation. It provides performance comparable to H.265.

[0167] 2.5 Introduction.

[0168] Almost all natural images / videos are in digital format. A grayscale digital image can be represented as in is a set of pixel values, m is the image height, and n is the image width. For example, is the normal setting, in which case Thus a pixel can be represented by an 8-bit integer. An uncompressed grayscale digital image has 8 bits per pixel (bpp), while compressed bits are certainly less.

[0169] Color images are usually represented in multiple channels to record color information. For example, in the RGB color space, an image can be represented by Representation, with three independent channels storing red, green, and blue information. Similar to an 8-bit grayscale image, an uncompressed 8-bit RGB image has 24bpp. Digital images / videos can be represented in different color spaces. Neural network-based video compression schemes have been developed primarily in the RGB color space, while traditional codecs typically use the YUV color space to represent video sequences. In the YUV color space, the image is decomposed into three channels, namely Y, Cb, and Cr, where Y is the luminance component and Cb / Cr are the chrominance components. Since the human visual system is less sensitive to the chrominance components, the benefit of this is that Cb and Cr are usually downsampled for pre-compression.

[0170] A color video sequence consists of multiple color images called frames to record scenes at different time stamps. For example, in the RGB color space, a color video can be composed of X = {x0, x1, ..., x t ,…,x T-1}, where T is the number of frames in the video sequence, If m = 1080, n = 1920, And the video has 50 frames per second (fps), then the data rate of the uncompressed video is 1920×1080×8×3×50=2,488,320,000 bits per second (bps), about 2.32 Gbps, which requires a lot of storage space, so it definitely needs to be compressed before transmission over the Internet.

[0171] Typically, for natural images, lossless methods can achieve a compression ratio of about 1.5 to 3, which is significantly lower than required. Therefore, lossy compression came into being to achieve a higher compression ratio, but at the expense of distortion. Distortion can be measured by calculating the average squared difference between the original image and the reconstructed image, i.e., the mean square error (MSE). For grayscale images, MSE can be calculated using the following formula.

[0172]

[0173] Therefore, the quality of the reconstructed image compared to the original image can be measured by the Peak Signal-to-Noise Ratio (PSNR):

[0174]

[0175] in yes The maximum value in , for example, for 8-bit grayscale images, the maximum value is 255. There are also other quality assessment metrics, such as structural similarity (SSIM) and multi-scale SSIM (MS-SSIM) [4].

[0176] To compare different lossless compression schemes, one only needs to compare the compression ratio for a given resulting rate, and vice versa. However, to compare different lossy compression methods, both the rate and the reconstruction quality must be considered. For example, a commonly used approach is to compute relative rates at several different quality levels and then average these computed rates; the average relative rate is called Bjontegaard's delta rate (BD-rate) [5]. There are other important aspects to evaluate image / video codec schemes, including encoding / decoding complexity, scalability, robustness, etc.

[0177] 3. Question

[0178] 3.1 Core Issues

[0179] State-of-the-art image compression networks usually involve synthesis networks to reconstruct images from bitstreams. Fig. 10A As shown, after passing through the synthetic network input, the spatial domain size increases significantly, such as in Fig. 10B In the case of , the number of pixels along one edge of the feature map can be 16 times that before the synthesis network. This can be a significant challenge when faced with limited memory or high image / video resolution, and can cause the decoder to crash.

[0180] 3.2 Problem Details

[0181] A typical feature map of a neural network is a three-dimensional (3D) array, where the three dimensions indicate channels, width, and height (note that different methods can be used to partition the array). When a feature map is input through a synthesis network, its spatial size is significantly enlarged. For example, the spatial size of the initial feature map is 4x4, and after 4 times of x2 upsampling, the spatial size reaches 64x64. In addition, the convolution kernel is a 4D array. As the spatial size increases, the computational resources required by the synthesis network grow exponentially. Today, high-definition video content is increasing rapidly, and more and more 2k / 4k content is emerging. For neural network-based decoders, decoding this content within a limited memory budget is challenging.

[0182] Fig.11 The detailed spatial size variation is shown as the feature map moves through the synthesis network.

[0183] 4. The present disclosure

[0184] The detailed techniques herein should be considered as examples to explain the general concept. These techniques should not be interpreted narrowly. Moreover, these techniques can be combined in any way.

[0185] 4.1 Objectives of this Disclosure

[0186] The goal of the present disclosure is to solve the potential memory shortage problem caused by limited memory or too large size of reconstructed image / video sequence.

[0187] 4.2 Core of this disclosure

[0188] The core of this disclosure provides a flexible and manageable feature map spatial size when reconstructing an image using a synthesis network. It should be noted that the methods described below can also be applied to certain areas within an image / frame / picture, for example, they can be applied to strips within an image.

[0189] 4.3p Example shows the slice segmentation of the feature map of the synthesis network

[0190] The technique described in this paper provides a flexible solution to control the spatial size of feature maps through patch segmentation. Fig.12 As shown in Figure 1, instead of inputting the entire feature map to the next convolutional layer, the proposed technique splits the feature map into multiple parts in the spatial domain to reduce memory consumption and concatenates them back into a single part after the calculation is completed. In addition, in order to reduce the distortion caused by boundary effects, each part has an overlapping area, i.e., slice segmentation.

[0191] Fig.13 An example of inserting patch segmentation into a synthesis network is shown.

[0192] In the following sections, "frame", "picture" or "image" may have the same meaning. Alternatively, the "frame / picture / image" hereinafter may be replaced by an area within the "frame / picture / image".

[0193] 1. Slice division can be horizontal, vertical, or both horizontal and vertical.

[0194] a. Fig.12 As shown in the leftmost sub-figure, the split can only be vertical.

[0195] i. Slice segmentation can be vertically segmented once or more than once.

[0196] b. Fig.12 As shown in the middle sub-figure, the segmentation can only be horizontal.

[0197] i. Slice partitioning can be horizontally partitioned once or more than once.

[0198] c. Fig.12 As shown in the rightmost sub-figure, the segmentation can be vertical and horizontal.

[0199] i. Slice partitioning can be performed recursively.

[0200] 2. The slice partitioning can be fixed for all frames in a sequence.

[0201] a. Alternatively, the slice partitioning may be different for two frames within a sequence.

[0202] 3. There are many ways to fill the fill area.

[0203] a. In one example, the fill area can be filled with adjacent feature map values. Fig.12 In the vertical segmentation of , the feature map values ​​in part 2 adjacent to the segmentation boundary can be used to fill part 1.

[0204] 4. Patch partitions can be inserted into one or more locations of the synthesis network.

[0205] a. In one example, a slice partition may be inserted into a location such as Fig.13 shown.

[0206] b. In one example, patch partitions can be inserted at multiple locations in the synthesis network.

[0207] 5. The filling size of each sub-section is controllable and can be the same or different.

[0208] 6. Associated parameters may be encoded into the bitstream, such as padding size, number of vertical partitions, or number of horizontal partitions.

[0209] 4.4 Signaling Notification

[0210] Additionally or alternatively, according to the present disclosure, the following may be implemented.

[0211] 1. An indication may be included in the bitstream to indicate the number of slices used to decode the bitstream.

[0212] a. In a specific embodiment, the bitstream includes at least two instructions: a portion of a neural network for generating a reconstructed image is segmented according to a first instruction, and a second portion of the neural network is segmented according to a second instruction.

[0213] 2. Additionally or alternatively, an indication may be included in the bitstream to indicate layer information (e.g., layer id, i.e., indicating which layer of the neural network to decode).

[0214] a. The layer id may indicate the starting layer after which slice segmentation is performed.

[0215] b. The layer id may indicate the starting layer after which slice segmentation stops.

[0216] c. The layer id may indicate the starting layer after which a specified number of slices applies. The number of slices may be indicated in the bitstream or may be predefined.

[0217] d. The layer id may indicate a starting layer, after which no more specified number of slices apply.

[0218] e. The layer id may be an index to a table, wherein the table includes information about the association of layers and indices of a decoding neural network.

[0219] 3. Additionally or alternatively, an indication may be included in the bitstream to indicate the size of the slice.

[0220] 4. Additionally or alternatively, an indication may be included in the bitstream to indicate the location of the slice.

[0221] 5. Additionally or alternatively, an indication may be included in the bitstream to indicate whether slice partitioning is applied. If the indication indicates that no slices are used, neural network-based image reconstruction (ie, decoding) is performed without applying slice partitioning.

[0222] 6. Additionally or alternatively, an indication may be included in the bitstream to indicate a minimum size according to which the number of slices is determined.

[0223] a. This minimum size and the size indicating the reconstructed image size can be used to determine the number of slices.

[0224] b. This minimum size and the size indicating the feature map size can be used to determine the number of patches.

[0225] 4.5 slices of extrapolation

[0226] Additionally or alternatively, according to the present disclosure, the following may be implemented.

[0227] 1. An indication may be included in the bitstream to indicate the size of the reconstructed image. The number of slices to be used is determined based on the size information.

[0228] 2. The number of slices to be used can be determined based on the size of the input feature map. A neural network-based decoder is usually composed of multiple processing layers, and the input of each layer (which is the output of the previous processing layer in order) is called a feature map. According to the present invention, the size of the feature map can be used to determine the number of slices to be used.

[0229] 3. A layer of a neural network may apply slice segmentation based on the size of its input feature map. A check may be performed before applying a layer to determine whether slice segmentation is applied.

[0230] 4. The layers of the neural network can apply slice partitioning based on the size of its input feature map. A check can be performed before applying the layer to determine the number of slices to use.

[0231] 4.6 Interactions between sheets

[0232] 5. In one example, the two slices can be decoded separately.

[0233] 6. In one example, the decoding of one slice may be dependent on the decoding of another slice.

[0234] 7. In one example, it may be signaled in the bitstream whether two slices are to be decoded independently.

[0235] 8. In one example, another process may be applied on-chip.

[0236] 9. For example, filtering can be performed on the boundary of two patches.

[0237] The schemes listed in this disclosure may be used to compress an image, compress a video, compress a portion of an image, or compress a portion of a video.

[0238] 4.7 Benefits of the Disclosure

[0239] The present disclosure provides a method for segmenting a feature map in a synthesis network into multiple parts in the spatial domain and performing calculations on each of the multiple parts in sequence. Therefore, the memory requirement is reduced. In addition, flexible control of slice segmentation parameters (e.g., the way to fill the padding area) provides another possibility for improving codec performance.

[0240] 5. Embodiments of the present disclosure

[0241] The following detailed techniques should be considered as examples to explain the general concept. These techniques should not be interpreted narrowly. Moreover, these techniques can be combined in any way.

[0242] 1. An image decoding method comprising the following steps: using an arithmetic decoder to obtain a reconstruction potential value The reconstruction potential value is input into the synthetic neural network; the output feature image is segmented into multiple parts based on the decoding parameters for the slice segmentation at one or more positions; each part is input into the next level of multiple convolutional layers to obtain the output spatial segmentation feature map; the spatial segmentation feature map is cropped and spliced ​​back to the overall feature map in the spatial domain; and the process is continued until the image is reconstructed.

[0243] 2. An image encoding method, comprising the following steps: obtaining quantized potential values ​​and slice segmentation parameters; and encoding the potential values ​​and the segmentation parameters into a bit stream.

[0244] Another one of the most beautiful places in the world:

[0245] [1]Z.Cheng,H.Sun,M.Takeuchi and J.Katto,“Learned image compressionwith discretized gaussian mixture likelihoods and attention modules,”inProceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition,pp.7939-7948,2020.

[0246] [2]B.Bross,J.Chen,S.Liu and Y.-K.Wang,“Versatile Video Draft(Draft10),”JVET-S2001,Jul.2020.

[0247] [3]RDDony and S.Haykin,“Neural Network Approaches to Image Compression,”Proceedings of the IEEE,vol.83,no.2,pp.288–303,1995.

[0248] [4]Z.Wang,ACBovik,HRSheikh,and EPSimoncelli,“Image QualityAssessment:From Error Visibility to Structural Similarity,”IEEE Transactionson Image Processing,vol.13,no.4,pp.600–612,2004.

[0249] [5]G.Bjontegaard,“Calculation of average PSNR differences between RD-curves,”VCEG,Tech.Rep.VCEG-M33,2001.

[0250] [6]C.E.Shannon,“A mathematical theory of communication,”Bell SystemTechnical Journal,vol.27,no.3,pp.379–423,1948.

[0251]

[17] G.E.Hinton and R.R.Salakhutdinov,“Reducing the dimensionality ofdata with neural networks,”Science,vol.313,no.5786,pp.504–507,2006.

[0252]

[18] G.Toderici,S.M.O’Malley,S.J.Hwang,D.Vincent,D.Minnen,S.Baluja,M.Covell,and R.Sukthankar,“Variable rate image compression with recurrentneural networks,”arXiv preprint arXiv:1511.06085,2015.

[0253]

[19] G.Toderici,D.Vincent,N.Johnston,S.J.Hwang,D.Minnen,J.Shor,andM.Covell,“Full resolution image compression with recurrent neural networks,”in CVPR,2017,pp.5306–5314.

[0254]

[20] N.Johnston,D.Vincent,D.Minnen,M.Covell,S.Singh,T.Chinen,S.JinHwang,J.Shor,and G.Toderici,“Improved lossy image compression with primingand spatially adaptive bit rates for recurrent networks,”in CVPR,2018,pp.4385–4393.

[0255]

[21] M.Covell,N.Johnston,D.Minnen,S.J.Hwang,J.Shor,S.Singh,D.Vincent,and G.Toderici,“Target-quality image compression with recurrent,convolutionalneural networks,”arXiv preprint arXiv:1705.06687,2017.

[0256]

[22] J.Ballé,V.Laparra,and E.P.Simoncelli,“End-to-end optimization ofnonlinear transform codes for perceptual quality,”in PCS.IEEE,2016,pp.1–5.

[0257]

[23] J.Ballé,“Efficient nonlinear transforms for lossy imagecompression,”in′PCS,2018,pp.248–252.

[0258]

[24] J.Ballé,V.Laparra and E.P.Simoncelli,“End-to-end optimized imagecompression,”in International Conference on Learning Representations,2017.

[0259]

[25] J.Ballé,D.Minnen,S.Singh,S.Hwang and N.Johnston,“Variationalimage compression with a scale hyperprior,”in International Conference onLearning Representations,2018.

[0260]

[26] D.Minnen,J.Ballé,G.Toderici,“Joint Autoregressive andHierarchical Priors for Learned Image Compression”,arXiv.1809.02736.1,2,3,4,7

[0261]

[27] Z.Cheng,H.Sun M.Takeuchi and J.Katto,“Learned image compressionwith discretized Gaussian mixture likelihoods and attention modules,”inProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition(CVPR),2020.

[0262]

[28] Github repository“CompressAI: https: / / github.com / InterDigitalInc / CompressAI ,”,InterDigital Inc,accessed Dec 2020.

[0263]

[29] T.Chen,H.Liu,Q.Shen,T.Yue,X.Cao,and Z.Ma,“DeepCoder:A deep neuralnetwork based video compression,”in VCIP.IEEE,2017,pp.1–4.

[0264]

[30] C.-Y.Wu,N.Singhal,and P.Krahenbuhl,“Video compression throughimage interpolation,”in Proceedings of the European Conference on ComputerVision(ECCV),2018,pp.416–431.

[0265]

[31] Z.Chen,T.He,X.Jin,and F.Wu,“Learning for video compression,”IEEETransactions on Circuits and Systems for Video Technology,DOI:10.1109 / TCSVT.2019.2892608,2019.

[0266]

[32] G.Lu,W.Ouyang,D.Xu,X.Zhang,C.Cai,and Z.Gao,“DVC:An end-to-enddeep video compression framework,”in Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition(CVPR),2019.

[0267]

[33] O.Rippel,S.Nair,C.Lew,S.Branson,A.Anderson and L.Bourdev,"LearnedVideo Compression,"2019 IEEE / CVF International Conference on Computer Vision(ICCV),Seoul,Korea(South),2019,pp.3453-3462,doi:10.1109 / ICCV.2019.00355.

[0268]

[34] G.Toderici,D.Vincent,N.Johnston,S.J.Hwang,D.Minnen,J.Shor,andM.Covell,“Full resolution image compression with recurrent neural networks,”in CVPR,2017,pp.5306–5314.

[0269]

[35] A.Habibian,T.Rozendaal,J.Tomczak and T.Cohen,“Video Compressionwith Rate-Distortion Autoencoders,”in Proceedings of the IEEE / CVFInternational Conference on Computer Vision(ICCV),2019,pp.7033-7042.

[0270]

[36] J.Lin,D.Liu,H.Li and F.Wu,“M-LVC:Multiple frames prediction forlearned video compression,”in Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition(CVPR),2020.

[0271]

[37] E.Agustsson,D.Minnen,N.Johnston,J.Ballé,S.J.Hwang andG.Toderici,"Scale-Space Flow for End-to-End Optimized Video Compression,"2020IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),Seattle,WA,USA,2020,pp.8500-8509,doi:10.1109 / CVPR42600.2020.00853.

[0272]

[38] X.Hu,Z.Chen,D.Xu,G.Lu,W.Ouyang and S.Gu,“Improving deep videocompression by resolution-adaptive flow coding,”in European Conference onComputer Vision(ECCV)2020.

[0273]

[39] B.Li,H.Li,L.Li and J.Zhang,"λDomain Rate Control Algorithm forHigh Efficiency Video Coding,"in IEEE Transactions on Image Processing,vol.23,no.9,pp.3841-3854,Sept.2014,doi:10.1109 / TIP.2014.2336550.

[0274]

[40] L.Li,B.Li,H.Li and C.W.Chen,"λDomain Optimal Bit AllocationAlgorithm for High Efficiency Video Coding,"in IEEE Transactions on Circuitsand Systems for Video Technology,vol.28,no.1,pp.130-142,Jan.2018,doi:10.1109 / TCSVT.2016.2598672.

[0275]

[41] Abdelaziz Djelouah,Joaquim Campos,Simone Schaub-Meyer,andChristopher Schroers.Neural inter-frame com-pression for video coding.InICCV,pages 6421–6429,October 2019.

[0276]

[42] F.Bossen,Common Test Conditions and Software ReferenceConfigurations,document Rec.JCTVC-J1100,Stockholm,Sweden,Jul.2012.

[0277] Fig.144000. The system 4000 may include an input 4002 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10 bit multi-component pixel values), or may be received in a compressed or encoded format. The input 4002 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, passive optical networks (PONs), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

[0278] System 4000 may include a codec component 4004 that can implement various codecs or encoding methods described in this document. Codec component 4004 can reduce the average bit rate of the video from input 4002 to the output of codec component 4004 to generate a codec representation of the video. Therefore, codec technology is sometimes referred to as video compression or video transcoding technology. The output of codec component 4004 can be stored or sent via the connected communication, as represented by component 4006. The bitstream (or codec) representation of the storage or communication of the video received at input 4002 can be used by component 4008 to generate pixel values ​​or displayable video sent to display interface 4010. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it should be understood that codec tools or operations are used at the encoder, and the corresponding decoding tools or operations of the results of the inverse codec will be performed by the decoder.

[0279] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or Displayport, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, etc. The techniques described in this document may be implemented in various electronic devices, such as mobile phones, laptop computers, smart phones, or other devices capable of digital data processing and / or video display.

[0280] Fig.154100. The apparatus 4100 may be used to implement one or more of the methods described herein. The apparatus 4100 may be implemented in a smart phone, a tablet, a computer, an Internet of Things (IoT) receiver, etc. The apparatus 4100 may include one or more processors 4102, one or more memories 4104, and a video processing circuit 4106. The processor(s) 4102 may be configured to implement one or more methods described in this document. The memory(s) 4104 may be used to store data and code that are used to implement the methods and techniques described herein. The video processing circuit 4106 may be used to implement some of the techniques described in this document in hardware circuits. In some embodiments, the video processing circuit 4106 may be at least partially included in a processor 4102 (e.g., a graphics coprocessor).

[0281] Fig.16 4200 is a flow chart of an example method 4200 of video processing. The method 4200 includes determining to apply a pre-processing function to visual media data as part of an image compression framework at step 4202. Performing conversion between the visual media data and a bitstream based on the image compression framework at step 4204. According to an example, the conversion of step 4204 may include encoding at an encoder or decoding at a decoder.

[0282] It should be noted that the method 4200 may be implemented in an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory having instructions thereon, for example, the video encoder 4400, the video decoder 4500, and / or the encoder 4600. In this case, the instructions, when executed by the processor, cause the processor to perform the method 4200. In addition, the method 4200 may be performed by a non-transitory computer-readable medium comprising a computer program product for use by a video codec device. The computer program product comprises computer-executable instructions stored on a non-transitory computer-readable medium, such that the computer-executable instructions, when executed by the processor, cause the video codec device to perform the method 4200.

[0283] Fig.17 43 is a block diagram illustrating an example video codec system 4300 that can utilize the techniques of the present disclosure. The video codec system 4300 may include a source device 4310 and a target device 4320. The source device 4310 generates encoded video data, which may be referred to as a video encoding device. The target device 4320 may decode the encoded video data generated by the source device 4310, which may be referred to as a video decoding device.

[0284] Source device 4310 may include a video source 4312, a video encoder 4314, and an input / output (I / O) interface 4316. Video source 4312 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system that generates video data, or a combination of these sources. Video data may include one or more pictures. Video encoder 4314 encodes video data from video source 4312 to generate a bitstream. The bitstream may include a sequence of bits that form a codec representation of video data. The bitstream may include a codec picture and associated data. The codec picture is a codec representation of a picture. Associated data may include sequence parameter sets, picture parameter sets, and other syntax elements. I / O interface 4316 includes a modulator / demodulator (modem) and / or a transmitter. The encoded video data may be directly sent to target device 4320 via network 4330 via I / O interface 4316. The encoded video data may also be stored on storage medium / server 4340 for access by target device 4320.

[0285] Target device 4320 may include an I / O interface 4326, a video decoder 4324, and a display device 4322. I / O interface 4326 may include a receiver and / or a modem. I / O interface 4326 may obtain encoded video data from source device 4310 or storage medium / server 4340. Video decoder 4324 may decode the encoded video data. Display device 4322 may display the decoded video data to a user. Display device 4322 may be integrated with target device 4320, or may be outside target device 4320 configured to be connected to an external display device.

[0286] The video encoder 4314 and the video decoder 4324 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other current and / or further standards.

[0287] Fig.18 is a block diagram showing an example of a video encoder 4400, which may be Fig.17 4300. Video encoder 4400 may be configured to perform any or all of the techniques of the present disclosure. Video encoder 4400 includes multiple functional components. The techniques described in the present disclosure may be shared between various components of video encoder 4400. In some examples, the processor may be configured to perform any or all of the techniques described in the present disclosure.

[0288] The functional components of the video encoder 4400 may include a segmentation unit 4401, a prediction unit 4402 (which may include a mode selection unit 4403, a motion estimation unit 4404, a motion compensation unit 4405 and an intra-frame prediction unit 4406), a residual generation unit 4407, a transform processing unit 4408, a quantization unit 4409, an inverse quantization unit 4410, an inverse transform unit 4411, a reconstruction unit 4412, a buffer 4413 and an entropy coding unit 4414.

[0289] In other examples, the video encoder 4400 may include more, fewer, or different functional components. In one example, the prediction unit 4402 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.

[0290] Furthermore, some components, such as the motion estimation unit 4404 and the motion compensation unit 4405 , may be highly integrated but are represented separately in the example of the video encoder 4400 for purposes of explanation.

[0291] The segmentation unit 4401 may segment the picture into one or more video blocks. The video encoder 4400 and the video decoder 4500 may support various video block sizes.

[0292] The mode selection unit 4403 may select one of the intra or inter coding modes, for example, based on the error result, and provide the resulting intra or inter coding block to the residual generation unit 4407 to generate residual block data and to the reconstruction unit 4412 to reconstruct the coding block for use as a reference picture. In some examples, the mode selection unit 4403 may select a combination of intra and inter prediction (CIIP) mode, where the prediction is based on an inter prediction signal and an intra prediction signal. The mode selection unit 4403 may also select a resolution of motion vectors (e.g., sub-pixel or integer pixel precision) for blocks in the case of inter prediction.

[0293] To perform inter-frame prediction on the current video block, the motion estimation unit 4404 may generate motion information of the current video block by comparing one or more reference frames from the buffer 4413 with the current video block. The motion compensation unit 4405 may determine a predicted video block for the current video block based on the motion information and decoded samples of a picture from the buffer 4413 (rather than a picture associated with the current video block).

[0294] The motion estimation unit 4404 and the motion compensation unit 4405 may perform different operations for the current video block, for example, performing different operations depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0295] In some examples, the motion estimation unit 4404 may perform unidirectional prediction of the current video block, and the motion estimation unit 4404 may search for a reference video block of the current video block in the reference picture of list 0 or list 1. The motion estimation unit 4404 may then generate a reference index indicating a reference video block in the reference picture of list 0 or list 1 and a motion vector indicating a spatial displacement between the current video block and the reference video block. The motion estimation unit 4404 may output the reference index, the prediction direction indicator, and the motion vector as motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current block based on the reference video block indicated by the motion information of the current video block.

[0296] In other examples, the motion estimation unit 4404 may perform bidirectional prediction of the current video block, and the motion estimation unit 4404 may search for a reference video block of the current video block in the reference picture of list 0 and may also search for another reference video block of the current video block in the reference picture of list 1. The motion estimation unit 4404 may then generate a reference index indicating a reference video block in the reference picture of list 0 or list 1 and a motion vector indicating a spatial displacement between the reference video block and the current video block. The motion estimation unit 4404 may output the reference index and the motion vector of the current video block as motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[0297] In some examples, motion estimation unit 4404 may output a full set of motion information for decoding processing by a decoder. In some examples, motion estimation unit 4404 may not output a full set of motion information for the current video. Instead, motion estimation unit 4404 may signal the motion information of the current video block with reference to the motion information of another video block. For example, motion estimation unit 4404 may determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.

[0298] In one example, the motion estimation unit 4404 may indicate in a syntax structure associated with the current video block a value that indicates to the video decoder 4500 that the current video block has the same motion information as another video block.

[0299] In another example, the motion estimation unit 4404 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 4500 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0300] As discussed above, the video encoder 4400 can predictively signal motion vectors. Two examples of predictive signaling techniques that can be implemented by the video encoder 4400 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0301] The intra prediction unit 4406 may perform intra prediction on the current video block. When the intra prediction unit 4406 performs intra prediction on the current video block, the intra prediction unit 4406 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a predicted video block and various syntax elements.

[0302] The residual generation unit 4407 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the predicted video block(s) of the current video block from the current video block. The residual data for the current video block may include residual video blocks corresponding to different sample point components of the samples in the current video block.

[0303] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 4407 may not perform a subtraction operation.

[0304] The transform processing unit 4408 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0305] After the transform processing unit 4408 generates a transform coefficient video block associated with the current video block, the quantization unit 4409 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0306] The inverse quantization unit 4410 and the inverse transform unit 4411 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 4412 may add the reconstructed residual video block to the corresponding samples from one or more predicted video blocks generated by the prediction unit 4402 to generate a reconstructed video block associated with the current block for storage in the buffer 4413.

[0307] After the reconstruction unit 4412 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0308] The entropy coding unit 4414 may receive data from other functional components of the video encoder 4400. When the entropy coding unit 4414 receives data, the entropy coding unit 4414 may perform one or more entropy coding operations to generate entropy coded data and output a bitstream including the entropy coded data.

[0309] Fig.19 is a block diagram showing an example of a video decoder 4500, which may be Fig.17 4300 in the system 4300 shown in FIG. 4324. Video decoder 4500 can be configured to perform any or all of the techniques of the present disclosure. In the example shown, video decoder 4500 includes multiple functional components. The techniques described in the present disclosure can be shared between various components of video decoder 4500. In some examples, the processor can be configured to perform any or all of the techniques described in the present disclosure.

[0310] In the example shown, the video decoder 4500 includes an entropy decoding unit 4501, a motion compensation unit 4502, an intra-frame prediction unit 4503, an inverse quantization unit 4504, an inverse transform unit 4505, a reconstruction unit 4506, and a buffer 4507. In some examples, the video decoder 4500 can perform a decoding process that is generally the inverse of the encoding process described with respect to the video encoder 4400.

[0311] The entropy decoding unit 4501 may retrieve a coded bitstream. The coded bitstream may include entropy coded video data (e.g., coded blocks of video data). The entropy decoding unit 4501 may decode the entropy coded video data, and based on the entropy decoded video data, the motion compensation unit 4502 may determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information. The motion compensation unit 4502 may determine such information, for example, by performing AMVP and merge modes.

[0312] The motion compensation unit 4502 may generate a motion compensated block, possibly interpolated based on an interpolation filter. An identifier of the interpolation filter to be used with sub-pixel precision may be included in the syntax element.

[0313] The motion compensation unit 4502 may calculate interpolated values ​​of a sub-integer number of pixels of the reference block using the interpolation filter used by the video encoder 4400 during encoding of the video block. The motion compensation unit 4502 may determine the interpolation filter used by the video encoder 4400 based on received syntax information and use the interpolation filter to generate a prediction block.

[0314] The motion compensation unit 4502 can use some syntax information to determine: the size of the blocks used to encode (multiple) frames and / or (multiple) slices of the coded video sequence, the partitioning information describing how each macroblock of the pictures of the coded video sequence is partitioned, the mode indicating how to encode each partition, one or more reference frames (and reference frame lists) for each inter-frame coding block, and other information for decoding the coded video sequence.

[0315] The intra prediction unit 4503 may form a prediction block from spatially neighboring blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 4504 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 4501. The inverse transform unit 4505 applies an inverse transform.

[0316] The reconstruction unit 4506 may sum the residual block with the corresponding prediction block generated by the motion compensation unit 4502 or the intra prediction unit 4503 to form a decoded block. As desired, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 4507, which provides reference blocks for subsequent motion compensation / intra prediction and also produces decoded video for presentation on a display device.

[0317] Fig. 20 4600 is a schematic diagram of an example encoder 4600. The encoder 4600 is suitable for implementing VVC technology. The encoder 4600 includes three loop filters, namely, a deblocking filter (DF) 4602, a sample adaptive offset (SAO) 4604, and an adaptive loop filter (ALF) 4606. Unlike the DF 4602 that uses a predefined filter, the SAO 4604 and the ALF 4606 utilize the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, and signaling the offset and filter coefficients using codec-side information. The ALF 4606 is located at the last processing stage of each picture and can be seen as a tool that attempts to capture and repair artifacts created by previous stages.

[0318] The encoder 4600 also includes an intra prediction component 4608 and a motion estimation / compensation (ME / MC) component 4610 configured to receive an input video. The intra prediction component 4608 is configured to perform intra prediction, and the ME / MC component 4610 is configured to perform inter prediction using a reference picture obtained from a reference picture buffer 4612. The residual block from the inter prediction or intra prediction is input to the transform (T) component 4614 and the quantization (Q) component 4616 to generate quantized residual transform coefficients, which are input to the entropy coding component 4618. The entropy coding component 4618 entropy codes and decodes the prediction result and the quantized transform coefficients and sends them to a video decoder (not shown). The quantized component output from the quantization component 4616 can be input to the inverse quantization (IQ) component 4620, the inverse transform component 4622, and the reconstruction (REC) component 4624. The REC component 4624 is capable of outputting images to the DF 4602 , SAO 4604 , and ALF 4606 for filtering before these pictures are stored in the reference picture buffer 4612 .

[0319] Fig.21 2100 is an image decoding method according to an embodiment of the present disclosure. The method 2200 may be implemented by a decoding device (eg, a decoder). In block 2102, the decoding device uses an arithmetic decoder to obtain a reconstruction potential value In box 2104, the decoding device inputs the reconstructed potential value into the synthetic neural network.

[0320] In box 2106, the decoding device divides the output feature image into multiple parts based on the decoding parameters of one or more positions. In box 2108, the decoding device inputs each of the multiple parts into the next level of multiple convolutional layers to obtain a spatially segmented feature map at the output. In box 2110, the decoding device crops the spatially segmented feature map and splices it back to the overall feature map in the spatial domain until the image is reconstructed.

[0321] Fig. 22 It is an image encoding method 2200 according to an embodiment of the present disclosure. The method 2200 can be implemented by an encoding device (e.g., an encoder). In box 2202, the encoding device obtains a quantized potential value. In box 2204, the encoding device obtains a parameter for slice segmentation. In box 2206, the encoding device encodes the potential value and the parameter for slice segmentation into a bitstream so that a decoder receiving the bitstream can crop the spatially segmented feature map and splice it back to the overall feature map in the spatial domain until the image is reconstructed.

[0322] A list of solutions preferred by some examples is provided next.

[0323] The following solutions illustrate examples of the techniques discussed herein.

[0324] 1. An image decoding method, comprising the steps of: using an arithmetic decoder to obtain a reconstruction potential value The reconstruction potential value is input into the synthetic neural network; the output feature image is segmented into multiple parts based on the decoding parameters for the slice segmentation at one or more positions; each part is input into the next level of multiple convolutional layers to obtain the output spatial segmentation feature map; the spatial segmentation feature map is cropped and spliced ​​back to the overall feature map in the spatial domain; and the process is continued until the image is reconstructed.

[0325] 2. An image encoding method, comprising the following steps: obtaining quantized potential values ​​and slice segmentation parameters; and encoding the potential values ​​and the segmentation parameters into a bit stream.

[0326] 3. A video data processing device, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method as claimed in any one of claims 1 to 2.

[0327] 4. A non-transitory computer-readable medium, comprising: a computer program product used by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, so that when the computer executable instructions are executed by a processor, the video codec device performs the method as described in any one of claims 1 to 2.

[0328] 5. A non-transitory computer-readable recording medium storing a bit stream of a video generated by a method performed by a video processing device, wherein the method includes the method according to any one of claims 1 to 2.

[0329] 6. A method for storing a bit stream of a video, comprising the method as claimed in any one of claims 1 to 2.

[0330] 7. A method, apparatus or system as described in this document.

[0331] In the solution described herein, an encoder can comply with the format rules by generating a codec representation according to the format rules. In the solution described herein, a decoder can parse syntax elements in the codec representation using the format rules, wherein the presence and absence of syntax elements are understood according to the format rules to produce decoded video.

[0332] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, during the conversion from a pixel representation of a video to a corresponding bitstream representation, a video compression algorithm may be applied, and vice versa. As defined by the syntax, the bitstream representation of the current video block may correspond, for example, to bits that are juxtaposed or interspersed in different places within the bitstream. For example, macroblocks may be encoded according to error residual values ​​of transforms and codecs and also using bits in headers and other fields in the bitstream. And, during conversion, the decoder may parse the bitstream based on this determination, knowing that some fields may or may not be present, as described in the above solution. Similarly, the encoder may determine whether to include or not include certain syntax fields, and generate the codec representation accordingly by including the syntax fields or not including the syntax fields in the codec representation.

[0333] The disclosed and other solutions, examples, embodiments, modules and functional operations described in this document can be implemented in digital electronic circuits, or in computer software, firmware or hardware (including the structures disclosed in this document and their structural equivalents), or in a combination of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, which are used to be executed by a data processing device or control the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that affect machine-readable propagation signals, or a combination of one or more of them. The term "data processing device" includes all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal generated to encode information for transmission to a suitable receiver device, such as a machine-generated electrical signal, an optical signal, or an electromagnetic signal.

[0334] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language (including compiled or interpreted languages), and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0335] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, and the apparatus can also be implemented as special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits).

[0336] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operably coupled to receive data from or transfer data to or from the one or more mass storage devices. However, a computer does not require such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and compact disk read-only memory (CD ROM) and digital versatile disk read-only memory (DVD-ROM) disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0337] Although this patent document contains many details, these details should not be interpreted as limitations on any subject matter or the scope of possible protection, but rather as descriptions of features of specific embodiments specified for specific technologies. Certain features described in the context of separate embodiments in this patent document may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be excluded from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0338] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed, to achieve the desired results. In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0339] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and shown in this patent document.

[0340] A first component is directly coupled to a second component when there are no intermediate components other than a line, trace, or another medium between the first and second components. A first component is indirectly coupled to a second component when there are intermediate components other than a line, trace, or another medium between the first and second components. The term "coupled" and its variations include direct coupling and indirect coupling. Unless otherwise specified, the use of the term "about" is meant to include a range of ±10% of the subsequent numerical value.

[0341] Although multiple embodiments are provided in the present disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. This example is considered to be illustrative rather than restrictive, and is not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0342] In addition, without departing from the scope of the present disclosure, the techniques, systems, subsystems, and methods described and shown as discrete or separate in the various embodiments may be combined or integrated with other systems, modules, techniques, or methods. Other items shown or discussed as coupled may be directly connected, or may be indirectly coupled or communicated through some interface, device, or intermediate component, whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations may be identified by those skilled in the art, and these changes, substitutions, and alterations may be made without departing from the spirit and scope disclosed herein.

[0343] The solutions listed in this disclosure may be used to compress an image, compress a video, compress a portion of an image, or compress a portion of a video.

[0344] In addition, the techniques, systems, subsystems, and methods described and illustrated in the various embodiments may be used to compress an image, compress a video, compress a portion of an image, or compress a portion of a video.

Claims

1. An image processing method comprising the following steps: Get reconstruction potential value inputting the reconstructed potential value into a synthetic neural network; Based on the parameters at one or more positions, the output feature image is divided into multiple parts; Inputting each of the plurality of parts into a next level of a plurality of convolutional layers respectively, so as to obtain a feature map of spatial domain segmentation at an output end; as well as The spatially segmented feature map is cropped and spliced ​​back to the overall feature map in the spatial domain until the image is reconstructed.

2. An image processing method comprising the following steps: Get quantified potential value; Get the parameters of slice segmentation; as well as The potential value and the slice segmentation parameters are encoded into a bitstream so that a decoder receiving the bitstream can crop the spatially segmented feature map and splice it back to the overall feature map in the spatial domain until the image is reconstructed.

3. The method according to any one of claims 1 to 2, wherein: The slice division is either horizontal or vertical.

4. The method according to any one of claims 1 to 2, wherein: The slice division is vertical only.

5. The method according to any one of claims 1 to 2, wherein: The tile segmentation is vertical, and wherein the tile segmentation is performed more than once.

6. The method according to any one of claims 1 to 2, wherein: The slice division is horizontal only.

7. The method according to any one of claims 1 to 2, wherein: The tile segmentation is horizontal, and wherein the tile segmentation is performed more than once.

8. The method according to any one of claims 1 to 2, wherein: The slice divisions are horizontal and vertical.

9. The method according to any one of claims 1 to 8, wherein: The slice segmentation is performed recursively.

10. The method according to any one of claims 1 to 9, wherein: The slice partitioning is fixed for all frames.

11. The method according to any one of claims 1 to 10, wherein: The slice partitioning is different for two or more frames within a sequence.

12. The method according to any one of claims 1 to 11, wherein: Feature map values ​​in a first portion of the plurality of portions adjacent to the segmentation boundary are used to fill a second portion of the plurality of portions.

13. The method according to any one of claims 1 to 12, wherein: The patch segmentation is inserted into a position of the synthetic neural network.

14. The method according to any one of claims 1 to 12, wherein: The patch segmentations are inserted into multiple locations of the synthetic neural network.

15. The method according to any one of claims 1 to 14, wherein: The filling size of each sub-portion is controllable, and the filling sizes of different sub-portions are the same or different.

16. The method according to claim 2, further comprising: One or more of a padding size, a number of vertical partitions, and a number of horizontal partitions are encoded into the bitstream.

17. The method according to any one of claims 1 to 16, wherein: The bitstream includes a number of slices used to decode the bitstream.

18. The method according to any one of claims 1 to 17, wherein: The bitstream includes a first indication and a second indication, wherein a portion of the synthetic neural network is segmented using the first indication, and wherein another portion of the synthetic neural network is segmented using the second indication.

19. The method according to any one of claims 1 to 17, wherein: The bitstream includes a layer identifier, and wherein the layer identifier identifies a layer of the synthetic neural network.

20. The method according to claim 19, wherein: The layer identifier indicates a starting layer after which the slice segmentation is performed.

21. The method according to claim 19, wherein: The layer identifier indicates a starting layer after which the slice segmentation stops.

22. The method according to claim 19, wherein: The layer identifier indicates a starting layer after which a specified number of slices applies, and wherein the specified number is indicated in the bitstream.

23. The method according to claim 19, wherein: The layer identifier indicates a starting layer after which a specified number of slices are no longer applied.

24. The method according to claim 19, wherein: The layer identifier comprises an index to a table, and wherein the table comprises information regarding an association of the index and the layer of the synthetic neural network.

25. The method according to any one of claims 1 to 24, wherein: The bitstream includes an indicator indicating the size of the slice.

26. The method according to any one of claims 1 to 25, wherein: The bitstream includes an indicator indicating a location of a slice.

27. The method according to any one of claims 1 to 26, wherein: The bitstream includes an indicator indicating whether the slice segmentation is applied, and wherein when the slice segmentation is not applied, a neural network based image reconstruction is performed.

28. The method according to any one of claims 1 to 27, wherein: The bitstream comprises an indicator indicating a minimum slice size, and wherein the minimum slice size is used to determine the number of slices.

29. The method according to claim 28, wherein: The size of the reconstructed image is also used to determine the number of slices.

30. The method of claim 28, wherein: The size of the output feature map is also used to determine the number of patches.

31. The method according to any one of claims 1 to 30, wherein: The bitstream includes an indicator indicating a size of the reconstructed image, and wherein the size of the reconstructed image is used to determine the number of slices.

32. The method according to any one of claims 1 to 31, wherein: The number of slices to use is based on the size of the input feature map.

33. The method according to any one of claims 1 to 32, wherein: A layer of the synthetic neural network applies a patch segmentation based on the size of the input feature map, and wherein before applying the layer, it is determined whether to apply the patch segmentation.

34. The method according to any one of claims 1 to 32, wherein: The layers of the synthetic neural network apply slice segmentation based on the size of the input feature map, and before applying the layer, the number of slices to be used is determined.

35. The method according to any one of claims 1 to 34, wherein: Two or more of the plurality of slices are independently decoded.

36. The method according to any one of claims 1 to 34, wherein: The decoding of the second slice depends on the decoding of the first slice.

37. The method according to any one of claims 1 to 34, wherein: The bitstream includes an indicator indicating whether two or more of the plurality of slices are independently decoded.

38. The method according to any one of claims 1 to 37, wherein: Filter the boundary between two patches.

39. The method according to any one of claims 1 to 38, wherein: The intermediate tensor size reaches its maximum value before the last deconvolution or pixel shuffling layer in the synthesis transformation.

40. The method according to any one of claims 1 to 39, wherein: right The sharding is performed with overlap to control the maximum memory size of the intermediate latent tensor, where Represents a latent space tensor of size [C,h4,w4] And where C represents the channels, h represents the height of the tensor, and w represents the width of the tensor.

41. The method according to any one of claims 1 to 40, wherein: Slice positions and slice sizes are signaled independently for primary and secondary components in the bitstream.

42. The method according to any one of claims 1 to 41, wherein: Each slice undergoes synthesis transformation independently.

43. The method according to any one of claims 1 to 42, wherein: The patches are concatenated to each other after reconstruction, and the overlapping regions of the patches are discarded.

44. The method according to any one of claims 1 to 43, wherein: The bitstream includes one or both of a slice-enable luma flag and a slice-enable chroma flag for slices of the primary component and the secondary component.

45. The method according to any one of claims 1 to 44, wherein: The bitstream includes one or both of a slice-size luma syntax element and a slice-size chroma syntax element for the primary component and the secondary component.

46. ​​The method according to any one of claims 1 to 45, wherein: The bitstream includes one or both of slice-overlapped luma syntax elements and slice-overlapped chroma syntax elements for the primary component and the secondary component.

47. The method according to any one of claims 1 to 46, wherein: The reconstructed potential value is obtained using an arithmetic decoder.

48. A video data processing device, comprising: processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 47.

49. A non-transitory computer-readable medium comprising: A computer program product for use by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer readable medium, such that when the computer executable instructions are executed by a processor, the video codec device performs the method as claimed in any one of claims 1 to 47.

50. A non-transitory computer-readable recording medium storing a bit stream of a video generated by a method performed by a video processing device, wherein: The method comprises the method of any one of claims 1 to 47.

51. A method of storing a bitstream of a video, comprising the method of any one of claims 1 to 47.

52. A method, apparatus or system as described in this document.

Images