Image and video compression using a learned dictionary of implicit neural representations

Abstract

Video image compression and decompression are achieved using an improved implicit neural representation (INR) compression method. Specifically, representations are learned locally for smaller image regions, eliminating spatiotemporal redundancies by reusing large portions of the representation. In one embodiment, a local INR corresponding to a segment of the image / video is calculated for each portion of the segment. Large portions of the representations for the neural network weights are reused by decomposing each representation into two parts: a head (preferably containing most of the weights) and a tail (a lightweight network designed to adapt the head to specific image patches). In another embodiment, large portions of the representations are learned over a large set of image patches to obtain a dictionary that is known on both the encoding and decoding sides, reducing the amount of data transmitted.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of European Application No. 22306530.1, filed October 11, 2022, which is incorporated herein by reference in its entirety.

[0002] FIELD OF THE INVENTION At least one of the present embodiments generally relates to a method or apparatus for compressing images and videos using neural network-based tools. [Background technology]

[0003] Compared to traditional approaches used in standardization, Machine Learning (ML) has emerged as a new tool to disrupt compression. The main idea is to learn the entire compression chain, including content description, quantization, entropy coding, and descriptor decompression. Summary of the Invention

[0004] At least one of the present embodiments generally relates to a method or apparatus in the context of image and video compression using a novel Neural Network (NN)-based tool. In particular, one objective of the described embodiment is to improve Implicit Neural Representation (INR) methods for compression. Specifically, representations are learned locally (for smaller image regions) and spatiotemporal redundancies are eliminated by reusing most of the representation. Local INR provides a better rate-distortion fit.

[0005] According to a first aspect, a method is provided, comprising: partitioning at least a portion of a video image into patches, determining at least one head network from global information of the at least a portion of the video image, determining a plurality of tail networks from the patches of the at least a portion of the video image, determining weights corresponding to the head networks, optimizing the tail layer weights by minimizing an implicit neural representation of the patches through learning the weights of the at least one head network, and encoding the tail layer weights into video data.

[0006] According to a second aspect, there is provided a method, comprising: analyzing video data for tail network weights for patches of the video data; and reconstructing the patches of the video data using the weights together with an optimized head network.

[0007] According to another aspect, an apparatus is provided, the apparatus comprising a processor, the processor may be configured to implement the general aspects by performing any of the described methods.

[0008] According to another general aspect of at least one embodiment, there is provided a device comprising an apparatus according to any of the decoding embodiments and at least one of (i) an antenna configured to receive a signal including the video block, (ii) a band limiter configured to limit the received signal to a band of frequencies including the video block, or (iii) a display configured to display an output representing the video block.

[0009] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated according to any of the described encoding embodiments or variations.

[0010] According to another general aspect of at least one embodiment, there is provided a signal including video data generated according to any of the described encoding embodiments or variations.

[0011] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, there is provided a computer program product including instructions that, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0013] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

[0014] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content, the data content including instructions for performing either an encoding method or a decoding method. [Brief explanation of the drawings]

[0015] [Figure 1] 1 illustrates an exemplary image parameterization. [Figure 2] Illustrate an example of an implicit neural representation. [Figure 3] 1 illustrates an example of an image fitting the SIREN network and the residuals. [Figure 4] Illustrate an example of multipatch learning. [Figure 5] Illustrates examples of a) ground truth patches, b) patches fitted by the shared network, and c) patches fine-tuned one by one. [Figure 6]Illustrates another example of a) ground truth patches, b) patches fitted by a shared network, and c) patches fine-tuned one by one. [Figure 7] Illustrates another example of a) ground truth patches, b) patches fitted by a shared network, and c) patches fine-tuned one by one. [Figure 8] Illustrates another example of a) ground truth patches, b) patches fitted by a shared network, and c) patches fine-tuned one by one. [Figure 9] Illustrates another example of a) ground truth patches, b) patches fitted by a shared network, and c) patches fine-tuned one by one. [Figure 10] Illustrates another example of a) ground truth patches, b) patches fitted by a shared network, and c) patches fine-tuned one by one. [Figure 11] Illustrates another example of a) ground truth patches, b) patches fitted by a shared network, and c) patches fine-tuned one by one. [Figure 12] Illustrates another example of a) ground truth patches, b) patches fitted by a shared network, and c) patches fine-tuned one by one. [Figure 13] Illustrates another example of a) ground truth patches, b) patches fitted by a shared network, and c) patches fine-tuned one by one. [Figure 14] Illustrates another example of a) ground truth patches, b) patches fitted by a shared network, and c) patches fine-tuned one by one. [Figure 15] 1 illustrates one embodiment of a method for encoding video using the described embodiments. [Figure 16] 1 illustrates one embodiment of a method for decoding video using the described embodiments. [Figure 17] 1 illustrates one embodiment of an apparatus for encoding or decoding using the described embodiments. [Figure 18] Illustrates standard common video compression schemes. [Figure 19] Illustrates a standard common video decompression scheme. [Figure 20] 1 illustrates a processor-based system for encoding / decoding under the general aspects described. DETAILED DESCRIPTION OF THE INVENTION

[0016] The context of the described embodiment is the compression of image / video content.

[0017] Among these tools, a new proposal based on implicit neural representation (INR) has recently emerged. The main idea is to represent an image function. For 2D images, this function maps 2D coordinates to signal RGB values. Once the function is overfitted using a neural network, transmitting the image is equivalent to transmitting the neural network weights. INR has been used for a variety of signals, including audio and 3D point clouds.

[0018] Currently, the performance of INR for compression, even with state-of-the-art methods, barely reaches the performance of the SOTA method. The described embodiments propose contributions to improve performance.

[0019] The described embodiments relate to improving state-of-the-art INR methods for compression. Specifically, representations are learned locally (for smaller image regions) and spatiotemporal redundancies are eliminated by reusing large portions of the representations. Although reference is made to partitioning video images into "patches," the present aspects and embodiments are not limited to patches and may relate to any irregular partitioning or segmentation of an image.

[0020] The described embodiments are concerned with finding a local INR that provides a better fit in terms of rate distortion. This corresponds to having a segment (regular or irregular) of the image / video to be compressed and calculating the INR for each part of the segment. To exploit spatio-temporal redundancy, a large portion of the representation (in terms of neural network weights) is assumed to be reused. More specifically, each representation is decomposed into two parts: a head (preferably containing most of the weights) and a tail (a lightweight network designed to adapt the head to a specific image patch). The described embodiments propose two approaches: - Large parts of the representation are used across different parts of the image. This is called multi-patch learning. For a given image, one or several head networks can be used. This set of head networks can also be used temporally. Alternatively, this bulk of the representation can be trained over a large set of image patches to obtain the dictionary of the head network, which, once trained, is known at the encoder and decoder side and is therefore no longer a transmission cost.

[0021] First, recall some background and known techniques regarding INR necessary for understanding the aspects and embodiments described herein.

[0022] An implicit or coordinate-based representation parameterizes a signal as a continuous function. In a 2D image, a pixel has coordinates (x,y) that describe its location in the image. The goal of a continuous function is to obtain the RGB value of a pixel based on its coordinates. Figure 1 illustrates a coordinate-based representation.

[0023] As shown in Figure 1, in a coordinate-based representation, each pixel has coordinates (x, y), such as x∈[0,1] and y∈[0,1], and a function f is parameterized by a neural network to obtain RGB values. An implicit representation takes pixel coordinates as input and parameterizes a function f by a neural network to obtain the corresponding RGB values.

[0024] The exact formula for this function is unknown, and a neural network is used to approximate this function, which is called an implicit neural representation. Figure 2 illustrates an implicit neural representation. Figure 2 shows a diagram of an implicit neural representation: the neural network takes pixel coordinates as input and gives an RGB image as output. During training, the network learns θ, which represent the neural network weights. Essentially, the neural network is learning a function that matches pixel coordinates to their RGB values.

[0025] INRs have several advantages. First, they are no longer tied to spatial resolution. This means that the image is independent of the number of pixels. INRs can be sampled at any spatial resolution, so they have "infinite resolution." Second, because INRs are independent of spatial resolution, memory requirements scale with the complexity of the underlying signal.

[0026] One major problem with traditional Multi-Layer Perceptrons (MPLs) is that the resulting functions are smooth. Therefore, it is difficult to approximate the high frequencies of the signal. To solve this, methods have been proposed that use either a sinusoidal activation function or Fourier embedding.

[0027] Consider I as an image to be fitted to a SIREN (sinunsoidal representation network) network. The image can be represented by the following notation I[x,y], where (x,y) represent each pixel coordinate. The SIREN network returns the RGB value at each pixel location (x,y). The SIREN function

[0028]

number

[0029] The goal is to achieve f under a certain distortion measure. θ to I[x,y] and using the mean squared error we get the following optimization problem:

[0030]

number

[0031] FIG. 3 provides an example of a reconstruction that can be achieved using Siren INR.

[0032] Figure 3 shows SIREN fitted to Image 1 from the Kodak dataset. On the left is the original image. In the center is the image reconstructed by SIREN. The residual image on the right is obtained by subtracting the reconstructed image from the original image. The PSNR of the reconstructed image is 22.45 dB for a model size of 0.6 bpp.

[0033] The Fourier method differs only in the fact that a layer is added after the image coordinates, which are mapped into a higher dimensional space using the known Fourier functions.

[0034] Multipatch Learning The INR is a function f parameterized by a neural network. θ We optimize the loss function in equation (1) over the entire image using . This implicitly captures redundancy within the image, but can better reduce redundancy when operating on local patches. Therefore, we optimize the loss function in equation (1) over the entire image using . This implicitly captures redundancy within the image, but can better reduce redundancy when operating on local patches.

[0035]

number

[0036]

number

[0037]

number

[0038] The same decomposition is applied to a set of frames to account for temporal redundancy. In the following, reference is made to image decomposition, but it should be understood that the method applies to any set of frames without loss of generality.

[0039] Image-specific multipatch INR

[0040]

number

[0041]

number

[0042] Figure 4 illustrates multi-patch learning. The input image is divided into patches, here six patches. All patches have the same shared network or shared head, and each patch has its own network. The tail network consists of patch-specific networks. The shared head learns a global representation of all patches, and the tail network adapts this global representation to each patch. An MPL network is trained on all patches in parallel, and the patches are merged to reconstruct the image.

[0043] The minimization of equation (2) is performed in two steps: (1) The weights of the head layer and all tail layers are trained together. (2) The weights in the head layer are frozen, and only the weights in the tail layer are optimized to adapt to the specific local content of each patch.

[0044] In this way, the first step ensures that the head layer captures the global information present in the image, while the second step ensures that the local content of each patch is adapted. The weights of the loss function are optimized using gradient descent or stochastic gradient descent. Once optimized, the weights are further coded in the bitstream using any entropy coding technique.

[0045] Image-independent multipatch INR In the previous section, each image has its own specific head, but here the heads are independent and common to all images. The weights are the head weights trained over a large set of patches: [P1,P2,...,P NLet ] be a large set of N diverse patches. The head layer is optimized with respect to the loss function

[0046]

number

[0047]

number

[0048] Once the head network weights are trained, for any given image to be encoded, we partition the image into local patches and add a tail network on top of the optimized head network (frozen) to adapt to the local content of the patches and optimize the tail network weights. The tail network weights can also be made sparsifiable by adding L1 regularization in the loss as in Equation (4). Because the head network is image-independent, it is known to both the encoding and decoding sides, and therefore only the tail network weights need to be transmitted.

[0049]

number

[0050] Without loss of generality, this applies to the temporal coding of video sequences, where the head is trained on the first frame of the sequence and used for the remaining frames.

[0051] Dictionary Learning Here, the goal is to learn a collection of head layers: D=[d1,d2,...d K ] is the dictionary to be learned, then each element d k is the weight of the head layer. After obtaining this dictionary, for a given image, select the optimal head layer from D.

[0052] The dictionary D is learned from a large set of image patches in two ways.

[0053] The patches are clustered into K clusters, and for each cluster, the weights of the head layer are learned. Therefore, each cluster can represent an image with different information content. Any existing clustering technique can be used for clustering. Clustering can be performed on image regions (pixels) or calculated on features extracted from an off-the-shelf deep neural network.

[0054] From the collection of head layers in the image-specific multi-patch INR description, we cluster the head layers and learn K elements of the dictionary.

[0055] During encoding, the selection of the head (dictionary) can be performed in several ways: In the first case, the selection of the head layer can be performed by calculating the distance between the image patch and the cluster center, and the cluster center with the smallest distance is selected as the head layer. The distance can be the mean square error.

[0056] The head layer that has the smallest loss between the reconstructed image and the original image may be selected as the head layer.

[0057] We train a classifier that takes the input as patches and outputs the index of the head to select.

[0058] In addition to the weights, we also code in the bitstream the index of the dictionary used as the head layer.

[0059] Weight Encoding As explained in the Multipatch INR section, once the weights are optimized, they are coded into the bitstream. To code the weights, the following scheme can be used: (1) Weights can be coded in half precision (16 bits) (2) The weights can be coded with 8-bit quantization with maximum absolute normalization. (3) The weights can be encoded using an explicit probability distribution by estimating the parameters of this distribution from the weights, e.g., a Gaussian distribution, where the mean and variance are estimated from the weights. (4) The weights can be 8-bit quantized coded using maximum absolute normalization and a boundary-aware entropy model. (5) Equations (2) and (4) can also include an entropy model, so that the weights are minimized using the entropy model, in which case equation (2) becomes:

[0060]

number

[0061]

number

[0062] During training, uniform noise is added to the weights to approximate the quantization error during test (encoding) time. During encoding, the weights are quantized to the nearest integer and coded into the bitstream by the learned CDF of p(.).

[0063] result Multipatch INR experiments on Kodak dataset images The multi-patch INR experiment was conducted using a FOUREN network, where the goal is to reconstruct an image with a small tail and transmit the tail weights to the receiver side. The head is already known by the encoder and decoder. The goal is to find the best network architecture for reconstructing an image with a 0.68 bpp tail.

[0064] Our multi-patch INR using FOUREN was implemented in PyTorch and all experiments were performed on a single Tesla M60 GPU with 8GB RAM. We used two steps to train our model. First, we trained the network across all patches, which resulted in a global representation across all patches. Then, we fine-tuned each patch individually to obtain a better representation for each patch.

[0065] The technique used for training is called freezing. In the context of neural networks, freezing a layer refers to controlling how weights are updated. A frozen layer means that its weights cannot be changed any more. In our case, during training, the head layers are frozen. The tail has the same number of sub-networks equal to the number of patches. To train a particular patch, we also freeze other sub-networks. This means that only the weights of the current patch are updated.

[0066] In the following experiments, the goal is to find the best architecture to transmit less information to the decoder. We use a tail size of 0.68 bpp and put all the network complexity in the head.

[0067] Multipatch INR applied to a set of patches We select 10 patches from several images in the Kodak dataset. We reconstruct these patches using a shared network and fine-tuned networks one by one. To train these networks, we introduce several parameters: a patch size of 64x64 pixels, a mapping size of 256. The networks are 2 -4 It has been trained for 15K iterations with a learning rate of

[0068] Figure 5 shows the ground truth patches and 10 patches of size 64 × 64px taken from several Kodak dataset images. (a): All original patches. (b): Patches fitted by the shared network. We set the layer width to 20 nodes and the learning rate to 2. -4 (c): The network was fine-tuned one by one, and for each patch, we set the learning rate to 0.5. * 2 -4 as 0.5 * The network was retrained for 15K iterations.

[0069] In Figure 5, it can be seen that fitting the patches to the same shared network introduces some artifacts into the patches, since the networks are learning to reconstruct the patches simultaneously using the same representation. Then, the weights in each tail are updated again, fine-tuned for each patch to obtain better patch reconstructions.

[0070] Table 1 shows further experimental results based on the network architecture. Each network has one tail layer of 0.68 bpp. By varying the number of head layers, we can observe some changes in network performance. If the head is too small or too large, the network performance is poor. We found that a head size of approximately 40-45 bpp yields a good PSNR average for 10 patches. The next step is to apply this method to the entire image from the Kodak dataset.

[0071] [Table 1]

[0072] Multipatch INR applied to the full image Following the same idea of ​​applying MPL to a set of patches, we apply the same method to images. Images are divided into patches of equal size. Experiments are performed with three different patch sizes: 64 * 64, 128 * 128 and 256 * The Kodak dataset images have the same size (768 x 512 pixels), giving 96 patches for the 64 patch size, 24 patches for the 128 patch size, and 6 patches for the 256 patch size. In theory, the network learns a global representation that matches all patches. It is desirable to keep the tail bpp as small as possible, and 0.68 bpp seems to be a good value for the tail. In these experiments, we reconstructed our images by varying the head size to find the best architecture for image representation.

[0073] Multi-patch INR at low head size Our first MPL network is built on a 64x64 patch size. This model has 10 head layers with a width of 64 layers and one tail layer with a width of 28 layers. The head size is 5.53 bpp and the tail size is 0.68 bpp. The Fourier feature mapping size is 256, and the learning rate is 2. -4 The network was trained for 10k iterations using the sparse network. The network was fed with reconstructed images from the shared network and fine-tuned reconstructed images for each patch. The results in Figures 6, 7, and 8 show that this network architecture does not perform well because the network trains on more than 96 patches, and the network size is too small to handle this. The PSNR value is below 30 dB, indicating that the head size needs to be increased, which means a larger patch size.

[0074] Figure 6 shows MPL fitted to Image 1 from the Kodak dataset. The network head size is 5.53 bpp and the tail size is 0.68 bpp. On the left is the original image. In the center is the image reconstructed by the shared network. On the right is the image reconstructed by fine-tuning the network patch by patch. The shared network reconstructed the image with a PSNR of 21.09 dB. Fine-tuning the network patch by patch yielded a PSNR of 21.10 dB.

[0075] Figure 7 shows MPL fitted on image 15 from the Kodak dataset. The network head size is 5.53 bpp and the tail size is 0.68 bpp. On the left is the original image. In the center is the image reconstructed by the shared network. On the right is the image reconstructed by fine-tuning the network patch by patch. The shared network reconstructed the image with a PSNR of 26.30 dB. Fine-tuning the network patch by patch yielded a PSNR of 26.33 dB.

[0076] Figure 8 shows MPL fitted to image 24 from the Kodak dataset. The network head size is 5.53 bpp and the tail size is 0.68 bpp. On the left is the original image. In the center is the image reconstructed by the shared network. On the right is the image reconstructed by fine-tuning the network patch by patch. The shared network reconstructed the image with a PSNR of 21.74 dB. Fine-tuning the network patch by patch yielded a PSNR of 21.75 dB.

[0077] Multi-patch INR on medium head size Our second MPL network is 128 * The model is built on a 128 patch size. The model has eight head layers with a width of 160 layers and one tail layer with a width of 110 layers. Here, the head size is 20.70 bpp and the tail size is 0.65 bpp. The Fourier feature mapping size is 256, and the learning rate is 2. -4The network was trained for 10k iterations as shown in Figure 6.9, Figure 6.10, and Figure 6.11. It can be seen that the PSNR value is about 30 dB, which is an acceptable quality reconstruction, with only 24 patches and a sufficiently large head size.

[0078] Figure 9 shows MPL fitted to Image 1 from the Kodak dataset. The network head size is 20.70 bpp and the tail size is 0.65 bpp. On the left is the original image. In the center is the image reconstructed by the shared network. On the right is the image reconstructed by fine-tuning the network patch by patch. The shared network reconstructed the image with a PSNR of 29.14 dB. Fine-tuning the network patch by patch yielded a PSNR of 29.22 dB.

[0079] Figure 10 shows MPL fitted on image 15 from the Kodak dataset. The network head size is 20.70 bpp and the tail size is 0.65 bpp. On the left is the original image. In the center is the image reconstructed by the shared network. On the right is the image reconstructed by fine-tuning the network patch by patch. The shared network reconstructed the image with a PSNR of 35.44 dB. Fine-tuning the network patch by patch yielded a PSNR of 35.59 dB.

[0080] Figure 11 shows MPL fitted to image 24 from the Kodak dataset. The network head size is 20.70 bpp and the tail size is 0.65 bpp. On the left is the original image. In the center is the image reconstructed by the shared network. On the right is the image reconstructed by fine-tuning the network patch by patch. The shared network reconstructed the image with a PSNR of 30.44 dB. Fine-tuning the network patch by patch yielded a PSNR of 30.61 dB.

[0081] Multi-patch INR for large head sizes For large head sizes, it is desirable to obtain near-perfect reconstruction of the image. The MPL network uses 256 * The model is built on a 256 patch size. The model has five head layers with a width of 512 layers and one tail layer with a width of 450 layers. Here, the head size is 104.29 bpp and the tail size is 0.66 bpp. The Fourier feature mapping size is 256, and the learning rate is 2. -4 The network was trained for 10k iterations with σ = 0. For larger head sizes, a PSNR value of approximately 40 dB was obtained, which is a perfect reconstruction quality of the image, as can be seen in Figures 6.12, 6.13, and 6.14.

[0082] Figure 12 shows MPL fitted to Image 1 from the Kodak dataset. The network head size is 104.29 bpp and the tail size is 0.66 bpp. On the left is the original image. In the center is the image reconstructed by the shared network. On the right is the image reconstructed by fine-tuning the network patch by patch. The shared network reconstructed the image with a PSNR of 40.12 dB. Fine-tuning the network patch by patch yielded a PSNR of 40.21 dB.

[0083] Figure 13 shows MPL fitted on image 15 from the Kodak dataset. The network head size is 104.29 bpp and the tail size is 0.66 bpp. On the left is the original image. In the center is the image reconstructed by the shared network. On the right is the image reconstructed by fine-tuning the network patch by patch. The shared network reconstructed the image with a PSNR of 40.95 dB. Fine-tuning the network patch by patch yielded a PSNR of 41.14 dB.

[0084] Figure 14 shows MPL fitted on image 24 from the Kodak dataset. The network head size is 104.29 bpp and the tail size is 0.66 bpp. On the left is the original image. In the center is the image reconstructed by the shared network. On the right is the image reconstructed by fine-tuning the network patch by patch. The shared network reconstructed the image with a PSNR of 39.73 dB. Fine-tuning the network patch by patch yielded a PSNR of 39.99 dB.

[0085] FIG. 15 illustrates one embodiment of a method 1500 for encoding video data. The method begins at start block 1501 and proceeds to block 1510 for partitioning at least a portion of a video image into patches. Control proceeds from block 1510 to block 1520 for determining at least one head network from global information of at least a portion of the video image. Control proceeds from block 1520 to block 1530 for determining multiple tail networks from patches of at least a portion of the video image. Control proceeds from block 1530 to block 1540 for determining weights corresponding to the head networks. Control proceeds from block 1540 to block 1550 for optimizing tail layer weights by minimizing an implicit neural representation of the patch through learning weights of at least one head network. Control proceeds from block 1550 to block 1560 for encoding the tail layer weights into the video data.

[0086] 16 illustrates one embodiment of a method 1600 for decoding video data. The method begins at start block 1601 and proceeds to block 1610 to analyze the video data for tail network weights for a patch of video data. Control proceeds to block 1620 to reconstruct the patch of video data using the weights with the optimized head network from block 1610.

[0087] 17 shows one embodiment of an apparatus 1700 for compressing, encoding, or decoding video using the methods described above. The apparatus includes a processor 1710, which may be interconnected through at least one port to a memory 1720. Both the processor 1710 and the memory 1720 may also have one or more additional interconnections to external connections.

[0088] The processor 1710 is also configured to insert or receive information in the bitstream and either compress, encode, or decode using the methods described above.

[0089] The embodiments described herein include various aspects, including tools, features, embodiments, models, techniques, and the like. Many of these aspects are described with specificity and, often in a manner that may sound limiting, at least to indicate their individual characteristics. However, this is for purposes of clarity of description and does not limit the applicability or scope of the aspects. In fact, all of the different aspects can be combined and substituted to provide further aspects. Furthermore, the aspects can be combined and substituted with aspects described in previous applications.

[0090] Aspects described and contemplated in this application can be implemented in many different forms. While Figures 18, 19, and 20 provide some embodiments, other embodiments are contemplated, and a discussion of Figures 18, 19, and 20 is not intended to limit the scope of implementations. At least one aspect generally relates to encoding and decoding video, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0091] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image," "picture," and "frame" may be used interchangeably. Typically, although not necessarily, the term "reconstructed" is used on the encoder side, and the terms "decoded" or "reconstructed" are used on the decoder side.

[0092] Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. The order and / or use of specific steps and / or actions may be varied or combined, unless a specific order of steps or actions is required for proper operation of the method. Additionally, terms such as “first,” “second,” and the like may be used in various embodiments to modify elements, components, steps, operations, etc., e.g., “first decode” and “second decode.” The use of such terms does not imply any ordering of the modified operations unless specifically required. Thus, in this example, the first decode need not be performed before the second decode, but may occur before, during, or during an overlapping period with the second decode.

[0093] Various methods and other aspects described herein can be used to modify modules, such as intra-prediction modules, entropy coding modules, and / or decoding modules (160, 360, 145, 330), of video encoder 100 and video decoder 200 as shown in Figures 18 and 19. Furthermore, aspects of the present disclosure are not limited to VVC or HEVC, but can also be applied to, for example, other standards and recommendations, whether existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically precluded, aspects described herein can be used alone or in combination.

[0094] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes and the described aspects are not limited to these specific values.

[0095] 18 illustrates an encoder 100. Variations of this encoder 100 are contemplated, but for clarity, the following describes the encoder 100 without describing all possible variations.

[0096] Before encoding, the video sequence may undergo pre-encoding processing (101), such as applying a color transformation to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components (e.g., using histogram equalization of one of the color components) to obtain a signal distribution that is more resilient to compression. Metadata can be associated with the pre-processing and added to the bitstream.

[0097] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (102) into units, e.g., CUs, for processing. Each unit is coded, e.g., using either intra mode or inter mode. When a unit is coded in intra mode, intra prediction (160) is performed. In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder determines (105) whether intra mode or inter mode should be used to code the unit, and indicates the intra or inter determination, e.g., by a prediction mode flag. A prediction residual is calculated (110), e.g., by subtracting the predicted block from the original image block.

[0098] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying the transform or quantization processes.

[0099] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150), and the prediction residual is decoded. The decoded prediction residual is combined (155) with the predicted block to reconstruct an image block. An in-loop filter (165) is applied to the reconstructed picture, for example, to perform deblocking / sample adaptive offset (SAO) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0100] Figure 19 illustrates a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by decoder elements, as described below. The video decoder 200 generally performs a decoding pass in the reverse direction from the encoding pass described in Figure 18. The encoder 100 also generally performs video decoding as part of encoding the video data.

[0101] In particular, the decoder's input includes a video bitstream, which may be generated by video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can partition the picture according to the decoded picture's partition information (235). The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined with a predicted block (255) to reconstruct an image block. The predicted block may be obtained from intra prediction (260) or motion-compensated prediction (i.e., inter prediction) (275) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0102] The decoded picture may further undergo post-decoding processing (285), such as an inverse color transform (e.g., YcbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0103] FIG. 20 illustrates a block diagram of an example system in which various aspects and embodiments can be implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, Internet appliances, and servers. Elements of system 1000, singly or in combination, may be embodied in a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or separate components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.

[0104] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein, for example, to implement various aspects described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes storage device(s) 1040, which may include non-volatile and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drives, and / or optical disk drives. Storage device(s) 1040 may include, by way of non-limiting example, internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0105] System 1000 includes an encoder / decoder module 1030 configured to process data to provide, for example, encoded or decoded video, which may include its own processor and memory. Encoder / decoder module 1030 represents a module that may be included in a device for performing encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, encoder / decoder module 1030 may be implemented as a separate element of system 1000 or may be incorporated within processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0106] Program code to be loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently read into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and computational logic.

[0107] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store, for example, the television's operating system. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET).

[0108] Inputs to the elements of system 1000 can be provided through various input devices as indicated in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted throughout a broadcast by a broadcaster, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Although not shown in FIG. 20, other embodiments include composite video.

[0109] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or bandlimiting a signal to a band of frequencies), (ii) downconverting the selected signal, (iii) bandlimiting again to a narrower band of frequencies to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulating the downconverted, bandlimited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, downconverting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and refiltering to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, omit some of these elements, and / or add other elements that perform similar or different functions. Adding elements can include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0110] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or within processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, as desired, in a separate interface IC or within processor 1010. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, which work in cooperation with memory and storage elements to process the data streams as needed for presentation on an output device.

[0111] The various elements of system 1000 may be provided within an integrated housing in which the various elements may be interconnected and transmit data therebetween using suitable connection arrangements, e.g., internal buses known in the art, including the Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0112] System 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented in a wired medium and / or a wireless medium, for example.

[0113] In various embodiments, data is streamed or otherwise provided to system 1000 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received via communication channel 1060 and communication interface 1050 adapted for Wi-Fi communication. Typically, communication channel 1060 in such embodiments is connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, streamed data is provided to system 1000 using a set-top box that delivers data via an HDMI connection in input block 1130. Still other embodiments provide streamed data to system 1000 using an RF connection in input block 1130. As noted above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0114] System 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be for a television, a tablet, a laptop, a mobile phone, or another device. Display 1100 can also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop). Other peripheral devices 1120, in various example embodiments, include one or more of a standalone digital video disc (or digital versatile disc) (both terms referred to as a DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 to provide functionality based on the output of system 1000. For example, a disc player performs the function of playing the output of the system 1000 .

[0115] In various embodiments, control signals are communicated between system 1000 and display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that allow control between devices with or without user intervention. Output devices can be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speakers 1110 may be integrated into a single unit with other components of system 1000 in an electronic device such as a television. In various embodiments, display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0116] Alternatively, the display 1100 and speakers 1110 may be separate from one or more of the other components, for example, if the RF portion of the input 1130 is part of a separate set-top box. In various embodiments in which the display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0117] The embodiments may be implemented by computer software implemented by the processor 1010, by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type appropriate to the technology environment and may be implemented using any suitable data storage technology, including, by way of non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, embedded memory, and removable memory. The processor 1010 may be of any type appropriate to the technology environment and may include, by way of non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a multi-core architecture-based processor.

[0118] Various implementations involve decoding. As used herein, "decoding" can encompass all or part of the processing performed on a received encoded sequence to generate a final output suitable for display, for example. In various embodiments, such processes include one or more of the processing typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by decoders in various implementations described herein.

[0119] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0120] Various implementations involve encoding. As can be seen from the above discussion of "decoding," "encoding," as used herein, can encompass all or part of the processing performed on an input video sequence to, for example, generate an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as partitioning, differential encoding, transforming, quantizing, and entropy coding. In various embodiments, such processes also, or alternatively, include processes performed by the encoders of the various implementations described herein.

[0121] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to the broader encoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0122] It should be noted that the syntax elements used herein are descriptive terms and therefore do not preclude the use of other syntax element names.

[0123] Where a figure is presented as a flow diagram, it should be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it should be understood that the figure also provides a flow diagram of the corresponding method / process.

[0124] Various embodiments may refer to parametric models or rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is usually considered, often due to computational complexity constraints. This can be measured through a Rate Distortion Optimization (RDO) metric, or through Least Mean Square (LMS), Mean of Absolute Error (MAE), or other such measures. Rate-distortion optimization is usually formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on an extensive examination of all encoding options, including all considered modes or coding parameter values, but with a thorough evaluation of their coding costs and the associated distortion of the reconstructed signal after coding and decoding. To reduce encoding complexity, more rapid approaches may also be used, particularly with an approximate distortion calculation based on a predicted or predicted residual signal rather than the reconstructed one. A mixture of these two approaches can also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for others. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform optimization, but the optimization is not necessarily a complete evaluation of both the coding cost and the associated distortion.

[0125] Implementations and aspects described herein may be implemented as, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed only in the context of a single implementation form (e.g., discussed only as a method), the implementation of the discussed feature may also be embodied in other forms (e.g., an apparatus or a program). For example, an apparatus may be implemented in appropriate hardware, software, and firmware. A method may be implemented, for example, in a processor, which refers to a general processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include, for example, communication devices such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0126] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment," or "in one implementation" or "in an implementation" in various places throughout this application, as well as other variations, are not necessarily all referring to the same embodiment.

[0127] Additionally, the application may refer to "determining" various information. Determining / determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0128] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0129] Additionally, the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" typically involves in some manner an operation such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0130] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded to include as many items as are listed, as would be apparent to one skilled in this and related arts.

[0131] Also, as used herein, the term "signal" specifically refers to indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a specific one of multiple transforms, coding modes, or flags. Thus, in some embodiments, the same transform, parameter, or mode is used at both the encoder and decoder sides. Thus, for example, an encoder can transmit a specific parameter to a decoder so that the decoder can use the same specific parameter (explicit signaling). Conversely, if the decoder already has the specific parameter as well as other parameters, signaling without transmission can be used to simply allow the decoder to know and select the specific parameter (implicit signaling). By avoiding transmitting any actual function, bit savings are realized in various embodiments. It will be understood that signaling can be achieved in a variety of ways. For example, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder in various embodiments. While the above relates to the verb form of the term "signal," the term "signal" can also be used as a noun herein.

[0132] As will be apparent to those skilled in the art, implementations can generate a variety of signals formatted to carry information that can be stored or transmitted, for example. Information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. For example, such a signal can be formatted as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored on a processor-readable medium.

[0133] The preceding sections describe several embodiments across various claim categories and types. Features of these embodiments may be provided alone or in any combination. Furthermore, embodiments may include one or more of the following features, devices, or aspects across various claim categories and types, alone or in combination:

[0134] At least one embodiment involves encoding and decoding video information using neural networks.

[0135] At least one embodiment includes determining weights corresponding to a tail network for the video image.

[0136] At least one embodiment includes using weights in conjunction with a head network for compression / decompression of video images.

[0137] At least one embodiment includes building a head layer dictionary for a neural network.

[0138] At least one embodiment involves clustering patches to learn head layer weights for each cluster, which can be performed in the image domain or on features extracted from a deep neural network.

[0139] At least one embodiment includes a bitstream or signal that includes one or more of the described syntax elements or variations thereof.

[0140] At least one embodiment includes a bitstream or signal including syntax conveying information generated according to any of the described embodiments.

[0141] At least one embodiment includes creating, and / or transmitting, and / or receiving, and / or decoding according to any of the described embodiments.

[0142] At least one embodiment involves analyzing the video data or bitstream to determine the operating point of the codec.

[0143] At least one embodiment includes a method, process, apparatus, instruction storage medium, data storage medium, or signal according to any of the described embodiments.

[0144] At least one embodiment involves inserting syntax elements into the signaling that allow the decoder to determine the decoding information in a manner that corresponds to the manner used by the encoder.

[0145] At least one embodiment includes creating, transmitting, receiving, and / or decoding a bitstream or signal that includes one or more of the described syntax elements, or variations thereof.

[0146] At least one embodiment includes a television, set-top box, mobile phone, tablet, or other electronic device that implements the conversion method according to any of the described embodiments.

[0147] At least one embodiment includes a television, set-top box, mobile phone, tablet, or other electronic device that performs the conversion method determination according to any of the described embodiments and displays the final image (e.g., using a monitor, screen, or other type of display).

[0148] At least one embodiment includes a television, set-top box, mobile phone, tablet, or other electronic device that selects, band-limits, or tunes a channel (e.g., using a tuner) to receive a signal containing the encoded image and performs a conversion method according to any of the described embodiments.

[0149] At least one embodiment includes a TV, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing the encoded image wirelessly (e.g., using an antenna) and performs the conversion method.

Claims

1. 1. A method comprising: Segmenting at least a portion of a video image into patches; determining at least one head network from global information of said at least a portion of said video image; determining a plurality of tail networks from the patches of the at least a portion of the video image; determining a weight corresponding to the head network; optimizing the weights of the tail layer by minimizing the implicit neural representation of the patch through training the weights of the at least one head network; encoding the tail layer weights into video data.

2. 1. An apparatus comprising: Memory and a processor, the processor Segmenting at least a portion of a video image into patches; determining at least one head network from global information of said at least a portion of said video image; determining a plurality of tail networks from the patches of the at least a portion of the video image; determining a weight corresponding to the head network; optimizing the weights of the tail layer by minimizing the implicit neural representation of the patch through training the weights of the at least one head network; encoding the tail layer weights into video data.

3. 1. A method comprising: analyzing the video data for tail network weights for patches of the video data; and reconstructing the patch of video data using the weights in conjunction with an optimized head network.

4. 1. An apparatus comprising: Memory and a processor, the processor analyzing the video data for tail network weights for patches of the video data; and reconstructing the patch of video data using the weights in conjunction with an optimized head network.

5. 5. The method of claim 1 or 3, or the apparatus of claim 2 or 4, wherein the head network and the tail network are part of a neural network.

6. 6. The method of claim 1, 3 or 5 or the apparatus of claim 2, 4 or 5, wherein minimization of a loss function of an implicit neural representation is used.

7. 7. A method according to claim 1 or 3 or 5 or 6, or an apparatus according to claim 2 or 4 or 5 or 6, wherein the portion of video data is part of a sequence of frames.

8. 8. The method according to claim 1 or any one of claims 5 to 7, or the apparatus according to claim 2 or any one of claims 5 to 7, wherein the head network is common to all images.

9. The method of any one of claims 1 or 5 to 8 or the apparatus of any one of claims 2 or 5 to 8, wherein the head network is independent of the video image.

10. 10. A method according to claim 1 or any one of claims 5 to 9, or an apparatus according to claim 2 or any one of claims 5 to 9, wherein the head network is trained on a first frame of a sequence of video data and used for subsequent video data.

11. A device, A device according to any one of claims 4 or 5 to 10; (i) an antenna configured to receive a signal including the video block; (ii) a band limiter configured to limit the received signal to a band of frequencies including the video block; and (iii) a display configured to display an output representing the video block.

12. A non-transitory computer readable medium containing data content generated according to the method of any one of claims 1 or 5 to 10 or by the apparatus of any one of claims 2 or 5 to 10 or 11 for playback using a processor.

13. A signal comprising video data produced according to the method of any one of claims 1 or 5 to 10 or by the apparatus of any one of claims 2 or 5 to 11 for playback using a processor.

14. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 or 3 or 5 to 10.

15. A non-transitory computer readable medium containing data content, the data content including instructions for carrying out the method of any one of claims 1 or 3 and 5-10.

Images