Video signal encoding and decoding

Abstract

A method for encoding a video signal includes receiving encoding instructions for encoding the video signal and encoding the video signal using a training loop for optimizing a metric of the encoded video signal, the training loop including a loss function and one or more of a neural encoder and a neural decoder, and the training loop is adapted based on the encoding instructions.

Description

[Technical Field]

[0001] The present invention relates to the field of video coding, and in particular to the field of layered video encoding. [Background technology]

[0002] Generally speaking, videos are captured for a variety of uses, such as human analysis, machine analysis, archiving, entertainment, security, or legal issues. These uses have different requirements related to distortion, realism, number of viewpoints, immersion level, bit rate, error tolerance, etc. In addition, videos are played on different classes of devices with different capabilities. Therefore, video coding methods that incorporate multiple uses are being sought. Summary of the Invention [Problem to be solved by the invention]

[0003] One such method is to have a separate bitstream for each application, but having a separate video bitstream for each application is expensive in terms of required encoder hardware, power consumption, and bitrate.

[0004] Another such approach is to compromise multiple uses with a single bitstream (e.g., from increasing distortion to lower bitrates), but compromising multiple uses with a single bitstream often leads to unsatisfactory results.

[0005] Therefore, there is a need for improved video coding processes that incorporate multiple applications.

[0006] "Layered Image Compression Using Scalable Auto-Encoder" by Jia Chuanmin et al. describes a novel convolutional neural network-based image compression framework via a scalable auto-encoder (SAE), which consists of multiple hierarchical coding layers, each of which is an end-to-end optimized auto-encoder. [Means for solving the problem]

[0007] The invention is defined by the claims.

[0008] According to an example embodiment of the present invention, there is provided a method for encoding a video signal, the method comprising: receiving an encoding instruction for encoding a video signal; encoding a video signal using a training loop for optimizing a metric of the encoded video signal, the training loop including a loss function and one or more of a neural encoder and a neural decoder; and adapting the training loop based on the encoded instructions.

[0009] The training loop may include a loss function and a neural decoder. The training loop may also include a neural encoder.

[0010] By receiving the encoding instructions, the encoder can optimize the metrics as required by the encoding instructions, thus allowing the client to request the manner (and therefore the application) in which the video signal is encoded.

[0011] Neural coding involves applying a neural encoder to a layer to obtain a latent representation (i.e., the encoded layer) and applying a neural decoder to the latent representation to obtain a reconstruction of the layer.

[0012] In neural coding, a loss function is used in the loss loop of the reconstruction to try to optimize a metric (e.g., distortion, rate, quality, etc.). In other words, the loss function approximates the optimal decoding coefficients of the neural decoder (i.e., the decoding coefficients that try to optimize the metric).

[0013] It should be noted that "optimizing a metric" is understood to mean "attempting to optimize a metric," since optimization is generally limited by time and processing resources.

[0014] In the case of neural coding, the decoding parameters generally contain information necessary for a neural decoder to decode latent representations that are not yet available to the decoder.

[0015] Encoding using neural coding is based on applying a neural encoder to a layer to obtain an encoded layer, applying a neural decoder to the encoded layer to obtain a reconstruction, and applying a loss function to the reconstruction, optionally based on the original layer (e.g., for distortion).

[0016] The encoding instructions may include a metric for the training loop to optimize. The particular metric used at each layer will, of course, depend on the particular use case and which optimized metric is desired at each layer.

[0017] Thus, when a client encodes a video signal, it can specify the metric to optimize, rather than relying on the encoding scheme used by the encoder.

[0018] The method may further include adapting the loss function based on a metric specified in the encoding instructions or selecting the loss function based on the metric.

[0019] The encoding instructions may include structural information for the neural decoder and / or neural encoder, which structural information represents a selection between possible configurations of the neural decoder and / or neural encoder.

[0020] Specifying structural information of the neural encoder / decoder may indirectly affect the metrics of interest. For example, adapting the structure of the neural encoder may affect the bitrate of the video signal (e.g., by reducing the dimensionality of the latent representation (i.e., the encoded video signal)).

[0021] The training loop may include a neural decoder, and the encoding instructions include decoder coefficients for the neural decoder.

[0022] The received decoder coefficients may correspond to a metric of interest.

[0023] The training loop may include a neural encoder, and the encoding instructions may include encoder coefficients of the neural encoder.

[0024] The received encoder coefficients may correspond to a metric of interest.

[0025] The encoding instructions may include tunable control parameters for the loss function.

[0026] The received tunable control parameter may correspond to a metric of interest.

[0027] The metric may include one of a realism measure, a distortion measure, a color accuracy measure, a rate measure, or an object detection accuracy measure.

[0028] The video signal may be a layered video signal having a first layer and a second layer, where the second layer is dependent on the first layer such that the second layer is decodable only after a reference frame in the first layer is decoded, and the step of encoding the video signal includes encoding the first layer and the second layer, where at least one of the first layer and the second layer is encoded using neural coding, the first layer being encoded using a first loss function to optimize a first metric, and the second layer being encoded using a second, different loss function to optimize a second metric.

[0029] The present invention also provides a method for decoding an encoded video signal, the method comprising: sending encoding instructions to an encoder; receiving an encoded video signal that is encoded based on encoding instructions and a training loop that includes a neural decoder; receiving decoding parameters for decoding the encoded video signal; and decoding the encoded video signal using the neural decoder and the decoding parameters.

[0030] The decoding parameters may include decoding coefficients used by a decoder (e.g., a neural decoder) to decode the encoded first layer, coding coefficients used by an encoder (e.g., a neural encoder) to code the first layer, a reconstruction of the first layer, and / or the encoded first layer (e.g., a latent representation of the first layer).

[0031] The encoding instructions may be based on the user's viewing position and / or viewing direction.

[0032] The encoding instructions may be based on a user zooming in or out.

[0033] The step of transmitting the encoding instructions may be triggered by one or more of a change in viewing position, a change in viewing direction, a user zooming in, and a user zooming out.

[0034] The encoding instructions may include structural information for the neural decoder and / or neural encoder in the training loop, the structural information representing a selection between possible configurations of the neural decoder and / or neural encoder, and one or more of a metric for which the training loop is to optimize, coding coefficients of the neural encoder, decoder coefficients of the neural decoder, and tunable control parameters for a loss function in the training loop.

[0035] The present invention also provides a computer program medium comprising computer program code which, when executed on a processing system, causes the processing system to perform all the steps of any of the methods described above.

[0036] The present invention also provides a processing system for reading computer program code in a computer program medium and executing the computer program code.

[0037] Also described herein is a method for encoding a layered video signal having a first layer and a second layer, where the second layer is dependent on the first layer. The method includes: encoding the first layer and the second layer, At least one of the first layer and the second layer is encoded using neural coding; an encoding step, wherein a first layer is encoded using a first loss function to optimize a first metric and a second layer is encoded using a second, different loss function to optimize a second metric; obtaining decoding parameters for decoding the first layer and the second layer; transmitting and / or storing the coded base layer and the coded enhancement layer together with the decoding parameters.

[0038] This allows for the transmission or storage of video (e.g., in a bitstream) that is essentially optimized for two different metrics (by two different loss functions) upon playback, allowing the user to decide how they want to view the video.

[0039] In the first example, the first layer is optimized for rate (bitrate) and the second layer is optimized for color accuracy, so the client can decide whether to stream the video at a lower bitrate (e.g., by decoding only the first layer) or to stream a more color accurate version of the video (possibly at a higher bitrate).

[0040] In a second example, the first layer is optimized for low distortion in detected objects, and the second layer is optimized for low distortion throughout the video. In this case, an object detection function is used in the first layer, and the detected objects are optimized for distortion, while the rest of the video is optimized for realism (which typically requires a lower bitrate). The object detection function and distortion function are part of the loss network in the first layer.

[0041] The first metric and the second metric may include one of a realism measure, a distortion measure, a color accuracy measure, a rate measure, or an object detection accuracy measure.

[0042] In one example, only the second layer has realism and distortion measures as metrics to be optimized, while the first layer has distortion measures but not realism measures to be optimized. Thus, the first layer can be considered to provide an acceptable image, and a second layer can be added if desired to provide more realistic image quality.

[0043] In another example, only the first layer has the realism measure and the distortion measure as metrics to be optimized, while the second layer has the distortion measure but not the realism measure to be optimized.

[0044] It will be appreciated that other metrics can be used (e.g., color accuracy, object detection or a less complex derivative thereof, a rate measure of one layer, a combined rate, etc.) The particular metric used at each layer will, of course, depend on the particular use case and which optimized metric is desired at each layer.

[0045] The color accuracy measure indicates the color accuracy of the reconstruction relative to the original layer. The object detection accuracy measure indicates the ability of the object identification algorithm to identify the same object in the reconstruction relative to the original layer.

[0046] The second layer encoding may be based on the first layer decoding parameters.

[0047] The decoding parameters may include decoding coefficients used by a decoder (e.g., a neural decoder) to decode the encoded first layer, coding coefficients used by an encoder (e.g., a neural encoder) to code the first layer, a reconstruction of the first layer, and / or the encoded first layer (e.g., a latent representation of the first layer).

[0048] At least one layer coded using neural coding may be further coded using a lossy network to optimize two or more metrics based on tunable control parameters.

[0049] For example, a tunable control parameter may be used to implement a trade-off between two or more metrics. In another example, a tunable control parameter may implement a limit on one or more metrics (e.g., a rate limit).

[0050] Both the first and second layers may be coded using neural coding.

[0051] The first and second layers may be further encoded using a first and second loss network, respectively, where one of the first or second loss network is for optimizing the distortion measure and the realism measure, and the other of the first and second loss network is for optimizing the distortion measure but not the realism measure.

[0052] For at least one layer that is coded using neural coding, the neural encoder and / or the neural decoder used for the neural coding may be retrained using the corresponding layer.

[0053] The method further includes receiving encoding instructions for the first layer and / or the second layer from the client, and adapting the encoding of the first layer and / or the second layer according to the encoding instructions.

[0054] For example, the encoding instructions include one or more of loss parameters for adapting the first and / or second loss functions, network parameters for adapting a neural decoder of the neural coding, and encoding parameters for adapting a neural encoder of the neural coding.

[0055] The encoding instructions may include one or more of structural information of a neural decoder and / or a neural encoder of the neural coding, one or more metrics for one of the first loss function and / or the second loss function to optimize, coding coefficients of the neural encoder, decoder coefficients of the neural decoder, and tunable control parameters for the loss functions.

[0056] Also described herein is a method for decoding an encoded layered video signal having an encoded first layer and an encoded second layer, where the second layer is dependent on the first layer. The method includes: receiving decoding parameters for a first layer and a second layer; decoding the encoded first layer using a first decoder function with decoding parameters for the first layer; and decoding the encoded second layer using a second, different decoder function using decoding parameters for the second layer; Either the first decoder function or the second decoder function decodes the corresponding layer using a neural decoder.

[0057] The step of decoding the encoded second layer using the second decoder function may be further based on decoding parameters for the first layer.

[0058] In one example, if the second layer is of a different modality than the first layer (e.g., a depth map, reflectance map, etc. versus an image), it is advantageous to use the decoded coefficients of the first layer rather than the coded base layer when decoding the second layer.

[0059] In some cases, both the first and second decoder functions are neural decoders, and the decoding parameters include the decoding coefficients of the neural decoders, such that the decoding coefficients of the first neural decoder (in the first layer) are used by the second neural decoder (in the second layer) to decode the second layer.

[0060] Alternatively, the first decoder function may be a conventional (non-neural) decoding process, and the decoding parameters may include motion vectors, coding trees, predictor selection, etc. of the first decoding process.

[0061] The decoding method may further include sending encoding instructions to an encoder and receiving the encoded first layer or the encoded second layer encoded according to the encoding instructions.

[0062] The method may further include determining a user's viewing characteristics (e.g., viewing position, viewing direction, zoom, etc.), determining a loss network (preferably in which the first layer or the second layer is encoded) from the viewing characteristics, and transmitting the determined loss network to the encoder, which may be updated when the viewing characteristics change.

[0063] The method may further include determining a viewing characteristic of the user and determining whether to decode the second layer based on the viewing characteristic.

[0064] The second layer may not be decoded, so the first layer may be the only layer output for video playback.

[0065] Both the first decoder function and the second decoder function may use a neural decoder to decode the corresponding layer.

[0066] Also provided herein is a computer program medium containing computer program code that, when executed on a processing system, causes the processing system to perform all of the steps of any of the methods described herein.

[0067] Also provided herein is a processing system that reads the computer program code in the computer program medium and executes the computer program code.

[0068] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief explanation of the drawings]

[0069] For a better understanding of the present invention and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which:

[0070] [Figure 1] FIG. 1 shows an example of a functional configuration of a neural network. [Figure 2] Figure 2 shows the neural network of Figure 1 split into two smaller networks. [Figure 3] Figure 3 shows the two layers of abstraction of the neural network of Figure 2. [Figure 4] Figure 4 shows the neural network being optimized. [Figure 5] Figure 5 shows the inputs being coded into the neural network. [Figure 6] FIG. 6 illustrates the use of a neural codec and lossy network using rate-distortion optimization to code the input. [Figure 7] FIG. 7 shows a lossy network with rate control. [Figure 8] FIG. 8 shows an example of a neural post-processor in video coding. [Figure 9] FIG. 9 shows layered video in a prediction structure. [Figure 10] FIG. 10 shows a neural post-processor applied to layered video coding. [Figure 11] FIG. 11 shows a neural codec structure for layered video. [Figure 12] FIG. 12 shows a neural codec structure for encoding reference and dependent frames. [Figure 13] FIG. 13 shows a frame that is coded according to the coding instructions. DETAILED DESCRIPTION OF THE INVENTION

[0071] The present invention will now be described with reference to the drawings.

[0072] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the devices, systems, and methods, are for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems, and methods of the present invention will become better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the figures are schematic representations only and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the figures to indicate the same or similar parts.

[0073] The present invention provides a method for encoding a video signal, the method including receiving encoding instructions for encoding the video signal and encoding the video signal using a training loop for optimizing a metric of the encoded video signal, the training loop including a loss function and one or more of a neural encoder and a neural decoder, the training loop adapted based on the encoding instructions.

[0074] The present invention utilizes neural networks for video coding, and therefore the general concepts of neural networks and neural coding are briefly described below.

[0075] Neural Networks FIG. 1 shows an example of a functional configuration of a neural network 100. The neural network has coefficients [c1, c2, ..., c j ], functions [f1, f2, …, f i ], and the inputs [x1, x2, ..., x k ], intermediate output [w1, w2, ..., w f ], and output [y1, y2, ..., y m 1 shows two inputs [x1, x2], eight functions [f1, f2, ..., f8] (with corresponding coefficients and weights), and two outputs [y1, y2] to neural network 100.

[0076] In this functional model of this neural network, each function has at least one input (either an input to the neural network 100 or an intermediate output from another function), zero or more coefficients, and one output. All variables are numbers or compositions of numbers. By applying composition rules, including the chain rule, the entire neural network 100 becomes automatically differentiable. Functions are often called nodes or neurons. Coefficients are often called weights.

[0077] The differentiable nature of the neural network 100 is used to optimize the coefficients. Optimizing the coefficients is often called training or learning. Typical variable types for inputs, outputs, and coefficients are scalars, tuples, vectors, matrices, n-dimensional matrices (called tensors), or any combination of the above data types.

[0078] Weight sharing is when two functions share a coefficient. For example, in Figure 1, f2 and f8 both use coefficient c2. Some functions do not have a coefficient. For example, in Figure 1, if w1 is a pair of values, f6 is the "primary" function that selects the first value and ignores the second: f6(a,b) → a

[0079] Figure 2 shows the neural network 100 of Figure 1 split into two smaller networks. Neural network 100 can be split into smaller networks by taking a full cut. In Figure 2, neural network 100 is split into two layers: layer 202 (left) and layer 204 (right).

[0080] Typically, a neural network is abstracted into a differentiable function F with coefficients c, input x, intermediate output w, and output y. Having no coefficients or intermediate outputs is equivalent to having an empty tuple as a data type, and having multiples is equivalent to having a composite data type.

[0081] The concept of abstracting neural networks can be extended to the partitioned network shown in Figure 2.

[0082] Figure 3 illustrates two layers of abstraction for the neural network 100 of Figure 2. That is, Figure 3 illustrates the neural network F=f A 〇f B 2 shows the two neural network layers 202 and 204 as follows. Most of the variables mentioned above are now tuples: c A =(c1,c2,c3) c B =(c2,c4) w A =w1 w B =(w2,w3,w4) y A =(w2,w3,w4)

[0083] Here, the output y of the first layer 302 is A contains a subset of the intermediate outputs (w2, w3, w4) of the entire neural network.

[0084] Neural networks often have millions of nodes based on a relatively small dictionary of functions. For a neural network to exhibit any meaningful behavior, it often needs a way to combine multiple inputs into a single output, which can be as simple as a linear combination with a bias. For example,

number

[0085] where:

number

[0086] In addition, one or more nonlinear functions are often required. The nonlinear functions can be as simple as the normalized linear unit (RLU): RLU:x→max(0,x) where:

number

[0087] Note that all of the above functions are differentiable.

[0088] A commonly used structure is the convolutional neural network (CNN), which in its simplest form has three types of layers: convolutional layers, nonlinear layers, and optional pooling layers. Convolutional layers are a generalization of convolutional image filters (FIR). The inputs and outputs of such layers are tensors, the domain of the function is local (e.g., 5x5 samples), and the weights are often shared. Nonlinear layers are element-wise applications of the RLU operation. Pooling layers fold or subsample intermediate outputs to reduce dimensionality.

[0089] Figure 4 shows a neural network f being optimized. To optimize the neural network f, we enter the neural network f into a loop with a loss function L and an optimizer Opt. The loss function is the ratio of the neural network output y to a criterion y ref Often, the criterion refers to the expected or desired outcome of a neural network. A loss function has a scalar output e (in terms of error, loss or cost), and the task of the optimizer is to find the optimal set of coefficients c opt For example, this can be achieved as follows:

number

[0090] This is generally the derivative ∂y i / ∂c j This is achieved by gradient descent based on the A well-known optimization method is stochastic gradient descent, which uses a pseudo-random subset of the dataset at each training iteration (epoch).

[0091] Neural Coding Coding a neural network requires training and encoding the neural network, transmitting it over a channel, and decoding the neural network. A decoder (or related module) can then apply the neural network to new data.

[0092] Transmission typically involves a protocol that specifies what is to be transmitted and how, but also what information is shared in advance between the encoder and decoder.

[0093] Neural network function f(x) → x a If the decoder knows the value of a before decoding the bitstream, a is called a constant; otherwise, a value must be transmitted. If a must be transmitted, it is called a coefficient.

[0094] The encoder starts with an existing network and optimizes (trains) only a relatively small number of coefficients, and the decoder receives occasional refreshes (e.g., at random access points) with the values ​​of all coefficients. Alternatively, there is a side channel for transmitting the initial network, e.g., a uniform resource locator (URL) from which the network can be downloaded.

[0095] Often the coefficients are floating-point numbers (real numbers up to machine precision). However, they can also be 8-bit or 16-bit integers. If the coefficients are floating-point numbers, they can be quantized for coding. Well-known coding techniques such as context-adaptive binary arithmetic coding (CABAC) are suitable for coding coefficients.

[0096] Structural information of the neural network is also relevant. Consider a neural network function that selects function j or k depending on the value s∈{0,1}:

number

[0097] The function either is differentiable with respect to x but not with respect to s. Therefore, s is not a coefficient. Instead, the value s represents a choice between two possible neural network configurations. In other words, s provides structural information about the neural network. The value of s can be transmitted to share the selected network with the decoder.

[0098] In general, we have s∈S, where S is the set of allowed neural network configurations. Since there is a discrete set of possibilities, s can be encoded using a bit pattern, e.g., it can be encoded using CABAC.

[0099] If structural information is transmitted, the neural network is said to have a parametric structure: for example, the size of some layers of the neural network can be adjusted depending on the image resolution or to account for different bit rates.

[0100] Knowing how to transmit a neural network makes it (in principle) possible to use it to encode a video frame (or image).

[0101] Figure 5 shows an input x being coded by a neural network. The first step is to define a neural network that reconstructs the input (i.e., a reconstruction network). In other words, the input to the neural network is x, the output of the neural network is x', and the goal is for x' to be as close to x as possible. The neural network is divided into a neural encoder f and a neural decoder g.

[0102] A neural encoder f converts an input x into a latent representation y, which is fed into a neural decoder g, which then outputs a reconstructed signal x'. A loss function L compares the original input x with the reconstruction x'. The error e output from the loss function is as simple as the root mean square error (RMSE) between x and x'. The idea here is that the decoder has access to the decoder network g.

[0103] The problem with this is that the only way to constrain the bitrate is to reduce the dimension of y relative to the input x. In many cases, this is a suboptimal solution. Note that references to dimension here refer to the number of values ​​(e.g., the number of scalar values ​​in a vector / matrix / tensor). For example, a matrix of NxM values ​​has dimension NM.

[0104] Therefore, the second step is to add a loss function based on rate-distortion optimization.

[0105] 6 illustrates the use of a neural codec 602 and loss network 604 using rate-distortion optimization to code an input x, where a function Q in the loss network 604 determines the distortion measure q, and a function R determines the rate measure r (estimated entropy) of the latent representation y. Furthermore, a variable parameter λ implements a tradeoff between the rate measure r and the distortion measure q in the loss function L. For example: L:(q,r)→q+λr

[0106] Note that for simplicity, the loss network 604 is defined (herein) as the configuration of functions sandwiched between the neural codec 602 (including the neural encoder f and the neural decoder g) and the optimizer Opt. For example, in Figure 6, the loss network 604 is composed of the functions Q, R, and L.

[0107] Similarly, variable control parameters are defined as parameters, such as λ, that enter into the functions of the loss network 604. The variable control parameters themselves are not optimized in the training loop.

[0108] Figure 7 shows a lossy network 700 with rate control. In practice, the target rate r that must not be exceeded is ref In this case, we can incorporate this into the loss network 700 and start with a low value of λ, gradually increasing it each epoch: L:(q,r)→q+λmax(0,rr ref )

[0109] If the target rate must not be exceeded, a rate control function RC is added to the loss network 700 to control the rate r to the target rate r ref We can automatically find a value for λ that keeps r ref replaces λ as the variable control parameter.

[0110] It is recognized that there are various options for training a neural codec. The first option is to train the neural codec on a large collection of image data, where information about image x is encoded in a latent space y. This concept is best matched with non-neural video codecs.

[0111] The second option is to pre-train the neural codec on a large collection of image data and then further train it on the video being encoded. The video is then used to train the neural codec on the latent space y and the decoder coefficients c g is encoded as

[0112] The third option is to use the decoder coefficients c g The goal is to encode all the information in , where the neural encoder f is replaced by a constant or random function. This best matches an autoregressive signal model. Essentially, the video is fully encoded in the decoder network, and the input signal adds image noise (e.g., for realism).

[0113] Neural Post Processor Current 2D video codecs are the result of decades of work, and dedicated hardware is ubiquitous. It may be a while before neural video codecs can compete in terms of rate and distortion (i.e., the difference between the original input video and the reconstruction), also taking power consumption into account.

[0114] On the other hand, practical neural implementations of frame interpolation (e.g., temporal upsampling), super-resolution (e.g., resolution upsampling), and denoising already exist, and it is expected that the aforementioned neural codec concepts will be realized as neural post-processors in the near future.

[0115] Figure 8 shows an example of a neural post-processor 802 in video coding. In this approach, an input x is coded to produce a bitstream b and a reconstruction x * There exists a non-differentiable video encoder h that outputs the reconstruction x * is the decoded version of bitstream b.

[0116] Here, the neural post-processor 802 refers to the functions (f, g, R, Q, L, and Opt) used after the video encoder h. The neural post-processor 802 generates the reconstruction x * The neural encoder f and neural decoder g are used to filter the input x to generate a reconstruction x'. A distortion measure function Q compares the neural reconstruction x' with the input x to determine the distortion measure q. This scheme works because everything from the video encoder h onwards is differentiable; this property makes it a post-processor. In this case, function R determines the combined rate r of the bitstream b and the latent representation y. As before, the loss function L uses a variable control parameter λ to determine the trade-off between r and q to determine the error e. The error e is input to the optimizer Opt. The loss function L, together with the optimizer Opt, determines the neural encoder coefficients c. f and the neural decoder coefficients cg We attempt to optimize the metrics q and r by adapting

[0117] The decoder consists of the decoder part of h (not shown), the neural decoder g, and the receiver. As mentioned above, we denote the frames y, c g You can choose to encode the information in y in x, or both. * is the enhancement information (feature) for c g x * represents the coefficients of the decoder function g fitted to

[0118] Note that if y is not sent, the coefficient c (used by the neural encoder f) f must also be sent (unless it is empty). This is because the decoder * Decoding x (which is what a decoder does anyway) gives * The decoder then applies the encoder function f to the coefficients c f This is because we can apply it to get y.

[0119] Generative Adversarial Networks (GANs) In recent years, advances in neural networks have led to the use of generative adversarial network (GAN) loss functions in video coding. The general idea is that an image patch no longer needs to locally resemble the actual image patch, but the genre of the video, for example, is preserved. For example, a picture with an ocean with waves in the background will be reconstructed such that the water is still there, but the exact position of the waves is no longer the same as in the source video.

[0120] With the increasing use of GAN losses in video coding, the decoded video loses functionality in some applications. For example, in forensic and medical imaging applications, spatially accurate pixel data is essential for analysts and algorithms to analyze the data. At the same time, a brief overview of the types of data is useful.

[0121] A GAN consists of a generative network that "hallucinates" the output, and a discriminative network that evaluates the output. The discriminative network is trained to distinguish between generated outputs (e.g., synthetic images of faces) and known real values ​​(e.g., photographs of faces).

[0122] In the loss network described herein, the identification network is part of the loss network.

[0123] The surprising strength of the GAN approach is that the two networks are put into a lockstep battle, pushing each other towards higher performance. Of course, as with all machine learning, if there isn't enough variety in the inputs or if training isn't allowed to continue for enough epochs, the output from the GAN may not be suitable.

[0124] The scalar output of a discriminative network is called a realism measure. It is completely different from a distortion measure, because the fact that an image looks good (or realistic) does not mean that it is the same as the encoder input. Humans are willing to tolerate some coding artifacts (blurring, blocks, fringes). However, a neural network that hallucinates new objects not present in the original scene may be unacceptable to observers.

[0125] The realism metric can be understood to serve as a visual fidelity index that quantifies the degree to which a decoded image faithfully reproduces a real-world scene. It takes into account factors such as color accuracy, texture detail, and overall consistency to assess the reliability of the generated video.

[0126] For example, the academic paper "Neural Video Compression Using GANs for Detail Synthesis and Propagation" by F Mentzer et al. (arXiv:2107.12038, November 23, 2021) presents the first neural video compression method based on GANs. This academic paper shows that the introduction of a GAN loss function significantly improves compression performance compared to traditional metrics such as Peak Signal-to-Noise Ratio (PSNR), Multi-Scale Structural Similarity Index Metric (MS-SSIM), and Video Multi-Method Assessment Fusion (VMAF).

[0127] Layered Video Codec Figure 9 shows layered video in a predictive structure. Predictive structures are used in video coding, where a frame is decoded using multiple (zero or more) reference frames. When a frame has no reference frame, it is called an intraframe or I-frame. When it has one reference frame, it is called a predicted frame or P-frame. When it has two reference frames (one after the other in presentation order), it is called a bidirectional frame or B-frame. The horizontal direction (left to right) in Figure 9 indicates the presentation order of the frames. Of course, it will be understood that a B-frame can only be presented after the frame it depends on has been decoded. Therefore, the decoding order may not be the same as the presentation order.

[0128] When the prediction structure is repeated every so many frames, it is called a Group of Pictures (GOP). In the example of Figure 9, the GOP length is 8. Alternatively, the prediction structure can be content dependent and does not need to be fixed. For example, I-frames are inserted (only) at shot cuts (e.g., scene changes).

[0129] There are many possible prediction structures, and a useful example to add is that of a layered video codec, where the bitstream has multiple layers, where the base layer is independently decodable while the dependent layer is decodable only after the base layer reference frame is decoded. This is shown as base layer a) and dependent layer b) in Figure 9. The class of layered codecs includes scalable video codecs, enhancement video codecs, multiview video codecs, and multiview+depth video codecs.

[0130] The main difference between a scalable video codec and an enhancement video codec is that all layers of a scalable video codec share the same video format, while the enhancement layers may have a different representation.

[0131] Multiview video codecs have different views at each layer, and when these views are close enough, they can reduce redundancy by predicting one view from another. For example, the central view may be in the base layer and all other views may be in dependent layers.

[0132] Multiview+depth video codecs encode both color and depth, and prediction from reference frames may include a view synthesis step.

[0133] Thus, the neural post-processor concept described above can be applied to layered video coding. Figure 10 shows a neural post-processor applied to layered video coding. In this case, the base layer x A are independently decodable and provide a low-fidelity video stream. Enhancement layer x B is coded depending on the base layer and can be used to increase the fidelity of the frame, where fidelity is related to having higher spatial resolution, higher temporal resolution, higher bit depth, deeper colors, greater dynamic range, less noise, etc.

[0134] There are often subtle differences between neural post-processors and neural codecs. These differences are usually based on how the resulting bitstream is structured. Neural post-processors generally do not require bitstream elements, or the information may be present in the bitstream but not in the form of layers. A video layer is a specific concept related to Network Abstraction Layer (NAL) units. A bitstream typically consists of a sequence of NAL units, each with a short header that can be used to divide the bitstream into layers. Therefore, it is possible to divide the bitstream into layers without parsing large portions of the bitstream. Neural post-processors work based on coefficients transmitted as enhancement layers.

[0135] where the neural encoder f A encodes the base layer, and the encoded base layer y A The neural decoder g A decodes the coded base layer and reconstructs the base layer x' A The neural decoder g A The reconstructed base layer and / or intermediate output w A is the enhancement layer x B can be used by neural post-processors for

[0136] Of course, a conventional encoder and decoder can be used for the base layer, and the base layer reconstruction x' (by the conventional decoder) A It will be appreciated that is used by the neural post-processor.

[0137] Enhancement Layer x B is the neural encoder f B and the neural encoder f B is the coefficient c f,B Using y B The neural decoder g B is the input yB and optionally an intermediate output w A and / or x' A and the coefficient c g,B as further input using the neural decoder g B is the reconstruction of the enhancement layer x' B Output the reconstruction x'. B is input to the function Q, where the enhancement layer x B and the distortion measure q B Furthermore, the encoded enhancement layer y B is input to the function R, which generates the rate measure r B is determined. Therefore, the rate measure r B and the distortion measure q B is the loss function L B The loss function L B is the parameter λ as mentioned above. B Using the error e B The error is input to the optimizer, which determines the coefficient c g,B and / or c f,B Once the neural post-processor is sufficiently optimized, the encoded enhancement layer y B and the optimized coefficient c g,B The base layer y A is sent along with

[0138] Note that the coded enhancement layer and coded base layer essentially refer to the underlying representation of a neural network consisting of a neural encoder and a neural decoder for each video layer.

[0139] Therefore, we propose to encode videos by simultaneously optimizing multiple metrics via layered coding, so that the metrics themselves can be used as differentiators to download videos depending on the use case.

[0140] This is based on the fact that when a neural network is transmitted, only the parts that are unknown to the decoder need to be transmitted. This can be anything from transmitting nothing, to transmitting single values ​​of coefficients, the structure of layers, or even the entire neural network. Thus, the more data (of the neural network) is transmitted, the more personalized the neural network can be (at the decoder side). This is especially important in video decoding, when trying to balance bitstream size with the quality of the decoded video.

[0141] Therefore, we propose to build an encoder for layered video sequences that has at least a first layer and a second layer, where the second layer depends on the first layer, and the first and second layers each have different loss networks to optimize different metrics.

[0142] In one example, the encoder encodes at least two layers (each layer encoded using a different loss function / network) and transmits at least two layers, so that the client can decide to decode only the layers that were encoded according to the loss function / network of interest. This provides a significant computational advantage to the client, as only the layers of interest (based on the loss function / network of interest) are downloaded / decoded, etc.

[0143] Figure 11 shows a neural codec structure for layered video. A is the neural encoder f A and then the neural decoder g A and the base layer is reconstructed by A In this example, the loss network L' A For example, reconstruct x' A The base layer x A By trying to optimize the first metric (e.g., realism) compared to g,A We train a decoder function by trying to optimize

[0144] The client is a neural decoder A Assuming we have access to the structure of A (i.e., the encoded base layer) is reconstructed by the client as x' A So that we can obtain the coefficient c g,A The client can then send the structure to the neural decoder g, for example by receiving it at the beginning of the video. A Alternatively, the structure (or parts of it) can be updated and sent as needed (for example, when a scene in the video changes).

[0145] Enhancement Layer x B is processed in the same way to generate the neural encoder f B , neural decoder g B , and the loss network L' B However, if we reconstruct the enhancement layer x', B When reconstructing to B is the coefficient c g,A As with the base layer, the latent representation y B is the coefficient c g,B can be sent together with

[0146] Here, the loss network L' A For example, we can try to optimize the realism measure using the A provides a scene that looks good but doesn't need to be completely accurate to the base layer (for example, the shape and position of waves in the ocean may not be accurate).

[0147] Furthermore, the loss network L' B can be used to try to optimize the distortion measure, for example. In this way, the client-side reconstructed enhancement layer x' B is the reconstructed base layer x' AIt can provide enhancements to scenes in order to make them appear more accurate compared to the base layer.

[0148] Therefore, if the user only wants to see the global view of the scene, the reconstructed base layer x' A may be sufficient. In this case, the client does not need to decode the enhancement layer. However, if the user wants to see an accurate representation of the scene (e.g., for forensic reasons), the reconstructed enhancement layer x' B can provide a more accurate version of the scene.

[0149] Therefore, it is also proposed to construct a receiver having a decoder for a layered video sequence having at least a first layer and a second layer, the second layer depending on the first layer, and the first and second layers having decoder functions, where decoder coefficients are also received at the first and second layers and are used by at least one of the decoder functions to decode the first and second layers.

[0150] In another example, the base layer is encoded with a conventional (non-differentiable) encoder and decoded by the client with a conventional (non-differentiable) decoder, while the enhancement layer is encoded and decoded using a neural encoder and decoder as described above. Of course, it is also possible to encode / decode the base layer with a neural encoder / decoder and the enhancement layer with a conventional (non-differentiable) encoder / decoder. This depends on the particular use case.

[0151] In general, the bitstream from the encoder contains inputs to the neural decoder g that are not already pre-shared, can be obtained from other inputs, or are known from the base layer decoding. A Furthermore, the bitstream should contain all inputs to the neural decoder g that are not already pre-shared, can be obtained from other inputs, or are known from the base layer decoding. B All inputs to should be included.

[0152] The examples herein are shown with a single base layer and a single enhancement layer. However, it will be understood that in practice any number of base layers and any number of enhancement layers may be used. The bitstream is organized so that any of the dependent layers can be filtered out while retaining the other layers.

[0153] The motivation for having different loss networks for the enhancement and base layers is that compromising multiple applications with a single bitstream is unsatisfactory. This is because it is usually not possible to have one bitstream that is optimized for all metrics of interest (e.g., low bitrate, low distortion, high realism, etc.). Furthermore, having separate video bitstreams for each application is expensive in terms of required encoder hardware, power consumption, and bitrate. Therefore, different loss networks enable multiple applications.

[0154] As previously mentioned, one (or both) of the loss networks may include a realism measure.

[0155] The term realism measure is a generalization of GAN loss. There is industry hope that neural video codecs using GAN loss can produce good-looking videos at low bitrates. Sometimes, when you zoom in, things look strange. However, in this case, other layers may have different loss networks that are better suited to zooming in.

[0156] For example, one layer of the loss network may contain a distortion measure and a realism measure, while the other layer may contain a loss network that contains a distortion measure but not a realism measure.

[0157] The metric / loss information (e.g., realism measure, distortion measure, rate measure, tuned rate control settings, etc.) used in the first loss network (e.g., in the base layer) can also be input into the encoding process of other layers (e.g., enhancement layers).

[0158] While providing advantages to the client, encoding multiple layers requires more resources (hardware or paper) than a single layer, which can be problematic in 1:N transmission schemes where N is low. Because the input signals to multiple layers are identical or at least related (such as nearby viewpoints or different modalities of the same pixel), and to some extent, the output signals are also related, it is possible that control information from the base layer training loop can be useful to speed up parameter tuning of dependent layers.

[0159] This can reduce the number of epochs required to train a neural network for a dependent layer. A specific example is the first estimate of lambda required to reach a certain rate. Even if these values ​​are not the same, when the tunable parameter λ of the first network changes over time due to content changes, they are good predictors of how the parameters / coefficients of the second network need to change to compensate as well. Knowing these changes in advance guides a "direct" gradient descent to the optimum, saving the number of epochs and thereby power consumption. Because the number of epochs is usually fixed due to hardware constraints, the network will likely be more optimal if it starts with better control parameters.

[0160] For example, one of the loss networks (in the first or second layer) is based on the realism and / or distortion measures of the first layer, the realism and / or distortion measures of the second layer, and the combined bit rate.

[0161] In many cases, it is possible to send the complete bitstream to an edge server (i.e., the server at the last hop to the client) and then send only the video track with the relevant layer to the client. In this case, only the bitrate of the video track is relevant (assuming the connection to the client is the bottleneck). However, for simpler transmission schemes, or when a client needs to receive multiple alternative layers / tracks, it is advantageous to perform simultaneous rate optimization for multiple layers. This balances the overall bitrate with the quality of each layer. Care may need to be taken to ensure that the quality of one layer is comparable to the quality of another layer. The quality of one layer can be scaled relative to another layer, making one layer more important than the other. More bits are allocated to layers whose quality importance is scaled up.

[0162] The encoder may perform at least one training epoch on a video frame with the goal of overfitting the loss network on this frame, and send updates to the decoder network coefficients, for example, within the access unit of the frame.

[0163] In one example, network information (e.g., network coefficients) is only transmitted at random access points (full refreshes), and then only the output of the encoder network (e.g., feature maps) is transmitted. It is advantageous to allow the encoder to tune the coefficients of the encoder and / or decoder network to better represent the frame. While transmitting updated coefficients requires bits, this can be offset by having a more compact output.

[0164] It works with any prediction scheme, including hierarchical GOP structures and inter-layer prediction.

[0165] As will be appreciated, the concepts described herein are applicable to both neural codecs (i.e., coding using a partitioned neural network) and neural post-processors (i.e., processing of encoded data using a neural network).

[0166] Figure 12 shows the reference frame x A and dependent frame x B As mentioned before, the encoder network f A and f B We use the latent representation y A and y B Neural decoder g A and g B Reconstruct the latent representation x' using A and x' B In this case, the loss network has a rate function [R A , R B ], distortion function [Q, Q B ], and the loss function [L A , L B ]. The loss function is composed of inputs from the rate and distortion functions and tunable parameters [λ A , λ B ] and the error [e A , e B ] is determined by the decoding coefficients [c g,A , c g,B ] and in this case the encoding parameters [c f,A , c f,B] The optimization is achieved by adapting the

[0167] Furthermore, in this case, the latent representation of the reference layer, y A , decoding coefficient c g,A , and reconstruction x' A One or more of the decoder functions g B Naturally, as mentioned before, the distortion measure q A and / or the reference layer rate measure r A is the loss network L of the dependent layerB Also used by

[0168] The structure looks the same for temporal prediction (GOP structure) or inter-layer prediction (such as inter-view prediction or scalable coding).

[0169] As a complement to the encoder, a receiver (more generally referred to as a client) is proposed that has a decoder for a layered video sequence having at least a first layer and a second layer, the second layer being dependent on the first layer, the first layer and the second layer having different decoder functions, at least one of which is a neural decoder as described above. The client receives, for example, decoder coefficients of the neural decoder.

[0170] Allowing multiple layers with different decoder functions helps solve the problem of having multiple bitstreams. Because the layers are interdependent, the additional bitrate required to provide an additional representation of the video is a fraction of that required for an independent stream. Because the layers have different decoder functions, a decoder is suited to decoding layers with different uses.

[0171] Note that the decoder functions may not be directly present in the bitstream, thereby reducing the total amount of data. Naturally, this means that some knowledge of the decoder network is required on the client side. As mentioned before, these can be sent periodically (or on demand), for example the entire structure could be sent at the start of the video and further updated as needed.

[0172] A client with a receiver can select which layers the decoder outputs (e.g., for playback). When the client switches tasks (e.g., from observation to analysis), it can almost instantly switch which layers to use for playback because all layers have already been received. For example, the first layer is useful for observing a scene, and the first layer enhanced by the second layer is useful for analyzing the scene.

[0173] The most recent frames (or parts thereof) of the first and second layers are stored in a (rotating) buffer, so that the second layer of the video can be decoded with a delay (e.g., depending on the application) upon client request.

[0174] For example, when a user zooms in, a request to switch from layers coded using a GAN loss to layers coded using a PSNR loss (i.e., a more accurate coding scheme) is triggered. For example, when a user zooms out, a request to switch from layers coded using a PSNR loss to layers coded using a GAN loss is triggered.

[0175] In interactive streaming scenarios, the relationship between the zoom ratio and the loss function can also be considered. In forensic applications, zooming in can mean switching from a GAN loss to a per-pixel loss (e.g., PSNR). If the PSNR loss is represented as a second layer, this layer is coded as tiles, and therefore only a small number of tiles are retrieved when zooming in.

[0176] With tiled or block image data, the base or enhancement layers representing a given loss function are sparsely available. For example, with a GAN base layer and a PSNR enhancement layer, a blue sky with clouds represents only the GAN base layer and no PSNR enhancement layer at all. It would be useful to indicate (visually) per region (e.g., using a grid overlay and text) which loss function was used for a particular spatial region. This could be suitable for forensic applications.

[0177] Loss functions are usually designed with certain requirements in mind: for example, the MSE loss function is intended to avoid large errors per pixel, while the goal of GAN losses is to avoid images starting to look unnatural.

[0178] There may be loss functions designed to keep images of a given object class as accurate as possible while tolerating errors in images of other object classes that appear in the same image. Such category-specific losses can be implemented as one or more independent enhancement layers. In this case, a client device can request a video stream in which car images are always coded with per-pixel PSNR, while other aspects of the scene (e.g., background, streets, houses) are coded using GAN losses. Again, the spatial region (not necessarily rectangular) used for each loss function can be communicated to the client side. Another example is an enhancement layer that always keeps images of faces accurate per-pixel using PSNR.

[0179] In use, the first layer may be the only layer output for playback (e.g., if there is no request from the client). This occurs when the user is simply watching the video without interacting with it (e.g., zooming in / out).

[0180] The first layer is decodable using a non-differentiable (e.g., fixed function, legacy) video decoding process, and the second layer is decodable using a neural video decoding process (i.e., neural coding). Thus, the first layer can be decodable using a conventional legacy video decoder, and the second layer (e.g., containing enhancement information) can be decodable using a neural decoder.

[0181] Essentially, this provides the neural post-processor decoding scheme described above in Figure 8, where the output of the (legacy) video encoder h corresponds to the first layer, and the latent representation y corresponds to the second layer, which is the decoder coefficients c g can be decoded with a neural decoder using

[0182] The advantage is that most of the decoding can be done using dedicated hardware support for legacy video codecs.

[0183] As explained in the encoder, the intermediate output of the first layer (output w shown in Figure 10) A ) can be used as input to the second layer neural decoder.

[0184] When the outputs of two video layers differ substantially, the output of the first layer is not predictive of the second layer. However, some intermediate outputs are predictive because there are neural network layers that relate to information in the content shared between the video layers. This is "free" information because the decoder can postpone removing the received decoder coefficients of the first layer until the second layer is decoded.

[0185] It will be appreciated that the (coded) frames of the first layer can also be used as input to the second layer.

[0186] Since the "coded frame" has not yet been decoded, all of the information transmitted in the first frame (e.g., the reference frame) is also used in the second frame (e.g., the dependent frame). Essentially, the second frame is decoded using a neural network placed after the neural network of the first frame.

[0187] Of course, it is not necessary to use the reconstructed frames of the first layer for the reconstruction of the second layer.

[0188] The first and second layers may contain frames of different modalities (depth maps, multispectral, object maps, reflectance, normal maps, etc.). When the modalities are different, it is useful to share intermediate outputs. However, sharing reconstructions of other frames may not be as useful.

[0189] For example, the first and second layers are images of a scene from different viewpoints. As before, when the viewpoints are different, it is useful to share intermediate outputs, but it may not be so useful to share outputs of other frames.

[0190] When using a conventional coding scheme in one of the layers, intermediate outputs of the conventional decoding process, such as the coding tree, motion vectors, or predictor selection, may be input to the neural decoder in the other layer. Essentially, this is a variant of the decoder's neural post-processor. In this case, the "intermediate outputs" include information such as the motion vectors, coding tree, and predictor selection. This information allows for a more compact decoder network in the dependent layer, resulting in a better output.

[0191] It is further proposed that the encoder receives encoding instructions (also called training instructions) from a client, configures a training loop according to the instructions, and uses the training loop to encode one of the layers.

[0192] Figure 13 shows a frame x being coded according to coding instructions 1302. In this case, the frame is coded by a neural encoder f, and a training loop is initialized whereby a neural decoder g decodes the latent representation y output by the neural encoder into a reconstruction x'. In this case, the reconstruction x' is compared with the original frame x by a loss network L', which then generates the decoding coefficients c for the next training loop. g Adapt the.

[0193] The loss network and / or neural codec are adapted based on the encoding instructions 1302 received from the client. One or more functions included in the loss network may be adapted based on the encoding instructions 1302. The structure of the neural encoder and / or neural decoder may also be adapted.

[0194] For example, in a 1:1 streaming scenario, if a video is being streamed and the client wants to switch from one use to another (e.g. observation versus detailed analysis), it would be convenient to be able to reconfigure the encoder's training loop for this use while streaming.

[0195] The received encoding instructions may include parts of the lossy network and / or neural codec, such as the network structure, decoder coefficients, and / or tunable control parameters.

[0196] To change the purpose of the video, the loss function can be changed, which causes the encoder to optimize the video differently. The encoder can retrain the neural codec less frequently, such as every frame or only when it receives new encoding instructions.

[0197] If the encoding instructions include a network structure, the training loop may change the type of network, with a list of options including GAN, PSNR, and other objective metrics (VMAF, SSIM).

[0198] If the encoding instructions include loop coefficients, the loop coefficients used may depend on the network structure. One particular coefficient that is useful for tuning is the relative weight of the realism measure and the distortion measure (if both are present) in a lossy network. The loop coefficients may also include decoding coefficients and / or coding coefficients.

[0199] Regarding tunable control parameters, two examples of these types of parameters are parameters that control quality versus bitrate (i.e., λ as shown in Figure 8) and the average / maximum bitrate for the entire video or for individual layers.

[0200] Note that distortion is generally an inverse measure of quality: lower distortion means lower reconstruction error, which generally also means higher quality.

[0201] The encoding instructions may include parts of the encoder or decoder network, such as the network structure, loop coefficients, or tunable control parameters.

[0202] The encoding instructions may also include instructions to adapt the neural encoder and / or decoder. In one example, a smaller decoder network requires fewer resources from the client. In a second example, different encoder networks may produce different types of video with the same loss function.

[0203] Combining the concept of multiple loss functions (as explained in Figures 11 and 12) with the concept of receiving encoding instructions has the advantage that the encoder can output a multi-layered video bitstream, allowing the client to later (at storage, years later) select the best layer for playback, and additionally allow some or all layers to be tuned to the client's preferences.

[0204] Thus, a transceiver can be constructed with a decoder for video sequences having one or more layers, allowing the transceiver to send encoding instructions (e.g., select a loss function) and receive a video sequence coded according to the loss function.

[0205] In this case, there is a signal path in the opposite direction (i.e., towards the encoder) that provides instructions for the neural video codec's training loop (e.g., the instructions include part of the training loop), so that the transmitted, and therefore received, video is more suitable for the client's purposes.

[0206] Each time a coding command (e.g., involving any part of the lossy network and / or rate control functions) is sent, the decoder is ready to receive updates about changes to the decoder network, with a delay after the request is sent. The decoder only allows network updates at random access points (i.e., during I-frames). However, the decoder is flexible enough to receive updates for any frame of any video layer.

[0207] The client sends updated encoding instructions (e.g., an updated loss function) in response to the viewer's viewing position and orientation.

[0208] As an example, a client may specify a loss function in its encoding instructions, send a request to a server, and receive compressed video that has been encoded according to the specified loss function.

[0209] In a second example, when a user zooms in, a request to switch from, say, a GAN loss to a PSNR loss is triggered. Similarly, when a user zooms out, a request to switch from a PSNR loss to a GAN loss is triggered.

[0210] The coding instructions specify gradual transitions when changing parts of the training loop (e.g., by gradually adding realism over time in many steps).

[0211] In a third example, the client specifies a switch from the GAN loss to the PSNR loss when the observer moves near an object in the scene.

[0212] The combination of latent representations and coefficients results in lower bitrates because frequent or frame-by-frame optimization of the neural decoder network results in more compact inputs. This is a bitrate trade-off. However, it is possible to update only a small portion of the decoder network (e.g., a few layers, some / all coefficients, etc.) to achieve significant benefits with only a small increase in the bitrate due to the transmission of the neural network. Generally, the initial part of the decoder network (known as the backbone) remains fixed (both its structure and coefficients).

[0213] Coding layers may include the decoder structure of a neural decoder. Instead of or in addition to retuning the neural network by updating the coefficients, it is also possible to replace some of the functions in the network with other functions from a predefined function dictionary.

[0214] A specific example is a decoder network whose shape (width, depth) is parameterized, and coefficients are shared between nodes such that shape changes are independent of coefficient updates.

[0215] This ability to update the structure helps adapt to changes in content and video coding requirements.

[0216] The coding of a dependent layer borrows / shares coefficients from at least one of the reference layers, the advantage of which is that it saves bitrate by transmitting fewer coefficients.

[0217] The neural decoder network of the dependent layer may share similarities with the neural decoder network of the reference layer, e.g., some network layers of the decoder function are the same, or there are some function blocks with coefficients applied in multiple positions.

[0218] The coding of the dependent layer borrows / shares intermediate outputs (i.e., latent representations) from at least one of the reference layers.

[0219] The encoder may send a supplemental enhancement information (SEI) message to the decoder with information about the loss function used.

[0220] Clients can receive feedback about which loss function is being used for the data currently being viewed. For example, in the corner of the screen overlay, text may say "MSE mode" or "GAN mode," meaning that mean squared error loss or GAN loss was used for coding.

[0221] The encoder may also send an object map (pixel-level or block-level) to indicate which loss function was used for which spatial region in one or both of the first and second layers.

[0222] The present invention is generally defined by the following embodiments.

[0223] Embodiments: Embodiment 1: A method for encoding a video signal (x), comprising: receiving an encoding instruction (1302) for encoding a video signal; encoding a video signal using a training loop for optimizing a metric of the encoded video signal, the training loop including a loss function (L) and one or more of a neural encoder (f) and a neural decoder (g); and adapting the training loop based on the encoded instructions.

[0224] Embodiment 2 is dependent on embodiment 1, in which the encoding instructions include a metric for which the training loop is optimized.

[0225] Embodiment 3, depending on embodiment 1 or 2, wherein the encoding instructions include structural information of the neural decoder and / or the neural encoder.

[0226] The training loop includes a neural decoder, and the encoding instructions are used to generate the decoder coefficients (c g Embodiment 4 is dependent on any of embodiments 1 to 3, including

[0227] The training loop includes a neural encoder, and the encoding instructions are the encoder coefficients (c f Embodiment 5 is dependent on any of embodiments 1 to 4, including

[0228]

[0023] Embodiment 6 according to any of embodiments 1 to 5, wherein the encoding instructions include tunable control parameters for the loss function.

[0229] Embodiment 7 according to any of embodiments 1 to 6, wherein the metric is one of a realism measure, a distortion measure (q), a color accuracy measure, a rate measure (r), or an object detection accuracy measure.

[0230] The video signal is transmitted through the first layer (x A ) and the second layer (x B ), wherein the second layer is dependent on the first layer, and wherein encoding the video signal includes encoding the first layer and the second layer, wherein at least one of the first layer and the second layer is encoded using neural coding, the first layer is encoded using a first loss function to optimize a first metric, and the second layer is encoded using a second, different loss function to optimize a second metric.

[0231] Embodiment 9 A method for decoding an encoded video signal (y), comprising the steps of: sending encoding instructions (1302) to an encoder; receiving an encoded video signal encoded based on the encoding instructions and a training loop including a neural decoder (g); receiving decoding parameters for decoding the encoded video signal; and decoding the encoded video signal using the neural decoder and the decoding parameters.

[0232]

[0033] Embodiment 10, which is dependent on embodiment 9, wherein the encoding instructions are based on the user's viewing position and / or viewing direction.

[0233]

[0033] Embodiment 11, which is dependent on embodiment 9 or 10, wherein the encoding instruction is based on zooming in or zooming out by a user.

[0234] An embodiment 12 that depends on any of embodiments 9 to 11, wherein the step of sending the encoding instruction is triggered by one or more of a change in viewing position, a change in viewing direction, a user zooming in, and a user zooming out.

[0235] The encoding instructions include information about the structure of the neural decoder and / or neural encoder in the training loop, the metrics for which the training loop is optimized, and the coding coefficients (c f ), the decoder coefficients of the neural decoder (c g ), and tunable control parameters for the loss function in the training loop.

[0236] Embodiment 14: A computer program medium comprising computer program code, which when executed on a processing system causes the processing system to perform all of the steps of the method according to any of embodiments 1 to 8 and / or any of the steps of the method according to any of embodiments 9 to 13.

[0237] Embodiment 15. A processing system for reading the computer program code in the computer program medium of embodiment 14 and executing the computer program code, as more particularly defined in the appended claims.

[0238] Variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality.

[0239] The functions performed by a processor may be performed by a single processor or by multiple separate processing units which together may be considered to constitute a "processor". Such processing units may be remote from each other and may communicate with each other via wired or wireless means.

[0240] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0241] The computer program may be stored / distributed on any suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.

[0242] It should be noted that when the term "adapted to" is used in the claims or description, it is intended to be equivalent to the term "configured to." When the term "apparatus" is used in the claims or description, it is intended to be equivalent to the term "system," and vice versa.

[0243] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. A method for encoding a video signal, comprising: receiving encoding instructions for encoding said video signal, the encoding instructions describing a method for encoding said video signal and / or an application for encoding said video signal; encoding the video signal using a training loop to optimize a metric of the encoded video signal, the training loop including a loss function and one or more of a neural encoder and a neural decoder; adapting the training loop based on the encoded instructions; A method comprising:

2. The method of claim 1 , wherein the encoded instructions include the metric for which the training loop is to optimize.

3. 3. The method of claim 1, wherein the encoding instructions include structural information of the neural decoder and / or the neural encoder, the structural information representing a selection between possible configurations of the neural decoder and / or the neural encoder.

4. The method of claim 1 , wherein the training loop includes the neural decoder and the encoding instructions include decoder coefficients for the neural decoder.

5. The method of claim 1 , wherein the training loop includes the neural encoder and the encoding instructions include encoder coefficients of the neural encoder.

6. The method of claim 1 , wherein the encoding instructions include tunable control parameters for the loss function.

7. The method of claim 1 , wherein the metric comprises one of a realism measure, a distortion measure, a color accuracy measure, a rate measure, or an object detection accuracy measure.

8. the video signal is a layered video signal having a first layer and a second layer; Since the second layer depends on the first layer, the second layer can be decoded only after a reference frame in the first layer is decoded; and encoding the video signal includes encoding the first layer and the second layer; at least one of the first layer and the second layer is encoded using neural coding; 8. The method of claim 1, wherein the first layer is encoded using a first loss function to optimize a first metric, and the second layer is encoded using a second, different loss function to optimize a second metric.

9. 1. A method for decoding an encoded video signal, comprising: sending encoding instructions to an encoder describing a method for encoding said video signal and / or an application for encoding said video signal; receiving the encoded video signal, the encoded video signal being coded based on the coding instructions and a training loop including a neural decoder; receiving decoding parameters for decoding the encoded video signal; decoding the encoded video signal using the neural decoder and the decoding parameters; A method comprising:

10. The method of claim 9 , wherein the encoding instructions are based on a user's viewing position and / or viewing direction.

11. The method of claim 9 or 10, wherein the encoding instructions are based on a user zooming in or zooming out.

12. The step of transmitting the encoded instruction comprises: Change of viewing position, Change viewing direction, The user zooms in, and User zooms out The method according to any one of claims 9 to 11, wherein the method is triggered by one or more of the following:

13. The encoded instruction is: structural information of the neural decoder and / or neural encoder in the training loop, the structural information representing a selection between possible configurations of the neural decoder and / or the neural encoder; a metric for which the training loop optimizes; the coding coefficients of the neural encoder; Decoder coefficients of the neural decoder; and Tunable control parameters for the loss function in the training loop 13. The method of any one of claims 9 to 12, comprising one or more of:

14. 14. A computer program medium comprising computer program code which, when executed on a processing system, causes the processing system to perform all of the steps of the method of any one of claims 1 to 8 and / or any one of the steps of the method of any one of claims 9 to 13.

15. 15. A processing system that reads the computer program code in the computer program medium of claim 14 and executes the computer program code.

Images