A general media neural network predictor with perceptually shaped noise and a generative model including such a predictor

Abstract

An aspect of the present disclosure relates to a computer implemented neural network system for predicting frequency coefficients of a media signal. The neural network system comprises a time predicting portion and a frequency predicting portion. The neural network system further comprises a noise generation stage, configured to generate noise, and a noise application stage. The noise application stage being configured to perform at least one of obtain first original coefficients representing one or several previous time frames and perturb the first original coefficients with the noise to form the first input coefficients and obtain second original coefficients representing one or several frequency bands adjacent to the specific frequency band in said current time frame and perturb the second original coefficients with the noise to form the second input coefficients. The neural network system further comprises an output stage.

Description

A GENERAL MEDIA NEURAL NETWORK PREDICTOR WITH PERCEPTUALLY SHAPED NOISE AND A GENERATIVE MODEL INCLUDING SUCH A PREDICTOR CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U.S. Provisional Application No.63 / 638,417, filed on 25 April 2024, and European Application No.24202461.0 filed on 25 September 2024, each of which is incorporated by reference herein in its entirety. TECHNICAL FIELD

[0002] The present invention relates to a generative model for media, in particular audio. Specifically, the present invention relates to a computer implemented neural network system for predicting frequency coefficients representing frequency content of a media signal. BACKGROUND

[0003] A generative model for high-quality media (and in particular audio) can enable many applications. Raw waveform generative models have been proven to successfully achieve high quality audio within certain signal categories e.g. speech and piano, but the quality for general audio is still lacking.

[0004] Recently attempts have been made to move away from the raw waveform domain, for example as discussed in the article “MelNet: A Generative Model for Audio in the Frequency Domain”, by Vasquez and Lewis, 2019.

[0005] Additionally, in International Patent Application No. PCT / US2021 / 054617 a general media neural network predictor is proposed. SUMMARY

[0006] Despite recent attempts to provide generative models for high-quality media even further improvements would be beneficial. It is an object of the present disclosure to provide an improved generative model for general media, and in particular general audio, i.e. not only specific categories of audio, like speech or piano music, but audio in general.

[0007] According to a first aspect of the present invention there is provided a computer implemented neural network system for predicting frequency coefficients of a media signal. The neural network system comprising a time predicting portion comprising at least one neural network trained to predict a first set of output variables representing a specific frequency band of a current time frame of the media signal given first input coefficients representing one or several previous time frames of the media signal. The neural network system further comprises afrequency predicting portion comprising at least one neural network trained to predict a second set of output variables representing a specific frequency band given second input coefficients representing one or several lower frequency bands to the specific frequency band in the current time frame, a noise generation stage, configured to generate noise, and a noise application stage, wherein the noise application stage is configured to perform at least one of (A) obtaining first original coefficients representing one or several previous time frames of the media signal and perturb the first original coefficients with the noise to form the first input coefficients, and (B) obtaining second original coefficients representing one or several lower frequency bands to the specific frequency band in the current time frame and perturb the second original coefficients with the noise to form the second input coefficients. The neural network system further comprising an output stage configured to provide a set of output parameters representing the frequency coefficients of the specific frequency band of the current time frame, based on the first and second set of output variables.

[0008] This type of neural network system forms a predictor capable of capturing both temporal and frequency dependencies occurring in time-frequency tiles of a media signal. The frequency predicting portion is designed to capture frequency dependency e.g. harmonic structures.

[0009] The output parameters may describe a distribution representing the frequency coefficients, whereby the distribution described by the parameters may be sampled to obtain the predicted frequency coefficients. Alternatively, the output parameters are equal to the frequency coefficients.

[0010] By further adding noise to the input to at least one of the time predicting portion and the frequency predicting portion, the predictor can output predicted output parameters that represent frequency coefficients that feature better harmonic structure. The noise is added to the input of one or both of the time predicting portion and frequency predicting portion, but not to the output of the neural network system, whereby the output parameters representing the frequency coefficients of the specific frequency band will generally be clean from noise or at least only feature a low noise level.

[0011] In some implementations, the noise application stage perturbs both the first and second original coefficients or only one of the first and second original coefficients. In implementations where the noise application stage perturbs only one of the first and second original coefficients the original coefficients which are not perturbed are used as input coefficients directly.

[0012] In some implementations, the noise application stage is configured to perturb the first and / or second original coefficients with noise of a noise level which varies with a frequency associated with each original coefficient.

[0013] For example, the noise application stage is configured to perturb the first and / or second original coefficients with a noise level according to a noise level function indicating a noise level as a function of frequency.

[0014] Addition of noise with a noise level that depends on the frequency may improve the final quality of the output parameters. In one example, the noise level function is a monotonically non-decreasing function with frequency. Since human hearing is less sensitive at higher frequencies the addition of proportionally more noise at higher frequencies allows the output to be devoid from noise primarily at frequencies where human hearing is most sensitive, which makes for improved perceived quality. In some implementations, the first set of output variables, predicted by the time predicting portion, are used as the second original coefficients for the frequency predicting portion, or the second set of output variables, predicted by the frequency predicting portion, are used as the first original coefficients for the time predicting portion.

[0015] That is, the time predicting portion may take the output of the frequency prediction portion as input in the form of original input coefficients or the frequency predicting portion may take the output of the time predicting portion as input in the form of original input coefficients.

[0016] According to a second aspect of the present invention there is provided a generative model for generating a target media signal. The generative model comprises a neural network system according to the first aspect, and a conditioning neural network trained to predict a set of conditioning variables given conditioning information describing the target media signal. Wherein the time predicting recurrent neural network being configured to combine the first set of input variables with at least a subset of the set of conditioning variables.

[0017] A third aspect of the present invention relates to a decoder comprising a generative model according to the second aspect.

[0018] A fourth aspect of the present invention relates to a computer program product comprising computer readable program code portions which, when executed by a computer, implements a generative model according to the second aspect.

[0019] A fifth aspect of the present invention relates to a method for training the neural network system according to the first aspect of the present invention. The method comprises providing a description of an actual media signal as conditioning information to the conditioning neural network, and predicting, using the neural network system, a set of frequency coefficients representing a specific frequency band of a current time frame. The method further comprisesminimizing a measure of the predicted set of frequency coefficients with respect to a true set of frequency coefficients representing the specific frequency band of the current time frame of the actual media signal.

[0020] A sixth aspect of the present invention relates to a method for generating an enhanced media signal using a generative model using a generative model according to the second aspect of the present invention. The method comprises obtaining a training noise level, indicating a noise level of noise used to perturb the first and / or second original coefficients in at least one frequency interval by the noise application stage during training of the generative model. The method further comprises generating an enhanced media signal by providing conditioning information to the conditioning neural network, and for each frequency band of a current time frame, using the frequency predicting recurrent neural network to predict the frequency coefficients of this frequency band, and providing the set of frequency coefficients to the frequency predicting recurrent neural network as the second original coefficients. The method further comprises providing the predicted sets of frequency coefficients representing all frequency bands of the current frame to the time predicting recurrent neural network as the first original coefficients, wherein while generating the enhanced media signal the noise application stage is configured to perturb the first and / or second original coefficients with noise of an inference noise level in the at least one frequency interval, wherein the inference noise level is different from the training noise level.

[0021] In some embodiments, the noise level for at least one frequency is higher during training than during inference.

[0022] The step of obtaining the noise level used during training may comprise accessing a prestored noise level. The prestored noise level may be implicitly defined by the configuration of the noise generator and / or noise application which has been configured to apply noise of a noise level which differs compared to what was used during training.

[0023] Optionally, prior to generating the enhanced signal in accordance with the sixth aspect of the present invention the method may comprise training the generative model in accordance with the fifth aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be described in more detail with reference to the appended drawings, showing embodiments of the invention.

[0025] Figures 1A-B show a high-level structure of a time / frequency predictor according to embodiments of the present invention.

[0026] Figure 2 shows a neural network system implementing the structure in FIG.1A.

[0027] Figure 3 shows the neural network system in FIG.2, operating in self-generation mode.

[0028] Figure 4 shows a generative model including the neural network system in FIG.2.

[0029] Figure 5 is a flow chart describing a method for training the neural network system according to some implementation.

[0030] Figure 6 is a flow chart describing a method for predicting a time frame of frequency coefficients using the neural network system as a generative model.

[0031] Figure 7 illustrates a schematic block diagram of an example device or architecture that may be used to implement various aspects of the present invention. DETAILED DESCRIPTION

[0032] FIG.1A and 1B schematically illustrate two examples of a high-level structure of a time / frequency predictor 1 according to an embodiment of the present disclosure.

[0033] The predictor operates on frequency coefficients representing frequency content of a media (e.g. audio) signal. In one example, the media signal is an audio signal carrying general audio. By general audio it is meant any type of audio and not only audio of specific categories, like speech or music. General audio may comprise one or more of speech content, music content, sound effects, noise or mixtures thereof. An audio signal may comprise a single audio object or a mix of multiple audio objects. Each audio object is an audio signal associated with parametric source descriptions of apparent source position and other parameters. Additionally or alternatively, the audio signal may comprise audio content associated with a channel of a multi- channel presentation (e.g. a stereo presentation or a 5.1 presentation) or a mixture of multiple channels.

[0034] The frequency coefficients may correspond to bins of a time-to-frequency transform of the media signal, such as a Discrete Cosine Transform (DCT) or a Modified Discrete Cosine Transform (MDCT). Alternatively, the frequency coefficients may correspond to samples of a filterbank representation of the media signal, for example a Quadrature Mirror Filter (QMF) filterbank.

[0035] In FIG.1A, the frequency coefficients (here sometimes referred to as “bins” or “coefficients”) of previous time frames are first grouped into a preselected number ^^ of frequency bands. Then the predictor 1 predicts bins 2 of a target band ^^ in a current time frame ^^ based on the band context collected from all previous time frames 3. The predictor 1 thenpredicts bins 2 of the target band ^^ based on all lower and ^^ higher bands (i.e. bands 1 … ^^ ^ ^^),where ^^ is between 1 and ^^ െ ^^. In FIG. 1A, ^^ is equal to 1, i.e. only one higher band b+1 istaken into account. Finally, the predictor predicts bins 2 in the target band ^^ based on all lower (previously predicted) frequency bands 5 in the current time frame ^^.

[0036] The joint probability density of frequency coefficients (e.g. MDCT bins) ^^௧^^^^ can be expressed as a product of conditional probabilities: ^^^^^^ ൌ ∏^ ∏௧ ^^൫^^௧^^^^ ห ^^^…௧ି^^1 … ^^ ^ ^^^, ^^௧^1 … ^^ െ 1^൯ , (1)in ^^ ^^ the number of, representscoefficients in bands 1 to ^^ ^ ^^ from time 1 to time ^^ െ 1, and finally ^^௧^1 ... ^^ െ 1^represents the bins in band 1 to band ^^ – 1 at time ^^. As an example, ^^ lower neighboringadjacent bands to band ^^ includes bands ^^ െ ^^, ^^ െ ^^ ^ 1, … ^^ and ^^ higher neighboringadjacent bands to band ^^ includes bands ^^ ^ 1, ^^ ^ 2, … ^^ ^ ^^.

[0037] As clear from the above description of the predictor in FIG.1A, the prediction is done first in the time dimension and then in the frequency dimension. This is quite normal in many applications, e.g. in an audio decoder, where the prediction is typically made in real-time of the next frame of a signal.

[0038] Generally speaking, however, for example if an entire signal is available off-line, the time / frequency predictor could operate in the opposite order. This, slightly less intuitive, process is illustrated in FIG.1B.

[0039] Here, first the bins in each lower band are grouped into a set of ^^ time frames. Then, the predictor 1’ predicts the bins 2’ of a target frame t in the current (next higher) frequency band b based on the band context collected from all lower frequency bands 3’. The predictor 1’ then predicts bins 2’ of the target frame ^^ based on the lower frequency bands in all preceding and ^^ subsequent (future) time frames (i.e. frames 1 … ^^ ^ 1), where ^^ here isbetween 1 and ^^ െ ^^. In FIG. 1B, ^^ is again equal to 1, i.e. one subsequent (future) frame istaken into account. Finally, the predictor predicts the bins 2’ in the target frame ^^ based on all preceding (previously predicted) time frames 5’ in the current frequency band ^^.

[0040] An example implementation of the predictor in FIG.1A in a neural network system 10 is illustrated as a block diagram in FIG.2. As explained in detail in the following, the neural network system 10 has a time predicting portion 8 and a frequency predicting portion 9.

[0041] In the time predicting portion 8, a convolution network 11 receives first input coefficients ^^^௧ି^(each coefficient associated with a bin) which are based on first originalprevious frame ^^௧ି^. The original coefficients may e.g. be individual MDCT, DCT or QMF coefficients of a media signal or a predicted version thereof as will be described below.

[0042] The convolution network 11 of the time predicting portion performs convolution of the first input coefficients ^^^௧ି^(bins) to group them into ^^ bands 12. As an example, ^^ is equal to 32. In one implementation, the convolution network 11 is implemented as a convolution layer having a kernel length,to 16 and a stride, ^^, equal to 8 (i.e.50% overlap). The convolutional network 11 implemented as one or more convolution layers with other kernel lengths K and strides S are also within the scope of this disclosure.

[0043] The ^^ bands 12 are fed into a time predicting recurrent neural network (RNN) 13 containing a set of recurrent layers, here in the form of Gated Recurrent Units (GRU). Other recurrent neural networks may also be used, such as Long short-term memories (LSTM), Quasi- Recurrent Neural Networks (QRNN), Bidirectional recurrent units, Continuous time recurrent networks (CTRNN), etc. The time predicting recurrent neural network 13 processes the ^^ bands 12 separately but with shared weights, obtaining individual hidden states 14 for each frequency band of the current (predicted) time frame. Each hidden state 14 includes a set of output variables, wherein the size of the set is determined by the internal dimension of the layers in the RNN 13. In the illustrated example, the internal dimension is 1024, so there are 1024 variablesrepresenting each frequency band of the current (predicted) time frame. With B ൌ 32, there arethus 32 x 1024 variables output from the time predicting RNN 13.

[0044] The ^^ hidden states 14 are then fed to another convolutional network 15 which mixes the variables of all lower and N higher bands (i.e. adjacent hidden states) in order toachieve a cross-band prediction ^^൫^^௧^^^^ ห^^^…௧ି^^1 … ^^ ^ ^^^൯. In one implementation, theconvolutional network 15 is implemented as a single convolution layer along the banddimension, where the kernel length is 2^^ ^ 1, with ^^ lower bands and ^^ higher bands. That is,in total 2^^ adjacent bands in addition to the specific band itself is processed with the kernel. Inanother implementation, the convolution layer kernel length is ^^ ^ 2 with one lower band and Nhigher bands. The output (hidden state) 16 is again ^^ sets of output variables, where the size of each set is determined by the internal dimension. In the present case again 32 x 1024 variables are output from the network 15.

[0045] In the frequency predicting portion 9, the hidden state 16 representing the current (predicted) time frame is fed to a summation point 17. A 1x1 convolution layer 18 receives second input coefficients ^^^௧ ^1^, ..., ^^ ^௧ ^^^ െ 1^ based on second original coefficientsrepresenting the frequency coefficients of lower frequency bands ^^௧^1^... ^^௧^^^ െ 1^ in thecurrent time frame, and projects them onto the internal dimension of the system, i.e.1024 in the present case. In one implementation, the second input coefficients represent all lower frequency bands (with respect to the specific frequency band ^^). Alternatively, the second input coefficients represent ^^ lower adjacent frequency bands.

[0046] The output of the summation point 17 is fed into a recurrent neural network (RNN) 19 containing a set of recurrent layers, here in the form of Gated Recurrent Units (GRU). Again, other recurrent neural networks may also be used, such as Long short-term memories (LSTM), Quasi-Recurrent Neural Networks (QRNN), Bidirectional recurrent units, Continuous time recurrent networks (CTRNN), etc. The RNN 19 takes the summation output and predicts a set of output variables 20 representing ^^௧^^^^. Finally, two output layers 21, 22 in the form of two 1x1 convolution layers (output dimension 1024 and 16, respectively), with ReLU activationpreceding each convolutional layer, serve to provide the final prediction of ^^௧^^^^(labeledത^ത^ത௧^^^^)according to the final prediction scheme ^^൫^^௧^^^^ ห ^^^…௧ି^^1 … ^^ ^ ^^^, ^^௧^1 … ^^ െ 1^൯. Thehidden state of RNN 19 is reset for every new time stamp.

[0047] The output from the two output layers 21, 22 comprises output parameters thatrepresent the frequency coefficients of the predicted bandത^ത^ത௧^^^^.

[0048] In one embodiment, the output parameters are predicted frequency coefficients ofthe predicted bandത^ത^ത௧^^^^.

[0049] In one embodiment, the output parameters represent each predicted frequencycoefficient of each frequency bandത^ത^ത௧^^^^ with two parameters, for example the system maypredict the parameters ^^ (location) and s (scale) of a Laplace distribution. In one implementation, log ^s^ is used instead of ^^ for computational stability. In other implementation, a Logistic distribution or a Gaussian distribution can be chosen as the target distribution for parameterization. The output dimension of the final output layer 22, (i.e. the number of output parameters) is therefore twice the number of bins. In the present case, the output dimension of layer 22 is 16, corresponding to eight bins in each frequency band.

[0050] In another embodiment, the output parameters parametrize the frequency coefficients of each band as a mix of distributions, where each parametrized distribution has an individual (normalized) weight. Each coefficient will then be represented by (number of distributions) x (number of distribution parameters +1) parameters. For example, in the specific case of mixing two Laplace distributions (each with two parameters), each coefficient will be represented by 2 x (2+1) = 6 parameters (two sets of weights (^^1 and ^^2), location (^^1, ^^2), and scale (^^1, ^^2), where ^^1 + ^^2 = 1). The output dimension of the output layer 22 will then be 8 x 6 = 48. The previously mentioned embodiment is a special case with only one distribution and weight equal to one.

[0051] The neural network system 10 further comprises a noise generator 51 and a noise application stage 52a, 52b for perturbing at least one of: the first original coefficients (e.g. the coefficients of the preceding frame ^^௧ି^) prior to its input to the time predicting portion 8 and the second original coefficients (e.g. the coefficients of the lower frequency bands^^ ^1^, ... ,^^ ^^^ െ 1^). The first and second input coefficients ^^^and ^^^^1^, ..., ^^ ^௧ ௧ ௧ ௧ ௧ ^^^ െ 1^ aretherefore based on the first and second original input coefficients but perturbed with noise generated by the noise generator 51.

[0052] In time domain, noise may be described as a signal which varies randomly with time. Many types of noise are sequentially uncorrelated, meaning that one sample of a time domain noise signal is uncorrelated to any other samples of the noise signal. Noise may have an amplitude distribution describing the probability distribution of the noise amplitude (e.g. the noise is gaussian noise with an amplitude distribution following the normal distribution) and noise may have a spectral distribution describing how the noise spectral power is distributed over frequencies.

[0053] Here, noise is generated and applied to the original coefficients (spectral coefficients) whereby each original coefficient is perturbed with a noise sample to form a noise perturbed input coefficient. Perturbing an original coefficient with a noise sample may comprise drawing a random noise sample from a noise distribution and adding the noise sample to the original coefficient to form the input coefficient. Each spectral coefficient is associated with a frequency and in some implementations the spectral power of the noise distribution varies as a function of frequency (as described below). For example, when drawing noise samples to perturb a spectral coefficient of a first frequency a first noise probability distribution is used and when drawing noise samples to perturb a spectral coefficient of a second frequency a second noise probability distribution is used whereby the first and second probability distributions are different from each other.

[0054] In one embodiment, noise generated by the noise generator 51 is applied by the noise application stage 52a, 52b to only one of the first and second original coefficients or to both of the first and second original coefficients.

[0055] In one embodiment, noise is only added to the first original coefficients ^^௧ି^at summation point 52a to form noise perturbed first input coefficients ^^^௧ି^which are fed to the time predicting portion 8. In this embodiment, the second input coefficients ^^^^1^, ^௧ ..., ^^௧ ^^^ െ1^ are equal to the second original coefficients ^^௧^1^, ... ,^^௧^^^ െ 1^) as no noise is added atsummation point 52b.

[0056] In one embodiment, noise is only added to the second original coefficients^^௧^1^, ... ,^^௧^^^ െ 1^) at summation point 52b to form noise perturbed second input coefficients^^^௧ ^1^, ..., ^^ ^௧ ^^^ െ 1^ which are fed to the frequency predicting portion 9. In this embodiment,the first input coefficients ^^^௧ି^are equal to the first original coefficients ^^௧ି^as no noise is added at summation

[0057] By introducing noise, the neural network system 10 becomes capable of generating much clearer harmonic structures, especially for tonal signals, without introducing audible noise or other artifacts in the output. It is noted that the noise may be added only prior to input to eachof the neural network portions, meaning that the final output (e.g.ത^ത^ത௧^^^^) of the neural networksystem 10 is not directly perturbed with noise.

[0058] The noise generator 51 is configured to generate noise for application to the firstand / or second original coefficients ^^௧ି^,^^௧^1^, ... ,^^௧^^^ െ 1^) to transform these originalcoefficients into (noise perturbed) input coefficients ^^^,^^^^1^, ... ^௧ି^ ௧ , ^^௧ ^^^ െ 1^ that are inputinto the time predicting portion 8 and the 9 respectively.

[0059] Application of the noise generated by51 to at least one of thefirst and / or second original coefficients ^^௧ି^,^^௧^1^, ... ,^^௧^^^ െ 1^) is performed by the noiseapplication stage 52a, 52b.

[0060] The noise generated by the noise generator 51 may be provided to the noise application stage 52a, 52b which applies the noise to at least one of the first and second original coefficients. Applying the noise to original coefficients may comprise adding the noise to the original coefficients.

[0061] The noise generated by the noise generator 51 has a noise level. The noise level indicates the power or amplitude of the noise. The noise generated by the noise generator 51 may further have a power spectrum (also referred to as noise shape over frequency or “noise color”).

[0062] For example, the noise generated by the noise generator 51 is in some implementation white noise. The power spectrum of white noise is evenly distributed over all frequencies.

[0063] If the noise generated by the noise generator 51 and applied by the noise application stage 52a, 52b is white noise, noise of the same noise power level will on average beadded to each (first and / or second) original coefficient ^^௧ି^,^^௧^1^, ... ,^^௧^^^ െ 1^, irrespective ofthe frequency each original coefficient is associated with.

[0064] In some implementations, the noise applied to at least one of the first and secondoriginal coefficients ^^௧ି^,^^௧^1^, ... ,^^௧^^^ െ 1^ is spectrally shaped noise. With spectrallyshaped noise it is meant noise with a noise level which has a non-uniform distribution over frequency. For example, spectrally shaped noise is different from white noise due to at least the average noise power at a first frequency is different from the average noise power at a different second frequency.

[0065] For example, the spectrally shaped noise may be pink noise (noise with a noise level which decreases with increasing frequency) or blue / violet noise (noise with a noise level which increases with increasing frequency).

[0066] Spectrally shaped noise may be generated by the noise generator 51 directly and then applied by the noise application stage 52a, 52b. Alternatively, the noise generator 51 may generate a first type of noise (e.g. white noise) whereby the noise application stage 52a, 52b applies a gain curve which shapes the noise into a desired spectral shape.

[0067] In some implementations, the spectrally shaped noise is perceptually shaped noise. With perceptually shaped noise it is meant that comparatively higher noise power levels are used at frequencies where human hearing is less sensitive (e.g. at frequencies below 1 kHz or above 4 kHz) and comparatively lower noise power levels are used at frequencies where human hearing is more sensitive (e.g. at frequencies between 2 and 4 kHz).

[0068] Shaped noise may be defined with a noise level function ^^^^^^ which describes how the noise level of the applied noise varies as a function of frequency.

[0069] In some implementations, the noise level function ^^^^^^ is monotonically non- decreasing with frequency over at least a frequency interval ^^^^,^^ଶ^. The frequency interval^^^^, ^^ଶ^ may be the full human hearing range whereby ^^^ ൌ 20 Hz and ^^ଶ ൌ 20 kHz or thefrequency interval may be a portion of the full human hearing range.

[0070] A monotonically non-decreasing function is different from a strictly monotonically increasing function in that a monotonically non-decreasing function may be constant on an interval whereas a strictly monotonically increasing is constantly increasing. That is, the noiselevel function ^^^^^^ may be a monotonically non-decreasing function fulfilling ^^^^^ ^ ^^^^^ ^^^^^^^ for positive ^^^^ on the frequency interval ^^^^, ^^ଶ^.

[0071] It is also envisaged that the noise level function ^^^^^^ may be a strictlymonotonically increasing function fulfilling ^^^^^ ^ ^^^^^ ^ ^^^^^^ for positive ^^^^on thefrequency interval ^^^^,^^ଶ^.

[0072] In some implementations, the noise level function ^^^^^^ is constant for frequencies below a knee frequency ^^^^^^and increasing for frequencies above the knee frequency ^^^^^^. For example, the noise level^^^^^^ increases linearly, exponentially or polynomial for frequencies exceeding the knee frequency ^^^^^^. In one example, the knee frequency ^^^^^^is between 3 and 6 kHz, such as 4 kHz or 5 kHz. Since human hearing is most sensitive around 3 – 4 kHz a knee frequency ^^^^^^lying above 4 kHz will introduce more noise for frequencies where human hearing is more sensitive.

[0073] A further example of a noise level function ^^^^^^ is based on the absolute hearing threshold in quiet as described in Painter, Ted & Spanias, Andreas. (2000) “Perceptual Coding of Digital Audio”. Proceedings of the IEEE.88.451 - 515. The absolute hearing threshold indicates the sound pressure level at each frequency at which an average listener will detect a pure tone stimulus in a noiseless environment. The noise level function ^^^^^^ may be proportional to (e.g.equal to or a linearly scale version of) the absolute hearing threshold in quiet for all audible frequencies or at least a portion of the audible frequences. The absolute hearing threshold in quiet described in Painter, Ted & Spanias, Andreas is one example and other measurements of the absolute hearing threshold in quiet may be used as an alternative.

[0074] The absolute hearing thresholds indicate that human hearing is most sensitive around 3.3 kHz whereby the sensitivity decreases for increasing and decreasing frequencies from around 3.3 kHz. Hereby, the noise level function ^^^^^^ may be a convex function with a minima between 2 kHz and 4 kHz, such as a minima around 3.3 kHz.

[0075] As an example of a noise level function ^^^^^^ based on the absolute hearing threshold in quiet, the absolute hearing threshold in quiet presented in Painter, Ted & Spanias, Andreas will here be denoted ^^^^^^, wherein: ^ మ ^ି^^ ସ ^^^^^^ ൌ 3.64^ ^ି^.଼ െ .^^భబబబିଷ.ଷ^ିଷ^^^^6.5^^ ^ 10 ^^^^^^ . (2) The noise level function ^^^^^^ may now be defined using the absolute hearing threshold in quiet ^^^^^^ as: ^^^^^^ ൌ ^ ^^^^^^^^^^ ^ ^^ ^^ ^ f^^^^^^^^^^ ^ ^^ ^^ ^ f(3) ^^^^in dBand f^^^^is a knee frequency expressed in Hz. In one example, G is between െ40 dB and െ45 dB. For example, ^^ ൌ െ45 dB. The frequency ^^^^^^ may be set to the frequency point with theminimal absolute hearing threshold in quiet, which for the above mentioned reference is around 3.3 kHz. In some implementations, ^^^^^^is between 2 kHz and 4 kHz.

[0076] Hereby the noise level function ^^^^^^according to equation 3 is constant for frequencies below ^^^^^^and increasing for frequencies above ^^^^^^. Generally, human hearing is less sensitive for higher frequencies than lower frequencies whereby a constant noise level at frequencies below ^^^^^^(which e.g. is around 3.3 kHz) may be used.

[0077] For any definition of the noise level function ^^^^^^an upper limit may be placed on the noise level that is applied to the first and / or second original coefficients ^^௧ି^,^^௧^1^, ... ,^^௧^^^ െ 1^ to avoid saturating the first and / or second input coefficients..., ^^ ^௧ ^^^ െ 1^ with noise. For example, the noise level is limited to ^^^^௫decibels) whereby the level of the noise applied at the noise level application stage is kept below ^^^^௫ decibels. For example, ^^^^௫ ൌ െ30 ^^^^.

[0078] With further reference to the flow chart of FIG.5, training of the neural network system 10 will now be described.

[0079] Training of the neural network system 10 can be done in “teacher forcing mode”. First, in step S1, ground truth frequency coefficients representing an “actual” (known) media signal are provided to the convolution network 11 of the time predicting portion 8 and to the convolution layer 18 of the frequency predicting portion as first and second original coefficients, respectively.

[0080] At step S2a noise is generated with the noise generator 51 and applied with the noise application stage 52a, 52b to at least one of the first and second original coefficients^^ ,^^ ^1^, ... ,^^ ^^^ െ 1^ to gene^^௧ି^ ௧ ௧ rate the first and second input coefficients ^^௧ି^,^^௧ ^1^, ...,^^^^^^ െ 1^, w^^௧ herein at least one of the first and second input coefficients ^^௧ି^,^^௧ ^1^, ...,^^^௧ ^^^ െ 1^ becomes perturbed with noise. In some implementations, the noise applied is ashaped noise meaning that the noise level varies as a function of frequency, as described above.

[0081] The output parameters (which e.g. describes probability distributions of the binsഥX௧^^^^ of a current time frame) are then predicted in step S2b. In step S3, the predicted binsഥX௧^^^^ are compared to the actual bins ^^௧^^^^ of the actual signal in order to determine a training measure. Finally, in step S4, the parameters (weights and bias) of the various neural networks 11, 13, 15, 18, 19, 21, 22 are chosen such that the training measure is minimized or at least reduced. The training procedure of steps S1-S4 may then be repeated for new sets of ground truth frequency coefficients representing a known media signal.

[0082] As an example, the training measure which should be minimized may be the negative log-likelihood (NLL), e.g. in the case of Laplace distribution:^^^^^^ ൌ log^2 ∗ ^^^ ^|ఓି௬|^ , (4) where µ and s are the model output predictions and y is the actual bin value. The NLL would look slightly different in case of a Gaussian or mixture distribution model.

[0083] FIG.3 illustrates the neural network system 10 in FIG.2 in an inferencing mode, also known as a “self-generation” mode, wherein a predictedഥX௧^^^^ is used as history to continuously generate new predictions. The neural network system in FIG.3 is referred to as a self-generating predictor 30. Such a predictor can be used in an encoder to compute a prediction error based on a prediction generated by the predictor. The prediction error can be quantized and included in the bitstream as a residual error. In the decoder, the predicted result can then be added to the quantized error to obtain a final result.

[0084] The predictor 30 here includes two feedback paths, 31, 32; a first feedback path 31 for the time predicting portion 8 of the system, and a second feedback path 32 for the frequency predicting portion 9 of the system.

[0085] More specifically, a predicted bandഥX௧^^^^ is added to a partially predicted currentframe ഥ X௧ so that it then includes bands ഥ X௧^1^ െ ഥ X௧^^^^. The partially predicted current frame Xഥ௧may be stored in a buffer 53b of the frequency predicting portion 9. The frequency coefficientsof the bands in the buffer 53b are used as the second original coefficients ^^௧^1^, ... ,^^௧^^^^which are provided to the noise application stage 52b that perturbs the second original coefficients with noise to form the second input coefficients ^^^^௧^1^, ..., ^^௧^^^^. The second input coefficients ^^^௧ ^1^, ..., ^^^௧^^^^ are provided as input to the convolutional network 18, andthen to summation point 17, in order to predict the next higher band, ഥ X௧^^^ ^ 1^. The next higherband ഥ X௧^^^ ^ 1^ is added to the buffer of the frequency predicting portion before the process isrepeated to predict yet another next higher band ഥ X௧^^^ ^ 2^.

[0086] When all bands in the current frameഥX௧ have been predicted are available in thebuffer 53b of the frequency predicting portion, this entire frameഥX௧is provided as first original coefficients to the convolutional network 11 of the time predicting portion 8, to enable predictionof the next time frameഥX௧ା^. The entire frameഥX௧ may be stored in an input buffer 52a of the timepredicting portion 8. The entire frameഥX௧is used as the first original coefficients ^^௧^1^ and are perturbed with noise (e.g. spectrally shaped noise) by the noise applicator 52a to form the first input coefficients ^^^௧.

[0087] As described above it is envisaged that both the first and second original coefficients are perturbed with noise or that only one of the first and second original coefficients are perturbed with noise. When only one of the first and second original coefficients are perturbed with noise the first or second original coefficients which are not perturbed with noise are used as input coefficients without modification.

[0088] Given that in some embodiments ^^ and ^^ are the output parameters from the proposed neural network, a sampling operation 33 is used to obtain predicted bin values(coefficients) of the predicted bandഥX௧^^^^. The sampling operation can be written as:ഥ X ൌ ^^ ^ ^^^^^, ^^^ (5)whereഥX is predicted bin value, ^^^^ is the sampling function determined by pre-chosendistribution and ^^ is a random sample from uniform distribution. For example, in a Laplace distribution case,^^ ൌ െ^^ ∗ ^^^^^^^^^^^^ ∗ ^^^^^^^1 െ 2 ∗ |^^|^,^^~^^^െ0.5, 0.5^. (6)

[0089] To reduce accumulation of sampling error, ^^^^ may be adapted with “truncation” and “temperature” (e.g. weighting on ^^). In one implementation, “truncation” is done bysampling ^^~U^െ0.49, 0.49^ which bounds sampling output to ^^^ െ 4 ∗ ^^, ^^ ^ 4 ∗ ^^^. In anotherembodiment, µ is taken directly (max sampling). The “temperature” may be done by multiplyingweight ^^ on ^^, and in one implementation the weight ^^ can be controlled by prior knowledge about the target signal, including e.g. spectral envelope and band tonality.

[0090] The neural network system 10 embodies a predictor as shown in FIG.1A, and may advantageously be conditioned by suitable conditioning signal, thereby forming a conditioned prediction:^^^^^^ ൌ ∏^ ∏௧ ^^^^^௧^^^^ | ^^^…௧ି^^1 … ^^ ^ ^^^, ^^௧^1 … ^^ െ 1^, ^^^ , (7)where ^^ represents the conditioning signal, including e.g. quantized (or otherwise distorted)frequency coefficients ^^^ .

[0091] It is noted that the same neural network system 10 (with the noise generator and application stage 51, 52a, 52b) is used during training and during inference. Hereby, noise is not only added during training to perturb the first and second original coefficients but also during inference with the neural network system 10.

[0092] The type of noise (e.g. the noise spectral shape) introduced may be the same during inference and during training. Alternatively, it is envisaged that different types of noise are used during training and during inference. For example, while the neural network system 10 remains unchanged more noise (i.e. a higher noise level) may be introduced in at least one frequency band during training compared to during inference.

[0093] In some implementations, a different spectral shape may be used during inference compared to during training. That is, different noise level functions ^^^^^^^^^^^^^^^, ^^௧^^^^^^^^^^^ may be employed during inference and training. In one example, both noise level functions areconstant at a common noise level up to the knee frequency ^^^^^^ meaning that ^^^^^^^^^^^^^^^ ൌ^^௧^^^^^^^^^^^ when ^^ ^ ^^^^^^ whereby for frequencies above the knee frequency ^^^^^^ the noiselevel function ^^௧^^^^^^^^^^^ for training for increases more rapidly than that for inference,resulting in ^^^^^^^^^^^^^^^ ^ ^^௧^^^^^^^^^^^ when ^^ ^ ^^^^^^.

[0094] FIG.4 shows a generative model 40 for generating a target media signal. The generative model 40 in FIG.4 includes a self-generating neural network system 30 according to figure 3, and a first and second conditioning neural network 41, 44.

[0095] The first conditioning neural network 41 is trained to predict a set of conditioning variables given first conditioning information 42 describing the target media signal. The first conditioning network 41 is here a 2-D convolutional neural network with a 2-D kernel (operating in the frequency direction and time direction).

[0096] In the illustrated case the first conditioning information 42 is two-channel and includes quantized frequency coefficients and a set of perceptual model coefficients. The quantized frequency coefficients ^^^௧..௧ା^represent a time frame t of the target media signal, and nlook-ahead frames. The set of perceptual model coefficients ^^^^^^^^^^ may be derived from a perceptual model, such as those occurring in audio codec systems. The perceptual model coefficients ^^^^^^^^^^ are computed per band and are preferably mapped onto the same resolution as the frequency coefficients to facilitate processing.

[0097] In the illustrated embodiment, the first conditioning network 41 is configured to concatenate ^^^௧..௧ା^and ^^^^^^^^^^, and the first conditioning network 41 is configured to take the concatenated input and provide an output with a dimension which is two times the internal dimension of the neural network system 30 (e.g.2x1024 in the present example).

[0098] The second conditioning neural network 44 is trained to predict a second set of conditioning variables given second conditioning information 45 describing a window sequence of the target media signal. In some implementations, the transform window size used (on the encoder side) to transform the time domain target media signal into a time-frequency domain (such as an MDCT domain) is adaptable allowing it to switch between long and short temporal windows, each window covering a number of time domain samples. For example, each short window covers 192 samples, and each long window covers 768 samples. Additionally, long or short vectors of coefficients may be used to represent each window. For example, in the time- frequency domain (e.g. the MDCT domain) 192 frequency bins or 768 frequency bins may be used to represent a window.

[0099] The type of window (length in number of time samples and number of frequency bins) used to form each frame may be included in the second conditioning information 45 which is provided to the generative model 40 via the second conditioning neural network 44. The type of windows may e.g. be signaled in the side information as one-hot coded vectors. For example, if there are two types of transform window length (short and long) and two types MDCT vector lengths (short and long) the condition information 45 may comprise a four element one-hot vector indicating which, out of these four window types, which has been used when encoding each frame.

[0100] In some implementations, when a frame of the short MDCT vector length is received it is up-sampled (e.g. by implementing a sample and hold up-sampling) to the long MDCT vector length prior to being used as conditioning information.

[0101] In some implementations, the spectral resolution of the noise sampling is that of the short MDCT vector length. To convert the noise sampling to the long MDCT vector length the sampled noise values may be up-sampled in the frequency dimension (e.g. by implementing a sample and hold up-sampling). Of course this is only one example, and it is envisaged that, for example, the noise can be sampled at the frequency resolution of the long MDCT vector length directly.

[0102] The second conditioning neural network 44 takes the (optionally one-hot coded) window sequence as an input and projects the information to an output with a dimension which is two times the internal dimension of the neural network system 30 (e.g.2x1024 in the present example). The output from the second conditioning neural network is provided to summation point 46 and summed with the output of the first conditioning neural network 41.

[0103] The generative model may operate using only the first conditioning neural network 41 and the first conditioning information 42 or using both conditioning neural networks 41, 44 and both types of conditioning information 42, 45.

[0104] A splitter 43 is arranged to split the “double-length” output channel from the conditioning neural networks 41, 44 along the feature channel dimension. One half of the output variables is then added or concatenated with the input variables 12 (at summation point 121) connected to the time predicting recurrent neural network 13. The second half of the output variables are then added or concatenated to the hidden states 14 output by the neural network 13 (at summation point 141) of the time predicting. It has been empirically shown that splitting operation helps overall optimization performance.

[0105] While summation points 121 and 141 illustrated as additive summation points it is understood that each of these summation points 121, 141 may alternatively denote tensor concatenation. Both summation points 121, 141 may involve addition or both summation points 121, 141 may involve tensor concatenation. It is also envisaged that one of the summation points 121, 141 involves additive summation and that the other summation point 121, 141 involves concatenation. For example, summation point 121 comprises addition and summation point 141 comprises tensor concatenation.

[0106] Alternatively, the conditioning networks 41, 44 are configured to operate in the same dimension as the predictor 40, and output only 1024 output variables. In that case, no splitter is used, and the same conditioning variables are provided to both recurrent neural networks 13, 19.

[0107] Again with reference to FIG.5, training of the generative model 40 can also be done in “teacher forcing mode”. First, in step S1, ground truth frequency coefficients representing an “actual” (known) media signal are provided as conditioning information to the conditioning network 41. In this case, the frequency coefficients are first quantized, or otherwise distorted, in the same way as they would be in the actual implementation. At step S2a noise is generated and applied to perturb at least one of the first and second original coefficients. At step S2b the output parameters (e.g. describing a probability distributions of the binsഥX௧^^^^) of a current time frame are predicted.

[0108] The method then goes to step S3 where the predicted binsഥX௧^^^^ are compared to the actual bins ^^௧^^^^ of the actual signal in order to determine a training measure. Finally, in step S4, the parameters (weights and bias) of the various neural networks 11, 13, 15, 18, 19, 21, 22 and 41 are chosen such that the training measure is minimized. As an example, the training measure which should be minimized may be the negative log-likelihood (NLL), e.g. in the case of Laplace distribution:^^^^^^ ൌ log^2 ∗ ^^^ ^|ఓି௬| ^ (8)

[0109] system and y is theor mixture distribution model.

[0110] The (trained) generative model 40 may advantageously be implemented in a decoder, e.g. in order to enhance a quantized (or otherwise distorted) input signal. Specifically, decoding performance may be improved with the same amount or even reduced amount of coding parameters. For example, spectral voids in the input signal may be filled by the neural network. As mentioned, the generative model may operate in the transform domain, which may be particularly useful in a decoder.

[0111] In some implementations, when the generative model 40 is implemented in a decoder, the noise generator and / or noise applicator is configured to apply a noise level for at least one frequency or a frequency interval which differs from what has used during training. Advantageously, a lower noise level is used for the at least one the frequency or the frequency interval when the generative model is implemented in the decoder.

[0112] In use, the generative model 40 operates as illustrated in FIG.6. First, in step S11, conditioning information, e.g. a set of quantized frequency coefficients and perceptual model data received by a decoder, is provided to the conditioning networks 41, 44. Then, in step S12a noise is generated by the noise generator 51 and applied to the second original input coefficients to form second input coefficients which are provided to the frequency predicting portion. At step S12b output parameters representing the frequency coefficientsഥX௧^^^^ of a specific band b of a current frame t are predicted and provided as input to the frequency predicting portion 9. The method then goes to step S14 and this step involves repeating steps S12a, S12b and S13 for each frequency band in the current frame. In step S15a, noise is generated with the noise generator 51 and applied to the predicted frequency coefficients of the entire frameഥX௧(serving as the first original coefficients) to form the second input coefficients. The second input coefficients are then provided to the time predicting portion 8 at step S15b thereby enabling continued prediction of the next frame.

[0113] In some implementations, noise is only added to one of the first and second original coefficients whereby one of steps S12a and S15a may be skipped. For example, noise is only added to the first original coefficients at step S15a whereby step S12a is skipped.

[0114] In other implementations, noise is added to both the first and second original coefficients whereby both steps S12a and S15a are performed.

[0115] As indicated above, the noise level and / or the noise shape of the noise added during training differs from the noise level and / or noise shape of the noise added during inference. To this end, operating the generative model 40 to continuously predict new frames may comprise step S16 of obtaining noise level indicating a first noise level of at least one specific frequency used during training of the neural network system of the generative model. Steps S12a and S15a may further comprise generating and applying noise during operation of the generative model which has a second noise level at the specific frequency, wherein the second noise level is different from the first.

[0116] In some implementations, the second noise level is higher than the first noise level. As a further example, the noise level used for a range of frequencies during training is obtained at step S16 and at step S12a, S15b noise is generated and applied which has a noise level which is higher for each frequency in said range of frequencies.

[0117] While it also is possible for the generative model to receive information indicating the noise level to apply (e.g. as side information in the bitstream) it is noted that this is not necessary. To conserve bitrate the noise level to apply is predetermined and different from what was used during training.

[0118] In some implementations, it is envisaged that information indicating the spectral shape of the noise and / or the noise level is received by the generative model as side information. Hereby, the noise level and / or the spectral shape of the noise to be applied by the noise generator 51 and the noise applicator 52a, 52b may be based on this noise side information.

[0119] Turning back to FIG.4 showing the generative model 40 it is envisaged that in addition to, or as an alternative to, the window sequence 45 and the perceptual envelope pEnvq the generative model 40 may obtain side information indicating the noise level and / or the spectral shape of the noise to be applied. This noise side information may be received for each frame of quantized coefficients or more seldom, such as once for a plurality of frames.

[0120] The noise side information may be determined on the encoder side where an appropriate noise level is determined for each frame, or each group of frames, and then transmitted to the generative model 40 on the decoder side whereby the generative model 40 applies the noise indicated by the side information for the relevant frame, or group of frames.Hereby, the noise level and / or noise spectral shape may dynamically change over the course of many frames as dictated by the noise side information.

[0121] FIG.7 shows a schematic block diagram of an example electronic device or architecture 200 (e.g., an apparatus 200) suitable for implementing example embodiments of the present disclosure. Architecture 200 includes but is not limited to servers and client devices, systems, and methods as will be described in reference to FIGS.2-4. As shown, the architecture 200 includes central processing unit (CPU) 201 which is capable of performing various processes in accordance with a program stored in, for example, read only memory (ROM) 202 or a program loaded from, for example, storage unit 208 to random access memory (RAM) 203. The CPU 201 may be, for example, an electronic processor 201, which may include one or more processor cores, and in some examples the processor 201 may be multiple processors. In RAM 203, the data required when CPU 201 performs the various processes is also stored, as required. CPU 201, ROM 202 and RAM 203 are connected to one another via bus 204. Input / output (I / O) interface 205 is also connected to bus 204.

[0122] The following components are connected to I / O interface 205: input unit 206, that may include a keyboard, a mouse, or the like; output unit 207 that may include a display such as a liquid crystal display (LCD) and one or more speakers; storage unit 208 including a hard disk, or another suitable storage device; and communication unit 209 which may include a network interface card such as a network card (e.g., wired or wireless).

[0123] In some implementations, input unit 206 includes one or more microphones in different positions (depending on the host device) enabling capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0124] In some implementations, output unit 207 include systems with various number of speakers. Output unit 207 (depending on the capabilities of the host device) can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).

[0125] In some embodiments, communication unit 209 is configured to communicate with other devices (e.g., via a network). Drive 210 is also connected to I / O interface 205, as required. Removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive or another suitable removable medium is mounted on drive 210, so that a computer program read therefrom is installed into storage unit 208, as required. A person skilled in the art would understand that although apparatus 200 is described as including the above-described components, in real applications, it is possible to add, remove, and / or replace some of these components and all these modifications or alteration all fall within the scope of the present disclosure.

[0126] In accordance with example embodiments of the present disclosure, the processes described above may be implemented as computer software programs or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program including program code for performing methods. In such embodiments, the computer program may be downloaded and mounted from the network via the communication unit 209, and / or installed from the removable medium 211, as shown in FIG.7.

[0127] Generally, various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits (e.g., control circuitry), software, logic or any combination thereof. For example, the units discussed above can be executed by control circuitry (e.g., CPU 201 in combination with other components of FIG.7), thus, the control circuitry may be performing the actions described in this disclosure. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, a processor and / or other computing device(s), which may include control circuitry. While various aspects of the example embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representation, it will be appreciated that the blocks, apparatus, systems, techniques, or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0128] Additionally, various blocks shown in the flowcharts may be viewed as method steps, and / or as operations that result from operation of computer program code, and / or as a plurality of coupled logic circuit elements constructed to carry out the associated function(s). For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program containing program codes configured to carry out the methods as described above.

[0129] In the context of the disclosure, a machine-readable medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine- readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transitory and may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] Computer program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to one or more processors of a general-purpose computer, special purpose computer, or other programmable data processing apparatus that has control circuitry, such that the program codes, when executed by one or more processors of the computer or other programmable data processing apparatus, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may execute entirely on a computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server or distributed over one or more remote computers and / or servers.

[0131] Various aspects of the present disclosure may be appreciated from the following Enumerated Example Embodiments (EEEs): EEE1. A computer implemented neural network system (10) for predicting frequency coefficients of a media signal, the neural network system comprising: a time predicting portion (8) comprising at least one neural network trained to predict a first set of output variables (16) representing a specific frequency band of a current time frame of the media signal given first input coefficients (^^^௧ି^) representing one or several previous time frames of the media signal,a frequency predicting portion (9) comprising at least one neural network trained to predict a second set of output variables (20) representing a specific frequency band given second input coefficients (^^^௧ ^^^ െ 1^ … ,^^^௧ ^1^) representing one or several lower frequency bands to the specific frequency band in said current time frame, a noise generation stage (51), configured to generate noise, a noise application stage (52), configured to perform at least one of: - obtain first original coefficients ^^^௧ି^) representing one or several previous time frames of the media signal and perturb the first original coefficients with the noise to form the first input coefficients (^^^௧ି^), and -obtain second original coefficients (^^௧^^^ െ 1^… ,^^௧^1^) representing one orseveral lower frequency bands to the specific frequency band in said current time frame and perturb the second original coefficients with the noise to form the second input coefficients (^^^^^^ െ 1^௧^, ...^^௧^1^), andan output stage (21, 22) configured to provide a set of output parameters representing the frequency coefficients of said specific frequency band of said current time frame, based on said first and second set of output variables. EEE2. The neural network system according to EEE1, wherein the noise application stage is configured to form both the first and the second input coefficients by perturbing the first and second original coefficients with noise, respectively. EEE3. The neural network system according to EEE1, wherein the noise application stage (52) is configured to obtain the first original coefficients ^^^௧ି^) and perturb the first original coefficients with the noise to form the first input coefficients (^^^௧ି^), and wherein thesecond original coefficients (^^ ^^^ െ 1^, ...^^ ^1^) and ^௧ ௧ the second (^^௧ ^^^ െ^1^, ...^^௧^1^) are equal. EEE4. The neural network system according to EEE1, wherein the noise applicationstage (52) is configured to obtain the second original coefficients (^^௧^^^ െ 1^, ...^^௧^1^) andperturb the second original coefficients with the noise to form the second input coefficients (^^^௧ ^^^ െ 1^, ...^^^௧^1^), and wherein the first original coefficients ^^^௧ି^) and the first input coefficients (^^^௧ି^) are equal. EEE5. The neural network system according to any one of the preceding EEEs, wherein the noise application stage is configured to perturb the first and / or second original coefficients with noise of a noise level which varies with a frequency associated with each original coefficient. EEE6. The neural network system according to EEE5, wherein the noise application stage is configured to perturb the first and / or second original coefficients with a higher noise level for an original coefficient associated with a higher frequency and with a lower noise level for an original coefficient associated with a lower frequency. EEE7. The neural network according to EEE5 or EEE6, wherein the noise application stage is configured to perturb the first and / or second original coefficients with a noise level according to a noise level function indicating a noise level as a function of frequency.EEE8. The neural network according to EEE7, wherein the noise level function is a monotonically non-decreasing function with frequency. EEE9. The neural network according to EEE7 or EEE8, wherein the noise level function is constant for frequencies below a predetermined knee-frequency and increasing for frequencies above the predetermined knee-frequency. EEE10. The neural network system according to any one of EEE7 to EEE9, wherein the noise level function is a perceptual noise level function proportional to an absolute threshold of hearing in quiet. EEE11. The neural network system according to any one of the preceding EEEs, wherein said first set of output variables (16), predicted by the time predicting portion (8), are used as the second original coefficients for the frequency predicting portion (9), or wherein said second set of output variables (20), predicted by the frequency predicting portion (9), are used as the first original coefficients for the time predicting portion (8). EEE12. The neural network system according to any one of the preceding EEEs, wherein the time predicting portion comprises: an input stage (11) comprising a neural network trained to predict a first set of input variables given the first input coefficients of a preceding time frame of said media signal. EEE13. The neural network system according to EEE12, wherein the time predicting portion further comprises: a time predicting recurrent neural network (13) comprising a plurality of neural network layers, said time predicting recurrent neural network being trained to predict an intermediate set of output variables representing the current time frame, given said first set of input variables representing a preceding time frame of the media signal, and a band mixing neural network (15) trained to predict said first set of output variables, wherein variables in the intermediate set are formed by mixing variables in said intermediate set representing said time predicted frequency band and a plurality of adjacent frequency bands. EEE14. The neural network system according to any one of EEE11 to EEE13, wherein the frequency predicting portion includes:a frequency predicting recurrent neural network (19) comprising a plurality of neural network layers, said frequency predicting neural network being trained to predict said second set of output variables (20), given a sum of said first set of output variables (16) and a second set of input variables representing the second input coefficients. EEE15. A generative model for generating a target media signal, comprising: a neural network system (10) according to EEE13, and a conditioning neural network (41) trained to predict a set of conditioning variables given conditioning information describing the target media signal, said time predicting recurrent neural network (13) being configured to combine said first set of input variables with at least a subset of said set of conditioning variables. EEE16. The generative model according to EEE15, wherein the conditioning information comprises at least one of: quantized frequency coefficients describing the target media signal, a perceptual envelope of the target media signal and data describing a window sequence of the media signal. EEE17. The generative model according to EEE15 or EEE16, wherein the neural network system includes a frequency predicting recurrent neural network (19) according to EEE14, and wherein said frequency predicting recurrent neural network (19) is configured to combine said sum with at least a subset of said set of conditioning variables. EEE18. A decoder comprising a generative model according to any one of EEE15 to EEE17. EEE19. A computer program product comprising computer readable program code portions which, when executed by a computer, implement a generative model according to any one of EEE15 to EEE17. EEE20. A method for training the generative model according to any one of EEE15 to EEE17, comprising: providing a description of an actual media signal as conditioning information to the conditioning neural network,predicting, using the neural network system, a set of frequency coefficients representing a specific frequency band of a current time frame, minimizing a measure of the predicted set of frequency coefficients with respect to a true set of frequency coefficients representing the specific frequency band of the current time frame of the actual media signal. EEE21. A method for generating an enhanced media signal using a generative model according to any one of EEE15 to EEE17, comprising: obtaining a training noise level, indicating a noise level of noise used to perturb the first and / or second original coefficients in at least one frequency interval by the noise application stage during training of the generative model, generating an enhanced media signal by: - providing conditioning information to the conditioning neural network, - for each frequency band of a current time frame, using said frequency predicting recurrent neural network to predict the frequency coefficients of this frequency band, and providing said set of frequency coefficients to the frequency predicting recurrent neural network as said second original coefficients, - providing the predicted sets of frequency coefficients representing all frequency bands of the current frame to the time predicting recurrent neural network as said first original coefficients, wherein while generating the enhanced media signal the noise application stage is configured to perturb the first and / or second original coefficients with noise of an inference noise level in said at least one frequency interval, wherein the inference noise level is different from the training noise level. EEE22. The method of EEE21, wherein the inference noise level is lower than the training noise level.

Claims

CLAIMS 1. A computer implemented neural network system (10) for predicting frequency coefficients of a media signal, the neural network system comprising: a time predicting portion (8) comprising at least one neural network trained to predict a first set of output variables (16) representing a specific frequency band of a current time frame of the media signal given first input coefficients (^^^௧ି^) representing one or several previous time frames of the media signal, a frequency predicting portion (9) comprising at least one neural network trained to predict a second set of output variables (20) representing a specific frequency band given second input coefficients (^^^௧ ^^^ െ 1^ … ,^^^௧ ^1^) representing one or several lower frequency bands to the specific frequency band in said current time frame, a noise generation stage (51), configured to generate noise, a noise application stage (52), configured to perform at least one of: - obtain first original coefficients ^^^௧ି^) representing one or several previous time frames of the media signal and perturb the first original coefficients with the noise to form the first input coefficients (^^^௧ି^), and -obtain second original coefficients (^^௧^^^ െ 1^… ,^^௧^1^) representing one orseveral lower frequency bands to the specific frequency band in said current time frame and perturb the second original coefficients with the noise to form the second input coefficients (^^^௧ ^^^ െ 1^, ...^^^௧^1^), and an output stage (21, 22) configured to provide a set of output parameters representing the frequency coefficients of said specific frequency band of said current time frame, based on said first and second set of output variables.

2. The neural network system according to claim 1, wherein the noise application stage is configured to form both the first and the second input coefficients by perturbing the first and second original coefficients with noise, respectively.

3. The neural network system according to claim 1, wherein the noise application stage (52) is configured to obtain the first original coefficients ^^^௧ି^) and perturb the first original coefficients with the noise to form the first input coefficients (^^^௧ି^), and wherein the secondoriginal coefficients (^^ ^^^ െ 1^, ...^^ ^1^) ^௧ ௧ and the second input coefficients (^^௧ ^^^ െ1^, ...^^^௧ ^1^) are equal.

4. The neural network system according to claim 1, wherein the noise application stage(52) is configured to obtain the second original coefficients (^^௧^^^ െ 1^, ...^^௧^1^) and perturb thesecond original coefficients with the noise to form the second input coefficients (^^^௧ ^^^ െ1^, ...^^^௧ ^1^), and wherein the first original coefficients ^^^௧ି^) and the first input coefficients (^^^௧ି^) are equal.

5. The neural network system according to any one of the preceding claims, wherein the noise application stage is configured to perturb the first and / or second original coefficients with noise of a noise level which varies with a frequency associated with each original coefficient.

6. The neural network system according to claim 5, wherein the noise application stage is configured to perturb the first and / or second original coefficients with a higher noise level for an original coefficient associated with a higher frequency and with a lower noise level for an original coefficient associated with a lower frequency.

7. The neural network system according to claim 5 or 6, wherein the noise application stage is configured to perturb the first and / or second original coefficients with a noise level according to a noise level function indicating a noise level as a function of frequency.

8. The neural network system according to claim 7, wherein the noise level function is a monotonically non-decreasing function with frequency.

9. The neural network system according to claim 7 or 8, wherein the noise level function is constant for frequencies below a predetermined knee-frequency and increasing for frequencies above the predetermined knee-frequency.

10. The neural network system according to any one of claims 7 – 9, wherein the noise level function is a perceptual noise level function proportional to an absolute threshold of hearing in quiet.

11. The neural network system according to any one of the preceding claims, wherein said first set of output variables (16), predicted by the time predicting portion (8), are used as the second original coefficients for the frequency predicting portion (9), orwherein said second set of output variables (20), predicted by the frequency predicting portion (9), are used as the first original coefficients for the time predicting portion (8).

12. The neural network system according to any one of the preceding claims, wherein the time predicting portion comprises: an input stage (11) comprising a neural network trained to predict a first set of input variables given the first input coefficients of a preceding time frame of said media signal.

13. The neural network system according to claim 12, wherein the time predicting portion further comprises: a time predicting recurrent neural network (13) comprising a plurality of neural network layers, said time predicting recurrent neural network being trained to predict an intermediate set of output variables representing the current time frame, given said first set of input variables representing a preceding time frame of the media signal, and a band mixing neural network (15) trained to predict said first set of output variables, wherein variables in the intermediate set are formed by mixing variables in said intermediate set representing said time predicted frequency band and a plurality of adjacent frequency bands.

14. The neural network system according to any one of claims 11 - 13, wherein the frequency predicting portion includes: a frequency predicting recurrent neural network (19) comprising a plurality of neural network layers, said frequency predicting neural network being trained to predict said second set of output variables (20), given a sum of said first set of output variables (16) and a second set of input variables representing the second input coefficients.

15. A generative model for generating a target media signal, comprising: a neural network system (10) according to claim 13, and a conditioning neural network (41) trained to predict a set of conditioning variables given conditioning information describing the target media signal, said time predicting recurrent neural network (13) being configured to combine said first set of input variables with at least a subset of said set of conditioning variables.

16. The generative model according to claim 15, wherein the conditioning information comprises at least one of: quantized frequency coefficients describing the target media signal, aperceptual envelope of the target media signal and data describing a window sequence of the media signal.

17. The generative model according to claim 15 or 16, wherein the neural network system includes a frequency predicting recurrent neural network (19) according to claim 14, and wherein said frequency predicting recurrent neural network (19) is configured to combine said sum with at least a subset of said set of conditioning variables.

18. A decoder comprising a generative model according to any one of claims 15 - 17.

19. A computer program product comprising computer readable program code portions which, when executed by a computer, implement a generative model according to one of claims 15 -17.

20. A method for training the generative model according to any one of claims 15 - 17, comprising: providing a description of an actual media signal as conditioning information to the conditioning neural network, predicting, using the neural network system, a set of frequency coefficients representing a specific frequency band of a current time frame, minimizing a measure of the predicted set of frequency coefficients with respect to a true set of frequency coefficients representing the specific frequency band of the current time frame of the actual media signal.

21. A method for generating an enhanced media signal using a generative model according to any one of claims 15 - 17, comprising: obtaining a training noise level, indicating a noise level of noise used to perturb the first and / or second original coefficients in at least one frequency interval by the noise application stage during training of the generative model, generating an enhanced media signal by: - providing conditioning information to the conditioning neural network, - for each frequency band of a current time frame, using said frequency predicting recurrent neural network to predict the frequency coefficients of this frequency band, and providing said set of frequency coefficients to the frequency predicting recurrent neural network as said second original coefficients,- providing the predicted sets of frequency coefficients representing all frequency bands of the current frame to the time predicting recurrent neural network as said first original coefficients, wherein while generating the enhanced media signal the noise application stage is configured to perturb the first and / or second original coefficients with noise of an inference noise level in said at least one frequency interval, wherein the inference noise level is different from the training noise level.

22. The method of claim 21, wherein the inference noise level is lower than the training noise level.

Images