General media neural network predictor and generative model comprising such predictor

Abstract

A neural network system for predicting frequency coefficients of a media signal, the neural network system comprising a temporal prediction part comprising at least one neural network trained to predict a first set of output variables representing a particular frequency band of a current time frame given the coefficients of one or several preceding time frames, and a frequency prediction part comprising at least one neural network trained to predict a second set of output variables representing the particular frequency band given the coefficients of one or several frequency bands adjacent to the particular frequency band in the current time frame. Such a neural network system forms a predictor capable of capturing both the time dependency and the frequency dependency occurring in a time-frequency tile of a media signal.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 092,552, filed October 16, 2020, and European Patent Application No. 20206729.4, filed November 10, 2020, both of which are incorporated by reference herein in their entirety. TECHNICAL FIELD

[0003] The present invention relates to a generative model for media, in particular for audio. In particular, the present invention relates to a computer-implemented neural network system for predicting frequency coefficients representing frequency content of a media signal. BACKGROUND

[0004] Generative models of high-quality media, in particular audio, can enable many applications. Raw waveform generative models have been shown to successfully enable high-quality audio within certain signal classes (e.g., speech and piano), but the quality of general audio is still lacking.

[0005] Attempts to move away from the raw waveform domain have recently been made, for example as discussed in the article “MelNet: A Generative Model for Audio in the Frequency Domain” by Vasquez and Lewis, 2019.

[0006] Nonetheless, further improvements would be beneficial. SUMMARY

[0007] Based on the above, it is therefore an object of the present invention to provide an improved generative model for general media, in particular for general audio (i.e., not just audio of specific classes, like speech or piano music, but general audio).

[0008] According to a first aspect of the application, this and other objects are achieved by a neural network system for predicting frequency coefficients of a media signal, the neural network system comprising: a temporal prediction part comprising at least one neural network trained to predict a first set of output variables representing a particular frequency band of a current time frame given the coefficients of one or several preceding time frames; and a frequency prediction part comprising at least one neural network trained to predict a second set of output variables representing the particular frequency band given the coefficients of one or several frequency bands adjacent to the particular frequency band in the current time frame; and an output stage configured to provide a set of frequency coefficients representing the particular frequency band of the current time frame based on the first set of output variables and the second set of output variables.

[0009] Such a neural network system forms a predictor capable of capturing both the time dependency and the frequency dependency occurring in a time-frequency tile of a media signal. The frequency prediction part is designed to capture the frequency dependency, e.g. a harmonic structure.

[0010] Such a predictor has shown promising results as a neural network decoder in audio coding applications. In addition, such a neural network can be used in other signal processing applications, like bandwidth extension, packet loss concealment and speech enhancement.

[0011] In principle, the time and frequency based predictions can be performed in any order, or even combined. However, in a typical on-line application, in case of a frame-by-frame processing, the temporal prediction is usually performed first (on multiple previous frames) and the output of this prediction is used for the frequency prediction.

[0012] According to one embodiment, the temporal prediction part comprises a temporal prediction recurrent neural network comprising a plurality of neural network layers, the temporal prediction recurrent neural network being trained to predict a set of intermediate output variables representing the current time frame given a set of first input variables representing a preceding time frame of the media signal.

[0013] Similarly, according to some embodiments, the frequency prediction part comprises a frequency prediction recurrent neural network comprising a plurality of neural network layers, the frequency prediction neural network being trained to predict the second set of output variables given a sum of the first set of output variables and a set of second input variables representing a lower frequency band of the current time frame.

[0014] Recurrent neural networks have shown to be particularly useful in this context.

[0015] The time prediction part can also be a frequency band hybrid neural network, which is trained to predict the first set of output variables, wherein the variables in the intermediate set are formed by mixing the variables in the intermediate set that represent the specific frequency band and multiple adjacent frequency bands.

[0016] This type of band-mixing neural network performs cross-band predictions, thereby avoiding (or at least reducing) aliasing distortion.

[0017] Each frequency coefficient can be represented by a set of distribution parameters, wherein the set of distribution parameters is configured to parameterize the probability distribution of the coefficient. The probability distribution can be one of a Laplace distribution, a Gaussian distribution, and a logistic distribution.

[0018] A second aspect of the invention relates to a generative model for generating a target media signal, the generative model comprising a neural network system according to the first aspect, and a conditioning neural network configured to predict a set of conditioning variables given conditioning information describing the target media signal.

[0019] In the case where the time prediction component includes a time prediction recurrent neural network, the time prediction recurrent neural network can be configured to combine the first set of input variables with at least one subset of the conditioning variable set.

[0020] In cases where the frequency prediction component includes a frequency prediction recurrent neural network, the frequency prediction recurrent neural network can be configured to combine the summation with at least one subset of the set of conditioning variables.

[0021] Conditioning information may include quantized (or otherwise distorted) frequency coefficients, thereby allowing neural network systems to predict dequantized (or otherwise enhanced) frequency coefficients representing media signals.

[0022] In some applications, such as neural network-based decoders in general audio codecs, quantized frequency coefficients can be combined with a set of perceptual model coefficients derived from a perceptual model. This conditioning information can further improve predictions.

[0023] In an empirical study, this generative model has been implemented in a general audio coding application, such that it receives quantized MDCT bins as input and predicts dequantized MDCT bins. It has been shown that spectral holes are filled with plausible structure and quantization errors are removed in the prediction. In a MUSHRA-style subjective evaluation of a "deep audio codec" operating at 20 kb / s using the generative model according to the second aspect of the application, the "deep audio codec" was rated overall comparable to a 32 kb / s MPEG-4 AAC codec compared to several prior art codecs at different bit rates. This represents a bit rate saving of 37%.

[0024] A third aspect of the application relates to a method for inferring an enhanced media signal using a generative model according to the second aspect of the application.

[0025] A fourth aspect of the application relates to a method for training a neural network system according to the first aspect of the application. BRIEF DESCRIPTION OF DRAWINGS

[0026] The application will be described in more detail with reference to the attached drawings, which show a currently preferred embodiment of the application.

[0027] Figures la-lb A high-level structure of a time / frequency predictor according to an embodiment of the application is shown.

[0028] Figure 2 A neural network system implementing the structure of Figure la is shown.

[0029] Figure 3 A neural network system in Figure 2 operating in a self-generative mode is shown.

[0030] Figure 4 A generative model including a neural network in Figure 2 is shown. DETAILED DESCRIPTION

[0031] Figure la and Figure lbTwo examples of high-level structures of a time / frequency predictor 1 according to embodiments of the application are schematically illustrated. The predictor operates on frequency coefficients representing the frequency content of a media (e.g., audio) signal. The frequency coefficients can correspond to bins of a time-frequency transform of the media signal, such as a discrete cosine transform (DCT) or modified discrete cosine transform (MDCT). Alternatively, the frequency coefficients can correspond to samples of a filter bank representation of the media signal, e.g., a Quadrature Mirror Filter (QMF) filter bank.

[0032] In Figure la , the frequency coefficients (sometimes referred to as "bins" herein) of previous time frames are first grouped into a pre-selected number B of frequency bands. Then, the predictor 1 predicts the bins 2 in a target frequency band b in the current time frame t based on the frequency band context collected from all previous time frames 3. Then, the predictor 1 predicts the bins 2 in the target frequency band b based on all lower frequency bands and N higher frequency bands (i.e., frequency bands 1,..., b+N), where N is between 1 and B-b. In Figure la , N is equal to 1, i.e., only one higher frequency band b+1 is considered. Finally, the predictor predicts the bins 2 in the target frequency band b based on all lower (previously predicted) frequency bands 5 in the current time frame t.

[0033] The joint probability density Xt(b) of the frequency coefficients (e.g., MDCT bins) can be expressed as a product of conditional probabilities:

[0034]

[0035] where X t (b) denotes the set of coefficients in frequency band b at time t, N denotes the number of adjacent frequency bands on each side (higher and lower side), X 1...t-1 (1...b+N) denotes the coefficients in frequency bands 1 to b+N from time 1 to time t-1, and finally X t (1...b-1) denotes the bins in frequency bands 1 to b-1 at time t.

[0036] It is clear from the above description of the predictor in Figure la that the prediction is first done in the time dimension and then in the frequency dimension. This is normal in many applications, e.g., in an audio decoder, where typically a next frame of the signal is predicted in real-time.

[0037] However, in general, the time / frequency predictor can operate in the reverse order, e.g., if the entire signal is available off-line. This slightly less intuitive process is illustrated in Figure lb .

[0038] Here, first the bins in each lower frequency band are grouped into a set of T time frames. Then, the predictor 1' predicts the bins 2' of the target frame t in the current (next higher) frequency band b based on the frequency band context collected from all lower frequency bands 3'. Then the predictor 1' predicts the bins 2' of the target frame t based on the lower frequency bands in all preceding time frames and N subsequent (future) time frames (i.e. frames 1,..., t+1), where here N is between 1 and T-t. In Figure lb In the example implementation in Fig. 1, N again equals 1, i.e. one subsequent (future) frame is considered. Finally, the predictor predicts the bins 2' in the target frame t based on all preceding (previously predicted) time frames 5' in the current frequency band b.

[0039] An example implementation of the predictor in the neural network system 10 is illustrated in Figure la Fig. 2 in the form of a block diagram. As explained in detail below, the network system 10 has a temporal prediction part 8 and a frequency prediction part 9. Figure 2

[0040] In the temporal prediction part 8, a convolutional network 11 receives the frequency transformed coefficients (bins) X t-1 of the preceding frames and performs a convolution of the frequency bins to group them into B frequency bands 12. As an example, B equals 32. In one implementation, the convolutional network 11 is implemented as a convolutional layer with a kernel length K equal to 16 and a stride S equal to 8 (i.e. 50% overlap).

[0041] The frequency bands 12 are fed to a temporal prediction recurrent neural network (RNN) 13 comprising a set of recurrent layers (here in the form of Gated Recurrent Units, GRUs). Other recurrent neural networks can also be used, such as Long Short-Term Memory (LSTM), Quasi-Recurrent Neural Network (QRNN), Bi-directional Recurrent Unit, Continuous Time Recurrent Network (CTRNN), etc. The network 13 processes the B frequency bands individually but with shared weights, resulting in individual hidden states 14 for each frequency band of the current (predicted) time frame. Each hidden state 14 comprises a set of output variables, where the size of the set is determined by the internal dimensionality of the layers in the RNN 13. In the illustrated example, the internal dimensionality is 1024, so there are 1024 variables representing each frequency band of the current (predicted) time frame. In case of B = 32, there are thus 32 x 1024 variables output from the RNN 13.

[0042] ​The B hidden states 14 are then fed to another convolutional network 15 that mixes the variables of all lower and N higher frequency bands, i.e. adjacent hidden states, in order to achieve cross-band prediction p(X t (b) | X 1...t-1 (1...b+N)). In one embodiment, the convolutional network 15 is implemented as a single convolutional layer along the frequency band dimension with a kernel length of 2N+1 with N lower and N higher frequency bands. In another embodiment, the convolutional layer kernel length is N+2 with one lower and N higher frequency bands. The output (hidden state) 16 is again a set of B output variables, where the size of each set is determined by the internal dimension. In the present case, the network 15 again outputs 32x1024 variables.

[0043] In the frequency prediction part 9, the hidden states 16 representing the current (predicted) time frame are fed to a summation point 17. A 1x1 convolutional layer 18 receives the frequency coefficients X t (1),..., X t (b-1) of the previous frequency band and projects them onto the internal dimension of the system, i.e. 1024 in the present case.

[0044] The output of the summation point 17 is fed to a recurrent neural network (RNN) 19 comprising a set of recurrent layers, here in the form of gated recurrent units (GRU). Again, other recurrent neural networks can be used, such as long short-term memory (LSTM), quasi-recurrent neural network (QRNN), bidirectional recurrent unit, continuous-time recurrent network (CTRNN), etc. The RNN 19 takes the summed output and predicts a set of output variables (hidden state) 20 representing X t (b). Finally, two output layers 21, 22 in the form of two 1x1 convolutional layers (with output dimensions 1024 and 16, respectively) with ReLU activation before each convolutional layer are used to provide the final prediction of X t (b) | X 1...t-1 (1...b+N), X t (1...b-1) according to the final prediction scheme p(X t (b). For each new time stamp, the hidden state 20 of the RNN 19 is reset.

[0045] In one embodiment, each frequency coefficient is represented by two parameters, e.g. the system can predict the parameters μ (location) and s (scale) of a Laplace distribution. In one implementation, for the computation of the stability, log(s) is used instead of s. In other implementations, a logistic distribution or a Gaussian distribution can be chosen as the target distribution to be parameterized. Thus, the output dimension of the final output layer 22 is twice the number of bins. In the present case, the output dimension of layer 22 is 16, corresponding to eight bins in each frequency band.

[0046] In another embodiment, the frequency coefficients are parameterized as a mixture of distributions, where each parameterized distribution has an individual (normalized) weight. Then, each coefficient will be represented by (number of distributions) x (number of distribution parameters + 1) parameters. For example, in the specific case of a mixture of two Laplace distributions (two parameters per distribution), each coefficient will be represented by 2 x (2 + 1) = 6 parameters (two sets of weights (wl and w2), location (ul, u2), and scale (sl, s2), where wl + w2 = 1). Then, the output dimension of output layer 22 will be 8 x 6 = 48. The previously mentioned embodiment is the special case where there is only one distribution and the weight is equal to one.

[0047] Reference Figure 5 The training of neural network system 10 can be done in a “teacher forcing mode”. First, in step S1, ground truth frequency coefficients representing the “actual” (known) media signal are provided to convolutional network 11 and convolutional layer 18, respectively. Then, in step S2, the probability distribution of the bins is predicted for the current time frame. In step S3, the predicted bins are compared to the actual bins X t of the actual signal (b) in order to determine a training metric. Finally, in step S4, the parameters (weights and biases) of the various neural networks 11, 13, 15, 18, 19, 21, 22 are selected such that the training metric is minimized. As an example, the training metric to be minimized can be the negative log-likelihood (NLL), e.g. in the case of a Laplace distribution:

[0048]

[0049] where μ and s are the model output predictions and y is the actual bin value. In the case of a Gaussian or a mixture distribution model, the NLL looks slightly different.

[0050] Figure 3 Fig. 4 illustrates neural network system 10 in inferencing mode. In this mode, the ground truth frequency coefficients are not provided to convolutional network 11 and convolutional layer 18. Instead, in step S5, the actual media signal is provided to convolutional network 11, which predicts the probability distribution of the bins Figure 2the neural network system 10 in FIG. 1, the inference mode is also referred to as "self-generated" mode, in which the prediction is used as history to continuously generate new predictions. Figure 3 The neural network system in FIG. 1 is referred to as a self-generated predictor 30. Such a predictor can be used in an encoder to compute a prediction error based on the prediction generated by the predictor. The prediction error can be quantized and included in the bitstream as a residual error. In a decoder, the prediction result can then be added to the quantized error to obtain the final result.

[0051] The predictor 30 here comprises two feedback paths 31, 32; the first feedback path 31 is for the temporal prediction part 8 of the system and the second feedback path 32 is for the frequency prediction part 9 of the system.

[0052] More specifically, the prediction is added to the current frame that is partially predicted, so that it then comprises the frequency bands These frequency bands are provided as input to the convolutional network 18 and then to the summation point 17 in order to predict the next higher frequency band When all the frequency bands in the current frame have been predicted, the entire frame is provided as input to the convolutional network 11 in order to be able to predict the next time frame

[0053] Given that μ and s are the prediction parameters from the proposed neural network, a sampling operation 33 is needed to obtain the predicted bin value. The sampling operation can be written as:

[0054]

[0055] where, is the predicted bin value, F() is a sampling function determined by a pre-selected distribution, and u is a random sample from a uniform distribution. For example, in the case of a Laplace distribution,

[0056] F = -s * sign(u) * log(l - 2 * |u|), u ~ U(-0.5, 0.5) (4)

[0057] To reduce the accumulation of sampling errors, F() can be adapted with "truncation" and "temperature" (e.g., weighting s). In one embodiment, "truncation" is done by sampling u ~ U(-0.49, 0.49) which limits the sampling output to (μ-4*s, μ+4*s). In another embodiment, μ is directly valued (max sampling). "Temperature" can be done by multiplying the weight w with s, and in one embodiment, the weight w can be controlled by prior knowledge about the target signal, including e.g. spectral envelope and band tonality.

[0058] The neural network system 10 implements a predictor as shown in Figure la and can advantageously be conditioned by a suitable conditioning signal, thereby forming a conditioned prediction:

[0059]

[0060] where c denotes the conditioning signal, including e.g. quantized (or otherwise distorted) frequency coefficients

[0061] Figure 4 A generation model 40 for generating a target media signal using such a conditioned predictor is shown. Figure 4 The model 40 in comprises a self-generating neural network system 30 according to Figure 3 and a conditioning neural network 41.

[0062] The conditioning neural network 41 is trained to predict a set of conditioning variables given conditioning information 42 describing the target media signal. The conditioning network 41 is here a 2-D convolutional neural network with 2-D kernels (frequency direction and time direction).

[0063] In the illustrated case, the conditioning information 42 is two-channel and comprises a set of quantized frequency coefficients and perceptual model coefficients. The quantized frequency coefficients represent a time frame t and n look-ahead frames of the target media signal. The set of perceptual model coefficients pEnvQ can be derived from a perceptual model, such as the one occurring in an audio codec system. The perceptual model coefficients pEnvQ are computed per band and are preferably mapped onto the same resolution as the frequency coefficients to facilitate processing.

[0064] In the illustrated embodiment, the conditioning network is configured to cascade and pEnvQ, and the conditioning network 41 is configured to take the concatenated input and provide an output having a dimension that is twice the internal dimension of the neural network system 30 (e.g. 2x1024 in the present example). The splitter 43 is arranged to separate the "double length" output channel along the feature channel dimension. Half of the output variables are added to the input variables connected to the time prediction recurrent neural network 13. The other half of the output variables are added to the input variables connected to the frequency prediction recurrent network 19. Experience has shown that the separation operation helps the overall optimisation performance.

[0065] Alternatively, the conditioning network 41 is configured to operate on the same dimension as the predictor 40, and only outputs 1024 output variables. In this case, no splitter is required, and the same conditioning variables are provided to both recurrent neural networks 13, 19.

[0066] Referring again to Figure 5 The training of the generative model 40 can also be done in "teacher forcing mode". First, in step S1, ground truth frequency coefficients representing the "actual" (known) media signal are provided as conditioning information to the conditioning network 41. In this case, the frequency coefficients are first quantised in the same way as they would be in an actual implementation, or otherwise distorted. Then, in step S2, the probability distribution of the bins X of the current time frame is predicted. In step S3, the predicted bins X are compared to the actual bins X t of the actual signal in order to determine a training metric. Finally, in step S4, the parameters (weights and biases) of the various neural networks 11, 13, 15, 18, 19, 21, 22 and 41 are selected such that the training metric is minimised. As an example, the training metric that should be minimised can be the negative log likelihood (NLL), for example in the case of a Laplace distribution:

[0067]

[0068] where μ and s are the model output predictions, and y is the actual bin value. In the case of a Gaussian or mixture distribution model, the NLL looks slightly different.

[0069] The generative model 40 can advantageously be implemented in a decoder, for example in order to enhance a quantised (or otherwise distorted) input signal. In particular, the decoding performance can be improved with the same amount or even a reduced amount of encoding parameters. For example, spectral voids in the input signal can be filled in by the neural network. As mentioned above, the generative model can operate in the transform domain, which can be particularly useful in a decoder.

[0070] In use, the generative model 40 is applied to an input signal as follows: Figure 6The illustrated operations. First, in step Sll, conditioning information (e.g. a set of quantized frequency coefficients received by the decoder and a set of perceptual model data) is provided to the conditioning network 41. Then, in steps S12 and S13, the frequency coefficients of a particular frequency band b of the current frame t are predicted and provided as input to the frequency prediction RNN 19. In step S14, steps S12 and S13 are repeated for each frequency band in the current frame. In step S15, the predicted frequency coefficients of the entire frame are provided to the temporal prediction RNN 13, enabling the prediction of the next frame to continue.

[0071] ***

[0072] In the foregoing, possible methods of training and operating a deep learning based system for determining an indication of audio quality of an input audio sample have been described as well as possible implementations of such a system. In addition, the present disclosure also relates to apparatuses for implementing these methods. An example of such an apparatus can comprise a processor (e.g. a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application-specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination of these) and a memory coupled to the processor. The processor can be adapted to perform some or all of the steps of the methods described throughout the disclosure.

[0073] An apparatus can be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, the present disclosure will refer to any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0074] The present disclosure further relates to a program (e.g., a computer program) comprising instructions, which, when executed by a processor, cause the processor to perform some or all of the steps of the methods described herein.

[0075] Still further, the present disclosure relates to a computer-readable (or machine-readable) storage medium storing the aforesaid program. Here, the term “computer-readable storage medium” includes, but is not limited to, data storage repositories in the form of solid-state memory, optical media, and magnetic media, for example.

[0076] Unless specifically stated otherwise, as apparent from the following discussion, it is appreciated that throughout the disclosure discussions utilizing terms such as "processing," "computing," "calculating," "determining," "analyzing" or the like, can refer to the action and / or processes of a computer or computing system, or similar electronic

[0077] In a similar manner, the term "processor" can refer to any device or portion of a device that processes electronic data from registers and / or memory to transform that electronic data into other electronic data that can be directly or indirectly stored in registers and / or memory. A "computer" or a "computing platform" or "computer machine" can include one or more processors.

[0078] In one example embodiment, the methods described herein can be performed by one or more processors that accept computer-readable (also called machine-readable) code, also referred to as instructions, written in any of a variety of computer-readable languages including, but not limited to, machine code, assembly code, an object-oriented programming language such as C++, and a high-level programming language such as FORTRAN, Pascal, C, Java, BASIC, Ada, or Python. The instructions are executed by one or more processors to perform the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken is included. Thus, one example is a typical processing system comprising one or more processors. Each processor can include one or more of CPUs, graphics processing units, and programmable DSP units. The processing system further can include a memory subsystem including main RAM and / or a static RAM, and / or ROM. A bus subsystem can be included for communicating between the elements. The processing system in some embodiments can also include a storage system, such as a disk drive, optical storage device, tape drive, etc. The processing system can also include input and output devices, such as a keyboard, mouse, etc. The processing system can further include a display device, such as a liquid crystal display (LCD) or a cathode ray tube (CRT) display. The processing system can also include a storage system, such as a disk drive, optical storage device, tape drive, etc. The processing system can also include a user input device, such as a keyboard, a mouse, a stylus, a microphone, touch screen, or any other input device by which a user can give input to the processing system. The processing system can also include a user output device, such as a display, a printer, speakers, a projector, or any other output device by which the processing system can give an output to a user. The memory subsystem thus includes a computer-readable carrier medium carrying computer-readable code (e.g., software) comprising a set of instructions to cause a processing system to perform any of the methods described herein. Note also that while the computer-readable carrier medium is shown in the memory, the computer-readable carrier medium can be embodied in any

[0079] In alternative example embodiments, the one or more processors operate as a standalone device or can be connected, e.g., networked to other processor(s), in a networked deployment, the one or more processors can operate in the capacity of a server or a user machine in server-user network environment, or as a peer machine in peer-to-peer or distributed network environment. The one or more processors can form a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a cellular telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, the one or more processors can be a component of a

[0080] It should be noted that the term "machine" should also be considered to include any set of machines that individually or jointly execute a set (or more sets) of instructions to perform any or more of the methods discussed herein.

[0081] Therefore, an example embodiment of each method described herein takes the form of a computer-readable carrier medium carrying a set of instructions, such as a computer program for execution on one or more processors (e.g., one or more processors as part of a web server arrangement). Thus, as those skilled in the art will recognize, example embodiments of this disclosure can be embodied as methods, apparatus such as dedicated devices, apparatus such as data processing systems, or computer-readable carrier media (e.g., computer program products). A computer-readable carrier medium carries computer-readable code comprising a set of instructions that, when executed on one or more processors, cause one or more processors to implement the method. Therefore, aspects of this disclosure can take the form of methods, entirely hardware example embodiments, entirely software example embodiments, or example embodiments combining software and hardware aspects. Furthermore, this disclosure can take the form of a carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied in the medium.

[0082] Software can be further sent or received over a network via a network interface device. While the carrier medium is a single medium in the example embodiment, the term "carrier medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The term "carrier medium" should also be considered to include any medium capable of storing, encoding, or carrying a set of instructions for execution by one or more processors and causing one or more processors to perform any one or more methods of this disclosure. The carrier medium can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical discs, magnetic disks, and magneto-optical discs. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including conductors containing a bus subsystem. Transmission media can also take the form of acoustic or optical waves, such as those generated during radio wave and infrared data communication. For example, the term "carrier medium" should therefore be considered to include, but is not limited to, solid-state storage; computer products embodied in optical and magnetic media; media carrying propagation signals that can be detected by at least one or more processors and represent a set of instructions, which, when executed, implement a method; and transmission media in a network that carry propagation signals that can be detected by at least one of one or more processors and represent the set of instructions.

[0083] It will be understood that, in one example embodiment, the steps of the methods discussed are performed by a suitable processor (or processors) in a processing (e.g., computer) system executing instructions (computer readable code) stored in a storage device. It will also be understood that the present disclosure is not limited to any particular implementation or programming technique, and that the present disclosure can be implemented using any appropriate techniques for implementing the functionality described herein. The present disclosure is not limited to any particular programming language or operating system.

[0084] Reference throughout this disclosure to "one example embodiment", "some example embodiments" or "example embodiments" means that a particular feature, structure, or characteristic described in connection with the example embodiment is included in at least one example embodiment of the present disclosure. Thus, the appearances of the phrase "in one example embodiment", "in some example embodiments" or "in an example embodiment" in various places throughout this disclosure are not necessarily all referring to the same example embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more example embodiments, as would be apparent to one of ordinary skill in the art from this disclosure.

[0085] As used herein, the use of the ordinal adjectives "first", "second", "third", etc. to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0086] In the following claims and the description herein, the terms "comprising", "comprised of", or "which comprises" any one of these terms is an open term, which is intended to mean at least the recited elements / features, but does not exclude additional elements / features. Accordingly, when the term "comprising" is used in the claims, the term "comprising" should not be interpreted as being limited to the means or elements listed thereafter. For example, the expression "a device comprising A and B" should not be interpreted as being limited to only devices that comprise elements A and B. As used herein, any one of the terms "including", "includes" or "included" or "includes that" is also an open term, which also means at least the recited elements / features, but does not exclude additional elements / features. Accordingly, "including" is synonymous with "comprising" and means "comprising".

[0087] It is recognized that, in the above description of example embodiments of the disclosure, various features of the disclosure are sometimes grouped together in a single example embodiment / figure or description thereof for the purpose of simplifying the disclosure and aiding in the understanding of one or more of the inventive aspects. However, the methods of the disclosure should not be construed as reflecting an intention that the claim require more features than are explicitly recited in each claim. To the contrary, the inventive aspects are directed to each individual feature individually recited in each claim, as reflected in the claims below. Thus, the claims following the description are expressly incorporated into this description, wherein each claim is independently a separate example embodiment of the disclosure.

[0088] Furthermore, while some example embodiments described herein include some features of other example embodiments and exclude other features thereof, as will be understood by those of ordinary skill in the art, combinations of features of the different example embodiments are intended to be within the scope of the disclosure and form different example embodiments. For example, in the appended claims, any of the example embodiments claimed can be used in any combination.

[0089] In the description provided herein, numerous specific details are set forth. However, it is understood that example embodiments of the disclosure can be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0090] Accordingly, although embodiments which are described herein are believed to be the best of their kind, those skilled in the art will recognize that other and further modifications can be made thereto without departing from the spirit of the present disclosure, and it is intended to claim all such changes and modifications as fall within the scope of the disclosure. For example, any of the formulas given above represent only processes that can be used. Functionality can be added or deleted from the block diagrams, and operations can be interchanged among functional blocks. Steps can be added or deleted to methods described within the scope of the disclosure. In particular, different layouts can be envisioned to implement the high-level predictor structure in Figure la

[0091] Various aspects of the application can be appreciated from a review of the following enumerated exemplary embodiment (EEE) list:

[0092] EEE1. A computer-implemented neural network system for predicting frequency coefficients of a media signal, the neural network system comprising:

[0093] ​a time prediction portion comprising at least one neural network trained to predict, given the coefficients of one or several previous time frames, a first set of output variables representative of a particular frequency band of a current time frame, and

[0094] a frequency prediction portion comprising at least one neural network trained to predict, given the coefficients of one or several frequency bands adjacent to the particular frequency band in the current time frame, a second set of output variables representative of the particular frequency band,

[0095] an output stage configured to provide, based on the first set of output variables and the second set of output variables, a set of frequency coefficients representative of the particular frequency band of the current time frame.

[0096] EEE2. The computer-implemented neural network system of EEE 1, wherein the first set of output variables predicted by the time prediction portion are used as input variables for the frequency prediction portion.

[0097] EEE3. The computer-implemented neural network system of EEE 2, wherein the time prediction portion comprises:

[0098] a time prediction recurrent neural network comprising a plurality of neural network layers trained to predict, given a first set of input variables representative of a previous time frame of the media signal, an intermediate set of output variables representative of the current time frame.

[0099] EEE4. The computer-implemented neural network system of EEE 3, wherein the time prediction portion further comprises:

[0100] an input stage comprising a neural network trained to predict the first set of input variables given the frequency coefficients of a previous time frame of the media signal.

[0101] EEE5. The computer-implemented neural network system of EEE 4, wherein the time prediction portion further comprises:

[0102] a band mixing neural network trained to predict the first set of output variables, wherein the variables in the intermediate set are formed by mixing the variables in the intermediate set representative of the particular frequency band and a plurality of adjacent frequency bands.

[0103] EEE6. The computer-implemented neural network system of EEE 5, wherein the frequency prediction portion comprises:

[0104] a frequency prediction recurrent neural network comprising a plurality of neural network layers, the frequency prediction neural network being trained to predict a second set of output variables given a sum of the first set of output variables and a second set of input variables representing lower frequency bands of the current time frame.

[0105] EEE7. The computer-implemented neural network system of EEE 6, wherein the frequency prediction portion further comprises:

[0106] one or several output layers trained to provide the set of frequency coefficients based on the second set of output variables.

[0107] EEE8. The computer-implemented neural network system of EEE 1, wherein each frequency coefficient is represented by a set of distribution parameters, wherein the set of distribution parameters is configured to parameterize a probability distribution of the coefficient.

[0108] EEE9. The computer-implemented neural network system of EEE 8, wherein the probability distribution is one of a Laplace distribution, a Gaussian distribution, and a logistic distribution.

[0109] EEE10. The computer-implemented neural network system of EEE 1, wherein the frequency coefficients correspond to bins of a time-frequency transform of the media signal.

[0110] EEE11. The computer-implemented neural network system of EEE 1, wherein the frequency coefficients correspond to samples of a filter bank representation of the media signal.

[0111] EEE12. A generative model for generating a target media signal, the generative model comprising:

[0112] a computer-implemented neural network system according to EEE 3, and

[0113] a conditioning neural network trained to predict a set of conditioning variables given conditioning information describing the target media signal,

[0114] the time prediction recurrent neural network being configured to combine the first set of input variables with at least a subset of the set of conditioning variables.

[0115] EEE13. The generative model of EEE 12, wherein the neural network system comprises a frequency prediction recurrent neural network according to EEE 6, and wherein

[0116] The frequency prediction recurrent neural network is configured to combine the sum with at least a subset of the conditioning variable set.

[0117] EEE14. The generative model according to EEE 13, wherein the conditioning variable set comprises variables that are twice the internal dimension of the neural network system, and wherein the time prediction recurrent neural network and the frequency prediction recurrent neural network are each provided with half of the conditioning variables.

[0118] EEE15. The generative model according to EEE 12, wherein the conditioning information comprises a set of distortion frequency coefficients.

[0119] EEE16. The generative model according to EEE 15, wherein the conditioning information additionally comprises a set of perceptual model coefficients.

[0120] EEE17. The generative model according to EEE 12, wherein the conditioning information comprises a spectral envelope.

[0121] EEE18. The generative model according to EEE 12, wherein the conditioning neural network comprises a convolutional neural network with 2D kernels operating in frequency direction and time direction.

[0122] EEE19. A method for training a computer-implemented neural network system according to EEE 7, the method comprising the steps of:

[0123] a) providing a set of frequency coefficients representing a previous time frame of an actual media signal as the first input variable set,

[0124] b) predicting a set of frequency coefficients representing a specific frequency band of a current time frame using the neural network system,

[0125] c) minimizing a measure of the predicted set of frequency coefficients with respect to a true set of frequency coefficients representing the specific frequency band of the current time frame of the actual media signal.

[0126] EEE20. The method according to EEE 19, wherein each frequency coefficient is represented by a set of distribution parameters, wherein the set of distribution parameters parameterizes a probability distribution for each frequency coefficient.

[0127] EEE21. The method according to EEE 20, wherein the measure is a negative log likelihood, NLL.

[0128] EEE22. A method for training a generative model according to EEE 12, the method comprising the steps of:

[0129] a) providing a description of an actual media signal as conditioning information to the conditioning neural network,

[0130] b) predicting, using the neural network system, a set of frequency coefficients representing a particular frequency band of a current time frame,

[0131] c) minimizing a measure of the predicted set of frequency coefficients with respect to a true set of frequency coefficients representing the particular frequency band of the current time frame of the actual media signal.

[0132] EEE23. The method according to EEE22, wherein the description comprises a set of distorted frequency coefficients representing the actual media signal.

[0133] EEE24. The method according to EEE22, wherein each frequency coefficient is represented by a set of distribution parameters, wherein the set of distribution parameters parameterizes a probability distribution of each frequency coefficient.

[0134] EEE25. The method according to EEE24, wherein the measure is a negative log-likelihood, NLL.

[0135] EEE26. A method for obtaining an enhanced media signal using the generative model according to EEE13, the method comprising the steps of:

[0136] a) providing conditioning information to the conditioning neural network,

[0137] b) for each frequency band of a current time frame, predicting, using the frequency prediction recurrent neural network, a set of frequency coefficients representing that frequency band, and providing the set of frequency coefficients as the second set of input variables to the frequency prediction recurrent neural network,

[0138] c) providing the predicted sets of frequency coefficients representing all frequency bands of the current frame as the first set of input variables to the temporal prediction RNN.

[0139] EEE27. The method according to EEE26, wherein the conditioning information comprises a set of distorted frequency coefficients representing the actual media signal.

[0140] EEE28. The method according to EEE26, wherein each frequency coefficient is represented by a set of distribution parameters, wherein the set of distribution parameters parameterizes a probability distribution of each frequency coefficient, the method further comprising:

[0141] sampling each probability distribution to obtain a frequency coefficient value.

[0142] EEE29. A decoder, the decoder comprising a generative model according to EEE12.

[0143] EEE30. A computer program product comprising computer readable program code portions, the computer readable program code portions implementing the computer-implemented neural network system according to EEE 12 when executed by a computer.

Claims

1. A computer-implemented neural network system for predicting the frequency coefficients of a media signal, the neural network system comprising: The time prediction section includes at least one neural network trained to predict a first set of output variables representing a specific frequency band of the current time frame, given coefficients from one or more previous time frames. The frequency prediction section includes at least one neural network trained to predict a second set of output variables representing a specific frequency band, given a first set of output variables and coefficients of one or more frequency bands adjacent to the specific frequency band in the current time frame. An output stage is configured to provide a set of frequency coefficients representing the specific frequency band of the current time frame, based on the first set of output variables and the second set of output variables.

2. The neural network system according to claim 1, wherein, The time prediction component includes: A time-predictive recurrent neural network, comprising multiple neural network layers, is trained to predict an intermediate set of output variables representing the current time frame, given a first set of input variables representing the previous time frame of the media signal. A frequency band hybrid neural network, which is trained to predict a first set of output variables, wherein the variables in the intermediate set are formed by mixing variables in the intermediate set representing the specific frequency band and multiple adjacent frequency bands.

3. The neural network system according to claim 2, wherein, The time prediction component further includes: The input stage includes a neural network trained to predict the first set of input variables given the frequency coefficients of a previous time frame of the media signal.

4. The neural network system according to claim 2, wherein, The frequency prediction section includes: A frequency prediction recurrent neural network, comprising multiple neural network layers, is trained to predict a second set of output variables given a first set of output variables and a summation of a second set of input variables representing a lower frequency band of the current time frame.

5. The neural network system according to claim 4, wherein, The frequency prediction section further includes: One or more output layers, said one or more output layers being trained to provide the set of frequency coefficients based on the second set of output variables.

6. The neural network system according to any one of the preceding claims, wherein, Each frequency coefficient is represented by a set of distribution parameters, wherein the set of distribution parameters is configured to parameterize the probability distribution of the coefficient. The specific frequency band of the current time frame is obtained by sampling the probability distribution of each frequency coefficient.

7. The neural network system according to claim 1, wherein, The frequency coefficients correspond to the time-frequency transformation of the media signal, or the frequency coefficients correspond to the samples represented by the filter bank of the media signal.

8. A generative model system for generating target media signals, said generative model system include: The neural network system according to claim 2, and A conditioning neural network is trained to predict a set of conditioning variables, given conditioning information describing the target media signal, wherein the conditioning information includes quantized frequency coefficients describing the target media signal. The time-prediction recurrent neural network is configured to combine the first set of input variables with at least one subset of the set of conditioning variables.

9. The generative model system according to claim 8, wherein, The neural network system includes the frequency prediction recurrent neural network according to claim 4, and wherein, The frequency prediction recurrent neural network is configured to combine the summation with at least one subset of the set of conditioning variables.

10. The generative model system according to claim 8 or 9, wherein, The conditioning information includes at least one of the following: a set of distortion frequency coefficients, a set of perception model coefficients, and a spectral envelope.

11. A method for obtaining an enhanced media signal using the generative modeling system according to claim 9, the method comprising the steps of: a) Provide conditioning information to the conditioning neural network. b) For each frequency band in the current time frame, the frequency prediction recurrent neural network is used to predict the set of frequency coefficients representing that frequency band, and the set of frequency coefficients is provided to the frequency prediction recurrent neural network as the second set of input variables. c) The set of frequency coefficients representing the predictions of all frequency bands in the current frame is provided to the time prediction RNN as the first set of input variables.

12. A decoder comprising the generative model system according to claim 8.

13. A computer program product comprising a computer-readable program code portion that, when executed by a computer, implements the generative modeling system according to any one of claims 8 to 10.

Images