Signal processing based on neural network

Abstract

In one embodiment, a method for processing an input audio signal includes adjusting a first neural network system with a representation of the input audio signal to predict a bit rate reduced representation of the processed input audio signal, the first neural network system is trained to generate a reduced bit rate representation of a processed version of a given audio signal, wherein the reduced bit rate representation has a format associated with a predefined audio encoding process; adjusting a second neural network system with the reduced bit rate representation to predict an enhanced representation of the processed audio signal, the second neural network system being trained to generate an enhanced representation of a given reduced bit rate audio representation, wherein the reduced bit rate representation has a format associated with the predefined audio encoding process; and transforming the enhanced representation of the processed audio signal into an output audio signal.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority of the following priority applications: U.S. Provisional Application Serial No. 63 / 391,124 filed on July 21, 2022, and European Patent Application No. 22188293.9 filed on August 2, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to audio signal processing using a generative model involving a neural network system.The signal processing may for example involve signal enhancement or source separation. Background Art

[0004] Generative models involving trained neural network systems have been used in various audio signal processing applications for some time. The general approach is to train the neural network system using ground truth data, after which the trained model can be used to infer the processed signal. Specially designed neural network systems have been developed for specific applications including decoding.

[0005] In some signal processing applications, although it is conceivable to successfully train a neural network to perform the desired signal processing, such a neural network system would become impractically complex. This would require a large amount of training data and also require huge computing resources during inference.

[0006] Some attempts have been made to alleviate this problem. In one approach, for example, Dhariwal et al., Jukebox: Agenerative model for music (2020), the input signal is first transformed to a vector quantized representation before being input to the generative model. The inferred signal is then synthesized back to the full representation. As a result of this approach, the generative model operates in a vector quantized space, significantly reducing computational complexity. Summary of the invention

[0007] A disadvantage of the methods discussed above is that vector quantization (or any other complexity reduction) produces a trade-off between complexity reduction and achievable quality. This trade-off is difficult to optimize. Another disadvantage is that vector quantization applied directly to the signal to be processed removes some information relevant to solving the processing problem, thereby limiting the achievable performance.

[0008] The present invention seeks to overcome these problems and to provide an improved method of audio signal processing using neural networks.

[0009] According to a first aspect of the invention, this object is achieved by a method for processing an input audio signal, the method comprising conditioning a first neural network system using a representation of the input audio signal to predict a bit-rate reduced representation of the processed input audio signal, the first neural network system being trained to generate a bit-rate reduced representation of a processed version of a given audio signal, wherein the bit-rate reduced representation has a format associated with a predefined audio encoding process; conditioning a second neural network system using the bit-rate reduced representation to predict an enhanced representation of the processed audio signal, the second neural network system being trained to generate an enhanced representation of the given bit-rate reduced audio representation, wherein the bit-rate reduced representation has a format associated with a predefined audio encoding process; and transforming the enhanced representation of the processed audio signal into an output audio signal.

[0010] The processing is thus performed in two stages, with an intermediate processing result having a reduced bit rate. This intermediate processing result is called the latent signal.

[0011] It is important to note that the intermediate processing results (latent signals) have a format associated with a predefined audio coding process. This means that for a given (known) pair of overall truth signals, i.e., the original signal and the processed (target) signal, the intermediate processing objectives of the first stage can also be determined deterministically. For example, in the context of supervised training, the training objective of the first stage can be defined as a function of the network output and the audio encoded target (not the unencoded target). This process facilitates the definition of the audio coding latent that is effectively decoupled from the stages. Therefore, the neural network system of each stage can be trained using separate loss functions (separately or jointly).

[0012] Separate training of the two neural network systems has possible advantages in that it can be simpler to perform and it is likely that the models implementing the networks can be relatively smaller. Joint training is expected to provide better overall performance, but it is likely that larger models and more training data will be required.

[0013] During inference, the first network solves the processing task by providing the result of the intermediate representation, while the second network provides the final processing result based on the intermediate representation. The fact that inference involves two specialized networks has a significant impact on the computational complexity (compared to an end-to-end system). The use of two specialized networks facilitates the decomposition of the processing problem into sub-problems that can be associated with their own training objectives. It is expected that a single network solving the processing task in an end-to-end setting will require significantly more trainable parameters and significantly larger amounts of training data.

[0014] Additionally, the potential bit rate reduction (quantization) performed according to a predefined audio coding process implements a trade-off between bit rate reduction and distortion according to a predefined audio coding process. The audio coding algorithm optimizes its bit rate distortion trade-off in a perceptually optimal manner, and its details depend on the coding algorithm. The potential format thus ensures an appropriate trade-off between the performance of the processing task performed by the first stage and the performance of the final synthesis task performed by the second stage.

[0015] According to a second aspect of the invention, this object is achieved by a system for processing an input audio signal, the system comprising: a first neural network system, which is trained to generate a reduced bit rate representation of a processed version of a given audio signal, wherein the reduced bit rate representation has a format associated with a predefined audio encoding process, wherein the first neural network system is conditioned by the representation of the input audio signal to predict the reduced bit rate representation of the processed input audio signal; a second neural network system, which is trained to generate an enhanced representation of a given reduced bit rate audio representation, wherein the reduced bit rate representation has a format associated with a predefined audio encoding process, wherein the second neural network system is conditioned by the reduced bit rate representation predicted by the first neural network system to predict the enhanced representation of the processed audio signal; and a processing stage for transforming the enhanced representation of the processed audio signal into an output audio signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The invention will now be described in more detail with reference to the accompanying drawings, which show currently preferred embodiments of the invention.

[0017] Figure 1 is a block diagram of a process according to an embodiment of the present invention.

[0018] Figure 2 shows a method with separate training objectives for the first level (S1) and the second level (S2). Figure 1 Training of neural network systems in .

[0019] Figure 3 It operates in the MDCT domain Figure 1 An example of a more detailed implementation of the procedure in .

[0020] Figure 4 Shows Figure 2 A first example of the audio encoding process in .

[0021] Figure 5 Shows Figure 2 A second example of the audio encoding process in . DETAILED DESCRIPTION

[0022] The systems and methods disclosed in this application can be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, the division of tasks does not necessarily correspond to division into physical units; on the contrary, a physical component can have multiple functions, and a task can be performed collaboratively by several physical components.

[0023] The computer hardware may be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smart phone, a web device, a network router, a switch or a bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by the computer hardware. In addition, the present disclosure shall relate to any collection of computer hardware that executes instructions, either individually or in combination, to perform any one or more of the concepts discussed herein.

[0024] Some or all components may be implemented by one or more processors that accept a computer-readable (also referred to as machine-readable) code of a set of instructions that execute at least one of the methods described herein when executed by one or more of the processors. Any processor that is capable of executing a set of instructions (sequential or otherwise) that specifies an action to be taken is included. Therefore, an example is a typical processing system (i.e., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may also include a memory subsystem that includes a hard drive, an SSD, a RAM, and / or a ROM. A bus subsystem may be included to communicate between components. The software may reside in the memory subsystem and / or in the processor during its execution by the computer system.

[0025] One or more processors may operate as independent devices or may be connected, for example, networked to other processors. Such a network may be built on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0026] The software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is well known to those skilled in the art, the term "computer storage media" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, various forms of physical (non-transitory) storage media, such as EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media (transitory) typically contain computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information delivery media.

[0027] Figure 1 The audio signal processing of an input signal y in a two-stage architecture with a latent signal associated with a finite bit rate is shown. The architecture has two separate processing stages 10 and 11. The first stage involves a first neural network system 12 which, given a representation of the input signal y, is trained according to a training objective (S1) of the first stage to generate the latent signal (See Figure 2 ). The potential signal generated is the processed input signal (in Figure 2 The bit rate reduced representation z (and hence also the predicted latent signal ) has a format associated with a predefined audio codec quantized to a desired bit rate. The nature of the processing predicted by the neural network system depends on the implementation. Examples include signal separation (e.g., separating a piano from a mixed music signal) and signal enhancement (e.g., dereverberation of speech).

[0028] The second stage involves a second neural network system 13 which is trained to predict a predicted bit rate reduced representation of the processed input signal Enhanced representation of In other words, given the quantized representation , this stage can be considered as the processed signal The enhanced representation may have the same format as the bit rate reduced representation, but may also be different. For example, the bit rate reduced representation may be represented in another domain (eg, MDCT) by transforming the reconstruction of the bit rate reduced representation.

[0029] The neural network systems 12, 13 included in stages 10 and 11 may be designed to operate entirely in the domain of the underlying signal (e.g. in a transform domain such as MDCT). In this case, the input signal is first transformed into this domain and the prediction of the second neural network is inversely transformed back (synthesized) into the time domain.

[0030] Figure 2 Shows Figure 1 Training of the neural network systems 12 and 13 in . For training (in the case of supervised training), an original audio signal y (signal to be processed) and a target signal x (processed signal) are obtained. The original signal may be, for example, mixed music including piano sounds, and the target signal may be the same piano sound as the isolated signal. In addition, by applying a predefined audio encoding process 14 to the target signal x (isolated piano sound signal), a target latent signal z is obtained with an appropriate degree of bit rate reduction (quantization). The target latent signal z is a bit rate reduced representation of the target signal x. The first neural network system 12 is trained using the representation of the original signal y to generate a latent signal having a first loss function S1 relative to the target latent signal z. The first neural network system can be trained in a regression setting, in which case the loss function S1 can be a weighted squared error, a weighted L-1 norm, a multi-resolution STFT loss, or a combination of L-2 and L-1 norms. Alternatively, the first neural network is trained in a generative setting, in which case the loss function S1 can be a negative log-likelihood (NLL). The second neural network system 13 uses the generated latent signal (in the case of joint training) or the actual latent signal z (in the case of independent training) to generate an output signal with a second loss function S2 relative to the target signal x. The second neural network system is trained in a generative setting, wherein the loss function S2 may be a negative log-likelihood (NLL). The training of the network systems may be performed separately or in combination.

[0031] Reference Figure 3 Describe in more detail Figure 1 A specific implementation of the system in . In this example, the audio codec format is in the transform domain, and more specifically in the MDCT domain. Both neural network systems are also designed to operate entirely in the MDCT domain. In the illustrated example, the MDCT lines are dynamically lowered (e.g., spectral flattening).

[0032] refer to Figure 3, the first stage 10 here includes an MDCT transform 21 to transform the input audio signal into the MDCT domain. The resulting MDCT line is supplied to an envelope estimator 23 to provide a spectral envelope. The MDCT line is then flattened by a flattening function 22 using the spectral envelope determined by the envelope estimator 23 to reduce the spectral dynamics of the signal. In some embodiments, the flattening function 22 can be implemented by estimating the spectral envelope (e.g., calculating the variance of the signal in a predefined number of sub-bands) and then normalizing the MDCT coefficients in each sub-band according to the value of the spectral envelope of these bands. Examples of such normalization include normalization towards unit variance (where we use an envelope value with an exponent of 1.0), or normalization towards a "pink domain" (where we use an envelope value with an exponent of 0.5).

[0033] The resulting representation of the input signal (flattened MDCT lines and envelope) is input to the first neural network system 24. In the illustrated case, the first neural network system 24 is configured according to the generation settings and predicts the underlying signal The probability distribution is sampled by the sampler 25 to obtain the potential signal In some embodiments, the first neural network system is instead configured according to a regression setting. The system will then directly provide the potential signal , and sampler 25 would not be required.

[0034] The second stage 11 here comprises a second neural network system 26 operating in a generative setting, which takes the latent signal and predict the enhanced (reconstructed) signal The probability distribution is sampled by the sampler 27 to obtain an enhanced signal representation . Like a potential signal Likewise, the enhanced signal indicates includes flattened MDCT lines and envelopes. However, the enhanced signal representation With potential In other words, the second neural network system 26 reduces the bit rate of the latent signal by quantizing it (reducing the bit rate). Signals that are modulated and predicted to be enhanced The enhanced signal indicates is inverse flattened by an inverse flattening function 28 (using the potential signal ), and the audio output x is finally synthesized through inverse MDCT transformation 29.

[0035] If S1 is configured according to the negative log-likelihood (NLL) loss (generative setting), then the first stage will provide a probability distribution at its output. In order to use this as a conditioning on the second stage, sampler 25 must be used. If S1 is configured in a regression setting, then the output of stage 1 is deterministic, and thus sampler 25 can be omitted. The S2 target of the second stage is always configured in a generative setting (according to the NLL loss), and sampler 27 will always be used.

[0036] exist Figure 3 In the process of FIG. 1 , where the transform domain is MDCT, the neural network systems 24 and 26 can be designed according to the topology discussed in PCT / US2021 / 054617 entitled “GENERAL MEDIA NEURAL NETWORK PREDICTOR AND A GENERATIVE MODEL INCLUDING SUCHA PREDICTOR,” which is incorporated herein by reference. In this topology, the neural network system includes an independent frequency prediction part and an independent time prediction part, where the output from one part is provided as an input to the other part.

[0037] However, it is noted that the neural network system in PCT / US2021 / 054617 is conditioned by samples of MDCT lines and generates sample MDCT lines. Figure 3 In the context of the process in , the first neural network system 24 will be conditioned by the MDCT lines and will predict not only the MDCT lines but also the envelope (variance vector) of the MDCT lines. The topology in PCT / US2021 / 054617 will therefore need to be modified in an appropriate manner. With respect to the second neural network system 26, it is noted that the enhanced signal representation The envelope will be related to the potential Therefore, the neural network system 26 can only pass the potential signal The MDCT lines are adjusted and only the enhanced signal representation is generated MDCT lines. The neural network system 26 can therefore essentially be an implementation of the topology in PCT / US2021 / 054617.

[0038] for Figure 3 In the process, the training of the neural network system will require the target latent signal z obtained using the audio coding process operating in the MDCT domain.

[0039] Figure 4An example of an audio coding process 30 is shown, which maps an input signal to a representation with a reduced bit rate by means of quantization using a waveform codec. In the illustrated case, the process 30 includes an MDCT transform 31 to transform the target signal into the MDCT domain. The MDCT transform 31 is configured to provide a perceptually excited division of the MDCT line, wherein the lower frequency band is narrower (i.e., the information is denser). The process also includes an envelope estimator 33 for determining the spectral envelope and a flattening function 32 for reducing the dynamics of the MDCT line using the spectral envelope. Finally, two quantizers 34a, 34b are provided for quantizing the flattened MDCT line and the envelope to a desired bit rate R. The quantizer is configured to distribute the distortion caused by the given bit rate reduction in a perceptually optimal manner (i.e., as unnoticeable as possible). Existing audio codec processes include such quantization algorithms.

[0040] Figure 5 Another example of an encoding process 40 is shown, which maps an input signal to a reduced bit rate representation by means of parametric encoding. Such an encoding process can be based on a sinusoidal analysis algorithm (e.g., a matching pursuit algorithm), in which the analysis is performed in a manner that selects sinusoidal components to minimize a certain perceptual criterion (e.g., a spectrally weighted mean square error). The process 40 includes a parameterization function 41 that parameterizes the input signal into a parameterized description having N sinusoidal waves and a phase parameter for each sinusoidal trajectory. The size of the number N determines the level of quantization. The larger N is, the higher the associated bit rate and the higher the fidelity of the reconstruction. In the illustrated example, the parameterized description is extended with envelope information. The spectral envelope can include a set of envelope values ​​associated with a certain frequency division (using perceptually excited sub-banding). In some embodiments, the spectral envelope can be described by means of linear prediction coefficients (LPC).

[0041] In order to Figure 3 , the target latent signal z needs to be in the MDCT domain. For this purpose, the encoding process 40 here also includes a reconstruction block 42 for reconstructing the time domain signal, and an MDCT transform 43 for obtaining MDCT lines. In principle, the parameterized representation can also be reconstructed directly into the MDCT domain, for example, by projecting a sinusoidal wave onto the MDCT basis. Figure 4 Similar to the encoding process in , the MDCT lines are flattened using the spectral envelope by a flattening function 44. The spectral envelope is obtained in block 45 by mapping the envelope information in the parameterized description onto the MDCT lines.

[0042] Unless otherwise specifically stated, as will be clear from the following discussion, it is appreciated that discussions throughout the disclosure utilizing terms such as "processing," "computing," "calculating," "determining," "analyzing," etc., refer to the actions and / or processes of computer hardware or computing systems or similar electronic computing devices that manipulate data represented as physical (such as electronic) quantities and / or transform data represented as physical (such as electronic) quantities into other data similarly represented as physical quantities.

[0043] It should be appreciated that in the above description of the exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of simplifying the disclosure and helping to understand one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly stated in each claim. More specifically, as reflected in the accompanying claims, the inventive aspects exist in less than all the features of a single previously disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into this specific embodiment, wherein each claim is independently a separate embodiment of this invention. In addition, although some embodiments described herein include some features included in other embodiments but do not include other features, as will be understood by those skilled in the art, the combination of features of different embodiments is intended to be within the scope of the invention and form different embodiments. For example, in the accompanying claims, any of the claimed embodiments may be used in any combination.

[0044] In addition, some of the embodiments are described herein as a combination of methods or elements of methods, which can be implemented by a processor of a computer system or by other means of performing functions. Therefore, a processor with instructions for executing such methods or elements of methods forms a means for executing methods or elements of methods. It is noted that when a method includes several elements (e.g., several steps), the order of such elements is not implied unless specifically stated. In addition, the elements described herein of the device embodiment are examples of means for executing the functions performed by the elements for the purpose of executing the embodiments of the invention. In the description provided herein, many specific details are set forth. However, it is understood that embodiments of the invention can be practiced without these specific details. In other cases, well-known methods, structures, and techniques have not yet been shown in detail so as not to confuse the understanding of this description.

[0045] Those skilled in the art will appreciate that the present invention is by no means limited to the preferred embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims. For example, other domains than MDCT may be considered.

[0046] The invention can be further understood from the following list of enumerated exemplary embodiments (EEE).

[0047] EEE1. A method for processing an input audio signal, comprising:

[0048] conditioning a first neural network system using the representation of the input audio signal to predict a bit rate reduced representation of the processed input audio signal, the first neural network system being trained to generate a bit rate reduced representation of a processed version of a given audio signal, wherein the bit rate reduced representation has a format associated with a predefined audio encoding process,

[0049] conditioning a second neural network system to predict an enhanced representation of the processed audio signal using the bit rate reduced representation, the second neural network system being trained to generate an enhanced representation of a given bit rate reduced audio representation, wherein the bit rate reduced representation has a format associated with the predefined audio encoding process, and

[0050] The enhanced representation of the processed audio signal is transformed into an output audio signal.

[0051] EEE2. The method according to EEE1, wherein the input audio signal and the output audio signal are in the time domain.

[0052] EEE3. A method according to EEE1 or EEE2, wherein the enhanced representation has a format associated with the predefined audio encoding process.

[0053] EEE4. A method according to any one of EEE1 to EEE3, wherein the representation of the input signal, the bit rate reduced representation, and the enhanced representation are all in one and the same transform domain. EEE4.

[0054] EEE5. A method according to any one of EEE1 to EEE4, wherein the transform domain is a waveform transform domain.

[0055] EEE6. The method according to any one of EEE1 to EEE5, wherein the representation of the input signal, the bit rate reduced representation, and the enhanced representation all comprise a set of MDCT lines and associated envelope information. EEE6.

[0056] EEE7. The method according to any one of EEE1 to EEE6, wherein the MDCT line has reduced signal dynamics.

[0057] EEE8. A method according to any one of EEE1 to EEE7, wherein the step of transforming comprises increasing the signal dynamics of the enhanced representation.

[0058] EEE9. A method according to any one of EEE1 to EEE8, wherein the first neural network system is trained and operated in a generative setting.

[0059] EEE10. A method according to any of the preceding EEEs, wherein the second neural network system is trained and operated in a generative setting.

[0060] EEE11. A method according to any of the preceding EEEs, wherein the input audio signal is a distorted audio signal, and the first neural network system predicts a bit rate reduced representation of a signal enhanced version of the input audio signal. EEE12. A method according to any of the preceding EEEs, wherein the input audio signal is a distorted audio signal, and the first neural network system predicts a bit rate reduced representation of a signal enhanced version of the input audio signal.

[0061] EEE12. A method according to any of the preceding EEEs, wherein the input audio signal is a mixed audio signal and the first neural network system predicts a bit rate reduced representation of a source separated version of the input audio signal. EEE13.

[0062] EEE13. A system for processing an input audio signal, comprising:

[0063] a first neural network system, the first neural network system being trained to generate a reduced bit rate representation of a processed version of a given audio signal, wherein the reduced bit rate representation has a format associated with a predefined audio encoding process, wherein the first neural network system is conditioned by the representation of the input audio signal to predict the reduced bit rate representation of the processed input audio signal,

[0064] a second neural network system trained to generate an enhanced representation of a given bit rate reduced audio representation, wherein the reduced bit rate representation has a format associated with the predefined audio encoding process, wherein the second neural network system is conditioned by the bit rate reduced representation predicted by the first neural network system to predict the enhanced representation of the processed audio signal, and

[0065] A processing stage for transforming the enhanced representation of the processed audio signal into an output audio signal.

[0066] EEE14. A system according to EEE13, wherein the input audio signal and the output audio signal are in the time domain. EEE14.

[0067] EEE15. A system according to EEE13 or EEE14, wherein the enhanced representation has a format associated with the predefined audio encoding process.

[0068] EEE16. A system according to any one of EEE13 to EEE15, wherein the representation of the input signal, the bit rate reduced representation, and the enhanced representation are all in a same transform domain. EEE16.

[0069] EEE17. A system according to any one of EEE13 to EEE16, wherein the transform domain is a waveform transform domain. EEE17.

[0070] EEE18. A system according to any one of EEE13 to EEE17, wherein the representation of the input signal, the bit rate reduced representation, and the enhanced representation all comprise a set of MDCT lines and associated envelope information. EEE18.

[0071] EEE19. A system according to any one of EEE13 to EEE18, wherein the MDCT lines have reduced signal dynamics. EEE19.

[0072] EEE20. A system according to any one of EEE13 to EEE19, wherein the step of transforming includes increasing the signal dynamics of the enhanced representation.

[0073] EEE21. A system according to any one of EEE13 to EEE20, wherein the first neural network system is trained and operated in a generative setting.

[0074] EEE22. A system according to any one of EEE13 to EEE21, wherein the second neural network system is trained and operated in a generative setting.

[0075] EEE23. A system according to any one of EEE13 to EEE22, wherein the input audio signal is a distorted audio signal and the first neural network system predicts a bit rate reduced representation of a signal enhanced version of the input audio signal.

[0076] EEE24. A system according to any one of EEE13 to EEE23, wherein the input audio signal is a mixed audio signal and the first neural network system predicts a bit rate reduced representation of a source separated version of the input audio signal.

[0077] EEE25. A computer program product, comprising computer program code portions, which are configured to perform a method according to one of EEE1 to EEE12 when executed on a computer processor. EEE25.

Claims

1. A method for processing an input audio signal, comprising: conditioning a first processing stage comprising a first neural network system using a representation of the input audio signal to generate a latent signal comprising a prediction of a bit rate reduced representation of a processed version of the input audio signal, the first neural network system being trained to generate a bit rate reduced representation of a target processed version of a given audio signal, wherein the bit rate reduced representation has a format associated with a predefined audio codec quantized to a desired bit rate, conditioning a second processing stage comprising a second neural network system to predict a processed version of the input audio signal using the latent signal, the second neural network system being trained to generate an enhanced representation of a given reduced bit rate audio representation of the processed version of the audio signal, wherein the reduced bit rate representation has a format associated with the predefined audio codec, and The predicted processed version of the input audio signal is transformed into an output audio signal. 2 . The method of claim 1 , wherein the input audio signal and the output audio signal are in the time domain.

3. The method according to claim 1 or 2, wherein the enhanced representation has a format associated with the predefined audio encoding process.

4. A method according to any one of claims 1 to 3, wherein the representation of the input signal, the bit rate reduced representation, and the enhanced representation are all in the same transform domain.

5. The method according to any one of claims 1 to 4, wherein the transform domain is a waveform transform domain.

6. The method of any one of claims 1 to 5, wherein the representation of the input signal, the bit rate reduced representation, and the enhanced representation all comprise a set of MDCT lines and associated envelope information. The method of claim 6 , wherein the MDCT lines have reduced signal dynamics.

8. A method according to any one of claims 1 to 7, wherein the step of transforming comprises increasing the signal dynamics of the enhanced representation.

9. The method of any one of claims 1 to 8, wherein the first neural network system is trained and operated in a production setting.

10. The method according to any of the preceding claims, wherein the second neural network system is trained and operated in a production setting.

11. A method according to any of the preceding claims, wherein the input audio signal is a distorted audio signal and the first neural network system predicts a bit rate reduced representation of a signal enhanced version of the input audio signal.

12. A method according to any one of the preceding claims, wherein the input audio signal is a mixed audio signal and the first neural network system predicts a bit rate reduced representation of a source separated version of the input audio signal.

13. A system for processing an input audio signal, comprising: comprising a first processing stage of a first neural network system, the first neural network system being trained to generate a reduced bit rate representation of a target processed version of a given audio signal, wherein the reduced bit rate representation has a format associated with a predefined audio codec quantized to a desired bit rate, wherein the first neural network system is conditioned by a representation of the input audio signal to generate a latent signal, the latent signal comprising a prediction of the reduced bit rate representation of the processed version of the input audio signal, comprising a second processing stage of a second neural network system, the second neural network system being trained to generate an enhanced representation of a given bit-rate reduced audio representation of a processed version of an audio signal, wherein the bit-rate reduced representation has a format associated with the predefined audio codec, wherein the second neural network system is conditioned by the latent signal predicted at the first neural network system to predict the processed version of the input audio signal, and A processing stage for transforming the predicted processed version of the input audio signal into an output audio signal.

14. The system of claim 13, wherein the input audio signal and the output audio signal are in the time domain.

15. A system according to claim 13 or claim 14, wherein the enhanced representation has a format associated with the predefined audio codec.

16. A system according to any one of claims 13 to 15, wherein the representation of the input signal, the bit rate reduced representation, and the enhanced representation are all in the same transform domain.

17. A system according to any one of claims 13 to 16, wherein the transform domain is a waveform transform domain.

18. A system according to any one of claims 13 to 17, wherein the representation of the input signal, the bit rate reduced representation, and the enhanced representation all comprise a set of MDCT lines and associated envelope information.

19. The system of claim 18, wherein the MDCT lines have reduced signal dynamics.

20. A system according to any one of claims 13 to 19, wherein the step of transforming comprises increasing the signal dynamics of the enhanced representation.

21. The system of any one of claims 13 to 20, wherein the first neural network system is trained and operates in a production setting.

22. The system of any one of claims 13 to 21, wherein the second neural network system is trained and operates in a production setting.

23. A system according to any one of claims 13 to claim 22, wherein the input audio signal is a distorted audio signal and the first neural network system predicts a bit rate reduced representation of a signal enhanced version of the input audio signal.

24. A system according to any one of claims 13 to claim 23, wherein the input audio signal is a mixed audio signal and the first neural network system predicts a bit rate reduced representation of a source separated version of the input audio signal.

25. A computer program product comprising computer program code portions configured to perform the method according to one of claims 1 to 12 when executed on a computer processor.

Images