Audio processor and method for audio processing

Abstract

Audio processor (10), comprising an audio enhancement module (12) configured to process an input audio signal to obtain an enhanced audio signal (ES) and a control unit (14) configured to generate a control signal (CS), wherein the audio enhancement module is adjusted dependent on the control signal (CS).

Description

[0001] Embodiments of the present invention refer to an audio processor and to a corresponding method. In general, embodiments refer to neural coding. Preferred embodiments refer to adjustable noise reduction for neural coding.

[0002] Most speech signals recorded under real-world conditions contain background noise additionally to the desired signal. State-of-the-art communication pipelines leverage neural speech codecs for deriving a compact discrete representation of the input signal which represents the bitstream to be transmitted. Here, the expected background noise has to be considered during training of the neural codec where the aim might be to reproduce or to remove the background noise at receiver side [1]. However, in any case, considering background noise for the training of the neural codec will consume model capacity and will, hence, require larger and more complex neural networks for maintaining speech quality achievable for clean speech. An alternative approach is to use a concatenation of a Speech Enhancement Module (SEM, in general Audio Enhancement Modul) and a neural codec, where the neural codec now might be trained solely on clean speech. Here, it is crucial that the SEM and the neural codec are aligned with each other, i.e., that distortions introduced by the SEM are seen during the training of the neural codec, which reduces flexibility and compatibility with existing solutions, and makes the training of the overall system more difficult. Therefore, there is a need for an improved approach.

[0003] An objective of the present application is to provide an improved concept for speech enhancement or in general for audio enhancement.

[0004] The objective is solved by the subject-matter of the independent claims.

[0005] An embodiment provides an audio processor comprising an audio enhancement module, like a speech enhancement module, configured to process an input audio signal to obtain an enhanced audio signal. The audio processor further comprises a control unit configured to generate a control signal, wherein the audio enhancement module is adjusted dependent on the control signal.

[0006] According to further embodiments, the audio processor may comprise a downstream task module configured to process the enhanced (audio) signal to obtain an output (audio) signal or, in general output data. For audio coding chosen as the downstream task, the output is a bitstream (if sender and receiver is treated together, the output signal is indeed an audio signal), for speech recognition the output is text, for scene classification the output of the downstream task are labels. Thus, the output of the audio processor and / or the downstream task is referred to as output data. For example, the downstream task module may comprise a neural audio / speech codec or encoder, or a system involving a neural audio / speech encoder and decoder. For example, the downstream task could be a speech transmission, which consists of a speech encoder and a channel coding and another processing step and / or speech decoder. According to embodiment, the control unit may be connected to the downstream task module or may be part of same, so that the control signal is derived from a signal of the downstream task module.

[0007] According to an embodiment, the audio enhancement module may be a speech enhancement module (SEM). According to a preferred embodiment, an audio enhancement module may be a speech enhancement module, where the downstream task module comprises a neural codec. Due to the control module, information generated by the neural codec can be used to control the speech enhancement module so as to improve the processing of the neural codec, e.g. during application. Background is that the SEM is adjusted during operation such that the neural codec can work well. Basically, this performs domain adaptation, i.e., the neural codec which may be trained for clean speech now also works with noisy speech without the need of retraining and without suffering from train-test mismatch due to unseen artifacts of the SEM provided to the codec. For example, by adjusting the SEM to the neural codec, possibly pretrained with clean speech data, the training process can be simplified significantly and compatibility with existing and deployed models is enabled.

[0008] In general, embodiments of the present invention are based on the principle, that a control unit, e.g. a control unit belonging or linked with a downstream task module can influence the processing performed by an audio enhancement module. In doing so, the audio enhancement module can beneficially be controlled to provide an enhancement audio signal having high quality, e.g. for further coding. Consequently, the control signal enables to inform the audio enhancement module on the amount of noise present in the coded signal, e.g.., if the input signal is clean or noisy, or on other information relevant for the audio enhancement.

[0009] Regarding the processing of the audio enhancement module, it should be noted that according to embodiments, the processing may be performed so as to reduce interferences, i.e., undesired signal components, of the input audio signal, i.e., the outputted enhanced audio signal having reduced interferences when compared to the input audio signal. Consequently, the audio enhancement module comprises a reduction of interferences.

[0010] In other words, this means that according to embodiments the audio processor primarily consists of a speech enhancer and a neural codec, wherein the speech enhancer may reduce interfering sources, e.g., background noise, but also, depending on the application, other undesired signal components such as reverberation etc. The output of the speech enhancer is the input of the neural codec that encodes, i.e., compresses this signal and prepares it for storage or transmission.

[0011] According to embodiments, the audio enhancement module, i.e. its processing, may be adapted dependent on the control signal. As mentioned above, the downstream module may in this way control and adjust the audio / speech enhancement module by use of the control unit. For example, the adjusting may comprise manipulating propagated RNN stages of the audio enhancement module in accordance with the control signal or wherein adjusting comprises providing embeddings to the audio enhancement module, e.g. sinusoidal embeddings or learned embeddings by use of the control signal, which comprises said embeddings. Furthermore, intermediate representations of the audio enhancement module may be manipulated by trainable network blocks that are controlled by the control signal including, e.g., TADE layers.

[0012] In the below embodiments, it will be discussed that either the audio enhancement module or an entity like a combiner, e.g. arranged more downstream to the audio enhancement module can be controlled by the control signal.

[0013] The control unit may control the audio enhancement module directly. According to a further embodiment, the control unit may control a component of the audio enhancement module, e.g. a combiner (arranged downstream to the central processor of the audio enhancement module). Thus, according to embodiments, the audio enhancement module comprises a combiner controlled by the control unit in accordance to the control signal. For example, the combiner may be configured to combine the enhanced audio signal and the input audio signal as a function of the control signal. The combiner is configured to combine and / or mix the enhanced audio signal and the input signal, e.g. as a function of the control signal. According to further embodiments, a combiner is configured to combine the enhanced audio signal and the input audio signal as a weighted sum. For example, the control signal comprises one or more weights for the weighted sum. According to further embodiments, the combiner is realized in a transform domain or time / frequency domain, such that a frequency selective combination of the enhanced audio signal and the input audio signal is achieved. According to further embodiments, the combiner is configured to adjust a mask derived by an audio enhancement module or a downstream task module. According an embodiment, the combination could also be done by ,styling' the enhanced signal with a (trainable) network block comprising, e.g., TADE layers.

[0014] According to a further embodiment, the adjusting may comprise an addition or concatenation of information dependent on the control signal to some intermediate representation of the speech enhancement module (SEM) to control its behavior. For example, embeddings are a representation of the control signal that is tailored to use in the audio or speech enhancement module. It might incorporated in various ways into the speech enhancement module, e.g. by addition, concatenation, etc. to some intermediate representation of the speech enhancement module. According to embodiments, the intermediate representations of the speech enhancement module may be manipulated by 'styling', e.g., with TADE layers, as well.

[0015] Below, different variants of how the control unit derives the control signal will be discussed. According to embodiments, there are different variants, partially based on signals or interim signals of a downstream unit like the downstream task module and / or partially based on external signals like a user input or an information obtained from another entity of the audio processor or an external entity.

[0016] According to further embodiments (main aspect), the control unit is configured to derive the control signal from an intermediate representation of a downstream task module, especially a neural audio / speech encoder or codec module or an output signal of a downstream task module, especially a neural audio / speech encoder or codec module. This is beneficial since, by doing so, a kind of feedback from the downstream task module to the audio enhancement module is given. It should be noted that preferably, but not necessarily, the downstream task module may be a neural encoder or uses a neural codec.

[0017] According to embodiments (side aspect), the downstream task module is configured to perform acoustic scene analysis, e.g. used for the retrieval of acoustic parameters of an acoustic scene. For example, acoustic scene parameters, like reverberation time, noise level or noisetype, can, according to embodiments, be extracted or determined using the downstream task module.

[0018] According to alternative embodiments, the control unit is configured to derive the control signal from an acoustic scene analysis and / or dependent on acoustic scene parameters characterizing an acoustic scene. For example, the acoustic scene parameters may comprise the reverberation time, noise level or noise type.

[0019] According to further embodiments, the control unit is realized by a neural network or is part of the downstream module, especially a neural audio / speech codec module or an encoder of said neural audio / speech codec module. It is noted that the combination of an adjustable SEM (Speech Enhancement Module) with a neural codec is beneficial since the coding can profit from the adjustment of the SEM having the purpose to output the enhanced signal to be easily coded (e.g. when having reduced interferences).

[0020] According to further embodiments, the control signal is dependent on the user input. In other words, this means that an additional control signal is used to control the audio enhancement module, wherein the additional control signal depends on a user input. This enables beneficially to control the audio enhancement module using user preferences. According to further embodiments, the control signal depends on an external signal (from an external entity) on information of another entity of the audio processor. Using an external signal enables beneficially that a transmission system could deliberatively switch off the speech enhancer since it is known (by signaling of states of transmission systems) that another previous speech enhancer or noise reduction entity was already applied in the processing chain.

[0021] According to another embodiment, control signals depend on an estimation of level of interferences in the input audio signal. This may, for example, be determined using the downstream task module. According to embodiments, the control signal is dependent on the estimation of the level of interferences in the input audio signal, wherein the control signal triggers the audio enhancement module to be bypassed and / or the input signal to be conveyed to a downstream task module if the estimate of the level of interferences is, e.g.., below a threshold.

[0022] According to embodiments, the audio processor comprises a neural audio codec or (neural) encoder or part of a (neural) decoder. For this, the downstream task module may comprise a neural audio / speech codec module or a neural network. According to embodiments, the downstream task module may be configured to process the enhanced audio signal and / or is connected to the audio enhancement module to receive the enhanced audio signal or a combined signal comprising the enhanced audio signal. According to further embodiments, the enhanced audio signal or a derivative of the enhanced audio signal, e.g., the output signal of the downstream task module, may be used for an application out of the following: automatic speech recognition; acoustic scene analysis; classification; adjustable speech enhancement; other processing.

[0023] Another embodiment provides a user interface, like a loudspeaker, headphone or another device comprising the audio processor; additionally or alternatively, or a device for sound acquisition, like a microphone, smart phone, that acquires the input signal, may comprise the audio processor. Furthermore, a communication device would be relevant that records the input signal, enhances it, compresses it and transmits it to the receiver side. Thus, an embodiment refers to an user interface, sound acquisition device or communication device comprising the audio processor. According to embodiments, the audio processor is integrated into a speech enhancer, especially a mask-based speech enhancer (UpHear speech enhancement)) or a neural encoder, especially an NESC encoder.

[0024] Another embodiment provides an audio processor comprising a downstream task module and the control module. The downstream task module is controlled by the control module in a manner as discussed above. For example, the control module determines the control signal based on the input signal. According to embodiments, the downstream task unit may be configured to perform acoustic scene analysis. For acoustic scene analysis the acoustic parameters may also be extracted directly from the input signal based on a neural network that estimates the acoustic parameters from the input waveform, i.e., the control unit directly observes the input signal. Thus, according to an embodiment, the control signal is derived by the control unit based on the input signal, i.e., may for example comprise the parameter estimation and the construction of the control signal in this case.

[0025] Another embodiment provides a method for processing. It comprises the following steps: performing signal processing of an input audio signal to obtain an enhanced audio signal by use of an audio enhancement module; generating a control signal by use of a control unit; wherein the audio enhancement module is adjusted dependent on the control signal.

[0026] Note according to embodiments, the audio enhancement module may not be adjusted but just its output (second of the two main approaches for adjusting the enhanced signal).

[0027] A further embodiment refers to a method for audio processing: the method comprises: performing signal processing of an enhanced audio signal to obtain an output signal or data; generating a control signal by use of a control unit, wherein the downstream task module is adjusted dependent on the control signal.

[0028] Although in the text above only the control of the SEM based on the state of the downstream task has been mentioned, it should be noted, according to another embodiment, the control signal could be derived from the input signal or any intermediate representation of the SEM in order to control the downstream module.

[0029] According to another embodiment, the method may be computer implemented, i.e., another embodiment provides a computer program comprising a program code for performing the steps of the above-mentioned method.

[0030] Embodiments of the present invention will subsequently be discussed referring to the enclosed figures, wherein: Fig. 1shows a schematic representation of an audio processor according to a basic implementation; Fig. 2shows a schematic representation of an audio processor according to enhanced embodiments to illustrate the control of internal states of a SEM by a control signal derived from internal representations of a neural codec which is derived by the control unit; and Fig. 3shows a schematic representation of an audio processor according to another enhanced embodiment for illustrating a control of the combination of input and output of an SEM module, where the control signals during the combination is derived from an internal representation of the encoder of the neural codec.

[0031] Below, embodiments of the present invention will subsequently be discussed referring to the enclosed figures, wherein identical reference numerals are provided to objects having identical or similar function, so that description thereof is interchangeable and mutually applicable.

[0032] Fig. 1 shows an audio processor 10 having an audio enhancement module 12 and a control unit 14. The control unit 14 is connected to the audio enhancer module so as to control same using a control signal CS.

[0033] The audio enhancement module can be a speech enhancement module and configured to process an input (audio) signal IS so as to obtain an extended signal ES.

[0034] The audio enhancement module 12 / SEM is an adjustable module controlled by the control signal CS output by the control module 14. For example, the control unit is configured to modify and / or skip and / or attenuate the signal processing performed by the speech or audio enhancement module in accordance with the control signal. For example, the control module may use signals from a downstream task module (not shown) or may generate the control signal CS based on user input or an external signal. Due to this information may be made available to the control unit, e.g., in form of the noisy input signal, user input, or the output signal / enhanced signal ES of the entity 12 based on same, the control signal CS is generated.

[0035] The entity 12 may be an audio enhancement module or speech enhancement module which can be configured to reduce interferences from the input audio signal IS to output the enhanced signal ES without or with reduced interferences. In other words, this means that the entity 12 may be configured to reduce interfering sources, e.g., background noise, but also reverberation, etc. This enables beneficially that the output of the audio enhancement module 12, i.e., the enhanced signal ES can be further processed by downstream task modules like neural codecs in an improved manner. According to further embodiments, the enhancement module may comprise an RNN stage where propagated RNN states are manipulated in accordance to the control signal. Alternatively, the audio enhancement module may be configured to provide embeddings, e.g., sinusoidal embeddings or learned embeddings, wherein the process of providing the embeddings is controlled using the control signal. For example, the embeddings may be comprised by the control signal. According to further embodiments, the adjusting according to the control signal may comprise adding or concentrating information dependent on the control signal.. This might be incorporated in various ways into the audio enhancement module 12, e.g., by addition, concatenation, etc. to some intermediate representation of the speech enhancement module, but not necessarily concentrated to the output signal of this speech or audio enhancement model 12.

[0036] The proposed approach is, when compared to the prior art more elegant as it adjusts the SEM to the neural codec possibly pretrained with clean speech data, which simplifies the training process and enables compatibility with existing and deployed models.

[0037] Fig. 2 shows an audio enhancement model or speech enhancement module 12 and a downstream task module 16, e.g., a neural codec encoder. Furthermore, the control unit 14 is arranged as a kind of feedback loop. The downstream task module 16 receives the enhanced signal ES from the audio enhancement module 12. The control unit receives the output signal or an interim signal of the downstream task module 16, the input signal IS or the output signal ES and generates the control signal CS. In other words, the control unit could take IS, ES, or BS or any other intermediate representation as input for generating the control signal CS.

[0038] The audio enhancement module 12 may be a speech enhancement module receiving a noisy signal as input signal IS and configured to denoised same. Consequently, the signal ES is a denoised signal. The denoised signal is then further processed by the neural codec encoder 16, so as to obtain the bitstream BS.

[0039] In other words, Fig. 2 shows the combination of an adjustable SEM 12 with a neural audio / speech codec 16 from which a control signal CS is generated for controlling the adjustment of the SEM 12.

[0040] Here, the control signal CS is derived from an intermediate representation of the encoder (activations of the encoder of the neural codec 16, the outputted bitstream, statistics of the quantization error or other internal representations as, e.g., propagated states of RNN layers) of the neural codec 16 by a control unit 14 that might be realized by a neural network itself or might be seen as part of the encoder of the neural audio codec 16. The control of the SEM 12 is done such that its output ES allows the neural codec 16 to decode high-quality clean speech. Furthermore, other information might be made available to the control unit 14, e.g., the noisy input signal, user input, or the output signal ES of the SEM 12.

[0041] According to embodiments, the control unit 14 acts directly on internal representations of the SEM to adjust its behavior. This might be accomplished by manipulating propagated RNN states by specific neural network units, e.g., TADE layers [3], by presenting embedded control signal information to the SEM 12, e.g., sinusoidal embeddings or learned embeddings, or by adding or concatenating (concatenating information may be is derived from the control signal) the information contained in the control signal.

[0042] According to other embodiments, besides the main aspect of controlling the SEM 12 by the neural codec, other side aspects include the control of the SEM 12 by a user to trade off aspects as, e.g., dereverberation, noise suppression, intelligibility, annoyance of artifacts etc. Furthermore, the SEM 12 might be controlled by acoustic scene analysis, i.e., a control unit 14 that derives the control signal from the noise input signal. Here, also acoustic parameters characterizing the acoustic scene, e.g., reverberation time, noise level, etc., might be estimated explicitly. Of course, also a combination of the techniques for deriving the control signal might be used as well.

[0043] According to embodiments it should be noted that the SEM 12 might also act as the encoder of the neural codec, i.e., controlling the encoding of the denoised signal depending on the state of the SEM. In this case, the control unit extracts information from the entity 12 so as to control the entity 16.

[0044] In other words, this means that the neural codec or, in general the downstream task module 16 is adjusted by the control unit 14 dependent on information, e.g., received from the speech enhancement module 12. The further optional features, like the usage of additional control signal may be applied to this embodiment as well.

[0045] According to embodiments, the proposed concept may include the use of adjustable SEM for other downstream task, as, e.g., automatic speech recognition, acoustic scene analysis or classification. This means that the entity 16 may be configured to perform automatic speech recognition, acoustic scene analysis and classification and / or adjustable speech enhancement or other tasks. Again here it might be possible, that these downstream tasks are adjusted by the entity 14 based on information obtained from the entity 12.

[0046] Embodiment of the invention might not be based on but not limited to the other technologies used in the field of neural coding, like the neural speech codec NESC and the family of speech enchancement technologies UpHear.

[0047] With respect to Fig. 3 another approach will be discussed, where the input (the observed noise audio signal) is combined with an output of the SEM. Fig. 3 shows the speech enhancement module 12 (representative for the audio enhancement module), the neural codec entity 16 (representative for a downstream task module) together with the control unit 14. Additionally, the speech enhancement module 12 comprises a combination module 18 which belongs to the speech enhancement module 12. The module 18 is arranged between the modules 12 and 16.

[0048] In other words, the combiner 18 is arranged downstream to the core of the speech enhancement module 12, i.e., receives the signal ES and has a bypass for the signal IS. The combiner is configured to combine the signal ES and IS dependent on the control signal CS. The result output by the combiner is a combination or weighted combination of ES and IS. In a special case of the more generic combination the combiner can act as switch picking one of the signals. According to embodiments, the combiner comprises a neural network block, especially a neural network block comprising TADE layers or neural network block controlled by the control signal. The controller uses the information on BS and / or other information, like the signal ES. The main advantage of this technique is that for both, the SEM 12 and neural codec 16 existing pretrained modus might be used making this technique compatible with various existing systems.

[0049] Realizations of the combination module might include a weighted sum of both signals, where the weights are determined by the control unit. This weighted sum might be realized in a transform domain such that a frequency-selective combination can be achieved allowing for differently combining various signal components. Moreover, the combination might be accomplished by another neural network block including, e.g., TADE layers, or, in case of a mask-based SEM, the control of the aggressiveness of the mask.

[0050] Another embodiment provides a system processing an input (audio) signal by an audio enhancement unit 12 and a downstream task unit 16. Here, the system includes a control unit adjusting the audio enhancement unit or is output in function of control signal CS. For example, the audio enhancement unit 12 removes or attenuates undesired interferences from the input signal IS, wherein the downstream task unit 16 receives as input the output ES of the audio enhancement unit 12, or a processed version of it.

[0051] According to embodiments, the control signal, or a part of it is provided by the downstream task unit 16. According to embodiments, a downstream task is speech or audio coding, performed by a (neural) audio or speech coder.

[0052] According to embodiments, the control unit 12 can modify and / or skip and / or attenuate the audio enhancement unit 12 effect in response to the control signal CS: According to further embodiments, a system includes a combiner (to be seen as part of the audio enhancement module 12). The combiner is configured to combine or mix the enhanced audio signal ES and the input signal IS in function of the control signal CS.

[0053] According to further embodiments a control unit receives as input a state or internal representation of the downstream task unit. Additionally or alternatively, the control signal may be based on information provided by a user or another application like an external application. This enables that an external entity, like another previous speech enhancer of noise reduction entity can deactivate the audio enhancement module. According to further embodiments the control unit receives as input the output or a state or an internal representation of the audio enhancement unit. Additionally or alternatively the control unit may receive the input and / or the output of the audio enhancement unit.

[0054] Another embodiment provides a system processing an (audio) signal by an audio enhancement module. Optionally, a downstream task unit may be provided. The system includes a control unit adjusting the output of the audio enhancement unit in a function of a control signal. According to embodiments, the output of the audio enhancement output is combined with the input signal according to the control signal. For this, a combiner may be integrated into the audio enhancement module.

[0055] According to another embodiment, the audio processor may be integrated into one of the following: Other downstream tasks: Automatic speech recognition, acoustic scene analysis and classification, user-defined speech enhancement Realizations of combiner Realizations of adjustment techniques of SEM

[0056] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

[0057] The inventive encoded audio signal can be stored on a digital storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0058] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

[0059] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0060] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.

[0061] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0062] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0063] A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitionary.

[0064] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.

[0065] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0066] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0067] A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0068] In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

[0069] The above described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.References

[0070] [1] Zeghidour, Neil, Alejandro Luebs, Ahmed Omran, Jan Skoglund, und Marco Tagliasacchi. "SoundStream: An End-to-End Neural Audio Codec". arXiv, 7. Juli 2021. http: / / arxiv.org / abs / 2107.03312. [2] Pia, Nicola, Kishan Gupta, Srikanth Korse, Markus Multrus, und Guillaume Fuchs. "NESC: Robust Neural End-2-End Speech Coding with GANs". arXiv, 7. Juli 2022. http: / / arxiv.org / abs / 2207.03282. [3] Mustafa, Ahmed, Nicola Pia, und Guillaume Fuchs. "StyleMelGAN: An Efficient High-Fidelity Adversarial Vocoder with Temporal Adaptive Normalization". In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 6034-38. Toronto, ON, Canada, 202.

Claims

1. Audio processor (10), comprising: an audio enhancement module (12) configured to process an input audio signal (IS) to obtain an enhanced audio signal (ES); and a control unit (14) configured to generate a control signal (CS); wherein the audio enhancement module is adjusted dependent on the control signal (CS).

2. Audio processor (10) according to claim 1, further comprising a downstream task module (16) configured to process the enhanced audio signal (ES) to obtain an output signal or data.

3. Audio processor (10) according to claim 2, wherein the downstream task module (16) comprises a neural audio / speech codec or encoder, or a system involving a neural audio / speech encoder and decoder.

4. Audio processor (10) according to one of the previous claims, wherein the processing of the audio enhancement (12) module comprises a reduction of interferences, i.e., undesired signal components, of the input audio signal.

5. Audio processor (10) according to one of the previous claims, wherein adjusting comprises manipulating propagated RNN states of the audio enhancement module in accordance with the control signal (CS) or wherein adjusting comprises providing embeddings to the audio enhancement module, e.g., sinusoidal embeddings or learned embeddings by use of the control signal (CS), which comprise said embeddings; or wherein adjusting comprises an addition or concatenation of information dependent on the control signal (CS) to some intermediate representation of the speech enhancement module.

6. Audio processor (10) according to one of the previous claims, wherein the audio enhancement module (12)comprises a combiner (18) controlled by the control unit (14) in accordance with the control signal (CS).

7. Audio processor (10) according to claim 6, wherein the combiner (18) is configured to combine the enhanced audio signal (ES) and the input audio signal (IS) as a function of the control signal (CS); and / or wherein the combiner (18) is configured to combine the enhanced audio signal (ES) and the input audio signal (IS) as weighted sum, wherein the control signal (CS) comprises one or more weights for the weighted sum, and / or wherein the combiner (18) is realized in a transform domain or time-frequency-domain, such that a frequency selective combination of the enhanced audio signal (ES) and the input audio signal (IS) is achieved, or wherein the combiner (18) is configured to adjust a mask derived by an audio enhancement module (12) and / or a downstream task module (16).

8. Audio processor (10) according to claim 6 or 7, wherein the combiner (18) comprises a neural network block, especially a neural network block comprising TADE layers or a neural network block controlled by the control signal (CS).

9. Audio processor (10) according to one of the previous claims, wherein the control unit (14) is configured to modify and / or skip and / or attenuate the signal processing performed by the audio enhanced module in accordance with the control signal (CS).

10. Audio processor (10) according to one of the previous claims, wherein the control unit (14) is configured to derive the control signal (CS) from an acoustic scene analysis and / or dependent on an acoustic scene parameter characterizing an acoustic scene and / or comprising a reverberation time, noise level or noise input signal.

11. Audio processor (10) according to one of the previous claims, wherein the control signal (CS) is derived from an intermediate representation of a downstream task module (16), especially a neural audio / speech encoder or codec module or an output signal of a downstream task module (16), especially a neural audio / speech encoder or codec module; and / or wherein the control unit (14) determines the control signal (CS) dependent on an input (IS) or an output (ES) of the audio enhancement module (12).

12. Audio processor (10) according to one of the previous claims, wherein the control unit (14) is realized by a neural network or is part of the downstream task module (16), especially a neural audio / speech codec module or an encoder of the neural audio / speech codec module.

13. Audio processor (10) according to one of the previous claims, wherein the control signal (CS) is dependent on a user input or wherein an additional control signal (CS) is used to control the audio enhancement module, wherein the additional control signal (CS) depends on a user input; and / or wherein the control signal (CS) is dependent on an external signaling or an information of another entity of the audio processor (10); and / or wherein the control signal (CS) is dependent on an estimation of the level of interferences in the input audio signal; or wherein the control signal (CS) is dependent on an estimation of the level of interferences in the input audio signal, wherein the control signal (CS) triggers the audio enhancement module (12) to be by-passed and / or the input signal to be conveyed to a downstream task module (16) if the estimate of the level of interferences is below a threshold.

14. Audio processor (10) according to one of the previous claims, wherein the downstream task module (16) comprises a neural audio / speech codec module or a neural network and / or is configured to process the enhanced audio signal and / or is connected to the audio enhancement module to receive the enhanced audio signal.

15. Audio processor (10) according to one of the previous claims, wherein the audio processor (10) is part of a neural audio codec or part of an encoder or part of a decoder; and / or wherein the enhanced audio signal (ES) or a derivative of the enhanced audio signal (ES) is used for an application out of the following: automatic speech recognition, acoustic scene analysis, classification, adjustable speech enhancement; and / or wherein the audio processor (10) is integrated into an user interface, especially a loudspeaker, headphones or any other playback device, or a device for sound acquisition especially a microphone, smart phone; or wherein the audio processor (10) comprises a speech enhancer, especially an masked-based speech enhancer or an neural coder, especially an NESC coder.

16. Audio processor (10) comprising a downstream task module (16) configured to process an enhanced audio signal (ES) to obtain an output signal or data; and a control unit (14) configured to generate a control signal (CS), wherein the downstream task module is adjusted dependent on the control signal (CS).

17. Method for audio processing, the method comprising: performing signal processing of an input audio signal to obtain an enhanced audio signal (ES) by use of an audio enhancement module; generating a control signal (CS) by use of a control unit (14); wherein the audio enhancement module (12) is adjusted dependent on the control signal (CS).

18. Computer program comprising a computer program code for performing the steps of the method according to claim 17.

Images