Neural transform-domain audio coding

Abstract

Audio encoder, configured to encode an input audio signal (IS), comprising: a first processing block (111) realizing a time-feature decomposition configured to transform the input audio signal (TIS) into a transform domain to obtain a transformed input audio signal (TIS); a second processing block (101) including a set of learnable and styling layers (TADE layers) configured to generate a quantized representation of the transformed input audio signal (IS).

Description

[0001] NEURAL TRANSFORM-DOMAIN AUDIO CODING

[0002] Description

[0003] Embodiments of the present invention refer to an audio encoder and audio decoder and to corresponding methods.

[0004] In general, embodiments are in the field of neural coders. Preferred embodiments use transform-domain neural codecs.

[0005] State-of-the-art (SOTA) neural speech coders [1 ,2,3] typically encode the time-domain waveform with a neural encoder and quantize the resulting representation, e.g., by vector quantization. The quantized representation is submitted to the receiver side and decoded with a neural decoder. The encoder and decoder network blocks are usually realized as residual networks and all three algorithmic blocks - encoder, quantizer, decoder - are trained together end-to-end.

[0006] This is illustrated by Fig. 1 showing an encoder 100, decoder 200 and quantizer 150 in between. The encoder 100 receives an input signal IS and outputs an encoded signal. At the encoder side, a quantization is performed by use of the quantizer 150. At the decoder side a dequantization is performed by use of the quantizer 150. The dequantized signal is then decoded by the decoder 200 so as to obtain the reconstructed signal RS. As mentioned above, the three entities 100, 150 and 200 can be trained together.

[0007] In signal processing, often transforms are applied to the signal that provide signal representations that are easier to interpret, process or reveal structures underlying the input signals. Hence, such transforms (e.g., the STFT, MDCT, PQMF, etc.) are used in a plethora of audio signal processing applications. However, for neural audio coding, such transforms are only rarely used so far, usually not designed in a stream-wise fashion (i.e. , not capable of processing an incoming signal frame by frame) and usually not making explicit use of the signal representation as complex numbers [4],

[0008] Therefore, there is a need for an improved approach. It is an objective of the present invention to improve coders, like neural audio coders with respect to efficiency in coding and learning. However, this objective is solved by the subject matter of the independent claims.

[0009] Encoder Aspects

[0010] Embodiments of an aspect of the present invention provide an audio encoder configured to encode an input audio signal. The encoder comprises a first processing block and a second processing block. The first processing block realizing a time-feature decomposition is configured to transform an input audio signal into a transform domain to obtain a transformed input audio signal. The second processing block includes a set of learnable and styling layers (TADE layers) and is configured to generate a quantized representation of the transformed input audio signal (based on the transformed input audio signal).

[0011] Embodiments of the present invention are based on the finding that transforming the time domain wave form into a transformed domain representation may add structural bias (e.g., by preferring harmonic structures) to algorithms operating in this transform domain. Similarly, the neural networks trained in such a transform domain may experience similar bias. Starting from this finding, the encoder or in general the codec may be designed to have a processing block realizing a time-feature decomposition for transforming the input audio signal into a transform domain together with a second processing block belonging to a neural codec, i.e. , having at least a learnable layer.

[0012] Another embodiment of another aspect provides an audio encoder which also comprises a first and a second processing block. The first processing block realizing a time-feature decomposition is configured to transform the input audio signal into a transform domain to obtain a transformed input audio signal. Here, the transformed input audio signal is divided into different subsets. The second processing block includes at least a learnable layer and is configured to generate a quantized representation of the transformed input audio signal by processing differently or separately the different subsets.

[0013] Embodiments of this aspect are based on the finding that the ability to differently process frequency components or treat the potentially complex-valued signal representation in transform domains separately by real / imaginary parts or magnitude / phase components provides benefit by allowing for weighting computational efforts appropriately or by deriving more interpretable and more efficient deep learning solutions. Therefore, the codec uses a first processing block realising the time-feature decomposition for performing the transformation into the transform domain, wherein the transformed input signal is divided into different subsets. The second processing entity comprising a learnable layer (i.e. , part of a neural coder) is then able to process differently or separately the different subsets so as to generate the quantized representation of the transformed input audio signal.

[0014] According to embodiments the second processing block includes as the at least one learnable layer a set of temporal adaptive DE-normalization layers (TADE) or styling layers and / or involves a learned linear modulation or learned affine transform.

[0015] According to embodiments the temporal adaptive DE-normalization layers (TADE) or styling layers are elements of a (TADE) residual block, where the temporal adaptive DE- normalization layers (TADE) or styling layers are part of a residual function and combined to a skip-connection or residual connection.

[0016] According to embodiments the set of learnable and styling layers enables skip connections from an audio encoder side to styling layers of an audio decoder.

[0017] According to embodiments the styling layers are conditioned by the output of previous learnable layers.

[0018] According to embodiments the second processing block comprises a DualPathConvRNN (DPRNN) neural network block containing a sequence of operations for combining different input features, especially by convolutional layers and / or learning temporal context by recurrent layers and again combining features by convolutional layers.

[0019] According to embodiments the second processing block is configured to apply conditioning feature parameters to generate the quantized representation or an intermediate representation of it.

[0020] According to embodiments the first processing block is configured to perform a timefrequency transformation (STFT, MDCT, PQMF, ...) of the input audio signal, especially to realize a filterbank with predetermined filters. According to embodiments the first processing block realizing the time-feature decomposition comprises one or more filters or one or more learned filters or convolutional neural network layers.

[0021] According to embodiments the different subsets of the transformed input signal represents different frequency bands and / or wherein the different subsets are transformed into different domains.

[0022] According to embodiments the transformed input audio signal is a complex-valued signal.

[0023] According to embodiments the second processing block is configured to use different data rates to encode the different subsets.

[0024] According to embodiments the second processing block is configured to use different computational power or different number of learnable parameters to process the different subsets.

[0025] According to embodiments the different subsets of the transformed input signal represent magnitude and phase of a complex-valued frequency-domain representation.

[0026] According to embodiments the (TADE) residual blocks are configured to provide intermediate representations of the transformed input audio signal to an output of the second processing block. For example, the TADE skips provide the intermediate representations to the output of the encoder.

[0027] According to embodiments the second processing block is configured to downsample the conditioning information of the TADE or styling layers subsequently, especially by lowering the time resolution or frequency resolution or channel dimension.

[0028] According to embodiments TADE or styling layers are configured to provide scale and shift parameters for each intermediate representation and successively apply them to the output as affine transformation.

[0029] According to embodiments the encoder may comprise an output comprising a skip connection configured to transmit the intermediate representations in parallel to an encoded and quantized transformed input audio signal. Note, if the skip connections are transmitted, they are quantized.

[0030] According to embodiments the encoder comprises a quantizer configured to quantize the output of the encoder.

[0031] According to embodiments different neural network blocks act on subsets of the transformdomain representation.

[0032] According to embodiments the subsets of the transform-domain representation, e.g., magnitude and phase, are transmitted separately by using potentially different quantizations, e.g., different resolutions.

[0033] According to embodiments different subsets of the transformed input audio signal, e.g., different frequency bands, or magnitude and phase components, are coded separately by different neural network components allowing for individually allocating computational complexity to the processing, e.g. of those different frequency components or of those magnitude and phase components.

[0034] According to embodiments different subsets of the transformed input audio signal, e.g., different frequency components, are quantized and coded at different resolutions.

[0035] According to embodiments all or a subset of all operations in the network is implemented as a complex-valued operation, e.g., interpreting intermediate signal representations as complex numbers.

[0036] According to embodiments all operations in the network are implemented in a causal manner, e.g., only relying on past data or limited lookahead data, thereby allowing for a stream-wise encoding of the audio signal.

[0037] Decoder Aspects

[0038] Another embodiment provides an audio decoder configured to decode a target representation of the audio signal. The decoder comprises a first processing block and a second processing block. The first processing block is configured for conditioning from a quantized representation a set of learnable layers including a styling layer and to obtain a decoded transformed representation. The second processing block realizing a time-feature fusion unit is configured to transform the decoded transformed representation to obtain a generated audio signal.

[0039] Another embodiment refers to an audio decoder configured to decode an audio signal, comprising a first processing block, a combiner and a second processing block. The first processing block is configured to process a quantized representation by a set of learnable layers to obtain different subsets of a decoded transformed representation. The combiner is configured to combine the different subsets to obtain the decoded transform representation. The second processing block realizing a time feature fusion unit is configured to transform the decoded transformed representation to obtain a generated audio signal.

[0040] The finding of these two decoder aspects are comparable to the finding forming the basis for the encoder aspects. Consequently, the coding can be improved with regard to efficiency when a neural decoder is combined with a signal transforming entity realized as time-feature fusion unit. The efficiency can be increased as well when the neural decoder uses different subsets as decoded transformed representation.

[0041] According to embodiments the first processing block includes as the set of learnable layers at least one temporal adaptive DE-normalization layer (TADE) or styling layer involving a learned linear modulation or learned affine transform, wherein the temporal adaptive DE- normalization layers (TADE) or styling layers are elements of a (TADE) residual block, where the temporal adaptive DE-normalization layers (TADE) or styling layers are part of a residual function and combined to a skip-connection or residual connection.

[0042] According to embodiments the set of learnable and styling layers enables skip connections from an audio encoder side to styling layers of an audio decoder.

[0043] According to embodiments the styling layers are conditioned by the input or output of previous learnable layers or by pre-processed conditioning signal derived from the quantized representation.

[0044] According to embodiments TADE or styling layers are configured to provide scale and shift parameters for each intermediate representation and successively apply them to the output as affine transformation. According to embodiments the second processing block is configured to perform timefrequency inverse transformation (inverse STFT, inverse MDCT, PQMF synthesis, ...).

[0045] According to embodiments the second processing block is configured to involve an inverse time-frequency transformation that only relies on the current or on the current and past decoded transformed representation.

[0046] According to embodiments the second processing block realizes a time-feature fusion unit with learned filters, e.g., by convolutional neural network layers.

[0047] According to embodiments the first block comprising the TADE or styling layers by subsequent upsampling a styling information, i.e. , by computing a representation of the styling information at increased time resolution or frequency resolution or channel dimension based on a representation of the quantized representation.

[0048] According to embodiments the decoder comprises a fusion network to combine the encodings of different frequencies and / or of different input features.

[0049] According to embodiments the first processing block comprises a network component fusing different encodings for different subsets of the target representation.

[0050] According to embodiments the first processing block is configured to apply conditioning feature parameters.

[0051] According to embodiments the first processing block is configured to use different computational power or different number of learnable parameters to process the different subsets.

[0052] According to embodiments the second processing block is configured to perform a transformation which is not an (approximate) inverse of the transformation performed at the encoder side.

[0053] According to embodiments the different subsets of the transformed input signal represents different frequency bands and / or wherein the different subsets are transformed into different domains. According to embodiments the different subsets of the transformed input signal represent magnitude and phase of a complex-valued frequency-domain representation.

[0054] According to embodiments the different subsets are coded using different data rates.

[0055] According to embodiments the encoder and decoder side transformation are inverse to each other and / or share same / similar parameters.

[0056] According to embodiments the encoder side transformation and decoder side transformation being parametrized differently, e.g., by different window lengths, transform window solutions or window overlap of the transform.

[0057] According to embodiments, the encoder and decoder side transformation, also referred to as conditioning information (input) are either learned, e.g., DPCRNN (DualPathConvRNN), or signal processing based. In other words, the DPCRNN may be seen as a way of learning this conditioning information from the quantized latent representation.

[0058] According to embodiments the second (or first) processing block comprises a DualPathConvRNN (DPCRNN) neural network block containing a sequence of operations for combining different input features, especially by convolutional layers, learning temporal context by recurrent layers and again combining features by convolutional layers.

[0059] According to embodiments the first processing block comprises a dequantizer configured to dequantize the quantizes representation.

[0060] According to embodiments different neural network blocks act on subsets of the transformdomain representation.

[0061] According to embodiments the subsets of the transform-domain representation, e.g., magnitude and phase, are received separately by using potentially different quantizations, e.g., different resolutions.

[0062] According to embodiments different subsets of the transformed input audio signal, e.g., different frequency bands, or magnitude and phase components, are decoded separately by different neural network components allowing for individually allocating computational complexity to the processing, e.g., of those different frequency components or of those magnitude and phase components.

[0063] According to embodiments received different subsets, e.g., different frequency components, are quantized and coded at different resolutions.

[0064] According to embodiments all or a subset of all operations in the network is implemented as a complex-valued operation, e.g., interpreting intermediate signal representations as complex numbers.

[0065] Method Aspects

[0066] According to further embodiments, a method for encoding the input audio signal is provided. The method comprises transforming by using a time-feature decomposition the input audio signal into a transform domain to obtain a transformed input audio signal; generating a quantized representation of the transformed input audio signal by using a set of learnable and styling layers (TADE layers).

[0067] According to a further embodiment, another method for encoding an input audio signal is provided. This method comprises transforming by using a time-feature decomposition the input audio signal into a transform domain to obtain a transformed input audio signal; wherein the transformed input signal is divided into different subsets, generating a quantized representation of the transformed input audio signal by processing differently the different subsets by using at least a learnable layer.

[0068] Another embodiment refers to a method for decoding an audio signal. Here, the method comprises conditioning from a quantized representation a set of learnable layers including a styling layer and obtaining a decoded transformed representation; transforming by using a time-feature fusion unit the decoded transformed representation to obtain the generated audio signal.

[0069] According to another embodiment the method comprises processing a quantized representation by a set of learnable layers to obtain different subsets of a decoded transformed representation; combining the different subsets to obtain the decoded transformed representation; transforming by using a time-feature fusion unit the decoded transformed representation to obtain the generated audio signal. Of course, the methods may be computer implemented. Therefore, further embodiments refer to a computer program for performing, when running on a computer, the steps of the corresponding methods.

[0070] Further

[0071] Further embodiments refer to a system comprising an encoder and decoder as discussed above. According to embodiments, the encoder and the decoder may be trained together. Therefore, another embodiment refers to an apparatus and method for training the above system.

[0072] Below, embodiments of the present invention will subsequently be discussed referring to the enclosed figures, wherein

[0073] Fig. 1 shows schematically a block diagram of a codec architecture, where encoder, decoder and quantizer may be trained together;

[0074] Fig. 2 shows a schematic block diagram of a codec architecture according to embodiments to illustrate the encoder according to embodiments and the decoder according to embodiments;

[0075] Fig. 3 shows a schematic block diagram of a codec architecture according to further embodiments to illustrate the encoder according to further embodiments and the decoder according to further embodiments;

[0076] Fig. 4 shows a schematic block diagram for illustrating the codec architecture according to embodiments according to another aspect;

[0077] Fig. 5 shows schematically a block diagram of a proposed neural audio codec architecture having variables in brackets denote the output channels and number of output samples / frames to illustrate optional features of embodiments;

[0078] Fig. 6a-c show schematically Convolutional network blocks: Fig. 6a) Design of ConvBlocks (BN is not used for the EncBlocks) used in the Fig. 6b) EncBlocks and Fig. 6c) DecBlocks to illustrate optional features of embodiments;

[0079] Fig. 7 shows a schematic block diagram of DPRNN network block (BN is not used on encoder side) to illustrate optional features of embodiments; and

[0080] Fig. 8a-b show schematically the TADE layer being a component of the DecBlocks (Fig. 8a) as well for the TADEResblock (Fig. 8b) to illustrate optional features of embodiments.

[0081] Below, embodiments of the present invention will subsequently be discussed referring to the enclosed figures, wherein identical reference numerals are provided to objects having identical or similar functions.

[0082] As discussed in context of Fig. 1 , a codec architecture is typically formed as encoder 100 in combination with a decoder 200, where a quantizer 150 may be arranged in between. Below, taking reference to Figs. 2 and 3, enhanced encoders 101 and 102 and enhanced decoders 201 and 202 according to different embodiments will be discussed. The respective encoder and decoder 101 plus 201 and 102 plus 202 form a codec architecture.

[0083] Fig. 2 shows a codec architecture comprising on the encoder side the encoder 101 which is also referred to as second processing block 101 in combination with a first processing block 111. The processing block 111 is a so-called transforming entity, e.g., based on STFT and pre-processes the input signal IS for the encoder 101.

[0084] The decoder 201 is also referred to as first processing block 201 on the decoder side and outputs the decoded transformed representation to a so-called second processing block 211. The second processing block 211 may realize an inverse transformation and further processes the decoded transformed representation to generate an audio signal RS. Between the encoder 101 and the decoder 201 the quantizer 150 is arranged which may be used on the encoder side and the decoder side as well, since for transferring the signal the signal is typically quantized on the encoder side and dequantized on the decoder side.

[0085] Below, the encoder side and the decoder side will be discussed independently. The encoder 101 is the second processing block on the encoder side and includes a set of learnable and styling layers, e.g., TADE layers. These are configured to generate a quantized representation of the transformed input audio signal.

[0086] The transformed input audio signal is provided by the first processing block 111. It realizes a time feature decomposition configured to transform the input audio signal IS into a transform domain to obtain a transformed input audio signal TIS.

[0087] In other words, this means that embodiments are used in input transform TIS for neural codecs 101 that map the input (time-domain) signal IS to a time-frequency representation or a more general transform domain TIS. The use of neural networks in this context permits to leverage the following design choices.

[0088] A computationally efficient codec can be designed by cleverly distributing the processing power of the neural network to different subsets of the transform-domain representation. Similarly, different resolutions for quantizing / coding the resulting latent data stream may be assigned to such subsets. Some examples: For the STFT transform more computational power can be assigned to the processing of the magnitude and less to the processing of the phase, which has been successfully used for neural speech enhancement, e.g., [5], These different parts of the signal representation may be transmitted separately and at different resolution, resulting in a possible trade-off in terms of data rate. For a time- frequency-domain representation, different amounts of computational power can be associated with different frequency bands of the signal representation. Similarly, to the previous point, different data rates may be spent for different frequency bands.

[0089] The transformation can be complex-valued, hence one can make use of complex neural networks that allow for sparse parameterizations resulting in computational efficient solutions.

[0090] Since the powerful mapping learned by the neural codec is able to transform its input to an appropriate representation of its output, the input and output transformation do not need to be approximately inverse to each other. More precisely one can use: Symmetric transformations with different parameters, e.g. STFT and iSTFT with different frame sizes or overlap to reduce possible high delay at the decoder side. Asymmetric fixed transformations, e.g., STFT at encoder side and PQMF synthesis at decoder side. Asymmetric learned transformations, e.g., the DPCRNN (DualPathConvRNN) ) at the encoder side and PQM F filter at the decoder side. As a particular case of each of the above the synthesis window (decoder side) may be implemented in a zero-latency way by applying it only to past signal samples, i.e. , by not employing look-ahead at signal synthesis.

[0091] According to embodiments, the encoder 101 is configured to use a first one-dimensional convolutional layer to adjust the input channels to see learned features. The learned representation, according to embodiments, may be downsampled in the time dimension and upscaled in the feature dimension via a series of dilated residual blocks and does not necessary change the dimensionality of the learned representation; it may learn more complex features via dilated conv layers, activations (e.g. ReLU, ELU, GELU, ...) and optionally via normalization layers (e.g., channel normalization), strided convld which downsamples the learned representation in the time dimension and upsamples the learned channels. Note, the features may directly be taken as the frequency components of the input transform. Furthermore, the upscaling in the feature dimension (possibly the frequency dimension) is also an option. In case of interpreting frequencies as features, downscaling the features may make sense as well.

[0092] To improve gradient flow, a new method for implementing skip connections is proposed on encoder side: Here, TADE layers (StyleMelGAN / NESC) can be employed to send information from different intermediate representations of TIS at the encoder to the output of the encoder.

[0093] Instead of simply adding intermediate results of encoder blocks, as done for typical residual network architectures, scale and shift parameters can be learned for each intermediate encoder representation and successively apply them to the output of the encoder as an affine transformation (shown in blue in the figure 3). These intermediate representations may be transmitted to the (symmetrical) decoder and may be used as additional conditioning there.

[0094] As illustrated by Fig. 1 a quantizer 150 may be combined with the encoder 101. The quantizer may be a VQ, SQ with or without projection of the learned features to a lower dimension representation.

[0095] Below, the decoder side will be discussed. The decoder side comprises the two processing blocks 201 (decoder) and 211 (inverse transform). The first processing block is configured to condition from a quantized representation a set of learnable layers including a styling layer and to obtain a decoded transform representation. For this, the first processing block 101 may include the dequantizer 150. The second processing block 211 is realized as time feature fusion unit configured to transform the decoded transform representation DTR to obtain a generated audio signal RS.

[0096] According to embodiments a quantized representation is upsampled in the time dimension and downsampled in the feature dimension via a series of dilated residual blocks, similar to the ones in the encoder; for example, transpose convld (one-dimensional convolutional) or interpolation + convld layers, which upsamples the learned representation in the time dimension and downsamples the learned channels, may be used.

[0097] According to further embodiments, a last convl d layer which brings the number of channels from C back to the number of input channels, usually activated via tanh. .

[0098] On decoder side we may use similar connections between intermediate representations of the decoder and the input to the decoder.

[0099] This discussed principle of using on the encoder side TADE connections and on the decoder side TADE connections while skipping the connections between the encoder and decoder side is illustrated by Fig. 3.

[0100] Fig. 3 shows the CNN encoder 102 and the CNN decoder 202. Each may be combined with the respective transformation unit (transformation unit on the encoder side 111 arranged at the input of the encoder 102 and inverse transformation unit 211 arranged at the output of the decoder 202). Here, the second processing block 102 on the encoder side may include a set of learnable and styling layers, TADE layers, configured to apply conditioning feature parameters to generate the target representation. For example, an efficient implementation of the TADE connections by subsequent downsampling the conditioning information from network block to network block may be used. As mentioned above, the first processing block 111 on the encoder side involves a time-frequency transform, like STFT or MDCT or PQMF, etc., of the input signal. The decoder, according to embodiments, uses the first processing block 201 for conditioning from the target representation data a set of learnable layers including a styling layer (TADE layer) configured to apply the conditioning feature parameters to a first generated data. The second processing block 211 involves a time-frequency inverse transform (STFT, MDCT or PQMF, ...) to obtain the generated audio signal. According to an efficient implementation of the TADE styling layers subsequently upsampling the styling information from block to block of the neural coder may be used. According to embodiments, the second processing block 211 involves a time-frequency transformation that only realises on past data, i.e. , not using any look-ahead. It is beneficial that same do not introduce any additional signal delay.

[0101] With respect to Fig. 4 a different coding architecture will be discussed. The basic principle having a first processing block 11 T and a second processing 10T on the encoder side and a first processing block 20T and a second processing block 21 T on the decoder side is comparable to the embodiment as discussed in context of Fig. 2. However, the entities 11 T, 10T, 20T and 21 T are implemented slightly differently.

[0102] On the encoder side the first processing block 11 T is enhanced with respect to the entity 111 with respect to the feature that the output transformed input signal is divided into different subsets. For example, they may differ with regard to the frequency bands. According to embodiments, the different subsets may be transformed into different domains. The second processing block 10T uses the different subset and further processes it. Here, the second processing block 10T includes at least a learnable layer configured to generate a quantized representation of the transformed input audio signal by processing differently the different subsets. Note, here it is not necessarily required that styling layers or TADE (temporal adaptive denormalization layers) are involved. Alternatively, learned linear modulation or learned affine transformation may be used by 10T.

[0103] According to embodiments, the encoder 10T may provide different encodings for different subsets, e.g., different frequency bands or - more general - input features. For example, different codings for different subsets of all available frequency bands may be used. In general, this means that the encoder 10T provides different encodings for different subsets of the input presentation in the transform domain, e.g., magnitude and phase. In Fig. 4 the different subsets are marked by the reference numerals TISA and TISB. On the decoder side the first processing block 20T is configured to process a quantized representation by a set of learnable layers to obtain different subsets of the decoded transformed representation. The different subsets are marked by the reference numeral DTRA and DTRB. The different subsets DTRA and DTRB are combined by the entity 215’, also referred to as combiner 215’, to obtain the decoded transform representation. After that, the second processing block 21 T, which is substantially equal to the processing block 211 (realising a time feature fusion unit), transforms the decoded transform representation to a obtain the generated audio signal RS.

[0104] Again the first processing block 20T may include a set of learnable layers, at least one temporal adaptive DE normalization layer or styling layer involving learned linear modulation or learned affine transformation but may also use different layers.

[0105] Regarding the combiner 215’, it should be noted that a fusion free net may be used to combined the encodings for the different frequency (more generally input future) components. The network component combines the different encodings for subsets of the input signal, e.g., magnitude and phase.

[0106] Below, optional features to be used for the codec according to Fig. 2 or the codec according to Fig. 4 may be used.

[0107] According to embodiments the data representation in the transform domain may be complex-valued.

[0108] According to embodiments the encoder- and decoder-side transformation may approximately be inverse to each other and share same / similar parameters.

[0109] According to embodiments the encoder-side transformation and decoder-side transformation being parameterized differently, e.g., by different window lengths, transform resolutions or window overlap of the transforms.

[0110] According to embodiments the encoder- and decoder-side transformations may be either learned (e.g., by a DPCRNN) or signal processing based. Note, the transformations are not necessarily the same kind on encoder and decoder side. According to embodiments different neural network blocks act on subsets of the transformdomain representation, e.g., the magnitude and the phase representation of the complexvalued signals. These network blocks may be of different computational complexity thereby allowing to trade-off the spent computational complexity for the representation of magnitude and phase representation.

[0111] According to embodiments subsets of the transform-domain representation, e.g., magnitude and phase, may be transmitted separately by using potentially different quantizations, e.g., at different resolutions.

[0112] According to embodiments different frequency components of the input signal may be coded separately by different neural network components allowing for individually allocating computational complexity to the processing of those different frequency components.

[0113] According to embodiments different frequency components may be quantized and coded at different resolutions.

[0114] According to embodiments all or a subset of all operations in the network may be implemented as complex-valued operations, i.e., interpreting intermediate signal representations as complex numbers. Thereby, in particular a sparse parameterization of the networks may be achieved, resulting in computational efficient solutions.

[0115] According to embodiments all operations in the network are implemented in a causal manner, i.e., only relying on past data, thereby allowing for stream-wise decoding of the output signal.

[0116] Below, an embodiment for simple and efficient quantization techniques for neural audio coding, will be discussed. The embodiments makes use of the above discussed teachings.

[0117] Network Architecture

[0118] In the following section, the proposed Deep Neural Network (DNN) architecture for neural speech coding is described. It building blocks are illustrated in Figures 5, 6a-c, 7, 8a-b, where DNN blocks are annotated with the corresponding output channels C and number of (temporal) output samples / frames N if they are changed by the respective layer, otherwise we will omit this annotation for readability (we always omit the notation of the batch size B for the same reason). All convolutional layers may be one-dimensional and causal and are denoted them by ConvK,S,D(C,N), where K, S and D denote the kernel size, the stride and the dilation. To simplify notation, S and D is omitted, if they are equal to one and denote convolution with K = 1 as Convl xl .

[0119] Overview

[0120] The DNN architecture of the proposed neural audio codec is depicted in Fig. 5 according to an embodiment. The input signal is processed by an analysis block 501 which may be realized by a signal processing or learned analysis filter bank. The analysis and synthesis filter banks may be realized by the Short-Time Fourier Transform (STFT). The input signal waveform is transformed into a complex-valued STFT representation Sf,n, where n and f denote the frame and frequency index, respectively. To avoid dominance of high-energy components, the STFT signals are compressed

[0121] |Sf,n|aeJSf’n, (l) where a > 0 is a compression factor (we use 0.3 in the following for example), j is the imaginary unit and ( ) denotes the phase. Subsequently, the compressed STFT signal is represented by real and imaginary part which are stacked along the frequency dimension, resulting in an input tensor X e RBX2F X N for t esubsequent neural network block, where B denotes the batch size, F the number of non-redundant STFT bins and N the number of frames. The feature representation outputted by the analysis block is successively downsampled by a cascade of encoder blocks 502, 503 (see Sec. Encoder Block), where the channel dimension coincides with the frequency dimension of the analysis block. Afterwards, a DPRNN layer 504 (see Sec. DPRNN) is applied to cover long-term signal relationships. To improve gradient flow and to effectively combine intermediate signal representations at the encoder before quantization 515, TADEResblocks 517 (see Sec. TADE and TADE-ResBlock) are appended conditioned on the outputs of the encoder blocks. The obtained quantized latent is processed by a decoder roughly symmetrical to the encoder, starting with a DPRNN 505 and followed by decoder blocks upsampling the latent successively (see 506 and 507). Here, each decoder block is conditioned on the quantized latent realizing a styling transform similar to StyleMelGAN. The output of the final decoder block is processed by a synthesis block 608 (symmetric to the analysis block) to reconstruct the time-domain signal waveform, where the inverse STFT is preprocessd by the the expansion

[0122] Encoder Block

[0123] According to embodiments, (the or) each encoder block potentially downsamples the input signal by a strided convolutional layer 601 that may also adjust the number of channels (see Fig. 6b). The result is further processed by two ConvBlocks 602 and 603 where the first one doubles the number of channels and the second restores the number of input channels again. Both ConvBlocks do not alter the time resolution.

[0124] Example of a ConvBlock is given by Fig. 6a. Each ConvBlock applies a GELLI activation 611 to the input (for the encoder blocks, no BN is used), which is followed by a channelwise convolution 612 with kernel size K and same number of output channels C. Then, ChannelNorm 613 is followed by a Conv1 *1 increasing the channels to Cup and a GELLI activation (see reference numeral 614). Finally, the channels are again changed to the number of input channels by another Conv1 *1 (see reference numeral 615).

[0125] The usage of ConvBlocks at the DecBlock is illustrated by Fig. 6c.

[0126] DPRNN

[0127] The DPRNN normalizes according to embodiments the input by BN (only on decoder side, on encoder side all BNs are dropped) and applies a GELLI activation (see reference numeral 701) . Channels are shuffled by Conv1 *1 (702) and the result is normalized by BN and activated by GELLI (cf. 703). A GRU 704 layer is used for modeling temporal context, which is followed by another BN, GELU (cf. 705) and Conv1 *1 (cf. 706).

[0128] TADE and TADE-ResBlock

[0129] The encoder output is according to embodiments styled by a sequence of TADERes- Blocks that are conditioned on the outputs of the EncBlocks, as illustrated by Fig. 8a and 8b. The input of the TADEResBlock 802 is styled by a TADE layer 801 and activated by softmax- gated tanh activation, i.e. , is processed in parallel by conv layers one with softmax and the other one with tanh activation where the results are multiplied together. This structure is repeated where the conv layers of the gated activation are now parameterized with dilation. Both structures together are then realized as a ResBlock.

[0130] For example, the TADE layer 801 is a building block of the TADEResBlocks as well as for the DecBlocks. It normalizes the input by ChannelNorm and adjusts it by an affine transform. The scale and shift parameters of the affine transform are learned from a conditioning signal that is preprocessed by up- or downsampling and a conv layer activated by LeakyReLU. From this, scale and shift parameters are calculated by conv layers.

[0131] Decoder Block

[0132] Similar as in StyleMelGAN the latent is according to embodiments styled by a sequence of decoder network blocks conditioned on the quantized latent, as illustrated by Fig. 6c. Similar to the EncBlocks, the signal is processed by two ConvBlocks (cf. 621 , 622), where the first one doubles the input channels and the second restores the number of input channels again. Differently to its use in the EncBlocks, the ConvBlocks in the DecBlocks use BN. The output of the ConvBlocks may be styled by a TADE layer 623 conditioned on the quantized latent. Finally, a transposed conv layer upsamples the signal (cf. 624).

[0133] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

[0134] The inventive encoded audio signal can be stored on a digital storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet. Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

[0135] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0136] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.

[0137] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0138] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0139] A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non- transitionary.

[0140] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet. A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0141] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0142] A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0143] In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

[0144] The above described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.

[0145] References

[0146] [1] Pia, Nicola et al., “NESC: Robust Neural End-2-End Speech Coding with GANs”, Soundstream: An End-to-End Neural Audio Codec”,

[0147] [3] Defossez Alexandre et al., “High Fidelity Neural Audio Compression”, https: / / arxiv.org / abs / 2210.13438

[0148] [4] Du, Zhihao et al., “FunCodec: A Fundamental, Reproducible and integrable open-source Toolkit for Neural Speech Codec”, 2023,

[0149] [5] Shetu, Shrishti Saha et al.: “Ultra Low Complexity Deep learning Based Noise Suppression”, 2023, https: / / arxiv.org / pdf / 2312.08132.pdf

Claims

Claims1. Audio encoder, configured to encode an input audio signal (IS), comprising: a first processing block (111) realizing a time-feature decomposition configured to transform the input audio signal (IS) into a transform domain to obtain a transformed input audio signal (TIS); a second processing block (101) including a set of learnable and styling layers (TADE layers) configured to generate a quantized representation of the transformed input audio signal (TIS).

2. Audio encoder, configured to encode an input audio signal (IS), comprising: a first processing block (11 T) realizing a time-feature decomposition configured to transform the input audio signal (IS) into a transform domain to obtain a transformed input audio signal (TIS); wherein the transformed input signal is divided into different subsets (TISA, TISB), a second processing block (10T) including at least a learnable layer configured to generate a quantized representation of the transformed input audio signal (TIS) by processing differently the different subsets (TISA, TISB).

3. Audio encoder according to claim 2, wherein the second processing block (101 , 10T) includes as the at least one learnable layer a set of Temporal Adaptive DE- normalization layers (TADE) or styling layers and / or involves a learned linear modulation or learned affine transform.

4. Audio encoder according to claims 1 or 3, wherein the Temporal Adaptive DE- normalization layers (TADE) or styling layers are elements of a (TADE) residual block, where the Temporal Adaptive DE-normalization layers (TADE) or stylinglayers are part of a residual function and combined to a skip-connection or residual connection.

5. Audio encoder according to claim 1 , 3 or 4, wherein the set of learnable and styling layers enables skip connections from an audio encoder side to styling layers of an audio decoder.

6. Audio encoder according to claims 1 or 3, 4 or 5, wherein the styling layers are conditioned by the output of previous learnable layers.

7. Audio encoder according to any one of the previous claims, wherein the second processing block (101 , 101’) comprises a DualPathConvRNN (DPCRNN) neural network block containing a sequence of operations for combining different input features, especially by convolutional layers and / or learning temporal context by recurrent layers and again combining features by convolutional layers.

8. Audio encoder according to any one of the previous claims, wherein the second processing block (101 , 101’) is configured to apply conditioning feature parameters to generate the quantized representation or an intermediate representation of it.

9. Audio encoder according to any one of the previous claims, wherein the first processing block (111 , 11 T) is configured to perform time-frequency transformation (STFT, MDCT, PQMF, ...) of the input audio signal (IS), especially to realize a filterbank with predetermined filters.

10. Audio encoder according to any one of the previous claims, wherein the first processing block (111 , 11 T) realizing the time-feature decomposition comprises one or more filters or one or more learned filters or convolutional neural network layers.

11. Audio encoder according to any one of the previous claims, wherein the different subsets (TISA, TISB) of the transformed input signal represents different frequency bands and / or wherein the different subsets (TISA, TISB) are transformed into different domains.

12. Audio encoder according to any one of the previous claims, wherein the transformed input audio signal (TIS) is a complex-valued signal.

13. Audio encoder according to any one of the previous claims, wherein the second processing block (101 , 10T) is configured to use different data rates to encode the different subsets (TISA, TISB).

14. Audio encoder according to any one of the previous claims, wherein the second processing block (101 , 101’) is configured to use different computational power or different number of learnable parameters to process the different subsets (TISA, TISB).

15. Audio encoder according to any one of the previous claims, wherein the different subsets (TISA, TISB) of the transformed input signal (TIS) represent magnitude and phase of a complex-valued frequency-domain representation.

16. Audio encoder according to any one of the claims 2 to 15, wherein the (TADE) residual blocks are configured to provide intermediate representations of the transformed input audio signal (TIS) to an output of the second processing block (101 , 10T).

17. Audio encoder according to any one of the previous claims, wherein the second processing block (101 , 10T) is configured to downsample the conditioning information of the TADE or styling layers subsequently, especially by lowering the time resolution or frequency resolution or channel dimension.

18. Audio encoder according to any one of the claims 2 to 17, wherein TADE or styling layers are configured to provide scale and shift parameters for each intermediate representation and successively apply them to the output as affine transformation.

19. Audio encoder according to any one of the previous claims, wherein the encoder comprises an output comprising a skip connection configured to transmit the intermediate representations in parallel to an encoded and quantized transformed input audio signal (TIS).

20. Audio encoder according to any one of the previous claims, wherein the encoder comprises a quantizer configured to quantize the output of the encoder.

21. Audio encoder according to any one of the previous claims, wherein different neural network blocks act on subsets (TISA, TISB) of the transform-domain representation.

22. Audio encoder according to any one of the previous claims, wherein the subsets (TISA, TISB) of the transform-domain representation, e.g., magnitude and phase, are transmitted separately by using potentially different quantizations, e.g., different resolutions.

23. Audio encoder according to any one of the previous claims, wherein different subsets (TISA, TISB) of the transformed input audio signal (TIS), e.g., different frequency bands, or magnitude and phase components, are coded separately by different neural network components allowing for individually allocating computational complexity to the processing, e.g. of those different frequency components or of those magnitude and phase components.

24. Audio encoder according to any one of the previous claims, wherein different subsets (TISA, TISB) of the transformed input audio signal (TIS), e.g., different frequency components, are quantized and coded at different resolutions.

25. Audio encoder according to any one of the previous claims, wherein all or a subset of all operations in the network is implemented as a complex-valued operation, e.g., interpreting intermediate signal representations as complex numbers.

26. Audio encoder according to any one of the previous claims, wherein all operations in the network are implemented in a causal manner, e.g., only relying on past data, thereby allowing for a stream-wise encoding of the audio signal.

27. Method for encoding an input audio signal (IS), comprising: transforming by using a time-feature decomposition the input audio signal (IS) into a transform domain to obtain a transformed input audio signal (TIS); generating a quantized representation of the transformed input audio signal (TIS) by using a set of learnable and styling layers (TADE layers).

28. Method for encoding an input audio signal (IS), comprising:transforming by using a time-feature decomposition the input audio signal (IS) into a transform domain to obtain a transformed input audio signal (TIS); wherein the transformed input signal is divided into different subsets (TISA, TISB, DTRA, DTRB), generating a quantized representation of the transformed input audio signal (TIS) by processing differently the different subsets (TISA, TISB, DTRA, DTRB) by using at least a learnable layer.

29. Computer program for performing, when running on a computer the method of claim 27 or 28.

30. Audio decoder configured to decode an audio signal, comprising: a first processing block (201) configured to condition from a quantized representation a set of learnable layers including a styling layer and to obtain a decoded transformed representation (DTR); a second processing block (211) realizing a time-feature fusion unit configured to transform the decoded transformed representation to obtain a generated audio signal (RS).

31. Audio decoder configured to decode an audio signal, comprising: a first processing block (20T) configured to process a quantized representation by a set of learnable layers to obtain different subsets (DTRA, DTRB) of a decoded transformed representation; a combiner (215’) configured to combine the different subsets (DTRA, DTRB) to obtain the decoded transformed representation;a second processing block (21 T) realizing a time-feature fusion unit configured to transform the decoded transformed representation to obtain a generated audio signal (RS).

32. Audio decoder according to claim 31 , wherein the first processing block (201 , 20T) includes as the set of learnable layers at least one temporal adaptive DE- normalization layer (TADE) or styling layer involving a learned linear modulation or learned affine transform.

33. Audio decoder according to claim 32, wherein the temporal adaptive DE- normalization layers (TADE) or styling layers are elements of a (TADE) residual block, where the temporal adaptive DE-normalization layers (TADE) or styling layers are part of a residual function and combined to a skip-connection or residual connection.

34. Audio decoder according to claim 30 or 33, wherein the set of learnable and styling layers enables skip connections from an audio encoder side to styling layers of the audio decoder.

35. Audio decoder according to claim 30, 32, 33 or 34, wherein the styling layers are conditioned by the output of previous learnable layers or a pre-processed conditioning signal derived from the quantized representation.

36. Audio decoder according to claim 30, 32, 33, 34 or 35, wherein TADE or styling layers are configured to provide scale and shift parameters for each intermediate representation and successively apply them to the output as affine transformation.

37. Audio decoder according to any one of the claims 30 to 36, wherein the second processing block (211 , 21 T) is configured to perform time-frequency inverse transformation (inverse STFT, inverse MDCT, PQMF synthesis, ...).

38. Audio decoder according to any one of the claims 30 to 37, wherein the second processing block (211 , 21 T) is configured to involve an inverse time-frequencytransformation that only relies on the current or on the current and past decoded transformed representation (DTR).

39. Audio decoder according to any one of the claims 30 to 38, wherein the second processing block (211 , 211’) realizes a time-feature fusion unit with learned filters, e.g., by convolutional neural network layers.

40. Audio decoder according to any one of the claims 30 to 39, wherein the first block comprising the TADE or styling layers by subsequent upsampling a styling information, i.e., by computing a representation of the styling information at increased time resolution or frequency resolution or channel dimension based on a representation of the quantized representation.

41. Audio decoder according to any one of the claims 30 to 40, wherein the decoder comprises a fusion network to combine the encodings of different frequencies and / or of different input features.

42. Audio decoder according to any one of the claims 30 to 41 , wherein the first processing block (201 , 201’) comprises a network component fusing different encodings for different subsets (DTRA, DTRB) of the target representation.

43. Audio decoder according to any one of the claims 30 to 42, wherein the first processing block (201 , 20T) is configured to apply conditioning feature parameters.

44. Audio decoder according to any one of the claims 30 to 43, wherein the first processing block (201 , 201’) is configured to use different computational power or different number of learnable parameters to process the different subsets (DTRA, DTRB).

45. Audio decoder according to any one of the claims 30 to 44, wherein the second processing block (211 , 21 T) is configured to perform a transformation which is not an (approximate) inverse of the transformation performed at the encoder side.

46. Audio decoder according to any one of the claims 30 to 45, wherein the different subsets (DTRA, DTRB) of the transformed input signal represents differentfrequency bands and / or wherein the different subsets (DTRA, DTRB) are transformed into different domains.

47. Audio decoder according to any one of the claims 30 to 46, wherein the different subsets (DTRA, DTRB) of the transformed input signal represent magnitude and phase of a complex-valued frequency-domain representation.

48. Audio decoder according to any one of the claims 30 to 47, wherein the different subsets (DTRA, DTRB) are coded using different data rates.

49. Audio decoder according to any one of the claims 30 to 48, wherein the encoder and decoder side transformation are inverse to each other and / or share same / similar parameters.

50. Audio decoder according to any one of the claims 30 to 49, wherein the encoderside transformation and decoder-side transformation being parametrized differently, e.g., by different window lengths, transform window solutions or window overlap of the transform.51 . Audio decoder according to any one of the claims 30 to 50, wherein the encoder and decoder side transformation are either learned, e.g., DPCRNN (DualPathConvRNN), or signal processing based, e.g., PQMF.

52. Audio decoder according to any one of the claims 30 to 51 , wherein the first or second processing block (201 , 20T) comprises a DualPathConvRNN (DPRNN) neural network block containing a sequence of operations for combining different input features, especially by convolutional layers, learning temporal context by recurrent layers and again combining features by convolutional layers.

53. Audio decoder according to any one of the claims 30 to 52, wherein the first processing block (201 , 20T) comprises a dequantizer configured to dequantize the quantizes representation.

54. Audio decoder according to any one of the claims 30 to 53, wherein different neural network blocks act on subsets (DTRA, DTRB) of the transform-domain representation.

55. Audio decoder according to any one of the previous claims, wherein the subsets (DTRA, DTRB) of the transform-domain representation, e.g., magnitude and phase, are received separately by using potentially different quantizations, e.g., different resolutions.

56. Audio decoder according to any one of the claims 30 to 54, wherein different subsets (DTRA, DTRB) of the transformed input audio signal (TIS) (IS) (TIS), e.g., different frequency bands, or magnitude and phase components, are decoded separately by different neural network components allowing for individually allocating computational complexity to the processing, e.g. of those different frequency components or of those magnitude and phase components.

57. Audio decoder according to any one of the claims 30 to 56, wherein received different subsets (DTRA, DTRB), e.g., different frequency components, are quantized and coded at different resolutions.

58. Audio decoder according to any one of the claims 30 to 57, wherein all or a subset of all operations in the network is implemented as a complex-valued operation, e.g., interpreting intermediate signal representations as complex numbers.

59. Method for decoding an audio signal, comprising: conditioning from a quantized representation a set of learnable layers including a styling layer and obtaining a decoded transformed representation; transforming by using a time-feature fusion unit the decoded transformed representation to obtain the generated audio signal.

60. Method for decoding an audio signal, comprising: processing a quantized representation by a set of learnable layers to obtain different subsets (DTRA, DTRB) of a decoded transformed representation; combining the different subsets (DTRA, DTRB) to obtain the decoded transformed representation;transforming by using a time-feature fusion unit the decoded transformed representation to obtain the generated audio signal.

61. Computer program for performing, when running on a computer the method of claim 59 or 60.

62. System comprising an audio encoder according to any one of the claims 1 to 26 and an audio decoder according to any one of the claims 30 to 58.

63. System, where the second processing block (211, 211’) of the audio encoder (101, 101’) and of the first processing block (201, 20T) of the audio decoder are trained together.

Images