Apparatus and method for decoding an encoded audio signal using a conditional flow matching based process

Abstract

An audio decoder for decoding an encoded audio signal (4) to obtain a decoded audio signal (40) comprises: an input interface (50) for receiving the encoded audio signal (4); and a learned conditional flow matching based processing stage (70) for transforming the encoded audio signal (4) into the decoded audio signal (40). In an embodiment, the learned conditional flow matching based processing stage (70) comprises a learned conditional flow matching core (7) comprising: a first input (6a) for first input data derived from the encoded audio signal (4); and a second input (6b) for second input data representing conditional data; and an output (7a) for decoded audio data.

Description

[0001] Apparatus and Method for Decoding an Encoded Audio Signal Using a Conditional Flow Matching Based Process Specification The present invention relates to audio encoding, audio decoding or a system consisting of an encoder and a decoder and is related to a novel neural audio codec for low bit rate audio coding based on flow matching (FM). Embodiments of FlowMAC (Flow Matching Audio Coding) are able to encode and decode wideband speech and music at 1.5, 3 and 6 kbps at high quality. In recent years, audio codecs based on Deep Neural Networks (DNNs) have become state of the art for low and mid bit rates. These models use so-called deep generative models, i.e., DNNs that model the probability distribution of the audio in the dataset. This modelling procedure enables sampling (synthesis) of new data points with similar characteristics. This synthesis procedure can be conditioned on side-information which specifies the content of the audio to generate. In the context of audio coding the conditioning information at the decoder side is usually the quantized bit stream, but could potentially be another feature extracted fromit, e.g., a decoded spectrum.In the modern digital world, audio coding systems are used on a day to day basis, hence every advancement in compression capabilities can lead to huge benefits. In recent years, deep neural networks (DNNs) have revolutionized the field of audio compression. Many of the early DNN-based approaches [1]-[3] control the compression at training time via entropy-based losses and ensure good quality via spectral reconstruction losses. However, quality at low bitrates was often unmatched to those of classical coding techniques. With the advent of deepgenerative models, the quality of neural codecs at bit rates lower than 6 kbps greatly improved.While for speech coding many different approaches were proven to be successful [4]-[7], for general audio coding the ground-breaking SoundStream codec [8] established the paradigmof training a VQ-VAE [9] via an additional GAN loss end-to-end (e2e).In e2e neural coding an encoder model extracts a learned latent from the input signal. This latent is then quantized via a residual vector quantizer and decoded via a neural network decoder. All the modules are jointly learned via a combination of multiple spectral reconstruction losses, VQ-VAE codebook and commitment losses and adversarial losses. This framework permits to train a codec for multiple bit rates thanks to the scalability offered by the residual quantizer.Various improvements on the design of SoundStream were proposed throughout the years.EnCodec

[0010] introduced the use of recurrent networks and improved on the compression via entropy coding based on language models on the quantizer. The Descript-Audio-Codec (DAC)

[0011] achieved unprecedented quality using a much bigger model based on innovative audio- specific activations

[0012] and scaling up the discriminator architecture. While the e2e VQ-GAN approach offers a great flexibility in the design and complexity of the codec

[0013] –

[0015] , it often entails a complicated and unstable training pipeline while sometimes failing to meet quality expectations for challenging signal types, such as music. To achievehigher quality at lower bit rates Denoising Diffusion Probabilistic Models (DDPMs) wereproposed recently for speech

[0016] and general audio

[0017] ,

[0018] . MultiBandDiffusion (MBD)

[0017] is a decoder model that permits high-quality decoding of the EnCodec latent at 1.5, 3 and 6 kbps. This model uses a time-domain subband-based decoding scheme and achieves state-of-the-art quality for music. The high complexity of this modelmakes it hard to use in embedded devices and its dependency on a pre-trained bit streammight limit its compression capabilities. All of these approaches entail either a highly involved training pipeline or use computationally heavy models. Recently, a new paradigm to train continuous normalizing flows (CNFs) called conditional flow matching (CFM) has emerged

[0019] and demonstrated state-of-the-art qualityfor both image

[0026] and audio generation

[0020] ,

[0021] . This approach offers simulation- freetraining of CNFs at much lower inference and training costs compared to DDPMs.Fig.1 illustrates a straightforward encoder / decoder scenario. Input audio data 20 are encodedby an encoder 100 in order to generate a bit stream 4. The bit stream 4 is transmitted to a decoder 200. In the decoder 200, the bit stream is decoded to obtain a decoded audio signal. One family of generative models often adopted are Generative Adversarial Networks (GANs) [E1,E2,E3]. In this context GANs allow end-to-end learning of encoder, quantizer and decoder architecture. These models are computationally efficient because each module is used a single time during synthesis. They are trained via a minimax game between generator and discriminator, where the generator synthesizes new samples and the discriminator “criticizes” their quality. In this way the two models bootstrap their capabilities via stochastic gradient descent. Denoising Diffusion Probabilistic Models (DDPMs) are another family of generative models that was recently applied in the context of audio coding [E4,E5,E6,E7]. These models learnhow to synthesize audio step-by-step, via progressively “denoising” a sample from a Gaussiandistribution (so called prior distribution), see Fig.2. While codecs based on these models could in principle be trained to jointly learn encoder, quantizer, and decoder, existing work only learns the decoder [E5,E6] or jointly learns a sub- module in the encoder [E7]. In particular, [E5] and [E6] learn to decode a fixed bit stream directly in time domain or in subband-time domain. To this end specific tricks for the denoising schedule are needed and it is unclear whether the bit stream can be learned jointly. In [E7] some sub-modules of the encoder architecture are jointly learned with the DDPM-based decoder. However additional sub-modules in the encoder are needed and the DDPM works in the latent domain of a Variational AutoEncoder (VAE). Hence for the generation of the mel- spectrogram it is needed to first encode the VAE latent, then run the DDPM synthesis (conditioned on the bit stream) and finally decode with the VAE decoder. It is an object of the present invention to provide an improved concept for audio encoding and / or audio decoding. This object is achieved by an audio decoder of claim 1, an audio processing system of claim35, a method of decoding of claim 36, a method of audio processing of claim 37, or a computerprogram of claim 40. In accordance with the present invention, an audio decoder is configured for decoding an encoded audio signal to obtain a decoded audio signal. The audio decoder comprises an input interface for receiving the encoded audio signal and a processing stage which is based on a learned conditional flow matching technology. The processing stage is configured for transforming the encoded audio signal into the decoded audio signal.In an embodiment, the input interface is configured to receive, as the encoded audio signal, abit stream, and the learned conditional flow matching base processing stage is configured to transform the bit stream or a signal derived from the bit stream such a spectral data into a time domain waveform as the decoded audio signal. The learning conditional flow matching base processing stage comprises a learned conditional flow matching core. The learned conditionalflow matching core comprises a first input for first input data derived from the encoded audiosignal and a second input for second input data representing conditional data and an outputfor the decoded audio data.Particularly, the learned conditional flow matching based processing stage is configured toimplement the functionalities of a spectral decoder to derive raw spectral data from theencoded audio signal and the learned conditional flow matching core is configured fortransforming the raw spectral data into fine spectral data, and a spectrum-time converter is provided for converting the fine spectral data into a time domain representation representingthe decoded audio signal. The raw spectral data can be distorted or coarsely quantizedspectral data and the fine spectral data can be restored or finely quantized spectral data. Generally, the fine spectral data approximate the original audio signal closer compared to the raw spectral data. In other words, the audio quality obtained by the raw spectral data is lower than the audio quality obtained by the fine spectral data. In embodiments, the raw spectral data are raw mel spectral data, and the fine spectral data are fine mel spectral data. Other embodiments rely on a time-frequency representation in the form of a short time Fourier transform representation with a sequence of complex-valuedspectra. Other embodiments rely, as the raw spectral data, on modified discrete cosinetransform data (MDCT) data or modified discrete sine transform (MDST) data as the rawspectral data and the fine spectral data as obtained from the learned conditional flow matching base processing stage. In particular, the learned conditional flow matching base processing core / stage comprises a learned ordinary differential equation estimator for estimating parameters for ordinary differential equations in response to the encoded audio signal and a subsequently connected ordinary differential equation solver for solving the ordinary differential equations determined by the parameters to obtain result data corresponding to the decoded audio signal, or from which the decoded audio signal is finally derived. In the first alternative, the ordinary differential equation solver already outputs the time domain waveform data. However, in the preferred latter alternative, the ordinary differential equation solver outputs spectral data such as mel spectral data or STFT spectral data or MDCT spectral data or MDST spectral data in corresponding sequences of frames and the final calculation of the time domain output data representing the decoded audio signal is done by a straightforward spectrum-time converter that, for the sake of simplicity, does not rely on any neural networks, but is a well-established spectrum-time converter known from conventional audio coding technologies. In preferred embodiments, the learned ordinary differential equation estimator and the ordinary differential equations solver are configured to operate iteratively in at least two steps, where, in the first step, first flow derivatives at the initial condition are provided depending on the encoded audio signal and first estimated solutions are provided based on the initial condition. In the second iteration step, the first estimated solution obtained by the ordinary differential equation solver at the end of the first iteration step is used as the new “initial condition” for providing second flow derivatives by the learned ordinary differential equation estimator and the result of the calculation of the second estimated solution by the ordinary differential equation solver is then the final result or is the input into the third iteration step. The secondflow derivatives are the flow derivatives for the second step, i.e., as obtained as a result of thesecond step. In each of the iterative steps, the encoded audio signal or a signal derived fromthe encoded audio signal is used as the input into the learned ordinary differential equationestimator as the first input data. The conditional data is used as the second input data at the initial condition for the first iteration step, the corresponding estimated solution of the earlier iteration step is used as the conditional data for the next iteration step. The iteration is performed until an iteration termination criterion is achieved. This iteration termination criterion can be a number of maximum iterations such as a number between 10 and 40 iteration steps or can be a criterion that evaluates the change of a current iteration step result with respect to an earlier iteration result. When this change is smaller than a predefined threshold, then the iteration is terminated.Typically, the iteration criterion will consist of more than one criterion. In such a case, thethreshold criterion and the number of iteration steps criterion can be combined with each other. It has been found that particularly the learned condition flow matching base processingtechnology is particularly suited when transforming an audio signal into spectral data on theencoder-side and for retransforming of the encoded audio data into decoded audio data on thedecoder-side. Although the technology based on the learned condition flow matchingtechnology can be used in connection with directly transforming the time domain audio signalinto a bit stream on the encoder-side and directly transforming the bit stream into the timedomain audio signal on the decoder-side, it is preferred to use additional stages so that the flow matching based processing is applied to spectral data. Particularly, the conditional flow matching based processing stage applied by the decoder is a fully learned deep neural network processor where, particularly, the neural network relying on trying parametric data is applied within the learned ordinary differential equation which generates a first order differentialequation representation or, generally, first flow derivatives based on the encoded audio signal,i.e., a raw spectral representation of the audio data as the decoded latent data and based on the extra conditional data representing the initial condition for the calculation of the first estimated solution. The application of the learned conditional flow matching technology on raw spectral data, i.e., a sequence of frames of raw spectral data being mel spectral data or STFT, MDCT, MDST or LPC spectral data provides particularly useful results. Therefore, the decoder architecture that applies a spectral decoder to derive a sequence of raw spectral data from the encoded audio signal which, for example, is available as a bit stream, a subsequently connected conditional flow matching and based processing core relying on a learned ordinary differential equation estimator and the subsequently connected ordinary differential equations solver, whichcooperate in an iterative manner and the subsequently connected spectrum time converterprovide high quality decoded audio signals in the time domain with high quality at low bitrates. In embodiments, the spectral decoder and the spectrum time converter can be implemented to also rely on neural network technology. However, in an embodiment, only the learned ordinary differential equation estimator relies on a deep neural network, which is learned by parametric data, and the other elements, i.e., the spectral decoder, the spectrum-time converter and the ordinary differential equations solver are implemented with conventional technology, i.e., without neural network implementations for the sake of efficiency and transparency and predictability of the whole process.Preferred embodiments of the present invention are discussed with respect to theaccompanying drawings, in which:Fig. 1 illustrates a general encoder / decoder overview, which can be based on neuralnetwork technology;Fig. 2 illustrates the process applied by DDPM for an audio synthesis, where the priorGaussian noise is sequentially “de-noised” in a step-by-step manner;Fig. 3 illustrates an example of a vector field;Fig. 4 illustrates a step-by-step synthesis procedure applied with a learned conditionalflow matching based processing stage decoder;Fig. 5 illustrates an overview over an encoder / decoder system based on conditionalflow matching technology;Fig. 6 illustrates a preferred embodiment of the encoder / decoder framework relyingon conditional flow matching based technology;Fig.7a illustrates a high level description of a CFM decoder in an embodiment;Fig.7b illustrates a more detailed representation of the conditional flow matching basedprocessing applying an ordinary differential equation estimator and an ordinary differential equation solver;Fig.8a illustrates a further embodiment for an encoder / decoder architecture relying onconditional flow matching (CFM) and using mel spectra;Fig.8b illustrates a preferred implementation of the spectral encoder for transformingthe sequence of mel spectra into a bit stream and an implementation of a spectral decoder for converting a bit stream into a sequence of mel spectra; Fig.8c illustrates a preferred implementation of the conditional flow matching procedure illustrated in Fig.8a; Fig.9 illustrates results for a MUSHRA listening test showing superior results for the flow matching technology at very low bitrates; Fig.10 illustrates results for P808 listening test; Fig.11 illustrates a preferred implementation of a specific spectral decoder for generating the decoded latent based on neural network technology relying on convolutional neural networks (CNN) for feature expansion on the one hand and channel expansion on the other hand connected by a transformer stage; Fig.12 illustrates a preferred implementation of an audio decoder in accordance with the present invention; Fig.13 illustrates a preferred embodiment of the audio decoder of Fig.12; Fig.14a illustrates a preferred implementation of the learned conditional flow matching core; Fig.14b illustrates a preferred implementation of the processing steps implementing an iterative architecture;Fig.14c illustrate a preferred implementation of the learned ordinary differential equation estimator of Fig.14a; Fig.15a illustrates a specific procedure for generating learned patches from a spectral representation for further processing with a transformed architecture;Fig.15b illustrates a procedure for unpatching to reconstruct the original spectrum out ofthe learned patches; Fig.16 illustrates a preferred embodiment for implementing the ODE estimator using the conditional as styling of a diffusion transformer implementation of the neural network in the CFM core; and Fig.17 illustrates another preferred embodiment for implementing the ODE estimator using the conditional as prior of a diffusion transformer implementation of the neural network in the CFM core. Fig. 12 illustrates an audio decoder for decoding an encoded audio signal 4 to obtain a decoded audio signal 40. The audio decoder comprises an input interface 50 for receiving the encoded audio signal 4 and a learned conditional flow matching based processing stage 70 for transforming the encoded audio signal obtained from the input interface 50 into the decoded audio signal 40. The encoded audio signal 4 may be a bit stream, a text, a MIDI file, or a representation of a time domain wave form signal, for example via a sequence of encoded spectral data. The learned conditional flow matching based processing stage (cf. M stage 70) is preferably configured to combine timely adjacent frames using an overlap and add procedure. This application of the combination of timely adjacent frames is useful when the encoded audio signal 4 represents mere spectral data or STFT, MDCT, or MDST spectral data. The decoded audio signal 40 is preferably a time domain waveform signal. Fig.13 illustrates a preferred implementation where the encoded audio signal 4 is first the input into a spectral decoder 4b, 5. The spectral decoder 4b, 5 can be, depending on certain implementations, an entropy decoder as conventionally known from state of the art audiotechnologies which converts a bit stream into a sequence of spectral data. Alternatively, thespectral decoder 4b, 5 can be a DNN based decoder or can be any other processing stage that converts a bit stream representing the encoded audio signal 4 into first input data for the conditional flow matching based processing stage. This first input data can be raw mel spectral data, general spectral data for one or more time frames or general time frame data, but are preferably spectral data.The output of the spectral decoder is input into the conditional flow matching core 7 thatadditionally receives second input data as conditional data. These second input data can be noise data such as Gaussian noise data, colored noise data, pseudo noise data, or data from a sampleable probability distribution, or samples from a probability distribution, or even the first input data. The result of the CFM core 7 output at the output thereof are fine spectral data such as fine mel spectral data or fine STFT data or any other preferable spectral data which are then input into the spectrum-time converter 9 which outputs the time domain decoded audio signal 40. The spectrum-time converter can be implemented as a pre-trained neural vocoder or as a straight forward frequency-time converter receiving an FFT representation, an MDST representation, an MDCT representation, an STFT representation or a mel or LPC spectral representation. Depending on the kind of input data, the spectrum-time converter is correspondingly implemented in order to output the decoded audio signal 40. All the elements 4b, 5, 7, 9 together represent a preferred implementation of the learned conditional flow matching based processing stage 70 of Fig.12.In a preferred implementation, the first input data 6a represent the encoded audio signal 4 asreceived by the input interface 50 or, in the implementation of Fig.13, the still encoded audio data at the output of the spectral decoder 4b, 5 which represent spectral data and, particularly, raw spectral data. This latent data 6a is input into the learned ordinary differential equation estimator 71 which cooperates with an ordinary differential equation solver 72 that solves the system of first order ordinary differential equations determined by block 71 for example by integration. Generally, any procedure for solving a system of differential equations can be used such as the Euler procedure, the Runge-Kutta, the dopri or the finite element method can be used for generating a decoded audio signal or generally, the basis for a decoded audio signal. The stages 71, 72 are controlled by an iteration controller 80 which controls both elements in order to cooperate in an iterative manner and which also controls these elements, so that for each iteration step, the corresponding data are provided at the corresponding inputs.Fig. 14b illustrates a flow chart for the iterative procedure controlled by the controller 80. Atline 73, initial condition data 6b are provided to block 710, 720. Block 710 represents the firstiteration step provided by the learned ordinary differential equation estimator based on the encoded / spectral audio signal 6a. The result of the processing step 710 is the representation of the first flow derivatives at the initial condition provided by line 73. Based on the first flow derivatives at the initial condition provided by step 710, step 720 generates a first estimated solution, i.e., is implemented by block 72 of Fig.14a. The result of the ordinary differential equation solving procedure in step 720 is used as the new initial condition for the second iteration step as indicated at line 73a. Based on the first estimated solution, and based on the encoded / spectral audio signal 6a, step 711 applied by block 71 of Fig.14a provides second flow derivatives at the first estimated solution 73a and step 721 implemented by block 72 of Fig.14a provides a second estimated solution 73b. Then, in analogy to Fig.14b, this second estimated solution 73b is used as the conditional data for the third iteration step, and this procedure is continued until an iteration criterion is reached such as a threshold criterion or a maximum number of iteration steps criterion. Fig.14c illustrates an implementation of the learned ordinary differential equation estimator 71 of Fig. 14b. Particularly, this element comprises a neural network 730 which relies on pre- trained parametric data, and based on the conditional data or second input data 6b and first input data 6a, the neural network 730 provides an information on a flow direction and / or flow magnitude which can then be used by the ordinary differential equations solver 72 of Fig.14a. Embodiments of the invention refer to FlowMAC, a novel neural audio codec for high-quality general audio coding at low bit rates based on Conditional Flow Matching (CFM). FlowMACgenerates the bit stream via a learned mel spectrogram encoder, and a vector quantizer. Thedecoder integrates a Continuous Normalizing Flow (CNF) via an ODE solver to generate a high-quality mel spectrogram. Finally, the decoded mel spectrogram is converted to the output waveform by a well-established neural mel-to-audio model. Training FlowMAC is extremely simple and memory efficient. The subjective evaluation shows that FlowMAC at 3 kbpsachieves the same quality of state-of-the-art GAN-based and DDPM-based neural audiocodecs at double the bit rate. Moreover, FlowMAC offers a tunable inference pipeline which permits coding faster than real-time coding on CPU. In an embodiment, the architecture uses a learned Mel spectrogram encoder that generatesfeatures which are quantized for the bit stream. The decoder defines an ODE-based (ODE =Ordinary Differential Equations) synthesis of a Mel spectrogram using the FM objective. The output waveform is generated by a well-established pre-trained neural vocoder method. A new family of generative models introduced in [E8] called Conditional Flow Matching (CFM)models learns an Ordinary Differential Equation (ODE) based on the dataset and synthesizesnew data via following the “flow” of the ODE. This is analogous to learning a “gravitation law”at training time which forces everything (e.g., noise samples) to move towards the“gravitationally heavy” statistics of the dataset (e.g., audio files). At synthesis time one needsto “follow the gravitation law” via integrating the ODE step-by-step, e.g., via finite elementsmethods implemented in ODE-Solvers. The Flow Matching Audio Codec (FlowMAC) is a new audio compression model based on CFM for low bit rate coding of general audio at 24 kHz audio. The proposed solutions learn a mel spectrogram encoder, residual VQ, and decoder via a combination of a simple reconstruction and CFM objectives. The CFM-based decoder generates realistic melspectrograms from the discrete latent, which is then converted to waveform domain via anefficient version of BigVGAN

[0022] . The model design is simple and the training pipeline is stable and efficient. Contributions of embodiments can be summarized as follows: ^A simple and efficient encoder-decoder architecture for mel spectrograms codingbased on CFMs^ Subjective test results show that FlowMAC achieves state-of-the-art quality at 3 kbpsmatching GAN-based and DDPM-based solutions at double the bit rate. ^An efficient version of FlowMAC able to code at high quality and faster than real timeon CPU.More precisely CFMs learn a time-dependent family of invertible transformations ^ → ^for ^ ∈ [0, 1]. This family of transformations is called a “flow”.There are many ways to define a flow, but one of the most useful ones is through a family oftime-dependent vector fields ^^: ^^ → ^^, i.e., for each ^ ∈ [0,1] one has a vector field ^^, i.e.,a mapping that associates to each point in ^^ a vector, i.e., a direction and a magnitude asdepicted in Figure 3. Once a time-dependent vector field ^^ is given then a corresponding flow^^is obtained via integrating the following system of Ordinary Differential Equations With initial conditions ^^(^) = ^.This is called an Initial Value Problem (IVP).Given a flow ^^ and an initial probability distribution ^^, one obtains a family of probabilitydistributions via push-forward Conditional Flow Matching models learn a specific vector field ^^, which transforms the priordistribution ^^ into the target data distribution ^^ = ^^^^^. While there are in principle noconstraints on the prior distribution, often a known noise distribution is chosen, e.g., Gaussian.To train ^^ so that the final distribution ^^ matches the wanted data distribution, one samples adatapoint ^^~^^^^^(^) and define a conditional probability path ^^(^|^^) so that ^^(^|^^) =^^(^), ^^(^|^^) approximates ^^^^^ near ^^ and ^^ is tractable, e.g., Gaussian. Via ^^(^|^^) aconditional vector field generating ^^(^|^^) is denoted.Then by marginalization will generate ^^(^) with the above properties. This “conditional” framework not only permits to efficiently train ^^(instead of the unknown ^^), but it defines it for each sample in an easy to compute form. In particular, it permits to learn the ODE associated to the Optimal Transport Conditional Vector Field, i.e. Where ^ is a small real number. A different choice of vector field subsumes Variance Preserving and Variance Exploding DDPMs in this framework. For more details, reference is made to Section 4.1 in [E8]. Subsequently, additional information relating to flow matching is summarized. Flow matching

[0019] describes a method to fit a time- dependent probability density path^ ^^: [0, 1] × ℝ → ℝ > 0 between a simple sampling distribution ^^(^) and the target datadistribution ^(^), where ^ ∈ [0, 1] and ^ ∈ ℝ^ . Suppose a vector field ^ : [0, 1] × ℝ^ → ℝ^^exists such that its flow ^ : [0, 1] × ℝ^^ → ℝ^ generates the probability path ^^. More precisely^^is the solution of the ODE With initial condition ^^(^) = ^, and the push-forward of ^^ via ^^ is ^^, i.e.Assume ^^(^) = ^(^; ^, ^) and ^^(^) close to the data distribution ^(^), then

[0019] defines theflow matching loss as where ^^(^, ^) outlines a neural network parametrized by ^ that regresses ^^, and the timestep^ ~ ^[0, 1] is sampled from a uniform distribution. As ^^ is unknown, it is intractable to obtain^^via this formulation. As a solution,

[0019] defines the conditional probability path where ^^ ~ ^(^^) is a data point and is a path satisfying ^^(^) = 0 and ^^(^) = ^. ^^ is afunction such that ^^(^) = 1 and ^^(^) ≪ 1. Given a corresponding vector field ^^(^|^^) whoseflow generates ^^(^|^^),

[0019] defines the tractable CFM objective as The optimal transport formulation defined in

[0019] is used for the flow where ^^is sampled from a standard normal distribution. The resulting target vector field can then be inferred as = ^^ − (1 − ^^^^)^^. During inference, an ODE solver isutilized to map from the sampling distribution to the target distribution. This is similar to DDPMs such that an iterative procedure is followed, however CFMs follow much simpler and straighter paths and hence are much more efficient to evaluate. Many works explore the power of such models for image synthesis [E9], Text-To-Speech (TTS) [E10,E11,E12], vocoding [E13], and more general Text-To-Audio (TTA) [E14,E15,E16,E17]. To use CFMs in accordance with the invention it is useful to condition the values of the vector field ^^onto additional information, e.g., a bit stream in the case of coding in accordance with preferred embodiments of the invention or, in general, the encoded audio signal. This additional conditioning can be inserted at various step in the neural network representing ^^, e.g., via concatenation at the input, or at the bottleneck, FiLM (Feature-wise Linear Modulation) conditioning layers as in [E1], etc. This additional conditional informs the vector field, end hence the generative process, about the part of the target distribution ^^^^^which is relevant for the current sample. The following formula explicitly makes clear all the dependencies for the vector field or ODE- Estimator (and hence the flow ^^(^) = ^(^, ^, ^, ^),where- ^ is a point in the domain ^^- ^ indicates the parameters of the neural network- ^ is the optional extra conditioning, e.g., the text or the bit stream- ^ is the current time.A CFM estimates a vector field at training time, at generation time one needs to integrate thisvector field. The usual way to do this is to use an ODE-Solver. On a high-level one canrepresent this procedure as described in Figure 4. In the Fig. 4 embodiment, the upper portion of Fig. 4 illustrates the first step in analogy to blocks 710, 720 of Fig. 14b, and the lower portion with elements 71b, 72b for the ODEestimator and the ODE solver represents an illustration in analogy to block 711, 721 of Fig.14b. Particularly, all the blocks illustrated in Fig.4 are implemented in item 7 of Fig.13, i.e., in the learned conditional flow matching core.It is visible from Fig.4 that at least the ODE solver 72a receives the initial condition 6b, 73 andin the embodiment in accordance with the present invention, this data is preferably also introduced into the ODE estimator 71a for the first step. The output of block 71a on the one hand the input into block 72a on the other hand are the flow derivatives at the initial condition or, generally, as outlined in Fig.14c, the information on a flow direction and / or flow magnitude. In the second step illustrated by block 71b, 72b, the estimated solution 73a from the first steprepresents the initial condition for the ODE solver 72b and preferably also for the ODEestimator 71b, and the output of block 71b is once again the flow derivatives at the current point, i.e., the estimated solution from the first step as illustrated at 73a. The estimated solution 73b then serves as the “second input” for the ODE estimator and / or ODE solver for the third step and, at the same time, the first input 6a is always the same, i.e., the encoded audio signal or the spectral representation of the audio signal corresponding to a raw spectral representation which is, in a sense, also still an encoded version of the final time domain audio signal. In the final step, the estimated solution is represented as being a decoded mel spectrum as a representation of a decoded audio feature 8, but in other embodiments, this can also be a decoded STFT representation, or an MDCT representation or an MDST representation or anyother spectral representation or an LPC representation, for example.There is complete freedom on how to use the output for the CFM model to solve the approximation problem for the final output. For example:^ Finite elements methods, e.g., Euler method, Runge-Kutta, dopri, …^ Approximation via neural networks and so-called neural-ODEs methodsThe ODE-Solver might use the vector field information provided by the ODE-Estimator atdifferent points ^ ∈ ^^ and not necessarily only at the initial condition 6b or the subsequentestimated solutions.Moreover, the sampling of the initial condition 6b can make use of different probabilitydistributions, e.g., Gaussian noise, uniform noise, a learned distribution related to the data oran auxiliary DNN-Model, e.g. the latent distribution of a VAE. At each step of the solving procedure this probability distribution can be sampled again to offer an updated estimate of the estimated solution. The ODE-Solver can also accept additional information as in the Classifier-Free Guidance(CFG) sampling method for DDPMs. More precisely, when using the CFG sampling methodan additional estimation of the vector field is obtained via applying the ODE-Estimator to witha value ^̃ of 6a unrelated to the conditioning latent ^. Typical values for ^̃ are zero, the meanof ^, a list of fixed values coming from a fixed codebook. This new estimation is used togetherwith the current estimation at ^ to “drift” the current solution into the general direction of thedata distribution. This drift is enforced at training time via dropping out ^ with a certainprobability so that the model both learns conditional and unconditional generation. The audio codec system is described in Figure 5. The system is composed of: 1. An Encoder model consisting ofa. an optional learned or fixed feature extractor 1 extracting an audio feature 2 thatcan be time-frequency transformation or a learned transformationb. a learned DNN-based model 3.2. A quantizer 4 that generates the bit stream.3. A Decoder model consisting ofa. An optional first learned block 5 that decodes the bit stream to a firstrepresentation 6a, b. A CFM-based decoder 7 which synthesizes the decoded audio feature, and / orc. An optional learned or fixed inverse transformation from feature space to audiospace 9.At synthesis time the CFM-decoder 7 takes as inputs the decoded latent 6a and additionalconditioning information 6b to synthesize the decoded audio-feature 8 step-by-step. Thisprocedure entails first generating an ODE using a neural network and the decoded latent 6a,evaluating this ODE at the initial condition 6b and solving it, e.g., via applying an ODE-Solveras described above. The learned neural network represents ^^ from above. The optionalmodules are specified with examples.The encoder module 1 and the corresponding decoder module 9 can represent:^ Time-frequency transformation: STFT (STFT = Short Time Fourier Transform) (e.g., fullcomplex spectrum, only magnitude spectrum or band limited magnitude spectrum), perceptually weighted STFT spectrum (e.g., mel, bark, ERB (Equivalent RectangularBandwidth) magnitude spectrograms), lapped transforms (e.g., MDCT / MSDT) (ModifiedDiscrete Cosine Transform / Modified Discrete Sine Transform),^ Time-domain subband transformation or PQMF (Pseudo Quadrature Mirror Filterbank),^ Wavelet transformations,^ The filterbanks from the previous points but sampled in a non-uniform way. E.g. thewindows of the transformation can have more frequency resolution for the lower bands and higher time resolution for the higher bands, and / or^ Learned transformation (e.g., acoustic feature from a learned codec [E1,E2,E3], semanticfeatures from the activations of a learned encoder).Notice that in this case 1 and 9 do not need to be inverse of each other. That is if the analysistransformation 1 is, e.g. an STFT, the synthesis transformation 9 need not be the iSTFT(inverse short time Fourier transform). In fact, both 1 and 9 could be learned transformations.There is the special case where the transformation 9 is a sort of “generalized vocoder”converting a compressed representation 8 into the waveform domain using a possiblygenerative neural network. An example of this could be a neural network based mel-to-audio model.Depending on whether the optional layers are present or not the CFM 7 can generate audiodirectly in time-domain or in one of the abovementioned representation domains.The DNN-encoder 3 and the corresponding optional DNN decoder 5 can be trained jointly andinclude CNNs (convolutional neural networks) ,RNNs (recurrent neural networks), Transformerlayers, FiLM Conditioning layers. Similarly, the CFM ODE estimator can be jointly trained andinclude CNNs, RNNs, Transformer layers, FiLM Conditioning layers, learned or fixedembeddings for the time-step as well as conditioning layers for the latent 6a. Theseconditioning layers can be also deactivated for unconditional generation or CFG.The Quantization 4a can entail:^ classical quantization techniques, like scalar quantization, vector quantization, residualvector quantization,^ lossless quantization techniques like Huffman coding, arithmetic coding, bits-back coding,and / or^ learned counterparts like VQ-VAE (vector quantization - variational auto encoder)codebooks, residual learned codebooks, learned scalar quantization and learned entropy coding, e.g. using Language Models.The conditioning latent 6a can be learned from a bit stream, text, midi, auxiliary contextualinformation, e.g., images / video, a decoded signal of lower quality. The extra conditioning 6b,which is usually used as initial condition in the Initial Value Problem, can be sampled from any known distribution, e.g. Gaussian noise, uniform noise, the uniform latent distribution of alearned VQ-VAE latent, the empirical distribution of 6a itself.The decoder of the system does not need to be learned together with the encoder. In fact, it can be trained to generate audio from the bit stream of a fixed legacy codec, e.g. MP3, USAC, OPUS, EVS, or a fixed learned encoder, e.g. NESC, DAC, SoundStream, EnCodec. To this end the CFM can act both as full decoder for the fixed encoder or as a post-filter in case a partor the full corresponding fixed decoder is applied. For example, the encoder-decoder pipelineof MP3 could be used to obtain the coded audio, and this could be given as conditioning to the CFM-model for generating an enhanced version directly in time-domain or in one of the abovementioned representation domains. Alternatively only part of the MP3 decoder could beused to generate a representation of the audio, e.g., in MDCT or PQMF domain and then theCFM-model can be used to enhance this representation of decoded data in the time domain.Fig. 5 illustrates a complete encoder / decoder system. However, it is to be noted that the encoder on the one hand and the decoder on the other hand can be implemented separate from each other. this will typically be the case, since the encoder is remote from the decoder and the encoder generates the bit stream 4 and forwards this bit stream to the decoder. In the Fig. 5 embodiment, the input audio data 20 are input into a feature extractor 1. Thefeature extractor 1 generates an audio feature 2. Then, the audio feature 2 is input into a DNNencoder 3 and a final step of the encoder is the quantization in block 4a. Then, a bit stream is either generated as the result of the quantization or, alternatively, a bit stream formatter is placed at the output of quantization block 4a. Preferably, the feature extractor is a time- spectrum converter and the audio feature 2 is a spectral representation of the audio signal such as a sequence of mel spectra or STFT spectra or MDCT spectra or MDC spectra. In a further embodiment, the feature extractor 10b in an LPC analysis filter and the audio feature is the LPC residual signal. Alternatively, the feature extractor can also comprise a processor for processing the LPC residual data such as for generating a sequence of spectral derived from the LPC residual signal. In a further embodiment, the DNN encoder can include the quantizer and the output of the DNN encoder / quantizer is already a complete bit stream. As illustrated in Fig.5, blocks feature extractor 1 and audio feature 2 are optional blocks and the input audio signal 20 can also be directly input into the DNN encoder 3. Furthermore, in a further embodiment, the encoder that encodes the input audio signal 20 to obtain the bit stream does not necessarily have to be any deep neural network (DNN) based encoder. Instead, a straightforward encoder such as an LLC encoder or a USAC encoder can be used as well to generate the bit stream 4. On the decoder-side, the bit stream is input, in the embodiment, into an inverse quantizer 4b and the subsequently connected DNN decoder 5 to generate the decoded draw spectrum or generally, the decoded latent is, for example, a dequantized feature as a general example for a raw spectrum. Additionally, the extra conditional such as Gaussian noise 6b is provided at the decoder for example via a noise table and is input into the conditional flow matching core 7 that generates the decoded audio feature 8 which is then input into an optional inverse feature extractor 9 to finally obtain the decoded (time domain) audio signal 40. As illustrated in Fig. 5, the bit stream can also be directly input into the conditional flow matching core without any preprocessing via an inverse quantizer 4b or a DNN decoder 5.Additionally, or alternatively, the CFM output can already be the decoded audio signal. In thisembodiment, block 9 is optional. Therefore, in such a general embodiment, the bit stream 4 is directly input into the CFM decoder as the decoded latent together with extra conditional data 6b to finally obtain the decoded audio signal 40. In preferred embodiments, however, decoded latent 6a is a spectral representation and the output of the CFM core is subjected to an inverse of the feature extraction such as a spectrum-time conversion or alternatively, an LPC synthesis filter for converting a decoded fine LPC residual signal into time domain audio signals using a correspondingly designed LPC synthesis filter. An embodiment of the audio codec system is described in Figure 6, which is a specific case of the system introduced before. The system is composed of ^An Encoder model consisting of a feature extractor 1 and a learned DNN-based model3, ^a Quantizer 4 that generates the bit stream, and / or^ a Decoder model consisting of a first learned block 5 that decodes the bit stream to alearned representation. This is given, together with a noise prior, to the CFM-based decoder 7, which synthesizes the decoded mel-spectrogram. Finally, a mel-to-audio model 9 is used to get the decoded audio.At synthesis time the CFM-decoder 7 takes as inputs the decoded latent 6a and additionalconditioning information 6b to synthesize the decoded mel-spectrogram 8 step-by-step. Thisprocedure entails first generating an ODE and then applying and ODE-Solver to solve it. Fig. 6 illustrates a preferred embodiment of the encoder of Fig. 5 and also a preferred embodiment of the decoder of Fig.5. Both, the encoder and the decoder constitute an audio processing system or a method of audio processing. In this preferred embodiment the feature extractor is a time-spectrum converter and the audio feature 2 is a spectrogram such as a mel spectrogram or an STFT spectrogram or any other spectrogram as discussed before. The converter for converting the spectrogram into a bit stream is a deep neural network based spectrum encoder such as an DNN mel encoder as illustrated at 3 in Fig.6. Correspondingly, the decoder comprises a DNN spectral decoder 5 to obtain the decoded latent 6a which is a decoded such as a mel spectral or STFT or MDCT, or MDST spectral representation which is input into the SFM decoder core 7 which outputs a decoded or fine spectral representation which is then input into a mel to audio “vocoder” which can also be inimplemented in deep neural network technology in order to obtain the decoded audio signal40.Subsequently, further embodiments are shown in the following Figure 7a and 7b. Fig. 7aillustrates a general representation of the CFM core having the first input 6a for receiving the decoded latent such as a quantized mel spectrum and having a second input for receiving extra conditional data such as Gaussian noise. At the output, the CFM core outputs exemplarily a mel spectrum representation of the audio signal or any other spectral representation. Fig 7b illustrates a preferred detailed representation of the CFM core 7 consisting of an ODE estimator 71 for calculating flow derivatives 74 at the current point, which are fed into an ODE solver 72 that finally outputs an estimated solution which is, in an embodiment, a decoded mel spectrum or any other spectral representation or LPC representation of the audio signal..An architecture of such an embodiment titled FlowMAC is illustrated in Fig.8a to 8c.A. Mel Encoder-Decoder (Fig. 8b)The 128 mel spectrogram bands are calculated on the input 24 kHz audio with hop size 512 and window of 2048 samples, hence yielding 47 frames per second. Mean and standard deviations are calculated offline for the whole dataset and used as fixed normalization factorsfor the input. The normalized mel spectrogram passes through a 1 × 1 convolutional layer with128 channels to extract features for the encoder. The encoder is a sequence of Multi-head Attention, Dropout, Layer normalization, feed-forward and dropout layers, producing a latentvector to be quantized. The number of repetitions is ^ = 6 and the dimension of the latentvector is 768. The decoder architecture follows the same structure as the encoder. Finally, a1 × 1 convolutional layer serves as a final projection layer to generate the decoded quantizedmel spectrogram. A combination of MSE and MAE losses denoted as ℒ^^^^^is used to push the network to generate a representation that is already close to the input mel spectrogram. As quantization technique a learned residual vector quantization based on Vector-Quantized VAE (VQ-VAE) [9] is used, with projections to small dimensional spaces similar to

[0011] . FlowMAC uses a codebook size of 256 and 8 quantizer levels and a downsampling dimension 16 for the 128-dimensional latent. Using 8 bits per level with 47 frames per second results in a total of 3 kbps.B. CFM Decoder (Fig.8c)The CFM decoder follows an architecture similar to

[0020] using a U-Net with residual 1D convolutional blocks and transformer blocks with snakebeta activations

[0022] . Finally, the output of the U-Net passes through a 1D Block consisting of a 1D convolution, group normalizationand a Mish activation

[0023] , after which a 1 × 1 convolutional layer creates the final output. Thecorresponding time-step embeddings were generated the same way as in

[0024] . The CFM decoder is conditioned on the decoded quantized mel spectrogram and takes Gaussian noise as input to estimate the corresponding vector field. As objective the MSE between the estimated and actual vector field was used. The training objective for the whole system is then =0.01 and ^^ = 0.25 denote weighting factors for the prior and VQ-VAE loss tobalance its influence.C. Mel-to-Audio Module shown at 9a in Fig. 8aAs mel-to-audio module, a smaller version of BigVGAN

[0022] is re-trained on available data. The mel spectrogram calculation is adapted to fit the setting described in Section III-A. Then, the decoder initial channels are decreased to 1024 and an additional upsampling layer is used. This yields a smaller architecture than the original BigVGAN, whose complexity is comparable with the DAC Decoder. Notice that the dependence of an embodiment of the system on this mel-to-audio module for the final audio synthesis leads to a highest achievable quality dictated by BigVGAN’s performance. This is saturated by a preferred mel spectrogram codec and subjective evaluations confirm this phenomenon. Fig.8a illustrates a further implementation of an audio processing system that consists of an encoder-side represented by blocks 2a, 101 to generate the bit stream 4 and that comprises a decoder-side consisting of blocks 201, 7, 8a, 9a in order to finally obtain the decoded audio signal 40. In the Fig. 8a embodiment, the input into a spectral encoder stage 101 is a mel spectrum representation of the audio signal or, generally, a time-frequency representation of the audio signal. The bit stream 4 is input into the bit stream-spectrum decoder 201 that corresponds to block 5 of, for example, Fig.5 or Fig.6. Correspondingly, block 101 of Fig.1a corresponds to block 1 of Fig.5 or Fig.6. Similarly, block 201 of Fig.1a corresponds to the combination of blocks 4b, 5 of Fig.5 or block 5 of Fig.6. Furthermore, block 9a corresponds to block 9 of Fig.5 or Fig.6. The left side of Fig. 8b illustrates an implementation of block 101 in an embodiment. Correspondingly, the right side of Fig. 8b illustrates a representation of the corresponding decoder stage 201 of Fig.8. It is to be noted that the encoder side or left side of Fig.8b can be implemented without the right side and vice versa. Therefore, blocks 101, 201 of Fig.8a or Fig.8b can be implemented independent from each other and in separate devices.Fig. 8c illustrates a preferred implementation of the neural network within the CFM core 7 ofFig. 8a and can also be implemented independent from the implementation of block 101 or 201 of Fig.8a or 8b. The Fig.8b implementation of the mel-encoder 101 comprises a 1 x 1 convolutional layer 102 with preferably 128 channels. The encoder comprises a multi-head attention functionality 103,a dropout and layer normalization functionality 104, a feed forward layer preferably togetherwith normalization 105and a dropout and layer normalization functionality 106 which produces a latent vector to be quantized. This quantization is done in the block 107. The decoder architecture 201 comprises a dequantization functionality 202 followed by a multi- head attention layer 203, a dropout and layer normalization layer 205, a feed forward layer preferably together with normalization functionality 205 and a dropout and layer normalization functionality 206 in order to obtain an output which is finally forwarded to the 1 x 1 convolutional layer 207 to obtain the latent data to be input into the conditional flow matching core 7. As is illustrated in Fig.8b, the blocks 103 to 106 for the encoder-side and the blocks 203 to 206 for the decoder-side are implemented N times. As illustrated, the number N illustrates the number of repetitions and is preferably between 4and 10 and in the described example equal to 6.Fig. 8c illustrates a preferred implementation of the neural network in the conditional flow matching core 7 that receives, at the first input, the decoded quantized mel spectral representation. Additionally, the decoder receives the conditional data in the form of Gaussian noise. The preferred embodiment, the CFM core comprises an input concatenator 701 followed by an input side pair of a residual one dimensional convolutional block 702 and a transformerblock 703 and at least one output side pair of a transformer block 714 and a residual onedimensional convolutional block 713, wherein the transformer block 703 of the first input side pair is configured to feed the residual one dimensional convolutional block 713 of the outputside pair, and wherein the one dimensional convolutional blocks 702, 713 of the input side pairand the output side pair are controlled by a time step embedding controller 708. As illustrated in Fig.8c, the neural network implemented in the CFM block 7 not only comprises a single input side pair and a single output side pair. Instead, three input side pairs and corresponding three output side pairs are illustrated. The second input side pair consists of the residual one dimensional convolutional block 704 of the input side and 711 of the output side. The third input side pair and output side pair comprise the residual one-dimensional convolutional blocks 706 for the input side and the 709 for the output side. As illustrated, the corresponding transformer block 705 or 707 of the input side is connected to the corresponding residual one-dimensional convolutional block 711 or 709 of the output side. And all residual one-dimensional convolutional blocks are controlled by the time imbedding controller 708. On the input-side, the connection of the transformer block is so that the transformer block of a lower input pair such as 703 feeds the residual one-dimensional convolutional block of the higher pair such as block 704. On the output-side, the situation is so that the corresponding transformer block 710, 712, 714 feeds the corresponding residual one-dimensional convolutional block of the subsequent output side pair. Hence, block 710 feeds block 711, block 712 feeds block 713 and block 714 feeds the output pair 715 / 716 to finally obtain the decoded fine spectral representation illustrated at 8 in Fig.8c. Hence, block 701 in Fig.8c represents an input layer of the neural network, and block 715, 716 represent an output layer of the neural network implemented in block 7. With respect to Fig.8c on the one hand and Fig.7b on the other hand, it is to be noted that the functionality of the ODE estimator 71 and particularly the neural network based parameterestimation of the system of differential equations is implemented by the cooperation of acorresponding input side pair and output side pair such as by the cooperation of block 706, 707 on the one hand and 709, 710 on the other hand. Subsequently, further preferred implementations and preferred parameters are outlined. The first input data derived from the encoded audio signal at the first input 6a comprises spectral data for one frame or for more than one frames preferably for a number of framesbelow 20 frames. The second input data at the second input comprise pseudo noise data,noise data, colored noise data or Gaussian noise data, and the frame is preferably a time frame or represents a group of consecutive time-domain samples or represents a group of LPC domain samples or covers a certain time span of the original audio signal or has a group of spectral values representing a certain time span of the original audio signal or a certain group of consecutive time domain samples.In an embodiment, the ordinary differential equations defines vector field in a space, and thelearned ordinary differential equation estimator provides, as the parameters for the ordinary differential equations, the information on a flow direction or flow magnitude as illustrated in Fig. 14c, wherein the information depends on trained parametric data of a neural network included in the learned ordinary differential equation estimator 71, a point in the space determined by the first input data 6a derived from the decoded audio signal and second input data 6b representing the conditional data. The learned neural network is configured to output, in response to the input, flow direction data of an ordinary differential equation system at a point in a vector space determined by the first input data derived from the encoded audio signal and the second input data, wherein the second input data are similar to corresponding data used as conditional data for determining the learned neural network in a training phase. The encoded audio signal is preferably organized in a sequence of frames, wherein the sequence of frames comprises at least two time sequential frames, where each frame of the at least two sequential frames has the same length in time, and wherein the frame is a time frame or represent a group of consecutive time-domain samples, or represents a group of LPC domain samples of covers a certain time span of the original audio signal or has a group of spectral values representing a certain time span of the original audio signal or a certain group of consecutive time domain samples. The sequence of frames preferably comprises at least two time sequential frames and even more preferably between 10 and 100 frames, wherein each frame of the at least two sequential frames has a length in time being greater than one millisecond and lower than 200 milliseconds and preferably between 10 millisecond and 50 milliseconds. The learned conditional flow matching based processing stage 70 is configured to generate a sequence of frames of a raw decoded audio signal and is configured to combine two timely adjacent frames of the raw decoded audio signal to obtain at least one frame of the decoded audio signal, wherein the two timely adjacent frames of the raw decoded signal correspond to two timely adjacent frames of the sequence of frames of the encoded audio signal. Similarly, when the sequence of frames of the encoded audio signal comprises spectral data derived from a short time spectral representation of a sequence of short time spectra of the original audio signal corresponding to timely overlapping frames of the original audio signal is occurring in an FFT representation and, particularly, in an MDCT or an MDST representation, the learned conditional flow matching based processing stage 70 combines the two timely adjacent frames of the raw decoded audio signal using an overlap-add operation in calculating sampling values of the decoded audio signal. The overlap-add operation that typically relies on synthesis windows may be implemented within the CFM core, but is preferably implemented within the spectrum-time converter connected at the output of the CFM core to finally obtain the decoded audio signal. A frame preferably comprises between 70 and 140 mel spectrogram bands derived from an original input audio signal having a sampling rate between 16kHZ and 48kHZ with a hop size between 128 and 1024 samples and a window having a length between 124 and 4096 samples. Subsequently an evaluation of the architecture is given.A. Experimental setupBoth FlowMAC and BigVGAN are trained on a combination of the full LibriTTS

[0025] clean and dev train subset as in

[0022] and an internal music database consisting of 640 hours of high- quality music of various genres. The sampling rate for all datapoints was 24 kHz. BigVGAN was trained following the official implementation for 1M iterations on a single A100GPU. FlowMAC was trained with the Adam optimizer with learning rate 1^ − 4, a segmentlength of 2 s and batch size of 128 on a single RTX3080. To improve the CFM training, thetimestep ^ is sampled according to a logit normal distribution

[0026] for each mini-batch.Moreover, a preferred model is trained with a classifier-free guidance (CFG) technique

[0027] ,where the decoded mel spectrogram condition is set to zero with a probability of ^^ = 0.2.This greatly enhances the quality of the signal at inference time when using CFG.B. Subjective EvaluationTo evaluate the proposed system, a MUSHRA

[0028] listening test was performed with expert listeners and a P.808 DCR

[0029] listening test was performed with naive listeners. To this end a test set of 12 carefully selected test items including 4 challenging clean and noisy speech samples (male, female, child and speech over music), 5 music items of various genres and 3 critical items (castanets, harpsichord and glockenspiel) were designed. It was given careful consideration that the test set is disjoint with the train set.For a P.808 DCR FlowMAC was compared to state-of- the-art DNN-based audio codecs anda well-known legacy codec. The GAN-based audio codecs DAC

[0011] and EnCodec

[0010] , and the DDPM-baseline MBD

[0017] were selected. The official implementations and trained weights for all those models were used. It was noticed that the comparison is not fully balanced as training sets vary strongly between the conditions. Still, it is considered useful to compare an embodiment of the system with well-established and robust codecs.As a measure of the highest achievable quality with Flow- MAC the copy-synthesis of thesignals via BigV-GAN was included. As a benchmark legacy-condition an internal implementation of the MPEG-D USAC Standard was used. This works on full-band audio, but the decoded signal was downsampled to 24 kHz to more closely measure the differences in the codecs at this sample rate, although this essentially leads to a suboptimal result for USAC. As a lower anchor a low-pass filter with cutoff frequency of 3.5 kHz was used.It was noticed that the absolute ratings and categories, e.g., ”excellent” and ”good”, are notcomparable among different tests. The P.808 DCR gives a good overall idea of the different average quality of the conditions in the test. The MUSHRA test provides more finely measures and comparison between a subset of the conditions. Therefore, the most promising conditions around 6 kbps, namely FlowMAC at 3 kbps, DAC at 6 kbps, MBD at 6 kbps and USAC at 8 kbps were selected.C. FlowMAC variantsIn the P.808 DCR various version of FlowMAC are included to test quality and complexity trade- offs. The table in Fig.9 summarizes the complexity figures for the most relevant models. The real time factor (RTF) is the ratio between the inference and the input duration measured on a CPU. Thanks to the residual vector quantizer bit rate scalability was achieved via dropping out codebook levels at inference time. Moreover, the iterative nature of the Euler method used for the inference enables some freedom on the Number of Function Evaluations (NFE) for the CFM decoder. A preferred baseline works at 1.5 and 3.0 kbps, uses 32 steps for the ODE solver and factor 1 for the classifier-free guidance (CFG), hence leading to a total of 64 NFE. In order to show the fast saturation of the highest possible quality achieved by BigVGAN a 6.0kbps version was separately trained, for which the same NFE was used. This is referred to asFlowMAC-CQ. Finally, the quality-complexity trade-off was tested via using a single step for the Euler method and no CFG, hence obtaining FlowMAC-LC and using 1 NFE. Informal listening showed that using significantly more than 64 NFE did not bring significant improvement in quality. Careful attention needs to be placed on the choice of CFG factor: values around 0 usually lead to noisy signals (except if the number of steps is smaller than 2)and values bigger that 2 tend to overestimate the energy of the signal and introduce unwanted artifacts. A comprehensive study of the finer effects of the ODE-Solver, the number of steps and the classifier-guidance hyperparameters is out of the scope of this publication. Fig.10 illustrates the results of the P.808 DCR listening test. The measurements from the naive listeners confirm that both FlowMAC and FlowMAC-LC are the best models at 3 kbps, being on average on par with EnCodec, MBD and DAC at 6 kbps. The copy-synthesis with BigVGAN offers the highest achievable quality for a preferred system, and FlowMAC-CQ quality is not significantly better than FlowMAC’s. FlowMAC-LC suffers from the low-complexity scenario, but the test confirms that it is a competitive baseline. For naive listeners the higher frequency resolution of DAC 44.1 kHz at 8 kbps offers only a slight advantage over the 24 kHz model.Overall, it was noticed that the DNN conditions achieve comparable quality with the legacyUSAC condition at similar bit rates, the only exceptions being FlowMAC at 3 kbps. The results of the MUSHRA test are illustrated in Fig.9. They refine measurements from the P.808 and closely compare conditions which achieve comparable quality at 6 and 3 kbps. WhileUSAC 8kbps clearly has a quality advantage here, this test confirms that FlowMAC at 3 kbpsperforms on par with DAC 6 kbps and both conditions outperform MBD 6 kbps. The copy- synthesis from BigVGAN does not perform significantly better than USAC 8 kbps, hence confirming that this module is a bottleneck for the ceiling quality of FlowMAC.FlowMAC represents an embodiment of the invention being a low bit rate neural audio codecfor wideband audio. This is achieved by constructing an efficient encoder-decoder structure to encode, quantize and decode mel spectrograms. The output then serves as input to a conditional flow matching model that generates realistic mel-spectrograms to be converted to waveform audio. Subjective evaluation demonstrated that FlowMAC outperform strong baselines at lower bitrates and is able to compress and generate high quality audio samples. In the future, it is intended to experiment with different input representations and causal architecture variants. Subsequently, further preferred implementations and embodiments of the present invention are provided. Particularly, the DNN encoder module 3 can be one of the following. One implementation is a transformer-based architecture over the reordered or reshaped output of the feature extractor 1. If the feature extractor 1 of Fig. 5 for example is an STFT procedure, then the transformer processes the time-frequency bins following the frequency dimension first. Such a transformer architecture preferable uses causal attention for low delay, for example in the case of an STFT feature, one can work with zero added delay. Furthermore, the technology with causal attention can be used on a shifted time-frequency signal to achieve look ahead, for example if the Short Time Fourier Transform (STFT) has 10 millisecond frames, one can have e.g. seven frames, look-ahead in order to, for example, implement a certain transient processing or in order to enhance the functionality of the neural network that not only refers to past data but also tofuture data from the look-ahead period.Attention mechanisms are preferably windowed to reduce the computational needs. Thewindowing is done to, for example, a temporal context of 0.5, 1, 2 or 5 seconds or in otherwords, to 50, 100, 20 or 500 STFT frames compared to the full duration of the audio signal.The transformer-based architecture can be implemented as described in A. Vaswani et al.,“Attention is All You Need”, in arXiv, 2017, https: / / arxiv.org / abs / 1076.03762. The DNN-encoder module can also use a learned model containing the DPCRNN layer from the publication from N. Pia, et al., “NESC: Robust Neural End-2-End Speech Coding with GANs” in arXiv, 2022, https: / / arxiv.org / abs / 2207.03282, for example in combination with a residual CNN model or a transformer-based model.Furthermore, the DNN coder module 3 can also be implemented as a fully strided CNN-basedmodel with total downsampling in time domain of a factor 320, 640, 960.The DNN-decoder module 5 can be a transformer-based architecture as described before, or a CNN-based architecture based on transposed convolution or can comprise a sequence oflearned layers that transform the one-dimensional bit stream to a tensor of three-dimension,similar to a complex time-frequency signal. More precisely, the sequence of quantized frames q = [q0… qt] is transformed into a three-dimensional sequence containing c channels (similar to real and imaginary parts), f learned frequency bins and t learned time frames. The sequence of learned layers preferably contains a series of CNN and transformer layers as illustrated in Fig.11.Fig.11 illustrates the bit stream 4 having a single dimension only which is the time dimension.The bit stream is introduced into a CNN block 5a for performing a feature expansion into twodimensions, where one dimension is h and the second dimension is the time dimension t. Thistwo-dimensional representation is input into the transformer block 5b outputting a two-dimensional representation. This output-side two-dimensional representation is then input intoa further CNN layer 5c performing a channel expansion to a three-dimensional representation having the c channels, the f frequency bins and the t time frames. In a preferred implementation, the CFM decoder core 7 is based on diffusion transformers as illustrated in the publication authored by W. Peebles and Saining Xie, “Scalable Diffusion Models with Transformers”, in arXiv, 2022, https: / / arxiv.org / abs / 2212.09748.Furthermore, the CFM decoder core 7 can preferably contain the TADE or FiLM layers fromthe above cited publication “NESC: Robust Neural End-2-End Speech Coding with GANs”. Furthermore, the CFM decoder preferably comprises a learned transposed convolution with orwithout overlapping between the patches.Fig. 15a illustrates a specific procedure for generating patches and Fig. 15b illustrates thecorresponding procedure for unpatching. The patching and unpatching are parts of a diffusiontransformer architecture. Fig. 16 illustrates a first version using the conditional as styling of a diffusion transformer implementation of the neural network in the CFM core, and particularly in the ODE estimator.The diffusion transformer comprises at least one Diffusion Transformer Block (DiT Block). Inthe preferred case of having more than one Diffusion Transformer Blocks (DiT Block) such asN blocks, the Diffusion Transformer is the concatenation of all the N Diffusion TransformerBlocks. Fig. 17 illustrates a second version using the conditional as prior of a diffusion transformerimplementation of the neural network in the CFM core, and particularly in the ODE estimator.The diffusion transformer comprises at least one Diffusion Transformer Block (DiT Block). Inthe preferred case of having more than one Diffusion Transformer Blocks (DiT Block) such asN blocks, the Diffusion Transformer is the concatenation of all the N Diffusion Transformer Blocks.With respect to Figs.16 and 17 on the one hand and Fig.7b on the other hand, it is to be notedthat the functionality of the ODE estimator 71 and particularly the neural network basedparameter estimation of the system of differential equations is implemented by functionalitiesillustrated in these figures based on a diffusion transformer implementation. Such diffusion transformer based implementations can be used instead of the Fig. 8c implementation of the neural network in the CFM core for generating the parameters for the system of differential equations.The inverse of the feature extraction illustrated in block 9 of Fig. 5 or Fig.6 can preferably bebased on the iSTFT optionally containing the usage of a decompression factor α that scalesthe time-frequency x representation via the formula β is a positive number, and │∙ │ denotes the magnitude of x and Φ(∙) its phase. The factors αand β can also be dependent on the time-frequency bin under consideration or can also belearned from data. In the latter case, the information is transmitted from the encoder to the decoder. In the case of only a frequency-dependency, then the factors α and β can be provided via for example a table known to the decoder so that this data does not have to be transmitted from the encoder to the decoder.Furthermore, block 9 can also rely on an inverse mu-law compression of the signal or can relyon an STFT framing implemented with 10, 20 or 40 milliseconds. It is to be mentioned here that all alternatives or aspects as discussed before and all aspects as defined by independent claims in the following claims can be used individually, i.e., without any other alternative or object than the contemplated alternative, object or independent claim. However, in other embodiments, two or more of the alternatives or the aspects or the independent claims can be combined with each other and, in other embodiments, all aspects, or alternatives and all independent claims can be combined to each other. An inventively encoded signal can be stored on a digital storage medium or a non-transitory storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet. Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Depending on certain implementation requirements, embodiments of the invention can beimplemented in hardware or in software. The implementation can be performed using a digitalstorage medium, for example a floppy disk, a DVD, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed. Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier. Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier or a non-transitory storage medium. In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer. A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computerprogram for performing one of the methods described herein.A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet. A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein. A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.In some embodiments, a programmable logic device (for example a field programmable gatearray) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus. The above described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.

[0002] References [E1] N. Zeghidour, et al. “SoundStream: An End-to-End Neural Audio Codec”, In arXiv, 2021 https: / / arxiv.org / abs / 2107.03312 [E2] A. Défossez, et al., “High Fidelity Neural Audio Compression”, In arXiv 2022 (https: / / arxiv.org / abs / 2210.13438) [E3] R. Kumar, et al, “”, In arXiv, 2023 (https: / / arxiv.org / pdf / 2306.06546)[E4] R.S. Roman, et al., "From Discrete Tokens to High-Fidelity Audio Using Multi-BandDiffusion Robin", In Workshop NIPS, 2023, (https: / / arxiv.org / pdf / 2308.02560). [E5] H. Yang, et al., "Generative de-quantization for neural speech codec via latent diffusion", In ICASSP, 2023, (https: / / arxiv.org / pdf / 2311.08330). [E6] Y.C. Wu, et al., "ScoreDec: a phase-preserving high-fidelity audio codec with a generalized score-based diffusion post-filter", In arXiv, 2024, [E7] H. Liu, et al., "SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound", In arXiv, 2024, (https: / / arxiv.org / pdf / 2405.00233). [E8] Y. Lipman, et al., “Flow Matching for Generative Modeling”, In arXiv, 2023 [E9] A. Sauer, et al., “Fast High-Resolution Image Synthesis with Latent Adversarial DiffusionDistillation”, In arXiv, 2024, (https: / / arxiv.org / pdf / 2403.12015 )[E10] Kim et al. “P-Flow: A Fast and Data-Efficient Zero-Shot TTS through Speech Prompting”, In NeurIPS, 2023 [E11] S. Metha, et al., Matcha-TTS: A fast TTS architecture with conditional flow matching, In ICASSP, 2024 (https: / / arxiv.org / abs / 2309.03199)[E12] S. E. Eskimez, et al., “E2TTS: Embarrassingly Easy fully non-autoregressive zero-shotTTS”, In arXiv 2024 (https: / / arxiv.org / pdf / 2406.18009)[E13] Liu et al., “RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction”, In arXiv 2024 (https: / / arxiv.org / pdf / 2403.05010) [E14] A. Vyas, et al., "AudioBox: Unified Audio Generation with Natural Language Prompts", In arXiv, 2023, (https: / / arxiv.org / pdf / 2312.15821). [E15] W. Guan, et al., "LAFMA: A Latent Flow Matching Model for Text-to-Audio Generation", In arXiv, 2024, (https: / / arxiv.org / pdf / 2406.08203). [E16] A. H. Liu, et al., "Generative pre-training for speech with flow matching", In ICLR, 2024, (https: / / arxiv.org / pdf / 2310.16338). [E17] K. R. Prajwal, et al., "MusicFlow: Cascaded Flow Matching for Text Guided Music Generation", In ICLR, 2024, (https: / / openreview.net / pdf?id=kOczKjmYum). [E18] J. Song, et al. “Denoising Diffusion Implicit Models”, In ICLR, 2021, (https: / / arxiv.org / pdf / 2010.02502) Additional References[1] S. Morishima, H. Harashima, and Y. Katayama, “Speech coding based on a multi-layerneural network,” in IEEE International Conference on Communications, Including Supercomm Technical Sessions. IEEE, 1990, pp.429–433.[2] S. Kankanahalli, “End-to-end optimized speech coding with deep neural networks,” in2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018.[3] K. Zhen, M. S. Lee, JZ Sung, S. Beack, and M. Kim, “Efficient and scalable neuralresidual waveform coding with collaborative quantization,” 2020. [Online]. Available: https: / / arxiv.org / abs / 2002.05604[4] W. B. Kleijn, F. S. C. Lim, A. Luebs, J. Skoglund, F. Stimberg, Q. Wang, and T. C.Walters, “WaveNet Based Low Rate Speech Coding,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp.676–680.[5] J. Valin and J. Skoglund, “A Real-Time Wideband Neural Vocoder at 1.6 kb / s UsingLPCNet,” in INTERSPEECH 2019, 20th Annual Conference of the International Speech Communication Association, 2019, pp.3406–3410.[6] A. Mustafa, J. Büthe, S. Korse, K. Gupta, G. Fuchs, and N. Pia, “A Streamwise GanVocoder for Wideband Speech Coding at Very Low Bit Rate,” in 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2021, pp.66- 70.[7] K. Zhen, J. Sung, M. Lee, S. Beack, and M. Kim, “Cascaded Cross- Module ResidualLearning Towards Lightweight End-to-End Speech Coding,” Proc. Interspeech 2019, 2019.[8] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream:An end-to-end neural audio codec,” IEEE / ACM Trans. Audio, Speech and Lang. Proc., vol.30, pp.495-507, 2021. [Online]. Available: https: / / doi.org / 10.1109 / TASLP.2021.3129994[9] A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural Discrete RepresentationLearning,” in Advances in Neural Information Processing Systems, vol.30, 2017.

[0010] A. Defossez, J. Copet, G. Synnaeve, and Y. Adi, “High fidelity neural audiocompression,” Transactions on Machine Learning Research, 2023.

[0011] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High- Fidelity AudioCompression with Improved RVQGAN,” in Advances in Neural Information ProcessingSystems, vol. 36, 2023, pp.27980– 27993.

[0012] L. Ziyin, T. Hartwig, and M. Ueda, “Neural Networks Fail to Learn Periodic Functionsand How to Fix It,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. CurranAssociates, Inc., 2020, pp.1583–1594.

[0013] X. Jiang, X. Peng, C. Zheng, H. Xue, Y. Zhang, and Y. Lu, “End-to-end neural speechcoding for real-time communications,” in 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.

[0014] N. Pia, K. Gupta, S. Korse, M. Multrus, and G. Fuchs, “NESC: Robust Neural End-2-End Speech Coding with GANs,” in Proc. Interspeech 2022, 2022, pp.4212–4216.

[0015] Z. Du, S. Zhang, K. Hu, and S. Zheng, “FunCodec: A Fundamental, Reproducible andIntegrable Open-Source Toolkit for Neural Speech Codec,” in IEEE Intern. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp.591–595.

[0016] H. Yang, I. Jang, and M. Kim, “Generative de-quantization for neural speech codec vialatent diffusion,” in ICASSP 2024 - 2024 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP), 2024, pp.1251–1255.

[0017] R. San Roman, Y. Adi, A. Deleforge, R. Serizel, G. Synnaeve, andA. Defossez, “From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion,”in Advances in Neural Information Processing Systems, vol.36, 2023, pp.1526–1538.

[0018] H. Liu, X. Xu, Y. Yuan, M. Wu, W. Wang, and M. D. Plumbley, “SemantiCodec: An UltraLow Bitrate Semantic Audio Codec for General Sound,” 2024. [Online]. Available: https: / / arxiv.org / abs / 2405.00233

[0019] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow Matching forGenerative Modeling,” in International Conference on Learning Representations (ICLR), 2023.

[0020] S. Mehta, R. Tu, J. Beskow, E. Szekely, and G. E. Henter, “Matchatts: A fast-ttsarchitecture with conditional flow matching,” in IEEE Intern. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp.11341–11345.

[0021] S. Kim, K. Shih, R. Badlani, J. F. Santos, E. Bakhturina, M. Desta, R. Valle, S. Yoon,and B. Catanzaro, “P-Flow: A Fast and Data-Efficient Zero-Shot TTS through Speech Prompting,” in Advances in Neural Information Processing Systems, 2023, pp.74213–74228.

[0022] S. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon, “BigVGAN: A UniversalNeural Vocoder with Large-Scale Training,” in International Conference on Learning Representations (ICLR), 2023.

[0023] D. Misra, “Mish: A self regularized non-monotonic activation function,” 2020. [Online].Available: https: / / arxiv.org / abs / 1908.08681

[0024] V. Popov, I. Vovk, V. Gogoryan, T. Sadekova, and M. Kudinov, “Grad-tts: A diffusionprobabilistic model for text-to-speech,” in Proceedings of the 38th InternationalConference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol.139. PMLR, 18–24 Jul 2021, pp.8599–8608. [Online]. Available: https: / / proceedings.mlr.press / v139 / popov21a.html

[0025] H. Zen, V. Dang, R. Clark, Y. Zhang, R. Weiss, Y. Jia, Z. Chen, and Y. Wu, “LibriTTS:A corpus derived from LibriSpeech for text- to-speech,” 2019, arXiv preprintarXiv:1904.02882.

[0026] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Mueller, H. Saini, Y. Levi, D. Lorenz,A. Sauer, F. Boesel, D. Podell, T. Dockhorn, Z. English, K. Lacey, A. Goodwin, Y.Marek, and R. Rombach, “Scaling rectified flow transformers for high-resolution image synthesis,” 2024. [Online]. Available: https: / / arxiv.org / abs / 2403.03206.

[0027] J. Ho and T. Salimans, “Classifier-free diffusion guidance,” in NeurIPS 2021 Workshopon Deep Generative Models and Downstream Applications, 2021.

[0028] International Telecommunication Union, “Recommendation ITU–R BS.1534-3 Methodfor the subjective assessment of intermediate quality level of audio systems,” 2015.

[0029] ——, “Recommendation ITU–T P.808 Subjective evaluation of speech quality with acrowdsourcing approach,” 2021.

Claims

Claims1. Audio decoder for decoding an encoded audio signal (4) to obtain a decoded audiosignal (40), comprising: an input interface (50) for receiving the encoded audio signal (4); and alearned conditional flow matching based processing stage (70) for transforming theencoded audio signal (4) into the decoded audio signal (40).

2. Audio decoder of claim 1, wherein the input interface (50) is configured to receive, asthe encoded audio signal (4), a bit stream, andwherein the learned conditional flow matching based processing stage (70) isconfigured to transform the bit stream into a time domain waveform as the decodedaudio signal (40).

3. Audio decoder of claim 1 or 2, wherein the learned conditional flow matching basedprocessing stage (70) comprises a learned conditional flow matching core (7)comprising: afirst input (6a) for first input data derived from the encoded audio signal (4); anda second input (6b) for second input data representing conditional data; andan output (7a) for decoded audio data.

4. Audio decoder of one of the preceding claims, wherein the learned conditional flowmatching based processing stage (70) comprises:a spectral decoder (4b, 5) to derive raw or distorted or coarsely quantized spectral data(6a) from the encoded audio signal (4); alearned conditional flow matching core (7) for transforming the raw spectral data (6a)into fine or restored or finely quantized spectral data (8); anda spectrum-time converter (9) for converting the fine spectral data into a time domainrepresentation representing the decoded audio signal (40).

5. Audio decoder of claim 4, wherein the raw spectral data (6a) are raw or distorted orcoarsely quantized mel spectral data, wherein the fine spectral data (8) are fine or restored or finely quantized mel spectral data, and wherein the spectrum-time converter (9) is configured to use a pre-trained neural vocoder to generated a time domainwaveform as the decoded audio signal (40).

6. Audio decoder of one of the preceding claims, wherein the learned conditional flowmatching based processing stage (70) comprises:a learned ordinary differential equation estimator (71) for estimating parameters for anordinary differential equations system in response to the encoded audio signal (4) or asignal (6a) derived from the encoded audio signal (4); and an ordinary differential equation solver (72) for solving the ordinary differentialequations determined by the parameters to obtain result data corresponding to the decoded audio signal (40), or from which the decoded audio signal (40) is derived.

7. Audio decoder of claim 6, wherein the learned ordinary differential equation estimator(71) and the ordinary differential equation solver (72) are configured to operateiteratively in at least two steps.

8. Audio decoder of claim 7,wherein, in a first step of the at least two steps, the learned ordinary differential equation estimator (71) is configured to provide,in response to an initial condition (73) and in response to the encoded audiosignal (4) or the signal (6a) derived from the encoded audio signal (4), first flow derivatives (74) at the initial condition (73), andthe ordinary differential equation solver (72) is configured to provide a firstestimated solution (73a) in response to the initial condition (73) and the first flow derivatives (74) at the initial condition (73), andwherein, in the second step of the at least two steps, the learned ordinary differential equation estimator (71) is configured to provide,in response to the first estimated solution (73a) and in response to the encodedaudio signal (4) or the signal (6a) derived from the encoded audio signal (4) second flow derivatives (74a) at the first estimated solution (73a), andthe ordinary differential equation solver (72) is configured to provide a secondestimated solution (73b) in response to the first estimated solution (73a) and thesecond flow derivatives (74a) at the first estimated solution (73a).

9. Audio decoder of one of claims 4 to 8, wherein the spectral decoder (4b, 5) is configuredto convert a frame of the encoded audio signal (4) into a spectrum having the rawspectral data, the spectrum having the raw spectral data representing the signal derived from the encoded audio signal (4), wherein the spectral decoder (4b, 5) is an entropydecoder, or a decoder in an entropy coding system, or a learned deep neural networkdecoder (5) for transforming a bit stream portion into a spectrum being an MDCTspectrum, an MDST spectrum, an STFT spectrum or a mel spectrum, wherein the frame is a time frame, or represents a group of consecutive time-domainsamples, or represents a group of LPC domain samples, or covers a certain time spanof the original audio signal, or has a group of spectral values representing a certaintime span of the original audio signal or a certain group of consecutive time domainsamples, orwherein the spectrum-time converter (9) is configured to transform an MDCT spectrum,an MDST spectrum, or an STFT spectrum, or a mel spectrum received from the learned conditional flow matching core as the fine spectral data into the decoded audio signal (40).

10. Audio decoder of claim 3, wherein the first input data derived from the encoded audiosignal (4) at the first input (6a) comprise spectral data for one frame or for more thanone frames, preferably for a number of frames below 20 frames, or wherein the second input data at the second input (6b) comprise pseudo noise data,noise data, colored noise data or Gaussian noise data, or data from a sampleable probability distribution, or samples from a probability distribution, wherein the frame is a time frame, or represents a group of consecutive time-domain samples, or represents a group of LPC domain samples, or covers a certain time span of the original audio signal, or has a group of spectral values representing a certain time span of the original audio signal or a certain group of consecutive time domain samples.

11. Audio decoder of one of claims 6 to 8, wherein the ordinary differential equations definea vector field in a space, and wherein the learned ordinary differential equation estimator (71) is configured to provide, as the parameters for the ordinary differentialequations, information on a flow direction or flow magnitude, wherein the informationon a flow direction or flow magnitude depends on trained parametric data of a neural network (730) included in the learned ordinary differential equation estimator (71), apoint in the space determined by first input data derived from the encoded audio signal (4), and second input data representing conditional data.

12. Audio decoder of claim 11, wherein the learned ordinary differential equation estimator(71) is configured to provide, as a result, the information on the flow direction or theflow magnitude represented by the following equation: ^^(^) = ^(^, ^, ^, ^),wherein ^^(^) is the information on the flow direction or the flow magnitude, ^ is a pointin the space ^^given by the second input data representing the conditional data, ^ indicates the trained parametric data, trained parameters or trained weights of theneural network, ^ is given by the first input data derived from the encoded audio signal(4), and ^ is a current time, orwherein ^^(^) represents the output values of the neural network when the inputs arex, c and t.

13. Audio decoder of claim 11 or 12, wherein the learned ordinary differential equationestimator (71) is configured to receive, as the first input data derived from the encodedaudio signal (4), one or more frame data of the encoded audio signal (4) or derivedfrom the encoded audio signal (4) and, as the second input data, the first input data, ornoise data, or Gaussian noise data or pseudo noise data or colored noise data or white noise data.

14. Audio decoder of one of claims 6 to 8, 11 - 13, wherein the ordinary differential equationsolver (72) is configured to integrate the ordinary differential equations system definedby the parameters for the ordinary differential equations provided by the learned ordinary differential equation estimator (71) and using initial data given by second inputdata representing conditional data.

15. Audio decoder of claim 14, wherein the ordinary differential equation solver (72) isconfigured to integrate the ordinary differential equations system using a finite elementmethod, an Euler method, a Runge-Kutta method, or a dopri method.

16. Audio decoder of one of claims 1 to 14, wherein the ordinary differential equation solver(72) is based on a neural network, such as an ordinary differential equation solver (72)being conditioned on additional information.

17. Audio decoder of one of the preceding claims, wherein the encoded audio signal (4)comprises a bit stream, a text, a midi file or any other representation of a time domainwaveform signal, or wherein the raw spectral data comprise mel spectral data, STFT data, MDCT data, MDST data or learned data derived from a bit stream representing the encoded audiosignal (4).

18. Audio decoder of one of claims 4, 6, 11, 12, 14, 15, 17, wherein the second input datacomprise pseudo noise data, colored noise, white noise or white Gaussian noise or a signal derived from a given distribution.

19. Audio decoder of one of claims 6 – 8, 11, 12, 14, 15, wherein the learned ordinarydifferential equation estimator (71) comprises a learned neural network (730) forreceiving, as an input, first input data derived from the encoded audio signal (4), and / or second input data, wherein the learned neural network (730) is configured to output, inresponse to the input, flow direction data of an ordinary differential equations systemat a point in a vector space determined by the first input data derived from the encoded audio signal (4) and / or the second input data, wherein the second input data are similarto corresponding data used as conditional data for determining the learned neural network (730) in a training phase, or a signal derived from the bit stream (4).

20. Audio decoder of one of the preceding claims, wherein the encoded audio signal (4) isorganized in a sequence of frames, wherein the sequence of frames comprises at least two time sequential frames, wherein each frame of the at least two sequential frameshas the same length in time, wherein the frame is a time frame, or represents a group of consecutive time-domain samples, or represents a group of LPC domain samples, or covers a certain time span of the original audio signal, or has a group of spectral values representing a certain time span of the original audio signal or a certain group of consecutive time domain samples.

21. Audio decoder of one of the preceding claims, wherein the encoded audio signal (4) isorganized in a sequence of frames, wherein the sequence of frames comprises at least two time sequential frames and preferably between 10 and 100 frames, wherein eachframe of the at least two sequential frames has a length in time being greater than 1ms and lower than 200 ms and preferably between 10 ms and 50 ms, wherein the frame is a time frame, or represents a group of consecutive time-domain samples, or represents a group of LPC domain samples, or covers a certain time span of the original audio signal, or has a group of spectral values representing a certain time span of the original audio signal or a certain group of consecutive time domain samples.

22. Audio decoder of one of the preceding claims, wherein the encoded audio signal (4) isorganized in a sequence of frames, wherein the sequence of frames comprises at least two time sequential frames, wherein the frame is a time frame, or represents a group of consecutive time-domain samples, or represents a group of LPC domain samples, or covers a certain time span of the original audio signal, or has a group of spectral values representing a certain time span of the original audio signal or a certain group of consecutive time domain samples, and wherein the learned conditional flow matching based processing stage (70) isconfigured to generate a sequence of frames of a raw decoded audio signal and to combine two timely adjacent frames of the raw decoded audio signal to obtain at least one frame of the decoded audio signal (40), wherein the two timely adjacent frames of the raw decoded signal correspond to two timely adjacent frames of the sequence of frames of the encoded audio signal (4).

23. Audio decoder of claim 22, in which each frame of the sequence of frames of theencoded audio signal (4) comprises spectral data derived from a short time spectrumof a sequence of short time spectra of an original audio signal corresponding to timely overlapping frames of the original audio signal, wherein the learned conditional flow matching based processing stage (70) isconfigured to combine the two timely adjacent frames of the raw decoded audio signal using an overlap-add-operation in calculating sampling values of the decoded audio signal (40).

24. Audio decoder of one of the preceding claims, wherein the encoded audio signal (4) isorganized in a sequence of frames, wherein the sequence of frames comprises at least two time sequential frames, wherein a frame comprises between 70 and 140 mel spectrogram bands derived from an original input audio signal having a sampling ratebetween 16 kHz and 48 kHz with a hop size between 128 and 1024 samples and awindow having a length between 1024 and 4096 samples, wherein the frame is a time frame, or represents a group of consecutive time-domain samples, or represents a group of LPC domain samples, or covers a certain time span of the original audio signal, or has a group of spectral values representing a certain time span of the original audio signal or a certain group of consecutive time domain samples.

25. Audio decoder of one of the preceding claims, wherein the learned conditional flowmatching based processing stage (70) comprises a learned ordinary differentialequation estimator (71) for estimating parameters for ordinary differential equations inresponse to the encoded audio signal (4), the learned ordinary differential equation estimator (71) comprising a neural network (730) comprising at least one feature of agroup of features comprising using a U-Net with residual 1D convolutional blocks, transformer blocks with snakebeta activations, a 1D Block consisting of a 1D convolution, group normalization and a Mish activation for an output of the U-Net, a 1×1 convolutional layer creating the data on the parameters for the differential equations, and / or corresponding time-step embeddings.

26. Audio decoder of one of the preceding claims, wherein the neural network comprisesat least one input side pair of a residual 1D convolutional block (702) and a transformer block (703), and at least one output side pair of a transformer block (714) and a residual1D convolutional block (713), wherein the transformer block (703) of the input side pairis configured to feed the residual 1D convolutional block (713) of the output side pair,and wherein the residual 1D convolutional blocks (702, 713)of the input side pair andthe output side pair are controlled by a time step embedding controller (708).

27. Audio decoder of claim 26, wherein the neural network comprises a second input sidepair of a residual 1D convolutional block (704) and a transformer block (705), and asecond output side pair of a transformer block (712) and a residual 1D convolutional block (711), wherein the transformer block (703) of the first input side pair is configuredto feed the residual 1D convolutional block (704) of the second input side pair, and wherein the transformer block (712) of the second output side pair is configured to feed the residual 1D convolutional block (713) of the first output side pair.

28. Audio decoder of claim 27, wherein the neural network comprises an inputconcatenation layer (701) receiving raw spectral data for one or more frames, or an output layer comprising a 1D block (715) and / or a 1×1 convolutional layer (716) for creating the parameters for the ordinary differential equations.

29. Audio decoder of claim 28, wherein the 1D block (715) of the output layer comprises atleast one of a 1D convolution process, a group normalization process, and a Mish activation process.

30. Audio decoder of claim 7 or 8, wherein the learned ordinary differential equationestimator (71) and the ordinary differential equation solver (72) are configured tooperate in only one step or iteratively in a number of steps between 16 steps and 64 steps.

31. Audio decoder of claim 4, wherein the spectral decoder comprises a feature expansionstage (5a), a subsequently connected transformer stage (5b) and a subsequentlyconnected channel expansion stage (5c).

32. Audio decoder of claim 31, wherein the feature expansion stage (5a) is configured toreceive a one-dimensional input and to output a two-dimensional output, or wherein the transformer stage (5b) is configured to receive a two-dimensional input and to output atwo-dimensional input, or wherein the channel expansion stage (5c) is configured toreceive a two-dimensional input and to output a three-dimensional output as input data input into the learned conditional flow matching core (7).

33. Audio decoder of claim 3, wherein the learned conditional flow matching core comprises adiffusion transformer or TADE or FiLM layers or a learned transposed convolution withor without overlapping between time-frequency-bin patches, wherein the diffusion transformer comprises at least one Diffusion Transformer Block or in case of more thanone Diffusion Transformer Blocks a concatenation of the more than one Diffusion Transformer Blocks.

34. Audio decoder of claim 4, wherein the spectrum-time converter is configured to apply adecompression factor for scaling a time-frequency representation, or wherein the decompression factor is frequency dependent .

35. Audio processing system comprising:an audio encoder (100) for encoding an audio input signal; and an audio decoder (200) of any one of claims 1 - 34.

36. Audio processing system of claim 35, wherein the audio encoder (100) comprises alearned encoder processor (1, 2, 3) for generating the encoded audio signal (4) fromthe input audio signal (20).

37. Apparatus of claim 36, wherein the learned encoder processor (3) comprises:a feature extractor (1) of extracting features from the input audio signal; and alearned encoder core (3) for generating the encoded audio signal (4) from theextracted features.

38. Method of decoding an encoded audio signal (4) to obtain a decoded audio signal (40),the method comprising: receiving the encoded audio signal (4); and transforming the encoded audio signal (4) into the decoded audio signal (40) using alearned conditional flow matching based processing.

39. Method of audio processing, the method comprising:encoding an audio input signal; and amethod of decoding of claim 38.

40. Computer program for performing, when running on a computer or a processor, amethod of claim 38 or claim 39.

Images