Post-processing of an audio signal according to a post-processing model resulting from optimised training

Abstract

The invention relates to a method for post-processing a decoded audio signal, wherein the post-processing (603) applies a post-processing model resulting from a training phase (602) carried out between a reference signal (x'), obtained by applying at least one pre-processing operation (601) to an input signal (x), and a decoded output signal (y'). The invention also relates to a post-processing device implementing the method.

Description

[0001] DESCRIPTION

[0002] Title: Post-processing of an audio signal according to a post-processing model derived from optimized training

[0003] Technical Field

[0004] The present invention relates to the general field of audio signal processing. The invention relates in particular to the compression of audio signals – hereafter referred to as “audio coding”. Particular interest lies in the improvement of existing codecs through post-processing (decoding).

[0005] This type of processing finds applications generally in the field of audio technologies, and more specifically in the field of audio decoding and rendering.

[0006] Previous technique

[0007] Post-processing is a well-known technique for improving the quality of a decoded signal in a backward-compatible way (without changing the format of the bitstream or the transport format in question). In the case of speech coding, historical methods based on signal processing include, for example:

[0008] - Improvement of decoded speech through post-filtering (of formant or pitch: "formant postfilter", "pitch postfilter"...)

[0009] - Improvement of the decoded music (by applying a frequency mask, reducing inter-harmonic noise, etc.)

[0010] An example of post-processing - incorporating several post-processing sub-modules - can be found in the 3GPP EVS codec (for "Enhanced Voice Services") and in particular in the decoding of the EVS AMR-WB 10 mode which is essentially an enhanced version of the original AMR-WB codec (for "Adaptive Multi-Rate- Wideband"), and which is described in the 3GPP TS 26.445 specification.

[0011] With the advent of deep learning, new methods based on neural networks have been proposed for the post-processing of decoded signals, using tools such as a convolutional network, or a mask-based approach, or even with approaches using antagonistic neural networks (GANs).

[0012] More recently, an approach that will be called "ScoreDec" proposing a diffusion-based post-processing, named SPF (for "score-based diffusion post-filter"), was presented in the article by Y.-C. Wu, D. Markovic, S. Krenn, ID Gebru and A. Richard, "ScoreDec: A Phase-Preserving High-Fidelity Audio Codec with a Generalized Score-Based Diffusion Post-Filter," Proc. ICASSP, 2024. This approach works in the complex spectrum domain and essentially consists of applying a diffusion model denoising method according to the article by J. Richter, S. Welker, I-M. Lemercier, B. Lay, and T. Gerkmann, "Speech enhancement and dereverberation with diffusion-based generative models," IEEE / ACM TASLP, vol. 31, pp. 2351-2364, 2023, to reduce the output encoding noise of existing AudioDec codecs (before GAN training) or Opus at 24 kbit / s. Thus, the broadcast model is only trained for a given encoding / decoding bitrate (here 24 kbit / s).

[0013] Figures 1a and 1b illustrate the operation of the method called "ScoreDec". The general principle is to apply denoising using a diffusion model

[0014] - according to the SGMSE method (for "Score-based Generative Models for Speech Enhancement") described later - as output from an existing codec (AudioDec or Opus in the article by Y.-C. Wu et al.). The ScoreDec method thus involves combining an existing codec with SGMSE denoising post-processing.

[0015] In Figure 1a, the AudioDec neural codec (block 100) takes an input audio signal x, sampled, for example, at 48 kHz; this signal is analyzed by an encoder (block 101) with a final neural projection layer P (block 102) to obtain a latent space z, which is then encoded (block 103) by residual vector quantization (multi-stage), and this discrete representation is decoded (block 104) to reconstruct the audio signal y. The AudioDec codec—here in its autoencoding version and not its vocoder version—is shown here.

[0016] - is normally trained in two phases: a phase 1 where the generator (block 100) is trained alone, then a phase 2 in a generator-anchored network (GAN) architecture with the addition of a discriminator providing an adversarial cost function. According to the aforementioned article by Y.-C. Wu et al., by adding post-processing (block 130), it is possible to implement only phase 1 of the training for block 100; the SPF post-processing (block 130) avoids the need for a GAN-type phase 2. Refer to the aforementioned article by Y.-C. Wu et al. for details of the training for block 130, with a training procedure illustrated here in block 120 and providing the parameters used in block 130 for inference. Block 130 here corresponds to a pre-trained SGMSE diffusion model with 6 weights. The training here takes a series of examples (x,y) where x is the input of block 100 and y is the (decoded) output of block 104.Thus, the SGMSE denoising method is repurposed here to reduce the coding noise induced by block 100. This SPF post-processing is also applied in the aforementioned article by Y.-C. Wu et al. to another codec, Opus, which is described in IETF RFC 6716 and IETF RFC 8251. In Figure 1b, the Opus codec (block 150), divided into an encoder (block 151) and a decoder (block 152), is followed by post-processing (block 180). As with AudioDec, a block 170 is highlighted here, illustrating that the training of the SPF post-processing block is done using a sequence of examples (x,y), where x is the input of block 150 and y is the (decoded) output of block 152; Thus, block 180 corresponds to a pre-trained SGMSE post-processing, using inference the 6 weights from block 170.

[0017] It should be noted that the aforementioned article by Y.-C. Wu et al. does not explicitly state that the input x is used by block 170, however the results presented in this article in terms of scale-invariant signal-to-noise ratio (SI-SNR) allow a person skilled in the art to verify that this is indeed the case.

[0018] Furthermore, in figure lb we explicitly highlight the high-pass pre-filter (denoted HPF for high-pass filter) of Opus at 160, while the rest of the Opus coding is done at 161. This distinction is not defined in the article by Y.-C. Wu et al. cited above, however we will see later that this pre-filter block can make the SPF post-processing suboptimal if it is not properly taken into account.

[0019] Figure 2a describes the operation of the SPF post-processing (which in fact corresponds exactly to the SGMSE denoising model according to the aforementioned article J. Richter et al with the following adaptations, as used in the ScoreDec method to improve the quality of the decoded signals after the AudioDec or Opus codec (blocks 130 or 180).

[0020] It should be noted that the SGMSE model has several versions, and in some cases the name is modified; for example, article 1-E. Ayilo, M. Sadeghi, R. Serizel, Diffusion-based speech enhancement with a weighted generative-supervised learning loss, Proc. ICASSP, 2024, explicitly distinguishes it from a version called "SGMSE+". In this document, we will retain the generic name SGMSE.

[0021] The degraded audio signal here corresponds to the decoded y signal (output of block 104 or 152). For the sake of brevity, the time indices n of the y(n) signal, sampled for example at 48 kHz, are not shown here.

[0022] This decoded signal y is first normalized (block 202) by a scaling factor g given by the maximum of the absolute value of y (determined non-causally over the entire processed signal): g = max|y(n)| n

[0023] Where the index n is applied to the entire signal y of limited duration. Then, this normalized signal y norm is transformed by an STFT-203 type transform (for "short time Fourier Transform" in English) to obtain the complex spectrogram Y of size K x F, where K is the number of time frames of analysis and F is the number of frequency lines.

[0024] This transformation corresponds to:

[0025] Where N FFT is the length of the FFT (typically N FFT = 510), h(n) is an analysis window (e.g., Hanning window) of length N win (typically N win = N FFT N hopis the number of new samples analyzed in each frame (typically N hop = 128). Some details, such as the centering of the analysis window ù(n) relative to the blocks of the input signal, are not addressed here. For N FFT = 510, we obtain F = 256 frequency lines. Furthermore, for a 16 kHz signal with a length of 2.048 seconds, we will have K = 256, which would give a spectrogram in the form of a square "image" with a complex coefficient. In practice, the value of K will depend on the effective length of the signal to be processed.

[0026] The amplitude of Y is transformed by default by amplitude compression to 204 (in order to reduce the spectral dynamic range):

[0027] Where the value of p is, for example, p = 0.15, k is the transformed frame index, f is the frequency line index, and, for example, a = 1 / 2. Again, for the sake of brevity, the indices k and f of the Y, Y spectrograms are not generally shown. trans ..., except for the equation below where these details are relevant to remove any ambiguity.

[0028] Finally, the transformed spectrum Y trans is denoised by sampling a diffusion model at 205 (inference phase) to obtain the signal x trans , which undergoes the inverse transformation (block 206) to block 204:

[0029] The resulting spectrogram X is time-transformed to 207 to obtain x norm which is scaled (block 208) by the g factor from block 201.

[0030] The denoising module in the STFT spectrogram domain at 205 can be implemented according to the article by J. Richter et al., but it can also be implemented according to the article by I-E. Ayilo et al., which extends the learning criterion of the "score" (in generative mode) to a mean squared error cost function on the signal (in supervised mode). A description of the 205 block according to J. Richter et al. is now given in relation to the backward diffusion process.

[0031] The SGMSE denoising model used by the ScoreDec method is described in detail in the aforementioned article by J. Richter et al. It is a generative speech enhancement model based on diffusion. Given a degraded speech signal, this model generates a "clean" speech sample, corresponding to a denoised version of the original signal. Unlike some neural approaches that attempt to learn a correspondence between noisy and "clean" speech, the SGMSE model learns to generate a "clean" speech sample through successive denoising operations from Gaussian noise, conditioning this generation on the noisy signal. The core idea of ​​the SGMSE model is as follows.

[0032] The starting point for training an SGMSE neural model is to create a database containing pure speech signals and their degraded versions. In denoising, the degraded version is obtained by adding noise at a specific signal-to-noise ratio; in the aforementioned article by J. Richter et al., the audio database is public and corresponds to VoiceBank-DEMAND (at 16 kHz). In the context of ScoreDec, in the article by Y.-C. Wu et al., pure speech corresponds to the public database "Valentini," while the degraded version is obtained by encoding and decoding using AudioDec or Opus.

[0033] The samples in the resulting audio database (pure speech, degraded version) are represented in the frequency domain after a complex short-term discrete Fourier transform (STFT). Working in this domain allows for the simultaneous improvement of both amplitude and phase spectra.

[0034] The pure speech spectrograms X, denoted x0 hereafter, are progressively noisy during a process (called "forwarding") that transforms them into a heavily corrupted version of the corresponding noisy data, denoted y. This process is parameterized by a continuous time variable t and governed by a stochastic differential equation (SDE). We denote x t the pure speech spectrogram at time t, following the "forward" process.

[0035] Theoretical results confirm that it is possible to reverse the previous process, that is, to return to a pure speech signal starting from a noisy speech signal. This process (called "backward") admits an explicit expression in the form of a speech-sensing differential equation (SSE). The "forward" process is now detailed. We work with complex spectrograms, generically denoted here as x. tindexed by the time variable t, which evolves in the interval [0, T] where t=0 is the initial time (when the speech signal is pure) and t=T is the final time (when the speech signal is buried in noise). Thus, %0 denotes a "pure" spectrogram, an element of the training set, and x T The spectrogram is degraded at the end of the forward scattering process. The forward process follows the dx EDS. t = fx t , y)dt + g(t)dw, where f is a drift term governing the average behavior of the equation, given by the expression f(x,y) = y(y - x), with y >0 a stiffness coefficient, g is a diffusion term controlling the amount of noise injected into the equation at each instant, w is a Wiener process (or Brownian motion), at the origin of the stochastic character of the equation.

[0036] The drift term leads x0 to y with a velocity controlled by the stiffness parameter. The noise added by diffusion leads x0 to a heavily corrupted version x T at the final time.

[0037] The interval [0,T] is discretized into N subintervals of length AT, and the backward equation is solved between t=T and t=t £ where t £ -0, using numerical methods such as the Euler-Maruyama method. The limitation to t £ This helps to avoid numerical instabilities that can occur for t close to 0. For example, we can take T=1 and N=30.

[0038] Several numerical methods for solving stochastic differential equations (SDEs) can be used during the "forward" phase, the most common being the Euler-Maruyama method, an extension of Euler's method for ordinary differential equations.

[0039] Figure 2b illustrates the NCSN++ (Noise Conditional Score Network) architecture used by default by the SGMSE method during training and inference. The NCSN++ architecture was introduced in the aforementioned article by Y. Song et al. The architecture has been adapted to handle complex spectrograms: the real and imaginary parts are treated as separate channels. Thus, the score estimation... e The output from the network is done separately for the real and imaginary parts.

[0040] The neural network has a multi-resolution "UNET" structure with different types of layers (Conv2D, DownLayer, UpLayer, BottleneckLayer, ProgDown, and ProgUp). Further details on the structure of this network are given in the article by J. Richter et al. (section IV, "Network architecture").

[0041] Figure 2c details the structure of the residual blocks present in the NCSN++ architecture during each undersampling / oversampling. A residual block includes Conv2D layers in 283, 287, and 289, group normalization (according to Y. Wu and K. He, "Group normalization," Proc. ECCV, 2018) in 280 and 285, finite impulse response filter over- and undersampling (according to R. Zhang, "Making convolutional networks shift-invariant again," Proc. ICML, 2019) in 282 and 288, and a Swish activation function (according to P. Ramachandran, B. Zoph, and QV Le, "Swish: a self-gated activation function," arXiv preprint arXiv: 1710.05941, 2017) in 281 and 286. To make the model time-dependent, information about this time variable (indicating the progress in the (diffusion process) is integrated into each residual block, by means of a Fourier embedding which maps the variable t to a vector temb (block 284). The information from the two branches is summed (block 290) and then normalized by a factor V2 (block 291).

[0042] Figure 3 illustrates the forward scattering process on the interval [0,T] discretized into N subintervals of length AT where N=30. It shows an example of a scattering process applied to a complex spectrogram X = x0. Only the amplitude spectrogram is shown here. The pure speech spectrogram x0 is progressively noisy by EDS as implemented in the SGMSE method for T=l, the result after a few iterations (x0, x 5ÙT , x 10ÙT , x 15ÙT , x T At the end of the "forward" process (t=T), the signal x T drifted towards Y but the result is almost totally drowned out by the noise (only a few energetic components of pure speech can be distinguished).

[0043] The "backward" process aims to solve the inverse SDE, and in the article by Richter et al., this solution is performed using the Predictor-Corrector sampling procedure introduced in the article by Y. Song, J. Sohl-Dickstein, DP Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations," Proc. ICLR, 2021, to iteratively generate the denoised signal estimate. The inverse SDE is given by: dx t = -[f(x t , ) + g(JV x ogp t x t \y)\dt + gt)dw Where w is the time-reversed Wiener process. This inverse SDE involves an a priori unknown coefficient - the score (which is an approximation of the derivative of the conditional log-probability) Xt logp t (x t \y ), depending on the conditional distribution p t ( t \y). We denote s0(x t ,y, t) the approximate score evaluated at a point xt at time t, where 0 represents the parameters of the approximation (weight of a neural network).

[0044] Using a neural network, the SGMSE model learns the score via a function s e x t ,y, t), where 0 denotes the network weights. It is then possible to approximately solve the SDE of the "backward" process by replacing the score with its approximate value.

[0045] Several neural network architectures, built around the U-Net architecture, have been proposed for SGMSE, for example DCUNET for "Deep Complex U-Net" or NCNS for "Noise Conditional Score Network".

[0046] Here again, several numerical methods for solving SDEs can be used during the "backward" phase, the most common being the Euler-Maruyama method.

[0047] Blocks 120 and 170 in figures aa and lb correspond to the training of the SPF post-processing. To do this, the pure / degraded signals x( ) and y( ) are both normalized (for example by taking the maximum value of the amplitude of y(n) to normalize both x(n) and y(n) by the same factor) and transformed in the frequency domain as in blocks 203 and 204 illustrated in figure 2a, to obtain the pure / degraded spectrograms.

[0048] At each stage of training, a time t is first sampled according to a uniform distribution over the interval [0, t £ ], then a pair (x0,y) of pure / degraded spectrograms is randomly chosen from the database. Knowing x0 and y, the distribution of x t can be determined explicitly: therefore, we can directly sample x tand calculate the corresponding scoring function. A distance of 12 (squared error) between the scoring model and the score is finally calculated, and then the neural network parameters are updated.

[0049] The ScoreDec method yields very good results for improving the AudioDec codec, limited to phase 1 of training (without the GAN architecture of phase 2). In the aforementioned article by YC Wu et al., the bitrate of AudioDec and Opus is set at 24 kbps (for signals sampled at 48 kHz). It is well known to those skilled in the art that Opus already achieves good speech quality at 24 kbps; the improvement provided by the SPF post-processing (block 170) is less pronounced for Opus than for AudioDec. The ScoreDec method treats the AudioDec and Opus codecs upstream as "black boxes." This simplification is acceptable for AudioDec, which is an end-to-end neural codec without "classical" signal processing and with virtually no algorithmic delay implementation.However, it is well known to those skilled in the art that the Opus codec uses a high-pass pre-filter (HPF) in block 160, introducing a phase shift on the input signal, and that the Opus codec causes an algorithmic delay (due to the "lookahead" anticipation process and internal filtering in block 161) which is by default 6.5 ms (312 samples at 48 kHz); however, these two types of delay are not taken into account in the ScoreDec method, which means that the SPF post-processing (block 170) will actually result in a phase lead (of 312 samples) and model the phase response of the HPF filter (block 161), whereas these two operations are actually undesirable (one can create problems for real-time processing and remains a non-causal operation, the other is not "perceptually" justified).

[0050] Furthermore, the ScoreDec method is not applied to existing widely deployed speech codecs, such as 3GPP AMR (for "Adaptive Multi-Rate" in English) or 3GPP AMR-WB (for "Adaptive Multi-Rate-Wideband" in English).

[0051] Furthermore, the ScoreDec method, as demonstrated in the aforementioned article by Y.-C. Wu et al., is trained at a single bitrate (24 kbit / s). It would be advantageous to have a method applicable to a multi-bitrate codec such as AMR-WB.

[0052] Furthermore, ScoreDec is typically trained and tested only on pure speech; it would be advantageous to have a method adapted to different types of content (pure speech, music, noisy speech, etc.).

[0053] We now recall the structure of examples of "traditional" codecs such as AMR-WB or its improved version in EVS (as EVS AMR-WB 10 mode).

[0054] The AMR-WB codec is described for example in the article by B. Bessette et al., "The adaptive multirate wideband speech codec (AMR-WB)," in IEEE Transactions on Speech and Audio Processing, vol. 10, no. 8, pp. 620-636, Nov. 2002.

[0055] Figures 4a and 4b describe the simplified architecture of the AMR-WB encoder and decoder respectively. This codec operates at 9 bit rates called "modes" (6.6, 8.85, 12.65, 14.25, 15.85, 18.25, 19.85, 23.05 and 23.85 kbit / s).

[0056] Figure 4a describes the simplified architecture of the AMR-WB encoder. The input signal is sampled at 16 kHz (% | 16fc| ), is generally with a PCM (for Power Code Modulation) format input, linearly truncated to 14 bits according to the 3GPP TS 26.171 standard. For the sake of simplification, figures 4a and 4b do not show the non-encoder / decoder blocks, such as voice activity detection (VAD), discontinuous transmission (DTX) or comfort noise generation (CNG).

[0057] The input signal is decimated at 12.8 kHz (block 401), then pre-filtered by a Butterworth high-pass filter with a cutoff at 31 Hz (block 402), and finally pre-emphasized by filter 1 - az -1 (block 403) where a = 0.68. The pre-emphasized low-band (0-6.4 kHz) signal x pre is analyzed by linear prediction (LP), the LPC coefficients are quantized and interpolated in 4(z) per subframe (block 404). An open loop pitch analysis (OL) is then performed (block 406) in the domain of the signal perceptually weighted by the filter (block 405):

[0058] 4(z / yi) W(z) =1

[0059] 1 — az 1

[0060] Where / i = 0.92. Next, the closed-loop search of the adaptive dictionary (block 407) and the fixed dictionary (block 408) is performed, before joint encoding of the dictionary gains (block 409). When the codec is operating at 23.85 kbit / s, high-band gain parameters (6.4–7 kHz) are also determined and encoded (block 420). All parameters encoded in the current frame are multiplexed (block 410).

[0061] Figure 4b describes the simplified architecture of the AMR-WB decoder.

[0062] During decoding, the parameters are demultiplexed (block 460), the LPC parameters are decoded and interpolated (block 452), the excitation combining the contributions of the fixed and adaptive dictionaries is decoded (451), before applying the LPC synthesis filter (block 454) per subframe, then the synthesis is de-emphasized (block 455) and resampled to 16 kHz (block 456). The decoded low-band signal is then... L [ 6k]is combined (block 458) with the reconstructed high-band signal y^ 6k] (at block 457) - if the decoder is operating at 23.85 kbit / s, the high band correction gains are decoded (block 453), otherwise the high band scaling factors are estimated "blindly" for all bit rates from 6.6 to 23.05 kbit / s.

[0063] Figure 5 describes the simplified architecture of the EVS encoder and decoder (including the AMR-WB IO mode).

[0064] Block 501 includes various preprocessing sub-blocks (such as a high-pass pre-filter "HP Filt.", a filter bank and resampling "FB & Res.", a pre-emphasis module "Pre-emp", a classification module "Class.", a linear prediction analysis module "LP Ana.", etc.). The individual preprocessing sub-blocks are not detailed here. Similarly, various post-processing sub-blocks are included in block 508 (such as a high-pass post-filter "HP Filt.", a filter bank and resampling "FB & Res.", a de-emphasis module "De-emp", a formant post-filter "Form. Post Filt.", a music enhancement module "Music Enhancer", etc.). These are not detailed here either. Block 502 handles the encoding of the "EVS Primary" mode, and block 506 handles the decoding of this mode. The sub-blocks that constitute them ("switch.", "LP-cod." or "LP-decod.", "MDCT cod." or "MDCT decod.", "DTX, CNG cod." or "DTX, CNG decod."" are not detailed here. Similarly, the transmitter (504) and the receiver in 505 are ignored here.

[0065] Block 503 concerns the encoding of the "AMR-WB IO" mode, block 507 concerns the decoding of this mode.

[0066] Unlike AMR-WB, a Butterworth-type high-pass pre-filter (called a "HP filter") with a 20 Hz cutoff is applied directly to the input signal in 501 (before resampling to 12.8 kHz for AMR-WB IO mode). It should also be noted that numerous 508 post-processing blocks are defined in the EVS decoder.

[0067] There is a need to be able to apply neural network post-processing (e.g., SGMSE) to existing CELP codecs such as AMR-WB or other types like Opus, and in an optimized manner. There is also a need to be able to apply neural network post-processing (e.g., SGMSE) that is suitable for multibitrates and different types of content (speech, music, etc.).

[0068] Description of the invention

[0069] The invention improves upon the existing state of the art.

[0070] To this end, the invention relates to a method for post-processing a decoded audio signal, in which the post-processing applies a post-processing model resulting from a training phase carried out between a reference signal and the decoded audio signal, the reference signal being obtained by applying at least one pre-processing to an input signal, a pre-processing being a processing from the following list:

[0071] - compensation for delay;

[0072] - phase compensation;

[0073] - a pre-sharpening treatment;

[0074] - a perceptual weighting treatment...

[0075] Thus, unlike state-of-the-art methods where the training phase is applied between the codec's input signal, without modification, and the output signal, the proposed post-processing method applies preprocessing to the input signal during the training phase. This allows us to leverage knowledge of the algorithmic delay and / or the internal structure of existing codecs to obtain a suitable reference signal and, consequently, a suitable preprocessing.

[0076] In one embodiment, a preprocessing applied to the input signal is a delay compensation.

[0077] Thus, in the case of a codec with an algorithmic delay, such as the Opus codec, the delay is compensated for during the training phase, resulting in a suitable post-processing model because the input and output signals are optimally synchronized. Furthermore, this prevents the post-processing from resulting in a phase lead, which is undesirable in real-time operation when processing successive blocks of output signals.

[0078] A preprocessing applied to the input signal can also be phase compensation.

[0079] Thus, in the case of codecs where a high-pass prefilter is applied to the input signal, training applied to the codec's input signal without taking this prefilter into account is not optimal. This is because it models the phase response of the prefilter, whereas the prefilter has virtually no perceptual effect but typically serves to remove or reduce the DC component that would introduce bias in internal codec analyses. Taking this prefiltering into account through phase compensation preprocessing allows the training phase to be adapted to this type of codec.

[0080] A preprocessing applied to the input signal for the training phase can be -

[0081] - a pre-sharpening treatment;

[0082] - a perceptual weighting treatment.

[0083] It can also be a resampling process between a first and a second frequency.

[0084] Thus, post-processed codecs are no longer considered black boxes. The internal structure of existing codecs is exploited here to optimize training, resulting in a post-processing model tailored to the codec and thereby improving the quality of the decoded signal, since the post-processing focuses on reducing the coding noise induced by the codec.

[0085] In some embodiments, the decoded audio signal is the audio signal decoded before inverse processing is applied to at least one preprocessor from the list. The post-processing is applied to the decoded signal before inverse processing is applied to at least one preprocessor from the list.

[0086] The reference signal used for the training phase is taken after an existing preprocessing in the codec's processing chain, and the output signal used for the training phase is taken before the inverse processing of the preprocessing in the codec's processing chain.

[0087] In one embodiment, the post-processing model is a diffusion denoising model.

[0088] The diffusion denoising model is adapted here by incorporating preprocessing to obtain a reference signal for the training phase. The denoising model can be of the SGMSE type with an NCSN++, DCUNET, or another model variant architecture, with different configurations of solvers, EDS resolution, etc., as described later.

[0089] According to one embodiment, the post-processing method uses a neural network for the training phase and for an inference phase corresponding to the application of the post-processing model.

[0090] According to a particular embodiment, the post-processing model depends on a conditioning value.

[0091] In example implementations, the conditioning value is a coding rate value or a value indicating a signal type.

[0092] Thus, the post-processing model is best adapted to the characteristics of the encoder or signal.

[0093] The invention relates to an audio signal post-processing device comprising a post-processing module applying a post-processing model to a decoded audio signal and a training module generating the post-processing model and applying a training phase between a reference signal and the decoded audio signal, the reference signal being obtained by applying at least one pre-processing to an input signal, a pre-processing being a processing from the following list:

[0094] - compensation for delay;

[0095] - phase compensation;

[0096] - a pre-sharpening treatment;

[0097] - a perceptual weighting treatment.

[0098] This post-processing device includes a processing circuit for implementing the steps of the post-processing method as described above. The invention relates to a computer program comprising instructions for implementing the post-processing method as described above, when executed by a processor.

[0099] Finally, the invention relates to a storage medium, readable by a processor, storing a computer program containing instructions for the execution of the post-processing method described above.

[0100] Brief description of the drawings

[0101] Other features and advantages of the invention will become more apparent from the following description of particular embodiments, given by way of simple illustrative and non-limiting examples, and the accompanying drawings, among which: [Fig la] and [Fig lb] illustrate the state-of-the-art method called ScoreDec, as described above, for improving the signal from an AudioDec codec and an Opus codec respectively;

[0102] [Fig 2a] illustrates the state-of-the-art SGMSE (for "Score-based Generative Models for Speech Enhancement") method, as described previously;

[0103] [Fig 2b] illustrates the architecture of a state-of-the-art NCSN++ neural network, as described previously;

[0104] [Fig 2c] illustrates a residual block of the state-of-the-art NSCN++ architecture, as described previously;

[0105] [Fig 3] illustrates the effects on the signal of a state-of-the-art diffusion process;

[0106] [Fig 4a] illustrates the simplified architecture of an AMR-WB encoder (for "Adaptive Multi-Rate-Wideband" in English);

[0107] [Fig 4b] illustrates the simplified architecture of an AMR-WB decoder;

[0108] [Fig 5] illustrates the simplified architecture of an EVS (for "Enhanced Voice services" codec) (coder and decoder);

[0109] [Fig6] illustrates a post-processing device and a post-processing method in a general implementation of the invention;

[0110] [Fig7a] illustrates a first embodiment of a post-processing device and a post-processing method according to the invention for an AMR WB type codec;

[0111] [Fig7b] illustrates a second embodiment of a post-processing device and a post-processing method according to the invention for an AMR WB type codec;

[0112] [Fig8] illustrates a third embodiment of a post-processing device and a post-processing method according to the invention for an AMR WB type codec; [Fig9] illustrates a fourth embodiment of a post-processing device and a post-processing method according to the invention for an AMR WB type codec; [Fig10] illustrates a fifth embodiment of a post-processing device and a post-processing method according to the invention for an AMR WB type codec; [Fig11] illustrates an embodiment of a post-processing device and a post-processing method according to the invention for an Opus type codec;

[0113] [Figl2a] illustrates an embodiment of a context-based conditioning device and a context-based conditioning method, with a residual block of the NSCN++ architecture modified according to the invention;

[0114] [Figl2b] illustrates an alternative embodiment of a context conditioning device and a context conditioning method, with a residual block of the NSCN++ architecture modified according to the invention;

[0115] [Fig 13] illustrates an example of a structural realization of a post-processing device according to an embodiment of the invention.

[0116] Description of the implementation methods

[0117] Figure 6 illustrates a post-processing device and a post-processing method in a general embodiment of the invention.

[0118] A preprocessing module 601 applies at least one preprocessing step to the input signal x, for example, the input signal of a codec. This preprocessing module can be part of the post-processing device 600 (module 601 within device 600), which is then added to an existing codec. Alternatively, the preprocessing module 601 (outside device 600) can be composed of the existing preprocessing modules within the codec to which the post-processing device is applied.

[0119] A preprocessing method applied by this module is, for example, a preprocessing method for the following list:

[0120] - compensation for delay;

[0121] - phase compensation;

[0122] - a high-pass type filter;

[0123] - a resampling process between a first and a second frequency;

[0124] - a pre-sharpening treatment;

[0125] - a perceptual weighting treatment.

[0126] The post-processing device 600 thus obtains a pre-processed signal x' which serves as a reference signal for a training phase operated by the training module 602. This training module implements a training phase between this reference signal and a decoded output signal y'. This training module implements, for example, a training phase using a "forward" and "backward" diffusion process as described previously and illustrated in Figure 3.

[0127] At the end of this training phase, a post-processing model defined by parameters 0 is obtained. These parameters are sent to the post-processing module 603, which applies this model to the decoded output signal y'. The application of this post-processing model constitutes an inference phase. The post-processing module 603 is, for example, as described in Figure 2a.

[0128] Depending on the preprocessing performed by the 601 module, the decoded output signal may be the signal before applying inverse processing to the preprocessing performed, with the post-processing being applied to the same decoded signal.

[0129] Several specific embodiments, applied to existing codecs, are now described.

[0130] Figure 7a shows a first embodiment of a post-processing device 600 and a post-processing method according to the invention for an AMR-WB type codec.

[0131] In the embodiment of Figure 7a, we find the coding blocks of Figure 4a (400, 401, 402, 403 and 420) and the decoding blocks of Figure 4b (450, 454, 455, 456, 457, 458) - blocks 400 and 450 are not detailed here for convenience to make the diagram more compact.

[0132] A post-processing unit 600, as described with reference to Figure 6, is added to the processing chain of this codec. It includes the training block 602 and the post-processing block 603. Blocks 710 and 711 constitute the pre-processing module applied to the input signal.

[0133] Block 603 is a diffusion model post-processing (in inference, with pre-trained weights) applied to the 16 kHz codec output.

[0134] Block 602 corresponds to the diffusion model drive; according to the invention, the drive takes as its reference signal ("clean") the signal % | 16fc| sampled at 16 kHz, filtered to 710 and delayed to 711 by D samples, and as a degraded ("noisy") signal y [16k] .

[0135] Two preprocessing steps are applied to the codec's input signal to obtain the reference signal for the training phase: phase compensation and delay compensation. Delay compensation using D samples is important here because it is assumed that % | 16fc| This corresponds to the new samples for each frame, before taking into account the "lookahead" anticipation process.

[0136] The delay D is, for example, 95 samples for AMR-WB. The phase compensation aims to reproduce at 16 kHz the response of block 402, which operates at 12.8 kHz. In a variant, block 710 is implemented as a second-order Butterworth IIR filter in the following form: with bo = b2= 0.99142886, 0! = -1.98285771 and ^ = -1.98278425 and a2= 0.98293118.

[0137] This example filter is obtained, for example, with the following Python pseudocode: `import scipy.signal as signal sos = signal.butter(2, 31 / 8000, 'highpass', output='sos') b=sos[0, 0:3] a=sos[0, 3:]`

[0138] In some variations, different coefficient values ​​are also possible.

[0139] Blocks 710 and 711 can be swapped and / or combined into a single treatment.

[0140] In another variant, block 711 is actually implemented in block 602, after an STFT transformation and by applying the H filter response HP 16kHZ z) to the spectrum of the reference signal at the output of 710. In this variant, the STFT transformation of the shifted input x | 16fc| (n - D) is always done in 602, but block 710 is not applied to x [16fe] (n), it is implemented in 602 in the frequency domain.

[0141] Figure 7b shows a second embodiment of a post-processing device 600 and a post-processing method according to the invention for an AMR-WB type codec.

[0142] As with Figure 7a, we find coding blocks from Figure 4a (400, 401, 402, 403 and 420) and decoding blocks from Figure 4b (450, 454, 455, 456, 457, 458).

[0143] A post-processing unit 600, as described with reference to Figure 6, is added to the processing chain of this codec. It includes the training block 602 and the post-processing block 603. In this embodiment, the pre-processing steps applied to obtain the reference signal for the training phase in 602 differ from those shown in Figure 7a. Several pre-processing steps are performed.

[0144] The signal x hpf is shifted to 713, with D2=76 samples at 12.8 kHz. Optionally, the signal % | 16fc|is shifted to 712 with D=95 samples at 16 kHz for AMR-WB, or the high-band signal xÿ F (obtained by high-pass filtering on the 6-7 kHz band) is shifted by D=95 samples at 16 kHz.

[0145] Late payment compensation for x hpf and xÿ F is important here because we consider that x hpf This corresponds to the new samples for each frame, before the "lookahead" anticipation process is taken into account. If these signals are taken at the level of the current frame encoded after the "lookahead" anticipation process, the delay in 713 and 714 is not necessary.

[0146] Next, two STFT analyses are performed and combined in 602, with windows defined in the same way and time-synchronized, in order to obtain:

[0147] • The spectrum of the 0-fc Hz band of x hpf shifted (sampled at 12.8 kHz) and the spectrum of xÿ Foffset (sampled at 16 kHz) for the high band part (after bandpass fc-7 kHz or high pass cutoff at fc kHz), where fc is for example fixed at fc=6 or 6.4 kHz.

[0148] • The spectrum of the 0 to fc kHz band of x hpf offset (sampled at 12.8 kHz) and the fc band spectrum at 8 kHz of % | 16fc| offset (sampled at 16 kHz) for the high band part, where fc is for example fixed at fc=1 kHz.

[0149] In both cases, this allows us to take into account the frequency response of the 402 filter. The use of xÿ F (high-band signal obtained by 6-7 kHz bandpass filtering of the original signal % | 16fc| The drawback of this is that it is impacted by the frequency response of the high-band extraction filter in AMR-WB, so the junction at 6.4 kHz is not optimal. It is therefore preferable to use the variant where the spectra of x hpf And % 116fc| are combined into a single input spectrum of 602.

[0150] Thus, the input corresponding to the reference signal in 602 is here a mix between a 12.8 kHz signal (after the following preprocessings: resampling in 401, HP in 402 and delay compensation in 713) and a signal reflecting the high band and resulting from the preprocessing in 714. The post-processing block 603 operates here at 16 kHz.

[0151] Figure 8 shows a third embodiment of a post-processing device 600 and a post-processing method according to the invention for an AMR-WB type codec. This figure therefore illustrates a variant of an audio signal processing chain with post-processing integrated into the AMR-WB encoding and decoding. In this variant, the post-processing does not operate on the 16 kHz decoding output signal, but only on the 12.8 kHz low band.

[0152] According to this embodiment, the 602 drive takes as its input, as the reference signal, the signal xhpf and as a degraded signal, the signal y (in both cases, we take the successive "current" frames). The signal x hpf is thus obtained after resampling preprocessing in 401 and high-pass (HP) filtering in 402. The signal y, taken as the degraded signal for the training phase, is an output signal before the application of the inverse processing to the resampling preprocessing performed in 401. Since the input of the training module 602 is taken after filtering in 402, the two signals x hpf and are "consistent" and no prefilter response compensation is necessary. Furthermore, no delay compensation is necessary because we consider here that x hpf This corresponds to the current coded frame.

[0153] For this variant, only the low band (0-6.4 kHz) is improved.

[0154] Figure 9 shows a fourth embodiment of a post-processing device 600 and a post-processing method according to the invention for an AMR-WB type codec.

[0155] In this variant, post-processing does not operate on the 16 kHz decoding output signal, but only on the 12.8 kHz low band, as in the previous embodiment. This time, the post-processing is applied in the pre-emphasized signal domain, that is, before the implementation of the de-emphasis processing, also known as de-emphasis.

[0156] According to this embodiment, the 602 drive takes as its input, as the reference signal, the signal x pre and as a degraded signal the signal y pre (In both cases, we take the successive "current" frames). The signal x preis thus obtained after preprocessing with resampling in 401, high-pass (HP) filtering in 402, and pre-emphasis, also called pre-accentuation, in 403. The signal y pre taken as a degraded signal for the training phase, this is an output signal before the application of inverse processing steps 455 and 456, respectively the pre-emphasis preprocessing in step 403 and resampling in step 401. Since the input to the training module 602 is taken after filtering in step 402, the two signals x pre and y pre are "consistent" and no prefilter response compensation is necessary. Furthermore, no delay compensation is necessary because we consider here that x pre This corresponds to the current coded frame.

[0157] For this variant, only the low band (0-6.4 kHz) is improved.

[0158] Figure 10 shows a fifth embodiment of a post-processing device 600 and a post-processing method according to the invention for an AMR-WB type codec.

[0159] In this variant, post-processing does not operate on the 16 kHz decoding output signal, but only on the 12.8 kHz low band. This time, post-processing is applied in the perceptually weighted signal domain, i.e., after filtering by M7(z) at 1011 and before the implementation of the inverse filtering V / -1 (z) in 1012. Here the M7(z) filter is defined as

[0160] 4(z / yi) W(z) = ,

[0161] 1 — az 1 where 4(z) is the LPC filter decoded in each 5ms subframe.

[0162] According to this embodiment, the 602 drive takes as its input, as the reference signal, the signal x preto which a perceptual weighting filtering W z) is applied at 1010 and as a degraded signal the signal y pre to which a perceptual weighting filter W(z) is applied in 1011. The reference input signal of the training module 602 is thus obtained after preprocessing of resampling in 401, high-pass filtering (HP) in 402, pre-emphasis in 403 and perceptual weighting in 1010. The signal taken as the degraded signal for the training phase is a perceptually weighted output signal before the application of the inverse processing 1012, 455 and 456 respectively of the preprocessing of perceptual weighting (1010 and 1011), pre-emphasis in 403 and resampling in 401.

[0163] The advantage of this embodiment is that it is in a signal domain consistent with low-band CELP coding, where the coding noise is much less colored.

[0164] For this variant, only the low band (0-6.4 kHz) is improved.

[0165] In variants of the invention with reference to Figures 7 to 10, the AMR-WB decoder (blocks 450 to 458) can be replaced by an EVS AMR-WB IO decoder (507), possibly with the deactivation of all or part of the post-processing blocks defined in AMR-WB IO (508), except for necessary blocks such as comfort noise generation, de-emphasis, and resampling. Indeed, in the same way as for the embodiments described above for the AMR-WB codec, the post-processing device 600 according to the invention can be added to the processing chain of the EVS codec as illustrated in Figure 5. One or more pre-processing steps included in module 501 can be applied to the input signal to obtain the reference signal for the drive phase of the post-processing device.

[0166] Thus, as with the examples described for the AMR-WB codec, training can be performed in a pre-filtered, pre-emphasized, or perceptually weighted signal domain. The output signal used as the degraded signal for the training phase is the signal before the inverse preprocessing is applied.

[0167] The post-processing performed for the EVS codec improves the quality of the output signal to such an extent that it is possible to disable certain post-processing steps included in block 508 of Figure 5 (such as, for example: "music enhancer", "bass postfilter", "formant postfilter", "LTP postfilter"). Indeed, these blocks were optimized on an ad hoc basis, whereas the post-processing according to the invention is optimized end-to-end and can replace the combination of several of these blocks.

[0168] Figure 11 illustrates an embodiment of a post-processing device 600 and a post-processing method according to the invention for an Opus type codec.

[0169] As described in figure lb, the Opus codec (block 150) comprises an encoder (block 151) and a decoder (block 152). The encoder includes a high-pass pre-filter (HPF) at 160.

[0170] In this embodiment, the reference signal is obtained by applying a delay compensation at 1110 to the pre-filtered signal x hpf This reference signal is provided to the training module 602 to define a post-processing model to be applied to the output signal y in 603. Here, we consider that x hpf corresponds to the new samples for each frame, before taking into account the anticipation process, it is therefore important to compensate for the algorithmic delay by this preprocessing which delays the input (to synchronize it with the decoded output).

[0171] In variants of the invention with reference to Figures 7 to 11, the STFT-based transformation (203, 204) may be replaced by an alternative time / frequency decomposition, for example by using an MDCT transform instead of an STFT. In this case, the amplitude compression (204) is modified by replacing phase conservation with sign conservation, and the diffusion model in the complex spectrogram domain must be adapted to a real (signed) signal. Similarly, the normalization (blocks 201, 202) may not be implemented or applied directly in the frequency domain.

[0172] In other embodiments of the invention, as shown in Figures 7 to 11, the SGMSE diffusion model can be modified by other diffusion-based denoising methods without altering the principle of the invention. In particular, the resolution of the EDS can be changed, for example, using the Milstein method instead of the Euler-Maruyama method, and the NCNS++ network can be replaced with a DCUNET network or another network used for denoising.

[0173] Figure 12a shows a variant embodiment of the neural network architecture applicable to all previously described embodiments of a 600 post-processing device and a post-processing method according to the invention for an AMR-WB or other type codec.

[0174] In this variant, post-processing is dependent on the bitrate used during audio encoding (block 1200). The bitrate is considered a categorical variable. For example, for AMR-WB limited to 6.6, 8.85, and 12.65 kbit / s, there are three categories or classes. During the training phase, the bitrate information (a class among the number of considered bitrates) is transformed into a vector. emb belonging to a higher-dimensional space by means of a learned transformation, then injected into the network at each residual block, by component-by-component multiplication between two layers of the main branch of the residual block (left branch in Figure 12a). In order to guarantee the operation of the model in the absence of conditioning, the information contained in c embis randomly masked before being integrated into the network. During inference, the model receives as input the degraded audio as well as any information on the bitrate used during encoding.

[0175] Figure 12b shows an alternative embodiment of the neural network architecture applicable to all previously described embodiments of a 600 post-processing device and a post-processing method according to the invention for an AMR-WB or other type codec.

[0176] In this variant, post-processing is dependent on the bitrate used during audio encoding (block 1200). Here, the bitrate is considered a numerical variable (just like time t). During the training phase, the information concerning the bitrate and time is transformed into a global vector. embbelonging to a higher-dimensional space by means of a learned transformation, then injected into the network in each residual block, by component-by-component addition. The global vector emb This can be obtained in two ways. One way is to perform the transformation on the concatenation of the bitrate and time. Another way is to sum the results of the individual bitrate and time transformations. During inference, the model receives as input the degraded audio as well as any information about the bitrate used during encoding.

[0177] In some variations, the conditioning may be given by the type of content (pure speech, noisy speech, music).

[0178] According to the invention, context information is global, it influences the network in the same way on all time frames - for example a fixed bit rate and / or a type of content on the entire spectrogram.

[0179] Figure 13 illustrates a post-processing device 600, as defined in the invention. The device 600 comprises a processing circuit typically including:

[0180] - a MEM memory for storing instruction data of a computer program as defined in the invention;

[0181] - an INT interface for receiving a reference audio signal x' and a decoded output signal y' to implement a training phase;

[0182] - a PROC processor to receive these signals and process them by executing the computer program instructions stored in the MEM memory, in order to perform post-processing; in particular, the processor being capable of driving the processing modules 601, 602 and 603 as described with reference to Figure 6 and Figures 7 to 12; and

[0183] - a COM communication interface to transmit the post-processed signals, resulting from the post-processing, to a network.

[0184] Of course, this Figure 13 illustrates an example of a structural embodiment of a post-processing device as defined in the invention. Figures 6 to 12, discussed above, describe in detail functional embodiments of this device.

[0185] Applications of this type of post-processing can be found, for example, in audio decoding, such as when an encoded audio message is recorded (on a mail server or elsewhere) and needs to be decoded and played back to a user. This type of process can also be used in real-time conversations, provided that the complexity constraints of the broadcast model, as described, are taken into account. The invention can also be applied to improve SGMSE denoising by using context conditioning, where the context would be the noise level estimated by an external device.

Claims

1. DEMANDS 1. A method for post-processing a decoded audio signal, in which the post-processing (603) applies a post-processing model resulting from a training phase (602) carried out between a reference signal (x') and the decoded audio signal (y'), the reference signal being obtained by applying at least one pre-processing (601) to an input signal (x), a pre-processing being a processing from the following list: - compensation for delay; - phase compensation; - a pre-sharpening treatment; - a perceptual weighting treatment.

2. A method according to claim 1, wherein a preprocessing applied to the input signal is further a resampling process between a first and a second frequency.

3. A method according to any one of claims 1 or 2, wherein the decoded audio signal is the decoded audio signal before applying inverse processing to at least one preprocessor from the list.

4. A method according to any one of the preceding claims, wherein the post-processing model is a diffusion denoising model.

5. A method according to any one of claims 1 to 4, wherein the post-processing method uses a neural network for the training phase and for an inference phase corresponding to the application of the post-processing model.

6. A method according to any one of claims 1 to 4, wherein the post-processing model depends on a conditioning value.

7. Method according to claim 6, wherein the conditioning value is a coding rate value or a value indicating a type of signal.

8. Audio signal post-processing device comprising a post-processing module (603) applying a post-processing model to a decoded audio signal and a training module generating the post-processing model and applying a training phase (602) between a reference signal and the decoded audio signal, the reference signal being obtained by applying at least one pre-processing (601) to an input signal, a pre-processing being a processing from the following list: - compensation for delay; - phase compensation; - a pre-sharpening treatment; - a perceptual weighting treatment.

9. Audio codec comprising an encoding and decoding module and a post-processing device according to claim 8.

10. Processor-readable storage medium storing a computer program comprising instructions for carrying out the process according to any one of claims 1 to 7.

Images