Post-processing of an audio signal according to a post-processing model derived from optimized training

Abstract

Post-processing of an audio signal according to a post-processing model derived from optimized training. The invention relates to a method for post-processing a decoded audio signal, in which the post-processing (603) applies a post-processing model derived from a training phase (602) performed between a reference signal (x') obtained by applying at least one pre-processing (601) to an input signal (x) and a decoded output signal (y'). The invention also relates to a post-processing device implementing the method. Abstract figure: Figure 6

Description

Title of the invention: Post-processing of an audio signal according to a post-processing model derived from optimized training technical field

[0001] The present invention relates to the general field of audio signal processing. The invention relates in particular to the compression of audio signals – hereafter referred to as “audio coding”. Particular interest is given to the improvement of existing coded signals through post-processing (decoding).

[0002] This type of processing finds applications generally in the field of audio technologies, and more specifically in the field of audio decoding and rendering. Previous technique

[0003] Post-processing is a well-known technique for improving the quality of a decoded signal in a backward-compatible manner (without modifying the format of the bitstream or the transport format in question). In the case of speech coding, historical methods based on signal processing include, for example:

[0004] - Improvement of decoded speech by post-filtering (of formant or pitch: " forming postfilter", "pitch postfilter"...)

[0005] - Improvement of decoded music (by applying a frequency mask, inter-harmonic noise reduction...)

[0006] An example of post-processing - integrating several post-processing sub-modules - is found in the 3GPP EVS coded (for "Enhanced Voice Services" in English) and in particular in the decoding of the EVS AMR-WB IO mode which is essentially an enhanced version of the original AMR-WB coded (for "Adaptive Multi-Rate-Wideband" in English), and which is described in the 3GPP TS 26.445 specification.

[0007] With the advent of deep learning, new methods based on neural networks have been proposed for the post-processing of decoded signals, using tools such as a convolutional network, or a mask-based approach, or even with approaches using antagonistic neural networks (GANs).

[0008] More recently, an approach that will be referred to hereafter as "ScoreDec," proposing a diffusion-based post-processing method called SPF (for "score-based diffusion post-filter"), was presented in the article by Y.-C. Wu, D. Markovié, S. Krenn, L.D. Gebru, and A. Richard, "ScoreDec: A Phase-Preserving High-Fidelity Audio Coded with a Generalized Score-Based Diffusion Post-Filter," Proc. ICASSP, 2024. This approach operates in the complex spectrum domain and consists of essentially involves applying a diffusion model denoising method, as described in the article by J. Richter, S. Welker, J.-M. Lemercier, B. Lay, and T. Gerkmann, “Speech enhancement and dereverberation with diffusion-based generative models,” IEEE / ACM TASLP, vol. 31, pp. 2351-2364, 2023, to reduce the output encoding noise of existing AudioDec (before GAN training) or Opus 24 kbit / s encoded audio. Thus, the diffusion model is only trained for a given encoding / decoding rate (here, 24 kbit / s).

[0009] Figures 1a and 1b illustrate the operation of the method called "ScoreDec". The general principle is to apply denoising using a diffusion model – according to the SGMSE method (for "Score-based Generative Models for Speech Enhancement") described later – to the output of an existing coded audio file (AudioDec or Opus in the article by Y.-C. Wu et al.). The ScoreDec method thus amounts to combining an existing coded audio file with SGMSE denoising post-processing.

[0010] In Figure 1a, the AudioDec neural encoder (block 100) takes an input audio signal x, sampled, for example, at 48 kHz; this signal is analyzed by an encoder (block 101) with a final neural projection layer P (block 102) to obtain a latent space z which is then encoded (block 103) by residual vector quantization (multi-stage) and this discrete representation is decoded (block 104) to reconstruct the audio signal y. The AudioDec encoder—here in its autoencoder version and not its vocoder version—is normally trained in two phases: a phase 1 where the generator (block 100) is trained alone, then a phase 2 in a generator-anchored network (GAN) architecture with the addition of a discriminator providing an adversarial cost function. According to the article by Y.-C. Wu et al.As mentioned above, by adding post-processing (block 130), it is possible to implement only phase 1 of the training for block 100. The SPF post-processing (block 130) avoids the need for a GAN-type phase 2. Refer to the aforementioned article by Y.-C. Wu et al. for details of the training of block 130, with a training procedure illustrated here in block 120 and providing the zero parameters used in 130 for inference. Block 130 corresponds here to a pre-trained SGMSE diffusion model with zero weights. The training takes a sequence of examples (x,y) where x is the input of block 100 and y is the (decoded) output of block 104. Thus, the SGMSE denoising method is repurposed here to reduce the coding noise induced by block 100.

[0011] This SPF post-processing is also applied in the aforementioned article by Y.-C. Wu et al. to another codec, Opus, which is described in IETF RFC 6716 and IETF RFC 8251. In Figure 1b, the Opus codec (block 150), divided into an encoder (block 151) and a decoder (block 152), is followed by post-processing (block 180). As with AudioDec, block 170 is highlighted here, illustrating the training of the post-processing block.

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023] SPF is done using a series of examples (x,y), where x is the input of block 150 and y is the (decoded) output of block 152; thus, block 180 corresponds to a pre-trained SGMSE post-processing, using inference the 0 weights from block 170. It should be noted that the aforementioned article by Y.-C. Wu et al. does not explicitly state that the input x is used by block 170, however the results presented in this article in terms of scale-invariant signal-to-noise ratio (SI-SNR) allow a person skilled in the art to verify that this is indeed the case. Furthermore, in [Fig.lb] the high-pass pre-filter (noted HPF for high-pass filter) of Opus is explicitly highlighted here in 160, while the rest of the Opus coding is done in 161. This distinction is not defined in the article by Y.-C. Wu et al. cited above, however we will see later that this pre-filter block can make the SPF post-processing suboptimal if it is not properly taken into account. Figure [Fig. 2a] describes the operation of the SPF post-processing (which in fact corresponds exactly to the SGMSE denoising model according to the aforementioned article by J. Richter et al with the following adaptations, as used in the ScoreDec method to improve the quality of the decoded signals after the AudioDec or Opus encoding (blocks 130 or 180). It should be noted that the SGMSE model has several versions, and in some cases the name is modified; for example, the article by J.-E. Ayilo, M. Sadeghi, and R. Serizel, "Diffusion-based speech enhancement with a weighted generative-supervised learning loss," Proc. ICASSP, 2024, explicitly distinguishes it from a version called "SGMSE+." In this document, we will retain the generic name SGMSE. The degraded audio signal here corresponds to the decoded y signal (output of block 104 or 152). For the sake of brevity, the time indices n of the y(n) signal, sampled for example at 48 kHz, are not shown here. This decoded signal y is first normalized (block 202) by a scaling factor g given by the maximum of the absolute value of y (determined non-causally over the entire processed signal): Where the index n applies to the entire signal y of limited duration. Then, this normalized signal y,mrm is transformed by a 203 STFT type transform (for "short time Fourier Transform" in English) to obtain the complex spectrogram Y of size K x F, where K is the number of time frames of analysis and F is the number of frequency lines. This transformation corresponds to: Where NFFT is the length of the FFT (typically NFFT = 510), ù(n) is an analysis window (e.g., Hanning's) of length (typically Nwin ~ Nfft), hop cst 'c number of new samples analyzed in each frame (typically N hop — 128). Some details, such as the centering of the analysis window ^(n) relative to the input signal blocks, are not addressed here. For Nfft = 510, we obtain F = 256 frequency lines. Furthermore, for a 16 kHz signal with a length of 2.048 seconds, we have K = 256, which would give a spectrogram in the form of a square "image" with complex coefficients. In practice, the value of K will depend on the effective length of the signal to be processed.

[0024] The amplitude of Y is transformed by default by amplitude compression to 204 (in order to reduce the spectral dynamic range):

[0025] Ytrans(k, f) - fi \ Y(k, f)\Va£

[0026] Where the value of fi is for example fi = 0.15, k is the transformed frame index, f is the frequency line index and for example « = 1 / 2. Here again, for the sake of brevity, the indices k and f of the spectrograms Y, ytrans^ are not shown in general except for the equation below where these details are relevant to remove any ambiguity.

[0027] Finally, the Ytruns transformed spectrum is denoised by sampling a diffusion model at 205 (inference phase) to obtain the signal which undergoes the inverse transformation (block 206) at block 204: LJX(k,f)=\X (kf)

[0029] The resulting spectrogram x is transformed in time into 207 to obtain y'™™ which is scaled back (block 208) by the factor g from block 201.

[0030] The denoising module in the STFT spectrogram domain at 205 can be implemented according to the article by J. Richter et al., but it can also be implemented according to the article by J.-E. Ayilo et al., which extends the learning criterion of the "score" (in generative mode) to a mean squared error cost function on the signal (in supervised mode). A description of block 205 according to J. Richter et al. is now given in relation to the backward diffusion process.

[0031] The SGMSE denoising model used by the ScoreDec method is described in particular in the aforementioned article by J. Richter et al. It is a generative speech enhancement model based on diffusion. Given a degraded speech signal, this model generates a "clean" speech sample, corresponding to a denoised version of the original signal. Unlike some neural approaches that seek to learn a correspondence between noisy and "clean" speech, the SGMSE model learns to generate a "clean" speech sample through successive denoising operations from Gaussian noise, conditioning this generation on the noisy signal. The idea behind the SGMSE model is as follows.

[0032] The starting point for training an SGMSE neural model is to create a database containing pure speech signals and their degraded versions. In denoising, the degraded version is obtained by adding noise at a certain signal-to-noise ratio; in the aforementioned article by J. Richter et al., the audio database is public and corresponds to VoiceBank-DEMAND (at 16 kHz). In the context of ScoreDec, in the article by Y.-C. Wu et al., pure speech corresponds to the public database "Valentini," while the degraded version is obtained by encoding and decoding using AudioDec or Opus.

[0033] The samples from the audio database thus created (pure speech, degraded version) are represented in the frequency domain after a complex short-term discrete Fourier transform (STFT). Working in this domain allows for the simultaneous improvement of the amplitude and phase spectra.

[0034] The pure speech spectrograms X, hereafter denoted xo, are progressively noisy during a process (called "forward") which transforms them into a highly corrupted version of the corresponding noisy data, denoted J. This process is parameterized by a continuous time variable t and governed by a stochastic differential equation (SDE). xt denotes the pure speech spectrogram at time t, following the "forward" process.

[0035] Theoretical results assure that it is possible to reverse the previous process, that is, starting from a noisy speech signal, to return to a pure speech signal. This process (called "backward") admits an explicit expression in the form of an EDS.

[0036] The "forward" process is now described in detail. We work with complex spectrograms, generically denoted here as xt, indexed by the time variable t, which evolves in the interval [0, T] where t=0 is the initial time (when the speech signal is pure) and t=T is the final time (when the speech signal is buried in noise). Thus, x0 denotes a "pure" spectrogram, an element of the training set, and xt the degraded spectrogram at the end of the "forward" diffusion process. The "forward" process follows the EDS

[0037] dxt = f(xt, y)dt + g(t)dw,

[0038] where f is a drift term governing the average behavior of the equation, given by the expression

[0039] f(x,y) = y(yx),

[0040] with >0 a stiffness coefficient, is a diffusion term controlling the amount of noise injected into the equation at each instant, w is a Wiener process (or Brownian motion), at the origin of the stochastic character of the equation.

[0041] The drift term leads x₀ towards x₀ with a velocity controlled by the stiffness parameter. The noise added by diffusion leads x₀ towards a highly corrupted version x₀ at the final time.

[0042] The interval [0,T] is discretized into N subintervals of length AT, and the backward equation is solved between t=T and t=tE, where 1, -0, using numerical methods such as the Euler-Maruyama method. Limiting the solution to tE avoids numerical instabilities that can occur for t close to 0. For example, T=1 and N=30 can be taken.

[0043] Several numerical methods for solving stochastic differential equations (SDEs) can be used during the "forward" phase, the most common being the Euler-Maruyama method, an extension of Euler's method for ordinary differential equations.

[0044] Figure 2b illustrates the NCSN++ (Noise Conditional Score Network) architecture used by default by the SGMSE method during training and inference. The NCSN++ architecture was introduced in the aforementioned article by Y. Song et al. The architecture has been adapted to handle complex spectrograms: the real and imaginary parts are treated as separate channels. Thus, the estimation of the score at the network output is performed separately for the real and imaginary parts.

[0045] The neural network has a multi-resolution "Unet" structure with different types of layers (Conv2D, DownLayer, UpLayer, BottleneckLayer, ProgDown and ProgUp). Further details on the structure of this network are given in the article by J. Richter et al. (section IV. "Network architecture").

[0046] Figure 2c details the structure of the residual blocks present in the NCSN++ architecture during each subsampling / oversampling. A residual block includes Conv2D layers in 283, 287 and 289, group normalization (according to the paper by Y. Wu and K. He, “Group normalization,” Proc. ECCV, 2018) in 280 and 285, suret subsampling by finite impulse response filter (according to the paper by R. Zhang, “Making convolutional networks shift-invariant again,” Proc. ICML, 2019) in 282 and 288, and a Swish activation function (according to the paper by P. Ramachandran, B. Zoph, and QV Le, “Swish: a self-gated activation function,” arXiv preprint arXiv:1710.05941, 2017) in 281 and 286.To make the model dependent on time t, information about this time variable (indicating the progress in the diffusion process) is integrated into each residual block, by means of a Fourier embedding which maps the variable t to a vector (block 284). The information from the two branches is summed (block 290) and then normalized by a factor of 2 (block 291).

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] Figure 3 illustrates the forward scattering process on the interval [0,T] discretized into N subintervals of length AT where N=30. It shows an example of a scattering process applied to a complex spectrogram X = xo. Only the amplitude spectrogram is shown here. The pure speech spectrogram xo is progressively noisy by EDS as implemented in the SGMSE method for T=l, the result after a few iterations (vvrvv Y A la end of the "forward" process (t=T), the signal xr has drifted towards Y but the result is almost totally drowned out by noise (only a few energetic components of pure speech can be distinguished). The "backward" process aims to solve the inverse SDE, and in the article by Richter et al., this solution is performed using the Predictor-Corrector sampling procedure introduced in the article by Y. Song, J. Sohl-Dickstein, DP Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations," Proc. ICLR, 2021, to iteratively generate the denoised signal estimate. The inverse SDE is given by: dxt = - [f(y) + g(f) 2V Xtlogpt(x)y)] dt + Where W is the Wiener process reversed in time. This inverse SDE involves a coefficient that is unknown a priori – the score (which is an approximation of the derivative of the conditional log-probability Xlogp[x^y^' depending on the conditional distribution notc yt \ 'c score approximate evaluated at a point xt at time t, where © represents the parameters of the approximation (weight of a neural network). Using a neural network, the SGMSE model learns the score via a function yt)' °where ® denotes the network weights. It is then possible to approximately solve the SDE of the "backward" process by replacing the score with its approximate value. Several neural network architectures, built around the U-Net architecture, have been proposed for SGMSE, for example DCUNET for "Deep Complex U-Net" or NCNS for "Noise Conditional Score Network".

[0054] Here again, several numerical methods for solving EDS can be used during the "backward" phase, the most common being the Euler-Maruyama method.

[0055] Blocks 120 and 170 of Figures 1a and 1b correspond to the training of the SPF post-processing. To do this, the pure / degraded signals x(n) and y(n) are both normalized (for example, by taking the maximum value of the amplitude of y(n) to normalize both x(n) and y(n) by the same factor) and transformed in the domain frequency as in blocks 203 and 204 illustrated in [Fig.2a], to obtain the pure / degraded spectrograms.

[0056] At each training stage, a time t is first sampled according to a uniform distribution over the interval [0, 1], then a pair yj of pure / degraded spectrograms is randomly chosen from the database. Knowing and 3, the distribution of xt can be determined explicitly: one can therefore directly sample xt and calculate the corresponding score function. A distance 12 (squared error) between the score model and the score is then calculated, and the neural network parameters are updated.

[0057] The ScoreDec method gives very good results for improving the AudioDec codec limited to phase 1 of training (without the GAN architecture of phase 2). In the aforementioned article by YC Wu et al., the bitrate of AudioDec and Opus is set at 24 kb / s (for signals sampled at 48 kHz). It is known to those skilled in the art that Opus already achieves good quality for speech at 24 kb / s; the improvement provided by the SPF post-processing (block 170) is less pronounced for Opus than for AudioDec.

[0058] The ScoreDec method treats the upstream AudioDec and Opus coded formats as "black boxes." This simplification is acceptable for AudioDec, which is an end-to-end neural coded format without "classical" signal processing and with virtually no algorithmic delay implementation. On the other hand, it is well known to those skilled in the art that the Opus coded format uses a high-pass pre-filter (HPF) in block 160, introducing a phase shift on the input signal, and that the Opus coded format causes an algorithmic delay (due to the "lookahead" anticipation process and internal filtering in block 161) which is 6 by default.5 ms (312 samples at 48 kHz); however, these two types of delay are not taken into account in the ScoreDec method, which means that the SPF post-processing (block 170) will actually result in a phase advance (of 312 samples) and model the phase response of the HPF filter (block 161), whereas these two operations are actually undesirable (one can create problems for real-time processing and remains a non-causal operation, the other is not "perceptually" justified).

[0059] Furthermore, the ScoreDec method is not applied to existing widely deployed speech codes, such as 3GPP AMR (for "Adaptive Multi-Rate" in English) or 3GPP AMR-WB (for "Adaptive Multi-Rate-Wideband" in English).

[0060] Furthermore, the ScoreDec method is trained in the aforementioned article by Y.-C. Wu et al. at a single bit rate (24 kbit / s). It would be advantageous to have a method applicable to a multi-bit rate encoded code such as AMR-WB.

[0061] Furthermore, ScoreDec is typically trained and tested only on pure speech; it would be advantageous to have a method adapted to different types of content (pure speech, music, noisy speech, etc.).

[0062] We now recall the structure of examples of "traditional" coded such as AMR-WB or its improved version in EVS (as EVS AMR-WB 10 mode).

[0063] The AMR-WB coded speech is described for example in the article by B. Bessette et al., "The adaptive multirate wideband speech coded (AMR-WB)," in IEEE Transactions on Speech and Audio Processing, vol. 10, no. 8, pp. 620-636, Nov. 2002.

[0064] Figures 4a and 4b describe the simplified architecture of the AMR-WB encoder and decoder respectively. This encoding operates at 9 rates called "modes" (6.6, 8.85, 12.65, 14.25, 15.85, 18.25, 19.85, 23.05 and 23.85 kbit / s).

[0065] Figure 4a describes the simplified architecture of the AMR-WB encoder. The input signal, sampled at 16 kHz, is generally a linear PCM (Plug Code Modulation) input truncated to 14 bits according to the 3GPP TS 26.171 standard. For the sake of simplicity, Figures 4a and 4b do not show the non-encoder / decoder blocks, such as Voice Activity Detection (VAD), Discontinuous Transmission (DTX), or Comfort Noise Generation (CNG).

[0066] The input signal is decimated at 12.8 kHz (block 401), then pre-filtered by a Butterworth high-pass filter with a cutoff at 31 Hz (block 402), and finally pre-emphasized by the L1 filter (block 403) where a ≠ 0.68. The pre-emphasized low-band signal (0-6.4 kHz) is analyzed by linear prediction (LP), the LPC coefficients are quantized and interpolated in μ per subframe (block 404). An open-loop pitch analysis (OL) is then performed (block 406) in the domain of the signal perceptually weighted by the filter (block 405):

[0067] 4 / „) W(z)

[0068] Where Pj = 0.92. Next, the closed-loop search of the adaptive dictionary (block 407) and the fixed dictionary (block 408) is performed, before joint encoding of the dictionary gains (block 409). When the encoded system is operating at 23.85 kbit / s, high-band gain parameters (6.4–7 kHz) are also determined and encoded (block 420). All parameters encoded in the current frame are multiplexed (block 410).

[0069] Fig. 4b describes the simplified architecture of the AMR-WB decoder.

[0070] During decoding, the parameters are demultiplexed (block 460), the LPC parameters are decoded and interpolated (block 452), the excitation combining the contributions of the fixed and adaptive dictionaries is decoded (451), before applying the LPC synthesis filter (block 454) per subframe, then the synthesis is de-emphasized (block 455) and resampled to 16 kHz (block 456). The decoded low-band signal is combined (block 458) with the reconstructed high-band signal dW] (at block 457) - if the HF The decoder operates at 23.85 kbit / s, the high band correction gains are decoded (block 453), otherwise the high band scaling factors are estimated "blindly" for all bit rates from 6.6 to 23.05 kbit / s.

[0071] Figure 5 describes the simplified architecture of the EVS encoder and decoder (including AMR-WB 10 mode).

[0072] Block 501 comprises various preprocessing sub-blocks (such as, for example, a high-pass prefilter "HP Filt.", a filter bank and resampling "FB&res.", a pre-emphasis module "pre-emp", a classification module "class.", a linear prediction analysis module "LP Ana.", etc.). The various preprocessing sub-blocks are not detailed here. Similarly, various post-processing sub-blocks are provided in block 508 (such as, for example, a high-pass postfilter "HP Filt.", a filter bank and resampling "FB&res.", a de-emphasis module "de-emp", a formant postfilter "form. Post Filt.", a music enhancement module "music Enh.", etc.). These are not detailed further here. Block 502 concerns the encoding of the "EVS Primary" mode, and block 506 concerns the decoding of this mode. The sub-blocks that constitute them ("switch.", "LP-cod." or "LP-decod.", "MDCT cod." or "MDCT decod.", "DTX, CNG cod." or "DTX, CNG decod."" are not detailed here. Similarly, the transmitter (504) and the receiver at 505 are ignored here.

[0073] Block 503 relates to the encoding of the "AMR-WB IO" mode, block 507 relates to the decoding of this mode.

[0074] Unlike AMR-WB, a high-pass pre-filter (called an "HP filter") of the " Butterworth, with a 20 Hz cutoff, is applied in 501 directly to the input signal (before resampling to 12.8 kHz for AMR-WB IO mode). It should also be noted that numerous 508 post-processing blocks are defined in the EVS decoder.

[0075] There is a need to be able to apply post-processing using neural networks (for example, of the SGMSE type) to existing CELP-type codes such as AMR-WB or other types such as Opus, and in an optimized manner. There is a need to be able to apply post-processing using neural networks (for example, of the SGMSE type) that is adapted to multi-bitrate cases and to different types of content (speech, music, etc.). Description of the invention

[0076] The invention improves upon the state of the art.

[0077] To this end, the invention relates to a method for post-processing a decoded audio signal, in which the post-processing applies a post-processing model derived from a phase training performed between a reference signal obtained by applying at least one preprocessing to an input signal and a decoded output signal.

[0078] Thus, unlike state-of-the-art methods where the training phase is applied between the input signal of the coded signal, without modification, and the output signal, the proposed post-processing method applies preprocessing to the input signal for the training phase. This makes it possible to exploit knowledge of the algorithmic delay and / or the existing internal structure of the coded signal to obtain a suitable reference signal and, consequently, a suitable preprocessing.

[0079] In one embodiment, a preprocessing applied to the input signal is a delay compensation.

[0080] Thus, in the case of a coded signal with an algorithmic delay, such as the Opus coded signal, the delay is compensated during the training phase, resulting in a suitable post-processing model because the input and output signals are optimally synchronized. Furthermore, this prevents the post-processing from resulting in a phase lead, which is undesirable in real-time operation when processing successive blocks of output signals.

[0081] A preprocessing applied to the input signal can also be further phase compensation.

[0082] Thus, in the case of a coded signal where a high-pass prefilter is provided and applied to the input signal, training applied to the input signal of the coded signal, without taking this prefilter into account, is not optimal since it models the phase response of the prefilter, whereas the prefilter has virtually no perceptual effect but typically serves to suppress or reduce the DC component that would introduce bias in internal analyses of the coded signal. Taking this prefiltering into account through phase compensation preprocessing makes it possible to adapt the training phase to this type of coded signal.

[0083] According to several embodiments, a preprocessing applied to the input signal for the training phase is a processing from the following list: - a resampling process between a first and a second frequency; - a pre-sharpening treatment; - a perceptual weighting treatment; and the decoded output signal is the signal before applying inverse processing to at least one preprocessor from the list, the post-processing being applied to the decoded signal before applying inverse processing to at least one preprocessor from the list.

[0084] Thus, post-processed coded signals are no longer considered black boxes. The internal structure of existing coded signals is exploited here to optimize the training, making it possible to obtain a post-processing model adapted to the coded signal and thus improve the quality of the decoded signal, since the post-processing focuses on reducing the coding noise induced by the coded signal.

[0085] The reference signal used for the training phase is taken after an existing preprocessing in the coded processing chain and the output signal used for the training phase is taken before the inverse processing of the preprocessing in the coded processing chain.

[0086] In one embodiment, the post-processing model is a diffusion denoising model.

[0087] The diffusion denoising model is adapted here by incorporating preprocessing to obtain a reference signal for the training phase. The denoising model may be of the SGMSE type with an NCSN++, DCUNET, or another model variant architecture, with different configurations of solvers, EDS resolution, etc., as described later.

[0088] According to one embodiment, the post-processing method uses a neural network for the training phase and for an inference phase corresponding to the application of the post-processing model.

[0089] According to a particular embodiment, the post-processing model depends on a conditioning value.

[0090] In exemplary embodiments, the conditioning value is a coding rate value or a value indicating a signal type.

[0091] Thus the post-processing model is best adapted to the characteristics of the encoder or the signal

[0092] The invention relates to an audio signal post-processing device comprising a post-processing module applying a post-processing model to a decoded audio signal and a training module generating the post-processing model and applying a training phase between a reference signal obtained by applying at least one pre-processing to an input signal and a decoded output signal.

[0093] This post-processing device includes a processing circuit for implementing the steps of the post-processing process as described above.

[0094] The invention relates to a computer program comprising instructions for implementing the post-processing method as described above, when executed by a processor.

[0095] Finally, the invention relates to a storage medium, readable by a processor, storing a computer program containing instructions for the execution of the post-processing method described above. Brief description of the drawings

[0096] Other features and advantages of the invention will become more apparent upon reading the following description of particular embodiments, given by way of simple illustrative and non-limiting examples, and the accompanying drawings, among which:

[0097] [Fig.la] and [Fig.lb] illustrate the state-of-the-art ScoreDec method, as described above, for improving the signal from an AudioDec coded and an Opus coded respectively;

[0098] [Fig.2a] illustrates the SGMSE method (for "Score-based Generative Models for Speech Enhancement » of the state of the art, as described previously;

[0099] [Fig.2b] illustrates the architecture of a state-of-the-art NCSN++ neural network, as described previously;

[0100] [Fig.2c] illustrates a residual block of the state-of-the-art NSCN++ architecture, such as described previously;

[0101] [Fig.3] illustrates the effects on the signal of a diffusion process according to the state of the art;

[0102] [Fig.4a] illustrates the simplified architecture of an AMR-WB (for "Adaptive") encoder Multi-Rate-Wideband (in English);

[0103] [Fig.4b] illustrates the simplified architecture of an AMR-WB decoder;

[0104] [Fig.5] illustrates the simplified architecture of an EVS (encoder and decoder) coded (for « Enhanced Voice services (in English);

[0105] [Fig.6] illustrates a post-processing device and a post-processing method in a general implementation of the invention;

[0106] [Fig.7a] illustrates a first embodiment of a post-processing device and of a post-processing method according to the invention for an AMR WB type coded;

[0107] [Fig.7b] illustrates a second embodiment of a post-processing device and of a post-processing method according to the invention for an AMR WB type coded;

[0108] [Fig.8] illustrates a third embodiment of a post-processing device and of a post-processing method according to the invention for an AMR WB type coded;

[0109] [Fig.9] illustrates a fourth embodiment of a post-processing device and of a post-processing method according to the invention for an AMR WB type coded;

[0110] [Fig. 10] illustrates a fifth embodiment of a post-processing device and a post-processing method according to the invention for an AMR WB type coded;

[0111] [Fig. 11] illustrates an embodiment of a post-processing device and a post-processing method according to the invention for an Opus type coded;

[0112] [Fig. 12a] illustrates an embodiment of a context conditioning device and a context conditioning method, with a residual block of the NSCN++ architecture modified according to the invention;

[0113] [Fig. 12b] illustrates an alternative embodiment of a context conditioning device and a context conditioning method, with a residual block of the NSCN++ architecture modified according to the invention;

[0114] [Fig. 13] illustrates an example of a structural realization of a post-processing device according to an embodiment of the invention. Description of the implementation methods

[0115] Figure 6 illustrates a post-processing device and a post-processing method in a general embodiment of the invention.

[0116] A preprocessing module 601 applies at least one preprocessing step to the input signal x, for example, the input signal of a coded signal. This preprocessing module may be part of the postprocessing device 600 (module 601 within the device 600), which is then added to an existing coded signal. The preprocessing module 601 (outside the device 600) may also be composed of the existing preprocessing modules within the coded signal to which the postprocessing device is applied.

[0117] A preprocessing applied by this module is, for example, a preprocessing of the following list:

[0118] - a compensation for delay; - phase compensation; - a high-pass type filter; - a resampling process between a first and a second frequency; - a pre-sharpening treatment; - a perceptual weighting treatment.

[0119] The post-processing device 600 thus obtains a pre-processed signal x' which serves as a reference signal for a training phase operated by the training module 602. This training module implements a training phase between this reference signal and a decoded output signal y'. This training module implements, for example, a training phase using a "forward" and "backward" diffusion process as described above and illustrated in [Fig. 3].

[0120] At the end of this training phase, a post-processing model defined by parameters 0 is obtained. These parameters are sent to the post-processing module 603, which applies this model to the decoded output signal y'. The application of this post-processing model constitutes an inference phase. The post-processing module 603 is, for example, as described in [Fig. 2a].

[0121] Depending on the preprocessing performed by the 601 module, the decoded output signal can be the signal before application of inverse processing to the preprocessing performed, the post-processing being applied to the same decoded signal.

[0122] Several particular embodiments, applied to existing coded systems, are now described.

[0123] Fig. 7a shows a first embodiment of a post-processing device 600 and a post-processing method according to the invention for an AMR-WB type coded.

[0124] In the embodiment of [Fig.7a], we find coding blocks of [Fig.4a] (400, 401, 402, 403 and 420) and decoding blocks of [Fig.4b] (450, 454, 455, 456, 457, 458) - blocks 400 and 450 are not detailed here for convenience to make the diagram more compact.

[0125] A post-processing device 600, as described with reference to [Fig. 6], is added to the processing chain of this coded device. It includes the training block 602 and the post-processing block 603. Blocks 710 and 711 constitute the pre-processing module applied to the input signal.

[0126] Block 603 is a diffusion model post-processing (in inference, with pre-trained weights) applied to the output of the 16 kHz encoded.

[0127] Block 602 corresponds to the drive of the diffusion model; according to the invention, the drive takes as a reference signal (“clean”) the signal sampled at 16 kHz, filtered in 710 and delayed in 711 by D samples, and as a degraded signal (“noisy”) the signal j[16k|.

[0128] Two preprocessing steps are applied here to the input signal of the encoded, to obtain the reference signal of the training phase, a phase compensation and a delay compensation.

[0129] The delay compensation by D samples is important here because x[16A] is considered to correspond here to the new samples for each frame, before taking into account the "lookahead" anticipation process.

[0130] The delay D is, for example, 95 samples for AMR-WB. The phase compensation aims to reproduce at 16 kHz the response of block 402, which operates at 12.8 kHz. In a variant, block 710 is implemented as a second-order Butterworth IIR filter in the form:

[0131] „ , s "HP_ 16kHZ U 7 ~ ir ^-2"

[0132] With = / a- 0.99142886, = -1.98285771 and = -1.98278425 and «2 = 0.98293118. This example filter is obtained, for example, with the Python pseudocode:

[0133] import scipy.signal as signal

[0134] sos = signal.butter(2, 31 / 8000, 'highpass', output='sos')

[0135] b=sos[0,0:3]

[0136] a=sos[0,3:]

[0137] In variants, different values ​​of coefficients are also possible.

[0138] Blocks 710 and 711 can be interchanged and / or combined into a single treatment.

[0139] In another variant, block 711 is actually implemented in block 602, after an STFT transformation and by applying the filter response to the spectrum of the reference signal at the output of 710. In this variant, the STFT transformation of the shifted input is still done in 602, but block 710 is not applied on x^6^h)> it is implemented in 602 in the frequency domain.

[0140] Fig. 7b shows a second embodiment of a post-processing device 600 and a post-processing method according to the invention for an AMR-WB type coded.

[0141] As with [Fig.7a], we find coding blocks from [Fig.4a] (400, 401, 402, 403 and 420) and the decoding blocks of [Fig.4b] (450, 454, 455, 456, 457, 458).

[0142] A post-processing device 600, as described with reference to [Fig. 6], is added to the processing chain of this coded device. It includes the training block 602 and the post-processing block 603.

[0143] In this embodiment, the pre-processing applied to obtain the reference signal for the training phase in 602 is different from that carried out in [Fig.7a]. Several pre-processing steps are performed.

[0144] The xhpf signal is shifted by 713, with D2=76 samples at 12.8 kHz. Optionally, the x^6^ signal is shifted by 712 with D=95 samples at 16 kHz for AMR-WB, or the high-band xhf signal (obtained by high-pass filtering on the 6-7 kHz band) is shifted by D=95 samples at 16 kHz.

[0145] The delay compensation for xhpf and xhf is important here because X^pf is considered to correspond to the new samples for each frame, before the lookahead anticipation process is taken into account. If these signals are taken at the level of the current frame encoded after the lookahead anticipation process, the delay in 713 and 714 is not necessary.

[0146] Next, two STFT analyses are performed and combined in 602, with windows defined in the same way and time-synchronized, in order to obtain: • The 0-fc Hz band spectrum of xltp^ shifted (sampled at 12.8 kHz) and the xhf shifted spectrum (sampled at 16 kHz) for the high band part (after bandpass fc-7 kHz or high pass cutoff at fc kHz), where fc is for example fixed at fc=6 or 6.4 kHz. • The spectrum of the band 0 at fc kHz of xhpf shifted (sampled at 12.8 kHz) and the spectrum of the band fc at 8 kHz of xt16^ shifted (sampled at 16 kHz) for the high band part, where fc is for example fixed at fc=1 kHz.

[0147] In both cases, this allows the frequency response of the 402 filter to be taken into account. The use of xhf (high-band signal obtained by 6-7 kHz bandpass filtering of the original signal xi16*) has the disadvantage of being impacted by the The frequency response of the high-band extraction filter in AMR-WB, therefore, is not optimal. The 6.4 kHz injunction is thus preferable. It is therefore better to use the variant where the spectra of xhpf and are combined into a single input spectrum of 602.

[0148] Thus, the input corresponding to the reference signal in 602 is here a mixture between a 12.8 kHz signal (after the following preprocessings: resampling in 401, HP in 402 and delay compensation in 713) and a signal reflecting the high band and resulting from the preprocessing in 714. The post-processing block 603 operates here at 16 kHz.

[0149] Figure 8 shows a third embodiment of a post-processing device 600 and a post-processing method according to the invention for an AMR-WB type coded.

[0150] This figure therefore illustrates a variant of an audio signal processing chain with post-processing integrated into the AMR-WB encoding and decoding. In this variant, the post-processing does not operate on the 16 kHz decoding output signal, but only on the 12.8 kHz low band.

[0151] According to this embodiment, the 602 training takes the x1^ signal as its reference input and the J' signal as its degraded input (in both cases, successive "current" frames are used). The x^f signal is thus obtained after resampling preprocessing in 401 and high-pass (HP) filtering in 402. The y signal, taken as the degraded signal for the training phase, is an output signal before the application of the inverse processing to the resampling preprocessing performed in 401. Since the input of the 602 training module is taken after 402 filtering, the two signals xhpf and y are "coherent," and no prefilter response compensation is necessary. Furthermore, no delay compensation is necessary because xhpf is considered here to correspond to the current encoded frame.

[0152] For this variant, only the low band (0-6.4 kHz) is improved.

[0153] Figure 9 shows a fourth embodiment of a post-processing device 600 and a post-processing method according to the invention for an AMR-WB type coded.

[0154] In this variant, the post-processing does not operate on the 16 kHz decoding output signal, but only on the 12.8 kHz low band, as in the previous embodiment. This time, the post-processing is applied in the pre-emphasized signal domain, that is, before the implementation of the deemphasis processing, also called de-emphasis.

[0155] According to this embodiment, the 602 training takes the xpre signal as its reference input and the ypre signal as its degraded input (in both cases, successive "current" frames are taken). The xp,e signal is thus obtained after preprocessing by resampling in 401, high-pass filtering (HP) in 402, and

[0156]

[0157]

[0158]

[0159]

[0160]

[0161]

[0162]

[0163]

[0164] Pre-emphasis, also called pre-stressing, is present in channel 403. The ypre signal, taken as the degraded signal for the training phase, is an output signal before the application of inverse processing steps 455 and 456, which are respectively the pre-emphasis pre-processing in channel 403 and resampling in channel 401. Since the input to the training module 602 is taken after filtering in channel 402, the two signals xp,e and ypre are "coherent," and no pre-filter response compensation is necessary. Furthermore, no delay compensation is necessary because xpre is considered to correspond to the current encoded frame. For this variant, only the low band (0-6.4 kHz) is improved. Figure 10 shows a fifth embodiment of a post-processing device 600 and a post-processing method according to the invention for an AMR-WB type coded. In this variant, post-processing does not operate on the 16 kHz decoding output signal, but only on the 12.8 kHz low band. This time, post-processing is applied in the perceptually weighted signal domain, that is, after filtering by jy^) in 1011 and before the implementation of the inverse filter W 1(^) in 1012. Here the filter yy(j) is defined as Azlr} W(z) = where is the decoded LPC filter in each 5ms subframe. According to this embodiment, the training module in 602 takes as its input the xpre signal as the reference signal, to which a perceptual weighting filter W(z) is applied in 1010, and as its degraded signal the ypre signal, to which a perceptual weighting filter |y(^) is applied in 1011. The reference input signal of the training module 602 is thus obtained after preprocessing with resampling in 401, high-pass filtering (HP) in 402, pre-emphasis in 403, and perceptual weighting in 1010. The signal taken as the degraded signal for the training phase is a perceptually weighted output signal before the application of the inverse processing steps 1012, 455, and 456, which respectively preprocess the perceptual weighting (1010 and 1011), pre-emphasis in 403, and resampling in 401. The advantage of this embodiment is that it is in a signal domain consistent with low-band CELP coding, where the coding noise is much less colored. For this variant, only the low band (0-6.4 kHz) is improved. In variants of the invention with reference to figures 7 to 10, the AMR-WB decoder (block 450 to 458) can be replaced by an EVS AMR-WB IO decoder (507), possibly with the deactivation of all or part of the post-processing blocks. defined in AMR-WB 10 (508), except for necessary blocks such as comfort noise generation, de-emphasis, resampling.

[0165] Indeed, in the same way as for the embodiments described above for the AMR-WB coded, the post-processing device 600 according to the invention can be added to the processing chain of the EVS coded as illustrated in [Fig. 5]. One or more pre-processing steps included in module 501 can be applied to the input signal to obtain the reference signal for the drive phase of the post-processing device.

[0166] Thus, as with the examples described for the AMR-WB encoded signal, training can be performed in a pre-filtered, pre-emphasized, or perceptually weighted signal domain. The output signal taken as the degraded signal for the training phase is the signal before the inverse preprocessing is applied.

[0167] The post-processing thus performed for the EVS encoded improves the quality of the output signal to such an extent that it is possible to deactivate certain post-processing steps provided in block 508 of [Fig. 5] (such as, for example: "music enhancer", "bass postfilter", "formant postfilter", "LTP postfilter"). Indeed, these blocks were optimized on an ad hoc basis, whereas the post-processing according to the invention is optimized end-to-end and can replace the combination of several of these blocks.

[0168] Fig. 11 illustrates an embodiment of a post-processing device 600 and a post-processing method according to the invention for an Opus type coded.

[0169] As described in [Fig.lb], the coded Opus (block 150) comprises an encoder (block 151) and a decoder (block 152). The encoder includes a high-pass pre-filter (HPF) at 160.

[0170] In this embodiment, the reference signal is obtained by applying a delay compensation in 1110 to the pre-filtered signal. This reference signal is provided to the training module 602 to define a post-processing model to be applied to the output signal y in 603. Here, ^Pf is considered to correspond to the new samples for each frame, before taking into account the anticipation process; it is therefore important to compensate for the algorithmic delay by this pre-processing which delays the input (to synchronize it with the decoded output).

[0171] In variants of the invention with reference to Figures 7 to 11, the STFT-based transformation (203, 204) may be replaced by an alternative time / frequency decomposition, for example by taking an MDCT transform instead of an STFT – in this case the amplitude compression (204) is modified by replacing phase conservation with sign conservation, and the diffusion model in the complex spectrogram domain must be adapted to a real (signed) signal. Similarly the standardization (block 201, 202) may not be implemented or applied directly in the frequency domain.

[0172] In other embodiments of the invention with reference to Figures 7 to 11, the SGMSE diffusion model can be modified by other diffusion-based denoising methods without changing the principle of the invention. In particular, the resolution of the EDS can be changed, for example with the "Milstein" method instead of the Euler-Maruyama method, and the NCNS++ network can be changed to a "DCUNET" network or another network used for denoising.

[0173] Fig. 12a shows a variant embodiment of the neural network architecture applicable to all previously described embodiments of a 600 post-processing device and a post-processing method according to the invention for an AMR-WB type coded or other.

[0174] In this variant, the post-processing is conditioned by the bitrate used during audio encoding (block 1200). The bitrate is considered here as a categorical variable. For example, for AMR-WB limited to 6.6, 8.85, and 12.65 kbit / s, there are 3 categories or classes. During the training phase, the bitrate information c (one class among the number of bitrates considered) is transformed into a vector cemb belonging to a higher-dimensional space by means of a learned transformation, and then injected into the network at each residual block by component-wise multiplication between two layers of the main branch of the residual block (left branch in Figure 12a). To ensure the model functions in the absence of conditioning, the information contained in c^nb is randomly masked before being integrated into the network.During inference, the model receives as input the degraded audio as well as any information about the bitrate used during encoding.

[0175] Fig. 12b shows a variant embodiment of the neural network architecture applicable to all previously described embodiments of a 600 post-processing device and a post-processing method according to the invention for an AMR-WB type coded or other.

[0176] In this variant, the post-processing is conditioned by the bitrate used during audio encoding (block 1200). Here, the bitrate is considered a numerical variable (just like time t). During the training phase, the information concerning the bitrate and time is transformed into a global vector belonging to a higher-dimensional space by means of a learned transformation, and then injected into the network in each residual block, by component-by-component addition. The global vector can be obtained in two ways. One way is to perform the transformation on the concatenation of the bitrate and the time. Another way is to sum the results of the individual transformations of the bitrate and time. During inference, the model receives as input the degraded audio as well as any information about the bitrate used during encoding.

[0177] In variants the conditioning may be given by the type of content (pure speech, noisy speech, music).

[0178] According to the invention, the context information is global, it influences the network in the same way on all time frames - for example a fixed bit rate and / or a type of content over the entire spectrogram.

[0179] A post-processing device 600, according to the invention, has been illustrated in [Fig. 13].

[0180] The device 600 includes a processing circuit typically including: - a MEM memory for storing instruction data of a computer program as defined in the invention; - an INT interface for receiving a reference audio signal x' and a decoded output signal y' to implement a training phase; - a PROC processor to receive these signals and process them by executing the computer program instructions stored in the MEM memory, in order to perform post-processing; in particular, the processor being capable of driving the processing modules 601, 602 and 603 as described with reference to [Fig. 6] and Figures 7 to 12; and - a COM communication interface to transmit the post-processed signals, resulting from the post-processing, to a network.

[0181] Of course, this [Fig. 13] illustrates an example of a structural embodiment of a post-processing device as defined in the invention. Figures 6 to 12, discussed above, describe in detail functional embodiments of this device.

[0182] Applications of this type of post-processing can be found, for example, in audio decoding, such as when an encoded audio message is recorded (on a mail server or elsewhere) and must be decoded and played back to a user. This type of process can also be used in real-time conversations, provided that the complexity constraints of the broadcast model as described are taken into account.

[0183] The invention can also be applied to improve SGMSE denoising by using conditioning by a context which would be the noise level estimated by an external device.

Claims

Demands

1. A method for post-processing a decoded audio signal, wherein the post-processing (603) applies a post-processing model from a training phase (602) carried out between a reference signal (x') obtained by applying at least one pre-processing (601) to an input signal (x) and a decoded output signal (y').

2. A method according to claim 1, wherein a preprocessing applied to the input signal is a delay compensation.

3. A method according to any one of the preceding claims, wherein a preprocessing applied to the input signal is further phase compensation.

4. A method according to any one of claims 1 to 2, wherein a preprocessing applied to the input signal to obtain a reference signal for the training phase is a processing from the following list: - a resampling process between a first and a second frequency; - a pre-emphasis process; - a perceptual weighting process; and the decoded output signal is the signal before application of an inverse processing to at least one preprocessing from the list, the postprocessing being applied to the decoded signal before application of an inverse processing to at least one preprocessing from the list.

5. A method according to any one of the preceding claims, wherein the post-processing model is a diffusion denoising model.

6. A method according to any one of claims 1 to 5, wherein the post-processing method uses a neural network for the training phase and for an inference phase corresponding to the application of the post-processing model.

7. A method according to any one of claims 1 to 5, wherein the post-processing pattern depends on a conditioning value.

8. Method according to claim 7, wherein the conditioning value is a coding rate value or a value indicating a type of signal.

9. Audio signal post-processing device comprising a post-processing module (603) applying a post-processing model to a decoded audio signal and a training module generating the post-processing model and applying a training phase (602) between a reference signal obtained by applying at least one pre-processing (601) to an input signal and a decoded output signal.

10. Coded audio comprising a coding and decoding module and a post-processing device according to claim 9.

11. A processor-readable storage medium storing a computer program containing instructions for carrying out the process according to any one of claims 1 to 8.

Images