Device and method for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder, and associated computer program

Abstract

Device and method for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder, and associated computer program The present invention relates to a device (12) for post-processing voice synthesis (11) produced by an existing vocoder (10), comprising: - a module (14) for receiving said voice synthesis; - an artificial intelligence model (16) comprising: - a projection module (18); - a voice reconstruction module (20); - a training module (22) configured to train said model to correct voice synthesis artifacts generated by said existing vocoder, comprising: - a first training tool (24) configured to train said model according to a supervised learning phase; - a second training tool (26) configured to re-train by adverse training, via at least one discriminator (D), said model provided by the first training tool.Figure for abstract: Figure 1.

Description

Title of the invention: Device and method for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder, and associated computer program

[0001] The present invention relates to a device for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder.

[0002] The present invention also relates to a method for post-processing the voice synthesis produced using a predetermined codec by an existing vocoder.

[0003] The invention also relates to a computer program comprising software instructions which, when implemented by a computer, implement such a method.

[0004] The invention relates to the field of speech coding also called "vocoding", voice synthesis and voice transmission.

[0005] Subsequently, according to the present invention "vocoding" and by extension the corresponding device "vocoder" is associated with the contraction of the English words Voice Coding respectively Voice coder, in other words voice coding by an electronic device for processing the sound signal, corresponding substantially to a method according to which, from a decomposition into important properties (eg main spectral components) of an input voice or another sound, synthesizes an associated synthetic voice at the output.

[0006] Digital communications offer a significant number of benefits compared to analog communications. In addition to being of better quality, they are often more robust to interfering noise during transmission. However, one of the major problems linked to the very nature of digital communications is the limitation of the flow rate. Indeed, the higher the flow rate, the more it is possible to send a large amount of information allowing for higher voice quality. In addition, digital transmissions suffer from a "binarization" of the received data. Indeed, either the transmission of the digital frames is correct and the communication is unchanged, or it is incorrect (i.e. missing frames) and all of the information contained in these missing frames at reception is lost.

[0007] Some transmission contexts impose constrained bit rates and the use of codecs (or coder-decoders) whose sound quality is limited. In these contexts, the codecs used are generally standardized such as the MELP codec (from the English Mixed-excitation linearprediction) or the improved MELPe codec (from the English Enhanced Mixed-excitation linear prediction). most often associated with a flow rate, for example MELP 600 corresponding to a MELP codec with a flow rate of 600 bits / s, MELP 1200, or MELPe 1200, corresponding to a MELP codec, respectively MELPe, with a flow rate of 1200 bits / s, MELP 2400 or MELPe 2400, corresponding to a MELP codec, respectively MELPe, with a flow rate of 2400 bits / s, etc.

[0008] Codecs are generally optimized to operate according to precise modeling (e.g. LPC (Linear Predictive Coder), MELP, etc.) for extracting parameters from the speech signal.

[0009] The resulting parametric model can be used to synthesize a speech signal. This is called a parametric coder or vocoder (to clarify that the model applies to the speech signal).

[0010] The quantization / dequantization step allows transmission over a constrained channel by restricting the flow rate. There are therefore two sources of degradation: the approximation of the model, and the quantization of the model parameters.

[0011] The different MELP bit rates correspond to different operating modes. It is the quantization scheme that is optimized for a given model (LPC, MELP, ...) and a target bit rate. If the bit streams are obviously different, the information is of the same type (associated with the MELP parameters for example).

[0012] A bitstream associated with the MELPe 1200 codec works with the MELPe 1200 vocoder and cannot work with a MELPe 2400 vocoder.

[0013] Indeed, a modification of the quantification scheme implies a loss of interoperability: a MELP 1200 is not interoperable with a MELP 2400.

[0014] Currently, the voice synthesis output from existing vocoders, in particular a MELP vocoder (according to all these existing versions (i.e. variants) such as the aforementioned examples MELP 600, MELP 1200, MELPe 1200, MELP 2400, MELPe 2400, etc.) very often presents artifacts.

[0015] To address this, vocoding solutions including AI artificial intelligence capabilities have been proposed. However, such solutions are generally very demanding in terms of input parameters, requiring, for example, an input number of around eighty Mel frequencies to provide voice synthesis. In addition, such AI solutions are generally not designed to allow real-time implementation, favoring operation adapted to uses such as text-to-speech transformation or voice assistants. Furthermore, the number of parameters generally associated with the implementation of such AI solutions, of the order of tens of millions of parameters, reduces the possibilities of implementation in embedded hardware with strong memory and computing capacity constraints.

[0016] To summarize, currently, the reduction in the number of input parameters is therefore systematically accompanied by a reduction in the quality of voice synthesis, and conventional low-bitrate voice synthesizers, such as MELP vocoders (according to all these existing versions (i.e. variants) such as the aforementioned examples MELP 600, MELP 1200, MELPe 1200, MELP 2400, MELPe 2400, etc.) cannot compensate for the loss of information linked to the reduction in the number of parameters, while AI voice synthesizers compensate for the loss of information but are often characterized by heavy numerical calculations, as well as by a large number of implementation parameters, these properties being inherent to the fact that AI voice synthesizers learn an a priori distribution to compensate for areas without information.

[0017] The aim of the invention is then to improve the sound quality (i.e. audio) of voice syntheses provided by existing vocoders using in particular low bit rate or even very low bit rate codecs used in certain transmission contexts which impose constrained bit rates, such as for example the MELP codec and its variants, while being suitable for being embedded (i.e. integrated) within existing radiocommunication products, and for operating in real time.

[0018] To this end, the invention relates to a device for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder, said post-processing device comprising:

[0019] - a module for receiving the voice synthesis produced by said existing vocoder;

[0020] - an artificial intelligence model comprising:

[0021] - a projection module configured to project via at least one network of neurons said voice synthesis produced by said existing vocoder, in a latent space of configurable dimension;

[0022] - a voice reconstruction module configured to reconstruct a synthesis corrected voice, via at least one other neural network, from said projection provided by said projection module;

[0023] - a training module for said artificial intelligence model;

[0024] said training module being configured to train said artificial intelligence model to correct the voice synthesis artifacts generated by said existing vocoder, from a database comprising a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of voice syntheses produced beforehand,

[0025] said training module comprising two training tools implemented successively:

[0026] - the first training tool being configured to train said model artificial intelligence according to a first phase of supervised learning;

[0027] - the second training tool being configured to re-train by training adverse, via at least one discriminator, according to a second learning phase, said artificial intelligence model previously trained and provided by the first training tool.

[0028] Such a post-processing device advantageously offers an improvement in the quality of the voice synthesis of an existing vocoder by focusing this time directly on the detection and then the correction of the artifacts of the voice synthesis provided by such existing vocoders, instead of seeking to replace them completely with AI voice synthesizers (i.e. so-called full AI).

[0029] The present invention is further advantageously applicable to several uses, namely increasing the audio quality while remaining at an equivalent bit rate compared to the voice synthesis provided by the existing vocoder, or even gaining bit rate at equivalent quality. Indeed, an improvement in audio quality is capable of being transcribed in terms of gain in bit rate compared to a given existing vocoder, such as a MELP vocoder for example.

[0030] In other words, at the output of an existing vocoder, the present invention proposes to use an artificial intelligence model specifically trained to focus on detecting the position of the artifacts and then correcting them.

[0031] Instead of the existing solution based on AI voice synthesizers associated with heavy digital calculations, it is therefore proposed here to keep the classic low-bitrate voice synthesizers and by post-processing according to the invention to correct only the artifacts that they produce by deleting them (i.e. by removing them, which amounts to a denoising task), which significantly lightens the computational load and allows real-time operation.

[0032] To do this, the proposed post-processing device comprises an artificial intelligence model which is specifically trained by a first training tool and then re-trained, by adverse training, by a second training tool.

[0033] According to other advantageous aspects of the invention, the post-treatment device comprises one or more of the following characteristics, taken individually or in all technically possible combinations:

[0034] - said at least one predetermined codec used to generate said voice synthesis and said database is the MELP code, or the improved MELPe code, and is associated with a flow rate belonging to the group comprising the following flow rates: 600 bits / s; 1200 bits / s; - 2400 bits / s,

[0035] said database being single-bit or multi-bit and / or single-language or multi-language;

[0036] - said first training tool is configured to optimize said model artificial intelligence by simultaneously minimizing four distinct loss functions;

[0037] - said four distinct loss functions are expressed in the following form:

[0038] - Lj (x, y) = ||x- jU , corresponding to the norm Ll of the difference between the signal input voice x of the existing vocoder and the output of the artificial intelligence model whose input is connected to the output of said existing vocoder;

[0039] - L2(x,y, params) = difference between the spectrogram ç ( of the input speech signal x of the vocoder 1 ° O params\y / ° existing and the spectrogram Sparamly) 'a sort^e of the artificial intelligence model whose input is connected to the output of said existing vocoder;

[0040] - L3(x, y) = X parants ), the combination of the loss function with spectrogram parameters from a predetermined list P of parameters Spectrogram Params;

[0041] - ​L4 ( X, y ) = lLmparams ^5 ( X, y, mparams )>the combination of the loss function £5 with mel-spectrogram parameters from a predetermined list Q of mel-spectrogram parameters, where L5 ( X, y, mparams ) - 11MSmparam^x) - MSmpûram(y) 119'is the normL2 of the difference between the mel-spectrogram MSmparam / x) of the input voice signal x of the existing vocoder and the mel-spectrogram MSmparamJ^y) 'at the output 'V of the artificial intelligence model whose input is connected to the output of said existing vocoder;

[0042] - the second training tool is configured to implement alternatively two distinct optimizations,

[0043] the first optimization minimizing both:

[0044] - said four distinct loss functions associated with said first phase supervised learning, and

[0045] - a first loss function LG associated with said second learning phase by adverse training, via said at least one discriminator D, with LcA^y') = l^(v) -1| ;

[0046] the second optimization, implemented successively to said first optimization, minimizing a second loss function LD associated with said second learning phase by adverse training, via said at least one discriminator D, with LD(x, y) - |D(y) | + |D(x) + 1|;

[0047] - said database is previously processed by dividing said plurality of voice synthesizers in two sets:

[0048] - a training data set itself divided into a number predetermined batches of voice synthesizers,

[0049] - a validation data set,

[0050] said second learning phase comprising a plurality of iterations,

[0051] during each iteration, said two distinct optimizations being applied, batch by batch, until the training data set of said database has been completely processed,

[0052] said complete processing being repeated until convergence according to a predetermined convergence criterion and determined using the validation data set;

[0053] - said predetermined convergence criterion depends on the PESQ score associated with the speech synthesis corrected via said re-trained artificial intelligence model and / or the result of said loss functions used by said training module;

[0054] - said projection module comprises at least three projection blocks implementing implements three successive treatments:

[0055] - a first projection block, convolutional, configured to take into account the times the correlation along a time dimension and along a frequency dimension of the data received as input;

[0056] - a second projection block receiving as input the output of the first block, said second block corresponding to a first temporal convolutional neural network capable of taking into account short and medium term temporal correlation;

[0057] - a third projection block receiving as input the output of the second block, the third block corresponding to a first recurring unit with a gate capable of taking into account a long-term temporal correlation;

[0058] and in which said reconstruction module, the input of which is connected to the output of the projection module, comprises at least four reconstruction blocks implementing four successive processing operations:

[0059] - a first reconstruction block corresponding to a second neural network temporal convolutional capable of taking into account short and medium term temporal correlation;

[0060] - a second reconstruction block receiving as input the output of the first block of reconstruction, said second reconstruction block corresponding to a second recurrent unit with a gate capable of taking into account a long-term temporal correlation;

[0061] - a third reconstruction block receiving as input the output of the second block of reconstruction, the third reconstruction block corresponding to a third temporal convolutional neural network capable of taking into account short and medium term temporal correlation;

[0062] - a fourth reconstruction block, convolutional, receiving as input the output of the third reconstruction block, the fourth reconstruction block being configured to implement the inverse operation of that implemented by the first projection block and provide a change in scale of said reconstructed and corrected speech synthesis.

[0063] The invention also relates to a method for post-processing the voice synthesis produced by means of a predetermined codec by an existing vocoder, said method being implemented by a post-processing device, said method comprising:

[0064] - receiving the voice synthesis produced by said existing vocoder;

[0065] and the following steps implemented during an inference phase implemented by an artificial intelligence model of said post-processing device previously trained during a prior training phase,

[0066] said inference phase comprising:

[0067] - the projection, via at least one neural network, of said voice synthesis produced by said existing vocoder, in a latent space of configurable dimension;

[0068] - the voice reconstruction of a corrected voice synthesis, via at least one other neural network, from said projection provided by said projection step;

[0069] said preliminary training phase of said artificial intelligence model training said artificial intelligence model to correct the voice synthesis artifacts generated by said existing vocoder, from a database comprising a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of voice syntheses produced beforehand, said training phase

[0070] said preliminary training phase comprising at least two training steps implemented successively and corresponding respectively to

[0071] - a first supervised learning phase; and

[0072] - a second learning phase during which said model artificial intelligence previously trained during the first supervised learning phase is re-trained by adversarial training, via at least one discriminator.

[0073] The invention also relates to a computer program comprising software instructions which, when executed by a computer, implement a method of post-processing the voice synthesis produced by means of a predetermined codec by an existing vocoder, as defined above.

[0074] The invention will appear more clearly on reading the description which follows, given solely by way of non-limiting example, and made with reference to the drawings in which:

[0075] [Fig-1] [Fig.l] is a general schematic representation of a device, according to the present invention, for post-processing the voice synthesis produced using a predetermined codec by an existing vocoder;

[0076] [Fig.2] [Fig.2] illustrates an example of the architecture of the artificial intelligence model of the post-processing device of [Fig.l];

[0077] [Fig.3] [Fig.3] is a flowchart of a method for post-processing speech synthesis produced using a predetermined codec by an existing vocoder implemented by the post-processing device of [Fig.l] according to one embodiment.

[0078] In the remainder of the description, the expression “substantially equal to” is understood as a relationship of equality to plus or minus 10%, that is to say with a variation of at most 10%, more preferably as a relationship of equality to plus or minus 5%, that is to say with a variation of at most 5%.

[0079] Furthermore, subsequently, we consider that a neural network comprises an ordered succession of layers of neurons, each of which takes its inputs from the outputs of the previous layer.

[0080] More precisely, each layer comprises neurons taking their inputs from the outputs of the neurons of the previous layer, or from the input variables for the first layer.

[0081] Alternatively, more complex neural network structures can be envisaged with a layer that can be connected to a layer further away than the immediately preceding layer.

[0082] Each neuron is also associated with an operation, i.e. a type of processing, to be carried out by said neuron within the corresponding processing layer.

[0083] Each layer is connected to the other layers by a plurality of synapses. A synaptic weight is associated with each synapse, and each synapse forms a link between two neurons. It is often a real number, which takes both positive and negative values. In some cases, the synaptic weight is a complex number.

[0084] Each neuron is capable of performing a weighted sum of the value(s) received from the neurons of the previous layer, each value then being multiplied by the respective synaptic weight of each synapse, or link, between said neuron and the neurons of the previous layer, then applying an activation function, typically a non-linear function, to said weighted sum, and delivering at the output of said neuron, in particular to the neurons of the following layer which are connected to it, the value resulting from the application of the activation function. The activation function makes it possible to introduce a non-linearity into the processing carried out by each neuron. The sigmoid function, the hyperbolic tangent function, the Heaviside function are examples of activation functions.

[0085] As an optional addition, each neuron is also capable of applying, in addition, a multiplicative factor, also called bias, to the output of the activation function, and the value delivered at the output of said neuron is then the product of the bias value and the value from the activation function.

[0086] A fully connected layer of neurons is a layer in which the neurons of said layer are each connected to all the neurons of the previous layer.

[0087] Such a type of layer is more often referred to as “fully connected” and sometimes referred to as a “dense layer”.

[0088] [Fig.l] illustrates, in general, an electronic chain for processing an input voice signal x (i.e. input sound signal comprising a human voice) processed by an existing vocoder 10 delivering at output a voice synthesis 11 classically very often presenting artifacts resulting in a reduction in the quality of the voice synthesis 11 at the output of the existing vocoder compared to the input voice signal x. To remedy this, it is proposed according to the present invention to add at the output of the existing vocoder 10 a post-processing device 12.

[0089] As can be seen in [Fig.l], said post-processing device 12 firstly comprises a module 14 for receiving the voice synthesis produced by said existing vocoder 10.

[0090] In other words, the reception module 14 of the post-processing device 12 receives as input the result 11 of the voice synthesis produced by the existing vocoder 10, such a result 11 classically very often presenting artifacts compared to the input voice signal x of the existing vocoder 10.

[0091] Furthermore, the post-processing device 12 also comprises an artificial intelligence model 16 comprising on the one hand a projection module 18 configured to project said voice synthesis via at least one neural network 11 produced by said existing vocoder 10, in a latent space of configurable dimension, and on the other hand a voice reconstruction module 20 configured to reconstruct a corrected voice synthesis y, via at least one other neural network, from said projection provided by said projection module.

[0092] According to the present invention, the post-processing device 12 also comprises a training module 22 for said artificial intelligence model 16.

[0093] According to the present invention, said training module 22 is configured to train said artificial intelligence model 16 to correct the voice synthesis artifacts 11 generated by said existing vocoder 10, from a database DB.

[0094] The DB database includes:

[0095] - a plurality of voice syntheses produced beforehand, by means of at least one predetermined coded, by at least one existing vocoder, and

[0096] - a plurality of reference voice signals used as input by said at least one an existing vocoder for producing said plurality of pre-produced voice syntheses.

[0097] The training module 22 comprises two training tools 24 and 26 implemented successively. More precisely, the first training tool 24 is configured to train said artificial intelligence model according to a first supervised learning phase by signal reconstruction, in particular using said database DB comprising, as indicated previously, both a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder, and a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of voice syntheses produced beforehand.

[0098] The second training tool 26 is configured to re-train by adversarial training, via at least one discriminator D, according to a second learning phase, said artificial intelligence model 16 previously trained and provided by the first training tool 24.

[0099] For example, the second training tool 26 uses two discriminators, namely for example a multi-scale discriminator MSD (from the English multi-scale discriminator), an example of which is described in the article by J. Kong et al entitled “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis” NeurIPS 2020, but applied in a context other than that of the present invention, namely to re-train an artificial intelligence model 16 previously trained via supervised learning, and a multi-scale convolutional discriminator MSCD (from the English multi-scale convolution discriminator) as introduced in the article by W. Li et al. entitled “Improvi ng

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] [YES] Generative Adversarial Network-based Vocoding throgh Multi-scale Convolution » ACM Transactions on Asian and Low-Resource Language Information Processing, Volume 22, number 9, article n°230, pages 1 to 10. It should be noted that once the learning phase has been carried out, such a training module 22 is no longer used during the inference phase using the artificial intelligence model 16 thus trained. According to a particular variant, said at least one predetermined codec used by said existing vocoder 10 to generate said voice synthesis 11 and said database DB is the MELP codec, or the improved MELPe codec, and is associated with a rate belonging to the group comprising the following rates: - 600 bits / s; - 1200 bits / s; - 2400 bits / s; said DB database being single-bit or multi-bit and / or single-language or multi-language. In other words, the artificial intelligence model 16 is capable of being trained for one flow rate or for several flow rates at the same time, and for a single language or several languages ​​at the same time, for example French, English and Arabic, etc. In other words, according to this variant, the post-processing device 12 is specifically configured to correct the artifacts of a voice synthesis 11 provided by an existing MELP vocoder 10 (according to all these existing versions (i.e. variants) such as the aforementioned examples MELP 600, MELP 1200, MELPe 1200, MELP 2400, MELPe 2400, etc.). As an optional addition, said first training tool 24 is configured to optimize said artificial intelligence model 16 by simultaneously minimizing four distinct loss functions. According to a specific variant of this optional complement, the said four distinct loss functions are expressed in the following form: - y ) = || X -y II , corresponding to the L1 norm of the difference between the input voice signal x of the existing vocoder 10 and the output of the artificial intelligence model 16 whose input is connected to the output of said existing vocoder 10; - £2(x, y, params) = |Jcorresponding to the L2 norm of the difference between the spectrogram ç of the input voice signal x of the vocoder r ° ^param&A'J ° existing 10 and the Sparam^y) spectrogram comes out of the intelligence model artificial 16 whose input is connected to the output of said existing vocoder 10;

[0112] - L3(x,y) = Hparmns(x,y,params), the combination of the L2 loss function with spectrogram parameters from a predetermined list P of parameters Spectrogram Params;

[0113] - ( x, y ) = "Lmparams ^5 ( x, y, mparams ), the combination of the loss function Ls with mel-spectrogram parameters from a predetermined list Q of mparams i mel-spectrogram parameters, where L5 ( x, y, mparams ) = |\MSmparamix} - MSmparamly} | [ 7'is the L2 norm of the difference between the mel-spectrogram MSmparaml.x) of the input voice signal x of the existing vocoder and the mV mel-spectrogram of the output y of the model of artificial intelligence whose input is connected to the output of said existing vocoder.

[0114] As an optional addition, the second training tool 26 is configured to alternately implement two distinct optimizations,

[0115] the first optimization minimizing both:

[0116] - said four distinct loss functions associated with said first phase supervised learning, and

[0117] - a first loss function LG associated with said second learning phase by adverse training, via said at least one discriminator D, with LG(x,y) = \D(y) -1| ;

[0118] the second optimization, implemented successively to said first optimization, minimizing a second loss function LD associated with said second learning phase by adverse training, via said at least one discriminator D, with Lj)(x, y) - |Z)(y) | + |D(x) -1|

[0119] Note that when for example two discriminators are used, such as the aforementioned multi-scale discriminator MSD and the aforementioned multi-scale convolutional discriminator MSCD, these two discriminators are used simultaneously and the loss function LD corresponds to the average of two loss functions LMSD and ^mscd which are respectively associated with them such that = \MSD(y) \ + |MSP(x)-lh* \MSCD(ÿ) \ + |MSD(x) -1|.

[0120] More precisely, said at least one discriminator is capable of receiving as input the input voice signal x (i.e. the real source signal with a human voice provided as input to the existing vocoder 10) as well as the output y of the artificial intelligence model 16 previously trained by the first training tool 24, and the first loss function LG allows the discriminator to minimize the difference between these two signals while the second loss function LD allows to find the areas where the artifacts, constituting differences between the input voice signal x and the output y of the artificial intelligence model 16, are present.

[0121] The purpose of the discriminator is to distinguish the real signal x from the signal y generated by the artificial intelligence model 16, with for example, D(y) =0, if the discriminator detects the signal y generated by the artificial intelligence model 16 and D(y) = 1 if the discriminator detects the real signal x.

[0122] The loss function LD(x, y) = \D(y) | + |D(.r) -1| aims to localize the artifact areas, while the loss function LG(x, y) = |£>(y ) -1| aims to fool the discriminator and minimize the differences in these localized areas via the LD loss function.

[0123] In the example of [Fig.l], the post-processing device 12 comprises an information processing unit 28 formed for example of a memory 30 and a processor 32 associated with the memory 30.

[0124] In the example of [Fig.l], the reception module 14, the artificial intelligence model 16 and the training module 22 are each produced in the form of software, or a software brick, executable by the processor 32. The memory 30 of the post-processing device 12 is then capable of storing reception software, an artificial intelligence model, and training software. The processor 32 is then capable of executing each of the software among the reception software, the artificial intelligence model, the training software.

[0125] In a variant not shown, the reception module 14, the artificial intelligence model 16 and the training module 22 are each produced in the form of a programmable logic component, such as an FPGA (Field Programmable Gate Array), or an integrated circuit, such as an ASIC (Application Specific Integrated Circuit).

[0126] When the post-processing device 12 is produced in the form of one or more software programs, that is to say in the form of a computer program, also called a computer program product, it is furthermore capable of being recorded on a medium, not shown, readable by a computer. The computer-readable medium is for example a medium capable of storing electronic instructions and of being coupled to a bus of a computer system. By way of example, the readable medium is an optical disk, a magneto-optical disk, a ROM memory, a RAM memory, any type of non-volatile memory (for example FLASH or NVRAM) or a magnetic card. A computer program comprising software instructions is then stored on the readable medium.

[0127] [Fig.2] illustrates an example of the architecture of the artificial intelligence model 16 of the post-processing device 12 of [Fig.l].

[0128] According to the example of [Fig.2], said projection module 18 comprises at least three projection blocks implementing three successive processing operations, namely first of all a first projection block 34, convolutional, configured to take into account both the correlation according to a time dimension and according to a frequency dimension of the data received as input.

[0129] The projection module 18 then comprises a second projection block 36 receiving as input the output of the first block 34, said second block 36 corresponding to a first temporal convolutional neural network TCN (from the English temporal convolutional network) capable of taking into account a short and medium term temporal correlation.

[0130] According to a particular example, this second boc 36 is composed of two convolutions with a kernel of size 1 x 1 to change the dimensions of the channel, and a dilated depthwise convolution to exploit the temporal correlations with low complexity. This TCN block is more efficient for exploring the short term and the medium term of the temporal evolution. By "short term", we mean taking into account the temporal evolution with a distance in time samples less than twenty, by "medium term" we mean taking into account the temporal evolution with a distance in time samples between twenty and two hundred, and by "long term" we mean taking into account the temporal evolution with a distance in time samples greater than two hundred.

[0131] The projection module 18 then comprises a third projection block 38 receiving as input the output of the second block 36, the third block 38 corresponding to a first recurrent unit with a GRU gate (from the English Gate Recurrent Unit) capable of taking into account a long-term temporal correlation.

[0132] As an alternative to the GRU unit, a layer with long short term memory such as LSTM (Long Short Term Memory) or even more conventionally a recurrent RNN (Recurrent Neural Network) layer is used.

[0133] Such a third block with the GRU unit is however advantageous because it has only two gates: a reset gate and an update gate, compared to three gates of LSTM. Therefore, the third block with the GRU unit is not too complex and involves fewer parameters compared to the use of a layer with long short-term memory type LSTM. The third block with the GRU unit is very effective to explore the long term of the time evolution, which represents an interesting property in addition to the second block 36 with TCN.

[0134] In other words, the present invention proposes to combine the TCN and GRU in the artificial intelligence model 16 in order to take advantage of their complementarity.

[0135] The reconstruction module 20, the input of which is connected to the output of the projection module 18, comprises, according to the example of [Fig.2], at least four reconstruction blocks implementing four successive processing operations, namely a first reconstruction block 40 corresponding to a second temporal convolutional neural network TCN capable of taking into account a short and medium term temporal correlation.

[0136] The reconstruction module 20 also comprises a second reconstruction block 42 receiving as input the output of the first reconstruction block 40, said second reconstruction block 42 corresponding to a second recurrent unit with GRU gate capable of taking into account a long-term temporal correlation.

[0137] The reconstruction module 20 also comprises a third reconstruction block 44 receiving as input the output of the second reconstruction block 42, the third reconstruction block 44 corresponding to a third temporal convolutional neural network TCN capable of taking into account a short and medium term temporal correlation.

[0138] The reconstruction module 20 also comprises a fourth reconstruction block 46, convolutional, receiving as input the output of the third reconstruction block 44, the fourth reconstruction block 46 being configured to implement the inverse operation of that implemented by the first projection block 34 and provide a change of scale of said reconstructed and corrected voice synthesis.

[0139] Advantageously, the number of parameters associated with the implementation of such an artificial intelligence model 16 remains less than two million parameters, which proves to be less demanding compared to the solutions, mentioned above in relation to the prior art, of vocoding comprising artificial intelligence AI capabilities, and then allows it to be embedded (i.e. integrated) within existing radiocommunication products, and to operate in real time.

[0140] A post-processing method 50 implemented via said post-processing device 12 is subsequently explained with reference to [Fig.3] showing a flowchart of the steps of this method 50.

[0141] Generally, said method 50 firstly comprises a training phase 52 training said artificial intelligence model, previously described in relation to FIGS. 1 and 2, to correct the voice synthesis artifacts generated by said existing vocoder, from the database DB comprising a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of voice signals of reference used as input by said at least one existing vocoder to produce said plurality of previously produced voice syntheses.

[0142] According to a step 54, said database DB is previously processed by dividing Div said plurality of voice syntheses into two sets, namely a training dataset (train dataset) on the one hand, itself divided into a predetermined number of batches of voice syntheses, and a validation dataset (valid dataset).

[0143] Said preliminary training phase 52 comprises at least two training steps 56 and 58 implemented successively and corresponding respectively to

[0144] a first supervised learning phase 56 E_Ab and a second learning phase 58 during which said artificial intelligence model previously trained during the first supervised learning phase E_Ai is re-trained by adversarial training, via at least one discriminator.

[0145] The first supervised learning phase 56 E_Ai uses, for supervision, each voice synthesis produced beforehand, by means of at least one predetermined codec, by said at least one existing vocoder and the associated reference voice signal used as input by said at least one existing vocoder to produce said voice synthesis.

[0146] During this first phase 56 of supervised learning E_Ai, said artificial intelligence model 16 is optimized by simultaneously minimizing four distinct loss functions, in particular the aforementioned loss functions L3 and L4.

[0147] The second learning phase 58 comprises two steps 60 and 62 of distinct optimizations Oi and O2 implemented alternately.

[0148] The first step 60 of optimization Oi minimizing both:

[0149] - said four distinct loss functions L2, L3 and L4 associated with said first phase 56 of supervised learning, and

[0150] - a first loss function LG associated with said second learning phase by adverse training, via said at least one discriminator D, with LG(.r,y) = |D(y)-l| .

[0151] This first optimization step 60 Oi makes it possible, compared to the first supervised learning phase 56 E_AB, to continue to bring the output of the artificial intelligence model 16 closer to the input voice signal x of the existing vocoder 10, by minimizing said four distinct loss functions Eb£2, ^3 and L4 while minimizing the number and size of the different zones between x and y detected by the discriminator thanks to LG.

[0152] The second step 62 of optimization O2, implemented successively to said first optimization Oi, minimizes a second loss function LD associated with said second phase of learning by adversarial training, via said at least one discriminator D, with LD(x, y) = |#(y) | + |Z>(x) + IJ.

[0153] This second step 62 of optimization O2 will make it possible to update the weights of said at least one discriminator D to strengthen it while continuing to detect different zones between x and y.

[0154] As an optional addition, said second learning phase 58 comprises a plurality of iterations I, said two distinct optimizations Oi and O2 being applied during each iteration, batch by batch (i.e. batch by batch), until complete processing, called epoch (from the English epoch) of the training data set of said database DB.

[0155] At each iteration, batch by batch (i.e. batch by batch) the weights of the artificial intelligence model are updated.

[0156] Said complete processing is itself repeated, epoch by epoch, until convergence according to a predetermined convergence criterion, evaluated (i.e. determined), at each epoch (from the English epoch), using the validation data set.

[0157] According to an optional aspect of this optional supplement, said predetermined convergence criterion depends on the PESQ score (i.e. the metric for perceptual evaluation of speech quality of English Perceptual Evaluation of Speech Quality) associated with the speech synthesis corrected via said retrained artificial intelligence model and / or the result of said loss functions L^L2, L3 and used by said training module.

[0158] In other words, at each epoch, the PESQ score and / or the result of said loss functions L2, L3 and L4 used by said training module is determined, and in the event of improvement compared to the previous epoch, said second learning phase 58 is repeated, batch by batch, during a following epoch. Otherwise (i.e. absence of improvement), the training phase 52 is stopped and provides a trained artificial intelligence model M_E.

[0159] Once the trained artificial intelligence model M_E has been obtained at the end of the training phase 52, the method 50 comprises, in real time, a step 64 of receiving R the voice synthesis E produced, in particular in real time, by said existing vocoder.

[0160] Then the method 50 comprises an inference phase 66 using the trained artificial intelligence model 16 M_E, and comprising at least one projection step 68 P, via at least one neural network of said projection module 18 of the trained artificial intelligence model M_E, of said voice synthesis E produced. by said vocoder existing in a latent space of configurable dimension, then at least one step 70 of voice reconstruction RV of a corrected voice synthesis, via at least one other neural network of the voice reconstruction module 20 of the trained artificial intelligence model M_E, from said projection in said latent space of configurable dimension provided by said projection step.

[0161] According to an aspect not shown, said voice synthesis E used in the inference phase 66 is suitable for being subsequently integrated into the database DB, in particular in the validation data set, to subsequently re-train (offline) said artificial intelligence model.

[0162] In other words, the artificial intelligence model is capable of being continuously improved via the alternation of the training phase 52 and at least one inference phase 66, the voice syntheses E received during successive inference phases 66 being capable of being integrated into the database DB for subsequent training.

[0163] Those skilled in the art will understand that the invention is not limited to the embodiments described, nor to the particular examples of the description, the embodiments and variants mentioned above being suitable for being combined with each other to generate new embodiments of the invention.

[0164] The present invention thus makes it possible to propose a “corrector” by post-processing the output of an existing vocoder. For example, the correction result obtained on the MELP 600 and MELP 1200 is substantial, with in particular an increase in the PESQ score (i.e. the PESQ metric) of 0 to 10% for the MELP 600, the MELP 1200 or even the MELP 2400, a gain of 6dB in range, while also allowing more conferences in parallel.

[0165] In other words, the present invention makes it possible to improve the synthesis produced by an existing vocoder, for example the MELP vocoder, with a significant margin of improvement in the audio quality. This solution is also suitable for being embedded in radiocommunication products and advantageously its execution is suitable for being implemented in real time and with very low latency.

[0166] This solution opens a new direction of improving MELP which is a popular coding with many applications in. The proposed solution allows to have a better quality of the audio compared to MELP, with a significant improvement in the number of parameters and the computation time compared to vocoding solutions including AI artificial intelligence capabilities (which globally implement a vocoding and not a correction of an existing vocoding).

Claims

Claims

1. Post-processing device (12) for the voice synthesis (11) produced by means of a predetermined codec by an existing vocoder (10), said post-processing device (12) comprising: - a reception module (14) for the voice synthesis produced by said existing vocoder; - an artificial intelligence model (16) comprising: - a projection module (18) configured to project via at least one neural network said voice synthesis produced by said existing vocoder, into a latent space of configurable dimension; - a voice reconstruction module (20) configured to reconstruct a corrected voice synthesis, via at least one other neural network, from said projection provided by said projection module; - a training module (22) for said artificial intelligence model;said post-processing device being characterized in that said training module is configured to train said artificial intelligence model to correct the speech synthesis artifacts generated by said existing vocoder, from a database (DB) comprising a plurality of speech syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference speech signals used as input by said at least one existing vocoder to produce said plurality of speech syntheses produced beforehand, said training module (22) comprising two training tools implemented successively: - the first training tool (24) being configured to train said artificial intelligence model (16) according to a first supervised learning phase;- the second training tool (26) being configured to re-train by adverse training, via at least one discriminator (D), according to a second learning phase, said artificial intelligence model previously trained and provided by the first training tool.;

2. Device (12) according to claim 1, wherein said at least one predetermined codec used to generate said speech synthesis and said database is the MELP code, or the improved MELPe code, and is associated with a rate belonging to the group comprising the following rates: - 600 bits / s; - 1200 bits / s; - 2400 bits / s, said database being single-rate or multi-rate and / or single-language or multi-language.

3. A device (12) according to claim 1 or 2, wherein said first training tool (24) is configured to optimize said artificial intelligence model (16) by simultaneously minimizing four distinct loss functions.

4. Device (12) according to claim 3, wherein said four distinct loss functions are expressed in the following form: - L^X^y) = j| xJ j| corresponding to the norm L1 of the difference between the input voice signal x of the existing vocoder and the output of the artificial intelligence model whose input is connected to the output of said existing vocoder; - L2(x, y, params) = ||Spa„Ux)-S^^^ corresponding to the norm L2 of the difference between the spectrogram ç (,4 of the r ° paramsy^ j input voice signal x of the existing vocoder and the spectrogram ç of the output y of the artificial intelligence model whose input is connected to the output of said existing vocoder; L3(X y) = * params), the combination of the loss function E? with spectrogram parameters from a predetermined list P of spectrogram parameters Params;L4 ( X, y ) = Y^mparams eQL5 ( x, y, mparams ) 'the combination of the L5 loss function with mel-spectrogram parameters from a predetermined list Q of mel-spectrogram parameters mparams, where L5(x, y, mparams) = || MSmparam / x)- MSmparamiy)\\^ is the L2 norm of the difference between the mel-spectrogram; MSmparam^x) input vocal s^ëna^ x of the existing vocoder and the mel-spectrogram MSmparam^y) 'a output of the artificial intelligence model whose input is connected to the output of said existing vocoder.

5. Device (12) according to claim 4, wherein the second training tool (26) is configured to alternately implement two distinct optimizations, the first optimization minimizing both: - said four distinct loss functions associated with said first supervised learning phase, and - a first loss function LG associated with said second learning phase by adversarial training, via said at least one discriminator D, with (x, y ) = [Z>(y ) -1|; the second optimization, implemented successively to said first optimization, minimizing a second loss function LD associated with said second learning phase by adversarial training, via said at least one discriminator D, with LD(x, y) = \D(y)\ + |D(x) + lJ.

6. Device according to claim 5, wherein said database is previously processed by dividing said plurality of voice syntheses into two sets: - a training data set itself divided into a predetermined number of batches of voice syntheses, - a validation data set, said second learning phase comprising a plurality of iterations, during each iteration, said two distinct optimizations being applied, batch by batch, until complete processing of the training data set of said database, said complete processing being reiterated until convergence according to a predetermined convergence criterion and determined using the validation data set.

7. Device according to claim 6, wherein said predetermined convergence criterion depends on the PESQ score associated with the speech synthesis corrected via said re-trained artificial intelligence model and / or the result of said loss functions used by said training module.

8. Device according to any one of the preceding claims wherein said projection module comprises at least three projection blocks implementing three successive processing operations: - a first projection block (34), convolutional, configured to take into account both the correlation according to a temporal dimension and according to a frequency dimension of the data received as input; - a second projection block (36) receiving as input the output of the first block, said second block corresponding to a first temporal convolutional neural network capable of taking into account a short and medium term temporal correlation; - a third projection block (38) receiving as input the output of the second block, the third block corresponding to a first gated recurrent unit capable of taking into account a long term temporal correlation;and wherein said reconstruction module (20), the input of which is connected to the output of the projection module (18), comprises at least four reconstruction blocks implementing four successive processing operations: - a first reconstruction block (40) corresponding to a second temporal convolutional neural network capable of taking into account a short and medium term temporal correlation; - a second reconstruction block (42) receiving as input the output of the first reconstruction block (40), said second reconstruction block (42) corresponding to a second gated recurrent unit capable of taking into account a long term temporal correlation; - a third reconstruction block (44) receiving as input the output of the second reconstruction block (42), the third reconstruction block (44) corresponding to a third temporal convolutional neural network capable of taking into account a short and medium term temporal correlation;- a fourth convolutional reconstruction block (46) receiving as input the output of the third reconstruction block (44), the fourth reconstruction block (46) being configured to implement the inverse operation of that implemented by the first projection block (34) and provide a change in scale of said reconstructed and corrected voice synthesis.;

9. Method (50) for post-processing the voice synthesis produced by means of a predetermined codec by an existing vocoder, said method being implemented by a post-processing device according to any one of the preceding claims, said method comprising: - receiving (64) the voice synthesis produced by said existing vocoder; and the following steps implemented during an inference phase (66) implemented by an artificial intelligence model of said post-processing device previously trained during a prior training phase (52), said inference phase (66) comprising: - projecting (68), via at least one neural network, said voice synthesis produced by said existing vocoder, into a latent space of configurable dimension; - voice reconstruction (70) of a corrected voice synthesis, via at least one other neural network, from said projection provided by said projection step;said preliminary training phase (52) of said artificial intelligence model training said artificial intelligence model to correct the speech synthesis artifacts generated by said existing vocoder, from a database (DB) comprising a plurality of speech syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference speech signals used as input by said at least one existing vocoder to produce said plurality of speech syntheses produced beforehand, said training phase said preliminary training phase (52) comprising at least two training steps implemented successively and corresponding respectively to - a first supervised learning phase (56);and - a second learning phase (58) during which said artificial intelligence model previously trained during the first supervised learning phase is re-trained by adverse training, via at least one discriminator.;

10. A computer program product, comprising software instructions which, when executed by a computer, implement a method according to claim 9.

Images