Device and method for post-processing the voice synthesis produced by means of a predetermined codec by an existing vocoder, and related computer program

Abstract

The present invention relates to a device for improving the voice synthesis (11) produced by an existing vocoder (10), comprising: - a module (14) for receiving the voice synthesis; - an artificial intelligence model (16) comprising: - a projection module (18); - a voice reconstruction module (20); and - a training module (22) configured to train the model to correct the voice synthesis artefacts generated by the existing vocoder and comprising: - a first training tool (24) configured to train the model according to a supervised learning phase; - a second training tool (26) configured to retrain said model by means of adversarial training, via at least one discriminator (D), the model being provided by the first training tool.

Description

[0001] TITLE: Device and method for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder, and associated computer program

[0002] The present invention relates to a device for post-processing speech synthesis produced using a predetermined codec by an existing vocoder.

[0003] The present invention also relates to a method for post-processing the speech synthesis produced using a predetermined codec by an existing vocoder.

[0004] The invention also relates to a computer program comprising software instructions which, when implemented by a computer, implement such a method.

[0005] The invention relates to the field of speech coding also called "vocoding", voice synthesis and voice transmission.

[0006] Subsequently, according to the present invention "vocoding" and by extension the corresponding device "vocoder" is associated with the contraction of the English words Voice Coding respectively Voice coder, in other words voice coding by an electronic device for processing the sound signal, corresponding substantially to a method according to which, from a decomposition into important properties (eg main spectral components) of an input voice or another sound, synthesizes an associated synthetic voice at the output.

[0007] Digital communications offer a number of significant benefits over analog communications. In addition to being of higher quality, they are often more robust to interference during transmission. However, one of the major problems linked to the very nature of digital communications is the limitation of the data rate. Indeed, the higher the data rate, the more information can be sent, allowing for higher voice quality. In addition, digital transmissions suffer from "binarization" of the received data. Indeed, either the transmission of digital frames is correct and the communication is unchanged, or it is incorrect (i.e. missing frames) and all the information contained in these missing frames at reception is lost.

[0008] Some transmission contexts impose constrained bit rates and the use of codecs (or coder-decoder) whose sound quality is limited. In these contexts, the codecs used are generally standardized such as the MELP codec (from the English Mixed-excitation linear prediction) or the improved MELPe codec (from the English enhanced Mixed-excitation linear prediction) most often associated with a bit rate, for example the MELP 600 corresponding to a MELP codec with a bit rate of 600 bits / s, the MELP 1200, or MELPe 1200, corresponding to a MELP codec, respectively MELPe, with a bit rate of 1200 bits / s, the MELP 2400 or MELPe 2400, corresponding to a MELP codec, respectively MELPe, with a bit rate of 2400 bits / s, etc.

[0009] Codecs are usually optimized to operate according to precise modeling (e.g. LPC (Linear Predictive Coder), MELP, etc.) in terms of extracting parameters from the speech signal.

[0010] The resulting parametric model can be used to synthesize a speech signal. This is called a parametric encoder or vocoder (to clarify that the model applies to the speech signal).

[0011] The quantization / dequantization step allows transmission over a constrained channel by restricting the flow rate. There are therefore two sources of degradation: the approximation of the model, and the quantization of the model parameters.

[0012] The different MELP bitrates correspond to different operating modes. It is the quantization scheme that is optimized for a given model (LPC, MELP, etc.) and a target bitrate. While the bitstreams are obviously different, the information is of the same type (associated with the MELP parameters, for example).

[0013] A bitstream associated with the MELPe 1200 codec works with the MELPe 1200 vocoder and cannot work with a MELPe 2400 vocoder.

[0014] Indeed, a modification of the quantization scheme implies a loss of interoperability: a MELP 1200 is not interoperable with a MELP 2400.

[0015] Currently, the speech synthesis output from existing vocoders, in particular a MELP vocoder (according to all these existing versions (i.e. variants) such as the aforementioned examples MELP 600, MELP 1200, MELPe 1200, MELP 2400, MELPe 2400, etc.) very often presents artifacts.

[0016] To address this, vocoding solutions including AI capabilities have been proposed. However, such solutions are generally very demanding in terms of input parameters, requiring, for example, an input number of around eighty Mel frequencies to provide voice synthesis. In addition, such AI solutions are generally not designed to allow real-time implementation, favoring operation adapted to uses such as text-to-speech transformation or voice assistants. Furthermore, the number of parameters generally associated with the implementation of such AI solutions, in the order of tens of millions of parameters, reduces the possibilities of implementation in embedded hardware with strong memory and computing capacity constraints.

[0017] To summarize, currently, the decrease in the number of input parameters is therefore systematically accompanied by a decrease in the quality of speech synthesis, and conventional low-bitrate speech synthesizers, such as MELP vocoders (according to all these existing versions (i.e. variants) such as the aforementioned examples MELP 600, MELP 1200, MELPe 1200, MELP 2400, MELPe 2400, etc.) cannot compensate for the loss of information linked to the decrease in the number of parameters, while AI speech synthesizers compensate for the loss of information but are often characterized by heavy numerical calculations, as well as by a large number of implementation parameters, these properties being inherent to the fact that AI speech synthesizers learn a priori distribution to compensate for areas without information.

[0018] The aim of the invention is then to improve the sound quality (i.e. audio) of voice syntheses provided by existing vocoders using in particular low bit rate or even very low bit rate codecs used in certain transmission contexts which impose constrained bit rates, such as for example the MELP codec and its variants, while being suitable for being embedded (i.e. integrated) within existing radiocommunication products, and for operating in real time.

[0019] To this end, the invention relates to a device for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder, said post-processing device comprising:

[0020] - a module for receiving the voice synthesis produced by said existing vocoder;

[0021] - an artificial intelligence model including:

[0022] - a projection module configured to project via at least one neural network said voice synthesis produced by said existing vocoder, into a latent space of configurable dimension;

[0023] - a voice reconstruction module configured to reconstruct a corrected voice synthesis, via at least one other neural network, from said projection provided by said projection module;

[0024] - a training module for said artificial intelligence model; said training module being configured to train said artificial intelligence model to correct the speech synthesis artifacts generated by said existing vocoder, from a database comprising a plurality of speech syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference speech signals used as input by said at least one existing vocoder to produce said plurality of speech syntheses produced beforehand, said training module comprising two training tools implemented successively:

[0025] - the first training tool being configured to train said artificial intelligence model according to a first supervised learning phase;

[0026] - the second training tool being configured to re-train by adversarial training, via at least one discriminator, according to a second learning phase, said artificial intelligence model previously trained and provided by the first training tool.

[0027] Such a post-processing device advantageously offers an improvement in the quality of the voice synthesis of an existing vocoder by focusing this time directly on the detection and then the correction of the artifacts of the voice synthesis provided by such existing vocoders, instead of seeking to replace them completely with AI voice synthesizers (i.e. so-called full AI).

[0028] The present invention is further advantageously applicable to several uses, namely increasing the audio quality while remaining at an equivalent bit rate compared to the voice synthesis provided by the existing vocoder, or even gaining bit rate at equivalent quality. Indeed, an improvement in audio quality is capable of being transcribed in terms of gain in bit rate compared to a given existing vocoder, such as a MELP vocoder for example.

[0029] In other words, at the output of an existing vocoder, the present invention proposes to use an artificial intelligence model specifically trained to focus on detecting the position of artifacts and then correcting them.

[0030] Instead of the existing solution based on AI voice synthesizers associated with heavy digital calculations, it is therefore proposed here to keep the classic low-bitrate voice synthesizers and by post-processing according to the invention to correct only the artifacts that they produce by deleting them (i.e. by removing them, which amounts to a denoising task), which significantly lightens the computing load and allows real-time operation.

[0031] To do this, the proposed post-processing device includes an artificial intelligence model which is specifically trained by a first training tool and then re-trained, by adversarial training, by a second training tool.

[0032] According to other advantageous aspects of the invention, the post-processing device comprises one or more of the following characteristics, taken in isolation or in all technically possible combinations: - said at least one predetermined codec used to generate said voice synthesis and said database is the MELP codec, or the improved MELPe codec, and is associated with a bit rate belonging to the group comprising the following bit rates:

[0033] 600 bits / s;

[0034] 1200 bits / s;

[0035] 2400 bits / s, said database being single-speed or multi-speed and / or single-language or multi-language;

[0036] - said first training tool is configured to optimize said artificial intelligence model by simultaneously minimizing four distinct loss functions;

[0037] - said four distinct loss functions are expressed in the following form:

[0038] - Li(x,y) = || -y||i, corresponding to the L1 norm of the difference between the input voice signal x of the existing vocoder and the output y of the artificial intelligence model whose input is connected to the output of said existing vocoder; corresponding to the L2 norm of the difference between the spectrogram S params (x) of the input voice signal of the existing vocoder and the spectrogram s params (y) the output y of the artificial intelligence model whose input is connected to the output of said existing vocoder;

[0039] - L3(x,y) =params E PL2(x,y,params), the combination of the loss function L2 with spectrogram parameters from a predetermined list P of spectrogram params; combination of the loss function L with mel-spectrogram parameters from a predetermined list Q of mel-spectrogram parameters mparams, is the L2 norm of the difference between the mel-MS spectrogrammparams (x) of the input voice signal of the existing vocoder and the MS mel-spectrogram mparams (y) the output y of the artificial intelligence model whose input is connected to the output of said existing vocoder;

[0040] - the second training tool is configured to alternately implement two distinct optimizations, the first optimization minimizing both:

[0041] - said four distinct loss functions associated with said first supervised learning phase, and

[0042] - a first loss function L G associated with said second phase of learning by adverse training, via said at least one discriminator D, with L G (x,y) = \D(y) - 1| ; the second optimization, implemented successively to said first optimization, minimizing a second loss function L Dassociated with said second phase of learning by adverse training, via said at least one discriminator D, with

[0043] - said database is previously processed by dividing said plurality of voice syntheses into two sets:

[0044] - a training data set itself divided into a predetermined number of batches of voice syntheses,

[0045] - a validation data set, said second learning phase comprising a plurality of iterations, during each iteration, said two distinct optimizations being applied, batch by batch, until complete processing of the training data set of said database, said complete processing being repeated until convergence according to a predetermined convergence criterion and determined using the validation data set;

[0046] - said predetermined convergence criterion depends on the PESQ score associated with the corrected voice synthesis via said re-trained artificial intelligence model and / or the result of said loss functions used by said training module;

[0047] - said projection module comprises at least three projection blocks implementing three successive processes:

[0048] - a first projection block, convolutional, configured to take into account both the correlation according to a time dimension and according to a frequency dimension of the data received as input;

[0049] - a second projection block receiving as input the output of the first block, said second block corresponding to a first temporal convolutional neural network capable of taking into account a short and medium term temporal correlation;

[0050] - a third projection block receiving as input the output of the second block, the third block corresponding to a first recurrent gate unit capable of taking into account a long-term temporal correlation; and in which said reconstruction module, the input of which is connected to the output of the projection module, comprises at least four reconstruction blocks implementing four successive processes:

[0051] - a first reconstruction block corresponding to a second temporal convolutional neural network capable of taking into account a short and medium term temporal correlation; - a second reconstruction block receiving as input the output of the first reconstruction block, said second reconstruction block corresponding to a second recurrent gate unit capable of taking into account a long term temporal correlation;

[0052] - a third reconstruction block receiving as input the output of the second reconstruction block, the third reconstruction block corresponding to a third temporal convolutional neural network capable of taking into account short and medium term temporal correlation;

[0053] - a fourth reconstruction block, convolutional, receiving as input the output of the third reconstruction block, the fourth reconstruction block being configured to implement the inverse operation of that implemented by the first projection block and provide a change in scale of said reconstructed and corrected voice synthesis.

[0054] The invention also relates to a method for post-processing speech synthesis produced using a predetermined codec by an existing vocoder, said method being implemented by a post-processing device, said method comprising:

[0055] - receiving the voice synthesis produced by said existing vocoder; and the following steps implemented during an inference phase implemented by an artificial intelligence model of said post-processing device previously trained during a prior training phase, said inference phase comprising:

[0056] - the projection, via at least one neural network, of said voice synthesis produced by said existing vocoder, into a latent space of configurable dimension;

[0057] - the vocal reconstruction of a corrected vocal synthesis, via at least one other neural network, from said projection provided by said projection step; said preliminary training phase of said artificial intelligence model training said artificial intelligence model to correct the vocal synthesis artifacts generated by said existing vocoder, from a database comprising a plurality of vocal syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference vocal signals used as input by said at least one existing vocoder to produce said plurality of vocal syntheses produced beforehand, said training phase said preliminary training phase comprising at least two training steps implemented successively and corresponding respectively to

[0058] - a first supervised learning phase; and - a second learning phase during which said artificial intelligence model previously trained during the first supervised learning phase is re-trained by adversarial training, via at least one discriminator.

[0059] The invention also relates to a computer program comprising software instructions which, when executed by a computer, implement a method of post-processing speech synthesis produced using a predetermined codec by an existing vocoder, as defined above.

[0060] The invention will appear more clearly on reading the description which follows, given solely by way of non-limiting example, and made with reference to the drawings in which:

[0061] [Fig. 1] Figure 1 is a general schematic representation of a device, according to the present invention, for post-processing the speech synthesis produced by means of a predetermined codec by an existing vocoder;

[0062] [Fig. 2] Figure 2 illustrates an example of the architecture of the artificial intelligence model of the post-processing device of Figure 1;

[0063] [Fig. 3] Figure 3 is a flowchart of a method for post-processing speech synthesis produced using a predetermined codec by an existing vocoder implemented by the post-processing device of Figure 1 according to one embodiment.

[0064] In the remainder of the description, the expression "substantially equal to" is understood as a relationship of equality to plus or minus 10%, that is to say with a variation of at most 10%, more preferably as a relationship of equality to plus or minus 5%, that is to say with a variation of at most 5%.

[0065] Furthermore, subsequently, we consider that a neural network comprises an ordered succession of layers of neurons, each of which takes its inputs from the outputs of the previous layer.

[0066] More precisely, each layer consists of neurons taking their inputs from the outputs of the neurons in the previous layer, or from the input variables for the first layer.

[0067] Alternatively, more complex neural network structures can be considered with a layer that can be connected to a layer further away than the immediately preceding layer.

[0068] Each neuron is also associated with an operation, that is, a type of processing, to be performed by the neuron within the corresponding processing layer. Each layer is connected to the other layers by a plurality of synapses. A synaptic weight is associated with each synapse, and each synapse forms a connection between two neurons. It is often a real number, which takes both positive and negative values. In some cases, the synaptic weight is a complex number.

[0069] Each neuron is capable of performing a weighted sum of the value(s) received from the neurons of the previous layer, each value then being multiplied by the respective synaptic weight of each synapse, or link, between said neuron and the neurons of the previous layer, then applying an activation function, typically a non-linear function, to said weighted sum, and delivering as output of said neuron, in particular to the neurons of the following layer connected to it, the value resulting from the application of the activation function. The activation function makes it possible to introduce non-linearity into the processing carried out by each neuron. The sigmoid function, the hyperbolic tangent function, the Heaviside function are examples of activation functions.

[0070] As an optional addition, each neuron is also able to apply, in addition, a multiplicative factor, also called bias, to the output of the activation function, and the value delivered at the output of said neuron is then the product of the bias value and the value from the activation function.

[0071] A fully connected layer of neurons is one in which the neurons in that layer are each connected to all the neurons in the previous layer.

[0072] Such a type of layer is more often referred to as "fully connected" and sometimes referred to as a "dense layer".

[0073] Figure 1 illustrates, in general, an electronic chain for processing an input voice signal x (i.e. input sound signal comprising a human voice) processed by an existing vocoder 10 delivering at output a voice synthesis 11 classically very often presenting artifacts resulting in a reduction in the quality of the voice synthesis 11 at the output of the existing vocoder compared to the input voice signal x. To remedy this, it is proposed according to the present invention to add at the output of the existing vocoder 10 a post-processing device 12.

[0074] As can be seen in FIG. 1, said post-processing device 12 firstly comprises a module 14 for receiving the voice synthesis produced by said existing vocoder 10.

[0075] In other words, the reception module 14 of the post-processing device 12 receives as input the result 11 of the voice synthesis produced by the existing vocoder 10, such a result 11 classically very often presenting artifacts compared to the input voice signal x of the existing vocoder 10.

[0076] Furthermore, the post-processing device 12 also comprises an artificial intelligence model 16 comprising on the one hand a projection module 18 configured to project via at least one neural network said voice synthesis 11 produced by said existing vocoder 10, into a latent space of configurable dimension, and on the other hand a voice reconstruction module 20 configured to reconstruct a corrected voice synthesis y, via at least one other neural network, from said projection provided by said projection module.

[0077] According to the present invention, the post-processing device 12 also comprises a training module 22 of said artificial intelligence model 16.

[0078] According to the present invention, said training module 22 is configured to train said artificial intelligence model 16 to correct the voice synthesis artifacts 11 generated by said existing vocoder 10, from a database DB.

[0079] The DB database includes:

[0080] - a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder, and

[0081] - a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of previously produced voice syntheses.

[0082] The training module 22 comprises two training tools 24 and 26 implemented successively. More precisely, the first training tool 24 is configured to train said artificial intelligence model according to a first supervised learning phase by signal reconstruction, in particular using said DB database comprising, as indicated previously, both a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder, and a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of voice syntheses produced beforehand.

[0083] The second training tool 26 is configured to re-train by adversarial training, via at least one discriminator D, according to a second learning phase, said artificial intelligence model 16 previously trained and provided by the first training tool 24.

[0084] For example, the second training tool 26 uses two discriminators, namely for example a multi-scale discriminator MSD (from the English multi-scale discriminator), an example of which is described in the article by J. Kong et al entitled “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis” NeurIPS 2020, but applied in a context other than that of the present invention, namely to re-train an artificial intelligence model 16 previously trained via supervised learning, and a multi-scale convolutional discriminator MSCD (from the English multi-scale convolution discriminator) as introduced in the article by W. Li et al. entitled “Improving Generative Adversarial Network-based Vocoding throgh Multi-scale Convolution” ACM Transactions on Asian and Low-Resource Language Information Processing, Volume 22, number 9, article no. 230, pages 1 to 10.

[0085] It should be noted that once the learning phase has been carried out, such a training module 22 is no longer used during the inference phase using the artificial intelligence model 16 thus trained.

[0086] According to a particular variant, said at least one predetermined codec used by said existing vocoder 10 to generate said voice synthesis 11 and said database DB is the MELP codec, or the improved MELPe codec, and is associated with a bit rate belonging to the group comprising the following bit rates:

[0087] - 600 bits / s;

[0088] - 1200 bits / s;

[0089] - 2400 bits / s; said DB database being single-bit or multi-bit and / or single-language or multi-language.

[0090] In other words, the artificial intelligence model 16 is capable of being trained for one flow rate or for several flow rates at the same time, and for a single language or several languages ​​at the same time, for example French, English and Arabic, etc.

[0091] In other words, according to this variant, the post-processing device 12 is specifically configured to correct the artifacts of a voice synthesis 11 provided by an existing MELP vocoder 10 (according to all these existing versions (i.e. variants) such as the aforementioned examples MELP 600, MELP 1200, MELPe 1200, MELP 2400, MELPe 2400, etc.).

[0092] As an optional addition, said first training tool 24 is configured to optimize said artificial intelligence model 16 by simultaneously minimizing four distinct loss functions.

[0093] According to a specific variant of this optional complement, the said four distinct loss functions are expressed in the following form:

[0094] - L^xy) = ||% -y||i, corresponding to the norm L1 of the difference between the input voice signal x of the existing vocoder 10 and the output y of the artificial intelligence model 16 whose input is connected to the output of said existing vocoder 10; - L2(x,y, params) = corresponding to the L2 norm of the difference between the spectrogram s params (x) of the input voice signal x of the existing vocoder 10 and the spectrogram s params (y) of the output y of the artificial intelligence model 16 whose input is connected to the output of said existing vocoder 10;

[0095] - L3(x,y) =params E l,L2(x,y,para s), the combination of the loss function L2 with spectrogram parameters from a predetermined list P of spectrogram params;

[0096] - L4(x,y) = ^mparams E qL5x,y,mparams), the combination of the loss function L5 with mel-spectrogram parameters from a predetermined list Q of mel-spectrogram parameters mparams, L2 of the difference between the mel-MS spectrogram mparams (x) of the input voice signal of the existing vocoder and the MS mel-spectrogram mparams (y) of the output y of the artificial intelligence model whose input is connected to the output of said existing vocoder.

[0097] As an optional addition, the second training tool 26 is configured to alternately implement two distinct optimizations, the first optimization minimizing both:

[0098] - said four distinct loss functions associated with said first supervised learning phase, and

[0099] - a first loss function L Gassociated with said second learning phase by adversarial training, via said at least one discriminator D, with Lc x ) = \D(y) - 1| ; the second optimization, implemented successively to said first optimization, minimizing a second loss function L D associated with said second phase of learning by adverse training, via said at least one discriminator D, with L D x,y) = |P(y)| + \D(x) - 1|

[0100] Note that when for example two discriminators are used, such as the aforementioned multi-scale discriminator MSD and the aforementioned multi-scale convolutional discriminator MSCD, these two discriminators are used simultaneously and the loss function L D corresponds to the average of two loss functions L MSD and L MSCD which are respectively associated with them such as L MSD (x,y) = \MSD(y)\ + \MSD(x) - 1|, and

[0101] LMSCD xy) = \MSCD(y)\ + \MSD(x) - 1|.

[0102] More precisely, said at least one discriminator is capable of receiving as input the input voice signal x (i.e. the real source signal with a human voice provided as input to the existing vocoder 10) as well as the output y of the artificial intelligence model 16 previously trained by the first training tool 24, and the first loss function L G allows the discriminator to minimize the difference between these two signals while the second loss function L D allows to find the areas where artifacts, constituting differences between the input voice signal x and the output y of the artificial intelligence model 16, are present.

[0103] The goal of the discriminator is to distinguish the real signal x from the signal y generated by the artificial intelligence model 16, with for example, D(y) = o, if the discriminator detects the signal y generated by the artificial intelligence model 16 and (y) = 1 if the discriminator detects the real signal x.

[0104] The loss function L0(x,y) = |D(y)| + W - 1| aims to locate artifact areas, while the loss function L G (x, y) = \D(y) - 1| aims to fool the discriminator and minimize the differences in these localized areas via the loss function L D .

[0105] In the example of Figure 1, the post-processing device 12 comprises an information processing unit 28 formed for example of a memory 30 and a processor 32 associated with the memory 30.

[0106] In the example of Figure 1, the reception module 14, the artificial intelligence model 16 and the training module 22 are each produced in the form of software, or a software brick, executable by the processor 32. The memory 30 of the post-processing device 12 is then capable of storing reception software, an artificial intelligence model, and training software. The processor 32 is then capable of executing each of the software among the reception software, the artificial intelligence model, the training software.

[0107] In a variant not shown, the reception module 14, the artificial intelligence model 16 and the training module 22 are each produced in the form of a programmable logic component, such as an FPGA (Field Programmable Gate Array), or an integrated circuit, such as an ASIC (Application Specific Integrated Circuit).

[0108] When the post-processing device 12 is produced in the form of one or more software programs, that is to say in the form of a computer program, also called a computer program product, it is furthermore capable of being recorded on a medium, not shown, readable by a computer. The computer-readable medium is for example a medium capable of storing electronic instructions and of being coupled to a bus of a computer system. By way of example, the readable medium is an optical disk, a magneto-optical disk, a ROM memory, a RAM memory, any type of non-volatile memory (for example FLASH or NVRAM) or a magnetic card. On the readable medium is then stored a computer program comprising software instructions. Figure 2 illustrates an example of the architecture of the artificial intelligence model 16 of the post-processing device 12 of Figure 1.

[0109] According to the example of figure 2, said projection module 18 comprises at least three projection blocks implementing three successive processing operations, namely first of all a first projection block 34, convolutional, configured to take into account both the correlation according to a time dimension and according to a frequency dimension of the data received as input.

[0110] The projection module 18 then comprises a second projection block 36 receiving as input the output of the first block 34, said second block 36 corresponding to a first temporal convolutional neural network TCN (from the English temporal convolutional network) capable of taking into account a short and medium term temporal correlation.

[0111] In a particular example, this second boc 36 consists of two convolutions with a kernel size of 1 x 1 to change the channel dimensions, and a dilated depthwise convolution to exploit temporal correlations with low complexity. This TCN block is more efficient for exploring the short and medium term of the time evolution. By "short term" we mean taking into account the time evolution with a distance in time samples less than twenty, by "medium term" we mean taking into account the time evolution with a distance in time samples between twenty and two hundred, and by "long term" we mean taking into account the time evolution with a distance in time samples greater than two hundred.

[0112] The projection module 18 then comprises a third projection block 38 receiving as input the output of the second block 36, the third block 38 corresponding to a first GRU gate recurrent unit (Gate Recurrent Unit) capable of taking into account a long-term temporal correlation.

[0113] As an alternative to the GRU unit, a layer with long short-term memory such as LSTM (Long Short Term Memory) or even more classically a recurrent RNN (Recurrent Neural Network) layer is used.

[0114] Such a third block with the GRU unit is however advantageous because it has only two gates: a reset gate and an update gate, compared to three gates of LSTM. Therefore, the third block with the GRU unit is not too complex and involves fewer parameters compared to the use of a layer with long short-term memory type LSTM. The third block with the GRU unit is very effective in exploring the long term of the temporal evolution, which represents an interesting property in complement compared to the second block 36 with TCN. In other words, the present invention proposes to combine the TCN and GRU in the artificial intelligence model 16 in order to take advantage of their complementarity.

[0115] The reconstruction module 20, the input of which is connected to the output of the projection module 18, comprises, according to the example of FIG. 2, at least four reconstruction blocks implementing four successive processing operations, namely a first reconstruction block 40 corresponding to a second temporal convolutional neural network TCN capable of taking into account a short and medium term temporal correlation.

[0116] The reconstruction module 20 also comprises a second reconstruction block 42 receiving as input the output of the first reconstruction block 40, said second reconstruction block 42 corresponding to a second recurrent unit with GRU gate capable of taking into account a long-term temporal correlation.

[0117] The reconstruction module 20 also comprises a third reconstruction block 44 receiving as input the output of the second reconstruction block 42, the third reconstruction block 44 corresponding to a third temporal convolutional neural network TCN capable of taking into account a short and medium term temporal correlation.

[0118] The reconstruction module 20 also comprises a fourth reconstruction block 46, convolutional, receiving as input the output of the third reconstruction block 44, the fourth reconstruction block 46 being configured to implement the inverse operation of that implemented by the first projection block 34 and provide a change of scale of said reconstructed and corrected voice synthesis.

[0119] Advantageously, the number of parameters associated with the implementation of such an artificial intelligence model 16 remains less than two million parameters, which proves to be less demanding compared to the solutions, mentioned above in relation to the prior art, of vocoding comprising AI artificial intelligence capabilities, and then allows it to be embedded (i.e. integrated) within existing radiocommunication products, and to operate in real time.

[0120] A post-processing method 50 implemented via said post-processing device 12 is subsequently explained with reference to FIG. 3 showing a flowchart of the steps of this method 50.

[0121] Generally, said method 50 firstly comprises a training phase 52 training said artificial intelligence model, previously described in relation to FIGS. 1 and 2, to correct the voice synthesis artifacts generated by said existing vocoder, from the database DB comprising a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of voice syntheses produced beforehand.

[0122] According to a step 54, said database DB is previously processed by dividing Div said plurality of voice syntheses into two sets, namely a training dataset (train dataset) on the one hand, itself divided into a predetermined number of batches of voice syntheses, and a validation dataset (valid dataset).

[0123] Said preliminary training phase 52 comprises at least two training steps 56 and 58 implemented successively and corresponding respectively to a first supervised learning phase 56 E_A1, and to a second learning phase 58 during which said artificial intelligence model previously trained during the first supervised learning phase E_A-, is re-trained by adverse training, via at least one discriminator.

[0124] The first supervised learning phase 56 E_A-, uses, for supervision, each voice synthesis produced beforehand, by means of at least one predetermined codec, by said at least one existing vocoder and the associated reference voice signal used as input by said at least one existing vocoder to produce said voice synthesis.

[0125] During this first phase 56 of supervised learning E_A-, said artificial intelligence model 16 is optimized by simultaneously minimizing four distinct loss functions, in particular the aforementioned loss functions L 1} L2, L3 and L4.

[0126] The second learning phase 58 comprises two steps 60 and 62 of distinct Ch and O2 optimizations implemented alternately.

[0127] The first step 60 of optimization Ch minimizing both:

[0128] - the said four distinct loss functions L 1}L2, L3 and L4 associated with said first phase 56 of supervised learning, and

[0129] - a first loss function L G associated with said second phase of learning by adverse training, via said at least one discriminator D, with L G (x,y) = (y) - l| .

[0130] This first optimization step 60 Ch makes it possible, compared to the first supervised learning phase 56 E_A1, to continue to bring the output y of the artificial intelligence model 16 closer to the input voice signal x of the existing vocoder 10, by minimizing said four distinct loss functions L 1} L2, L3 and L4 while minimizing the number and size of the different areas between x and y detected by the discriminator thanks to L G .

[0131] The second step 62 of optimization O2, implemented successively to said first optimization Oi, minimizes a second loss function L Dassociated with said second phase of learning by adverse training, via said at least one discriminator D, with L D (x,y) = |D(y)| + |Z>(x) + 1|.

[0132] This second step 62 of optimization O2 will make it possible to update the weights of said at least one discriminator D to strengthen it while continuing to detect different zones between x and y.

[0133] As an optional addition, said second learning phase 58 comprises a plurality of iterations I, said two distinct optimizations Ch and O2 being applied during each iteration, batch by batch (i.e. batch by batch), until complete processing, called epoch (from the English epoch) of the training data set of said database DB.

[0134] At each iteration, batch by batch (i.e. batch by batch) the weights of the artificial intelligence model are updated.

[0135] The said complete processing is itself repeated, epoch by epoch, until convergence according to a predetermined convergence criterion, evaluated (i.e. determined), at each epoch (from the English epoch), using the validation data set.

[0136] According to an optional aspect of this optional supplement, said predetermined convergence criterion depends on the PESQ score (i.e. the Perceptual Evaluation of Speech Quality metric) associated with the speech synthesis corrected via said retrained artificial intelligence model and / or the result of said loss functions L 1} L2, L3 and L4 used by said training module.

[0137] In other words, at each epoch, the PESQ score and / or the result of the said loss functions L 1}L2, L3 and L4 used by said training module is determined, and in the event of improvement compared to the previous epoch, said second learning phase 58 is repeated, batch by batch, during a following epoch. Otherwise (i.e. absence of improvement), the training phase 52 is stopped and provides a trained artificial intelligence model M_E.

[0138] Once the trained artificial intelligence model M_E has been obtained at the end of the training phase 52, the method 50 comprises, in real time, a step 64 of receiving R the voice synthesis E produced, in particular in real time, by said existing vocoder.

[0139] Then the method 50 comprises an inference phase 66 using the trained artificial intelligence model 16 M_E, and comprising at least one projection step 68 P, via at least one neural network of said projection module 18 of the trained artificial intelligence model M_E, of said voice synthesis E produced by said existing vocoder in a latent space of parameterizable dimension, then at least one step 70 of voice reconstruction RV of a corrected voice synthesis, via at least one other neural network of the voice reconstruction module 20 of the trained artificial intelligence model M_E, from said projection in said latent space of parameterizable dimension provided by said projection step.

[0140] According to an aspect not shown, said voice synthesis E used in the inference phase 66 is suitable for subsequently being integrated into the database DB, in particular in the validation data set, to subsequently re-train (offline) said artificial intelligence model.

[0141] In other words, the artificial intelligence model is capable of being continuously improved via the alternation of the training phase 52 and at least one inference phase 66, the voice syntheses E received during successive inference phases 66 being capable of being integrated into the database DB for subsequent training.

[0142] Those skilled in the art will understand that the invention is not limited to the embodiments described, nor to the particular examples of the description, the embodiments and variants mentioned above being suitable for being combined with each other to generate new embodiments of the invention.

[0143] The present invention thus makes it possible to propose a "corrector" by post-processing the output of an existing vocoder. For example, the correction result obtained on the MELP 600 and MELP 1200 is substantial, with in particular an increase in the PESQ score (i.e. the PESQ metric) of 0 to 10% for the MELP 600, the MELP 1200 or even the MELP 2400, a gain of 6dB in range, while also allowing more conferences in parallel.

[0144] In other words, the present invention makes it possible to improve the synthesis produced by an existing vocoder, for example the MELP vocoder, with a significant margin of improvement in audio quality. This solution is also suitable for being embedded in radiocommunication products and advantageously its execution is suitable for being implemented in real time and with very low latency.

[0145] This solution opens a new direction for improving MELP, which is a popular coding with many applications in. The proposed solution allows for better audio quality compared to MELP, with a significant improvement in the number of parameters and computation time compared to vocoding solutions including AI capabilities (which implement global vocoding and not a correction of an existing vocoding).

Claims

CLAIMS 1. Post-processing device (12) for the voice synthesis (11) produced by means of a predetermined codec by an existing vocoder (10), said post-processing device (12) comprising: - a reception module (14) for the voice synthesis produced by said existing vocoder; - an artificial intelligence model (16) comprising: - a projection module (18) configured to project via at least one neural network said voice synthesis produced by said existing vocoder, into a latent space of configurable dimension; - a voice reconstruction module (20) configured to reconstruct a corrected voice synthesis, via at least one other neural network, from said projection provided by said projection module; - a training module (22) for said artificial intelligence model; said post-processing device being characterized in that said training module is configured to train said artificial intelligence model to correct the speech synthesis artifacts generated by said existing vocoder, from a database (DB) comprising a plurality of speech syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference speech signals used as input by said at least one existing vocoder to produce said plurality of speech syntheses produced beforehand, said training module (22) comprising two training tools implemented successively: - the first training tool (24) being configured to train said artificial intelligence model (16) according to a first supervised learning phase; - the second training tool (26) being configured to re-train by adverse training, via at least one discriminator (D), according to a second learning phase, said artificial intelligence model previously trained and provided by the first training tool, said projection module comprising at least three projection blocks implementing three successive processing operations: - a first projection block (34), convolutional, configured to take into account both the correlation according to a time dimension and according to a frequency dimension of the data received as input; - a second projection block (36) receiving as input the output of the first block, said second block corresponding to a first temporal convolutional neural network capable of taking into account a short and medium term temporal correlation; - a third projection block (38) receiving as input the output of the second block, the third block corresponding to a first recurrent gate unit capable of taking into account a long-term temporal correlation; said reconstruction module (20), the input of which is connected to the output of the projection module (18), comprising at least four reconstruction blocks implementing four successive processes: - a first reconstruction block (40) corresponding to a second temporal convolutional neural network capable of taking into account a short and medium term temporal correlation; - a second reconstruction block (42) receiving as input the output of the first reconstruction block (40), said second reconstruction block (42) corresponding to a second recurrent gate unit capable of taking into account a long-term temporal correlation; - a third reconstruction block (44) receiving as input the output of the second reconstruction block (42), the third reconstruction block (44) corresponding to a third temporal convolutional neural network capable of taking into account a short and medium term temporal correlation; - a fourth convolutional reconstruction block (46) receiving as input the output of the third reconstruction block (44), the fourth reconstruction block (46) being configured to implement the inverse operation of that implemented by the first projection block (34) and provide a change in scale of said reconstructed and corrected voice synthesis.

2. Device (12) according to claim 1, wherein said at least one predetermined codec used to generate said voice synthesis and said database is the MELP codec, or the improved MELPe codec, and is associated with a rate belonging to the group comprising the following rates: 600 bits / s; 1200 bits / s; 2400 bits / s, said database being single-bit or multi-bit and / or single-language or multi-language.

3. Device (12) according to claim 1 or 2, wherein said first training tool (24) is configured to optimize said artificial intelligence model (16) by simultaneously minimizing four distinct loss functions.

4. Device (12) according to claim 3, wherein said four distinct loss functions are expressed in the following form: - L^xy) = ||% -y||i, corresponding to the L1 norm of the difference between the input voice signal x of the existing vocoder and the output y of the artificial intelligence model whose input is connected to the output of said existing vocoder; - L2(x,y, params) = corresponding to the L2 norm of the difference between the spectrogram s params(x) of the input voice signal x of the existing vocoder and the spectrogram s params (y) the output y of the artificial intelligence model whose input is connected to the output of said existing vocoder; - I ,L2(X, y, params , the combination of the L2 loss function with spectrogram parameters from a predetermined list P of spectrogram parameters; - L4(x,y) =mparams e QL5(x,y,mparams), the combination of the loss function L5 with mel-spectrogram parameters from a predetermined list Q of mel-spectrogram mparams parameters, L2 of the difference between the mel-MS spectrogram mparams (x) of the input speech signal x of the existing vocoder and the mel-spectrogram MS mparams (y) of the output y of the artificial intelligence model whose input is connected to the output of said existing vocoder.

5. Device (12) according to claim 4, in which the second training tool (26) is configured to alternately implement two distinct optimizations, the first optimization minimizing both: - said four distinct loss functions associated with said first supervised learning phase, and - a first loss function L G associated with said second phase of learning by adverse training, via said at least one discriminator D, with L G (x,y) = \D(y) - 1| ; the second optimization, implemented successively to said first optimization, minimizing a second loss function L D associated with the said second phase learning by adversarial training, via said at least one discriminator D, with L D (x,y) = |D(y)| + \D(x) + 1|.

6. Device according to claim 5, wherein said database is pre-processed by dividing said plurality of voice syntheses into two sets: - a training data set itself divided into a predetermined number of batches of voice syntheses, - a validation data set, said second learning phase comprising a plurality of iterations, during each iteration, said two distinct optimizations being applied, batch by batch, until complete processing of the training data set of said database, said complete processing being repeated until convergence according to a predetermined convergence criterion determined using the validation data set.

7. Device according to claim 6, wherein said predetermined convergence criterion depends on the PESQ score associated with the corrected voice synthesis via said re-trained artificial intelligence model and / or the result of said loss functions used by said training module.

8. Method (50) for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder, said method being implemented by a post-processing device according to any one of the preceding claims, said method comprising: - receiving (64) the voice synthesis produced by said existing vocoder; and the following steps implemented during an inference phase (66) implemented by an artificial intelligence model of said post-processing device previously trained during a prior training phase (52), said inference phase (66) comprising: - the projection (68), via at least one neural network, of said voice synthesis produced by said existing vocoder, into a latent space of configurable dimension; - the vocal reconstruction (70) of a corrected vocal synthesis, via at least one other neural network, from said projection provided by said projection step; said preliminary training phase (52) of said artificial intelligence model training said artificial intelligence model to correct the vocal synthesis artifacts generated by said existing vocoder, from a database (DB) comprising a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of voice syntheses produced beforehand, said training phase said preliminary training phase (52) comprising at least two training steps implemented successively and corresponding respectively to - a first supervised learning phase (56); and - a second learning phase (58) during which said artificial intelligence model previously trained during the first supervised learning phase is re-trained by adverse training, via at least one discriminator, said projection comprising three successive treatments: - a first projection processing, via a first convolutional projection block, taking into account both the correlation according to a time dimension and according to a frequency dimension of the data received as input; - a second projection processing, via a second projection block receiving as input the output of the first block, and corresponding to a first temporal convolutional neural network, taking into account a short and medium term temporal correlation; - a third projection processing, via a third projection block receiving as input the output of the second block and corresponding to a first recurrent gated unit, taking into account a long-term temporal correlation; said voice reconstruction successive to said projection, comprising four successive processing operations: - a first voice reconstruction processing via a first reconstruction block corresponding to a second temporal convolutional neural network, taking into account a short and medium term temporal correlation; - a second voice reconstruction processing via a second reconstruction block receiving as input the output of the first reconstruction block, said second reconstruction block corresponding to a second gated recurrent unit, taking into account a long-term temporal correlation; - a third voice reconstruction processing via a third reconstruction block receiving as input the output of the second reconstruction block, the third reconstruction block corresponding to a third temporal convolutional neural network, taking into account a short and medium term temporal correlation; - a fourth voice reconstruction processing via a fourth convolutional reconstruction block, receiving as input the output of the third reconstruction block, corresponding to the inverse operation of that implemented by the first projection block and providing a change of scale of said reconstructed and corrected voice synthesis.

9. A computer program product, comprising software instructions which, when executed by a computer, implement a method according to claim 8.

Images