Device and method for post-processing the voice synthesis produced by means of a predetermined codec by an existing vocoder, and related computer program
Patent Information
- Application Number
- AE202603239
- Authority / Receiving Office
- AE · AE
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-03-28
- Filing Date
- 2025-03-25
Smart Images
Figure ABST_ABST
Abstract
Description
TITLE: Device and method for post-processing the voice synthesis produced by means of a predetermined codec by an existing vocoder, and related computer program The present invention relates to a device for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder.The present invention also relates to a method for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder.The invention also relates to a computer program comprising software instructions which, when implemented by a computer, implement such a method. The invention relates to the field of speech coding also referred to as “vocoding”, voice synthesis, and voice transmission.Hereinafter, according to the present invention, “vocoding” and, by extension, the corresponding “vocoder” device is associated with the contraction of the English words Voice Coding and Voice coder, respectively, in other words voice coding by an electronic sound signal processing device, substantially corresponding to a method according to which, from a decomposition into important properties (e.g. main spectral components) of an input voice or another sound, an associated synthetic voice is synthesized at the output.Digital communications have a significant number of benefits compared with analog communications. In addition to being of better quality, they are often more robust to interfering noise during transmission. However, one of the major problems related to the very nature of digital communications is the bitrate limitation. Indeed, the higher the bitrate, the more it is possible to send a large amount of information enabling greater voice quality. Furthermore, digital transmissions suffer from a “binarization” of the received data. Indeed, either the transmission of the digital frames is correct and the communication is unchanged, or it is incorrect (i.e. missing frames) and all the information contained in these frames missing at reception is lost.Certain transmission contexts impose constrained bitrates and the use of codecs (or coder-decoder, from the English coder-decoder) whose sound quality is limited. In these contexts, the codecs used are generally standardized, such as the MELP codec (from the English Mixed-excitation linear prediction) or the enhanced MELPe codec (from the English enhanced Mixed-excitation linear prediction) most often associated with a bitrate, for example MELP 600 corresponding to a MELP codec with a bitrate of 600 bits / s, MELP 1200, or MELPe 1200, corresponding to a MELP codec, respectively MELPe, with a bitrate of 1200 bits / s, MELP 2400 or MELPe 2400, corresponding to a MELP codec, respectively MELPe, with a bitrate of 2400 bits / s, etc.The codecs are generally optimized to operate according to a precise modeling (for example LPC (from the English Linear Predictive Coder), MELP, etc.) with regard to the extraction of the speech signal parameters.The obtained parametric model can be used to synthesize a speech signal. This is referred to as a parametric coder or a vocoder (to specify that the model applies to the speech signal).The quantization / dequantization step enables transmission over a constrained channel by restricting the bitrate. There are therefore two sources of degradation: model approximation, and quantization of the model parameters.The different MELP bitrates correspond to different operating modes. It is the quantization scheme that is optimized for a given model (LPC, MELP, …) and a target bitrate. While the bitstreams are obviously different, the information is of the same type (associated with MELP parameters for example).A bitstream associated with the MELPe 1200 codec operates with the MELPe 1200 vocoder and cannot operate with a MELPe 2400 vocoder.Indeed, a modification of the quantization scheme entails a loss of interoperability: a MELP 1200 is not interoperable with a MELP 2400.Currently, the voice synthesis output by existing vocoders, in particular a MELP vocoder (according to all these existing versions (i.e. variants) such as the aforementioned examples MELP 600, MELP 1200, MELPe 1200, MELP 2400, MELPe 2400, etc.) very often has artifacts.To remedy this, vocoding solutions comprising artificial intelligence AI capabilities have been proposed. However, such solutions are generally very demanding in terms of input parameters, requiring, for example, as input a number on the order of eighty Mel frequencies to provide voice synthesis. Furthermore, such AI solutions are generally not designed to enable real-time implementation, favoring operation adapted to text-to-speech (from the English text-to-speech) or voice assistant uses. Moreover, the number of parameters generally associated with the implementation of such AI solutions, on the order of tens of millions of parameters, reduces the possibilities of implementations in embedded hardware with strong memory and computing capacity constraints.To summarize, currently, a reduction in the number of input parameters is therefore systematically accompanied by a reduction in voice synthesis quality, and conventional low-bitrate voice synthesizers, such as MELP vocoders (according to all these existing versions (i.e. variants) such as the aforementioned examples MELP 600, MELP 1200, MELPe 1200, MELP 2400, MELPe 2400, etc.) cannot compensate for the loss of information related to the reduction in the number of parameters, whereas AI voice synthesizers compensate for the loss of information but are often characterized by heavy numerical computations, as well as by a significant number of implementation parameters, these properties being inherent in the fact that AI voice synthesizers learn an a priori distribution to compensate for areas without information.The purpose of the invention is thus to improve the sound (i.e. audio) quality of voice syntheses provided by existing vocoders using in particular low-bitrate or even very-low-bitrate codecs used in certain transmission contexts that impose constrained bitrates, such as for example the MELP codec and its variants, while being suitable for being embedded (i.e. integrated) within existing radiocommunication products, and for operating in real time.For this purpose, the invention relates to a device for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder, said post-processing device comprising:- a module for receiving the voice synthesis produced by said existing vocoder;- an artificial intelligence model comprising:- a projection module configured to project, via at least one neural network, said voice synthesis produced by said existing vocoder, into a latent space of parameterizable dimension;- a voice reconstruction module configured to reconstruct corrected voice synthesis, via at least one other neural network, from said projection provided by said projection module;- a training module for said artificial intelligence model;said training module being configured to train said artificial intelligence model to correct the voice synthesis artifacts generated by said existing vocoder, from a database comprising a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of voice syntheses produced beforehand,said training module comprising two training tools implemented successively:- the first training tool being configured to train said artificial intelligence model according to a first supervised learning phase;- the second training tool being configured to retrain, by adversarial training, via at least one discriminator, according to a second learning phase, said artificial intelligence model previously trained and provided by the first training tool.Such a post-processing device advantageously provides an improvement in the quality of the voice synthesis of an existing vocoder by this time focusing directly on the detection and then the correction of the artifacts of the voice synthesis provided by such existing vocoders, instead of seeking to replace them completely with AI voice synthesizers (i.e. referred to as full AI).The present invention is furthermore advantageously applicable to several uses, namely increasing audio quality while remaining at an equivalent bitrate relative to the voice synthesis provided by the existing vocoder, or alternatively gaining bitrate at equivalent quality. Indeed, an improvement in audio quality is capable of being transcribed in terms of bitrate gain relative to a given existing vocoder, such as a MELP vocoder for example.In other words, at the output of an existing vocoder, the present invention proposes using an artificial intelligence model specifically trained to focus on detecting the position of artifacts and then on correcting them.Instead of the existing solution based on AI voice synthesizers associated with heavy numerical computations, it is therefore proposed here to retain conventional low-bitrate voice synthesizers and, by post-processing according to the invention, to correct only the artifacts that they produce by removing them (i.e. removing them, which amounts to a denoising task), which substantially reduces the computational load and enables real-time operation.To this end, the proposed post-processing device comprises an artificial intelligence model that is specifically trained by a first training tool and then retrained, by adversarial training, by a second training tool.According to other advantageous aspects of the invention, the post-processing device comprises one or more of the following features, taken individually or according to all technically possible combinations:- said at least one predetermined codec used to generate said voice synthesis and said database is the MELP codec, or alternatively the enhanced MELPe codec, and is associated with a bit rate belonging to the group comprising the following bit rates:600 bits / s;1200 bits / s;2400 bits / s,said database being single-rate or multi-rate and / or monolingual or multilingual;- said first training tool is configured to optimize said artificial intelligence model by simultaneously minimizing four distinct loss functions;- said four distinct loss functions are expressed in the following form:- , corresponding to the L1 norm of the difference between the input voice signal of the existing vocoder and the output of the artificial intelligence model, the input of which is connected to the output of said existing vocoder;- , corresponding to the L2 norm of the difference between the spectrogram of the input voice signal of the existing vocoder and the spectrogram of the output of the artificial intelligence model, the input of which is connected to the output of said existing vocoder;- , the combination of the loss function with spectrogram parameters from a predetermined list P of spectrogram parameters ;- , the combination of the loss function with mel-spectrogram parameters from a predetermined list Q of mel-spectrogram parameters , wherein , is the L2 norm of thedifference between the mel-spectrogram of the input voice signal of the existing vocoder and the mel-spectrogram of the output of the artificial intelligence model, the input of which is connected to the output of said existing vocoder;- the second training tool is configured to alternatively implement two distinct optimizations,the first optimization minimizing both:- said four distinct loss functions associated with said first supervised training phase, and- a first loss function associated with said second adversarial training phase, via said at least one discriminator D, with ;the second optimization, implemented successively to said first optimization, minimizing a second loss function associated with said second adversarial training phase, via said at least one discriminator D, with ;- said database is previously processed by dividing said plurality of voice syntheses into two sets:- a training data set itself divided into a predetermined number of batches of voice syntheses,- a validation data set,said second training phase comprising a plurality of iterations,during each iteration, said two distinct optimizations being applied, batch by batch, until complete processing of the training data set of said database,said complete processing being repeated until convergence according to a predetermined convergence criterion and determined using the validation data set;- said predetermined convergence criterion depends on the PESQ score associated with the voice synthesis corrected via said retrained artificial intelligence model and / or on the result of said loss functions used by said training module;- said projection module comprises at least three projection blocks implementing three successive processings:- a first, convolutional, projection block configured to take into account both correlation along a temporal dimension and along a frequency dimension of the data received at the input;- a second projection block receiving at the input the output of the first block, said second block corresponding to a first temporal convolutional neural network suitable for taking into account short- and medium-term temporal correlation;- a third projection block receiving at the input the output of the second block, the third block corresponding to a first gated recurrent unit suitable for taking into account long-term temporal correlation;and wherein said reconstruction module, the input of which is connected to the output of the projection module, comprises at least four reconstruction blocks implementing four successive processings:- a first reconstruction block corresponding to a second temporal convolutional neural network suitable for taking into account short- and medium-term temporal correlation;- a second reconstruction block receiving at the input the output of the first reconstruction block, said second reconstruction block corresponding to a second gated recurrent unit suitable for taking into account long-term temporal correlation;- a third reconstruction block receiving as input the output of the second reconstruction block, the third reconstruction block corresponding to a third temporal convolutional neural network suitable for taking into account a short- and medium-term temporal correlation;- a fourth, convolutional reconstruction block, receiving as input the output of the third reconstruction block, the fourth reconstruction block being configured to implement the inverse operation of that implemented by the first projection block and to provide a scaling of said reconstructed and corrected voice synthesis.The invention also relates to a method for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder, said method being implemented by a post-processing device, said method comprising:- receiving the voice synthesis produced by said existing vocoder;and the following steps implemented during an inference phase implemented by an artificial intelligence model of said post-processing device previously trained during a prior training phase,said inference phase comprising:- projecting, via at least one neural network, said voice synthesis produced by said existing vocoder, into a latent space of parameterizable dimension;- voice reconstruction of a corrected voice synthesis, via at least one other neural network, from said projection provided by said projection step;said prior training phase of said artificial intelligence model training said artificial intelligence model to correct the voice synthesis artifacts generated by said existing vocoder, from a database comprising a plurality of voice syntheses previously produced, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of previously produced voice syntheses, said training phasesaid prior training phase comprising at least two training steps implemented successively and corresponding respectively to- a first supervised learning phase; and- a second learning phase during which said artificial intelligence model previously trained during the first supervised learning phase is re-trained by adversarial training, via at least one discriminator.The invention also relates to a computer program comprising software instructions which, when they are executed by a computer, implement a method for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder, as defined above. The invention will appear more clearly upon reading the following description, given solely by way of non-limiting example, and made with reference to the drawings in which:[Fig. 1] Figure 1 is a general schematic representation of a device, according to the present invention, for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder;[Fig. 2] Figure 2 illustrates an example of an architecture of the artificial intelligence model of the post-processing device of Figure 1;[Fig. 3] Figure 3 is a flowchart of a method for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder implemented by the post-processing device of Figure 1 according to one embodiment. In the remainder of the description, the expression “substantially equal to” is understood as an equality relationship to within plus or minus 10%, that is to say with a variation of at most 10%, more preferably still as an equality relationship to within plus or minus 5%, that is to say with a variation of at most 5%.Moreover, hereinafter, it is considered that a neural network comprises an ordered succession of neuron layers, each of which takes its inputs from the outputs of the preceding layer.More precisely, each layer comprises neurons taking their inputs from the outputs of the neurons of the preceding layer, or from the input variables for the first layer.Alternatively, more complex neural network structures may be contemplated with a layer which may be connected to a layer more distant than the immediately preceding layer.Associated with each neuron is also an operation, that is to say a type of processing, to be performed by said neuron within the corresponding processing layer.Each layer is connected to the other layers by a plurality of synapses. A synaptic weight is associated with each synapse, and each synapse forms a connection between two neurons. It is often a real number, which takes both positive and negative values. In some cases, the synaptic weight is a complex number.Each neuron is suitable for performing a weighted sum of the value(s) received from the neurons of the preceding layer, each value then being multiplied by the respective synaptic weight of each synapse, or connection, between said neuron and the neurons of the preceding layer, then for applying an activation function, typically a non-linear function, to said weighted sum, and for delivering at the output of said neuron, in particular to the neurons of the next layer that are connected thereto, the value resulting from the application of the activation function. The activation function makes it possible to introduce a non-linearity into the processing performed by each neuron. The sigmoid function, the hyperbolic tangent function, and the Heaviside function are examples of an activation function.As an optional addition, each neuron is also capable of furthermore applying a multiplicative factor, also called bias, to the output of the activation function, and the value delivered at the output of said neuron is then the product of the bias value and the value resulting from the activation function.A fully connected neuron layer is a layer in which the neurons of said layer are each connected to all the neurons of the preceding layer.Such a type of layer is more often referred to by the English term “fully connected”, and is sometimes designated by the name “dense layer”. Figure 1 illustrates, generally, an electronic chain for processing an input voice signal x (i.e. input audio signal comprising a human voice) processed by an existing vocoder 10 delivering as output a voice synthesis 11 classically very often having artifacts resulting in a decrease in the quality of the voice synthesis 11 at the output of the existing vocoder compared with the input voice signal x. To remedy this, it is proposed according to the present invention to add, at the output of the existing vocoder 10, a post-processing device 12.As is visible in Figure 1, said post-processing device 12 first comprises a module 14 for receiving the voice synthesis produced by said existing vocoder 10.In other words, the receiving module 14 of the post-processing device 12 receives as input the result 11 of the voice synthesis produced by the existing vocoder 10, such a result 11 classically very often having artifacts compared with the input voice signal x of the existing vocoder 10.Furthermore, the post-processing device 12 also comprises an artificial intelligence model 16 comprising, on the one hand, a projection module 18 configured to project, via at least one neural network, said voice synthesis 11 produced by said existing vocoder 10, into a latent space of parameterizable dimension, and, on the other hand, a voice reconstruction module 20 configured to reconstruct a corrected voice synthesis y, via at least one other neural network, from said projection provided by said projection module.According to the present invention, the post-processing device 12 also comprises a training module 22 of said artificial intelligence model 16.According to the present invention, said training module 22 is configured to train said artificial intelligence model 16 to correct the voice synthesis artifacts 11 generated by said existing vocoder 10, from a DB database (from the English database).The DB database comprises:- a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder, and- a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of voice syntheses produced beforehand.The training module 22 comprises two training tools 24 and 26 implemented successively. More precisely, the first training tool 24 is configured to train said artificial intelligence model according to a first supervised learning phase by signal reconstruction, using in particular said DB database comprising, as indicated previously, both a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder, and a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of voice syntheses produced beforehand.The second training tool 26 is configured to re-train, by adversarial training, via at least one discriminator D, according to a second learning phase, said artificial intelligence model 16 previously trained and provided by the first training tool 24.For example, the second training tool 26 uses two discriminators, namely for example a multi-scale discriminator MSD (from the English multi-scale discriminator), an example of which is described in the article by J. Kong et al. entitled “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Voice synthesis” NeurIPS 2020, but applied in a context other than that of the present invention, namely to re-train an artificial intelligence model 16 previously trained via supervised learning, and a multi-scale convolutional discriminator MSCD (from the English multi-scale convolution discriminator) as introduced in the article by W. Li et al. entitled “Improving Generative Adversarial Network-based Vocoding throgh Multi-scale Convolution” ACM Transactions on Asian and Low-Resource Language Information Processing, Volume 22, issue 9, article No. 230, pages 1 to 10.It should be noted that once the learning phase has been carried out, such a training module 22 is no longer used during the inference phase using the artificial intelligence model 16 thus trained.According to a particular variant, said at least one predetermined codec used by said existing vocoder 10 to generate said voice synthesis 11 and said DB database is the MELP codec, or alternatively the enhanced MELPe codec, and is associated with a bit rate belonging to the group comprising the following bit rates:- 600 bits / s;- 1200 bits / s;- 2400 bits / s;said DB database being single-bit-rate or multi-bit-rate and / or monolingual or multilingual.In other words, the artificial intelligence model 16 is suitable for being trained for one bit rate or for several bit rates at once, and for a single language or several languages at once, for example French, English and Arabic, etc.In other words, according to this variant, the post-processing device 12 is specifically configured to correct the artifacts of a voice synthesis 11 provided by an existing MELP vocoder 10 (according to all these existing versions (i.e. variants) such as the aforementioned examples MELP 600, MELP 1200, MELPe 1200, MELP 2400, MELPe 2400, etc.).As an optional addition, said first training tool 24 is configured to optimize said artificial intelligence model 16 by simultaneously minimizing four distinct loss functions.According to a specific variant of this optional addition, said four distinct loss functions are expressed in the following form:, corresponding to the L1 norm of the difference between the input voice signal of the existing vocoder 10 and the output of the artificial intelligence model 16, the input of which is connected to the output of said existing vocoder 10;, corresponding to the L2 norm of the difference between the spectrogram of the input voice signal of the existing vocoder 10 and the spectrogram of the output of the artificial intelligence model 16, the input of which is connected to the output of said existing vocoder 10;- , the combination of the loss function with spectrogram parameters from a predetermined list P of spectrogram parameters ;- , the combination of the loss function with mel-spectrogram parameters from a predetermined list Q of mel-spectrogram parameters , where , is the L2 norm of thedifference between the mel-spectrogram of the input voice signal of the existing vocoder and the mel-spectrogram of the output of the artificial intelligence model, the input of which is connected to the output of said existing vocoder.As an optional addition, the second training tool 26 is configured to alternatively implement two distinct optimizations,the first optimization minimizing both:- said four distinct loss functions associated with said first supervised learning phase, and- a first loss function associated with said second learning phase by adversarial training, via said at least one discriminator D, with ;the second optimization, implemented successively to said first optimization, minimizing a second loss function associated with said second learning phase by adversarial training, via said at least one discriminator D, with It should be noted that when, for example, two discriminators are used, such as the aforementioned multi-scale discriminator MSD and the aforementioned multi-scale convolutional discriminator MSCD, these two discriminators are used simultaneously and the loss function corresponds to the average of two loss functions and respectively associated therewith, such as , and More precisely, said at least one discriminator is suitable for receiving as input the input voice signal x (i.e. the real source signal with a human voice provided as input to the existing vocoder 10) as well as the output of the artificial intelligence model 16 previously trained by the first training tool 24, and the first loss function enables the discriminator to minimize the difference between these two signals while the second loss function enables finding the areas where the artifacts, constituting differences between the input voice signal x and the output of the artificial intelligence model 16, are present.The purpose of the discriminator is to distinguish the real signal x from the signal y generated by the artificial intelligence model 16, with, for example, , if the discriminator detects the signal y generated by the artificial intelligence model 16 and if the discriminator detects the real signal x.The loss function aims to locate the artifact areas, while the loss function aims to deceive the discriminator and to minimize the differences in these areas located via the loss function .In the example of Figure 1, the post-processing device 12 comprises an information processing unit 28 formed, for example, by a memory 30 and a processor 32 associated with the memory 30.In the example of Figure 1, the receiving module 14, the artificial intelligence model 16 and the training module 22 are each implemented in the form of software, or of a software component, executable by the processor 32. The memory 30 of the post-processing device 12 is then capable of storing receiving software, an artificial intelligence model, and training software. The processor 32 is then capable of executing each of the software items among the receiving software, the artificial intelligence model, the training software.In an alternative not shown, the receiving module 14, the artificial intelligence model 16 and the training module 22 are each implemented in the form of a programmable logic component, such as an FPGA (from the English Field Programmable Gate Array), or alternatively an integrated circuit, such as an ASIC (from the English Application Specific Integrated Circuit).When the post-processing device 12 is implemented in the form of one or more software programs, that is to say in the form of a computer program, also called a computer program product, it is furthermore capable of being recorded on a computer-readable medium, not shown. The computer-readable medium is for example a medium capable of storing electronic instructions and of being coupled to a bus of a computer system. By way of example, the readable medium is an optical disc, a magneto-optical disc, a ROM memory, a RAM memory, any type of non-volatile memory (for example FLASH or NVRAM) or a magnetic card. A computer program comprising software instructions is then stored on the readable medium.Figure 2 illustrates an example of architecture of the artificial intelligence model 16 of the post-processing device 12 of Figure 1.According to the example of Figure 2, said projection module 18 comprises at least three projection blocks implementing three successive processing operations, namely first a first convolutional projection block 34, configured to take into account both the correlation along a time dimension and along a frequency dimension of the data received at the input.The projection module 18 then comprises a second projection block 36 receiving at the input the output of the first block 34, said second block 36 corresponding to a first temporal convolutional neural network TCN (from the English temporal convolutional network) suitable for taking into account a short- and medium-term temporal correlation.According to one particular example, this second boc 36 consists of two convolutions with a kernel of size 1 × 1 for changing the channel dimensions, and of a dilated depthwise convolution (from the English dilated depthwise convolution) for exploiting temporal correlations with low complexity. This TCN block is more effective for exploring the short term and the medium term of the temporal evolution. By “short term” is meant taking into account the temporal evolution with a distance in temporal samples of less than twenty, by “medium term” is meant taking into account the temporal evolution with a distance in temporal samples comprised between twenty and two hundred, and by “long term” is meant taking into account the temporal evolution with a distance in temporal samples greater than two hundred.The projection module 18 then comprises a third projection block 38 receiving at the input the output of the second block 36, the third block 38 corresponding to a first gated recurrent unit GRU (from the English Gate Recurrent Unit) suitable for taking into account a long-term temporal correlation.As an alternative to the GRU unit, a long short-term memory layer of the LSTM type (from the English Long Short Term Memory) or alternatively, more conventionally, a recurrent RNN layer (from the English Recurrent Neural Network) is used.Such a third block with the GRU unit is however advantageous because it has only two gates: a reset gate and an update gate, as compared with three LSTM gates. Consequently, the third block with the GRU unit is not overly complex and involves fewer parameters as compared with the use of a long short-term memory layer of the LSTM type. The third block with the GRU unit is very effective for exploring the long term of the temporal evolution, which constitutes an interesting property in addition as compared with the second block 36 with TCN.In other words, the present invention proposes combining TCNs and GRUs in the artificial intelligence model 16 in order to take advantage of their complementarity.The reconstruction module 20, the input of which is connected to the output of the projection module 18, comprises, according to the example of Figure 2, at least four reconstruction blocks implementing four successive processing operations, namely a first reconstruction block 40 corresponding to a second temporal convolutional neural network TCN suitable for taking into account a short- and medium-term temporal correlation.The reconstruction module 20 also comprises a second reconstruction block 42 receiving at the input the output of the first reconstruction block 40, said second reconstruction block 42 corresponding to a second gated recurrent unit GRU suitable for taking into account a long-term temporal correlation.The reconstruction module 20 also comprises a third reconstruction block 44 receiving at the input the output of the second reconstruction block 42, the third reconstruction block 44 corresponding to a third temporal convolutional neural network TCN suitable for taking into account a short- and medium-term temporal correlation.The reconstruction module 20 also comprises a fourth convolutional reconstruction block 46, receiving at the input the output of the third reconstruction block 44, the fourth reconstruction block 46 being configured to implement the inverse operation of that implemented by the first projection block 34 and to provide a change of scale of said reconstructed and corrected voice synthesis.Advantageously, the number of parameters associated with the implementation of such an artificial intelligence model 16 remains below two million parameters, which proves to be less resource-intensive as compared with the solutions, mentioned above in relation to the prior art, of vocoding comprising artificial intelligence AI capabilities, and thus enables it to be embedded (i.e. integrated) within existing radiocommunication products, and to operate in real time.A post-processing method 50 implemented via said post-processing device 12 is subsequently explained with reference to Figure 3 presenting a flowchart of the steps of this method 50.In general, said method 50 first comprises a training phase 52 training said artificial intelligence model, previously described in relation to Figures 1 and 2, to correct voice synthesis artifacts generated by said existing vocoder, from the database DB comprising a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of voice syntheses produced beforehand.According to a step 54, which said database DB is pre-processed by dividing Div said plurality of voice syntheses into two sets, namely a training dataset (from the English train dataset) on the one hand, itself divided into a predetermined number of batches (from the English batch) of voice syntheses, and a validation dataset (from the English valid dataset).Said prior training phase 52 comprises at least two training steps 56 and 58 implemented successively and corresponding respectively to a first supervised learning phase 56 E_A1, and to a second learning phase 58 during which said artificial intelligence model previously trained during the first supervised learning phase E_A1 is re-trained by adversarial training, via at least one discriminator.The first supervised learning phase 56 E_A1 uses, for supervision, each voice synthesis produced beforehand, by means of at least one predetermined codec, by said at least one existing vocoder and the associated reference voice signal used as input by said at least one existing vocoder to produce said voice synthesis.During this first supervised learning phase 56 E_A1 said artificial intelligence model 16 is optimized by simultaneously minimizing four distinct loss functions, in particular the aforementioned loss functions , , and .The second learning phase 58 comprises two steps 60 and 62 of distinct optimizations O1 and O2 implemented alternately.The first optimization step 60 O1 minimizing both:- said four distinct loss functions , , and associated with said first supervised learning phase 56, and- a first loss function associated with said second learning phase by adversarial training, via said at least one discriminator D, with .This first optimization step 60 O1 enables, as compared with the first supervised learning phase 56 E_A1, the output of the artificial intelligence model 16 to continue to be brought closer to the input voice signal of the existing vocoder 10, by minimizing said four distinct loss functions , , and while minimizing the number and the size of the different areas between x and y detected by the discriminator thanks to .The second optimization step 62 O2, implemented successively to said first optimization O1, minimizes a second loss function associated with said second learning phase by adversarial training, via said at least one discriminator D, with This second optimization step 62 O2, will enable the weights of said at least one discriminator D to be updated in order to reinforce it while continuing to detect different areas between x and y.As an optional complement, said second learning phase 58 comprises a plurality of iterations I, said two distinct optimizations O1 and O2 being applied during each iteration, batch by batch (i.e. batch by batch), until complete processing, called an epoch (from the English epoch) of the training dataset of said database DB.At each iteration, batch by batch (i.e. batch by batch), the weights of the artificial intelligence model are updated.Said complete processing is itself repeated, epoch by epoch, until convergence according to a predetermined convergence criterion, evaluated (i.e. determined), at each epoch (from the English epoch), using the validation dataset.According to an optional aspect of this optional complement, said predetermined convergence criterion depends on the PESQ score (i.e. the perceptual evaluation metric of voice quality of the English Perceptual Evaluation of Speech Quality) associated with the corrected voice synthesis via said retrained artificial intelligence model and / or on the result of said loss functions , , and used by said training module.In other words, at each epoch (from the English epoch), the PESQ score and / or the result of said loss functions , , and used by said training module is determined, and in the event of improvement relative to the preceding epoch, said second learning phase 58 is repeated, batch by batch, during a following epoch. Otherwise (i.e. absence of improvement), the training phase 52 is stopped and provides a trained artificial intelligence model M_E.Once the trained artificial intelligence model M_E has been obtained at the end of the training phase 52, the method 50 comprises, in real time, a step 64 of receiving R the voice synthesis E produced, in particular in real time, by said existing vocoder.Then, the method 50 comprises an inference phase 66 using the trained artificial intelligence model M_E 16, and comprising at least one projection step 68 P, via at least one neural network of said projection module 18 of the trained artificial intelligence model M_E, of said voice synthesis E produced by said existing vocoder into a latent space of parameterizable dimension, then at least one voice reconstruction step 70 RV of a corrected voice synthesis, via at least one other neural network of the voice reconstruction module 20 of the trained artificial intelligence model M_E, from said projection into said latent space of parameterizable dimension provided by said projection step.According to an aspect not shown, said voice synthesis E used in the inference phase 66 is suitable for subsequently being integrated into the database DB, in particular into the validation data set, in order to subsequently retrain (offline from the English offline) said artificial intelligence model.In other words, the artificial intelligence model is suitable for being continuously improved via the alternation of the training phase 52 and of at least one inference phase 66, the voice syntheses E received during successive inference phases 66 being suitable for being integrated into the database DB for subsequent training.A person skilled in the art will understand that the invention is not limited to the embodiments described, nor to the particular examples of the description, the embodiments and the variants mentioned above being suitable for being combined with one another in order to generate new embodiments of the invention.The present invention thus makes it possible to provide a “corrector” by post-processing the output of an existing vocoder. For example, the correction result obtained on MELP 600 and MELP 1200 is substantial, with in particular an increase in the PESQ score (i.e. in the PESQ metric) from 0 to 10% for MELP 600, MELP 1200 or else MELP 2400 a gain of 6dB in range, while furthermore allowing more conferences in parallel.In other words, the present invention makes it possible to improve the synthesis produced by an existing vocoder, for example the MELP vocoder with a significant margin for improving audio quality. This solution is furthermore suitable for being embedded within radiocommunication products and advantageously its execution is suitable for being implemented in real time and with very low latency.This solution opens a new direction for improving MELP, which is a popular coding with many applications in. The proposed solution makes it possible to have better audio quality compared with MELP, with a significant improvement in the number of parameters and in computation time compared with vocoding solutions comprising artificial intelligence AI capabilities (which globally implement vocoding and not correction of an existing vocoding).
Claims
1. Post-processing device (12) for voice synthesis (11) produced by means of a predetermined codec by an existing vocoder (10), said post-processing device (12) comprising:- a module (14) for receiving the voice synthesis produced by said existing vocoder;- an artificial intelligence model (16) comprising:- a projection module (18) configured to project, via at least one neural network, said voice synthesis produced by said existing vocoder, into a latent space of parameterizable dimension;- a voice reconstruction module (20) configured to reconstruct a corrected voice synthesis, via at least one other neural network, from said projection provided by said projection module;- a training module (22) of said artificial intelligence model;said post-processing device being characterized in that said training module is configured to train said artificial intelligence model to correct voice synthesis artifacts generated by said existing vocoder, from a database (DB) comprising a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of voice syntheses produced beforehand,said training module (22) comprising two training tools implemented successively:- the first training tool (24) being configured to train said artificial intelligence model (16) according to a first supervised learning phase;- the second training tool (26) being configured to retrain, by adversarial training, via at least one discriminator (D), according to a second learning phase, said artificial intelligence model previously trained and provided by the first training tool,said projection module comprising at least three projection blocks implementing three successive processings:- a first projection block (34), convolutional, configured to take into account both correlation along a time dimension and along a frequency dimension of the data received as input;- a second projection block (36) receiving as input the output of the first block, said second block corresponding to a first temporal convolutional neural network suitable for taking into account short- and medium-term temporal correlation;- a third projection block (38) receiving as input the output of the second block, the third block corresponding to a first gated recurrent unit suitable for taking into account long-term temporal correlation;said reconstruction module (20), the input of which is connected to the output of the projection module (18), comprising at least four reconstruction blocks implementing four successive processings:- a first reconstruction block (40) corresponding to a second temporal convolutional neural network suitable for taking into account short- and medium-term temporal correlation;- a second reconstruction block (42) receiving as input the output of the first reconstruction block (40), said second reconstruction block (42) corresponding to a second gated recurrent unit suitable for taking into account long-term temporal correlation;- a third reconstruction block (44) receiving as input the output of the second reconstruction block (42), the third reconstruction block (44) corresponding to a third temporal convolutional neural network suitable for taking into account short- and medium-term temporal correlation;- a fourth reconstruction block (46), convolutional, receiving as input the output of the third reconstruction block (44), the fourth reconstruction block (46) being configured to implement the inverse operation of that implemented by the first projection block (34) and to provide a scaling of said reconstructed and corrected voice synthesis.
2. Device (12) according to claim 1, wherein said at least one predetermined codec used to generate said voice synthesis and said database is the MELP codec, or else the enhanced MELPe codec, and is associated with a bit rate belonging to the group comprising the following bit rates:600 bits / s;1200 bits / s ;2400 bits / s,said database being single-rate or multi-rate and / or monolingual or multilingual.
3. Device (12) according to claim 1 or 2, wherein said first training tool (24) is configured to optimize said artificial intelligence model (16) by simultaneously minimizing four distinct loss functions.
4. Device (12) according to claim 3, wherein said four distinct loss functions are expressed in the following form :, corresponding to the L1 norm of the difference between the input voice signal of the existing vocoder and the output of the artificial intelligence model the input of which is connected to the output of said existing vocoder;, corresponding to the L2 norm of the difference between the spectrogram of the input voice signal of the existing vocoder and the spectrogram of the output of the artificial intelligence model the input of which is connected to the output of said existing vocoder;- , the combination of the loss function with spectrogram parameters from a predetermined list P of spectrogram parameters ;- , the combination of the loss function with mel-spectrogram parameters from a predetermined list Q of mel-spectrogram parameters , where , is the L2 norm of thedifference between the mel-spectrogram of the input voice signal of the existing vocoder and the mel-spectrogram of the output of the artificial intelligence model the input of which is connected to the output of said existing vocoder.
5. Device (12) according to claim 4, wherein the second training tool (26) is configured to alternately implement two distinct optimizations,the first optimization minimizing both:- said four distinct loss functions associated with said first supervised learning phase, and- a first loss function associated with said second adversarial training learning phase, via said at least one discriminator D, with ;the second optimization, implemented successively to said first optimization, minimizing a second loss function associated with said second adversarial training learning phase, via said at least one discriminator D, with .
6. Device according to claim 5, wherein said database is preprocessed by dividing said plurality of voice syntheses into two sets:- a training data set itself divided into a predetermined number of batches of voice syntheses,- a validation data set,said second learning phase comprising a plurality of iterations,during each iteration, said two distinct optimizations being applied, batch by batch, until complete processing of the training data set of said database,said complete processing being reiterated until convergence according to a predetermined convergence criterion and determined using the validation data set.
7. Device according to claim 6, wherein said predetermined convergence criterion depends on the PESQ score associated with the corrected voice synthesis via said retrained artificial intelligence model and / or on the result of said loss functions used by said training module.
8. Method (50) for post-processing voice synthesis produced by means of a predetermined codec by an existing vocoder, said method being implemented by a post-processing device according to any one of the preceding claims, said method comprising:- receiving (64) the voice synthesis produced by said existing vocoder ;and the following steps implemented during an inference phase (66) implemented by an artificial intelligence model of said post-processing device previously trained during a prior training phase (52),said inference phase (66) comprising:- projecting (68), via at least one neural network, said voice synthesis produced by said existing vocoder, into a latent space of parameterizable dimension ;- voice reconstruction (70) of a corrected voice synthesis, via at least one other neural network, from said projection provided by said projection step ;said prior training phase (52) of said artificial intelligence model training said artificial intelligence model to correct voice synthesis artifacts generated by said existing vocoder, from a database (DB) comprising a plurality of voice syntheses produced beforehand, by means of at least one predetermined codec, by at least one existing vocoder and a plurality of reference voice signals used as input by said at least one existing vocoder to produce said plurality of voice syntheses produced beforehand, said training phasesaid prior training phase (52) comprising at least two training steps implemented successively and corresponding respectively to- a first supervised learning phase (56) ; and- a second learning phase (58) during which said artificial intelligence model previously trained during the first supervised learning phase is retrained by adversarial training, via at least one discriminator;said projection comprising three successive processings:- a first projection processing, via a first convolutional projection block, taking into account both correlation along a temporal dimension and along a frequency dimension of the input received data;- a second projection processing, via a second projection block receiving as input the output of the first block, and corresponding to a first temporal convolutional neural network, taking into account short- and medium-term temporal correlation;- a third projection processing, via a third projection block receiving as input the output of the second block and corresponding to a first gated recurrent unit, taking into account long-term temporal correlation;said voice reconstruction subsequent to said projection, comprising four successive processing operations:- a first voice reconstruction processing via a first reconstruction block corresponding to a second temporal convolutional neural network, taking into account short- and medium-term temporal correlation;- a second voice reconstruction processing via a second reconstruction block receiving as input the output of the first reconstruction block, said second reconstruction block corresponding to a second gated recurrent unit, taking into account long-term temporal correlation;- a third voice reconstruction processing via a third reconstruction block receiving as input the output of the second reconstruction block, the third reconstruction block corresponding to a third temporal convolutional neural network, taking into account short- and medium-term temporal correlation;- a fourth voice reconstruction processing via a fourth, convolutional, reconstruction block receiving as input the output of the third reconstruction block, corresponding to the inverse operation of that implemented by the first projection block and providing a scale change of said reconstructed and corrected voice synthesis.
9. Computer program product, comprising software instructions which, when executed by a computer, implement a method according to claim 8.