Neural vocoder for generating synthetic speech signals by implementing a speaker-adaptive model and method for training the same

Abstract

To provide a training method for a neural vocoder actualized by a computer.SOLUTION: A training method includes steps of setting, as an initial value, a weight from a speaker-independently trained source model for audio data sets from a plurality of speakers, the audio data sets each having a first size, and generating an updated weight for the initial value by training an audio data set of second size smaller than the first size not from the plurality of speakers, but from a target speaker. The updated weight is used for generating a synthesized audio signal corresponding to the target speaker.SELECTED DRAWING: Figure 17

Description

[Technical field]

[0001] The following description relates to a method for generating a synthetic speech signal using a neural vocoder, a neural vocoder, and a method for training the neural vocoder.

[0002] The following description also relates to a neural vocoder that utilizes a speaker adaptive model to generate a synthetic speech signal for a target speaker, and a method for training the neural vocoder to implement the speaker adaptive model. [Background technology]

[0003] Speech synthesis technology is a technology that creates synthetic speech similar to human speech based on input data. One example is Text to speech (TTS), which converts input text into human speech and provides it.

[0004] Such synthetic speech is generated by a vocoder that generates a speech signal based on input acoustic parameters. In recent years, with the development of artificial intelligence and deep learning technology, neural vocoders that utilize neural networks to generate synthetic speech have been proposed. Neural vocoders are trained with speech data from speakers in a speaker-independent or speaker-dependent manner, and generate a synthetic speech signal for the input acoustic parameters by using the training results.

[0005] In order for a neural vocoder to generate a synthetic voice signal corresponding to a specific target speaker, the neural vocoder must be trained using the voice data of the target speaker. In order to generate a synthetic voice signal with a certain level of quality, several hours or more of voice data including the target speaker's voice recording are generally required. If there is insufficient voice data, the quality of the generated synthetic voice signal may be degraded or distortion may occur. When the target speaker is not an ordinary person but a celebrity such as an entertainer or famous person, it is often difficult to secure several hours or more of voice recording as training data. Therefore, there is a need for a neural vocoder system that can improve the quality of a synthetic voice signal while minimizing the amount of target speaker's voice data used for training.

[0006] Meanwhile, in generating a synthetic voice signal based on a voice signal, it is difficult for a neural network (e.g., CNN) to fully capture the dynamic characteristics of the voice signal. In particular, spectral distortion is likely to occur in the high frequency region of the voice signal, which may lead to a deterioration in the quality of the synthetic voice signal. Therefore, there is a demand for a neural vocoder system that can reduce the spectral distortion in the high frequency region to improve the quality of the synthetic voice signal and further simplify the process of training voice data.

[0007] Patent document 1 (Korean Patent Application Publication No. 10-2018-0113325 (Publication date: October 16, 2018)) describes a speech synthesis device and method that provides a function of synthesizing a modulated speech waveform by encoding a speech model of a speech synthesizer, converting the speech model code, and decoding the speech model so that the voice of the synthesized sound is modulated as intended by a developer or user when the speech synthesis device synthesizes a speech waveform.

[0008] The information presented above is merely intended to aid in the understanding of the present invention and may contain material that does not form part of the prior art, nor may it contain material that the prior art would present to one of ordinary skill in the art. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Korean Patent Application Publication No. 10-2018-0113325 Summary of the Invention [Problem to be solved by the invention]

[0010] The objective of the present invention is to provide a method for generating a speech signal using a neural vocoder, which comprises obtaining a plurality of acoustic parameters including a spectrum-related parameter and an excitation-related parameter, estimating an excitation signal based on the plurality of acoustic parameters, and applying a linear synthesis filter based on at least one of the spectrum-related parameters to the estimated excitation signal to generate a target speech signal.

[0011] Another object is to provide a method for training a neural vocoder, which sets weights from a source model that has been trained speaker-independently on speech data sets from multiple speakers as initial values, and generates updated weights by training a speech data set from a target speaker on the corresponding initial values. [Means for solving the problem]

[0012] In one aspect, a speech signal generation method is provided that is executed by a computer-implemented neural vocoder, the speech signal generation method including: acquiring a plurality of acoustic parameters including a spectral-related parameter and an excitation-related parameter separated by a periodicity of an excitation; estimating an excitation signal based on the plurality of acoustic parameters; and generating a target speech signal by applying a linear synthesis filter based on at least one of the spectral-related parameters to the estimated excitation signal.

[0013] The excitation-related parameters may include a first excitation parameter indicative of excitation below a predetermined cutoff frequency, and a second excitation parameter indicative of excitation above the cutoff frequency.

[0014] The first excitation parameter may be indicative of a harmonic spectrum of the excitation and the second excitation parameter may be indicative of another part of the excitation.

[0015] The spectrum-related parameters may include a frequency parameter indicating the pitch of the speech signal, an energy parameter indicating the energy of the speech signal, a parameter indicating whether the speech signal is voiced or unvoiced, and a parameter indicating the Line Spectral Frequency (LSF) of the speech signal.

[0016] The step of generating the target speech signal may include converting parameters indicative of the LSF into a Linear Predictive Coding (LPC) code, and applying the linear synthesis filter based on the converted LPC to the estimated excitation signal.

[0017] The acoustic parameters may be generated by an acoustic model based on an input text or an input speech signal.

[0018] The neural vocoder may be trained based on a speech signal input for training, the training including a step of separating an excitation signal from the input speech signal by applying a linear prediction analysis filter to the input speech signal, and a step of modeling a probability distribution of the separated excitation signal, and the step of estimating the excitation signal may include estimating the excitation signal for the plurality of acoustic parameters using the probability distribution of the modeled excitation signal.

[0019] The step of separating the excitation signal may include a step of converting parameters indicative of the LSF of the input speech signal into a Linear Predictive Coding (LPC) code, and a step of applying to the input speech signal the linear predictive analysis filter based on the converted LPC of the input speech signal.

[0020] The separated excitation signal may be a residual component of the input speech signal.

[0021] In another aspect, a computer-implemented method for training a neural vocoder is provided, comprising the steps of: receiving an input speech signal; extracting from the input speech signal a plurality of acoustic parameters, the acoustic parameters including spectral-related parameters and excitation-related parameters separated by a periodicity of excitation; separating an excitation signal from the input speech signal by applying a linear predictive analysis filter to the input speech signal based on at least one of the spectral-related parameters; and modeling a probability distribution of the separated excitation signal.

[0022] The step of separating the excitation signal may include a step of converting a parameter among the spectral-related parameters indicative of the LSF of the input speech signal into an LPC, and a step of applying to the input speech signal the linear predictive analysis filter based on the converted LPC of the input speech signal.

[0023] The excitation-related parameters may include a first excitation parameter indicative of excitation below a predetermined cutoff frequency, and a second excitation parameter indicative of excitation above the cutoff frequency.

[0024] In yet another aspect, a neural vocoder is provided, the neural vocoder including: a parameter acquisition unit that acquires a plurality of acoustic parameters including a spectral-related parameter and an excitation-related parameter divided by a periodicity of excitation; an excitation signal estimation unit that estimates an excitation signal based on the plurality of acoustic parameters; and a speech signal generation unit that generates a target speech signal by applying a linear synthesis filter based on at least one of the spectral-related parameters to the estimated excitation signal.

[0025] The speech signal generation unit may include a conversion unit that converts a parameter indicating an LSF of the speech signal among the spectrum-related parameters into a Linear Predictive Coding (LPC) code, and applies the linear synthesis filter based on the converted LPC to the estimated excitation signal.

[0026] The neural vocoder may be trained based on a speech signal input for training, and the neural vocoder may further include an excitation signal separation unit that separates an excitation signal from the input speech signal by applying a linear prediction analysis filter to the input speech signal, and a modeling unit that models a probability distribution of the separated excitation signal, and the excitation signal estimation unit may estimate an excitation signal for the plurality of acoustic parameters using the probability distribution of the modeled excitation signal.

[0027] The excitation signal separation unit may include a conversion unit that converts parameters indicating the LSF of the input speech signal into a Linear Predictive Coding (LPC) code, and may apply the linear predictive analysis filter based on the converted LPC of the input speech signal to the input speech signal.

[0028] In yet another aspect, a computer-implemented method for training a neural vocoder is provided, comprising initializing weights from a source model trained on a speech data set from a plurality of speakers, and generating updated weights for the initial weights by training on a speech data set from a target speaker, the updated weights being used to generate a synthetic speech signal corresponding to the target speaker.

[0029] The weights from the source model are values ​​indicating global characteristics that are not distinguished for each speaker included in the speech dataset, and the step of generating the updated weights may generate the updated weights by adjusting the weights from the source model so as to reflect the unique characteristics of the target speaker included in the speech dataset from the target speaker.

[0030] The size of each of the speech data sets from the multiple speakers may be greater than the size of the speech data set from the target speaker.

[0031] The method for training the neural vocoder further includes constructing a source model for speaker-independently training speech data sets from the multiple speakers, and obtaining the weights from the source model, which may be used as an initializer of a model for training speech data sets from the target speaker.

[0032] A speech signal generation method is provided, which is executed by a neural vocoder trained by the training method, and includes the steps of: acquiring a plurality of acoustic parameters, including spectral-related parameters generated by an acoustic model and excitation-related parameters separated by a periodicity of excitation, based on an input text or an input speech signal; estimating an excitation signal based on the plurality of acoustic parameters; and generating a target speech signal by applying a linear synthesis filter based on at least one of the spectral-related parameters to the estimated excitation signal, wherein the target speech signal is a synthetic speech corresponding to the target speaker.

[0033] The excitation-related parameters may include a first excitation parameter indicative of excitation below a predetermined cutoff frequency, and a second excitation parameter indicative of excitation above the cutoff frequency.

[0034] The step of generating the target speech signal may include converting a parameter indicating the LSF of the speech signal among the spectral-related parameters into a Linear Predictive Coding (LPC) code, and applying the linear synthesis filter based on the converted LPC to the estimated excitation signal.

[0035] The step of estimating the excitation signal may be performed by a method including estimating an excitation signal for the plurality of acoustic parameters using a probability distribution of the modeled excitation signal, and modeling the probability distribution of the excitation signal includes separating an excitation signal from an input speech signal for training by applying a linear prediction analysis filter to the input speech signal, and modeling the probability distribution of the separated excitation signal.

[0036] The step of separating the excitation signal may include a step of converting parameters indicative of the LSF of the input speech signal into a Linear Predictive Coding (LPC) code, and a step of applying to the input speech signal the linear predictive analysis filter based on the converted LPC of the input speech signal.

[0037] In yet another aspect, a neural vocoder is provided, the neural vocoder including a speaker-adaptive model construction unit that sets weights from a source model that has been speaker-independently trained on speech data sets from a plurality of speakers as initial values, and constructs a speaker-adaptive model that generates updated weights for the initial values ​​by training on a speech data set from a target speaker, the updated weights generated by the speaker-adaptive model being used to generate synthetic speech corresponding to the target speaker.

[0038] The neural vocoder may further include a source model construction unit that constructs a source model for speaker-independently training the speech datasets from the multiple speakers, and the source model may act as an initializer for a model for training the speech dataset from the target speaker.

[0039] The neural vocoder may further include a parameter acquisition unit that acquires a plurality of acoustic parameters including spectral-related parameters generated by an acoustic model and excitation-related parameters classified by a periodicity of excitation based on an input text or an input speech signal, an excitation signal estimation unit that estimates an excitation signal based on the plurality of acoustic parameters, and a speech signal generation unit that generates a target speech signal by applying a linear synthesis filter based on at least one of the spectral-related parameters to the estimated excitation signal, and the target speech signal may be a synthetic speech corresponding to the target speaker.

[0040] The speech signal generation unit may include a conversion unit that converts a parameter indicating an LSF of the speech signal among the spectrum-related parameters into a Linear Predictive Coding (LPC) code, and applies the linear synthesis filter based on the converted LPC to the estimated excitation signal.

[0041] The neural vocoder may further include an excitation signal separation unit that separates an excitation signal from an input speech signal by applying a linear prediction analysis filter to the input speech signal for training, and a modeling unit that models a probability distribution of the separated excitation signal, and the excitation signal estimation unit may estimate an excitation signal for the plurality of acoustic parameters using the modeled probability distribution of the excitation signal.

[0042] The excitation signal separation unit may include a conversion unit that converts parameters indicating the LSF of the input speech signal into a Linear Predictive Coding (LPC) code, and may apply the linear predictive analysis filter based on the converted LPC of the input speech signal to the input speech signal. Effect of the Invention

[0043] The neural vocoder performs estimation using an excitation signal as a target, and generates a target speech signal by applying a linear prediction filter to the estimated excitation signal, thereby improving the quality of the generated target speech signal, and in particular reducing spectral distortion in the high frequency region of the speech signal.

[0044] By training a speech dataset from a target speaker using weights from a speaker-independently trained source model as initial values ​​that are not random, it is possible to generate high-quality synthetic speech (synthetic speech signal) of the target speaker even when training only a relatively small (i.e., short) speech dataset. [Brief description of the drawings]

[0045] [Figure 1] FIG. 2 illustrates a method for generating a synthetic speech signal based on an input text or speech signal in one embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating the architecture of a neural vocoder system in one embodiment. [Diagram 3] FIG. 2 is a block diagram illustrating the processor architecture of a neural vocoder system in one embodiment. [Figure 4] 4 is a flow chart illustrating a method for generating an audio signal in one embodiment. [Diagram 5] 1 is a flow chart illustrating a method for training a neural vocoder in one embodiment. [Figure 6] FIG. 2 illustrates a method for constructing a speaker adapted model to generate synthetic speech for a target speaker in one embodiment. [Figure 7] FIG. 2 is a block diagram illustrating the structure of a processor of a neural vocoder in one embodiment. [Figure 8]1 is a flow chart illustrating a method for training a neural vocoder to build a speaker adapted model in one embodiment. [Figure 9] FIG. 2 illustrates speech and excitation signals and their relationship in one example. [Figure 10a] FIG. 1 illustrates a Statistical Parametric Speech Synthesis (SPSS) system for generating synthetic speech signals using different types of vocoders. [Figure 10b] FIG. 1 illustrates a Statistical Parametric Speech Synthesis (SPSS) system for generating synthetic speech signals using different types of vocoders. [Figure 10c] FIG. 1 illustrates a Statistical Parametric Speech Synthesis (SPSS) system for generating synthetic speech signals using different types of vocoders. [Figure 11] FIG. 2 illustrates a method for training a neural vocoder by separating an excitation signal from a speech signal input for training in one embodiment. [Figure 12] FIG. 2 illustrates a method for estimating an excitation signal from acoustic parameters generated by an acoustic model based on an input text to generate a synthetic speech signal in one embodiment. [Figure 13] FIG. 2 illustrates a method for training a neural vocoder by separating an excitation signal from a speech signal input for training in one embodiment. [Figure 14] FIG. 2 illustrates a method for estimating an excitation signal from acoustic parameters generated by an acoustic model based on an input text to generate a synthetic speech signal in one embodiment. [Figure 15]1 is a graph showing the difference between the use of parameters classified according to excitation periodicity as acoustic parameters of negative log-likelihood (NLL) obtained in the training process / synthetic speech signal generation process in one example; [Figure 16] 1 is a diagram illustrating speaker-dependent and speaker-independent features of speech signals for speech signals from multiple speakers in one example; [Figure 17] FIG. 1 illustrates a method for generating synthetic speech for a target speaker using a source model constructed by training speech data sets from multiple speakers and a speaker adapted model constructed by training speech data sets from the target speaker in one example. [Figure 18] FIG. 13 illustrates the results of a comparative evaluation of the quality of synthesized speech signals generated with and without the application of a speaker adaptation algorithm in one example. [Figure 19] FIG. 13 illustrates the results of a comparative evaluation of the quality of synthesized speech signals generated with and without the application of a speaker adaptation algorithm in one example. [Figure 20] FIG. 1 shows the results of a Mean Opinion Score (MOS) evaluation of the ExcitNet vocoder and other vocoders in one example. [Figure 21] FIG. 13 is a diagram showing the change in performance of a neural vocoder that constructs a speaker adapted model when the F0 scaling factor is changed in one example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0046] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0047] FIG. 1 illustrates a method for generating a synthetic speech signal based on an input text or speech signal in one embodiment.

[0048] Although the audio signal refers to audio, in the detailed description below, for convenience of explanation, the terms "audio signal" and "audio" may be used interchangeably.

[0049] The acoustic model 110 may generate acoustic parameters(s) from an input text or speech signal for generating a synthetic speech signal. The acoustic model 110 may be designed in a deep learning based Statistical Parametric Speech Synthesis (SPSS) system. The acoustic model 110 may be composed of multiple feed-forward and long short-term memory layers trained to exhibit a nonlinear mapping function between the linguistic input and the acoustic output parameters. The acoustic model 110 may be, for example, a DNN TTS module. The acoustic parameters may be features used to generate a synthetic speech signal or parameters used to configure the features.

[0050] The vocoder 120 may generate a synthetic voice signal by converting the acoustic parameters generated by the acoustic model 110 into a voice signal. The vocoder 120 may be a neural vocoder. The neural vocoder may be trained by a deep learning model. The neural vocoder may be, for example, WaveNet, SampleRNN, or WaveRNN. The neural vocoder may also be a general generative model, without being limited thereto.

[0051] "Neural vocoder" may be used to denote a device that includes a trained model (e.g., WaveNet, SampleRNN, WaveRNN, or a general model) and various filters for the generation of a (synthetic) speech signal.

[0052] The vocoder 120 may estimate an excitation signal of the speech signal based on the acoustic parameters obtained from the acoustic model 110. That is, the excitation signal of the speech signal may be a target of the vocoder 120.

[0053] The excitation signal may be a component of the speech signal that indicates speech tremors and may be distinct from a component (spectral component) that indicates changes in the speech signal that change with the shape of the speaker's mouth. Changes in the excitation signal may be limited only by vocal cord movement of the speaker. The excitation signal may be a residual signal of the speech signal.

[0054] A target speech signal (i.e., a synthetic speech signal) may be generated by applying a Linear Prediction filter generated based on acoustic parameters indicating the spectral components of the speech signal to the excitation signal estimated by the vocoder 120.

[0055] By the vocoder 120 targeting an excitation signal that is not a speech signal and applying a linear prediction filter to the estimated excitation signal to generate a target speech signal, the quality of the generated target speech signal can be improved, and in particular, spectral distortion in the high frequency region of the speech signal can be reduced.

[0056] A more specific method of generating a target speech signal by estimating an excitation signal, and a more specific method of training a neural vocoder to estimate the excitation signal, will be described in more detail with reference to FIGS. 2-5.

[0057] FIG. 2 is a block diagram illustrating the architecture of a neural vocoder system in one embodiment.

[0058] A more detailed configuration of the neural vocoder system 200 will be described with reference to Fig. 2. The neural vocoder system 200 shown in the figure may represent a computer (computer system) configured to include a neural vocoder.

[0059] The neural vocoder system 200 may be a fixed terminal or a mobile terminal realized by a computer system. For example, the neural vocoder system 200 may be realized by an AI speaker, a smartphone, a mobile phone, a navigation system, a personal computer (PC), a notebook PC, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet, a game console, a wearable device, an Internet of Things (IoT) device, a virtual reality (VR) device, an augmented reality (AR) device, or the like. The neural vocoder system 200 may also be realized by a server or other computing device that communicates with the above-mentioned terminals via a network.

[0060] The neural vocoder system 200 may include a memory 210, a processor 220, a communication module 230, and an input / output interface. The memory 210 is a non-transitory computer-readable recording medium and may include a persistent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, or the like. Here, a persistent mass storage device such as a ROM, an SSD, a flash memory, or a disk drive may be included in the neural vocoder system 200 as a persistent storage device separate from the memory 210. The memory 210 may also store an operating system and at least one program code (for example, a code for a browser installed and executed in the neural vocoder system 200, an application installed in the neural vocoder system 200 to provide a specific service, or the like). Such software components may be loaded from a computer-readable recording medium separate from the memory 210. Such another computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory 210 through the communication module 230 that is not a computer-readable recording medium. For example, at least one program may be loaded into the memory 210 based on a computer program that is installed by a file provided by a developer or via a file distribution system (for example, an external server) that distributes installation files for an application.

[0061] The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication module 230. For example, the processor 220 may be configured to execute instructions received according to program code stored in a storage device such as the memory 210.

[0062] The communications module 230 may provide functionality for the neural vocoder system 200 to communicate with other electronic devices or other servers over a network. The communications module 230 may be a hardware module, such as a network interface card, a network interface chip, and a networking interface port of the neural vocoder system 200, or a software module, such as a network device driver or a networking program.

[0063] The input / output interface 240 may be a means for interfacing with an input / output device (not shown). For example, the input device may include a device such as a keyboard, a mouse, a microphone, a camera, and the output device may include a device such as a display, a speaker, a haptic feedback device, and the like. As another example, the input / output interface 240 may be a means for interfacing with a device that integrates functions for input and output into one, such as a touch screen. The input / output device 215 may be a component of the neural vocoder system 200. When the neural vocoder system 200 is realized as a server, the neural vocoder system 200 may not include an input / output device and an input / output interface.

[0064] Also, in other embodiments, neural vocoder system 200 may include more components than those shown in the figure, however, most prior art components are not necessary to explicitly show in the figure and are therefore omitted.

[0065] With reference to FIG. 3, a more detailed configuration of the processor 220 will now be described, along with a method for generating a target speech signal by estimating an excitation signal, and a method for training a neural vocoder to estimate the excitation signal.

[0066] The technical features explained above with reference to FIG. 1 can also be applied to FIG. 2, so duplicated explanations will be omitted.

[0067] FIG. 3 is a block diagram illustrating the processor architecture of the neural vocoder system in one embodiment.

[0068] Each of the configurations 310 to 340 of the processor 220 described below may be realized by one or more software modules and / or hardware modules. Depending on the embodiment, the components of the processor 220 may be selectively included or excluded from the processor 220. Also, depending on the embodiment, the components of the processor 220 may be separated or combined to express the functions of the processor 220.

[0069] The components of the processor 220 may be representations of different functions of the processor 220 that are performed by the processor 220 according to instructions provided by program code recorded in the neural vocoder system 200 .

[0070] The parameter acquisition unit 310 of the processor 220 may acquire a plurality of acoustic parameters including a spectral-related parameter and an excitation-related parameter classified according to the periodicity of the excitation. The acoustic parameters acquired by the parameter acquisition unit 310 may be generated by an acoustic model based on a text input by a user or a speech signal input by a speaker.

[0071] The excitation signal estimator 320 of the processor 220 may estimate an excitation signal based on a plurality of acoustic parameters. The excitation signal estimator 320 (neural vocoder) may be trained based on a speech signal input for training. The excitation signal estimator 320 may estimate an excitation signal for the plurality of acoustic parameters using a probability distribution of the excitation signal modeled by training.

[0072] The processor 220 may include a configuration 340 for performing training of a neural vocoder. An excitation signal separation unit 342 of the processor 220 may separate an excitation signal from a speech signal input for training by applying a linear prediction analysis filter to the speech signal input for training. The excitation signal separation unit 342 may include a conversion unit 343 that converts a parameter indicating a line spectral frequency (LSF) of the speech signal input for training into a linear predictive coding (LPC). The linear predictive analysis filter may be based on a parameter indicating the LSF and may be generated based on the converted LPC. A modeling unit 344 of the processor 220 may model a probability distribution of the separated excitation signal.

[0073] The speech signal generator 330 of the processor 220 may generate the target speech signal by applying a linear (predictive) synthesis filter based on at least one of the spectrum-related parameters to the excitation signal estimated by the excitation signal estimator 320. The target speech signal may be a synthesized speech signal.

[0074] The speech signal generating unit 330 may include a conversion unit 332 that converts a parameter indicating the LSF of the speech signal among the acquired spectrum-related parameters into a Linear Predictive Coding (LPC). The linear predictive synthesis filter is based on the parameter indicating the LSF of the speech signal among the acquired spectrum-related parameters, and may be generated based on the converted LPC. In other words, the speech signal generating unit 330 may generate a target speech signal by applying a linear predictive synthesis filter based on the converted LPC to an estimated excitation signal.

[0075] A more specific method for generating a target speech signal by estimating an excitation signal is described in more detail with reference to FIG. 4, and a more specific method for training a neural vocoder to estimate an excitation signal is described in more detail with reference to FIG. 5.

[0076] The technical features explained above with reference to FIGS. 1 and 2 can also be applied to FIG. 3, so duplicated explanations will be omitted.

[0077] FIG. 4 is a flow chart illustrating a method for generating an audio signal, in one embodiment.

[0078] In step 410, the parameter acquisition unit 310 may acquire a plurality of acoustic parameters including spectrum-related parameters and excitation-related parameters classified according to the periodicity of the excitation. The acoustic parameters acquired by the parameter acquisition unit 310 may be generated by an acoustic model based on a text input by a user or a speech signal input by a speaker. That is, the parameter acquisition unit 310 may receive the acoustic parameters from an acoustic model.

[0079] The spectrum-related parameters may be parameters indicative of the spectral components constituting the speech signal. The excitation-related parameters may be parameters indicative of components corresponding to a residual signal (excitation signal) obtained by removing the spectral components from the speech signal. The spectral component signals may indicate parts of the speech signal that change according to the shape of the speaker's mouth. The excitation signal may indicate parts of the speech signal that show voice tremors. Changes in the excitation signal may be limited only by the movement of the speaker's vocal cords.

[0080] The spectrum-related parameters may include, for example, a frequency parameter (F0) indicating the pitch of the audio signal, an energy parameter indicating the energy of the audio signal (as an example, a parameter indicating gain), a parameter (v / uv) indicating whether the audio signal is voiced or unvoiced, and a parameter indicating the line spectral frequency (LSF) of the audio signal.

[0081] The excitation-related parameters may include parameters that are differentiated by the periodicity of the excitation. The excitation-related parameters may, for example, be Time-Frequency Trajectory Excitation (TFTE) parameters. The TFTE may describe the spectral shape of the excitation along the frequency axis and the evolution of such shape along the time axis. The excitation-related parameters may include a first excitation parameter (a Slowly Evolving Waveform (SEW) parameter) that describes the components of the excitation signal that vary more slowly in the time-frequency axis, and a second excitation parameter (a Rapidly Evolving Waveform (REW) parameter) that describes the components of the excitation signal that vary more rapidly in the time-frequency axis.

[0082] The first excitation parameter may indicate an excitation below a predetermined cutoff frequency, and the second excitation parameter may indicate an excitation above the cutoff frequency. The first excitation parameter may indicate a harmonic spectrum of the excitation, and the second excitation parameter may indicate the other part of the excitation. For example, the first excitation parameter (SEW parameter) corresponding to the harmonic excitation spectrum may be obtained by low-pass filtering each frequency component of the TFTE along the time domain axis (at the predetermined cutoff frequency). The residual noise spectrum above the predetermined cutoff frequency may be obtained as the second excitation parameter (REW parameter) by subtracting the SEW from the TFTE. The first excitation parameter (SEW parameter) and the second excitation parameter are used to more effectively represent the periodicity of the excitation. The first excitation parameter and the second excitation parameter may correspond to an Improved Time-Frequency Trajectory Excitation (ITFTE) parameter.

[0083] In step 420, the excitation signal estimator 320 may estimate an excitation signal based on the acoustic parameters. That is, the excitation signal estimator 320 may estimate the excitation signal using the spectrum-related parameters and the excitation-related parameters as input. The estimated excitation signal may be a time sequence of the excitation signal.

[0084] The excitation signal estimator 320 may be trained based on a speech signal input for training, and may estimate an excitation signal for the acquired acoustic parameters by using a probability distribution of the excitation signal modeled by training. A training method for a neural vocoder including the excitation signal estimator 320 will be described in more detail with reference to FIG. 5.

[0085] The excitation signal estimator 320 may be realized by, for example, WaveNet, SampleRNN, or WaveRNN. Also, the excitation signal estimator 320 may be realized by a general generative model without being limited thereto.

[0086] In step 430, the speech signal generator 330 may generate a target speech signal by applying a linear (predictive) synthesis filter based on at least one of the spectrum-related parameters to the excitation signal estimated by the excitation signal estimator 320. The target speech signal may be a synthesized speech signal. Step 430 will be described in more detail with reference to steps 432 and 434.

[0087] In step 432, the conversion unit 332 may convert the parameter indicating the LSF of the speech signal among the acquired spectrum-related parameters into a Linear Predictive Coding (LPC). The linear predictive synthesis filter is based on the parameter indicating the LSF of the speech signal among the acquired spectrum-related parameters, and may be generated based on the converted LPC.

[0088] In step 434, the speech signal generator 330 may generate a target speech signal by applying the LPC-based linear prediction synthesis filter transformed in step 432 to the estimated excitation signal.

[0089] The target voice signal generated by steps 410 to 430 has superior quality compared to a voice signal generated by directly estimating a voice signal without estimating an excitation signal as a target, and in particular, can reduce spectral distortion in the high frequency region of the voice signal.

[0090] The technical features explained above with reference to FIGS. 1 to 3 can also be applied to FIG. 4 as they are, so duplicated explanations will be omitted.

[0091] FIG. 5 is a flow chart illustrating a method for training a neural vocoder in one embodiment.

[0092] With reference to FIG. 5, a method for modeling the probability distribution of the excitation signal, which can be estimated based on the acquired acoustic parameters, will now be described in detail.

[0093] At step 510, neural vocoder system 200 may receive a training speech signal. The training speech signal may be input directly to neural vocoder system 200 from a speaker, or may be input to neural vocoder system 200 by data containing the speech signal being transmitted from an electronic device that received the speech signal.

[0094] At step 520, the neural vocoder system 200 may extract a plurality of acoustic parameters from the input speech signal, including spectrum-related parameters and excitation-related parameters that are differentiated by the periodicity of the excitation. The neural vocoder system 200 may extract the plurality of acoustic parameters from the speech signal by speech analysis. For example, the neural vocoder system 200 may extract the plurality of acoustic parameters from the speech signal using a parametric vocoder that exists inside or outside the neural vocoder system 200.

[0095] The spectrum-related parameters may include, for example, a frequency parameter (F0) indicating the pitch of the speech signal, an energy parameter (for example, a parameter indicating gain) indicating the energy of the speech signal, a parameter (v / uv) indicating whether the speech signal is voiced or unvoiced, and a parameter indicating the line spectral frequency (LSF) of the speech signal. The excitation-related parameters may include parameters classified according to the periodicity of the excitation. The excitation-related parameters may be, for example, a Time-Frequency Trajectory Excitation (TFTE) parameter. The TFTE may indicate the spectral shape of the excitation along the frequency axis and the evolution of such shape along the time axis. The excitation-related parameters may include a SEW parameter indicating components of the excitation signal that vary more slowly on the time-frequency axis, and a REW parameter indicating components of the excitation signal that vary more quickly on the time-frequency axis. The SEW parameter may indicate an excitation below a certain cutoff frequency, and the REW parameter may indicate an excitation above the cutoff frequency. The SEW parameter may indicate the harmonic spectrum of the excitation, and the REW parameter may indicate the other part of the excitation. For example, the SEW parameter corresponding to the harmonic excitation spectrum may be obtained by low-pass filtering each frequency component of the TFTE along the time domain axis (at a predetermined cutoff frequency). The residual noise spectrum beyond the predetermined cutoff frequency may be obtained as the REW parameter by subtracting the SEW from the TFTE.

[0096] Stages 510 and 520 described above may be performed by processor 220 of neural vocoder system 200, as may stages 530 and 540 described below.

[0097] In step 530, the excitation signal separation unit 342 may separate an excitation signal from the input speech signal by applying a linear prediction analysis filter based on at least one of the spectrum-related parameters to the input speech signal. The linear prediction analysis filter may be a filter that separates a spectral formant structure from the speech signal. The separated excitation signal may be a residual component (i.e., a residual signal) of the input speech signal. The excitation signal may be an approximation of the residual signal by at least one of various types of excitation models such as pulse or noise (PoN), band aperiodicity (BAP), glottal excitation, and time-frequency trajectory excitation (TFTE) model to reduce the amount of information.

[0098] With reference to steps 532 and 534, the method for separating the excitation signal from the speech signal will be described in more detail.

[0099] In step 532, the conversion unit 343 of the excitation signal separation unit 342 may convert the parameter indicating the LSF of the input speech signal among the spectrum-related parameters into an LPC. The linear predictive analysis filter is based on the parameter indicating the LSF of the speech signal among the acquired spectrum-related parameters, and may be generated based on the converted LPC.

[0100] In step 534, the excitation signal separation unit 342 may separate the excitation signal from the speech signal by applying the LPC-based linear predictive analysis filter to the input speech signal.

[0101] In step 540, the modeling unit 344 may model the probability distribution of the separated excitation signal. The modeling unit 344 may be realized by, for example, WaveNet, SampleRNN, or WaveRNN. The modeling unit 344 may also be realized by a general generative model, including but not limited to these.

[0102] The excitation signal estimator 320 may perform the estimation of the excitation signal of step 420 described above by using the probability distribution of the excitation signal modelled by the modeller 344 .

[0103] The neural vocoder of the embodiment described with reference to FIGS. 1-4 may be named an ExcitNet vocoder in that it trains on an excitation signal and estimates the excitation signal to generate a synthetic speech signal.

[0104] Since the variation of the excitation signal is limited only by the movement of the vocal cords of the speaker, the process of training the excitation signal can be performed much more simply (compared to training the speech signal). In addition, the ITFTE parameters are used as conditional features that effectively indicate the degree of periodicity of the excitation signal, which can greatly improve the accuracy of the probability distribution modeling of the excitation signal.

[0105] The technical features explained above with reference to FIGS. 1 to 4 can also be applied to FIG. 5, and therefore a duplicated explanation will be omitted.

[0106] Below, with reference to Figures 6 to 8, we will explain a method for generating a synthetic speech of a target speaker by constructing a speaker-adapted model that generates high-quality synthetic speech of a target speaker using only a small amount (i.e., a short period) of speech data from the target speaker.

[0107] FIG. 6 illustrates a method for constructing a speaker adapted model to generate synthetic speech for a target speaker in one embodiment.

[0108] In the detailed description below, a speech data set may refer to a speech signal or data that includes a speech signal. For example, a speech data set may refer to a speech signal recorded from a speaker over a period of time.

[0109] The source model 610 may be an acoustic model trained on speech data sets from multiple speakers. The source model 610 may be an acoustic model trained speaker-independently on multiple speakers. For example, the source model 610 may be an acoustic model trained speaker-independently using one-hour speech data sets from each of ten speakers. The source model 610 may be designed in a deep learning-based Statistical Parametric Speech Synthesis (SPSS) system. The acoustic model 110 may be, for example, a DNN TTS module.

[0110] The source model 610, which is speaker-independently trained with speech datasets from multiple speakers, may be used as an initializer for the speaker adaptation model 620. In other words, weights from the source model 610 may be set as initial values ​​in training the speaker adaptation model 620 on speech datasets from a target speaker. The weights from the source model 610 may correspond to, for example, the acoustic parameters described above.

[0111] The speaker adaptation model 620 may be realized by a neural vocoder. The neural vocoder may be trained based on a deep learning model. The neural vocoder may be, for example, WaveNet, SampleRNN, ExcitNet, or WaveRNN. The neural vocoder may also be a general generative model, including but not limited to the above.

[0112] The speaker adaptation model 620 may be speaker-dependently trained for a particular speaker by applying a speaker adaptation algorithm. For example, the speaker adaptation model 620 may be speaker-dependently trained for a particular target speaker (e.g., a celebrity such as an entertainer, a famous person, etc.). The speaker adaptation model 620 may generate updated weight(s) by training on a speech dataset from the target speaker.

[0113] The speaker adaptation model 620 can generate a high-quality synthetic speech (synthetic speech signal) of the target speaker even when training a relatively small (i.e., short) speech data set by using the weights from the source model 610, which is not random and is trained independently of the speaker, as initial values. For example, the speaker adaptation model 620 can generate a high-quality synthetic speech of the target speaker even when training a speech data set of the target speaker of about 10 minutes.

[0114] In some embodiments, for celebrities for whom it is difficult to secure a speech dataset of several hours to several tens of hours, it is possible to construct a speaker adaptation model 620 that generates high-quality synthetic speech of a target speaker simply by securing a speech dataset of around 10 minutes and using this as training data.

[0115] The technical features described above with reference to FIGS. 1 to 5 can also be applied to FIG. 6 as they are, so duplicated explanations will be omitted.

[0116] FIG. 7 is a block diagram illustrating the architecture of a neural vocoder processor in one embodiment.

[0117] The processor 220 described with reference to FIG. 7 may correspond to the processor 220 described with reference to FIG. 3. Each of the configurations 710-720 of the processor 220 described below may be realized by one or more software modules and / or hardware modules. Depending on the embodiment, the components of the processor 220 may be selectively included or excluded from the processor 220. Also, depending on the embodiment, the components of the processor 220 may be separated or combined to represent the functions of the processor 220. The configurations 710-720 may represent different functions of the processor 220, which are executed by the processor 220 according to instructions provided by program code recorded in the neural vocoder system 200.

[0118] The processor 220 may include a speaker-adaptive model constructor 720. The speaker-adaptive model constructor 720 may set weights from the source model 610, which is speaker-independently trained on speech data sets from multiple speakers, as initial values, and may construct a speaker-adaptive model 620 that generates updated weights by training on speech data sets from a target speaker for the set initial values. The updated weights generated by the speaker-adaptive model 620 may be used to generate a synthetic voice corresponding to the target speaker.

[0119] The processor 220 may further include a source model builder 710. The source model builder 710 may build a source model 610 for speaker-independent training on speech datasets from multiple speakers. The built source model 610 may act as an initializer of a model for training on speech datasets from a target speaker.

[0120] The source model builder 710 may not be included in the processor 220 and may be implemented in a device separate from the neural vocoder system 200. The speaker adaptive model builder 720 may obtain weights from the source model 610 built by the source model builder 710 implemented in such a separate device and train the target speaker's speech dataset for building the speaker adaptive model 620.

[0121] The technical features explained above with reference to FIGS. 1 to 6 can also be applied to FIG. 7, and therefore duplicated explanations will be omitted.

[0122] FIG. 8 is a flow diagram illustrating a method for training a neural vocoder to build a speaker adapted model in one embodiment.

[0123] In step 810, the source model builder 710 may build a source model 610 that is speaker-independently trained on speech data sets from multiple speakers. The multiple speakers may be any users that provide speech data sets for training the source model 610.

[0124] In step 820, the speaker adaptive model builder 720 may obtain weights from the source model 610. The weights from the source model 610 may indicate values ​​indicative of global characteristics not distinguished by speaker contained in the speech data set from multiple speakers. The global characteristics may indicate, for example, formant characteristics or amplitude-frequency characteristics (patterns) for a particular pronunciation (such as, for example, "ah" or "ee"). In other words, the source model 610 may train speaker-independent global characteristics of such speech using speech data sets from multiple speakers.

[0125] In step 830, the speaker adapted model builder 720 may set the weighting values ​​obtained from the source model 610 as initial values. In other words, the source model 610 may be used as an initializer for the speaker adapted model 620 built by the speaker adapted model builder 720.

[0126] In step 840, the speaker adaptive model builder 720 may generate updated weights by training the speech data set from the target speaker against the obtained initial values. In other words, the speaker adaptive model builder 720 may build a speaker adaptive model 620 that is adapted to the target speaker (i.e., dependent on the target speaker) by training the speech data set from the target speaker against the initial values ​​from the source model 610.

[0127] The speaker adaptive model builder 720 may generate updated weights by adjusting the weights from the source model 610 to reflect the target speaker's unique characteristics contained in the speech data set from the target speaker. For example, the speaker adaptive model builder 720 may generate updated weights by fine-tuning the values ​​indicating the global characteristics not distinguished by speaker from the source model 610 to include the target speaker's unique characteristics by training the speech data set from the target speaker.

[0128] The generated updated weights may be used to generate a synthetic speech signal corresponding to the target speaker, which may be, for example, a synthetic speech of a celebrity corresponding to the target speaker.

[0129] The size of each of the speech datasets from multiple speakers for training the source model 610 (i.e., the length of the recorded speech signal, e.g., one hour or more) may be greater than the size of the speech dataset from the target speaker (i.e., the length of the recorded speech signal, e.g., 10 minutes).

[0130] A fine-tuning mechanism for the adaptation process as described in step 830 may capture the unique characteristics of a target speaker from a speech data set from the target speaker. Thus, the method of the described embodiment can improve vocoding performance even when there is an insufficient speech data set from the target speaker for training.

[0131] The neural vocoder training method described with reference to Figures 6-8 may be combined with the neural vocoder training method and synthetic speech signal generation method of the embodiments described with reference to Figures 1-4. For example, the ExcitNet vocoder described above may be combined with the embodiments described with reference to Figures 6-8.

[0132] As an example, the neural vocoder trained by performing steps 810-840 may correspond to the neural vocoder system 200 described with reference to Figures 1-4. The target speech signal generated in step 430 may be a synthetic speech signal corresponding to the target speaker trained by the speaker adapted model 620.

[0133] By combining the neural vocoder training method described with reference to Figures 6-8 with the technical features of the ExcitNet model described with reference to Figures 1-4, the quality of the synthesized speech corresponding to the target speaker can be improved.

[0134] The technical features explained above with reference to FIGS. 1 to 7 can also be applied to FIG. 8 as they are, so duplicated explanations will be omitted.

[0135] FIG. 9 illustrates an example of a speech signal and an excitation signal and their relationship.

[0136] As shown in the figure, assuming that the speech signal is S(n) and the excitation signal contained in S(n) is e(n), the relationship between S(n) and e(n) may be expressed as the following Equation (1).

[0137]

number

[0138] According to the relationship of Equation (1), a target speech signal (S(n)) may be generated by applying a linear prediction synthesis filter (i.e., h(n)) to the excitation signal (i.e., e(n)) estimated by step 420 of Fig. 4. A specific example of the linear prediction synthesis filter will be described in further detail with reference to Fig. 14.

[0139] The relationship of Equation (1) may be similarly applied to the separation of the excitation signal (i.e., e(n)) in step 530 of Fig. 5. In other words, a linear predictive analysis filter may be applied to the speech signal (S(n)) input for training, thereby separating the excitation signal (e(n)) from the speech signal (S(n)). A specific example of the linear predictive analysis filter will be described in more detail with reference to Fig. 13.

[0140] The technical features explained above with reference to FIGS. 1 to 8 can also be applied to FIG. 9, and therefore duplicated explanations will be omitted.

[0141] 10a-10c are diagrams illustrating Statistical Parametric Speech Synthesis (SPSS) systems for generating synthetic speech signals using different types of vocoders.

[0142] 10a shows a framework for speech synthesis including an acoustic model 1010 and an LPC synthesis module 1020 that generates a speech signal by LPC (Linear Predictive Coding) synthesis of acoustic features (acoustic parameters) from the acoustic model 1010. The LPC synthesis module 1020 is an LPC vocoder, which may correspond, for example, to the linear predictive synthesis filter described above.

[0143] FIG. 10b illustrates a framework for speech synthesis that includes an acoustic model 1010 and a WaveNet vocoder 1022, a neural vocoder that estimates a speech signal based on acoustic features (acoustic parameters) from the acoustic model 1010.

[0144] Figure 10c shows a framework for speech synthesis using an ExcitNet vocoder 1024, as described in Figures 1 to 5. The structure shown in Figure 10c may be a combination of the LPC coder 1020 of Figure 10a and the WaveNet vocoder 1022 of Figure 10b.

[0145] In the structure of Figure 10c, an ExcitNet vocoder 1024 may estimate an excitation signal based on acoustic features (acoustic parameters) from an acoustic model 1010. The estimated excitation signal may be converted to a target speech signal by Linear Predictive Coding (LPC) synthesis by a linear predictive synthesis filter 1030.

[0146] A more detailed example of the structure of FIG. 10c is described in further detail with reference to FIGS.

[0147] The technical features explained above with reference to Figs. 1 to 9 can also be applied to Figs. 10a to 10c as they are, so duplicated explanations will be omitted.

[0148] 11 and 13 illustrate a method for training a neural vocoder in one embodiment by separating an excitation signal from a speech signal input for training.

[0149] 11, a parametric vocoder 1110 may extract acoustic parameters from an input speech signal for training. A linear predictive analysis filter 1140 generated based on the spectral-related parameters among the extracted acoustic parameters may be applied to the input speech signal to separate an excitation signal from the input speech signal.

[0150] The WaveNet vocoder 1130 may receive and configure 1120 the extracted acoustic parameters as auxiliary features. The auxiliary features may include the spectrum-related parameters and excitation-related parameters described above. The WaveNet vocoder 1130 may model a probability distribution of the excitation signal based on the auxiliary features and the separated excitation signal. The WaveNet vocoder 1130 may be realized by the ExcitNet vocoder or other general generative model neural vocoders.

[0151] The structure of FIG. 11 will be described in more detail with reference to FIG. 13. An input speech signal for training may be subjected to speech analysis 1310 to extract acoustic features (acoustic parameters). Among the acoustic parameters, parameters indicative of LSF may be converted 1320 to LPC. A linear predictive analysis filter 1340 may be implemented based on the converted LPC. The linear predictive analysis filter 1340 may be applied to the input speech signal to separate an excitation signal from the input speech signal. The separated excitation signal may be input to an ExcitNet model (i.e., an ExcitNet vocoder) 1350. Meanwhile, the acoustic parameters may be configured 1330 as auxiliary features, and the auxiliary features may be input to the ExcitNet model 1350. The ExcitNet model 1350 may model a probability distribution of the excitation signal based on the input auxiliary features (i.e., acoustic parameters) and the separated excitation signal. In the illustrated example, e n may correspond to separate excitation signals.

[0152] 12 and 14 illustrate a method for estimating an excitation signal from acoustic parameters generated by an acoustic model based on an input text to generate a synthetic speech signal in one embodiment.

[0153] As shown in FIG. 12, the acoustic model 1150 may generate acoustic parameters based on the received language parameters. The WaveNet vocoder 1170 may receive the acoustic parameters by configuring them as auxiliary features 1160. The auxiliary features may include the above-mentioned spectrum-related parameters and excitation-related parameters. The WaveNet vocoder 1170 may estimate an excitation signal based on the acoustic parameters. The WaveNet vocoder 1170 may be realized by an ExcitNet vocoder or other general generative model neural vocoders. A linear predictive synthesis filter 1180 generated based on the spectrum-related parameters among the extracted acoustic parameters may be applied to the estimated excitation signal to generate a target synthetic speech.

[0154] The structure of FIG. 12 will be described in more detail with reference to FIG. 14. A text analysis 1410 may be performed on the input text for generating a synthetic speech signal to extract linguistic features (corresponding to the above-mentioned linguistic parameters). In extracting the linguistic features, a duration model 1420 may be further used to estimate phoneme durations, as shown in the figure. An acoustic model 1430 may generate acoustic features (acoustic parameters) from the extracted linguistic features. Among the acoustic parameters, parameters indicative of LSF may be converted 1440 to LPCs. A linear predictive synthesis filter 1470 may be realized based on the converted LPCs. The acoustic parameters may be configured 1450 as auxiliary features, which may be input to an ExcitNet model (i.e., an ExcitNet vocoder) 1460. The ExcitNet model 1460 may estimate an excitation signal based on the input auxiliary features (i.e., the acoustic parameters). A transformed LPC-based linear predictive synthesis filter 1470 may be applied to the estimated excitation signal to generate the target speech signal.

[0155]

number

[0156]

number

[0157] The technical features explained above with reference to Figs. 1 to 10c can also be applied to Figs. 11 to 14, and therefore a duplicated explanation will be omitted.

[0158] FIG. 15 is a graph showing the difference in the use or non-use of parameters classified according to the periodicity of excitation as acoustic parameters of negative log-likelihood (NLL) obtained in the training process / synthetic speech signal generation process in one example.

[0159] In the training process, the lower the NLL, the higher the modeling accuracy. In the graph shown in the figure, it can be seen that the NLL is lower when ITFTE parameters such as the SEW and REW parameters mentioned above are used than when they are not.

[0160] In addition, in the validation process of the synthetic speech signal, the lower the NLL, the better the quality of the synthetic speech generated. In the graph shown in the figure, it can be seen that the NLL is lower when ITFTE parameters such as the SEW parameter and the REW parameter are used than when they are not used.

[0161] In other words, the graphs shown in the figures confirm the fact that the use of ITFTE parameters in training a neural vocoder can significantly reduce errors in modeling the probability distribution of the excitation signal, and that the use of ITFTE parameters in estimating an excitation signal for generating synthetic speech can significantly reduce errors in generating a synthetic speech signal.

[0162] The technical features explained above with reference to FIGS. 1 to 14 can also be applied to FIG. 15, and therefore overlapping explanations will be omitted.

[0163] Figure 16 illustrates speaker-dependent and speaker-independent features of speech signals for speech signals from multiple speakers in one example, and Figure 17 illustrates a method for generating synthetic speech for a target speaker using a source model constructed by training speech data sets from multiple speakers and a speaker-adapted model constructed by training speech data sets from the target speaker in one example.

[0164] As shown in Fig. 16, the speaker-independent features may be features common to the voices of the speakers (speakers 1 to 3). In other words, the speaker-independent features may indicate global features that are not distinguished by speaker. The speaker-dependent features may indicate characteristics unique to each speaker.

[0165] As shown in FIG. 17, a source model 610 may be constructed by speaker-independently training speech data sets from multiple speakers, and a target speaker-dependent speaker-adapted model 620 may be constructed by training a speech data set from a target speaker based on weights from the source model 610. The weights from the source model 610 may be fine-tuned to reflect the unique characteristics of the target speaker by training the speech data set from the target speaker with the speaker-adapted model 620. As shown, the source model 610 and the speaker-adapted model 620 may be implemented using an ExcitNet model. As shown, in some embodiments, a speaker adaptation algorithm may be applied to the neural vocoder. Although not shown, a speaker adaptation algorithm may be applied to the acoustic model (e.g., DNNTTS) corresponding to the source model 610 in a similar manner.

[0166] The technical features explained above with reference to Figs. 1 to 15 can also be applied to Figs. 16 and 17, and therefore overlapping explanations will be omitted.

[0167] 18 and 19 show the results of a comparative evaluation of the quality of synthesized speech signals generated with and without the application of a speaker adaptation algorithm in one example.

[0168] 18 and 19 show the average scores given by evaluators who listened to and evaluated the audio signals. Here, RAW may correspond to the original audio signal.

[0169] 18, it can be seen that the quality of the synthetic speech signal was highly evaluated when the speaker adaptation algorithm was applied to both the WaveNet model and the ExcitNet model. In other words, as described with reference to FIGS. 6 to 8, it can be seen that the case where the speaker adaptation model 620 is constructed to generate a synthetic speech signal (w / speaker adaptation) shows superior performance compared to the case where it is not (w / o speaker adaptation).

[0170] A more detailed comparative evaluation of the quality of the synthesized speech signals is shown in Figure 19, which is described in more detail below.

[0171] The technical features explained above with reference to Figs. 1 to 17 can also be applied to Figs. 18 and 19, and therefore overlapping explanations will be omitted.

[0172] In the following, we will explain the ExcitNet model described above with reference to Figs. 1 to 5 in more detail and further explain the results of comparative experiments with other models.

[0173] The ExcitNet model (ExcitNet vocoder) may be a WaveNet-based neural excitation model for statistical parametric speech synthesis (SPSS) systems. The WaveNet-based neural vocoder system significantly improves the recognition quality of the synthesized speech signal, but may not be able to capture the complex time-varying characteristics of the speech signal and may output noise. The ExcitNet-based neural vocoder system can be trained by separating the residual components (i.e., the excitation signal) using an adaptive embossing filter that separates the spectral components from the speech signal (e.g., within the WaveNet framework), and can estimate the excitation signal as a target to generate the synthetic speech signal. In this manner, the spectral components of the speech signal are better represented by the deep learning framework, and the residual components are efficiently generated by the WaveNet framework, thereby improving the quality of the synthesized speech signal.

[0174] The following experiments also show that a neural vocoder system based on ExcitNet (trained both speaker-dependently and speaker-independently) outperforms conventional linear predictive vocoders and WaveNet vocoders.

[0175] For testing, three phonetically and prosodically rich speech corpora were used to train the acoustic models and the speaker-dependent (SD) ExcitNet vocoder. Each corpus was recorded by an expert Korean female (KRF) and Korean male (KRM). The speech signals were sampled at 24 kHz and each sample was quantized at 16 bits. Table 1 below shows the number of utterances in each set. To train the speaker-independent (SI) ExcitNet vocoder, speech corpora recorded by five Korean females and five Korean males were used. A total of 6,422 (10 hours) and 1,080 (1.7 hours) utterances were used for training and validation, respectively. Speech samples recorded by the same KRF and KRM speakers that are not included in the SI dataset were used for testing.

[0176] [Table 1] The following Tables 2 and 3 are the results of objective testing, showing the distortion between the original speech and the generated speech in terms of LSD (Log-Spectral Distance) (dB) and F0 RMSE (Root Mean Square Error) (Hz), respectively. WN denotes the WaveNet vocoder, WN-NS denotes the WaveNet vocoder with the noise shaping method applied, and ExcitNet denotes the ExcitNet vocoder. The parts with the lowest errors are marked in bold. Tables 2 and 3 may be measured results for voiced speech.

[0177] [Table 2]

[0178] [Table 3] As shown in Tables 2 and 3, in most cases of SD and SI, it can be seen that the ExcitNet vocoder exhibits the lowest distortion between the original speech and the generated speech.

[0179] Table 4 below shows the measured LSD (dB) for unvoiced sounds and transition regions.

[0180] [Table 4] As shown in Table 4, in all cases of SD and SI, the ExcitNet vocoder shows the lowest distortion between the original speech and the generated speech.

[0181] Tables 5 and 6 below show the subjective test results, with the preference test results (%). The parts that the listeners showed high preference for are shown in bold. It can be seen that the recognition quality of the synthesized speech is significantly better in the case of the ExcitNet vocoder than the rest. The evaluators were 12 Korean native listeners, and the test was conducted on 20 randomly selected utterances.

[0182] [Table 5]

[0183] [Table 6] FIG. 20 shows the Mean Opinion Score (MOS) evaluation results between the ExcitNet vocoder and other vocoders in one example.

[0184] The results were evaluated for analysis / synthesis (A / S), where acoustic features are extracted from recorded speech, and for SPSS, where acoustic features are generated from an acoustic model.

[0185] In S / A, the SI-ExcitNet vocoder performed similarly to the ITFTE vocoder, but appeared much better than the WORLD system. In all cases, the SD-ExcitNet vocoder showed the best recognition quality (4.35 and 4.47 MOS for KRF and KRM speakers, respectively). Due to the difficulty of representing high-pitched female voices, the MOS results for the KRF speaker were less good than those for the KRM speaker for the SI vocoders (WORLD, ITFTE, and SI-ExcitNet). On the other hand, the results for the KRF speaker with SD-ExcitNet were similar to those for the KRM speaker, indicating that the characteristics of each speaker must be modeled to effectively represent high-pitched voices. In the SPSS aspect, both the SD and SI-ExcitNet vocoders showed much better recognition quality than the parametric ITFTE vocoder. Although the acoustic model produced overly flat speech parameters, the ExcitNet vocoder was able to mitigate the smoothing effect by directly estimating the time-domain excitation signal. As a result, the SPSS system using the SD-ExcitNet vocoder achieved 3.78 and 3.85 MOS for the KRF and KRM speakers, respectively. The SI-ExcitNet vocoder achieved 2.91 and 2.89 MOS for the KRF and KRM speakers, respectively.

[0186] In the following, the neural vocoder that constructs the speaker adaptation model 620 described with reference to Figures 6 to 8 will be described in more detail, and comparative test results with other models will be further described. The neural vocoder of the embodiment can construct a high-quality speech synthesis system even when training data from a target speaker is insufficient, such as a speech dataset of only 10 minutes.

[0187] To solve the problem of lack of target speaker-related information caused by limited training data for a target speaker, the neural vocoder of the embodiment utilizes weights from a speaker-independently trained source model 610 that extracts universal characteristics across multiple speakers. The weights from such a source model 610 can be used to initialize the training of a speaker-adapted model 620 and fine-tuned to represent the unique characteristics of the target speaker. This adaptation process allows the deep neural network to capture the characteristics of the target speaker, thereby reducing the discontinuity problem that occurs in speaker-independent models. Experimental results described below also show that the neural vocoder of the embodiment significantly improves the recognition quality of synthesized speech compared to conventional methods.

[0188] SD indicates a speaker-dependently trained model (without using weights from the source model 610 as initial values), SI indicates a speaker-independently trained model, and SA indicates a speaker-adaptively trained model as described with reference to Figures 6 to 8 (i.e., a speaker-dependently trained model with weights from the source model 610 as initial values).

[0189] A speech corpus recorded by a Korean female speaker was used for the SD and SA models. The speech signal was sampled at 24 kHz, and each sample was quantized at 16 bits. A total of 90 (10 min), 40 (5 min), and 130 (15 min) utterances were used for training, validation, and testing. To train the SI model, speech data recorded by five Korean male speakers and five Korean female speakers, not included in the SD and SA model training, were used. For this purpose, 6,422 (10 h) and 1,080 (1.7 h) utterances were used for training and validation, respectively. The test sets of the SD and SA models were also used to evaluate the SI model.

[0190] Tables 7 and 8 below show the results of objective testing, showing the distortion between the original speech and the generated speech in terms of LSD (Log-Spectral Distance) (dB) and F0 RMSE (Root Mean Square Error) (Hz), respectively. Table 7 shows the analysis / synthesis results evaluation (A / S) when the acoustic features extracted from the recorded speech are directly used to construct the auxiliary features. Table 8 shows the evaluation of the results in SPSS. The parts with the lowest errors are marked in bold.

[0191] [Table 7]

[0192] [Table 8] It can be seen from Tables 7 and 8 that for both the WaveNet vocoder and the ExcitNet vocoder, the SA case exhibits the lowest distortion between the original speech and the generated speech.

[0193] FIG. 21 is a diagram showing a change in performance of a neural vocoder that constructs a speaker adapted model when the F0 scaling factor is changed in one example.

[0194] To verify the effectiveness of the SA-based training method of the embodiment, we investigated the change in the performance of a neural vocoder when F0 was manually changed. The SI model has been shown to be effective in generating synthetic speech with corrected pitch. Since the SA model also utilizes the SI model, it is expected to show higher performance than the SD approach.

[0195] In the test, the F0 trajectories were generated by the SPSS framework and then multiplied by scaling factors (0:6, 0:8, 1:0, and 1:2) to correct the auxiliary feature vectors. The speech signal was synthesized by a neural vocoder system.

[0196] Figure 21 shows the F0 RMSE (Hz) test results for different F0 scaling factors. It can be seen from Figure 21 that the SA model has a much lower modification error than the traditional SD model. Compared to the SI model, the SA-ExcitNet vocoder maintains the same quality even though all weights are optimized to match the characteristics of the target speaker.

[0197] We also observe that the ExcitNet vocoder performs much better than the WaveNet vocoder, which is trained on vocal fold changes (changes in the excitation signal) and is therefore considered to be more flexible in reconstructing F0-corrected speech segments than the WaveNet-based approach.

[0198] Fig. 19 shows the subjective test results, which are MOS evaluation results between SD, SI, and SA vocoders. Evaluation was performed for the analysis / synthesis (A / S) results, where acoustic features are extracted from recorded speech, and the results in SPSS, where acoustic features are generated from an acoustic model.

[0199] In the A / S results, the SD-WaveNet vocoder performed the worst because it was not possible to train the characteristics of the target speaker with limited training data. The SI-WaveNet vocoder performed similarly to the ITFTE vocoder and better than the WORLD system. It was confirmed that the use of SA in all WaveNet vocoders showed excellent performance. The results for the ExcitNet vocoder showed a similar trend to that of the WaveNet vocoder, but the ExcitNet vocoder performed much better overall because it improves the modeling accuracy of the remaining signal by isolating the formant components of the speech signal using an LP in-pass filter. As a result, the SA-ExcitNet vocoder achieved 4.40 MOS in the A / S results.

[0200] In the SPSS results, the SI-WaveNet vocoder and the SI-ExcitNet vocoder provided better recognition quality than the parametric ITFTE vocoder. As a result, it was confirmed that the SA training model of the embodiment significantly improved the quality of the synthesized speech signal compared to the conventional speaker-dependent and speaker-independent methods. As in the A / S results, the ExcitNet vocoder performed better than the WaveNet vocoder in the SPSS results. Although the acoustic model generated overly flat speech parameters, the ExcitNet vocoder was able to mitigate the smoothing effect by directly estimating the time-domain excitation signal. As a result, the SPSS system with the SA-ExcitNet vocoder achieved 3.77 MOS.

[0201] The above-mentioned devices may be realized by hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be realized using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the OS. The processing device may also respond to the execution of the software to access, record, manipulate, process, and generate data. For convenience of understanding, one processing device may be described as being used, but one skilled in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0202] The software may include computer programs, codes, instructions, or a combination of one or more of these, and may configure a processing device to operate as desired or instruct the processing device, either independently or collectively. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed and stored or executed in a distributed manner on computer systems connected by a network. The software and data may be stored on one or more computer-readable storage media.

[0203] The method according to the embodiment may be realized in the form of program instructions executable by various computer means and recorded on a computer-readable medium. Here, the medium may be one that continuously records a computer-executable program or one that temporarily records the program for execution or download. The medium may be one of various recording or storage means in the form of a single or multiple hardware devices combined, and may not be limited to a medium directly connected to a certain computer system, but may be one that is distributed over a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROMs, RAMs, flash memories, etc., configured to record program instructions. Other examples of the medium include recording media or storage media managed by application stores that distribute applications, or sites, servers, etc. that supply or distribute various other software.

[0204] Although the embodiments have been described based on the limited embodiments and drawings as above, those skilled in the art will be able to make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from that described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a manner different from that described, or may be counterbalanced or replaced by other components or equivalents, and still achieve suitable results.

[0205] Therefore, different embodiments are within the scope of the appended claims provided they are equivalent to the claims.

[0206] The following additional note is added: (Supplementary Note 1) A method for generating a speech signal implemented by a computer-implemented neural vocoder, comprising: obtaining a plurality of acoustic parameters including spectrum-related parameters and excitation-related parameters separated by a periodicity of the excitation; estimating an excitation signal based on the plurality of acoustic parameters; and generating a target speech signal by applying a linear synthesis filter based on at least one of the spectrally related parameters to the estimated excitation signal; 16. A method for generating an audio signal, comprising: (Additional Note 2) The excitation-related parameters include a first excitation parameter indicating an excitation equal to or lower than a predetermined cutoff frequency, and a second excitation parameter indicating an excitation exceeding the cutoff frequency. 2. A method for generating an audio signal as described in claim 1. (Appendix 3) The first excitation parameter is indicative of a harmonic spectrum of the excitation, and the second excitation parameter is indicative of another portion of the excitation. 3. A method for generating an audio signal as described in claim 2. (Supplementary Note 4) The spectrum-related parameters are a frequency parameter indicative of the pitch of the speech signal, an energy parameter indicative of the energy of the speech signal, a parameter indicative of whether the speech signal is voiced or unvoiced, and a parameter indicative of the line spectral frequency (LSF) of the speech signal; 2. A method for generating an audio signal as described in claim 1. (Supplementary Note 5) The step of generating the target voice signal includes: converting the parameters indicative of the LSF into a linear predictive code (LPC); and applying the linear synthesis filter based on the transformed LPC to the estimated excitation signal. Including, 5. A method for generating an audio signal as described in claim 4. (Supplementary Note 6) The plurality of acoustic parameters are generated by an acoustic model based on an input text or an input speech signal. 2. A method for generating an audio signal as described in claim 1. (Supplementary Note 7) The neural vocoder is trained based on a speech signal input for training, The training includes: separating the excitation signal from the input speech signal by applying a linear predictive analysis filter to the input speech signal; and Modeling a probability distribution of the separated excitation signal. Including, The step of estimating the excitation signal comprises: estimating excitation signals for the plurality of acoustic parameters using the modeled excitation signal probability distribution; 2. A method for generating an audio signal as described in claim 1. (Supplementary Note 8) The step of separating the excitation signal includes: converting a parameter indicating a line spectral frequency (LSF) of the input speech signal into a linear predictive code (LPC); applying the linear predictive analysis filter based on the transformed LPC of the input speech signal to the input speech signal. Including, 8. The method for generating an audio signal as described in claim 7. (Supplementary Note 9) The separated excitation signal is a residual component of the input speech signal. 8. The method for generating an audio signal as described in claim 7. (Supplementary Note 10) A computer-implemented method for training a neural vocoder, comprising: receiving an input of an audio signal; extracting from the input speech signal a plurality of acoustic parameters including spectrum-related parameters and excitation-related parameters separated according to a periodicity of excitation; separating an excitation signal from the input speech signal by applying a linear predictive analysis filter based on at least one of the spectrally related parameters to the input speech signal; and Modeling a probability distribution of the separated excitation signal. A method for training a neural vocoder, comprising: (Supplementary Note 11) The step of separating the excitation signal includes: converting a parameter indicating a line spectral frequency (LSF) of the input speech signal among the spectrum-related parameters into a linear predictive code (LPC); applying to the input speech signal the linear predictive analysis filter based on the transformed LPC of the input speech signal; Including, 11. A method for training a neural vocoder as described in claim 10. (Supplementary Note 12) The excitation-related parameters include a first excitation parameter indicating an excitation below a predetermined cutoff frequency, and a second excitation parameter indicating an excitation exceeding the cutoff frequency. 11. A method for training a neural vocoder as described in claim 10. (Supplementary Note 13) A neural vocoder, a parameter acquisition unit for acquiring a plurality of acoustic parameters including a spectrum-related parameter and an excitation-related parameter classified according to a periodicity of the excitation; an excitation signal estimator for estimating an excitation signal based on the plurality of acoustic parameters; and a speech signal generator for generating a target speech signal by applying a linear synthesis filter based on at least one of the spectrally related parameters to the estimated excitation signal; Including, a neural vocoder. (Supplementary Note 14) The speech signal generation unit includes a conversion unit that converts a parameter indicating a line spectral frequency (LSF) of the speech signal, among the spectrum-related parameters, into a linear predictive code (LPC); applying the linear synthesis filter based on the transformed LPC to the estimated excitation signal; 14. The neural vocoder of claim 13. (Supplementary Note 15) The neural vocoder is trained based on a speech signal input for training, an excitation signal separation unit that separates an excitation signal from the input speech signal by applying a linear predictive analysis filter to the input speech signal; a modeling unit for modeling a probability distribution of the separated excitation signal; Further comprising: the excitation signal estimator estimating an excitation signal for the plurality of acoustic parameters using a probability distribution of the modeled excitation signal; 14. The neural vocoder of claim 13. (Appendix 16) A computer-implemented method for training a neural vocoder, comprising: initializing weights from a source model trained on a speech dataset from multiple speakers; and generating updated weights for the initial values ​​by training a speech data set from a target speaker; Including, the updated weights are used to generate a synthetic speech signal corresponding to the target speaker. How to train a neural vocoder. (Supplementary Note 17) The weights from the source model are values ​​indicating global characteristics contained in the speech dataset that are not distinguished by speakers, The step of generating the updated weights comprises: generating the updated weights by adjusting weights from the source model to reflect unique characteristics of the target speaker contained in a speech data set from the target speaker; 17. A method for training a neural vocoder as described in appended claim 16. (Supplementary Note 18) A step of constructing a source model that trains the speech data set from the multiple speakers in a speaker-independent manner; and Obtaining the weights from the source model. Further comprising: The source model is used as an initializer for a model to train a speech dataset from the target speaker. 17. A method for training a neural vocoder as described in appended claim 16. (Appendix 19) A speech signal generation method executed by a neural vocoder trained by the training method according to claim 16, comprising: obtaining a plurality of acoustic parameters based on an input text or an input speech signal, the acoustic parameters including spectrum-related parameters generated by an acoustic model and excitation-related parameters separated by a periodicity of the excitation; estimating an excitation signal based on the plurality of acoustic parameters; and generating a target speech signal by applying a linear synthesis filter based on at least one of the spectrally related parameters to the estimated excitation signal; Including, the target speech signal is a synthetic speech corresponding to the target speaker; A method for generating an audio signal. (Supplementary Note 20) The excitation-related parameters include a first excitation parameter indicating an excitation below a predetermined cutoff frequency, and a second excitation parameter indicating an excitation exceeding the cutoff frequency. 20. The audio signal generation method of claim 19.

Claims

1. 1. A computer-implemented method for training a neural vocoder, comprising: initializing weights from a source model trained in a speaker-independent manner on speech data sets from a plurality of speakers, each of the speech data sets having a first size; and generating updated weights for the initial values ​​by training a speech data set of a second size, the second size being smaller than the first size, from a target speaker not from the plurality of speakers; Including, the updated weights are used to generate a synthetic speech signal corresponding to the target speaker. A method for generating a speech signal performed by a neural vocoder trained by a method for training a neural vocoder, comprising: obtaining a plurality of acoustic parameters based on an input text or an input speech signal, the acoustic parameters including spectrum-related parameters generated by an acoustic model and excitation-related parameters separated by a periodicity of the excitation; estimating an excitation signal based on the plurality of acoustic parameters, the estimating step including estimating an excitation signal for the plurality of acoustic parameters using a probability distribution of a modeled excitation signal; and generating a target speech signal by applying a linear synthesis filter based on at least one of the spectrally related parameters to the estimated excitation signal; Including, the target speech signal is a synthetic speech corresponding to the target speaker; A method for generating an audio signal.

2. The weights from the source model are values ​​indicative of global features contained in the speech dataset that are not differentiated by speaker, The step of generating the updated weights comprises: generating the updated weights by adjusting weights from the source model to reflect unique characteristics of the target speaker contained in a speech data set from the target speaker; 2. The method of claim 1.

3. constructing a source model that is speaker-independently trained on the speech datasets from the multiple speakers; and Obtaining the weights from the source model. Further comprising: The source model is used as an initializer for a model to train a speech dataset from the target speaker.

3. The method for generating an audio signal according to claim 1 or 2.

4. The excitation-related parameters include a first excitation parameter indicative of an excitation below a predetermined cutoff frequency, and a second excitation parameter indicative of an excitation above the cutoff frequency. A method for generating an audio signal according to any one of claims 1 to 3.

Images