An end-to-end speech splicing synthesis method

Abstract

The application provides an end-to-end voice splicing synthesis method, adopts a completely end-to-end acoustic model to model acoustic distribution of a splicing unit, the model directly takes a phoneme sequence as input, does not need expert knowledge to design an acoustic model, and greatly reduces the complexity of acoustic modeling. In addition, the end-to-end acoustic model has stronger sequence modeling capability than the traditional acoustic model, and can output smoother acoustic parameters. Further, by using a hybrid probability network as the output of the end-to-end acoustic model, the acoustic model can more accurately describe the distance of high-dimensional spectral features.

Description

TECHNICAL FIELD

[0001] The present application relates to speech synthesis technology, in particular to an end-to-end speech concatenation synthesis method. BACKGROUND

[0002] Speech synthesis, that is, artificial synthesis of human voice, is widely used in various fields from assistants to games, entertainment, etc. Recently, with speech recognition, speech synthesis has become an indispensable part of voice assistants.

[0003] Nowadays, two speech synthesis technologies are mainly used in the industry, concatenation synthesis and parameter synthesis:

[0004] Concatenation synthesis speech is to select audio segments in the audio library according to certain rules for concatenation, and when having high enough quality recordings, it can synthesize the highest quality speech, and therefore it is the most commonly used speech synthesis technology in commercial products. However, traditional concatenation synthesis greatly relies on expert knowledge to design acoustic models, and the modeling ability of acoustic models is weak, and the synthesized audio is too smooth and easy to produce concatenation jumps. In addition, the complexity of the audio library is high, and the audio library reading efficiency is low, which also restricts the industrialization landing of concatenation synthesis.

[0005] Parameter synthesis directly models the audio, which can provide high quality of concatenation synthesis technology and flexibility of parameter selection technology. However, this synthesis technology has very high computational cost and is not suitable for deployment on mobile devices. SUMMARY

[0006] To solve the problems of large computation cost of parameter synthesis and weak modeling ability of traditional concatenation synthesis, the present application combines the strong modeling ability of parameter synthesis and the high naturalness of concatenation synthesis to provide an end-to-end speech concatenation synthesis method with low computation and comparable human voice.

[0007] The following gives a brief summary of one or more aspects to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all contemplated aspects, and neither is it intended to identify key or critical elements of all aspects nor to delineate the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to the more detailed description given later.

[0008] According to an aspect of the present application, an end-to-end speech concatenation synthesis method is provided, comprising the following steps:

[0009] S1, inputting a text sequence to be synthesized;

[0010] S2, performing front-end text analysis on the text to be synthesized to obtain phoneme and prosody context information of the text, and converting the text sequence to be synthesized into a phoneme sequence with prosody annotation;

[0011] S3, performing first-level rough screening on the concatenation units in the phonetic library according to the phonetic and prosodic contexts, to obtain candidate units meeting the context conditions;

[0012] S4, using an end-to-end hybrid density network as an acoustic model, taking the phonetic sequence in S2 as input, and outputting an acoustic parameter feature distribution of the text to be synthesized;

[0013] S5, calculating acoustic costs of the candidate units on the acoustic parameter feature distribution, performing second-level fine screening on the candidate units, and obtaining a decoding network composed of candidate units of all phonemes of the text to be synthesized;

[0014] S6, in the decoding network, calculating selection costs and connection costs of the candidate units, and obtaining a globally optimal phonetic sequence in the decoding network through dynamic scaling;

[0015] S7, smoothing and concatenating the optimal phonetic sequence to obtain synthesized speech.

[0016] In an embodiment, the end-to-end hybrid density network in S4 includes an encoder, a decoder, and a duration predictor;

[0017] The encoder is composed of a multi-layer DFSMN network, which is used to model the context of each phonetic sequence; the duration predictor is composed of a multi-layer DFSMN network, which is used to predict the duration of each phoneme, and the input is the output of the encoder; the decoder is composed of two layers of LSTM, which is used to smooth the context output by the encoder, and the output is a hybrid density network.

[0018] In an embodiment, the hybrid density network models 56-dimensional features of the starting point, midpoint, and ending point positions of the phoneme, including the fundamental frequency lf0, the mel-frequency cepstrum MFCC, the differential fundamental frequency Δlf0 at the beginning of the phoneme, and ΔMFCC, respectively, and predicts the mean μ i and variance σ i of the Gaussian distribution of the above 56-dimensional features of each phoneme.

[0019] In an embodiment, the loss function of the hybrid density network is:

[0020]

[0021] Where x i is the acoustic parameter of the training sample, μ i and σ i represent the mean and standard deviation of the corresponding acoustic parameter, respectively, the inv gamma part is a penalty term of inverse gamma distribution, which is used to constrain the convergence of the model; α and λ are hyperparameters of the inverse gamma distribution, which are used to control the shape of the gamma distribution.

[0022] In an embodiment, the acoustic cost is calculated as:

[0023]

[0024] wherein, is the spectral cost, ω mfcc is the weight corresponding to the spectral cost; is the fundamental frequency cost, ω lfo is the weight corresponding to the fundamental frequency cost; is the duration cost, ω dur is the weight corresponding to the duration cost.

[0025] In an embodiment, the spectral cost is calculated as:

[0026]

[0027] wherein x mfcc represents the mel-cepstral coefficient, μ mfcc and σ mfcc represent the mean and variance of the spectrum predicted by the mixture density network, respectively.

[0028] In an embodiment, the fundamental frequency part models the multi-space distribution of Δlf0and lf0:

[0029]

[0030] wherein x lfo represents the log fundamental frequency, w uv and w v represent the probability of the fundamental frequency being in the unvoiced space and the voiced space, respectively, w uv + w v = 1.

[0031] The fundamental frequency cost is calculated as:

[0032]

[0033] wherein w uv is the weight of the voiced space predicted by the model when modeling the fundamental frequency.

[0034] In an embodiment, the duration cost is calculated as:

[0035]

[0036] wherein d is the duration of the target unit predicted by the duration predictor, d candi is the duration of the alternative unit.

[0037] In an embodiment, in S6, the selection cost is calculated as:

[0038]

[0039] wherein, ωs is the spectrum cost, mfcc ωs is the spectrum cost corresponding weight; ωb is the base frequency cost, lfo ωb is the base frequency cost corresponding weight; ωd is the duration cost, dur ωd is the duration cost corresponding weight;

[0040] Connection cost The calculation formula of the connection cost is:

[0041]

[0042] wherein, ωsc is the spectrum connection cost, Δmfcc ωsc is the spectrum connection cost corresponding weight;

[0043] ωbc is the base frequency connection cost, Δlfo ωbc is the base frequency connection cost corresponding weight.

[0044] In an embodiment, the minimum concatenation unit in the sound bank is a half-syllable coda.

[0045] The beneficial effects of the embodiment of the present application are: the method directly uses a mixed density network to describe the distribution of the acoustic parameters of the target unit, greatly improving the accuracy of acoustic modeling. By using an end-to-end modeling scheme, the acoustic parameters are directly generated from the phoneme sequence without the need for expert knowledge to design a set of front-end problems of linguistics prosody, greatly reducing the complexity of acoustic modeling. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor by reading the drawings.

[0047] The above features and advantages of the present application can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components having similar related properties or features can have the same or similar reference numerals.

[0048] Figure 1 is a method flowchart of the embodiment of the present application;

[0049] Figure 2 is a schematic diagram of an end-to-end hybrid density network model of an embodiment of the present application. DETAILED DESCRIPTION

[0050] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are merely exemplary and should not be understood as limiting the scope of protection of the present application in any way.

[0051] Splicing synthesis is to select suitable phoneme segments from a phonetic database according to certain rules to splice into continuous and natural long sentences. The phonetic database stores the waveforms and acoustic characteristics of each phoneme in different phoneme contexts and different prosodic phenomena. At the same time, the phonetic database maintains an index table which stores the phoneme contexts and prosodic context information of each phoneme, and the splicing system uses the index table to find phonemes.

[0052] The phonetic database used by the present method will be described first as follows:

[0053] For the preparation of the phonetic database, the present application uses semi-consonant-vowel as the smallest splicing unit, which ensures that the splicing position of the splicing synthesis system is in the stable segment of pronunciation, reduces the discontinuity of the formant between the splicing units, and greatly improves the naturalness of splicing synthesis. In order to reduce the redundancy of the phonetic database, the phonetic database should cover as many phoneme co-pronunciations and prosodic phenomena as possible with as little corpus as possible, so a greedy algorithm is used to select the text when recording the phonetic database.

[0054] The phonemes used in the phonetic database of the present application are consonant-vowels. However, unlike the traditional splicing scheme, in order to weaken the discontinuity of the formant between the splicing units, the cutting position of the phoneme segment in the phonetic database is in the stable segment of phoneme pronunciation, that is, all phonemes are "semi-consonant-vowel", which will retain the latter half of the initial and the former half of the vowel for each phoneme. When splicing units, the splicing position is in the stable segment of the initial or the vowel, which greatly improves the naturalness of splicing synthesis.

[0055] Since the phonetic database for splicing synthesis is large, it poses a great challenge to the IO and storage of terminal devices, so the phonetic database needs to be trimmed while ensuring the storage continuity of the units in the phonetic database as much as possible. Specifically, a large-scale synthesis test can be performed on the splicing system, and the unit IDs used in the decoding network are recorded. Finally, these IDs are compared with the phonetic database to find the unused units in the phonetic database, and the unused units are trimmed.

[0056] In the storage of the phonetic database, phonemes with the same phoneme context are stored continuously, which can ensure the reading efficiency of splicing.

[0057] Based on the above phonetic database, an embodiment of the present application provides an end-to-end speech splicing synthesis method, as shown inFigure 1 As shown in the method comprises the following steps:

[0058] S1, input the text sequence to be synthesized.

[0059] S2, perform front-end text analysis on the text to be synthesized to obtain phoneme and prosody context information of the text, and convert the text sequence to be synthesized into a phoneme sequence with prosody annotation.

[0060] Specifically, the front-end text analysis includes: performing text normalization, segmenting the text to be synthesized; performing multi-phonetic character disambiguation to determine the correct pronunciation of the text; and performing prosody analysis to predict the prosody level of the syllable boundary.

[0061] S3, according to the phoneme and prosody context, performing first-level coarse screening on the concatenation units in the phonetic library, that is, according to the context index table, screening to obtain the candidate units that meet the context conditions. Since the number of candidate units screened out is large, in order to reduce the search complexity of the decoding network, we need to finely screen the candidate units to further reduce the calculation amount of selecting concatenation units through the acoustic model.

[0062] S4, using an end-to-end hybrid density network as an acoustic model, taking the phoneme sequence in S2 as input, and outputting the acoustic parameter feature distribution of the text to be synthesized.

[0063] As shown in the figure, Figure 2 The end-to-end hybrid density network includes an encoder, a decoder, and a duration predictor;

[0064] The encoder is composed of a multi-layer DFSMN network, which has a long receptive field and can model the context of each phoneme sequence.

[0065] The duration predictor is composed of a multi-layer DFSMN network, which is used to predict the duration of each phoneme, and the input is the output of the encoder.

[0066] The decoder is composed of two layers of LSTM, which is used to smooth the context output by the encoder, and the output is a hybrid density network. By directly using a hybrid density network to describe the acoustic parameter distribution of the target unit, the need for manual design of language features is avoided, and the accuracy of acoustic modeling is greatly improved.

[0067] The hybrid density network models the 56-dimensional features of the starting point, midpoint, and endpoint positions of the phoneme, the fundamental frequency lf0, the mel-frequency cepstrum MFCC, the differential fundamental frequency Δlf0 at the beginning of the phoneme, and the ΔMFCC, respectively. And predict the mean μ i and variance σ i of the Gaussian distribution of each phoneme in the above 56-dimensional features.

[0068] The loss function of the hybrid density network is:

[0069]

[0070] where x i is the acoustic parameter of the training sample. During the training process of the model, the variance will diverge so that the training of the model will fail, thus the method uses an inverse gamma distribution as a penalty term (invgamma term) to constrain the model convergence. The alpha and lambda in the penalty term are hyperparameters of the inverse gamma distribution, used to control the shape of the gamma distribution.

[0071] S5, calculate the acoustic cost of the candidate unit on the acoustic parameter feature distribution, perform second-level fine screening on the candidate unit, and obtain a decoding network composed of candidate units of all phonemes of the text to be synthesized.

[0072] where the acoustic cost is calculated according to the following formula:

[0073]

[0074] In the formula, f is the spectrum cost, ω mfcc is the spectrum cost corresponding weight; is the fundamental frequency cost, ω lfo is the fundamental frequency cost corresponding weight; is the duration cost, ω dur is the duration cost corresponding weight, and each weight needs to be adjusted according to the development set.

[0075] The spectrum cost is used to measure the matching degree of the acoustic parameters of the candidate unit and the target, and the calculation formula is:

[0076]

[0077] where x mfcc represents the mel-frequency cepstral coefficient, μ mfcc and σ mfcc represent the mean and variance of the spectrum predicted by the hybrid density network, respectively.

[0078] For the fundamental frequency, the fundamental frequency of the unvoiced part does not exist (at this time, the fundamental frequency is set to a negative number, that is, x lf0 < 0), therefore the multi-space distribution is used to model Δlf0 and lf0:

[0079]

[0080] where x lfo represents the logarithmic fundamental frequency, w uv and w vrespectively represent the probability of the fundamental frequency being in the unvoiced space and the voiced space, w uv +w v = 1.

[0081] By modeling the fundamental frequency with multiple spaces, the continuous smooth assumption is not needed to model the fundamental frequency, and the problem of inaccurate calculation of the connection cost of the fundamental frequency between the front and rear unvoiced consonants is solved.

[0082] Specifically, in addition to predicting the mean μ lf0 and the variance σ lf0 , the model also predicts the weight w uv of the voiced space when modeling the fundamental frequency, and the acoustic cost of the fundamental frequency is calculated using the following formula:

[0083]

[0084] For the phoneme duration, the Euclidean distance is used to measure whether the length of the candidate unit is appropriate, and the calculation formula of the acoustic cost is specifically as follows:

[0085]

[0086] where d is the duration of the target unit predicted by the duration predictor, and d candi is the duration of the candidate unit.

[0087] By calculating the acoustic cost a of each candidate phoneme, it can be finally determined whether the candidate unit meets the prosody, intonation, speech rate, stress, etc. of the current phoneme position.

[0088] Through step S5, we obtain all the candidate units of the phonemes of the text to be synthesized, and all the candidate units constitute a decoding network.

[0089] S6, in the decoding network, the selection cost and the connection cost of the candidate units are calculated, and the globally optimal phoneme sequence in the decoding network is obtained through dynamic scaling;

[0090] The calculation formula of the selection cost is specifically as follows:

[0091]

[0092] wherein, is the spectrum cost, ω mfcc is the weight corresponding to the spectrum cost; is the fundamental frequency cost, ω lfo is the weight corresponding to the fundamental frequency cost; is the duration cost, ω dur is the weight corresponding to the duration cost;

[0093] It indicates whether the candidate units of the current phoneme can meet the expected listening experience in terms of fundamental frequency and spectrum.

[0094] Connection cost The calculation formula is:

[0095]

[0096] This indicates whether the concatenation of the current phoneme as a candidate unit can maintain the smoothness of the previous and subsequent concatenations. If the connection cost is too high, the concatenation synthesis will produce abrupt changes that are uncomfortable to the human ear.

[0097] Connection cost The calculation formula is:

[0098]

[0099] in, For the connection cost of the spectrum, ω Δmfcc The weights corresponding to the spectrum connectivity costs;

[0100] For the cost of baseband connection, ω Δlfo The weights corresponding to the baseband connection cost.

[0101] Furthermore, the total cost can be adopted. To comprehensively represent the cost of choice and connection cost Total cost The calculation formula is:

[0102]

[0103] w t With w c The weights for selection cost and connection cost are respectively, and they also need to be manually adjusted based on the development set.

[0104] After calculating the total cost, the Viterbi algorithm can be used to select the optimal phoneme sequence in the decoding network.

[0105] S7. Smoothly concatenate the optimal phoneme sequence to obtain the synthesized speech. In this embodiment, the smooth concatenation uses the WSOLA (Waveform Similarity Overlap-Add) algorithm.

[0106] In summary, this application provides a splicing and synthesis system with low computational cost and comparable to human voice, and also provides a corresponding sound library preparation scheme.

[0107] Specifically, the application adopts a complete end-to-end acoustic model to model the acoustic distribution of the concatenation unit, the model directly takes the phoneme sequence as the input, and does not need expert knowledge to design the acoustic model, which greatly reduces the complexity of acoustic modeling. In addition, the end-to-end acoustic model has stronger sequence modeling ability than the traditional acoustic model, and can output smoother acoustic parameters.

[0108] In order to more accurately describe the acoustic parameters of the concatenation unit, the application innovatively uses a hybrid probability network as the output of the end-to-end acoustic model, so that the acoustic model has more accurate distance description for high-dimensional spectral features.

[0109] For the preparation of the sound library, the application adopts a semi-vowel as the smallest concatenation unit, which ensures that the concatenation position of the concatenation synthesis system is in the stable pronunciation segment, reduces the discontinuity of the resonance peaks between the concatenation units, and greatly improves the naturalness of the concatenation synthesis.

[0110] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0111] The foregoing description of the present disclosure is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to the present disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the spirit or scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the examples and designs described herein, but should be granted the broadest scope consistent with the principles and novel features disclosed herein.

[0112] The above is only a preferred example of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An end-to-end speech splicing and synthesis method, characterized in that, Includes the following steps: S1. Input the text sequence to be synthesized; S2. Perform front-end text analysis on the text to be synthesized to obtain the phonemes and prosodic context information of the text, and convert the text sequence to be synthesized into a phoneme sequence with prosodic annotation. S3. Based on phonemes and prosodic context, perform a first-level coarse screening of the splicing units in the phonetic library to obtain candidate units that meet the context conditions. S4. Use an end-to-end hybrid density network as the acoustic model, take the phoneme sequence in S2 as input, and output the acoustic parameter feature distribution of the text to be synthesized. S5. Calculate the acoustic cost of the candidate units in the acoustic parameter feature distribution, perform a second-level fine screening of the candidate units, and obtain the decoding network composed of candidate units of all phonemes of the text to be synthesized. S6. In the decoding network, calculate the selection cost and connection cost of candidate units, and obtain the globally optimal phoneme sequence in the decoding network through dynamic regularization. S7. Smoothly splice the optimal phoneme sequence to obtain the synthesized speech.

2. The end-to-end speech splicing and synthesis method according to claim 1, characterized in that, The end-to-end hybrid density network in S4 includes an encoder, a decoder, and a duration predictor. The encoder consists of a multi-layer DFSMN network, which is used to model the context of each phoneme sequence. The duration predictor consists of a multi-layer DFSMN network and is used to predict the duration of each phoneme. Its input is the output of the encoder. The decoder consists of two LSTM layers, which smooth the context of the encoder output. Its output is a hybrid density network.

3. The end-to-end speech splicing and synthesis method according to claim 2, characterized in that, The hybrid density network models 56 features, including the fundamental frequency lf0 at the start, mid, and end points of the phoneme, the Mel-frequency cepstral (MFCC), the differential fundamental frequency Δlf0 at the beginning of the phoneme, and ΔMFCC. It then predicts the mean μ of the Gaussian distribution of each phoneme within these 56 features. i and variance σ i .

4. The end-to-end speech splicing and synthesis method according to claim 3, characterized in that, The loss function of the hybrid density network is: Where, x i For the acoustic parameters of the training samples, μ i With σ i α and λ represent the mean and standard deviation of the corresponding acoustic parameters, respectively. The invgamma part is the penalty term of the inverse gamma distribution, which is used to constrain model convergence. α and λ are hyperparameters of the inverse gamma distribution, which are used to control the shape of the gamma distribution.

5. The end-to-end speech splicing and synthesis method according to claim 4, characterized in that, In S5, the acoustic cost The calculation formula is: in, For the spectral cost, ω mfcc The weights correspond to the spectrum cost; For the cost of the fundamental frequency, ω lfo The weight corresponding to the base frequency cost; For the cost of time, ω dur The weight corresponds to the time cost.

6. The end-to-end speech splicing and synthesis method according to claim 5, characterized in that, The formula for calculating the spectral cost is as follows: Where x mfcc Represents the Mel-frequency cepstral coefficients, μ mfcc With σ mfcc denoted as mean and variance, respectively, of the predicted spectrum of the hybrid density network.

7. The end-to-end speech splicing and synthesis method according to claim 5, characterized in that, The fundamental frequency component is modeled using a multi-spatial distribution for Δlf0 and lf0: Where, x lfo w represents the logarithmic fundamental frequency. uv with w v Let w represent the probability that the fundamental frequency is in the unvoiced and voiced spaces, respectively. uv +w v =1; The formula for calculating the fundamental frequency cost is:

8. The end-to-end speech splicing and synthesis method according to claim 5, characterized in that, The formula for calculating the time cost is as follows: Where d is the duration of the target unit predicted by the duration predictor, d candi The duration of the alternative unit.

9. The end-to-end speech splicing and synthesis method according to claim 5, characterized in that, In S6, the formula for calculating the selection cost is: in, For the spectral cost, ω mfcc The weights correspond to the spectrum cost; For the cost of the fundamental frequency, ω lfo The weight corresponding to the base frequency cost; For the cost of time, ω dur The weight corresponding to the time cost; Connection cost The calculation formula is: in, For the connection cost of the spectrum, ω Δmfcc The weights correspond to the spectrum connectivity costs; For the cost of baseband connection, ω Δlfo The weights correspond to the baseband connection cost.

10. The end-to-end speech splicing and synthesis method according to claim 1, characterized in that, The smallest splicing unit in the sound library is a semi-vowel.

Images