Phoneme duration prediction method based on GA-IMV

Abstract

The present invention discloses a GA-IMV-based phoneme duration prediction method. The method is based on a GA-IMV-based speech synthesis model. The speech synthesis model includes an encoder module, a GA-IMV module, and a decoder module. The encoder module inputs a phoneme sequence, which is obtained by transliterating text into phonemes. The speech synthesis model outputs an attention weight matrix and a spectral feature matrix. The phoneme duration prediction method calculates the phoneme duration sequence based on the attention weight matrix. The GA-IMV-based speech synthesis model reduces the average phoneme duration deviation of the synthesized audio, thereby improving the quality of the synthesized audio.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech synthesis, and in particular relates to a phoneme duration prediction method based on GA-IMV. Background Art

[0002] Speech synthesis technology uses computer programs to simulate human pronunciation and convert text input into understandable speech output. Currently, there are two main models in speech synthesis technology: autoregressive models and non-autoregressive models.

[0003] Autoregressive speech synthesis models are a relatively traditional type of speech synthesis method. These models use a step-by-step prediction approach, where the output of the previous step becomes the input for the next step. While this serial prediction approach can generate high-quality speech, the step-by-step prediction process is slow and cannot meet real-time requirements.

[0004] Unlike autoregressive speech synthesis models, the core feature of non-autoregressive speech synthesis models is that they can generate complete speech sequences in parallel at once, without the need for step-by-step prediction required by autoregressive models. This significantly increases generation speed and makes them more suitable for real-time speech synthesis. However, to achieve this, non-autoregressive models must clearly define the duration of each phoneme (the basic unit of speech) during the generation phase. This requires the ability to predict phoneme duration. Training a phoneme duration model requires extracting the duration characteristics of each phoneme.

[0005] However, there are two major problems with current phoneme duration feature extraction methods:

[0006] 1. The existing methods are not accurate enough. The existing phoneme duration prediction methods fail to fully consider the contextual dependencies between phonemes and the complex mapping relationship between phoneme sequences and acoustic sequences, resulting in inaccurate duration prediction. For example, the patent with publication number CN102222501A discloses a method for generating duration parameters in speech synthesis, which is used to generate state duration parameters in speech synthesis based on a hidden Markov model. The method generates a more natural state residence time by combining the state duration model with the duration overall variance model. By creating a special training corpus and using a duration overall variance analyzer to generate training samples, the duration overall variance model is further optimized. This method effectively solves the problem of over-average state duration in traditional methods, making the synthesized speech more natural and expressive in duration distribution. Since the state sequence in the hidden Markov model is an abstract coarse-grained model of the actual pronunciation process, it cannot fully utilize the global context information, which makes it difficult for the phoneme sequence to accurately match the real acoustic sequence, and thus causes the extracted phoneme duration features to be inaccurate.

[0007] Second, existing methods lack scalability. Current phoneme duration extraction methods are mostly designed for non-autoregressive speech synthesis models and cannot be applied to autoregressive speech synthesis models, which limits the scope of application of these technologies. For example, patent publication number CN202410117821 discloses an improved phoneme duration prediction method based on the VITS neural spline stream. By improving the random duration predictor in the VITS model and introducing a neural spline stream-based method, the model performs a detailed analysis of the input data distribution and divides the block intervals. This enables more efficient coupled transformation within each block interval, thereby extracting the phoneme duration and improving the quality of VITS synthesized speech. However, this solution only improves the random duration predictor module of the VITS (Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech) streaming speech synthesis model and is difficult to apply to autoregressive speech synthesis models, which limits its scope of application. Summary of the Invention

[0008] In view of the problems existing in the prior art, the purpose of the present invention is to provide a speech synthesis model based on GA-IMV, which is improved based on the autoregressive speech synthesis model Tacotron2 (Jonathan S, Ruoming P, Ron JW, Mike S, Navdeep J, Zongheng Y, Zhifeng C, Yu Z, Yuxuan W, RJ S, Rif AS, Yannis A, Yonghui W, et al. Natural TTS Synthesis by Conditioning Wavenet on MELSpectrogram Predictions[J], 2018 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING(ICASSP), 2018: 4779-4783.), and replaces the position-sensitive attention mechanism in the autoregressive speech synthesis model Tacotron2 with a guided attention mechanism GA and index mapping vector IMV constraints.

[0009] Another object of the present invention is to provide a phoneme duration prediction method based on GA-IMV, which can extract phoneme duration more accurately.

[0010] The purpose of the present invention is achieved through the following technical solutions.

[0011] A speech synthesis model based on GA-IMV includes: an encoder module, a GA-IMV module and a decoder module, wherein the input of the encoder module is a phoneme sequence and the output is a phoneme global feature matrix S;

[0012] The input of the GA-IMV module is the phoneme global feature matrix S and T hidden states, and the output of the GA-IMV module is T groups of attention weight column vectors and T mapping vectors;

[0013] The decoder module output includes: T first stop signs and T groups of spectral features. The T hidden states are output one by one by the decoder module. After the decoder module outputs the t-1th hidden state to the GA-IMV module, the GA-IMV module outputs the tth group of attention weight column vectors and the tth mapping vector. The decoder module receives the tth mapping vector and then outputs the tth hidden state, the tth first stop sign and the tth group of spectral features. t=1, ..., T, so that t increases in the range of 1 to T until the decoder module outputs T first stop signs and T groups of spectral features and the GA-IMV module outputs T groups of attention weight column vectors; the T first stop signs are composed of the first stop sign sequence B in the order of increasing t, the T groups of spectral features are composed of the spectral feature matrix E in the order of increasing t, and the T groups of attention weight column vectors are composed of the attention weight matrix A in the order of increasing t;

[0014] The tth mapping vector C t For: C t =α t ′S, α t ′ represents the attention weight column vector α of the tth group t The transposed vector of

[0015] α t =softmax(V·tanh(W q h t-1 +W k S+W f ReLU(Conv(α′ t-1 ))+b);

[0016] Where, α t is the column vector of the t-th group of attention weights, softmax() represents the normalization function, α′ t-1 Represents the attention weight column vector α of the t-1 group t-1 The transposed vector, when t=1,α′ 1-1 is a zero vector, ReLU(Conv(α′ t-1 )) represents the extraction of α′ through convolution and nonlinear activation function ReLU t-1 Information, V, Wq 、W k and W f are all learnable parameter matrices, b is the bias term, h t-1 is the t-1th hidden state. When t=1, the hidden state h t-1 is the zero vector.

[0017] In the above technical solution, the encoder module includes: an embedding module, a three-layer convolution module and a bidirectional long short-term memory network module connected in sequence. The embedding module inputs a phoneme sequence and outputs a phoneme embedding matrix. The three-layer convolution module inputs a phoneme embedding matrix and outputs a phoneme local feature matrix. The bidirectional long short-term memory network module inputs a phoneme local feature matrix and outputs a phoneme global feature matrix S.

[0018] In the above technical solution, the phoneme global feature matrix corresponding to a phoneme sequence is H is the dimension of each phoneme after encoding, and N is the length of the phoneme sequence.

[0019] In the above technical solution, the decoder module includes: a 2-layer long short-term memory network module, a linear module, and a 5-layer convolution module. The 2-layer long short-term memory network module is used to input the mapping vector and the first stop mark and output the hidden state; the linear module inputs the hidden state and outputs the first stop mark; the 5-layer convolution module inputs the hidden state and outputs the spectral features; wherein:

[0020] When the 2-layer long short-term memory network module inputs the t-th mapping vector, the 2-layer long short-term memory network module outputs the t-th hidden state. The linear module inputs the t-th hidden state and outputs the t-th first stop mark and controls whether the 2-layer long short-term memory network module outputs the t+1-th hidden state when it inputs the t+1-th mapping vector. The 5-layer convolution module inputs the t-th hidden state and outputs the t-th spectral feature. The GA-IMV module inputs the t-th hidden state and is used to calculate the t+1-th group of attention weight column vectors and the t+1-th mapping vector.

[0021] In the above technical solution, the column vector α of the tth group of attention weights is t for Attention weight matrix corresponding to a phoneme sequence N is the length of the phoneme sequence.

[0022] A training method for a speech synthesis model based on GA-IMV includes the following steps:

[0023] S1. Prepare a training set. The training set consists of multiple samples. Each sample includes: a phoneme sequence, a mel-spectrogram feature matrix, and a second stop mark sequence (the mel-spectrogram feature matrix in the sample is the true value of the spectrum feature matrix E, and the second stop mark sequence in the sample is the true value of the first stop mark sequence B). Each phoneme sequence is obtained by phoneme transcription of a text. An audio segment corresponding to the text is divided into several short time frames. A second stop mark is set for each short time frame. The second stop mark of each short time frame is used to indicate whether there is a subsequent short time frame after the short time frame. All the second stop marks of each audio segment constitute a second stop mark sequence.

[0024] S2, the phoneme sequence, Mel spectrum feature matrix and second stop sign sequence of each sample in the training set are input into the speech synthesis model to train the speech synthesis model based on GA-IMV. During the training process, the speech synthesis model performs loss calculation through the Loss calculation module. ALL Calculate and get the loss value Loss ALL , use the back propagation algorithm to optimize the parameters, when the loss value Loss ALL If it is less than 1, the training is completed, where:

[0025] Loss ALL =Loss GA +Loss IMV +Loss MEL +Loss STOP-TOKEN

[0026] W is the guidance matrix, the dimension of W is the same as the dimension of the attention weight matrix A, and N is the number of phonemes in the phoneme sequence;

[0027] Loss IMV =Loss mono +Loss cont +Loss comp ,in,

[0028]

[0029] Loss comp =|IMV1|+|IMV T -N|

[0030]

[0031] Among them, n is the index number of the nth phoneme in the phoneme sequence, A n，t is the element in the nth row and tth column of the attention weight matrix A;

[0032]

[0033] The first stop flag and the second stop flag are binary values ​​respectively. When the first stop flag controls decoding, the first stop flag is assigned a value of 0, and when the first stop flag controls decoding to stop, the first stop flag is assigned a value of 1; when the second stop flag indicates that there is a subsequent short time frame after the short time frame, the second stop flag is assigned a value of 0, and when the second stop flag indicates that there is no subsequent short time frame after the short time frame, the second stop flag is assigned a value of 1; is the value assigned to the tth first stop mark, Assign the value of the second stop flag of the tth.

[0034] Loss MEL is calculated as follows:

[0035]

[0036] in, Represents all elements in the tth column of the Mel spectrum feature matrix, represents the spectrum characteristics of group t.

[0037] A phoneme duration prediction method based on GA-IMV includes the following steps:

[0038] S1, input the phoneme sequence to be predicted into the trained speech synthesis model to obtain the attention weight matrix A and the spectral feature matrix E;

[0039] S2, calculate each group of row vectors β in the attention weight matrix A n The dynamic threshold of n is the nth row vector in the attention weight matrix A, β n The dynamic threshold is G n , G n The calculation method is as follows:

[0040] G n =μ n +σ n ;

[0041] Among them, μ n is the row vector β n The mean of all elements in , σ n is the row vector β n The standard deviation of all elements in ;

[0042] S3, for row vector β n Medium and larger than G n And count the consecutive elements to get the row vector β nThe total number of consecutive elements of the row vectors is formed into a sequence of consecutive element totals f according to the increasing order of n value. n ; Calculate D total , and then calculate the phoneme duration d=D corresponding to each element total / T,D total The calculation formula is as follows:

[0043]

[0044] Where L is the frame length of a segment of audio into several short time frames, M is the frame shift of a segment of audio into several short time frames, and F is the sampling frequency of the audio.

[0045] S4, the total number of consecutive elements sequence f n Multiplying by the phoneme duration d, the phoneme duration sequence D of the phoneme sequence to be predicted is obtained = {D1, D2, D3, ..., D N}, each element D in the phoneme duration sequence D n The predicted value of the phoneme duration of a phoneme in the phoneme sequence to be predicted.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. The phoneme duration prediction method of the present invention adopts a combination of GA and IMV to enable the speech synthesis model to learn a more accurate attention weight matrix, and then extract more precise phoneme duration features.

[0048] 2. The phoneme duration prediction method of the present invention enhances the ability of the speech synthesis model to learn the mapping relationship between the phoneme sequence and the spectral feature matrix, and further improves the quality of the synthesized speech of the autoregressive speech synthesis model.

[0049] 3. Using the phoneme duration features extracted by the phoneme duration prediction method of the present invention to train a non-autoregressive speech synthesis model can provide more accurate time alignment information for the speech synthesis model, further enhancing the quality of the synthesized speech of the non-autoregressive speech synthesis model.

[0050] 4. The phoneme duration features extracted by the phoneme duration prediction method of the present invention are applicable to autoregressive speech synthesis models and non-autoregressive speech synthesis models, and have high generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a structural diagram of the speech synthesis model;

[0052] Figure 2 This is the training flow chart of the GA-IMV-based speech synthesis model. DETAILED DESCRIPTION

[0053] The phoneme duration prediction method of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0054] Example 1

[0055] like Figure 1 As shown, a speech synthesis model based on GA-IMV includes: an encoder module, a GA-IMV module and a decoder module, wherein the input of the encoder module is a phoneme sequence and the output is a phoneme global feature matrix S;

[0056] The input of the GA-IMV module is the phoneme global feature matrix S and T hidden states. The output of the GA-IMV module is T groups of attention weight column vectors and T mapping vectors, where the dimension of each group of attention weight column vectors is N×1;

[0057] The output of the decoder module includes: T first stop signs and T groups of spectral features. T hidden states are output one by one by the decoder module. When the decoder module outputs the t-1th hidden state (h t-1 ), the GA-IMV module outputs the t-th group of attention weight column vector (α t ) and output the t-th mapping vector (C t ), the decoder module then outputs the tth hidden state (h t ), the tth first stop mark (p t ) and the t-th group of spectral features (e t , column vector), t = 1, ..., T, so that t increases in the range of 1 to T, until the decoder module outputs T first stop signs and T groups of spectral features, and the GA-IMV module outputs T groups of attention weight column vectors; the T first stop signs are composed of the first stop sign sequence B in the order of t increasing, and the T groups of spectral features are composed of the spectral feature matrix E in the order of t increasing (the tth column in the spectral feature matrix E is the spectral feature e t ), the T groups of attention weight column vectors are formed into the attention weight matrix A in the order of t increasing (the tth column in the attention weight matrix A is the attention weight column vector α t ). The attention weight column vector α of the tth group t for A phoneme sequence corresponds to N is the sequence length of the phoneme sequence (the number of phonemes).

[0058] The tth mapping vector C t For: C t =α t ′S, α t ′ represents the attention weight column vector α of the tth group tThe transposed vector of

[0059] α t =softmax(V·tanh(W q h t-1 +W k S+W f ReLU(Conv(α′ t-1 ))+b)

[0060] Where, α t is the column vector of the t-th group of attention weights, softmax() represents the normalization function, α′ t-1 Represents the attention weight column vector a of the t-1 group t-1 The transposed vector, when t=1,α′ 1-1 is a zero vector, ReLU(Conv(α′ t-1 )) represents the extraction of α′ through convolution and nonlinear activation function ReLU t-1 Information, V, W q 、W k and W f are all learnable parameter matrices, b is the bias term, h t-1 is the t-1th hidden state. When t=1, the hidden state h 1-1 is the zero vector.

[0061] Example 2

[0062] A speech synthesis model based on GA-IMV, based on Example 1, wherein the encoder module includes: an embedding module, a three-layer convolution module and a bidirectional long short-term memory network module connected in sequence, the embedding module is used to input a phoneme sequence, the output of the embedding module is a phoneme embedding matrix, the embedding module is used to convert the phoneme sequence into a digital sequence and expand the digital sequence into a higher-dimensional phoneme embedding matrix, the three-layer convolution module inputs the phoneme embedding matrix and outputs the phoneme local feature matrix, the three-layer convolution module is used to capture the local features of the phoneme embedding matrix (the local features are mainly word boundaries and syllable structures), which helps the model understand the local dependencies in the input sequence, the bidirectional long short-term memory network module inputs the phoneme local feature matrix and outputs the phoneme global feature matrix S, and the bidirectional long short-term memory network module is used to extract global context information (forward and reverse information) in the phoneme local feature matrix.

[0063] The phoneme global feature matrix corresponding to a phoneme sequence H is the dimension of each phoneme after encoding, and H is set to 256 in this embodiment.

[0064] The decoder module includes: a 2-layer long short-term memory network module, a linear module, and a 5-layer convolution module. The 2-layer long short-term memory network module is used to input the mapping vector and the first stop mark, and the 2-layer long short-term memory network module outputs the hidden state; the linear module inputs the hidden state and outputs the first stop mark; the 5-layer convolution module inputs the hidden state and outputs the spectral features;

[0065] When the 2-layer LSTM network module inputs the t-th mapping vector (C t ) (At this time, the 2-layer long short-term memory network module also inputs the t-1th first stop mark (p t-1 ) and controls the 2-layer long short-term memory network module to decode), the 2-layer long short-term memory network module outputs the t-th hidden state (h t ), the linear module inputs the tth hidden state (h t ) after the output of the t-th first stop mark (p t ) and control the 2-layer long short-term memory network module to input the t+1th mapping vector (C t+1 ) whether to output the t+1th hidden state (h t+1 ), that is, controlling the 2-layer long short-term memory network module to input the t+1th mapping vector C t Then decode or stop decoding; the 5-layer convolution module inputs the tth hidden state (h t ), output the tth spectrum feature (e t ); GA-IMV module inputs the tth hidden state (h t ) and used for the t+1th group of attention weight column vectors (α t+1 ) and the t+1th mapping vector (C t+1 ) calculation.

[0066] The first stop flag prevents the decoder module from generating too many or too few spectral features. A two-layer long short-term memory network captures contextual relationships, ensuring temporal coherence and consistency in the generated speech features (hidden states). A five-layer convolutional module extracts local time-frequency information (detailed features along the time axis) from the hidden states layer by layer, outputting more detailed spectral features.

[0067] Example 3

[0068] Based on Example 2, a dataset D consisting of X samples is prepared. Each sample includes: a phoneme sequence, a mel-spectrogram feature matrix, and a second stop sign sequence. Each phoneme sequence is composed of English letters (phonemes), and each phoneme sequence is obtained by phoneme transliteration of a text (phoneme transliteration method: converting Chinese text into a phoneme sequence composed of English letters according to pinyin rules);

[0069] The method for obtaining each Mel spectrum feature matrix is ​​as follows: a segment of audio (each segment of audio is a continuous audio signal, and the sampling frequency of the audio signal is F) is divided into several short time frames according to the frame length L and the frame shift M, and each short time frame is Fourier transformed to obtain the frequency domain features of each short time frame. Then, the logarithm of all the frequency domain features of the audio segment is taken to obtain a Mel spectrum feature matrix;

[0070] The method for obtaining each second stop mark sequence is as follows: a second stop mark is set for each short time frame, and all the second stop marks of each audio segment constitute a second stop mark sequence. The second stop mark of each short time frame is used to indicate whether there is a subsequent short time frame after the short time frame.

[0071] Each sample has corresponding text and audio.

[0072] In this embodiment, X=9179, F=16000 Hz, frame length L=512, frame shift M=128, and H=256.

[0073] Example 4

[0074] like Figure 2 As shown, a training method for a speech synthesis model based on GA-IMV includes the following steps:

[0075] S1, prepare a training set: divide all samples in the dataset D of Example 3 into a training set and a test set in a ratio of 8:2.

[0076] S2, the phoneme sequence, Mel spectrum feature matrix and second stop sign sequence of each sample in the training set are input into the GA-IMV based speech synthesis model of Example 2, so that the GA-IMV based speech synthesis model is trained. During the training process, the speech synthesis model is subjected to a loss value Loss calculation module. ALL Calculate and get the loss value Loss ALL , using the back propagation algorithm (Ian Goodfellow, Joshua Bengio, Aaron Courville. (2017). Deep Learning [M]. Zhao Shenjian, Li Yujun, Fu Tianfan, Li Kai, translated. Beijing: People's Posts and Telecommunications Press.) to optimize the parameters, when the loss value Loss ALL If it is less than 1, the training is completed.

[0077] Loss ALL =Loss GA +Loss IMV +Loss MEL +Loss STOP-TPLEN ;in,

[0078]

[0079] W is the guide matrix (Hideyuki Tachibana, Katsuya Uenoyama, Shunsuke Aihara. Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention[J], Computing Research Repository, 2018: 4784-4788.), the dimension of W is the same as that of A, and N is the sequence length (number of phonemes) of the phoneme sequence.

[0080] IMV loss function Loss IMV is calculated as follows:

[0081] Loss IMV =Loss mono +Loss cont +Loss comp ,in,

[0082]

[0083] Loss comp =|IMV1|+|IMV T -N|

[0084]

[0085] Among them, n is the index number of the nth phoneme in the phoneme sequence, A n，t is the element in the nth row and tth column of the attention weight matrix A.

[0086] Loss STOP-TOKEN is calculated as follows:

[0087]

[0088] The first stop flag is a predicted value of the linear module, and the second stop flag is a true value of the short-time frame. The first stop flag and the second stop flag are binary values. When the first stop flag controls decoding, the first stop flag is assigned a value of 0, and when the first stop flag controls decoding to stop, the first stop flag is assigned a value of 1; when the second stop flag indicates that there is a subsequent short-time frame after the short-time frame, the second stop flag is assigned a value of 0, and when the second stop flag indicates that there is no subsequent short-time frame after the short-time frame, the second stop flag is assigned a value of 1; is the value assigned to the tth first stop mark, Assign the value of the second stop flag of the tth.

[0089] Loss MEL is calculated as follows:

[0090]

[0091] in, Represents all elements in the tth column of the Mel spectrum feature matrix, represents the spectrum characteristics of group t.

[0092] In this embodiment, the dimension of the phoneme global feature matrix S is N×256, C t The dimension of is 1×256, and the dimension of a set of spectral features is 256*1.

[0093] Example 5

[0094] A training method for a speech synthesis model based on GA-IMV is basically the same as that in Example 4, except that, in order to improve the training efficiency of the speech synthesis model based on GA-IMV, samples are input to the encoder module in batches, and 16 samples are selected from the training set for each batch. max The maximum length of the phoneme sequence in the current batch of 16 samples. max The phoneme sequence is padded with 0 at the end to make the sequence length N max , that is, let N=N in Example 4 max .

[0095] Example 6

[0096] On the basis of Example 5, a phoneme duration prediction method based on GA-IMV includes the following steps:

[0097] S1, sequentially input the 15 phoneme sequences to be predicted in the test set into the GA-IMV-based speech synthesis model trained in Example 5 to obtain the attention weight matrix A and the spectral feature matrix E;

[0098] S2, calculate the dynamic threshold of each row vector in the attention weight matrix A, where β n is the nth row vector in the attention weight matrix A, β n The dynamic threshold is G n , G n The calculation method is as follows:

[0099] G n =μ n +σ n ;

[0100] Among them, μ n is the row vector β nThe mean of all elements in , σ n is the row vector β n The standard deviation of all elements in ;

[0101] S3, for row vector β n Medium and larger than G n And count the consecutive elements to get the row vector β n The total number of consecutive elements of the row vectors is formed into a sequence of consecutive element totals f according to the increasing order of n value. n ; Calculate D total , and then calculate the phoneme duration d=D corresponding to each element total / T,D total The calculation formula is as follows:

[0102]

[0103] Among them, the sampling frequency F=16000 Hz, the frame length L=512, and the frame shift M=128.

[0104] S4, the total number of consecutive elements sequence f n Multiplying by the phoneme duration d, the phoneme duration sequence D of the phoneme sequence to be predicted is obtained = {D1, D2, D3, ..., D N}, each element D in the phoneme duration sequence D n The predicted value of the phoneme duration of a phoneme in the phoneme sequence to be predicted.

[0105] In this embodiment, the phoneme sequence is "nih ao", the phoneme sequence length N = 4, the dimension of the phoneme global feature matrix S is 4×256, T = 122, the dimension of the attention weight matrix A is 4×122, the dimension of the mapping vector is 1*256, the dimension of the spectrum feature is 256×1, f n ={25, 56, 10, 31}, d=0.008 seconds, D={0.2, 0.45, 0.1, 0.25} (unit: seconds).

[0106] The prediction accuracy of the phoneme duration prediction method of the present invention is evaluated, and the evaluation index is the average phoneme duration deviation value AvgPDM:

[0107]

[0108] Where N is the length of the phoneme sequence, and the phoneme sequence is composed of the manually labeled phoneme duration sequence P = {P1, P2, P3..., P N}, the predicted start time of the nth phoneme is: (When n=1, ), the actual start time of the nth phoneme is: (When n=1, ), the predicted end time of the nth phoneme is: The actual end time of the nth phoneme is:

[0109] The experimental results of the phoneme duration prediction method of the present invention are shown in Table 1. Table 1 shows the average value of the average phoneme duration deviation values ​​of 15 samples in the test set, which is only 0.0121 seconds. Therefore, the phoneme duration prediction method of the present invention captures the mapping relationship between the phoneme feature sequence and the acoustic feature sequence, thereby more accurately predicting the start and end time of the phoneme.

[0110] Table 1

[0111]

[0112] The mean opinion score (MOS) and Mel-Cepstral Distortion (MCD) are used to evaluate the improvement in the quality of the speech synthesized by the phoneme duration prediction method of the present invention (the spectral feature matrix E output by the phoneme duration prediction method is processed by a vocoder to obtain the synthesized speech):

[0113] Listeners rate the audio using a 5-point mean opinion score (MOS). "4.0-5.0" represents "excellent" (clear speech, natural rhythm, and strong emotion), "3.5-4.0" represents "good" (acceptable; speech is clearly audible, but rhythm is unstable and emotion is lacking), "3.0-3.5" represents "fair" (slightly poor; speech is barely audible, lacks rhythm and emotion), "1.5-3.0" represents "poor" (barely audible), and "0-1.5" represents "extremely poor" (unintelligible; mostly noise). The MOS result for audio is the average of the listener ratings.

[0114] Mel-cepstral distortion (MCD) is the difference between the spectral feature matrix E and the mel-spectral feature matrix.

[0115] Table 2

[0116]

[0117] The experimental results are shown in Table 2. The MOS value of the speech obtained by the phoneme duration prediction method of the present invention increased by 0.11. Within the numerical range of the highest score of 5, the improvement of 0.11 fully demonstrates that the phoneme duration prediction method of the present invention can effectively improve the quality of speech; the MCD value decreased by 0.21 (slight changes in the MCD value correspond to significant improvements in the actual listening experience. Under normal circumstances, every 0.1 to 0.2 reduction in the MCD will bring about a more obvious improvement in the naturalness of the listening experience). The two indicators MOS and MCD fully demonstrate that the phoneme duration prediction method of the present invention effectively improves the quality of the audio synthesized by the autoregressive speech synthesis model (Tacotron2).

[0118] Example 7

[0119] GA-IMV: The phoneme sequences of all samples in the dataset D of Example 3 and the phoneme duration sequences of the phoneme sequences in all samples predicted by the phoneme duration prediction method of the present invention are used as the input of the FastSpeech2 model (Yi R, Chenxu H, Xu T, Tao Q, Sheng Z, Zhou Z, Tie-Yan L, et al. FastSpeech2: Fast and High-Quality End-to-End Text to Speech[C], International Conference on LearningRepresentations, 2020, abs / 2006.04558). The output of FastSpeech2 is the spectral feature matrix, and the Mel-spectrum feature matrix in the samples of dataset D is used as the true value of the spectral feature matrix.

[0120] FastSpeech2 (MFA): Use the phoneme duration extraction tool MFA in the FastSpeech2 model paper to extract the phoneme duration sequences of the phoneme sequences of all samples in dataset D. The phoneme sequences of all samples in dataset D in Example 3 and the phoneme duration sequences extracted by MFA are used as the input of the FastSpeech2 model. The output of FastSpeech2 is a spectral feature matrix, and the Mel-spectrum feature matrix of the samples in dataset D is used as the true value of the spectral feature matrix.

[0121] The mean opinion score (MOS) and Mel-cepstral distortion (MCD) (RF KUBICHEK. Mel-cepstral Distance Measure for Objective Speech Quality Assessment [C], Pacific Rim Conference on Communications, Computers and Signal Processing, 1993, 1:125-128 vol. 1.) were used to evaluate the improvement in audio quality generated by the non-autoregressive FastSpeech2 model using the phoneme duration prediction method of the present invention. The experimental results are shown in Table 3. The MOS value of the audio generated by the FastSpeech2 model using GA-IMV is 3.63, and the MCD value is 7.65, which is superior to the audio quality synthesized by the FastSpeech2 (MFA) model. The phoneme duration prediction method of the present invention effectively improves the quality of the audio synthesized by the FastSpeech2 model.

[0122] Table 3

[0123]

[0124] The above is an exemplary description of the present invention. It should be noted that, without departing from the core of the present invention, any simple deformation, modification or other equivalent replacement that can be made by other skilled in the art without expending creative labor falls within the scope of protection of the present invention.

Claims

1. A speech synthesis model based on GA-IMV, characterized in that: include: Encoder module, GA-IMV module and decoder module, where the input of the encoder module is the phoneme sequence and the output is the phoneme global feature matrix S; The input of the GA-IMV module is the phoneme global feature matrix S and T hidden states, and the output of the GA-IMV module is T groups of attention weight column vectors and T mapping vectors; The decoder module output includes: T first stop signs and T groups of spectral features. The T hidden states are output one by one by the decoder module. After the decoder module outputs the t-1th hidden state to the GA-IMV module, the GA-IMV module outputs the tth group of attention weight column vectors and the tth mapping vector to the decoder module. The decoder module then outputs the tth hidden state, the tth first stop sign and the tth group of spectral features, t=1, ..., T, so that t increases in the range of 1 to T until the decoder module outputs T first stop signs and T groups of spectral features and the GA-IMV module outputs T groups of attention weight column vectors; the T first stop signs are composed of the first stop sign sequence B in the order of increasing t, the T groups of spectral features are composed of the spectral feature matrix E in the order of increasing t, and the T groups of attention weight column vectors are composed of the attention weight matrix A in the order of increasing t; The tth mapping vector C t For: C t =α t ′S, α t ′ represents the attention weight column vector α of the tth group t The transposed vector of α t =soft max(V·tanh(W q h t-1 +W k S+W f ReLU(Conv(α′ t-1 ))+b); Where, α t is the column vector of the t-th group of attention weights, soft max() represents the normalization function, α′ t-1 Represents the attention weight column vector α of the t-1 group t-1 The transposed vector, when t=1,α′ 1-1 is a zero vector, ReLU(Conv(α′ t-1 )) represents the extraction of α′ through convolution and nonlinear activation function ReLU t-1 Information, V, W q 、W k and W f are all learnable parameter matrices, b is the bias term, h t-1 is the t-1th hidden state. When t=1, the hidden state is a zero vector.

2. The speech synthesis model according to claim 1, characterized in that The encoder module includes: an embedding module, a three-layer convolution module and a bidirectional long short-term memory network module connected in sequence. The embedding module inputs a phoneme sequence and outputs a phoneme embedding matrix. The three-layer convolution module inputs a phoneme embedding matrix and outputs a phoneme local feature matrix. The bidirectional long short-term memory network module inputs a phoneme local feature matrix and outputs a phoneme global feature matrix S.

3. The speech synthesis model according to claim 1, characterized in that The phoneme global feature matrix corresponding to a phoneme sequence H is the dimension of each phoneme after encoding, and N is the length of the phoneme sequence.

4. The speech synthesis model according to claim 1, characterized in that The decoder module includes: a 2-layer long short-term memory network module, a linear module, and a 5-layer convolution module. The 2-layer long short-term memory network module is used to input the mapping vector and the first stop mark, and the 2-layer long short-term memory network module outputs the hidden state; the linear module inputs the hidden state and outputs the first stop mark; the 5-layer convolution module inputs the hidden state and outputs the spectral features; wherein: When the 2-layer long short-term memory network module inputs the t-th mapping vector, the 2-layer long short-term memory network module outputs the t-th hidden state. The linear module inputs the t-th hidden state and outputs the t-th first stop mark and controls whether the 2-layer long short-term memory network module outputs the t+1-th hidden state when it inputs the t+1-th mapping vector. The 5-layer convolution module inputs the t-th hidden state and outputs the t-th spectral feature. The GA-IMV module inputs the t-th hidden state and is used to calculate the t+1-th group of attention weight column vectors and the t+1-th mapping vector.

5. The speech synthesis model according to claim 1, characterized in that The column vector α of the attention weight of the tth group t for A phoneme sequence corresponds to N is the length of the phoneme sequence.

6. A training method for a speech synthesis model based on GA-IMV, characterized in that: The following steps are involved: S1. Prepare a training set. The training set consists of multiple samples, each of which includes: a phoneme sequence, a mel-spectrogram feature matrix, and a second stop mark sequence. Each phoneme sequence is obtained by transliterating a text. An audio segment corresponding to the text is divided into several short time frames. A second stop mark is set for each short time frame. The second stop mark of each short time frame is used to indicate whether there is a subsequent short time frame after the short time frame. All the second stop marks of each audio segment constitute a second stop mark sequence. S2, inputting the phoneme sequence, Mel spectrum feature matrix and second stop mark sequence of each sample in the training set into the speech synthesis model of claim 1, so that the speech synthesis model is trained. During the training process, the speech synthesis model performs a loss value Loss calculation module. ALL Calculate and get the loss value Loss ALL , when the loss value Loss AL If it is less than 1, the training is completed and a trained speech synthesis model is obtained, where: Loss ALL =Loss GA +Loss IMV +Loss MEL +Loss STOP-TOKEN ; W is the guidance matrix, the dimension of W is the same as the dimension of the attention weight matrix A, and N is the number of phonemes in the phoneme sequence; Loss IMV = Loss mono + Loss cont + Loss comp , where, Loss comp =|IMV1|+|IMV T -N| Among them, n is the index number of the nth phoneme in the phoneme sequence, A n，t is the element in the nth row and tth column of the attention weight matrix A; The first stop flag and the second stop flag are binary values ​​respectively. When the first stop flag controls decoding, the first stop flag is assigned a value of 0, and when the first stop flag controls decoding to stop, the first stop flag is assigned a value of 1; when the second stop flag indicates that there is a subsequent short time frame after the short time frame, the second stop flag is assigned a value of 0, and when the second stop flag indicates that there is no subsequent short time frame after the short time frame, the second stop flag is assigned a value of 1; is the value assigned to the tth first stop mark, Assign the value of the second stop flag of the tth. Loss MEL is calculated as follows: in, Represents all elements in the tth column of the Mel spectrum feature matrix, represents the spectrum characteristics of group t.

7. The training method according to claim 6, characterized in that When samples are input into the speech synthesis model of claim 1 in batches, N is N max , N max The maximum length of the phoneme sequence in the current batch of samples, for sequences less than N max The phoneme sequence is padded with 0 at the end to make the sequence length N max .

8. A phoneme duration prediction method based on GA-IMV, characterized in that: The following steps are involved: S1, inputting the phoneme sequence to be predicted into the trained speech synthesis model obtained in claim 6 to obtain the attention weight matrix A; S2, calculate the dynamic threshold of each row vector in the attention weight matrix A, where β n is the nth row vector in the attention weight matrix A, β n The dynamic threshold is G n , G n The calculation method is as follows: G n =μ n +s n ; Among them, μ n is the row vector β n The mean of all elements in , σ n is the row vector β n The standard deviation of all elements in ; S3, for row vector β n Medium and larger than G n And count the consecutive elements to get the row vector β n The total number of consecutive elements of , and in the order of increasing n value, the total number of consecutive elements of all row vectors are combined into a sequence of the total number of consecutive elements f n ; Calculate D total , and then calculate the phoneme duration d=D corresponding to each element total / T,D total The calculation formula is as follows: Wherein, L is the frame length of the audio frame, M is the frame shift of the audio frame, and F is the sampling frequency of the audio; S4, the total number of consecutive elements sequence f n Multiplying by the phoneme duration d, the phoneme duration sequence D of the phoneme sequence to be predicted is obtained = {D1, D2, D3, ..., D N }.

Images