Phoneme duration prediction method based on GA-IMV

Abstract

The invention discloses a phoneme duration prediction method based on GA-IMV, the phoneme duration prediction method is based on a speech synthesis model based on GA-IMV, the speech synthesis model comprises an encoder module, a GA-IMV module and a decoder module, the input of the encoder module is a phoneme sequence, and the phoneme sequence is obtained by phoneme transliteration conversion of a text. The output of the speech synthesis model is an attention weight matrix and a spectrum feature matrix, and the phoneme duration prediction method calculates a phoneme duration sequence according to the attention weight matrix. According to the speech synthesis model based on GA-IMV, the average phoneme duration deviation value of the synthesized audio is reduced, and the quality of the synthesized audio is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of speech synthesis, and in particular relates to a phoneme duration prediction method based on GA-IMV. Background Art

[0002] Speech synthesis technology is a technology that uses computer programs to simulate human pronunciation and convert input text information into understandable speech output. Currently, there are two main models in speech synthesis technology: autoregressive model and non-autoregressive model.

[0003] The autoregressive speech synthesis model is a relatively traditional speech synthesis method. This type of model uses a step-by-step prediction method, that is, the output of the previous step will be used as the input of the next step. Although this serial prediction method can generate high-quality speech, the generation process is slow because the prediction needs to be carried out step by step, which makes it difficult to meet real-time requirements.

[0004] Unlike the autoregressive speech synthesis model, the core feature of the non-autoregressive speech synthesis model is that it can generate a complete speech sequence in parallel at one time, without the need for step-by-step prediction like the autoregressive model, thereby greatly improving the generation speed and being more suitable for real-time speech synthesis. However, to achieve this, the non-autoregressive model needs to clarify the duration of each phoneme (the basic unit of speech) during the generation phase, which requires the non-autoregressive speech synthesis model to have the ability to predict the duration of the phoneme. To train a phoneme duration model, it is necessary to extract the duration feature of each phoneme.

[0005] However, there are two major problems with the current phoneme duration feature extraction methods:

[0006] 1. The existing methods are not accurate enough. The existing phoneme duration prediction methods fail to fully consider the contextual dependencies between phonemes and the complex mapping relationship between phoneme sequences and acoustic sequences, resulting in inaccurate duration prediction. For example, the patent with publication number CN102222501A discloses a method for generating duration parameters in speech synthesis, which is used to generate state duration parameters in speech synthesis based on a hidden Markov model. The method generates a more natural state residence time by combining the state duration model with the duration overall variance model. The duration overall variance model is further optimized by creating a special training corpus and using a duration overall variance analyzer to generate training samples. This method effectively solves the problem of overly average state duration in traditional methods, making the synthesized speech more natural and expressive in duration distribution. Since the state sequence in the hidden Markov model is an abstract coarse-grained model of the actual pronunciation process, it is impossible to make full use of the global context information, which makes it difficult for the phoneme sequence to accurately match the real acoustic sequence, and thus causes the extracted phoneme duration features to be inaccurate.

[0007] Second, the scalability of existing methods is insufficient. Most of the current phoneme duration extraction methods are designed for non-autoregressive speech synthesis models and cannot be applied to autoregressive speech synthesis models, which limits the scope of application of these technologies. For example, the patent with publication number CN202410117821 discloses a phoneme duration prediction method based on the VITS neural spline stream improvement. By improving the random duration predictor in the VITS model, introducing a method based on the neural spline stream, and carefully analyzing the input data distribution and dividing the block interval, the model can perform more effective coupling transformations in each block interval, thereby extracting the duration of the phoneme and improving the quality of VITS synthesized speech. However, this solution is only based on the random duration predictor module of the streaming speech synthesis model VITS (Conditional Variational Autoencoder with AdversarialLearning for End-to-End Text-to-Speech) to improve, and it is difficult to apply to the autoregressive speech synthesis model, which limits its scope of application. Summary of the invention

[0008] In view of the problems existing in the prior art, the object of the present invention is to provide a speech synthesis model based on GA-IMV, which is improved based on the autoregressive speech synthesis model Tacotron2 (Jonathan S, Ruoming P, Ron JW, Mike S, Navdeep J, Zongheng Y, Zhifeng C, Yu Z, Yuxuan W, RJ S, Rif AS, Yannis A, Yonghui W, et al. Natural TTS Synthesis by Conditioning Wavenet on MELSpectrogram Predictions[J], 2018 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING(ICASSP), 2018: 4779-4783.), and the position-sensitive attention mechanism in the autoregressive speech synthesis model Tacotron2 is replaced by a guided attention mechanism GA and an index mapping vector IMV constraint.

[0009] Another object of the present invention is to provide a phoneme duration prediction method based on GA-IMV, which can extract the phoneme duration more accurately.

[0010] The purpose of the present invention is achieved through the following technical solutions.

[0011] A speech synthesis model based on GA-IMV, comprising: an encoder module, a GA-IMV module and a decoder module, wherein the input of the encoder module is a phoneme sequence and the output is a phoneme global feature matrix S;

[0012] The input of the GA-IMV module is the phoneme global feature matrix S and T hidden states, and the output of the GA-IMV module is T groups of attention weight column vectors and T mapping vectors;

[0013] The decoder module output includes: T first stop signs and T groups of spectral features. The T hidden states are output one by one by the decoder module. After the decoder module outputs the t-1th hidden state to the GA-IMV module, the GA-IMV module outputs the tth group of attention weight column vectors and the tth mapping vector. The decoder module receives the tth mapping vector and then outputs the tth hidden state, the tth first stop sign and the tth group of spectral features. t=1, ..., T, so that t increases in the range of 1 to T until the decoder module outputs T first stop signs and T groups of spectral features, and the GA-IMV module outputs T groups of attention weight column vectors; the T first stop signs are formed into a first stop sign sequence B in the order of increasing t, the T groups of spectral features are formed into a spectral feature matrix E in the order of increasing t, and the T groups of attention weight column vectors are formed into an attention weight matrix A in the order of increasing t;

[0014] The tth mapping vector C t For: C t =α t ′S,α t ′ represents the attention weight column vector α of the tth group t The transposed vector of ;

[0015] α t =softmax(V·tanh(W q h t-1 +W k S+W f ReLU(Conv(α′ t-1 ))+b);

[0016] In the formula, α t is the column vector of the t-th group of attention weights, softmax() represents the normalization function, α′ t-1 Represents the attention weight column vector α of the t-1th group t-1 The transposed vector of α′ 1-1 is a zero vector, ReLU(Conv(α′ t-1 )) represents the extraction of α′ through convolution and nonlinear activation function ReLU t-1 Information, V, Wq , W k and W f are all learnable parameter matrices, b is the bias term, h t-1 is the t-1th hidden state. When t=1, the hidden state h t-1 is the zero vector.

[0017] In the above technical solution, the encoder module includes: an embedding module, a three-layer convolution module and a bidirectional long short-term memory network module connected in sequence, the embedding module inputs a phoneme sequence and outputs a phoneme embedding matrix, the three-layer convolution module inputs a phoneme embedding matrix and outputs a phoneme local feature matrix, and the bidirectional long short-term memory network module inputs a phoneme local feature matrix and outputs a phoneme global feature matrix S.

[0018] In the above technical solution, the phoneme global feature matrix corresponding to a phoneme sequence H is the dimension of each phoneme after encoding, and N is the sequence length of the phoneme sequence.

[0019] In the above technical solution, the decoder module includes: a 2-layer long short-term memory network module, a linear module and a 5-layer convolution module, the 2-layer long short-term memory network module is used to input the mapping vector and the first stop mark and output the hidden state; the linear module inputs the hidden state and outputs the first stop mark; the 5-layer convolution module inputs the hidden state and outputs the spectrum feature; wherein:

[0020] When the 2-layer long short-term memory network module inputs the tth mapping vector, the 2-layer long short-term memory network module outputs the tth hidden state, the linear module inputs the tth hidden state and outputs the tth first stop mark and controls whether the 2-layer long short-term memory network module outputs the t+1th hidden state when the t+1th mapping vector is input; the 5-layer convolution module inputs the tth hidden state and outputs the tth spectrum feature; the GA-IMV module inputs the tth hidden state and is used to calculate the t+1th group of attention weight column vectors and the t+1th mapping vector.

[0021] In the above technical solution, the t-th group attention weight column vector α t for Attention weight matrix corresponding to a phoneme sequence N is the sequence length of the phoneme sequence.

[0022] A training method for a speech synthesis model based on GA-IMV comprises the following steps:

[0023] S1, prepare a training set, the training set consists of multiple samples, each sample includes: a phoneme sequence, a Mel spectrum feature matrix and a second stop mark sequence (the Mel spectrum feature matrix in the sample is the true value of the spectrum feature matrix E, and the second stop mark sequence in the sample is the true value of the first stop mark sequence B), each phoneme sequence is obtained by phoneme transcription conversion of a text, a section of audio corresponding to the text is divided into a number of short time frames, a second stop mark is set for each short time frame, the second stop mark of each short time frame is used to indicate whether there is a subsequent short time frame after the short time frame, and all the second stop marks of each audio segment constitute a second stop mark sequence;

[0024] S2, input the phoneme sequence, Mel spectrum feature matrix and second stop sign sequence of each sample in the training set into the speech synthesis model, so that the speech synthesis model based on GA-IMV is trained. During the training process, the speech synthesis model is calculated through the loss calculation module. ALL Calculate and get the loss value Loss ALL , use the back propagation algorithm to optimize the parameters, when the loss value Loss ALL If it is less than 1, the training is completed, where:

[0025] Loss ALL =Loss GA +Loss IMV +Loss MEL +Loss STOP-TOKEN

[0026] W is the guidance matrix, the dimension of W is the same as the dimension of the attention weight matrix A, and N is the number of phonemes in the phoneme sequence;

[0027] Loss IMV =Loss mono +Loss cont +Loss comp ,in,

[0028]

[0029] Loss comp =|IMV1|+|IMV T -N|

[0030]

[0031] Where n is the index number of the nth phoneme in the phoneme sequence, A n，t is the element in the nth row and tth column of the attention weight matrix A;

[0032]

[0033] The first stop mark and the second stop mark are binary values ​​respectively. When the first stop mark controls decoding, the first stop mark is assigned a value of 0, and when the first stop mark controls decoding to stop, the first stop mark is assigned a value of 1; when the second stop mark indicates that there is a subsequent short-time frame after the short-time frame, the second stop mark is assigned a value of 0, and when the second stop mark indicates that there is no subsequent short-time frame after the short-time frame, the second stop mark is assigned a value of 1; is the value assigned to the tth first stop mark, Assign the value of the tth second stop flag.

[0034] Loss MEL The calculation method is as follows:

[0035]

[0036] in, represents all elements in the tth column of the Mel spectrum feature matrix, Represents the spectrum characteristics of the tth group.

[0037] A phoneme duration prediction method based on GA-IMV comprises the following steps:

[0038] S1, input the phoneme sequence to be predicted into the trained speech synthesis model to obtain the attention weight matrix A and the spectrum feature matrix E;

[0039] S2, calculate each group of row vectors β in the attention weight matrix A n The dynamic threshold of n is the nth row vector in the attention weight matrix A, β n The dynamic threshold is G n , G n The calculation method is as follows:

[0040] G n =μ n +σ n ;

[0041] Among them, μ n is the row vector β n The mean of all elements in , σ n is the row vector β n The standard deviation of all elements in ;

[0042] S3, for row vector β n Medium greater than G n And count the consecutive elements to get the row vector β nThe total number of consecutive elements of the row vectors is then converted into a sequence of consecutive element totals f in the order of increasing n values. n ; Calculate D total , and then calculate the phoneme duration d=D corresponding to each element total / T,D total The calculation formula is as follows:

[0043]

[0044] Wherein, L is the frame length of dividing a segment of audio into several short time frames, M is the frame shift of dividing a segment of audio into several short time frames, and F is the sampling frequency of the audio;

[0045] S4, the total number of consecutive elements sequence f n Multiply it by the phoneme duration d to obtain the phoneme duration sequence D = {D1, D2, D3, ..., D N}, each element D in the phoneme duration sequence D n It is the predicted value of the phoneme duration of a phoneme in the phoneme sequence to be predicted.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. The phoneme duration prediction method of the present invention adopts a combination of GA and IMV to enable the speech synthesis model to learn a more accurate attention weight matrix, and then extract more precise phoneme duration features.

[0048] 2. The phoneme duration prediction method of the present invention enhances the learning ability of the speech synthesis model on the mapping relationship between the phoneme sequence and the spectrum feature matrix, and further improves the quality of the synthesized speech of the autoregressive speech synthesis model.

[0049] 3. Using the phoneme duration features extracted by the phoneme duration prediction method of the present invention to train a non-autoregressive speech synthesis model can provide more accurate time alignment information for the speech synthesis model, further enhancing the quality of the synthesized speech of the non-autoregressive speech synthesis model.

[0050] 4. The phoneme duration features extracted by the phoneme duration prediction method of the present invention are applicable to autoregressive speech synthesis models and non-autoregressive speech synthesis models, and have high generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a structural diagram of the speech synthesis model;

[0052] Figure 2 This is a training flowchart of the GA-IMV-based speech synthesis model. DETAILED DESCRIPTION

[0053] The phoneme duration prediction method of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0054] Example 1

[0055] like Figure 1 As shown, a speech synthesis model based on GA-IMV includes: an encoder module, a GA-IMV module and a decoder module, wherein the input of the encoder module is a phoneme sequence and the output is a phoneme global feature matrix S;

[0056] The input of the GA-IMV module is the phoneme global feature matrix S and T hidden states. The output of the GA-IMV module is T groups of attention weight column vectors and T mapping vectors, where the dimension of each group of attention weight column vectors is N×1.

[0057] The output of the decoder module includes: T first stop signs and T groups of spectral features. The T hidden states are output one by one by the decoder module. When the decoder module outputs the t-1th hidden state (h t-1 ), the GA-IMV module outputs the t-th group of attention weight column vectors (α t ) and outputs the tth mapping vector (C t ), the decoder module then outputs the tth hidden state (h t ), the tth first stop mark (p t ) and the tth group of spectral features (e t , column vector), t = 1, ..., T, so that t increases in the range of 1 to T, until the decoder module outputs T first stop signs and T groups of spectral features, and the GA-IMV module outputs T groups of attention weight column vectors; the T first stop signs are formed into a first stop sign sequence B in the order of increasing t, and the T groups of spectral features are formed into a spectral feature matrix E in the order of increasing t (the tth column in the spectral feature matrix E is the spectral feature e t ), the T groups of attention weight column vectors are formed into an attention weight matrix A in the order of increasing t (the tth column in the attention weight matrix A is the attention weight column vector α t ). The column vector α of the tth group of attention weights t for A phoneme sequence corresponds to N is the sequence length (number of phonemes) of the phoneme sequence.

[0058] The tth mapping vector C t For: C t =α t ′S,α t ′ represents the attention weight column vector α of the tth group tThe transposed vector of ;

[0059] α t =softmax(V·tanh(W q h t-1 +W k S+W f ReLU(Conv(α′ t-1 ))+b)

[0060] In the formula, α t is the column vector of the t-th group of attention weights, softmax() represents the normalization function, α′ t-1 Represents the attention weight column vector a of the t-1th group t-1 The transposed vector of α′ 1-1 is a zero vector, ReLU(Conv(α′ t-1 )) represents the extraction of α′ through convolution and nonlinear activation function ReLU t-1 Information, V, W q , W k and W f are all learnable parameter matrices, b is the bias term, h t-1 is the t-1th hidden state. When t=1, the hidden state h 1-1 is the zero vector.

[0061] Example 2

[0062] A speech synthesis model based on GA-IMV, based on Example 1, the encoder module includes: an embedding module, a three-layer convolution module and a bidirectional long short-term memory network module connected in sequence, the embedding module is used to input a phoneme sequence, the output of the embedding module is a phoneme embedding matrix, the embedding module is used to convert the phoneme sequence into a digital sequence and expand the digital sequence into a higher-dimensional phoneme embedding matrix, the input of the three-layer convolution module is the phoneme embedding matrix, and the output is a phoneme local feature matrix, the three-layer convolution module is used to capture the local features of the phoneme embedding matrix (the local features are mainly word boundaries and syllable structures), which helps the model understand the local dependencies in the input sequence, the input of the bidirectional long short-term memory network module is the phoneme local feature matrix, and the output is the phoneme global feature matrix S, and the bidirectional long short-term memory network module is used to extract global context information (forward and reverse information) in the phoneme local feature matrix.

[0063] The phoneme global feature matrix corresponding to a phoneme sequence H is the dimension of each phoneme after encoding, and H is set to 256 in this embodiment.

[0064] The decoder module includes: a 2-layer long short-term memory network module, a linear module and a 5-layer convolution module. The 2-layer long short-term memory network module is used to input the mapping vector and the first stop mark, and the 2-layer long short-term memory network module outputs the hidden state; the linear module inputs the hidden state and outputs the first stop mark; the 5-layer convolution module inputs the hidden state and outputs the spectrum feature;

[0065] When the 2-layer LSTM network module inputs the tth mapping vector (C t ) (At this time, the 2-layer long short-term memory network module also inputs the t-1th first stop mark (p t-1 ) and controls the 2-layer long short-term memory network module to decode), the 2-layer long short-term memory network module outputs the tth hidden state (h t ), the linear module inputs the tth hidden state (h t ) after the output of the tth first stop mark (p t ) and control the 2-layer long short-term memory network module to input the t+1th mapping vector (C t+1 ) whether to output the t+1th hidden state (h t+1 ), that is, control the 2-layer long short-term memory network module to input the t+1th mapping vector C t Then decode or stop decoding; the 5-layer convolution module inputs the tth hidden state (h t ), output the tth spectrum feature (e t ); the GA-IMV module inputs the tth hidden state (h t ) and used for the t+1th group of attention weight column vectors (α t+1 ) and the t+1th mapping vector (C t+1 ) calculation.

[0066] The first stop sign is used to prevent the decoder module from generating too many or too few spectral features. The 2-layer long short-term memory network is used to capture contextual relationships so that the generated speech features (hidden states) are coherent and consistent in time. The 5-layer convolution module is used to extract the local time-frequency information (detailed features on the time axis) of the hidden state layer by layer and output more detailed spectral features.

[0067] Example 3

[0068] On the basis of Example 2, a data set D consisting of X samples is prepared, each sample including: a phoneme sequence, a Mel spectrum feature matrix and a second stop mark sequence, each phoneme sequence is composed of English letters (phonemes), and each phoneme sequence is obtained by phoneme transliteration of a text (phoneme transliteration method: converting Chinese text into a phoneme sequence composed of English letters according to pinyin rules);

[0069] The method for obtaining each Mel spectrum feature matrix is ​​as follows: a segment of audio (each segment of audio is a continuous audio signal, and the sampling frequency of the audio signal is F) is divided into several short time frames according to the frame length L and the frame shift M, and each short time frame is subjected to Fourier transform to obtain the frequency domain features of each short time frame, and then the logarithm of all the frequency domain features of the segment of audio is taken to obtain a Mel spectrum feature matrix;

[0070] The method for obtaining each second stop mark sequence is as follows: a second stop mark is set for each short time frame, and all the second stop marks of each audio segment constitute a second stop mark sequence, and the second stop mark of each short time frame is used to indicate whether there is a subsequent short time frame after the short time frame.

[0071] Each sample has a corresponding text and audio.

[0072] In this embodiment, X=9179, F=16000 Hz, frame length L=512, frame shift M=128, and H=256.

[0073] Example 4

[0074] like Figure 2 As shown, a training method for a speech synthesis model based on GA-IMV includes the following steps:

[0075] S1, prepare a training set: divide all samples in the data set D of Example 3 into a training set and a test set in a ratio of 8:2.

[0076] S2, inputting the phoneme sequence, Mel spectrum feature matrix and second stop mark sequence of each sample in the training set into the GA-IMV-based speech synthesis model of Example 2, so that the GA-IMV-based speech synthesis model is trained. During the training process, the speech synthesis model is subjected to a loss value Loss calculation module. ALL Calculate and get the loss value Loss ALL , using the back propagation algorithm (Ian Goodfellow, Joshua Bengio, Aaron Courville. (2017). Deep Learning [M]. Zhao Shenjian, Li Yujun, Fu Tianfan, Li Kai, Translated. Beijing: People's Posts and Telecommunications Press.) to optimize the parameters, when the loss value Loss ALL If it is less than 1, the training is completed.

[0077] Loss ALL =Loss GA +Loss IMV +Loss MEL +Loss STOP-TPLEN ;in,

[0078]

[0079] W is the guide matrix (Hideyuki Tachibana, Katsuya Uenoyama, Shunsuke Aihara. Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention[J], Computing Research Repository, 2018: 4784-4788.), the dimension of W is the same as that of A, and N is the sequence length (number of phonemes) of the phoneme sequence.

[0080] IMV loss function Loss IMV The calculation method is as follows:

[0081] Loss IMV =Loss mono +Loss cont +Loss comp ,in,

[0082]

[0083] Loss comp =|IMV1|+|IMV T -N|

[0084]

[0085] Where n is the index number of the nth phoneme in the phoneme sequence, A n，t is the element in the nth row and tth column of the attention weight matrix A.

[0086] Loss STOP-TOKEN The calculation method is as follows:

[0087]

[0088] The first stop mark is the predicted value of the linear module, and the second stop mark is the real value of the short-time frame. The first stop mark and the second stop mark are binary values ​​respectively. When the first stop mark controls decoding, the first stop mark is assigned a value of 0, and when the first stop mark controls decoding to stop, the first stop mark is assigned a value of 1; when the second stop mark indicates that there is a subsequent short-time frame after the short-time frame, the second stop mark is assigned a value of 0, and when the second stop mark indicates that there is no subsequent short-time frame after the short-time frame, the second stop mark is assigned a value of 1; is the value assigned to the tth first stop mark, Assign the value of the tth second stop flag.

[0089] Loss MEL The calculation method is as follows:

[0090]

[0091] in, represents all elements in the tth column of the Mel spectrum feature matrix, Represents the spectrum characteristics of the tth group.

[0092] In this embodiment, the dimension of the phoneme global feature matrix S is N×256, C t The dimension of is 1×256, and the dimension of a set of spectral features is 256*1.

[0093] Example 5

[0094] A training method for a speech synthesis model based on GA-IMV is basically the same as Example 4, except that: in order to improve the training efficiency of the speech synthesis model based on GA-IMV, samples are input into the encoder module in batches, and 16 samples in the training set are selected for each batch. max is the maximum length of the phoneme sequence of the current batch of 16 samples, for sequences less than N max The phoneme sequence is padded with 0 at the end to make the sequence length N max , that is, let N=N in Example 4 max .

[0095] Example 6

[0096] On the basis of Example 5, a phoneme duration prediction method based on GA-IMV includes the following steps:

[0097] S1, inputting 15 phoneme sequences to be predicted in the test set into the GA-IMV-based speech synthesis model trained in Example 5 in sequence, and obtaining an attention weight matrix A and a spectrum feature matrix E;

[0098] S2, calculate the dynamic threshold of each group of row vectors in the attention weight matrix A, where β n is the nth row vector in the attention weight matrix A, β n The dynamic threshold is G n , G n The calculation method is as follows:

[0099] G n =μ n +σ n ;

[0100] Among them, μ n is the row vector β nThe mean of all elements in , σ n is the row vector β n The standard deviation of all elements in ;

[0101] S3, for row vector β n Medium greater than G n And count the consecutive elements to get the row vector β n The total number of consecutive elements of the row vectors is then converted into a sequence of consecutive element totals f in the order of increasing n values. n ; Calculate D total , and then calculate the phoneme duration d=D corresponding to each element total / T,D total The calculation formula is as follows:

[0102]

[0103] Among them, the sampling frequency F=16000 Hz, the frame length L=512, and the frame shift M=128.

[0104] S4, the total number of consecutive elements sequence f n Multiply it by the phoneme duration d to obtain the phoneme duration sequence D = {D1, D2, D3, ..., D N}, each element D in the phoneme duration sequence D n It is the predicted value of the phoneme duration of a phoneme in the phoneme sequence to be predicted.

[0105] In this embodiment, the phoneme sequence is "nih ao", the phoneme sequence length N = 4, the dimension of the phoneme global feature matrix S is 4×256, T = 122, the dimension of the attention weight matrix A is 4×122, the dimension of the mapping vector is 1*256, the dimension of the spectrum feature is 256×1, f n ={25, 56, 10, 31}, d=0.008 seconds, D={0.2, 0.45, 0.1, 0.25} (unit: seconds).

[0106] The prediction accuracy of the phoneme duration prediction method of the present invention is evaluated, and the evaluation index is the average phoneme duration deviation value AvgPDM:

[0107]

[0108] Where N is the length of the phoneme sequence, and the phoneme sequence is composed of the manually annotated phoneme duration sequence P = {P1, P2, P3..., P N}, the predicted start time of the nth phoneme is: (When n=1, ), the actual start time of the nth phoneme is: (When n=1, ), the predicted end time of the nth phoneme is: The actual end time of the nth phoneme is:

[0109] The experimental results of the phoneme duration prediction method of the present invention are shown in Table 1. Table 1 is the average value of the average phoneme duration deviation values ​​of 15 samples in the test set, which is only 0.0121 seconds. Therefore, the phoneme duration prediction method of the present invention captures the mapping relationship between the phoneme feature sequence and the acoustic feature sequence, thereby more accurately predicting the start and end time of the phoneme.

[0110] Table 1

[0111]

[0112] The mean opinion score (MOS) and Mel-cepstral distortion (MCD) are used to evaluate the improvement of the quality of the speech synthesized by the phoneme duration prediction method of the present invention (the spectrum feature matrix E output by the phoneme duration prediction method is processed by a vocoder to obtain the synthesized speech):

[0113] The mean opinion score (MOS) is a 5-point scale in which listeners rate audio. "4.0-5.0" represents "excellent" (clear speech, natural rhythm, and full emotions), "3.5-4.0" represents "good" (okay, the voice can be heard clearly, the rhythm is not stable, and the emotions are not good), "3.0-3.5" represents "fair" (slightly poor, the speech can be barely heard, no rhythm, no emotions), "1.5-3.0" represents "poor" (basically unclear), and "0-1.5" represents "very poor" (unable to perceive human pronunciation, mostly noise). The MOS result of audio is the average of the listeners' scores.

[0114] The Mel-cepstral distortion (MCD) is the difference between the spectral feature matrix E and the Mel-spectral feature matrix.

[0115] Table 2

[0116]

[0117] The experimental results are shown in Table 2. The MOS value of the speech obtained by the phoneme duration prediction method of the present invention is increased by 0.11. Within the numerical range of the highest score of 5, the improvement of 0.11 fully demonstrates that the phoneme duration prediction method of the present invention can effectively improve the quality of speech; the MCD value is reduced by 0.21 (a slight change in the MCD value corresponds to a significant improvement in the actual listening experience. Under normal circumstances, every 0.1 to 0.2 reduction in the MCD will bring about a more obvious improvement in the naturalness of the listening experience). The two indicators MOS and MCD fully demonstrate that the phoneme duration prediction method of the present invention effectively improves the quality of the synthesized audio of the autoregressive speech synthesis model (Tacotron2).

[0118] Example 7

[0119] GA-IMV: The phoneme sequences of all samples in the data set D of Example 3 and the phoneme duration sequences of the phoneme sequences in all samples predicted by the phoneme duration prediction method of the present invention are used as the input of the FastSpeech2 model (Yi R, Chenxu H, Xu T, Tao Q, Sheng Z, Zhou Z, Tie-Yan L, et al. FastSpeech2: Fast and High-Quality End-to-End Text to Speech[C], International Conference on LearningRepresentations, 2020, abs / 2006.04558). The output of FastSpeech2 is the spectral feature matrix, and the Mel-spectrum feature matrix in the samples of the data set D is used as the true value of the spectral feature matrix.

[0120] FastSpeech2 (MFA): The phoneme duration extraction tool MFA in the paper of the FastSpeech2 model is used to extract the phoneme duration sequences of the phoneme sequences of all samples in the dataset D. The phoneme sequences of all samples in the dataset D of Example 3 and the phoneme duration sequences extracted by MFA are used as inputs of the FastSpeech2 model. The output of FastSpeech2 is a spectral feature matrix, and the Mel-spectrogram feature matrix in the samples of the dataset D is used as the true value of the spectral feature matrix.

[0121] The mean opinion score (MOS) and Mel-cepstral distortion (MCD) (RF KUBICHEK. Mel-cepstral Distance Measure for Objective Speech Quality Assessment [C], Pacific Rim Conference on Communications, Computers and Signal Processing, 1993, 1: 125-128 vol. 1.) are used to evaluate the improvement of the audio quality generated by the non-autoregressive model FastSpeech2 by the phoneme duration prediction method of the present invention. The experimental results are shown in Table 3. The MOS value of the audio generated by the FastSpeech2 model using GA-IMV is 3.63, and the MCD value is 7.65, which is better than the audio quality synthesized by the FastSpeech2 (MFA) model. The phoneme duration prediction method of the present invention effectively improves the quality of the audio synthesized by the FastSpeech2 model.

[0122] Table 3

[0123]

[0124] The present invention is described above by way of example. It should be noted that, without departing from the core of the present invention, any simple deformation, modification or other equivalent replacement that can be made by those skilled in the art without inventive effort falls within the protection scope of the present invention.

Claims

1. A speech synthesis model based on GA-IMV, characterized in that: include: An encoder module, a GA-IMV module and a decoder module, wherein the input of the encoder module is a phoneme sequence and the output is a phoneme global feature matrix S; The input of the GA-IMV module is the phoneme global feature matrix S and T hidden states, and the output of the GA-IMV module is T groups of attention weight column vectors and T mapping vectors; The decoder module output includes: T first stop signs and T groups of spectral features. The T hidden states are output one by one by the decoder module. After the decoder module outputs the t-1th hidden state to the GA-IMV module, the GA-IMV module outputs the tth group of attention weight column vectors and outputs the tth mapping vector to the decoder module. The decoder module then outputs the tth hidden state, the tth first stop sign and the tth group of spectral features. t=1, ..., T, so that t increases in the range of 1 to T until the decoder module outputs T first stop signs and T groups of spectral features, and the GA-IMV module outputs T groups of attention weight column vectors; the T first stop signs are formed into a first stop sign sequence B in the order of increasing t, the T groups of spectral features are formed into a spectral feature matrix E in the order of increasing t, and the T groups of attention weight column vectors are formed into an attention weight matrix A in the order of increasing t; The tth mapping vector C t For: C t =α t ′S,α t ′ represents the attention weight column vector α of the tth group t The transposed vector of ; α t =soft max(V·tanh(W q h t-1 +W k S+W f ReLU(Conv(α′ t-1 ))+b); In the formula, α t is the column vector of the t-th group of attention weights, soft max() represents the normalization function, α′ t-1 Represents the attention weight column vector α of the t-1th group t-1 The transposed vector of α′ 1-1 is a zero vector, ReLU(Conv(α′ t-1 )) represents the extraction of α′ through convolution and nonlinear activation function ReLU t-1 Information, V, W q , W k and W f are all learnable parameter matrices, b is the bias term, h t-1 is the t-1th hidden state. When t=1, the hidden state is a zero vector.

2. The speech synthesis model according to claim 1, characterized in that: The encoder module includes: an embedding module, a three-layer convolution module and a bidirectional long short-term memory network module connected in sequence. The embedding module inputs a phoneme sequence and outputs a phoneme embedding matrix. The three-layer convolution module inputs a phoneme embedding matrix and outputs a phoneme local feature matrix. The bidirectional long short-term memory network module inputs a phoneme local feature matrix and outputs a phoneme global feature matrix S.

3. The speech synthesis model according to claim 1, characterized in that: The phoneme global feature matrix corresponding to a phoneme sequence H is the dimension of each phoneme after encoding, and N is the sequence length of the phoneme sequence.

4. The speech synthesis model according to claim 1, characterized in that: The decoder module includes: a 2-layer long short-term memory network module, a linear module and a 5-layer convolution module. The 2-layer long short-term memory network module is used to input the mapping vector and the first stop mark, and the 2-layer long short-term memory network module outputs the hidden state; the linear module inputs the hidden state and outputs the first stop mark; the 5-layer convolution module inputs the hidden state and outputs the spectrum feature; wherein: When the 2-layer long short-term memory network module inputs the tth mapping vector, the 2-layer long short-term memory network module outputs the tth hidden state, the linear module inputs the tth hidden state and outputs the tth first stop mark and controls whether the 2-layer long short-term memory network module outputs the t+1th hidden state when inputting the t+1th mapping vector; the 5-layer convolution module inputs the tth hidden state and outputs the tth spectral feature; the GA-IMV module inputs the tth hidden state and is used to calculate the t+1th group of attention weight column vectors and the t+1th mapping vector.

5. The speech synthesis model according to claim 1, characterized in that: The column vector α of the attention weight of the tth group t for A phoneme sequence corresponds to N is the sequence length of the phoneme sequence.

6. A training method for a speech synthesis model based on GA-IMV, characterized in that: The following steps are involved: S1, prepare a training set, the training set consists of multiple samples, each sample includes: a phoneme sequence, a Mel spectrum feature matrix and a second stop mark sequence, each phoneme sequence is obtained by phoneme transcription conversion of a text, a segment of audio corresponding to the text is divided into a number of short time frames, a second stop mark is set for each short time frame, the second stop mark of each short time frame is used to indicate whether there is a subsequent short time frame after the short time frame, and all the second stop marks of each audio segment constitute a second stop mark sequence; S2, inputting the phoneme sequence, the Mel spectrum feature matrix and the second stop mark sequence of each sample in the training set into the speech synthesis model in claim 1, so that the speech synthesis model is trained. During the training process, the speech synthesis model calculates the loss value Loss through the Loss calculation module. ALL Calculate and get the loss value Loss ALL , when the loss value Loss AL If it is less than 1, the training is completed and a trained speech synthesis model is obtained, where: Loss ALL =Loss GA +Loss IMV +Loss MEL +Loss STOP-TOKEN ; W is the guidance matrix, the dimension of W is the same as the dimension of the attention weight matrix A, and N is the number of phonemes in the phoneme sequence; Loss IMV = Loss mono + Loss cont + Loss comp , where, Loss comp =|IMV1|+|IMV T -N| Where n is the index number of the nth phoneme in the phoneme sequence, A n，t is the element in the nth row and tth column of the attention weight matrix A; The first stop mark and the second stop mark are binary values ​​respectively. When the first stop mark controls decoding, the first stop mark is assigned a value of 0, and when the first stop mark controls decoding to stop, the first stop mark is assigned a value of 1; when the second stop mark indicates that there is a subsequent short-time frame after the short-time frame, the second stop mark is assigned a value of 0, and when the second stop mark indicates that there is no subsequent short-time frame after the short-time frame, the second stop mark is assigned a value of 1; is the value assigned to the tth first stop mark, Assign the value of the tth second stop flag. Loss MEL The calculation method is as follows: in, represents all elements in the tth column of the Mel spectrum feature matrix, Represents the spectrum characteristics of the tth group.

7. The training method according to claim 6, characterized in that: When samples are input into the speech synthesis model of claim 1 in batches, N is N max , N max is the maximum length of the phoneme sequence in the current batch of samples. max The phoneme sequence is padded with 0 at the end to make the sequence length N max .

8. A phoneme duration prediction method based on GA-IMV, characterized in that: The following steps are involved: S1, inputting the phoneme sequence to be predicted into the trained speech synthesis model obtained in claim 6 to obtain an attention weight matrix A; S2, calculate the dynamic threshold of each group of row vectors in the attention weight matrix A, where β n is the nth row vector in the attention weight matrix A, β n The dynamic threshold is G n , G n The calculation method is as follows: G n =μ n +s n ; Among them, μ n is the row vector β n The mean of all elements in , σ n is the row vector β n The standard deviation of all elements in ; S3, for row vector β n Medium greater than G n And count the consecutive elements to get the row vector β n The total number of consecutive elements of the row vectors is then converted into a sequence of consecutive element totals f in the order of increasing n values. n ; Calculate D total , and then calculate the phoneme duration d=D corresponding to each element total / T,D total The calculation formula is as follows: Wherein, L is the frame length of the audio frame, M is the frame shift of the audio frame, and F is the sampling frequency of the audio; S4, the total number of consecutive elements sequence f n Multiply it by the phoneme duration d to obtain the phoneme duration sequence D = {D1, D2, D3, ..., D N }.

Images