A deep learning-based audiobook prosodic speech synthesis method

Abstract

This invention discloses a deep learning-based method for synthesizing audiobook prosody. First, a speaker encoder module extracts audio features from reference audio. Second, the audio text to be synthesized is fed into an encoder and duration predictor, where the reference audio features are integrated to generate an audio code. The prosody predictor and decoder then output the synthesized audio spectrum. Finally, a vocoder converts the spectrum into synthesized audio. By introducing lightweight convolution and a prosody predictor, and training a model based on phoneme duration and prosodic features, this method controls the global prosody of the generated audio, addressing the issues of prone pronunciation errors and monotonous, unchanging prosody.

Description

Technical Field

[0001] The present invention mainly relates to speech synthesis technology in the field of natural language processing, and provides a deep learning-based audio book prosodic speech synthesis method. It uses three modules: speaker encoder, acoustic model and vocoder to perform audio synthesis of text content, helping people enjoy the achievements of modern science and technology more comfortably and get a better experience when listening to books. Background Art

[0002] With the development of modern society, electronic reading has gradually become popular. However, prolonged staring at electronic screens can easily lead to visual fatigue, leading to a surge in popularity of listening to books. With the advent of a technological society, people have begun exploring methods for synthesizing audio using machines. We can now see speech synthesis applications everywhere, such as voice navigation and intelligent operation guidance. Initially, people's expectations for speech synthesis were for accurate pronunciation. However, as people's quality of life has improved, their expectations for generated speech quality have also increased, leading to the pursuit of synthesized pronunciation that is both highly intelligible and pleasant to the ear. Initial concatenative speech synthesis methods required a large corpus and suffered from artifacts during concatenation. Improved parametric speech synthesis methods can improve pronunciation intelligibility, but they require complex pipeline design and extensive audio and video processing knowledge, and the generated electronic sound can be unpleasant to the ear. With the development of deep learning networks, deep learning-based speech synthesis methods have gradually become mainstream. Many successful algorithms have been developed for speech synthesis tasks, such as SpeedySpeech. These algorithms primarily generate mel-spectrograms through neural network training, followed by decoding of the sound waveform using a vocoder. This simplifies the speech synthesis pipeline and improves synthesis efficiency. However, these algorithms can only synthesize audio with a neutral tone, lacking rhythmic quality. Adding rhythmic variations to reading can better tell a story and make the reader more enjoyable. Therefore, improving the rhythmic performance of synthesized audio has become a key research direction in speech synthesis. Summary of the Invention

[0003] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide an audio book rhythmic speech synthesis method based on deep learning (the audio book rhythmic synthesis method refers to a speech synthesis method that generates rhythmically rich audio in the audio book reading scenario).

[0004] To achieve the above objectives, the technical solutions provided by the present invention are:

[0005] A deep learning-based audiobook prosodic speech synthesis method synthesizes audio content from text content through three modules: a speaker encoder, an acoustic model, and a vocoder. The method includes the following steps:

[0006] Step S1: Select reference audio Wav from the dataset reference , which is converted into the reference spectrum G after preprocessing reference , construct the speaker identity label and map it into vector S;

[0007] Step S2: The reference spectrum G reference Input into a long short-term memory network, and then obtain the output of the speaker encoder through the linear layer, and output the structure vector S pred Perform a cosine similarity loss operation on the speaker identity label vector S in the training set to reduce the distance between vectors of the same speaker;

[0008] Step S3: Repeat the operation of step S2 to optimize the network structure until the number of iterations is reached and the speaker vector encoder is Training completed;

[0009] Step S4: Randomly select a reference spectrum G from the data set reference , and select the training text Text and the corresponding real audio Wav at the same time; after preprocessing, convert the Text into the original phoneme sequence phonemes, and extract the real phoneme duration sequence T and the real Mel spectrum G of the audio; send the original phoneme sequence phonemes to the encoder Encoder to obtain the robustness sequence Z of the phonemes;

[0010] Step S5: Input the robustness sequence Z of the phoneme into the duration predictor DurationPrediction to obtain the duration of each phoneme, and output the predicted phoneme duration sequence T pred During training, the predicted phoneme duration sequence T is calculated by smoothing the L1 loss function pred The error between the actual phoneme duration sequence T and the actual phoneme duration sequence T is used to optimize the network parameters through back propagation.

[0011] Step S6: Combine the robustness sequence Z of the phoneme and the predicted phoneme duration sequence T pred They are sent to the expansion module for copying and expansion, and the position encoding PositionEmbedding and speaker encoding SpeakingEmbedding are added to obtain an extended phoneme sequence K with the same length as the synthesized audio;

[0012] Step S7: Send the extended phoneme sequence K to the prosody predictor ProsyPrediction to obtain the prosody sequence of the phonemes one by one, and finally output the phoneme prosody sequence P;

[0013] Step S8: Combine the extended phoneme sequence K and the phoneme prosody sequence P as the input of the decoder, and finally decode the Mel spectrum G predDuring training, the Adam optimizer, SSIM loss function and L1 loss function are used to optimize the model.

[0014] Step S9: Repeat steps S4-S8 to optimize the network structure until the maximum number of iterations is reached and the acoustic model structure f is trained;

[0015] Step S10: Send the real Mel spectrum G to the generator g of the vocoder to obtain the corresponding audio sequence; then the discriminator generates the output audio Wav pred Scoring, reducing the generator to generate audio Wav pred The difference between the real audio Wav is used to optimize the generator structure;

[0016] Step S11: Repeat step S10 to optimize the network structure until the maximum number of iterations is reached and the vocoder structure g training is completed;

[0017] Step S12: Convert the reference audio Wav reference Preprocessing to obtain the reference spectrum G reference , and passed into the speaker encoder Extract the speaker encoding vector SpeakEmbedding;

[0018] Step S13: Convert the text to be synthesized into a phoneme sequence phonemes, input the phoneme sequence phonemes into the acoustic model f and combine the speaker encoding vector SpeakEmbedding and position encoding PositionEmbedding to obtain the predicted Mel spectrum G pred ;

[0019] Step S14: Finally, the Mel spectrum G pred Input the generator g of the vocoder model to get the predicted audio Wav pred .

[0020] Furthermore, the step S1 is specifically as follows: obtaining the true mel spectrum G through Fourier transform, extracting the true phoneme duration T as a training label through the pre-trained dilated convolution combined with the autoregressive teacher model of the self-attention structure; at the same time, the audio clips of different speakers in the dataset are divided into short audio clips and converted into mel spectrum G reference , the relevant audio and the corresponding speaker together constitute the speaker identity label, and the speaker identity label is mapped to a vector S.

[0021] Furthermore, the loss function used in step S2 is as follows:

[0022]

[0023] represents the output vector, S i is the training dataset label, a i Represents similarity, 1 indicates similarity, -1 indicates dissimilarity, i represents the i-th vector in the sequence, and margin represents a constant.

[0024] Furthermore, in step S4, the robustness sequence Z of the phonemes is obtained using the following formula:

[0025] Z=Encoder(phonemes)

[0026] Encoder represents the encoder module, and phonemes represents the original phoneme sequence.

[0027] Furthermore, in step S5, the predicted phoneme duration sequence T pred Calculate it using the following formula:

[0028] T pred =DurationPrediction(Z)

[0029] DurationPrediction represents the duration predictor module, Z represents the robust sequence of phonemes;

[0030] The smooth L1 loss function during training is as follows:

[0031]

[0032] tp j Indicates the predicted duration of the jth phoneme in the input sequence, t j Indicates the actual duration of each phoneme, l j Represents the error between the prediction and the true value of each phoneme.

[0033] Furthermore, the step S6 is decomposed into predicting the robustness sequence Z of the phoneme according to the phoneme duration sequence T pred By expanding by copying, the expanded phoneme sequence K is obtained, and the formula is as follows:

[0034] K=copy(Z,T pred )+PositionEmbedding+SpeakEmbedding.

[0035] Furthermore, in step S7, the predicted prosodic phoneme sequence P is obtained by the following formula:

[0036] P = ProsyPrediction(K)

[0037] ProsyPrediction represents the prosody predictor, and K represents the expanded phoneme sequence.

[0038] Furthermore, in step S8, the predicted Mel spectrum G pred Calculate it using the following formula:

[0039] G pred =Decoder(cat(K,P))

[0040] K represents the expanded phoneme sequence, and P represents the predicted prosodic phoneme sequence;

[0041] The loss function is decomposed into the predicted Mel spectrum G pred Comparison of SSIM and L1 loss with the real Mel spectrum G:

[0042]

[0043]

[0044] μ g 、μ m Represents the mean of the Gaussian function after the real Mel spectrum and the predicted spectrum are fitted, σ g , σ m Represents the variance of the Gaussian function after the real Mel spectrum and the predicted spectrum fitting respectively; f(x i ) represents the predicted Mel spectrum G pred Frame i, y i Represents the i-th frame of the true Mel spectrum G, where i represents the position in the frame sequence; Loss ssim represents the structural similarity loss function, represents the mean absolute error function.

[0045] Furthermore, the step S10 is further refined as follows: the discriminator of the vocoder scores the output audio of the generator three times, obtains the final score by normalizing the average value, optimizes the generator structure, and completes the training of the vocoder part.

[0046] Compared with the prior art, the principles and advantages of the present invention are as follows:

[0047] 1. This invention introduces lightweight convolution to address semantic errors caused by non-autoregressive models' failure to consider the associations between contextual phonemes. Lightweight convolution uses a fixed window with equal context and a softmax function to determine the importance of elements within the window. This window is shifted forward in time. By focusing on the important phonemes at each moment, the model can generate more accurate pronunciations.

[0048] 2. The present invention introduces a rhythm prediction module and combines the phoneme duration and phoneme rhythm features to train the model, thereby controlling the global rhythm of the generated audio, alleviating the one-to-many problem that is prone to occur in non-autoregressive speech generation models, and generating audio that is easily accepted by the human ear. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the services required for use in the embodiments or the prior art descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is a principle flow chart of a method for synthesizing audiobook prosody based on deep learning in the present invention;

[0051] Figure 2 This is a training flow chart of three network modules in a method for synthesizing audiobook prosody based on deep learning according to the present invention;

[0052] Figure 3 This is a comparison chart of the effects of the method of the present invention and other methods. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with specific embodiments:

[0054] like Figure 1-Figure 2 As shown, the method for synthesizing audiobook prosody based on deep learning described in this embodiment specifically includes the following steps:

[0055] Step S1: Select reference audio Wav from the dataset reference , which is converted into the reference spectrum G after Fourier transformation reference The audio in the dataset is divided into 1.6-second audio examples and related speaker identity tags to construct speaker identity tags, which are mapped into vectors S and used as training datasets for speaker encoding.

[0056] Step S2: The reference spectrum G reference Input into a long short-term memory network, and then obtain the output of the speaker encoder through the linear layer, and output the structure vector S pred Perform cosine similarity loss operation with the training dataset label S to reduce the loss and perform module training. i Refers to the similarity between the two, 1 means similar, -1 means dissimilar;

[0057]

[0058] S i pred represents the output vector, S i is the training dataset label, a i Represents similarity, 1 indicates similarity, -1 indicates dissimilarity, i represents the i-th vector in the sequence, and margin represents a constant.

[0059] Step S3: Repeat the operation of step S2 and continuously optimize the network structure until the number of iterations reaches 100, and the speaker vector encoder is completed. train.

[0060] Step S4: Randomly select a reference spectrum G from the data set reference , and select the training text Text and the corresponding real audio Wav. After preprocessing, the Text is converted into the original phoneme sequence phonemes={p1,p2,...,p N}, and extract the real phoneme duration sequence T = {t1, t2, ..., t N} and the true Mel spectrum G={g1,g2,g3,...,g 80}, construct the training data set of the acoustic model. At the same time, the original phoneme sequence phonemes={p1,p2,...,p N} is sent to the encoder to obtain the robust sequence of phonemes Z = {z1,z2,z3,...,z N}.

[0061] Step S5: The robust sequence of phonemes Z = {z1, z2, z3, ..., z N} Input the duration predictor DurationPrediction to obtain the duration of each phoneme, and output the predicted phoneme duration sequence T pred ={tp1,tp2,...,tp N During training, the predicted phoneme duration sequence T is calculated by smoothing the L1 loss function. pred The error between the actual phoneme duration sequence T and the actual phoneme duration sequence T is used to optimize the network parameters through back propagation. The duration predictor is trained using the smooth L1 loss function.

[0062] T pred =DurationPrediction(Z)

[0063]

[0064] Step S6: Combine the robustness sequence Z of the phoneme and the predicted phoneme duration sequence T pred They are sent to the expansion module respectively, according to the predicted phoneme duration sequence Tpred ={tp1,tp2,...,tp N} Expand the robust phoneme sequence Z and add the sine and cosine position coding PositionEmbedding and speaker coding SpeakingEmbedding to obtain an extended phoneme sequence K = {k1, k2, k3, ..., k L}.

[0065] K=copy(Z,T pred )+PositionEmbedding+SpeakEmbedding

[0066] Step S7: Extend the phoneme sequence K={k1,k2,k3,...,k L} is sent to the prosody predictor ProsyPrediction to obtain the prosodic sequence of phonemes, and finally the phoneme prosodic sequence P is output.

[0067] P = ProsyPrediction(K)

[0068] Step S8: Combine the extended phoneme sequence K and the phoneme rhythm sequence P and input them into the decoder to perform Mel spectrum G pred Predict, perform L1 and SSIM loss operations on the real Mel spectrum G, reduce the loss and perform module training. g 、μ m Represents the true Mel spectrum G and the predicted Mel spectrum G respectively pred The mean of the fitted Gaussian function, σ g , σ m They represent the variance of the Gaussian function after fitting the real Mel spectrum and the predicted spectrum, c1 and c2 are constants; f(x i ) represents the predicted Mel spectrum G pred Frame i, y i Represents the i-th frame of the true Mel spectrum G, where i represents the position in the frame sequence; Loss ssim Represents the structural similarity loss function, and Lossl1 represents the mean absolute error function.

[0069] G pred =Decoder(cat(K,P))

[0070]

[0071]

[0072] Step S9: Repeat the operations of steps S3-S8 to continuously optimize the network structure until the number of iterations reaches 100, and the training of the acoustic model structure f is completed.

[0073] Step S10: Send the real Mel spectrum G to the generator g of the vocoder to obtain the corresponding audio sequence. Then the discriminator performs the output audio Wav of the generator respectively. pred , the audio h sampled once by Average Pool and the audio sampled twice by Average Pool are scored three times to obtain the normalized average value and reduce the generator to generate audio Wav pred The difference between the real audio Wav is used to optimize the generator structure.

[0074] Step S11: Repeat step S10 to optimize the network structure until the maximum number of iterations reaches 100, completing the training of the vocoder structure g;

[0075] Step S12: Convert the reference audio Wav reference Preprocessing to obtain the reference spectrum G reference , and passed into the speaker encoder Extract the speaker encoding vector SpeakEmbedding.

[0076] Step S13: Convert the text to be synthesized into a phoneme sequence phonemes, input the phoneme sequence phonemes into the acoustic model f and combine the speaker encoding vector SpeakEmbedding and position encoding PositionEmbedding to obtain the predicted Mel spectrum G pred .

[0077] Step S14: Finally, the Mel spectrum G pred Input the generator g of the vocoder model to get the predicted audio Wav pred .

[0078] In order to verify the effectiveness of the method of the present invention, the speech synthesis effect of the method is compared with that of Speedyspeech. Figure 3The specific Mel-spectrogram synthesis effects of the above two speech synthesis methods on the text "As has been pointed out, the Commission has not resolved all the proposals which could be made. The Commission never-theless is confident that" in the ESD (Emotional Speech Database) dataset are demonstrated. Through comparison, it can be found that the method proposed in the present invention can help generate acoustic features that are closer to the actual Mel-spectrogram, thereby generating audio with more accurate pronunciation.

[0079] The embodiments described above are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, any changes made based on the shape and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for synthesizing audiobook prosody based on deep learning, characterized in that: Audio synthesis of text content is performed through three modules: speaker encoder, acoustic model, and vocoder. The steps include: Step S1: Select reference audio Wav from the dataset reference , will refer to the audio Wav reference After preprocessing, it is converted into the reference spectrum G reference , construct the speaker identity label and map it into vector S; Step S2: The reference spectrum G reference Input into a long short-term memory network, and then obtain the output of the speaker encoder through the linear layer, and convert the output vector S of the speaker encoder into pred Perform a cosine similarity loss operation on the speaker identity label vector S in the training set to reduce the distance between vectors of the same speaker; Step S3: Repeat the operation of step S2 to optimize the network structure until the number of iterations is reached and the speaker vector encoder is Training completed; Step S4: Randomly select a reference spectrum G from the data set reference , and select the training text Text and the corresponding real audio Wav at the same time; after preprocessing, convert the Text into the original phoneme sequence phonemes, and extract the real phoneme duration sequence T and the real Mel spectrum G of the audio; send the original phoneme sequence phonemes to the encoder Encoder to obtain the robustness sequence Z of the phonemes; Step S5: Input the robustness sequence Z of the phoneme into the duration predictor DurationPrediction to obtain the duration of each phoneme, and output the predicted phoneme duration sequence T pred During training, the predicted phoneme duration sequence T is calculated by smoothing the L1 loss function pred The error between the actual phoneme duration sequence T and the actual phoneme duration sequence T is used to optimize the network parameters through back propagation. Step S6: Combine the robustness sequence Z of the phoneme and the predicted phoneme duration sequence T pred They are sent to the expansion module for copying and expansion, and the position encoding PositionEmbedding and speaker encoding SpeakingEmbedding are added to obtain an extended phoneme sequence K with the same length as the synthesized audio; Step S7: Send the extended phoneme sequence K to the prosody predictor ProsyPrediction to obtain the prosody sequence of the phonemes one by one, and finally output the phoneme prosody sequence P; Step S8: Combine the extended phoneme sequence K and the phoneme prosody sequence P as the input of the decoder, and finally decode the Mel spectrum G pred During training, the Adam optimizer, SSIM loss function and L1 loss function are used to optimize the model. Step S9: Repeat steps S4-S8 to optimize the network structure until the maximum number of iterations is reached and the acoustic model structure f is trained; Step S10: Send the real Mel spectrum G to the generator g of the vocoder to obtain the corresponding audio sequence; then the discriminator generates the output audio Wav pred Scoring, reducing the generator to generate audio Wav pred The difference between the real audio Wav is used to optimize the generator structure; Step S11: Repeat step S10 to optimize the network structure until the maximum number of iterations is reached and the vocoder structure g training is completed; Step S12: Convert the reference audio Wav reference Preprocessing to obtain the reference spectrum G reference , and passed into the speaker encoder Extract the speaker encoding vector SpeakEmbedding; Step S13: Convert the text to be synthesized into a phoneme sequence phonemes, input the phoneme sequence phonemes into the acoustic model f and combine the speaker encoding vector SpeakEmbedding and position encoding PositionEmbedding to obtain the predicted Mel spectrum G pred ; Step S14: Finally, the Mel spectrum G pred Input the generator g of the vocoder model to get the predicted audio Wav pred .

2. The method for synthesizing audiobook prosody based on deep learning according to claim 1, characterized in that: The step S1 specifically comprises: obtaining the true Mel spectrum G through Fourier transform, extracting the true phoneme duration T as a training label through the pre-trained dilated convolution combined with the autoregressive teacher model of the self-attention structure; at the same time, cutting the audio clips of different speakers in the dataset into short audio clips and converting them into reference spectrum G reference , the relevant audio and the corresponding speaker together constitute the speaker identity label, and the speaker identity label is mapped to a vector S.

3. The method for synthesizing audiobook prosody based on deep learning according to claim 1, characterized in that: The loss function used in step S2 is as follows: represents the output vector, S i is the training dataset label, a i Represents similarity, 1 indicates similarity, -1 indicates dissimilarity, i represents the i-th vector in the sequence, and margin represents a constant.

4. The method for synthesizing audiobook prosody based on deep learning according to claim 1, characterized in that: In step S4, the robustness sequence Z of the phonemes is obtained using the following formula: Z=Encoder(phonemes) Encoder represents the encoder module, and phonemes represents the original phoneme sequence.

5. The method for synthesizing audiobook prosody based on deep learning according to claim 1, characterized in that: In step S5, the predicted phoneme duration sequence T pred Calculate it using the following formula: T pred =DurationPrediction(Z) DurationPrediction represents the duration predictor module, Z represents the robust sequence of phonemes; The smooth L1 loss function during training is as follows: tp j Indicates the predicted duration of the jth phoneme in the input sequence, t j Indicates the actual duration of each phoneme, l j Represents the error between the prediction and the true value of each phoneme.

6. The method for synthesizing audiobook prosody based on deep learning according to claim 1, characterized in that: The step S6 is decomposed into predicting the robust sequence Z of the phoneme according to the phoneme duration sequence T pred By expanding by copying, the expanded phoneme sequence K is obtained, and the formula is as follows: K=copy(Z,T pred )+PositionEmbedding+SpeakEmbedding。 7. The method for synthesizing audiobook prosody based on deep learning according to claim 1, characterized in that: In step S7, the predicted prosodic phoneme sequence P is obtained by the following formula: P = ProsyPrediction(K) ProsyPrediction represents the prosody predictor, and K represents the expanded phoneme sequence.

8. The method for synthesizing audiobook prosody based on deep learning according to claim 1, characterized in that: In step S8, the predicted Mel spectrum G pred Calculate it using the following formula: G pred =Decoder(cat(K,P)) K represents the expanded phoneme sequence, P represents the predicted prosodic phoneme sequence, and Decoder represents the decoder; The loss function is decomposed into the predicted Mel spectrum G pred Comparison of SSIM and L1 loss with the real Mel spectrum G: μ g 、μ m Represents the mean of the Gaussian function after the real Mel spectrum and the predicted spectrum are fitted, σ g , σ m Represents the variance of the Gaussian function after the real Mel spectrum and the predicted spectrum fitting respectively; c1 and c2 are constants; f(x i ) represents the predicted Mel spectrum G pred Frame i, y i Represents the i-th frame of the true Mel spectrum G, where i represents the position in the frame sequence; Loss ssim Represents the structural similarity loss function, Loss L1 represents the mean absolute error function.

9. The method for synthesizing audiobook prosody based on deep learning according to claim 1, characterized in that: The step S10 is further refined as follows: the discriminator of the vocoder scores the output audio of the generator three times, obtains the final score by normalizing the average value, optimizes the generator structure, and completes the training of the vocoder part.

Images