A prosody transfer method and system for speech synthesis

Abstract

The present application relates to the technical field of speech synthesis, and specifically discloses a prosody migration speech synthesis method and system, which comprises the following steps: obtaining Chinese speech data with text annotation and speaker label; preprocessing the Chinese speech data and corresponding text data to obtain phoneme sequence, frame-level mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy; constructing a speech synthesis model based on prosody migration; inputting the speaker label, phoneme sequence, frame-level mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy into the speech synthesis model to obtain speaker classification results and prediction values; training the speech synthesis model based on the frame-level mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy, and the speaker classification results and prediction values to obtain a customized speech synthesis model; and synthesizing unknown speaker speech based on the customized speech synthesis model.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and more specifically to a prosodic transfer speech synthesis method and system. Background Technology

[0002] With the development of speech synthesis models, the quality of synthesized speech is constantly improving, and the similarity between speakers is getting higher and higher. However, speech synthesis has a one-to-many problem where one text can map to multiple speech. Simply synthesizing the voice of the target speaker is no longer enough for us. We need more methods and means to regulate the entire model so that the synthesized speech can be more controlled.

[0003] Prosody transfer-based speech synthesis has emerged, which forces the model to synthesize speech that conforms to the current prosodic form rather than arbitrarily synthesizing a learned prosodic form by taking input speech. Past prosody transfer-based speech synthesis models often used single acoustic features such as Mel spectrum to predict prosodic information, but this approach does not have sufficient input features and does not take into account the differences in pitch and energy of different speakers, resulting in poor synthesized speech quality. Summary of the Invention

[0004] To address the aforementioned problems, one objective of this invention is to provide a prosodic transfer speech synthesis method. This method uses multiple input features (including Mel spectrum, pitch information, energy information, and speaker features) to predict prosody, and is supplemented by a gradient-inverted speaker classifier to remove speaker information from the features encoded by Mel spectrum, thereby enhancing the effect of target speaker transfer.

[0005] A second objective of this invention is to provide a prosodic transfer speech synthesis system.

[0006] The first technical solution adopted in this invention is: a prosodic transfer speech synthesis method, comprising the following steps:

[0007] S100: Acquire Chinese speech data with text annotations and speaker labels; preprocess the Chinese speech data and its corresponding text data to obtain phoneme sequences, frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy.

[0008] S200: Construct a speech synthesis model based on prosodic transfer. The prosodic transfer-based speech synthesis model includes a speaker feature table, a phoneme feature table, a prosodic encoder, a conditional encoder, an adaptive predictor group, and a decoder. The prosodic encoder includes a linear layer, a gradient inversion layer, and a speaker classifier. Input the speaker labels, phoneme sequences, frame-level Mel spectra, phoneme-level duration, phoneme-level pitch, and phoneme-level energy into the prosodic transfer-based speech synthesis model to obtain speaker classification results, predicted Mel spectra, predicted duration, predicted energy, and predicted pitch.

[0009] S300: The prosody-transfer-based speech synthesis model is trained based on the frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch and phoneme-level energy, as well as the speaker classification results, predicted Mel spectrum, predicted duration, predicted energy and predicted pitch, to obtain a customized speech synthesis model;

[0010] S400: Obtain the text of an unknown speaker, input the text of the unknown speaker into the customized speech synthesis model to obtain the predicted Mel spectrum of the unknown speaker; synthesize the speech of the unknown speaker based on the predicted Mel spectrum of the unknown speaker.

[0011] Preferably, the preprocessing in step S100 includes the following sub-steps:

[0012] S110: Convert the text data into a phoneme sequence;

[0013] S120: The phoneme sequence and the corresponding Chinese speech data are fed into the alignment model to obtain the phoneme-level duration;

[0014] S130: Predict the pitch of the Chinese speech data to obtain frame-level pitch; average the frame-level pitch based on the phoneme-level duration to obtain phoneme-level pitch;

[0015] S140: The speech waveform of the Chinese speech data is processed by short-time Fourier transform and Mel spectrum transformation to generate frame-level Mel spectrum and frame-level energy; the frame-level energy is averaged based on the phoneme-level duration to obtain the phoneme-level energy.

[0016] Preferably, the adaptive predictor group in step S200 includes an adaptive duration predictor, an adaptive pitch predictor, an adaptive energy predictor, and a length normalizer.

[0017] Preferably, in step S200, the predicted Mel spectrum, predicted duration, predicted energy, and predicted pitch are obtained through the following sub-steps:

[0018] S210: Based on the speaker tags, query the speaker feature table to generate speaker features;

[0019] S220: Input the speaker features, frame-level Mel spectrum, phoneme-level pitch and phoneme-level energy into the prosody encoder to obtain prosody features and speaker classification results;

[0020] S230: Based on the phoneme sequence, query the phoneme feature table to generate initial phoneme features; input the prosodic features and the initial phoneme features into the conditional encoder to obtain the phoneme-level first hidden features;

[0021] S240: Input the phoneme-level first hidden feature, prosodic feature, phoneme-level duration, phoneme-level pitch, and phoneme-level energy into the adaptive predictor group to obtain the prediction duration, prediction energy, prediction pitch, and frame-level second hidden feature.

[0022] S250: Input the frame-level second hidden feature and prosodic feature into the decoder to obtain the predicted Mel spectrum.

[0023] Preferably, step S240 includes:

[0024] The phoneme-level first hidden feature is input into the adaptive duration predictor, the adaptive pitch predictor, and the adaptive energy predictor, respectively, to obtain the predicted duration, predicted pitch, and predicted energy.

[0025] The phoneme-level pitch and phoneme-level energy are added to the phoneme-level first hidden feature to generate the phoneme-level second hidden feature.

[0026] Based on the phoneme-level duration, the phoneme-level second hidden feature is expanded into a frame-level second hidden feature using a length warper.

[0027] Preferably, step S300 includes the following sub-steps:

[0028] S310: The speaker classification results, predicted Mel spectrum, predicted duration, predicted energy and pitch, as well as frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch and phoneme-level energy are used to calculate the loss through a loss function, and the prosody transfer-based speech synthesis model is pre-trained until convergence.

[0029] S320: Fine-tune the pre-trained prosody transfer-based speech synthesis model to obtain a customized speech synthesis model.

[0030] Preferably, the loss function in step S310 is expressed by the following formula:

[0031] Loss Adaptor =λ p*Loss P +λ e *Loss E +λ d *Loss D

[0032] Loss = Loss Adapto r+Loss Mel +Loss spk-_classifier

[0033] In the formula, Loss represents the total loss; Mel Loss is the Mel-spectral loss. Adaptor λ is the loss of the adaptive predictor group; p , λ e , λ d The weights for the adaptive pitch predictor, adaptive energy predictor, and duration predictor are all set to 1; Loss P For pitch loss; Loss E Loss is energy loss. D Loss is due to time loss. spk-_classifier This represents the cross-entropy loss.

[0034] Preferably, the mean squared error is used to calculate the predicted Mel spectrum and the frame-level Mel spectrum to obtain the Mel spectrum loss;

[0035] The mean squared error is used to calculate the predicted duration and phoneme-level duration to obtain the duration loss;

[0036] The predicted energy and phoneme-level energy are calculated using the mean absolute error to obtain the energy loss;

[0037] The predicted pitch and phoneme-level pitch are calculated using the mean absolute error to obtain the pitch loss;

[0038] Calculate the cross-entropy loss based on the speaker classification results.

[0039] The second technical solution adopted in this invention is: a prosodic transfer speech synthesis system, comprising a preprocessing module, a model building module, a training module, and a speech synthesis module.

[0040] The preprocessing module is used to acquire Chinese speech data with text annotations and speaker labels; and to preprocess the Chinese speech data and its corresponding text data to obtain phoneme sequences, frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy.

[0041] The model building module is used to construct a speech synthesis model based on prosodic transfer. The speaker labels, phoneme sequences, frame-level Mel spectra, phoneme-level duration, phoneme-level pitch, and phoneme-level energy are input into the prosodic transfer-based speech synthesis model to obtain speaker classification results, predicted Mel spectra, predicted duration, predicted energy, and predicted pitch. The prosodic transfer-based speech synthesis model includes a speaker feature table, a phoneme feature table, a prosodic encoder, a conditional encoder, an adaptive predictor group, and a decoder. The prosodic encoder includes a linear layer, a gradient inversion layer, and a speaker classifier.

[0042] The training module is used to train the prosody transfer-based speech synthesis model based on the frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch and phoneme-level energy, as well as the speaker classification results, predicted Mel spectrum, predicted duration, predicted energy and predicted pitch, to obtain a customized speech synthesis model.

[0043] The speech synthesis module is used to acquire the text of an unknown speaker, input the text of the unknown speaker into the customized speech synthesis model to obtain the predicted Mel spectrum of the unknown speaker, and synthesize the speech of the unknown speaker based on the predicted Mel spectrum of the unknown speaker.

[0044] Preferably, the model building module obtains the predicted Mel spectrum, predicted duration, predicted energy, and predicted pitch through the following operations:

[0045] Based on the speaker tags, the speaker feature table is queried to generate speaker features;

[0046] The speaker features, frame-level Mel spectrum, phoneme-level pitch, and phoneme-level energy are input into the prosodic encoder to obtain prosodic features and speaker classification results.

[0047] Based on the phoneme sequence, the phoneme feature table is queried to generate initial phoneme features; the prosodic features and the initial phoneme features are input into the conditional encoder to obtain the phoneme-level first hidden features;

[0048] The phoneme-level first hidden feature, prosodic feature, phoneme-level duration, phoneme-level pitch, and phoneme-level energy are input into the adaptive predictor group to obtain the prediction duration, prediction energy, prediction pitch, and frame-level second hidden feature.

[0049] The frame-level second hidden feature and prosodic feature are input into the decoder to obtain the predicted Mel spectrum.

[0050] The beneficial effects of the above technical solution are as follows:

[0051] (1) The prosody transfer speech synthesis method disclosed in this invention uses multiple input features (including Mel spectrum, pitch information, energy information, and speaker features) to predict prosody, and is supplemented by a gradient inversion speaker classifier to remove the speaker information content in the features after Mel spectrum encoding, thereby enhancing the effect of target speaker transfer.

[0052] (2) The present invention uses a method of inputting a speech to control the model to generate target speech with the same prosodic rhythm as the prosodic sample speech; specifically, it involves using mel-spec, speaker features, energy, and pitch as input to predict global prosodic features, and using the prosodic features as conditions to feed into encoder (CFT*4), decoder (AFT*4), and adaptive predictor group to generate high-quality speech that is highly close to the input prosodic. Attached Figure Description

[0053] Figure 1 A schematic flowchart of a prosodic transfer speech synthesis method provided in one embodiment of the present invention;

[0054] Figure 2 A schematic diagram of the structure of a speech synthesis model based on prosody transfer provided in an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a prosody encoder provided in one embodiment of the present invention;

[0056] Figure 4 A schematic diagram of the structure of a CFT provided in one embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the structure of an adaptive predictor group provided in one embodiment of the present invention;

[0058] Figure 6 A schematic diagram of the adaptive duration predictor, adaptive pitch predictor, and adaptive energy predictor provided in an embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of the structure of an AFT provided in one embodiment of the present invention;

[0060] Figure 8 This is a schematic diagram of the structure of a prosodic transfer speech synthesis system provided in one embodiment of the present invention. Detailed Implementation

[0061] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following detailed description of the embodiments and the accompanying drawings are used to illustrate the principles of the present invention by way of example, but should not be used to limit the scope of the present invention. That is, the present invention is not limited to the described preferred embodiments, and the scope of the present invention is defined by the claims.

[0062] In the description of this invention, it should be noted that, unless otherwise stated, "a plurality of" means two or more; the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance; those skilled in the art can understand the specific meaning of the above terms in this invention as appropriate.

[0063] Phoneme level, syllable level, and frame level refer to different levels of sequences (phoneme sequence, syllable sequence, and frame sequence), and the lengths of sequences at different levels are different: syllable sequence < phoneme sequence < frame sequence; for example, a speech with 5 Chinese characters has a syllable sequence of 5 pinyin, which is 5 in length; the phoneme sequence might be 10 or 12, etc., if we go down one level; but the frame sequence is segmented from the speech. For example, a 16000Hz speech with 5 characters, assuming it takes 2 seconds, has 32,000 sample points, and 32000 / 256 (the length of the segmented frames, called hop_size) = 125 frames.

[0064] Example 1

[0065] like Figure 1 As shown, one embodiment of the present invention provides a prosodic transfer speech synthesis method, comprising the following steps:

[0066] S100: Acquire Chinese speech data with text annotations and speaker labels; preprocess the Chinese speech data and its corresponding text data to obtain phoneme sequences, frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy.

[0067] (1) Obtain Chinese speech data with text annotations and speaker labels, including:

[0068] We collected and obtained Chinese speech data with text annotations and speaker labels from the open-source clean speech dataset ESD Chinese speech data and AISHELL-3. Speaker labels were obtained by attaching an index to each speaker in the Chinese speech data. For example, speaker labels are generally labeled as 0, 1, 2, 3, 4, 5 according to the data processing order.

[0069] (2) Preprocessing the Chinese speech data and its corresponding text data includes:

[0070] S110: Convert text data into a phoneme sequence;

[0071] After processing the text data corresponding to the Chinese speech data through text regularization, it is then processed into pinyin to obtain a pinyin sequence (i.e., a syllable sequence); the pinyin sequence is converted into a phoneme sequence using a preset dictionary; the pinyin processing includes: using the pypinyin library to perform pinyin matching on the text after text regularization to obtain a pinyin sequence.

[0072] In addition to the standard 1st, 2nd, 3rd, and 4th tones, Chinese Pinyin also has a fifth tone (neutral tone). This invention uses the officially provided MFA toolkit and a Chinese Pinyin to phoneme dictionary to convert the Pinyin sequence into a phoneme sequence, mapping the input Pinyin sequence to the output phoneme sequence.

[0073] S120: The phoneme sequence and the corresponding Chinese speech data are fed into the alignment model to obtain the phoneme-level duration;

[0074] The above phoneme sequence and the corresponding Chinese speech data are input into the alignment model (obtained using the Montreal ForceAlignment (MFA) tool) for forced alignment. The time portion corresponding to the phoneme is obtained through the forced alignment result. The number of frames corresponding to each phoneme in the phoneme sequence is determined by frame shifting. The total number of frames is counted to obtain the phoneme-level duration.

[0075] S130: Use the PYWORDLD library to predict the pitch of Chinese speech data to obtain frame-level pitch; average the frame-level pitch based on the phoneme-level duration to obtain phoneme-level pitch;

[0076] S140: Based on the speech waveform of the Chinese speech data, a frame-level Mel spectrum and a frame-level energy are generated through short-time Fourier transform (STFT) and Mel spectrum conversion; the frame-level energy is averaged based on the phoneme-level duration to obtain the phoneme-level energy.

[0077] Based on the parameters in Table 1, the Chinese speech data is first resampled, and the sampling rate of all Chinese speech data is converted to 22050 Hz. The resampled speech data is then pre-emphasized at 0.97. Then, it is processed by the Short Time Fourier Transform (STFT) algorithm with a frame shift of 256, a window length, and a frame length of 1024. Finally, Mel spectrum conversion is performed to obtain the frame-level Mel spectrum. The Mel filter uses an 80-Melt filter bank, boosting from a minimum frequency of 0 to a maximum frequency of 8000 to maintain consistency with the settings of the HiFi-GAN vocoder.

[0078] Table 1 Audio Parameters

[0079]

[0080]

[0081] S200: Construct a speech synthesis model based on prosody transfer, and input the speaker label, phoneme sequence, frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy into the speech synthesis model based on prosody transfer to obtain speaker classification results, predicted Mel spectrum, predicted duration, predicted energy, and predicted pitch.

[0082] like Figure 2 As shown, the speech synthesis model based on prosodic transfer includes a prosodic encoder, a conditional encoder, an adaptive predictor group, and a decoder.

[0083] The prosody transfer-based speech synthesis model also includes a speaker feature table and a phoneme feature table. The phoneme feature table contains 256 hidden units, and the speaker feature table contains 256 hidden units.

[0084] The speaker labels, phoneme sequences, frame-level Mel spectra, phoneme-level durations, phoneme-level pitches, and phoneme-level energy are input into a prosodic transfer-based speech synthesis model to obtain speaker classification results, predicted Mel spectra, predicted phoneme-level durations, predicted energy, and predicted pitches. This includes the following sub-steps:

[0085] S210: Query the speaker feature table based on the speaker label to generate speaker features;

[0086] S220: Input speaker features, frame-level Mel spectrum, phoneme-level pitch and phoneme-level energy into the prosody encoder to obtain prosodic features and speaker classification results;

[0087] like Figure 3 As shown, the prosody encoder includes a linear layer, a gradient inversion layer, and a speaker classifier.

[0088] Speaker classifier = linear layer * 1 (the predicted output of speaker classification can be obtained by mapping the linear layer). In the figure, all Conv1d have channels = 256, kernel_size = 3, and stride = 1.

[0089] The Add operation is simply adding two or three inputs together. The prosodic encoder can predict the prosodic feature P and the speaker classification result (used for loss calculation).

[0090] Conv1d->Relu->LN: First calculate Conv1d, then activate the Relu function, and then normalize using layer norm.

[0091] A linear layer transforms the dimension of the input features (input_hidden) to the dimension of the output features (output_hidden) using a single matrix. For example, if the input features are 128-dimensional and the output features are 256-dimensional, and the input is [1, input_hidden], then the weight matrix of the linear layer is W = [input_hidden, output_hidden]; the output = Input * W (matrix multiplication), so the output = [1, output_hidden].

[0092] Gradient Reversal Layer: After passing through this layer, the backpropagated gradient will be multiplied by -1 (that is, the gradient itself makes the whole model move in the direction of continuously decreasing loss, thus achieving training; the gradient multiplied by -1 makes the whole model move in the direction of increasingly large loss, thus removing speaker information from the input features); this will make the model loss of the speaker classifier, which should be getting smaller and smaller, become larger and larger, achieving the effect of removing speaker information from frame-level Mel-spectral features.

[0093] The output dimension of the speaker classifier is the number of speakers, and the output dimension of the linear layer that outputs prosodic features P is equal to the speaker feature dimension.

[0094] S230: Based on the phoneme sequence, query the phoneme feature table to generate initial phoneme features; input the prosodic features and the initial phoneme features into the conditional encoder to obtain the phoneme-level first hidden features;

[0095] like Figure 4 As shown, the conditional encoder uses four CFT (condition forward transform) modules to model long sequences and encode text content. Each CFT module contains a self-attention (SA) layer with 256 hidden units, a two-layer 1D convolutional network, a residual connection, and a CLN layer. The hidden units are all 1024, the kernel sizes are 3 and 9 respectively, and the stride is 1. The ReLU activation function is used inside the SA layer and inside the two-layer convolutional network.

[0096] The prosodic features and initial phoneme features are input into a conditional encoder, and then passed through a self-attention layer and a 1D convolution to generate phoneme-level first hidden features.

[0097] S240: Input the phoneme-level first hidden feature, prosodic feature, phoneme-level duration, phoneme-level pitch, and phoneme-level energy into the adaptive predictor group to obtain the prediction duration, prediction energy, prediction pitch, and frame-level second hidden feature.

[0098] like Figure 5As shown, the adaptive predictor group includes three identical adaptive predictors: an adaptive duration predictor, an adaptive pitch predictor, and an adaptive energy predictor; the adaptive predictor group also includes a length warper.

[0099] like Figure 6 As shown, the adaptive duration predictor, adaptive pitch predictor, and adaptive energy predictor have the same structure, all using a two-layer 1D convolutional network structure connected by the ReLU activation function; the stride of both layers is 1, the kernel size is 3 and 9, and the hidden units are 1024 and 256; then the results are output through a CLN with 256 hidden units.

[0100] The adaptive duration predictor, adaptive pitch predictor, and adaptive energy predictor make predictions in parallel. Each predictor only takes the first hidden feature at the phoneme level as input and outputs the corresponding predicted pitch (i.e., predicted phoneme-level pitch), predicted duration (i.e., predicted phoneme-level duration), and predicted energy (i.e., predicted phoneme-level energy).

[0101] Phoneme-level pitch and phoneme-level energy are added to the phoneme-level first hidden feature to generate the phoneme-level second hidden feature. Based on the phoneme-level duration, the phoneme-level second hidden feature is expanded into a frame-level second hidden feature through a length regulator (the duration information is the corresponding integer duration of each phoneme, and the length regulator expands a single feature by the corresponding n times to form the frame-level second hidden feature). (In the training phase, phoneme-level duration, phoneme-level pitch, and phoneme-level energy are used, and in the testing phase, predicted pitch, predicted energy, and predicted duration are used).

[0102] S250: Input the frame-level second hidden feature and prosodic feature into the decoder (AFT*4) to obtain the predicted Mel spectrum.

[0103] like Figure 7 As shown, the decoder consists of four sequentially connected AFT (adaptive forward transform) structures. Each AFT structure includes an adaptive-self-attention (ASA) module with 256 hidden units, a two-layer 1D convolutional network, a residual connection, and a CLN layer with 256 hidden units. The hidden units are all 1024, the kernel sizes are 3 and 9 respectively, and the stride is 1. The ReLU activation function is used inside the ASA module and the two-layer convolutional network.

[0104] The frame-level second hidden feature and prosodic feature are input into the decoder. The ASA layer in the decoder first performs CLN conditional normalization on the input frame-level second hidden feature, with the prosodic feature P as the condition. Then, the conditionally normalized frame-level second hidden feature is used as the parameter variables Q, K, and V of the adaptive self-attention module and input into the self-attention mechanism inside the ASA layer. Among them, V needs to undergo an additional CLN conditional normalization operation. The self-attention mechanism (module) generates a weighted average of Q, K, and V. Finally, the weighted average result and V are used to perform residual calculation to obtain the predicted Mel spectrum; specifically, the formula is as follows:

[0105] X = CLN (Frame-level Second Hidden Feature, P)

[0106] Q = W q *CLN(X, P)⊙Scale(W) q )

[0107] K = W k *CLN(X, P)⊙Scale(W) k )

[0108] V = CLN(W) v *CLN(X, P)⊙Scale(W) v ), P)

[0109]

[0110] In the formula, X is the second hidden feature after conditional normalization; P is the prosodic feature; CLN represents conditional normalization, with P as the condition for CLN; W is the trainable weight matrix; W q W k W v These represent the subordinate modules corresponding to W; * represents matrix multiplication; ⊙ represents the dot product within the matrix; Scale represents scaling the result, making it easier to train the model; Q, K, and V are the normalized second hidden features; the softmax function normalizes the result, making large values ​​larger and small values ​​smaller; K T d is the transpose of the calculated matrix; q For Q ASA The feature dimensions.

[0111] S300: Based on frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch and phoneme-level energy, as well as speaker classification results, predicted Mel spectrum, predicted duration, predicted energy and predicted pitch, train a prosodic transfer-based speech synthesis model to obtain a customized speech synthesis model;

[0112] S310: Pre-training: The speaker classification results, predicted Mel spectrogram, predicted duration, predicted energy, and pitch, as well as frame-level Mel spectrogram, phoneme-level duration, phoneme-level pitch, and phoneme-level energy, are used to calculate the loss through a loss function. This process is then used to pre-train the prosodic transfer-based speech synthesis model until convergence. The loss function is expressed by the following formula:

[0113] Loss Adaptor =λ p *Loss P +λ e *Loss E +λ d *Loss D

[0114] Loss = Loss Adaptor +Loss Mel +Loss spk-_classifier

[0115] In the formula, Loss represents the total loss; Mel Loss is the Mel-spectral loss. Adaptor λ is the loss of the adaptive predictor group; p , λ e , λ d The weights for the adaptive pitch predictor, adaptive energy predictor, and duration predictor are all set to 1; Loss P For pitch loss; Loss E Loss is energy loss. D Loss is due to time loss. spk-_classifier Cross-entropy loss;

[0116] Among them, pitch loss and energy loss can be calculated using MAE (mean absolute error) loss, while duration loss and Mel spectrum loss can be calculated using MSE (mean squared error) loss; cross-entropy loss is calculated based on speaker classification results;

[0117] During training, mean squared error is used to calculate the predicted Mel spectrum and frame-level Mel spectrum to obtain Mel spectrum loss; mean squared error is used to calculate the predicted duration and phoneme-level duration to obtain duration loss; mean absolute error is used to calculate the predicted energy and phoneme-level energy to obtain energy loss; mean absolute error is used to calculate the predicted pitch and phoneme-level pitch to obtain pitch loss; all weight coefficients are 1.

[0118] The cross-entropy loss is calculated based on the speaker classification results. Cross-entropy loss = (softmax(predict), real). The number of categories in the speaker classification results is the total number of speakers, that is, 228 speakers have 228 categories. The prediction is a 228-dimensional model output. After the softmax() function, it can be summarized that the sum of 228 values ​​is 1. The real is [0, 0, 0, 0, 0, 1, 0, 0], which is a true label sequence with only 1.

[0119] The cross-entropy loss function is expressed by the following formula:

[0120]

[0121] In the formula, Loss spk-_classifier For cross-entropy loss; N is the number of samples in the current training iteration (representing how many samples are used in the current training iteration), for example, N=3, and i=1,2,...,N; L i y is the cross-entropy loss for sample i; M is the number of classes; ic The sign function (0 or 1) is set to 1 if the true class of sample i equals class c, and 0 otherwise; p ic Let be the predicted probability that observed sample i belongs to category c.

[0122] The optimizer used was the Adam optimizer with beta values ​​of 0.9 and 0.98. The total data consisted of approximately 100,000 speech data points used for training, which converged after 100,000 iterations. Each step had 64 samples, and each selection was randomized within the current round. In addition, 10 speakers were randomly selected without being used for training, and this data was used for testing.

[0123] S320: Fine-tune the pre-trained prosody transfer-based speech synthesis model to obtain a customized speech synthesis model;

[0124] Acquire a small amount of speech training data (including text and corresponding audio data) from an unknown speaker and preprocess it (the preprocessing method is the same as in step S100) to obtain frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy. Freeze all weights in the pre-trained prosodic transfer-based speech synthesis model except for the internal CLN layer weights and the speaker feature table. Randomly initialize the speaker features of the unknown speaker, and input the obtained relevant data of the unknown speaker (frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy) into the pre-trained prosodic transfer-based speech synthesis model for fine-tuning training again (the fine-tuning training process is the same as the pre-training process), and converge quickly. The unknown speaker will be randomly initialized using a preset normal distribution. After approximately 5000 iterations of fine-tuning training, the model will converge.

[0125] Furthermore, in one embodiment, the method also includes testing and evaluating the customized speech synthesis model;

[0126] Ten unknown speakers from the test dataset were selected for testing. Ten native Chinese speakers were invited to conduct subjective evaluations, assessing the synthesized speech quality (MOS) and speaker similarity (SMOS). The evaluation results are shown in Table 2. An international standard 5-point scoring system was used, with 0 to 5 representing: Very poor (the emotion is completely unrelated to the target emotion, and the emotional expression is extremely poor); Very poor (the emotion is basically related to the target emotion, and the emotional expression is extremely poor); Average (the emotion is relatively related to the target emotion, and the emotional expression is relatively good); Good (the emotion is relatively related to the target emotion, and the emotional expression is sufficient); Excellent (the emotion is related to the target emotion, and the emotional expression is outstanding). Each 0.5 point represents one interval.

[0127] Table 2 Evaluation Results

[0128] Real Voice GT 4.12 4.23 This invention synthesizes speech. 3.95 4.05

[0129] As shown in Table 2, the speech synthesis method of the present invention was compared with real speech. The experiment proved that the speech synthesized by the speech synthesis method of the present invention is closer to real speech.

[0130] S400: Obtain the text of an unknown speaker, input the text of the unknown speaker into a customized speech synthesis model to obtain the predicted Mel spectrum of the unknown speaker; synthesize the speech of the unknown speaker based on the predicted Mel spectrum of the unknown speaker.

[0131] (1) The customized speech synthesis model predicts the Mel spectrum of the unknown speaker based on the input text of the unknown speaker, including:

[0132] The text from an unknown speaker is converted into a phoneme sequence. Based on this phoneme sequence, a phoneme feature table is queried to generate initial phoneme features. Speaker features, frame-level Mel spectrum, phoneme-level pitch, and phoneme-level energy are input into a prosodic encoder to obtain prosodic features. The prosodic features and initial phoneme features are input into a conditional encoder to obtain a first phoneme-level hidden feature. The first phoneme-level hidden feature is then input into an adaptive duration predictor, an adaptive pitch predictor, and an adaptive energy predictor to obtain predicted pitch, predicted duration, and predicted energy. The predicted pitch and predicted energy are added to the first phoneme-level hidden feature to generate a second phoneme-level hidden feature. Based on the predicted duration, the second phoneme-level hidden feature is expanded into a frame-level second hidden feature using a length normalizer. The frame-level second hidden feature and prosodic features are then input into a decoder to obtain a predicted Mel spectrum.

[0133] (2) Synthesizing unknown speaker speech based on predicted Mel spectrum of unknown speaker;

[0134] The predicted Mel spectrum of the unknown speaker is input into a pre-set vocoder to synthesize the unknown speaker's speech. The vocoder used in the speech synthesis stage is a pre-trained HiFi-GAN vocoder. This vocoder is not fine-tuned and is only used for the final speech synthesis, that is, to directly generate the speech waveform.

[0135] Example 2

[0136] like Figure 8 As shown, one embodiment of the present invention provides a prosodic transfer speech synthesis system, including a preprocessing module, a model building module, a training module, and a speech synthesis module:

[0137] The preprocessing module is used to acquire Chinese speech data with text annotations and speaker labels; and to preprocess the Chinese speech data and its corresponding text data to obtain phoneme sequences, frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy.

[0138] The model building module is used to build a speech synthesis model based on prosody transfer. The speaker label, phoneme sequence, frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy are input into the speech synthesis model based on prosody transfer to obtain speaker classification results, predicted Mel spectrum, predicted duration, predicted energy, and predicted pitch.

[0139] The training module is used to train the prosody transfer-based speech synthesis model based on the frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch and phoneme-level energy, as well as the speaker classification results, predicted Mel spectrum, predicted duration, predicted energy and predicted pitch, to obtain a customized speech synthesis model.

[0140] The speech synthesis module is used to acquire the text of an unknown speaker, input the text of the unknown speaker into the customized speech synthesis model to obtain the predicted Mel spectrum of the unknown speaker, and synthesize the speech of the unknown speaker based on the predicted Mel spectrum of the unknown speaker.

[0141] The model building module obtains the predicted Mel spectrum, predicted duration, predicted energy, and predicted pitch through the following operations:

[0142] Based on the speaker tags, the speaker feature table is queried to generate speaker features;

[0143] The speaker features, frame-level Mel spectrum, phoneme-level pitch, and phoneme-level energy are input into the prosodic encoder to obtain prosodic features and speaker classification results.

[0144] Based on the phoneme sequence, the phoneme feature table is queried to generate initial phoneme features; the prosodic features and the initial phoneme features are input into the conditional encoder to obtain the phoneme-level first hidden features;

[0145] The phoneme-level first hidden feature, prosodic feature, phoneme-level duration, phoneme-level pitch, and phoneme-level energy are input into the adaptive predictor group to obtain the prediction duration, prediction energy, prediction pitch, and frame-level second hidden feature.

[0146] The frame-level second hidden feature and prosodic feature are input into the decoder to obtain the predicted Mel spectrum.

[0147] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0150] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0151] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A prosodic transfer speech synthesis method, characterized in that, Includes the following steps: S100: Acquire Chinese speech data with text annotations and speaker labels; The Chinese speech data and its corresponding text data are preprocessed to obtain phoneme sequences, frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy. S200: Construct a speech synthesis model based on prosodic transfer. The prosodic transfer-based speech synthesis model includes a speaker feature table, a phoneme feature table, a prosodic encoder, a conditional encoder, an adaptive predictor group, and a decoder. The prosodic encoder includes a linear layer, a gradient inversion layer, and a speaker classifier. Input the speaker labels, phoneme sequences, frame-level Mel spectra, phoneme-level duration, phoneme-level pitch, and phoneme-level energy into the prosodic transfer-based speech synthesis model to obtain speaker classification results, predicted Mel spectra, predicted duration, predicted energy, and predicted pitch. S300: The prosody-transfer-based speech synthesis model is trained based on the frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch and phoneme-level energy, as well as the speaker classification results, predicted Mel spectrum, predicted duration, predicted energy and predicted pitch, to obtain a customized speech synthesis model; S400: Obtain the text of an unknown speaker, input the text of the unknown speaker into the customized speech synthesis model to obtain the predicted Mel spectrum of the unknown speaker; synthesize the speech of the unknown speaker based on the predicted Mel spectrum of the unknown speaker; In step S200, the predicted Mel spectrum, predicted duration, predicted energy, and predicted pitch are obtained through the following sub-steps: S210: Based on the speaker tags, query the speaker feature table to generate speaker features; S220: Input the speaker features, frame-level Mel spectrum, phoneme-level pitch and phoneme-level energy into the prosody encoder to obtain prosody features and speaker classification results; S230: Based on the phoneme sequence, query the phoneme feature table to generate initial phoneme features; input the prosodic features and the initial phoneme features into the conditional encoder to obtain the phoneme-level first hidden features; S240: Input the phoneme-level first hidden feature, prosodic feature, phoneme-level duration, phoneme-level pitch, and phoneme-level energy into the adaptive predictor group to obtain the prediction duration, prediction energy, prediction pitch, and frame-level second hidden feature. S250: Input the frame-level second hidden feature and prosodic feature into the decoder to obtain the predicted Mel spectrum.

2. The prosodic transfer speech synthesis method according to claim 1, characterized in that, The preprocessing in step S100 includes the following sub-steps: S110: Convert the text data into a phoneme sequence; S120: The phoneme sequence and the corresponding Chinese speech data are fed into the alignment model to obtain the phoneme-level duration; S130: Predict the pitch of the Chinese speech data to obtain frame-level pitch; The frame-level pitch is averaged based on the phoneme-level duration to obtain the phoneme-level pitch. S140: The speech waveform of the Chinese speech data is processed by short-time Fourier transform and Mel spectrum transformation to generate frame-level Mel spectrum and frame-level energy; the frame-level energy is averaged based on the phoneme-level duration to obtain the phoneme-level energy.

3. The prosodic transfer speech synthesis method according to claim 1, characterized in that, The adaptive predictor group in step S200 includes an adaptive duration predictor, an adaptive pitch predictor, an adaptive energy predictor, and a length normalizer.

4. The prosodic transfer speech synthesis method according to claim 1, characterized in that, Step S240 includes: The phoneme-level first hidden feature is input into the adaptive duration predictor, the adaptive pitch predictor, and the adaptive energy predictor, respectively, to obtain the predicted duration, predicted pitch, and predicted energy. The phoneme-level pitch and phoneme-level energy are added to the phoneme-level first hidden feature to generate the phoneme-level second hidden feature. Based on the phoneme-level duration, the phoneme-level second hidden feature is expanded into a frame-level second hidden feature using a length warper.

5. The prosodic transfer speech synthesis method according to claim 1, characterized in that, Step S300 includes the following sub-steps: S310: The speaker classification results, predicted Mel spectrum, predicted duration, predicted energy and pitch, as well as frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch and phoneme-level energy are used to calculate the loss through a loss function, and the prosody transfer-based speech synthesis model is pre-trained until convergence. S320: Fine-tune the pre-trained prosody transfer-based speech synthesis model to obtain a customized speech synthesis model.

6. The prosodic transfer speech synthesis method according to claim 5, characterized in that, The loss function in step S310 is expressed by the following formula: In the formula, Total loss; For Mel spectrum loss; The loss is for the adaptive predictor group; , , The weights for the adaptive pitch predictor, adaptive energy predictor, and duration predictor are all 1. For pitch loss; For energy loss; For time loss; This represents the cross-entropy loss.

7. The prosodic transfer speech synthesis method according to claim 6, characterized in that, The mean squared error is used to calculate the predicted Mel spectrum and the frame-level Mel spectrum to obtain the Mel spectrum loss; The mean squared error is used to calculate the predicted duration and phoneme-level duration to obtain the duration loss; The predicted energy and phoneme-level energy are calculated using the mean absolute error to obtain the energy loss; The predicted pitch and phoneme-level pitch are calculated using the mean absolute error to obtain the pitch loss; Calculate the cross-entropy loss based on the speaker classification results.

8. A prosodic transfer speech synthesis system, characterized in that, It includes a preprocessing module, a model building module, a training module, and a speech synthesis module: The preprocessing module is used to acquire Chinese speech data with text annotations and speaker labels; and to preprocess the Chinese speech data and its corresponding text data to obtain phoneme sequences, frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch, and phoneme-level energy. The model building module is used to construct a speech synthesis model based on prosodic transfer. The speaker labels, phoneme sequences, frame-level Mel spectra, phoneme-level duration, phoneme-level pitch, and phoneme-level energy are input into the prosodic transfer-based speech synthesis model to obtain speaker classification results, predicted Mel spectra, predicted duration, predicted energy, and predicted pitch. The prosodic transfer-based speech synthesis model includes a speaker feature table, a phoneme feature table, a prosodic encoder, a conditional encoder, an adaptive predictor group, and a decoder. The prosodic encoder includes a linear layer, a gradient inversion layer, and a speaker classifier. The training module is used to train the prosody transfer-based speech synthesis model based on the frame-level Mel spectrum, phoneme-level duration, phoneme-level pitch and phoneme-level energy, as well as the speaker classification results, predicted Mel spectrum, predicted duration, predicted energy and predicted pitch, to obtain a customized speech synthesis model. The speech synthesis module is used to acquire the text of an unknown speaker, input the text of the unknown speaker into the customized speech synthesis model to obtain the predicted Mel spectrum of the unknown speaker, and synthesize the speech of the unknown speaker based on the predicted Mel spectrum of the unknown speaker. The model building module obtains the predicted Mel spectrum, predicted duration, predicted energy, and predicted pitch through the following operations: Based on the speaker tags, the speaker feature table is queried to generate speaker features; The speaker features, frame-level Mel spectrum, phoneme-level pitch, and phoneme-level energy are input into the prosodic encoder to obtain prosodic features and speaker classification results. Based on the phoneme sequence, the phoneme feature table is queried to generate initial phoneme features; the prosodic features and the initial phoneme features are input into the conditional encoder to obtain the phoneme-level first hidden features; The phoneme-level first hidden feature, prosodic feature, phoneme-level duration, phoneme-level pitch, and phoneme-level energy are input into the adaptive predictor group to obtain the prediction duration, prediction energy, prediction pitch, and frame-level second hidden feature. The frame-level second hidden feature and prosodic feature are input into the decoder to obtain the predicted Mel spectrum.

Images