A personalized speech synthesis method and system with few-sample fine-tuning
By adding the coarse and fine-grained identification features of the speaker in personalized speech synthesis and fine-tuning parameters for specific levels, the problem of overfitting the model in small samples is solved, and high-quality personalized speech synthesis is achieved.
Patent Information
- Application Number
- CN202111628385.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-28
- Publication Date
- 2025-03-18
- Estimated Expiration
- 2041-12-28
AI Technical Summary
In personalized speech synthesis using few samples for fine-tuning of parameters, due to the small samples and large parameters, the model is prone to overfitting and the synthesis effect is poor.
By adding the coarse-grained identification features, fine-grained characteristics of the speaker, and fine-tuning parameters are only performed on the coarse-grained feature memory layer, fine-grained feature memory layer, speaker embedding layer, and language embedding layer, the fine-tuning parameters are reduced and the model is overfitted.
Only a very small number of target speaker samples are needed to obtain a high-quality personalized speech synthesis model, avoiding the overfitting problem caused by too few samples.
Smart Images
Figure CN114360486B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech processing technology, and in particular to a personalized speech synthesis method and system for fine-tuning with a small number of samples. Background Art
[0002] Speech synthesis is a technology that converts text information into speech information, that is, converts text information into any audible speech. It involves many disciplines such as acoustics, linguistics, and computer science. However, in personalized speech synthesis using a small number of samples for parameter fine-tuning, the model is often prone to overfitting due to the small number of samples and large parameters, resulting in poor synthesis effect. Summary of the invention
[0003] In order to solve the above technical problems, the present invention provides a personalized speech synthesis method and system with few-sample fine-tuning, which solves the problem of too large parameters and easy overfitting in personalized speech synthesis with few-sample fine-tuning. By adding the speaker's coarse-grained recognition features and fine-grained features and only fine-tuning the parameters of the coarse-grained feature memory layer, the fine-grained feature memory layer, the speaker embedding layer, and the language embedding layer, the fine-tuning parameters are greatly reduced, and the problem of model overfitting caused by too few samples is avoided. Only a very small number of target speaker samples are needed to obtain a high-quality personalized speech synthesis model.
[0004] In order to achieve the above technical effects, the present invention adopts the following technical solutions:
[0005] A personalized speech synthesis method and system for few-sample fine-tuning includes a training phase, a fine-tuning phase, and an inference phase. The training phase includes the following steps:
[0006] S11. Obtain multi-speaker training corpus, including audios of different speakers and corresponding texts;
[0007] S12. Extracting real spectral features from the audio obtained in step S11, normalizing the text obtained in step S11 and generating a text vector;
[0008] S13. construct a speaker embedding layer for the speaker in S12 and generate a speaker recognition vector, construct a language embedding layer for the text in S12 and generate a language recognition vector, construct a speaker fine-grained feature predictor for the real spectrum features in S12, and obtain fine-grained recognition features;
[0009] S14. Input the speaker recognition and language recognition vectors obtained in step S13 into the coarse-grained feature memory layer and output the coarse-grained recognition vector, and input the fine-grained recognition features obtained in step S13 into the fine-grained feature memory layer and output the fine-grained recognition vector;
[0010] S15. Train the speech synthesis model using the text vector of step 12, the coarse-grained recognition vector of step S14, the fine-grained recognition vector of step S14, and the real spectrum features extracted in step S12;
[0011] The fine-tuning phase includes the following steps:
[0012] S21. Obtain a small amount of training corpus of the target speaker, including the audio and corresponding text of the target speaker;
[0013] S22. The same as steps S12-S13, obtaining the target speaker's text vector, speaker recognition vector, language recognition vector, fine-grained recognition vector and true spectrum features from the training corpus of step S21;
[0014] S23. Fix all parameters except the speaker embedding layer, language embedding layer, coarse-grained feature memory layer, and fine-grained feature memory layer in the pre-trained speech synthesis model of step S15;
[0015] S24. Fine-tune the parameters of the speech synthesis model in step S23 using the text vector, speaker recognition vector, language recognition vector, fine-grained recognition vector, and true spectrum features extracted in step S22 to obtain a personalized speech synthesis model for the target speaker;
[0016] The inference phase consists of the following steps:
[0017] S31. normalize the synthesized text and generate a text vector;
[0018] S32. Generate a speaker recognition vector, a language recognition vector, and a fine-grained recognition feature from the speaker embedding layer, the language embedding layer, and the fine-grained feature predictor in the personalized speech synthesis model in step S23;
[0019] S33. Input the speaker recognition and language recognition vectors obtained in step S32 into the coarse-grained feature memory layer and output the coarse-grained recognition vector, and input the fine-grained recognition features obtained in step S32 into the fine-grained feature memory layer and output the fine-grained recognition vector;
[0020] S34. Input the text vector of step S31, the coarse-grained recognition vector of step S33, and the fine-grained recognition vector into the personalized speech synthesis model of step S23 to obtain a predicted spectrum;
[0021] S35. The predicted spectrum obtained in step S34 is passed through a vocoder to obtain synthesized audio;
[0022] Furthermore, in order to obtain training corpus of multiple speakers, step S11 further includes:
[0023] Acquiring multi-speaker training corpus including but not limited to a public multi-speaker speech synthesis training dataset or a self-recorded multi-speaker speech synthesis training dataset;
[0024] Furthermore, in order to extract training features, step S12 also includes:
[0025] The extracted real spectral features of speech include but are not limited to linear spectral features and Mel spectral features; the text is normalized to filter out illegal syllables, and legal input is segmented and POS tagged, etc., and the extracted comprehensive linguistic features are input into the prosody prediction model to obtain pause level annotations; the processed text is converted into ID representation, and a text vector is generated from the embedding vector space;
[0026] Furthermore, in order to extract speaker features, step S13 further includes:
[0027] Select the size of the speaker space and build a speaker embedding layer, and generate the corresponding speaker recognition vector according to the speaker ID; select the size of the language space and build a language embedding layer, and generate the language recognition vector of the corresponding text according to the text language tag; the fine-grained feature predictor includes but is not limited to convolutional neural network and recurrent neural network structure; the fine-grained features include but are not limited to phoneme prosodic features and syllable prosodic features;
[0028] Furthermore, in order to reduce fine-tuning parameters, step S14 further includes:
[0029] The coarse-grained feature memory layer and the fine-grained feature memory layer are linear structures, including but not limited to fully connected neural networks; the coarse-grained feature recognition vector and the fine-grained recognition vector output by the coarse-grained feature memory layer and the fine-grained feature memory layer need to expand the vector dimension to be consistent with the text vector;
[0030] Furthermore, in order to train the speech synthesis model, step S15 further includes:
[0031] The speech synthesis model adopts an encoder-decoder architecture. The output result of the text vector after the encoder is input into the decoder. At the same time, the coarse-grained recognition vector and the fine-grained recognition vector are added to the decoder, and the predicted spectrum is finally output. The speech synthesis model parameters are trained through the loss function of the real spectrum and the predicted spectrum.
[0032] Furthermore, in order to extract the target speaker training features, step S21 further includes:
[0033] The target speaker is not limited to the speakers in the training set, but can be any speaker;
[0034] Furthermore, in order to extract the target speaker training features, step S22 further includes:
[0035] Same as steps S12-S13 in the training phase;
[0036] Furthermore, in order to fine-tune the parameters, steps S23-S24 also include:
[0037] The output result of the text vector after the encoder is input into the decoder, and the coarse-grained recognition vector and the fine-grained recognition vector are added to the decoder, and the predicted spectrum is finally output; the parameters of the speaker embedding layer, language embedding layer, coarse-grained feature memory layer, and fine-grained feature memory layer in the speech synthesis model are fine-tuned through the loss function of the real spectrum and the predicted spectrum; other parameters in the speech synthesis model are not involved in the training;
[0038] Furthermore, in order to synthesize personalized audio for the synthesized text, steps S31-S34 also include:
[0039] The parameters of the personalized speech synthesis model in the inference phase are obtained in the training phase, and the network structure is consistent; the processing method of the text to be synthesized in the inference phase is consistent with that in the training phase;
[0040] Furthermore, in order to convert the predicted spectral features into synthetic audio, step S35 further includes:
[0041] The vocoders used include but are not limited to WavNET, WavRNN, and MelGAN.
[0042] The present invention also proposes a personalized speech synthesis system with few-sample fine-tuning, comprising:
[0043] The text processing module is used to normalize the text and generate text vectors;
[0044] Specifically, the text processing module is used to normalize the text, filter out illegal syllables, perform word segmentation and part-of-speech tagging on legal input, and input the extracted comprehensive linguistic features into the prosody prediction model to obtain pause level annotations; convert the processed text into ID representation, and generate text vectors from the embedded vector space.
[0045] A speaker recognition module, used to generate a coarse-grained recognition vector and a fine-grained recognition vector for the speaker;
[0046] Specifically, the size of the speaker space is selected and a speaker embedding layer is constructed, and a corresponding speaker recognition vector is generated according to the speaker ID; the size of the language space is selected and a language embedding layer is constructed, and a language recognition vector of the corresponding text is generated according to the text language tag; and a fine-grained recognition vector is obtained through a fine-grained feature predictor.
[0047] Speaker memory module, used to memorize speaker identification features;
[0048] Specifically, in order to reduce parameters during fine-tuning training, the speaker recognition vector and the language recognition vector are output as coarse-grained recognition vectors through the coarse-grained feature memory layer, and the fine-grained recognition features are output as fine-grained recognition vectors through the fine-grained feature memory layer.
[0049] The parameter fine-tuning module uses a small amount of target speaker corpus to fine-tune the parameters of the speaker memory module;
[0050] Specifically, the parameters of the speaker embedding layer, language embedding layer, coarse-grained feature memory layer, and fine-grained feature memory layer in the speech synthesis model are fine-tuned; other parameters in the speech synthesis model do not participate in the training.
[0051] A spectrum prediction module, used to generate a predicted spectrum of the target speaker;
[0052] Specifically, the speech synthesis model adopts an encoder-decoder architecture, inputs the output result of the text vector after the encoder into the decoder, and adds coarse-grained recognition vectors and fine-grained recognition vectors to the decoder, and finally outputs the predicted spectrum.
[0053] The vocoder module generates synthesized audio from an input spectrum.
[0054] Optionally, the adopted vocoder includes but is not limited to WavNET, WavRNN, and MelGAN.
[0055] Compared with the prior art, the present invention has the following beneficial effects: by adding the speaker's coarse-grained recognition features and fine-grained features and only fine-tuning the parameters of the coarse-grained feature memory layer, the fine-grained feature memory layer, the speaker embedding layer, and the language embedding layer, the fine-tuning parameters are greatly reduced, and the model overfitting problem caused by too few samples is avoided. Only a very small number of target speaker samples are needed to obtain a high-quality personalized speech synthesis model. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 The figure is a flowchart of the personalized speech synthesis with fine-tuning of a small number of samples according to an embodiment of the present invention.
[0057] Figure 2 This is a flowchart of training, fine-tuning, and inference according to an embodiment of the present invention. DETAILED DESCRIPTION
[0058] The embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the embodiments described here.
[0059] Example 1
[0060] The method for personalized speech synthesis with few-sample fine-tuning described in the embodiment of the present invention is as follows: Figure 1 As shown, the training phase includes the following steps:
[0061] S11. Obtain multi-speaker training corpus, including audios of different speakers and corresponding texts;
[0062] Optionally, the multi-speaker training corpus obtained includes but is not limited to a public multi-speaker speech synthesis training dataset or a self-recorded multi-speaker speech synthesis training dataset.
[0063] S12. Extracting real spectral features from the audio obtained in step S11, normalizing the text obtained in step S11 and generating a text vector;
[0064] Optionally, the extracted real spectral features of the speech include but are not limited to linear spectral features and Mel spectral features; the text is normalized to filter out illegal syllables, and legal input is segmented and tagged with parts of speech, etc., and the extracted comprehensive linguistic features are input into a prosody prediction model to obtain pause level annotations; the processed text is converted into an ID representation, and a text vector is generated from the embedding vector space.
[0065] For example, "￥#% is sold at 180 yuan per box." In this example, illegal characters such as "￥#%" are first filtered out, and the Arabic numeral "180" is converted into the corresponding pronunciation Chinese characters according to the situation. Then, the legal text to be synthesized is input into the prosody prediction model to obtain pause level annotations, and then the Chinese characters are converted into phoneme markers to obtain "sh ou4 pp0 j ia4 pp2 i4pp0 b ai3 pp0 b a1 pp0 sh iii2 pp0 van2 pp1 i1 pp0 h e2 pp4", find the corresponding ID through the phoneme set and get [217,206,209,188,132,211,127,209,93,75,209,93,68,209,217,160,209,269,210,124,209,123,98,213]. Extract the text vector from the embedding vector space according to the ID. Take the batch size as 16 and the hidden variable dimension as 256, then the text vector dimension is (16, 24, 256).
[0066] S13. construct a speaker embedding layer for the speaker in S12 and generate a speaker recognition vector, construct a language embedding layer for the text in S12 and generate a language recognition vector, construct a speaker fine-grained feature predictor for the real spectrum features in S12, and obtain fine-grained recognition features;
[0067] Optionally, the size of the speaker space is selected and a speaker embedding layer is constructed, and a corresponding speaker recognition vector is generated according to the speaker ID; the size of the language space is selected and a language embedding layer is constructed, and a language recognition vector of the corresponding text is generated according to the text language tag; the fine-grained feature predictor includes but is not limited to a convolutional neural network and a recurrent neural network structure; the fine-grained features include but are not limited to phoneme prosodic features and syllable prosodic features.
[0068] For example, there are 500 speakers used for training here, and the dimension of the speaker recognition vector is 256, so the size of the speaker space is (500, 256), and the corresponding speaker recognition vectors are extracted from them according to the speaker ID, with a dimension of (1, 256); the data used for training here contains two languages including Chinese and English, and the dimension of the language recognition vector is 256, so the size of the language space is (2, 256), and the language recognition vector is extracted from it according to the ID corresponding to the language text, with a dimension of (24, 256); the fine-grained recognition vector dimension obtained from the fine-grained feature identifier is (24, 256).
[0069] S14. Input the speaker recognition and language recognition vectors obtained in step S13 into the coarse-grained feature memory layer and output the coarse-grained recognition vector, and input the fine-grained recognition features obtained in step S13 into the fine-grained feature memory layer and output the fine-grained recognition vector;
[0070] It is understandable that the coarse-grained feature memory layer and the fine-grained feature memory layer are linear structures, including but not limited to fully connected neural networks; the coarse-grained feature recognition vectors and fine-grained recognition vectors output by the coarse-grained feature memory layer and the fine-grained feature memory layer need to expand the vector dimension to be consistent with the text vector.
[0071] S15. Train the speech synthesis model using the text vector of step 12, the coarse-grained recognition vector of step S14, the fine-grained vector of step S14 and the real spectrum features extracted in step S12.
[0072] Optionally, the speech synthesis model adopts an encoder-decoder architecture, and the output result of the text vector after the encoder is input into the decoder. At the same time, coarse-grained recognition vectors and fine-grained recognition vectors are added to the decoder, and finally a predicted spectrum is output; the speech synthesis model parameters are trained through the loss function of the real spectrum and the predicted spectrum.
[0073] Specifically, the model training process is as follows: Figure 2 shown.
[0074] The fine-tuning phase includes the following steps:
[0075] S21. Obtain a small amount of training corpus of the target speaker, including the audio and corresponding text of the target speaker;
[0076] Optionally, the target speaker is not limited to speakers in the training set, but can be any speaker;
[0077] S22. The same as steps S12-S13, obtaining the target speaker's text vector, speaker recognition vector, language recognition vector, fine-grained recognition vector and true spectrum features from the training corpus of step S21;
[0078] It can be understood that it is the same as steps S12-S13 in the training phase;
[0079] S23. Fix all parameters except the speaker embedding layer, language embedding layer, coarse-grained feature memory layer, and fine-grained feature memory layer in the pre-trained speech synthesis model of step S15;
[0080] S24. Fine-tune the parameters of the speech synthesis model in step S23 using the text vector, speaker recognition vector, language recognition vector, fine-grained recognition vector, and true spectrum features extracted in step S22 to obtain a personalized speech synthesis model for the target speaker;
[0081] It can be understood that the output result of the text vector after the encoder is input into the decoder, and the coarse-grained recognition vector and the fine-grained recognition vector are added to the decoder, and finally the predicted spectrum is output; the parameters of the speaker embedding layer, language embedding layer, coarse-grained feature memory layer, and fine-grained feature memory layer in the speech synthesis model are fine-tuned through the loss function of the real spectrum and the predicted spectrum; other parameters in the speech synthesis model are not involved in the training.
[0082] Specifically, the fine-tuning stage is as follows: Figure 2 As shown, the red boxes are the parameters that need to be fine-tuned.
[0083] The inference phase consists of the following steps:
[0084] S31. normalize the synthesized text and generate a text vector;
[0085] S32. Generate a speaker recognition vector, a language recognition vector, and a fine-grained recognition feature from the speaker embedding layer, the language embedding layer, and the fine-grained feature predictor in the personalized speech synthesis model in step S23;
[0086] S33. Input the speaker recognition and language recognition vectors obtained in step S32 into the coarse-grained feature memory layer and output the coarse-grained recognition vector, and input the fine-grained recognition features obtained in step S32 into the fine-grained feature memory layer and output the fine-grained recognition vector;
[0087] S34. Input the text vector of step S31, the coarse-grained recognition vector of step S33, and the fine-grained recognition vector into the personalized speech synthesis model of step S23 to obtain a predicted spectrum;
[0088] It can be understood that the parameters of the personalized speech synthesis model in the reasoning stage are obtained in the training stage, and the network structure is consistent; the processing method of the text to be synthesized in the reasoning stage is consistent with that in the training stage;
[0089] S35. The predicted spectrum obtained in step S34 is passed through a vocoder to obtain synthesized audio;
[0090] Optionally, the adopted vocoder includes but is not limited to WavNET, WavRNN, and MelGAN.
[0091] Specifically, the reasoning process is as follows Figure 2 shown.
[0092] Through the personalized speech synthesis method with few sample fine-tuning provided in this embodiment 1, by adding the speaker's coarse-grained recognition features and fine-grained features and only fine-tuning the parameters of the coarse-grained feature memory layer, the fine-grained feature memory layer, the speaker embedding layer, and the language embedding layer, a high-quality personalized speech synthesis model can be obtained with only a very small amount of target speaker corpus.
[0093] Example 2
[0094] A personalized speech synthesis system with few-sample fine-tuning according to an embodiment of the present invention includes:
[0095] The text processing module is used to normalize the text and generate text vectors;
[0096] Optionally, the text is normalized to filter out illegal syllables, and legal input is segmented and tagged with parts of speech, etc. The extracted comprehensive linguistic features are input into a prosody prediction model to obtain pause level annotations; the processed text is converted into an ID representation, and a text vector is generated from the embedding vector space.
[0097] A speaker recognition module, used to generate a coarse-grained recognition vector and a fine-grained recognition vector for the speaker;
[0098] Optionally, select the size of the speaker space and construct a speaker embedding layer, and generate a corresponding speaker recognition vector based on the speaker ID; select the size of the language space and construct a language embedding layer, and generate a language recognition vector for the corresponding text based on the text language tag; and obtain a fine-grained recognition vector through a fine-grained feature predictor.
[0099] Speaker memory module, used to memorize speaker identification features;
[0100] It is understandable that in order to reduce the parameters during fine-tuning training, the speaker recognition vector and the language recognition vector are output as coarse-grained recognition vectors through the coarse-grained feature memory layer, and the fine-grained recognition features are output as fine-grained recognition vectors through the fine-grained feature memory layer.
[0101] The parameter fine-tuning module uses a small amount of target speaker corpus to fine-tune the parameters of the speaker memory module;
[0102] It can be understood that the parameters of the speaker embedding layer, language embedding layer, coarse-grained feature memory layer, and fine-grained feature memory layer in the speech synthesis model are fine-tuned; other parameters in the speech synthesis model do not participate in the training.
[0103] A spectrum prediction module, used to generate a predicted spectrum of the target speaker;
[0104] Optionally, the speech synthesis model adopts an encoder-decoder architecture, inputs the output result of the text vector after the encoder into the decoder, and adds coarse-grained recognition vectors and fine-grained recognition vectors to the decoder, and finally outputs a predicted spectrum.
[0105] The vocoder module generates synthesized audio from an input spectrum.
[0106] Optionally, the adopted vocoder includes but is not limited to WavNET, WavRNN, and MelGAN.
[0107] The personalized speech synthesis system with few-sample fine-tuning provided in this embodiment 2, by adding a speaker memory module and a parameter fine-tuning module, only a very small amount of target speaker corpus is needed to obtain a high-quality personalized speech synthesis model.
[0108] Although the present invention is described herein with reference to the illustrative embodiments of the present invention, the above embodiments are only preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and embodiments, which will fall within the scope and spirit of the principles disclosed in this application.
Claims
1. A personalized speech synthesis method with few-sample fine-tuning, characterized in that: It includes the training phase, fine-tuning phase and inference phase. The training phase includes the following steps: S11. Obtain multi-speaker training corpus, including audios of different speakers and corresponding texts; S12. Extracting real spectral features from the audio obtained in step S11, normalizing the text obtained in step S11 and generating a text vector; S13. construct a speaker embedding layer for the speaker in S12 and generate a speaker recognition vector, construct a language embedding layer for the text in S12 and generate a language recognition vector, construct a speaker fine-grained feature predictor for the real spectrum features in S12, and obtain fine-grained recognition features; S14. Input the speaker recognition and language recognition vectors obtained in step S13 into the coarse-grained feature memory layer and output the coarse-grained recognition vector, and input the fine-grained recognition features obtained in step S13 into the fine-grained feature memory layer and output the fine-grained recognition vector; S15. Train the speech synthesis model using the text vector, the coarse-grained recognition vector, the fine-grained recognition vector and the real spectrum feature to obtain a pre-trained speech synthesis model; The fine-tuning phase includes the following steps: S21. Obtain a small amount of training corpus of the target speaker, including the audio and corresponding text of the target speaker; S22. The same as steps S12-S13, obtaining the target speaker's text vector, speaker recognition vector, language recognition vector, fine-grained recognition vector and true spectrum features from the training corpus of step S21; S23. Fix all parameters except the speaker embedding layer, language embedding layer, coarse-grained feature memory layer, and fine-grained feature memory layer in the pre-trained speech synthesis model of step S15; S24. Fine-tune the parameters of the speech synthesis model in step S23 using the text vector, speaker recognition vector, language recognition vector, fine-grained recognition vector, and true spectrum features extracted in step S22 to obtain a personalized speech synthesis model for the target speaker; The inference phase consists of the following steps: S31. normalize the synthesized text and generate a text vector; S32. Generate a speaker recognition vector, a language recognition vector, and a fine-grained recognition feature from the speaker embedding layer, the language embedding layer, and the fine-grained feature predictor in the personalized speech synthesis model in step S23; S33. Input the speaker recognition and language recognition vectors obtained in step S32 into the coarse-grained feature memory layer and output the coarse-grained recognition vector, and input the fine-grained recognition features obtained in step S32 into the fine-grained feature memory layer and output the fine-grained recognition vector; S34. Input the text vector of step S31, the coarse-grained recognition vector of step S33, and the fine-grained recognition vector into the personalized speech synthesis model of step S23 to obtain a predicted spectrum; S35. The predicted spectrum obtained in step S34 is passed through a vocoder to obtain synthesized audio.
2. The personalized speech synthesis method with few-sample fine-tuning according to claim 1, characterized in that: The multi-speaker training corpus obtained in step S11 includes but is not limited to a public multi-speaker speech synthesis training data set or a self-recorded multi-speaker speech synthesis training data set.
3. The personalized speech synthesis method with few-sample fine-tuning according to claim 1, characterized in that: The real spectral features of the speech extracted in step S12 include but are not limited to linear spectral features and Mel spectral features; the text is normalized to filter out illegal syllables, perform word segmentation and part-of-speech tagging on legal input, and input the extracted comprehensive linguistic features into a prosody prediction model to obtain pause level annotations; the processed text is converted into an ID representation, and a text vector is generated from the embedded vector space.
4. The personalized speech synthesis method with few-sample fine-tuning according to claim 1, characterized in that: Step S13 also includes: The size of the speaker space is selected and a speaker embedding layer is constructed, and a corresponding speaker recognition vector is generated according to the speaker ID; the size of the language space is selected and a language embedding layer is constructed, and a language recognition vector of the corresponding text is generated according to the text language tag; the fine-grained feature predictor includes but is not limited to a convolutional neural network and a recurrent neural network structure; the fine-grained features include but are not limited to phoneme prosodic features and syllable prosodic features.
5. The personalized speech synthesis method with few-sample fine-tuning according to claim 1, characterized in that: Step S14 also includes: the coarse-grained feature memory layer and the fine-grained feature memory layer are linear structures, including but not limited to fully connected neural networks; the coarse-grained feature recognition vectors and fine-grained recognition vectors output by the coarse-grained feature memory layer and the fine-grained feature memory layer need to expand the vector dimension to be consistent with the text vector.
6. The personalized speech synthesis method with few-sample fine-tuning according to claim 1, characterized in that: Step S15 also includes: the speech synthesis model adopts an encoder-decoder architecture, the output result of the text vector after the encoder is input into the decoder, and the coarse-grained recognition vector and the fine-grained recognition vector are added to the decoder, and finally the predicted spectrum is output; the speech synthesis model parameters are trained through the loss function of the real spectrum and the predicted spectrum.
7. The personalized speech synthesis method with few-sample fine-tuning according to claim 1, characterized in that: The target speaker in step S21 may be a speaker in the training set or any speaker.
8. The personalized speech synthesis method with few-sample fine-tuning according to claim 1, characterized in that: The personalized speech synthesis model parameters in the reasoning stage are obtained in the training stage, and the network structure is consistent; the processing method of the text to be synthesized in the reasoning stage is consistent with that in the training stage.
9. The personalized speech synthesis method with few-sample fine-tuning according to claim 1, characterized in that: The vocoder is obtained through pre-training; the network structure includes but is not limited to WaveNET, WavRNN, and MelGAN.
10. A personalized speech synthesis system with few-sample fine-tuning, characterized in that: A personalized speech synthesis method for implementing a few-sample fine-tuning as described in any one of claims 1 to 9, the system comprising: The text processing module is used to normalize the text and generate text vectors; A speaker recognition module, used to generate a coarse-grained recognition vector and a fine-grained recognition vector for the speaker; Speaker memory module, used to memorize speaker identification features; The parameter fine-tuning module uses a small amount of target speaker corpus to fine-tune the parameters of the speaker memory module; A spectrum prediction module, used to generate a predicted spectrum of the target speaker; The vocoder module generates synthesized audio from an input spectrum.
Citation Information
Patent Citations
Tone and accent controllable Chinese and English speech synthesis method and device thereof
CN113362805A