Speech synthesis method and system

Abstract

The present invention provides a speech synthesis method, comprising: obtaining target spectral features of a target speaker; processing the target spectral features based on a recognition model to obtain a speech recognition vector for the target speaker; processing the target spectral features based on a prediction model to obtain target acoustic prediction vectors at the phoneme, syllable, and sentence levels; processing the text to be synthesized based on an encoding model to obtain a text encoding vector; processing the phoneme, syllable, and sentence-level target acoustic prediction vectors and the text encoding vector based on a decoding model to obtain predicted spectral features; and processing the predicted spectral features based on a vocoding model to obtain synthesized audio. This method can improve sound quality and naturalness in the task of few-shot personalized speech synthesis.

Description

Technical Field

[0001] The present invention relates to the technical field of speech synthesis, and in particular to a speech synthesis method and system. Background Art

[0002] Speech synthesis is a technology that converts text into audible speech. It involves multiple disciplines, including acoustics, linguistics, and computer science. However, personalized speech synthesis with a small sample size is both a hot topic and a challenge. Limited sample size often results in low sound quality and similarity using existing acoustic models. Summary of the Invention

[0003] The purpose of the present invention is to provide a speech synthesis method and system, in order to solve the technical problems existing in the background technology.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A speech synthesis method, comprising:

[0006] Obtain target spectrum features of the target speaker;

[0007] Processing the target spectral features based on a recognition model to obtain a speech recognition vector of the target speaker;

[0008] Processing the target spectral features based on the prediction model to obtain phoneme-level, syllable-level, and sentence-level target acoustic prediction vectors respectively;

[0009] Process the synthesized text based on the encoding model to obtain a text encoding vector;

[0010] Processing the phoneme-level, syllable-level, and sentence-level target acoustic prediction vectors and the text encoding vector based on a decoding model to obtain predicted spectral features;

[0011] The predicted spectral features are processed based on a vocoding model to obtain synthesized audio.

[0012] In some embodiments, the recognition model is a machine learning model, and the training samples of the recognition model include audio of multiple different speakers and corresponding texts.

[0013] In some embodiments, the target spectrum feature includes at least one of the following: a linear spectrum feature and a Mel spectrum feature.

[0014] In some embodiments, the text to be synthesized is a pre-processed synthesized text, and the pre-processing includes screening out illegal syllables, performing word segmentation, and part-of-speech tagging.

[0015] In some embodiments, the pre-processing further comprises:

[0016] Extracting comprehensive linguistic features from the synthesized text, and inputting the extracted comprehensive linguistic features into a prosody prediction model to obtain pause level annotations;

[0017] The Chinese characters of the synthesized text are converted into corresponding pinyin phonemes.

[0018] In some embodiments, processing the target spectrum feature further includes: obtaining an alignment result between the spectrum feature and the text to be synthesized by a forced alignment algorithm;

[0019] According to the alignment results, the target spectral features are averaged for each frame at the phoneme level and the syllable level;

[0020] The coding model includes a phoneme-level coding model, a syllable-level coding model, and a sentence-level coding model;

[0021] The output dimension of the syllable-level encoding model is expanded according to the phoneme step size and is consistent with the phoneme step size.

[0022] In some embodiments, the phoneme-level, syllable-level, sentence-level target acoustic prediction vectors and the text encoding vector are first processed as follows before being input into the decoding model:

[0023] The speech recognition vector, the text encoding vector, and the phoneme-level, syllable-level, and sentence-level target acoustic prediction vectors are added together, and the added vectors are expanded based on the alignment results between the spectral features and the text to be synthesized.

[0024] In some embodiments, the decoding model includes but is not limited to CNN and Transformer.

[0025] In some embodiments, the vocoding model includes but is not limited to WaveNET, WavRNN, and MelGAN.

[0026] At the same time, the present invention also discloses a speech synthesis system, comprising:

[0027] An acquisition module, used to obtain target spectrum features of a target speaker;

[0028] A recognition module, configured to process the target spectral features based on a recognition model to obtain a speech recognition vector of the target speaker;

[0029] A prediction module, configured to process the target spectral features based on a prediction model to obtain target acoustic prediction vectors at the phoneme level, syllable level, and sentence level respectively;

[0030] An encoding module, used to process the synthesized text based on the encoding model to obtain a text encoding vector;

[0031] A decoding module, configured to process the phoneme-level, syllable-level, and sentence-level target acoustic prediction vectors and the text encoding vector based on a decoding model to obtain predicted spectral features;

[0032] The vocoding module is used to process the predicted spectral features based on the vocoding model to obtain synthesized audio.

[0033] At the same time, the present invention also discloses a speech synthesis device, which includes a processor and a memory; the memory is used to store instructions, and when the instructions are executed by the processor, the device implements any of the above-mentioned speech synthesis methods.

[0034] At the same time, the present invention also discloses a computer-readable storage medium, which stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer runs any of the above-mentioned speech synthesis methods.

[0035] Beneficial effects

[0036] Compared with the prior art, the present invention has the following significant advantages:

[0037] The solution of the present invention adds an adaptive acoustic information prediction module to the multi-speaker speech synthesis model, predicts the acoustic coding information of the input text at the phoneme level, syllable level, and sentence level, obtains the target predicted spectrum through the decoding layer of the input speaker recognition vector, text coding vector and predicted acoustic coding vector, and finally converts the target predicted spectrum into target synthesized audio through the vocoder, thereby improving the sound quality and similarity of small-sample personalized speech synthesis.

[0038] At the same time, by adding the phoneme-level, syllable-level, and sentence-level acoustic feature prediction vectors obtained by the adaptive acoustic feature predictor to the multi-speaker speech synthesis model, the sound quality and naturalness are improved in the task of few-sample personalized speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a schematic diagram of a speech synthesis system involved in this embodiment;

[0040] Figure 2 is a flowchart of the speech synthesis method involved in this embodiment;

[0041] Figure 3 This is a training flow chart of a model according to an embodiment of the present invention;

[0042] Figure 4 This is an application flow chart of the model of the embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0044] On the contrary, this application covers any alternatives, modifications, equivalents, and solutions made within the spirit and scope of this application as defined by the claims. Furthermore, to facilitate a better understanding of this application, certain specific details are described in detail below in the detailed description of this application. Those skilled in the art will be able to fully understand this application without these details.

[0045] The following will be combined Figure 1-2 A speech synthesis method according to an embodiment of the present application is described in detail. It should be noted that the following embodiments are only used to explain the present application and do not constitute a limitation of the present application.

[0046] Example 1

[0047] like Figure 1 As shown, a speech synthesis method is used to achieve speech synthesis based on the joint action of multiple models or multiple layers.

[0048] This example mainly describes the training method of the model:

[0049] First, obtain the training corpus, which contains the speech of multiple speakers, as well as the audio of different speakers and the corresponding text.

[0050] Specifically, obtaining multi-speaker training corpus includes but is not limited to a public multi-speaker speech synthesis training dataset or a self-recorded multi-speaker speech synthesis training dataset.

[0051] Then, feature extraction is performed on the acquired training corpus to extract its true spectrum features, and a speaker recognition vector is extracted from the true spectrum features using a pre-trained speaker recognition model.

[0052] Optionally, the extracted real speech spectrum features include but are not limited to linear spectrum features and Mel spectrum features; the pre-trained speaker recognition model includes but is not limited to current mainstream speaker recognition models, such as GE2E, I-Vector, etc.

[0053] Then, the text corresponding to the speech of different speakers is preprocessed, and the text encoding vector is obtained through the text encoding model.

[0054] Optionally, preprocessing may include screening out illegal syllables, performing word segmentation and part-of-speech tagging on legal input, and inputting the extracted comprehensive linguistic features into a prosody prediction model to obtain pause level annotations; converting Chinese characters into corresponding pinyin phonemes; wherein the text encoding layer includes but is not limited to a CNN network structure and a Transformer network structure.

[0055] The real spectral features are then input into the phoneme-level, syllable-level, and sentence-level acoustic coding layers respectively to obtain phoneme-level, syllable-level, and sentence-level acoustic coding vectors. At the same time, the real spectral features are passed through the phoneme-level, syllable-level, and sentence-level acoustic predictors respectively to obtain phoneme-level, syllable-level, and sentence-level acoustic prediction vectors.

[0056] Then the speaker recognition vector, text encoding vector, and phoneme-level, syllable-level, and sentence-level acoustic prediction vectors are input into the decoding layer to obtain the predicted spectral features.

[0057] Preferably, in order to adaptively obtain acoustic information at different levels, the following operations may be performed:

[0058] The alignment results between the spectral features and the text are obtained through a forced alignment algorithm. Before the spectral features are input into the phoneme-level and syllable-level acoustic coding layers, the spectral features are averaged for each frame at the phoneme level and syllable level respectively according to the alignment results between the spectral features and the text. The output dimension of the syllable-level coding layer is expanded according to the phoneme step size and is consistent with the phoneme step size.

[0059] Among them, the phoneme-level, syllable-level, and sentence-level acoustic coding layers include but are not limited to CNN network structures; the last layer of the sentence-level acoustic coding layer network structure is a pooling layer, and the output dimension of the sentence-level acoustic coding layer is expanded according to the phoneme step length and is consistent with the phoneme step length; the acoustic predictor is consistent with the acoustic coding layer network structure; the input of the acoustic predictor is a text encoding vector.

[0060] Furthermore, in order to obtain the predicted spectrum features, the following operations are also included:

[0061] By adding the speaker recognition vector, text encoding vector, and phoneme-level, syllable-level, and sentence-level acoustic encoding vectors, the added vector is expanded based on the alignment results between the spectral features and the text, and then input into the decoding layer; the decoding layer network structure includes but is not limited to CNN, Transformer and other network structures; the predicted spectral features are consistent with the actual spectral features.

[0062] Finally, the parameters of the text encoding layer, acoustic encoding layer, and decoding layer are trained through the loss function between the real spectral features and the predicted spectral features, and the parameters of the acoustic predictor are trained through the loss function between the acoustic encoding vector and the acoustic prediction vector.

[0063] Specifically, the loss function between the true spectral features and the predicted spectral features, and the loss function between the acoustic coding vector and the acoustic prediction vector include but are not limited to the mean square error loss function; in the early stage of training, the acoustic predictor parameters and the gradient of this part are fixed, and in the later stage of training, the entire network is trained together.

[0064] Example 2

[0065] This example mainly explains how to apply the trained model, as follows:

[0066] First, the text to be synthesized is normalized and the real spectrum features of the target speaker audio sample are extracted.

[0067] Secondly, the target spectral features are input into the pre-trained speaker recognition model to obtain the target speaker recognition vector. At the same time, the target spectral features are input into the trained acoustic predictor to obtain the phoneme-level, syllable-level, and sentence-level target acoustic prediction vectors.

[0068] Then, the text to be synthesized is input into the trained encoding layer to obtain the text encoding vector;

[0069] Next, the obtained phoneme-level, syllable-level, sentence-level acoustic prediction vectors and text encoding vectors are input into the trained decoding layer to obtain the predicted spectral features;

[0070] Among them, the parameters of the speech synthesis model are obtained in the training phase, and the network structure is consistent; the text processing method in the inference phase is consistent with the training phase; the training phase uses the alignment results of the real spectral features and the text to expand the text coding vector, acoustic coding vector, and speaker recognition vector, while the inference phase uses the prediction results of the duration predictor to expand the text coding vector, acoustic coding vector, and speaker recognition vector; the acoustic coding layer is not used in the inference phase.

[0071] Finally, the predicted spectral features are passed through a pre-trained vocoder to generate synthetic audio.

[0072] Among them, the adopted vocoders include but are not limited to WavNET, WavRNN, and MelGAN.

[0073] Example 3

[0074] A speech synthesis system, comprising:

[0075] An acquisition module, used to obtain target spectrum features of a target speaker;

[0076] A recognition module, configured to process the target spectral features based on a recognition model to obtain a speech recognition vector of the target speaker;

[0077] A prediction module, configured to process the target spectral features based on a prediction model to obtain target acoustic prediction vectors at the phoneme level, syllable level, and sentence level respectively;

[0078] An encoding module, used to process the synthesized text based on the encoding model to obtain a text encoding vector;

[0079] A decoding module, configured to process the phoneme-level, syllable-level, and sentence-level target acoustic prediction vectors and the text encoding vector based on a decoding model to obtain predicted spectral features;

[0080] The vocoding module is used to process the predicted spectral features based on the vocoding model to obtain synthesized audio.

[0081] Implementation 4

[0082] A method for speech synthesis, specifically including a model training method and a model application method, wherein, Figure 3 and Figure 4 As shown in the figure, the overall architecture of the model adopts an encoding and decoding network framework; the text encoding layer and decoding layer adopt a Transformer network structure, the acoustic encoding layer and acoustic predictor adopt a CNN network structure, and finally the predicted spectral features are output through the decoding layer.

[0083] like Figure 2 As shown, the model training method (i.e., the training phase) includes the following steps:

[0084] S11. Obtain multi-speaker training corpus, including audio of different speakers and corresponding text;

[0085] Optionally, the multi-speaker training corpus obtained includes but is not limited to a public multi-speaker speech synthesis training dataset or a self-recorded multi-speaker speech synthesis training dataset.

[0086] S12. Extracting true spectral features from the audio obtained in step S11, and extracting a speaker recognition vector from the true spectral features using a pre-trained speaker recognition model;

[0087] Optionally, the extracted real speech spectrum features include but are not limited to linear spectrum features and Mel spectrum features; the pre-trained speaker recognition model includes but is not limited to current mainstream speaker recognition models, such as GE2E, I-Vector, etc.

[0088] For example, 80-dimensional Mel spectrum features are extracted from the acquired audio, and a multi-speaker recognition model is trained using GE2E loss. The trained speaker recognition model is used to extract speaker recognition vectors from the Mel spectrum features.

[0089] S13. After normalizing the text obtained in step S11, a text encoding vector is obtained through a text encoding layer;

[0090] Optionally, the Chinese text is normalized to filter out illegal syllables, and legal input is segmented and tagged with parts of speech, etc., and the extracted comprehensive linguistic features are input into the prosody prediction model to obtain pause level annotations; Chinese characters are converted into corresponding pinyin phonemes; optionally, the text encoding layer includes but is not limited to a CNN network structure and a Transformer network structure.

[0091] For example, "Jiuzhitang Ejiao Blood-Replenishing Granules, priced at 180 yuan per box." In this example, illegal syllables such as "," and "." are first filtered out, and the Arabic numeral "180" is converted into the corresponding pronunciation Chinese characters according to the situation. Then, the legal text to be synthesized is input into the rhythm prediction model to obtain pause level annotations, and then the Chinese characters are converted into pinyin tags, that is, ['jiu3','zhi1','tang2','#2','e1','jiao1','#1','bu3','xue4','#2','ke1','li4','#1','shou4','jia4','#2','yi1','bai3','ba1','shi2','yuan2','#1','yi1','he2','#4'], where "#" is a pause tag; then the pinyin tags are converted into phoneme tags to obtain "j iou3 zh iii1 t ang2 #2 e1j iao1 #1 b u3 x ve4 #2 k e1 l i4 #1 sh ou4 j ia4 #2 i1 b ai3 b a1 sh iii2van2 #1 i1 h e2#4”; The text encoding layer adopts the CNN structure, and the normalized text sequence is input into the text encoding layer to obtain the text encoding vector.

[0092] S14. The real spectral features extracted in step S12 are input into the phoneme-level, syllable-level, and sentence-level acoustic coding layers to obtain phoneme-level, syllable-level, and sentence-level acoustic coding vectors, and the spectral features extracted in step S12 are respectively passed through the phoneme-level, syllable-level, and sentence-level acoustic predictors to obtain phoneme-level, syllable-level, and sentence-level acoustic prediction vectors;

[0093] It is understandable that in order to obtain acoustic coding vectors at the phoneme level, syllable level, and sentence level and add them to the text coding vector, it is necessary to keep the dimensions of the acoustic coding vectors at the phoneme level, syllable level, and sentence level consistent with the text coding vector (i.e., the phoneme step length). Optionally, the alignment results between the spectral features and the text are obtained by a forced alignment algorithm; before the spectral features are input into the phoneme level and syllable level acoustic coding layers, the spectral features are averaged for each frame at the phoneme level and syllable level according to the alignment results between the spectral features and the text; the output dimension of the syllable level coding layer is expanded according to the phoneme step length and is consistent with the phoneme step length; optionally, the phoneme level, syllable level, and sentence level acoustic coding layers all adopt a CNN network structure; the last layer of the sentence level acoustic coding layer network structure is a pooling layer, and the output dimension of the sentence level acoustic coding layer is expanded according to the phoneme step length and is consistent with the phoneme step length; the acoustic predictor is consistent with the acoustic coding layer network structure; it is understandable that the input of the acoustic predictor is the text coding vector.

[0094] Specifically, the extraction process of acoustic coding vector and acoustic prediction vector is as follows: Figure 2 shown.

[0095] S15. The speaker recognition vector obtained in step S12, the text encoding vector obtained in step S13, and the phoneme-level, syllable-level, and sentence-level acoustic encoding vectors obtained in step S14 are input into the decoding layer to obtain predicted spectral features.

[0096] It can be understood that by adding the speaker recognition vector, text encoding vector, and phoneme-level, syllable-level, and sentence-level acoustic encoding vectors, the added vector is expanded based on the alignment result between the spectral features and the text and then input into the decoding layer; the decoding layer network structure includes but is not limited to CNN, Transformer and other network structures; the predicted spectral features are consistent with the real spectral features.

[0097] Among them, the text encoding layer includes but is not limited to CNN network structure and Transformer network structure; the speaker recognition module includes but is not limited to speaker recognition models such as GE2E; the used vocoders include but are not limited to WaveNET, WavRNN, and MelGAN.

[0098] like Figure 2 As shown, the application method of the model (i.e., the inference phase) includes the following steps:

[0099] S21. Normalize the synthesized text and extract true spectral features from the target speaker audio sample;

[0100] S22. Input the target spectral features extracted in step S21 into the pre-trained speaker recognition model in step S12 to obtain a target speaker recognition vector. Input the target spectral features extracted in step S21 into the acoustic predictor trained in step S14 to obtain phoneme-level, syllable-level, and sentence-level target acoustic prediction vectors.

[0101] S23. Input the text obtained in step S21 into the encoding layer trained in step S13 to obtain a text encoding vector;

[0102] S24. Input the phoneme-level, syllable-level, and sentence-level acoustic prediction vectors obtained in step S22 and the text encoding vector obtained in step S23 into the decoder layer trained in step S15 to obtain predicted spectral features;

[0103] S25. The predicted spectral features obtained in step S24 are used to generate synthesized audio by a pre-trained vocoder;

[0104] It is understandable that the parameters of the speech synthesis model in the inference stage are obtained in the training stage, and the network structure is consistent; the text processing method in the inference stage is consistent with that in the training stage; the training stage uses the alignment results of the real spectral features and the text to expand the text encoding vector, acoustic encoding vector, and speaker recognition vector, while the inference stage uses the prediction results of the duration predictor to expand the text encoding vector, acoustic prediction vector, and speaker recognition vector; the acoustic coding layer is not used in the inference stage.

[0105] At the same time, the present invention also discloses a speech synthesis device, which includes a processor and a memory; the memory is used to store instructions, and when the instructions are executed by the processor, the device implements any of the above-mentioned speech synthesis methods.

[0106] At the same time, the present invention also discloses a computer-readable storage medium, which stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer runs any of the above-mentioned speech synthesis methods.

[0107] In summary, the technical solution of the present application improves the sound quality and naturalness in the task of few-sample personalized speech synthesis by adding the phoneme-level, syllable-level, and sentence-level acoustic feature prediction vectors obtained by the adaptive acoustic feature predictor to the multi-speaker speech synthesis model.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Obtain target spectrum features of the target speaker; Processing the target spectral features based on a recognition model to obtain a speech recognition vector of the target speaker; Processing the target spectral features based on the prediction model to obtain phoneme-level, syllable-level, and sentence-level target acoustic prediction vectors respectively; Process the synthesized text based on the encoding model to obtain a text encoding vector; Processing the phoneme-level, syllable-level, and sentence-level target acoustic prediction vectors and the text encoding vector based on a decoding model to obtain predicted spectral features; Processing the predicted spectral features based on a vocoding model to obtain synthesized audio; The processing of the target spectrum feature further includes: obtaining an alignment result between the spectrum feature and the text to be synthesized by a forced alignment algorithm; According to the alignment results, the target spectral features are averaged for each frame at the phoneme level and the syllable level; The coding model includes a phoneme-level coding model, a syllable-level coding model, and a sentence-level coding model; The output dimension of the syllable-level encoding model is expanded according to the phoneme step size and is consistent with the phoneme step size.

2. The method according to claim 1, characterized in that The recognition model is a machine learning model, and the training samples of the recognition model include audios of multiple different speakers and corresponding texts.

3. The method according to claim 1, characterized in that The target spectrum feature includes at least one of the following: a linear spectrum feature and a Mel spectrum feature.

4. The method according to claim 1, wherein The text to be synthesized is a pre-processed synthesized text, wherein the pre-processing includes screening out illegal syllables, performing word segmentation, and part-of-speech tagging.

5. The method according to claim 4, characterized in that The pre-processing further comprises: Extracting comprehensive linguistic features from the synthesized text, and inputting the extracted comprehensive linguistic features into a prosody prediction model to obtain pause level annotations; The Chinese characters of the synthesized text are converted into corresponding pinyin phonemes.

6. The method according to claim 1, characterized in that The phoneme-level, syllable-level, sentence-level target acoustic prediction vectors and the text encoding vector are first processed as follows before being input into the decoding model: The speech recognition vector, the text encoding vector, and the phoneme-level, syllable-level, and sentence-level target acoustic prediction vectors are added together, and the added vectors are expanded based on the alignment results between the spectral features and the text to be synthesized.

7. The method according to any one of claims 1 to 6, characterized in that: The decoding model includes but is not limited to CNN and Transformer.

8. The method according to any one of claims 1 to 6, characterized in that: The vocoding model includes but is not limited to WaveNET, WavRNN, and MelGAN.

9. A speech synthesis system, characterized in that: include: An acquisition module, used to obtain target spectrum features of a target speaker; A recognition module, configured to process the target spectral features based on a recognition model to obtain a speech recognition vector of the target speaker; A prediction module, configured to process the target spectral features based on a prediction model to obtain target acoustic prediction vectors at the phoneme level, syllable level, and sentence level respectively; An encoding module, used to process the synthesized text based on the encoding model to obtain a text encoding vector; A decoding module, configured to process the phoneme-level, syllable-level, and sentence-level target acoustic prediction vectors and the text encoding vector based on a decoding model to obtain predicted spectral features; A vocoding module, configured to process the predicted spectral features based on a vocoding model to obtain synthesized audio; The processing of the target spectrum feature further includes: obtaining an alignment result between the spectrum feature and the text to be synthesized by a forced alignment algorithm; According to the alignment results, the target spectral features are averaged for each frame at the phoneme level and the syllable level; The coding model includes a phoneme-level coding model, a syllable-level coding model, and a sentence-level coding model; The output dimension of the syllable-level encoding model is expanded according to the phoneme step size and is consistent with the phoneme step size.

Images