Chinese speech synthesis method and device, terminal and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GREAT WALL MOTOR CO LTD
- Filing Date
- 2023-01-13
- Publication Date
- 2026-05-29
AI Technical Summary
In existing Chinese speech synthesis technology, phoneme sequences are used as input to the acoustic model, which makes it difficult to learn pronunciation rules and results in poor speech synthesis speed and quality.
Sub-character sequences with clear meanings are obtained through sub-character recognition. These sequences are then combined with phoneme and positional information and input into a trained Chinese speech synthesis model. The model utilizes a progressively optimized recurrent network, a language encoder, a phoneme word embedding sub-model, and a prosodic recognition sub-model to generate Mel spectrograms for speech synthesis.
It improves the speed and quality of Chinese speech synthesis, reduces the difficulty for the model to learn pronunciation rules, increases the amount of input information, and enhances the speech synthesis effect.
Smart Images
Figure CN116052636B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech synthesis technology, and in particular to a Chinese speech synthesis method, apparatus, terminal and storage medium. Background Technology
[0002] Chinese speech synthesis refers to the conversion of Chinese sentences into speech. A typical speech synthesis system mainly consists of three parts: a TTS (Text To Speech) front-end, an acoustic model, and a vocoder.
[0003] Currently, in Chinese speech synthesis, phoneme sequences from Hanyu Pinyin (e.g., the Chinese phonetic format of BOPOMOFO) are typically used directly as input to the acoustic model. However, phoneme sequences do not explicitly contain the meaning of Chinese sentences, increasing the difficulty for the acoustic model to learn pronunciation rules, resulting in poor speech synthesis speed and quality. Summary of the Invention
[0004] This application provides a Chinese speech synthesis method, apparatus, terminal, and storage medium to address the problem that current methods, which only use phoneme sequences as input to acoustic models, make it difficult for acoustic models to learn pronunciation rules, resulting in poor speech synthesis speed and quality.
[0005] In a first aspect, embodiments of this application provide a Chinese speech synthesis method, including:
[0006] The Chinese sentence is obtained, and sub-character recognition is performed on the Chinese sentence to obtain a sub-character sequence; where each sub-character in the sub-character sequence contains the meaning of the corresponding Chinese character;
[0007] Phoneme conversion is performed on the sub-word sequence to obtain the phoneme sequence and the phoneme position sequence;
[0008] Word embedding is performed on the sub-character sequence to obtain sub-character vectors;
[0009] The phoneme sequence, phoneme position sequence, and sub-character vector are input into the trained Chinese speech synthesis model to obtain the Mel spectrum corresponding to the Chinese sentence; the Mel spectrum is used to synthesize the speech corresponding to the Chinese sentence.
[0010] In one possible implementation, the Chinese speech synthesis model includes a progressively optimized recurrent network sub-model, a language encoder, a phoneme word embedding sub-model, a prosodic recognition sub-model, and an acoustic sub-model.
[0011] In one possible implementation, the phoneme sequence, phoneme position sequence, and sub-word vector are input into a trained Chinese speech synthesis model to obtain the Mel spectrum corresponding to the Chinese sentence, including:
[0012] Based on phoneme sequences, phoneme position sequences, sub-character vectors, a trained progressively optimized recurrent network sub-model, a trained language encoder, a trained phoneme word embedding sub-model, and a trained prosodic recognition sub-model, a prosodic insertion vector is obtained; the prosodic insertion vector contains semantic information, phoneme information, and prosodic information of the Chinese sentence.
[0013] By inputting the prosody insertion vector into the trained acoustic sub-model, the Mel spectrum corresponding to the Chinese sentence is obtained.
[0014] In one possible implementation, a prosodic insertion vector is obtained based on a phoneme sequence, a phoneme position sequence, sub-word vectors, a trained progressively optimized recurrent network sub-model, a trained language encoder, a trained phoneme word embedding sub-model, and a trained prosodic recognition sub-model, including:
[0015] Input the sub-character vector into the trained progressively optimized recurrent network sub-model to obtain the transformed first vector;
[0016] Obtain the positional encoding of the sub-character sequence, and input the vector obtained by adding the positional encoding of the sub-character sequence to the first vector into the trained language encoder to obtain the third vector;
[0017] The phoneme sequence is input into the trained phoneme word embedding sub-model to obtain the phoneme vector;
[0018] Based on the phoneme position sequence, the third vector is sliced to obtain the fourth vector, and the fourth vector is added to the phoneme vector to obtain the fifth vector;
[0019] The fifth vector is input into the trained prosody recognition sub-model to obtain prosody information;
[0020] The prosody information is inserted into the fifth vector to obtain the prosody insertion vector.
[0021] In one possible implementation, before obtaining the Mel spectrum corresponding to the Chinese sentence by inputting the phoneme sequence, phoneme position sequence, and sub-word vector into the trained Chinese speech synthesis model, the Chinese speech synthesis method further includes:
[0022] Obtain the training sample set;
[0023] Based on the training sample set, the recurrent network sub-model, language encoder, phoneme word embedding sub-model, prosody recognition sub-model, and acoustic sub-model are trained simultaneously to obtain a well-trained Chinese speech synthesis model.
[0024] In one possible implementation, the acoustic sub-model includes the FastSpeech2 model.
[0025] In one possible implementation, the sub-word sequence is phoneme-converted to obtain a phoneme sequence and a phoneme position sequence, including:
[0026] The pinyin of each character in the sequence is divided into its corresponding initial consonant, final vowel, and tone.
[0027] Based on the initials, finals, and tones of each sub-character in the sub-character sequence, as well as the punctuation marks in Chinese sentences, a phoneme sequence is obtained.
[0028] Obtain the position number of the Chinese character to which each phoneme belongs in the phoneme sequence within the Chinese sentence to obtain the phoneme position sequence.
[0029] Secondly, embodiments of this application provide a Chinese speech synthesis device, including:
[0030] The sub-character recognition module is used to acquire Chinese sentences and perform sub-character recognition on the Chinese sentences to obtain a sub-character sequence; where each sub-character in the sub-character sequence contains the meaning of the corresponding Chinese character;
[0031] The phoneme conversion module is used to convert sub-character sequences into phonemes, resulting in phoneme sequences and phoneme position sequences.
[0032] The sub-character word embedding module is used to perform word embedding processing on sub-character sequences to obtain sub-character vectors;
[0033] The speech synthesis module is used to input phoneme sequences, phoneme position sequences, and sub-character vectors into a trained Chinese speech synthesis model to obtain the Mel spectrum corresponding to the Chinese sentences; the Mel spectrum is used to synthesize the speech corresponding to the Chinese sentences.
[0034] Thirdly, embodiments of this application provide a terminal, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the Chinese speech synthesis method as described in the first aspect or any possible implementation of the first aspect.
[0035] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the Chinese speech synthesis method as described in the first aspect or any possible implementation thereof.
[0036] This application provides a Chinese speech synthesis method, apparatus, terminal, and storage medium. By performing sub-character recognition on Chinese sentences, a sub-character sequence composed of sub-characters containing the meaning of the corresponding Chinese characters is obtained. Sub-character vectors containing the meaning of the Chinese characters are obtained from the sub-character sequence. Finally, the sub-character vectors containing the Chinese meaning, the phoneme sequence containing phoneme information, and the phoneme position sequence containing phoneme position information are input into a trained Chinese speech synthesis model to obtain the Mel spectrum corresponding to the Chinese sentence. By using the sub-character vectors containing the Chinese meaning, the phoneme sequence containing phoneme information, and the phoneme position sequence containing phoneme position information as input, the difficulty of the Chinese speech synthesis model in learning pronunciation rules can be reduced, and the speed and quality of speech synthesis can be improved. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0038] Figure 1 This is a flowchart illustrating the implementation of a Chinese speech synthesis method according to an embodiment of this application;
[0039] Figure 2 This is a flowchart illustrating the implementation of a Chinese speech synthesis method according to another embodiment of this application;
[0040] Figure 3 This is a schematic diagram of the structure of a Chinese speech synthesis device provided in an embodiment of this application;
[0041] Figure 4 This is a schematic diagram of a terminal provided in one embodiment of this application. Detailed Implementation
[0042] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.
[0044] See Figure 1, which shows an implementation flowchart of the Chinese speech synthesis method provided by an embodiment of the present application. The execution subject of the Chinese speech synthesis method may be a terminal.
[0045] The above Chinese speech synthesis method is described in detail as follows:
[0046] In S101, a Chinese sentence is obtained, and sub-character recognition is performed on the Chinese sentence to obtain a sub-character sequence; wherein, each sub-character in the sub-character sequence contains the meaning of the corresponding Chinese character.
[0047] A Chinese character often has multiple meanings, and in a specific context, it shows one of its specific meanings. For example: in "a long time", "long" is pronounced as cháng, indicating a long time; in "not grow up", "long" is pronounced as zhǎng, indicating growth.
[0048] For a specific meaning of a certain Chinese character, it is called a sub-character, that is, a sub-character contains the meaning of the corresponding Chinese character in a specific sentence.
[0049] In this embodiment, a Chinese sentence to be converted into speech is obtained. By performing sub-character recognition on the Chinese sentence, a sub-character sequence of the Chinese sentence can be obtained. Each sub-character in the sub-character sequence contains the meaning of the corresponding Chinese character in the above Chinese sentence. A sub-character is usually represented in the form of adding a number after the corresponding Chinese character, and this number can represent the nth meaning of the Chinese character in the dictionary.
[0050] Exemplarily, see Figure 2 , assuming the Chinese sentence S is "Make a little progress every day", the sub-character sequence S* is (every-2, day-1, make progress-3, step-1, a little-2, point-1, point-1). Every-2 means that the meaning of "every" in the Chinese sentence S is the 2nd meaning in the dictionary. The meanings of other sub-characters are similar and will not be elaborated.
[0051] Among them, the above-mentioned sub-character recognition of the Chinese sentence to obtain the sub-character sequence may include:
[0052] Traverse and combine all sub-characters corresponding to each Chinese character in the above Chinese sentence to obtain all sub-character combinations of the above Chinese sentence;
[0053] Calculate the occurrence probability of each sub-character combination of the above Chinese sentence, and take the sub-character combination with the maximum occurrence probability as the sub-character sequence corresponding to the above Chinese sentence.
[0054] Because each sub-character has a unique pronunciation, after sub-character recognition, by applying the pinyin of the sub-character, the task of disambiguating polyphonic characters can be completed, and there is no need to further disambiguate polyphonic characters.
[0055] Each Chinese character, based on the number of meanings it possesses, can have a corresponding number of sub-characters, each sub-character corresponding to one of the meanings of the Chinese character. A combination of one sub-character for each Chinese character in a Chinese sentence is called a sub-character combination. For example, assuming a Chinese sentence contains two Chinese characters, and each character has two sub-characters, then the Chinese sentence has 2*2=4 sub-character combinations.
[0056] The above calculation of the probability of occurrence of each sub-character combination of the above Chinese sentence, and taking the sub-character combination with the highest probability as the sub-character sequence corresponding to the above Chinese sentence, may include:
[0057] Based on the n-gram model, the probability of occurrence of each sub-character combination in the above Chinese sentence is calculated;
[0058] By applying maximum likelihood estimation, the sub-character combination with the highest probability of occurrence among the various sub-character combinations of the above Chinese sentence is taken as the sub-character sequence corresponding to the above Chinese sentence.
[0059] The formula for calculating the probability of occurrence of each sub-character combination is as follows:
[0060]
[0061] in, Let m be any combination of sub-characters in a Chinese sentence, and m be the number of sub-characters in that combination. w represents the probability of this sub-character combination occurring. k for The k-th sub-character; For w k Follow The conditional probability follows; s is the larger of 1 and k-n+1; n is the value of n in the n-gram model, which can be set according to actual needs, and no specific restrictions are made here.
[0062] The above-mentioned calculation of the probability of occurrence of each sub-character combination in the Chinese sentence based on the n-gram model can be achieved by using sub-character corpus and the n-gram model. The sub-character corpus can be obtained from Chinese text corpus. Specifically, the Chinese text corpus can be cleaned and segmented to obtain a dataset; then, a pre-model is applied to the dataset to obtain a word embedding set; finally, k-means clustering is performed on the word embedding set to obtain the embedding of each sub-character and the sub-character representation of each sentence; the collection of sub-character representations for each sentence constitutes the sub-character corpus.
[0063] In S102, the sub-word sequence is converted into phonemes to obtain the phoneme sequence and the phoneme position sequence.
[0064] In this embodiment, by performing phoneme conversion on each sub-character in the sub-character sequence, a phoneme sequence composed of the phoneme information of each sub-character and a phoneme position sequence composed of the position information of each phoneme in the phoneme sequence can be obtained.
[0065] The phoneme sequence can include the initial consonant, final vowel, and tone of each sub-character in the sub-character sequence, as well as punctuation marks in Chinese sentences. The order of phonemes in the phoneme sequence can be arranged according to the order of sub-characters in the sub-character sequence, that is, according to the order of Chinese characters in Chinese sentences.
[0066] The phoneme position sequence can include the position number of the Chinese character to which each phoneme belongs in the phoneme sequence within the Chinese sentence.
[0067] In S103, word embedding is performed on the sub-character sequence to obtain the sub-character vector.
[0068] Sub-word vectors include word embeddings of each sub-word in the sub-word sequence.
[0069] Word embedding refers to converting each sub-character into a corresponding vector representation. This embodiment can use any implementable method to perform word embedding processing on the sub-character sequence to obtain sub-character vectors, and there are no specific limitations on this.
[0070] In S104, the phoneme sequence, phoneme position sequence, and sub-character vector are input into the trained Chinese speech synthesis model to obtain the Mel spectrum corresponding to the Chinese sentence; the Mel spectrum is used to synthesize the speech corresponding to the Chinese sentence.
[0071] A well-trained Chinese speech synthesis model can output the Mel spectrum corresponding to a Chinese sentence based on the phoneme sequence, phoneme position sequence, and sub-character vector. The speech corresponding to the Chinese sentence can then be synthesized based on the Mel spectrum.
[0072] This embodiment identifies sub-characters in Chinese sentences to obtain a sequence of sub-characters containing the meaning of the corresponding Chinese characters. Sub-character vectors containing the meaning of the Chinese characters are then obtained from the sub-character sequences. Finally, the sub-character vectors containing Chinese meaning, the phoneme sequence containing phoneme information, and the phoneme position sequence containing phoneme position information are input into a trained Chinese speech synthesis model to obtain the Mel spectrum corresponding to the Chinese sentence. By using the sub-character vectors containing Chinese meaning, the phoneme sequence containing phoneme information, and the phoneme position sequence containing phoneme position information as input, the difficulty of learning pronunciation rules in the Chinese speech synthesis model can be reduced, improving the speed and quality of speech synthesis. Furthermore, this embodiment uses Chinese sentences instead of Pinyin as the basis for speech synthesis, which increases the amount of input information for the Chinese speech synthesis model and improves the speech synthesis effect.
[0073] In some embodiments, the Chinese speech synthesis model described above includes a progressive recurrent network (PreNet) sub-model, a language encoder, a phoneme word embedding sub-model, a prosody recognition sub-model, and an acoustic sub-model.
[0074] This embodiment does not impose specific restrictions on the specific structures of the language encoder, phoneme word embedding sub-model, prosodic recognition sub-model, and acoustic sub-model; any model that can achieve the corresponding functions is acceptable.
[0075] In some embodiments, S104 may include:
[0076] Based on phoneme sequences, phoneme position sequences, sub-character vectors, a trained progressively optimized recurrent network sub-model, a trained language encoder, a trained phoneme word embedding sub-model, and a trained prosodic recognition sub-model, a prosodic insertion vector is obtained; the prosodic insertion vector contains semantic information, phoneme information, and prosodic information of the Chinese sentence.
[0077] By inputting the prosody insertion vector into the trained acoustic sub-model, the Mel spectrum corresponding to the Chinese sentence is obtained.
[0078] Wherein, the prosody insertion vector is Figure 2 In H(2), the prosodic insertion vector is a vector containing semantic, phonetic, and prosodic information of the Chinese sentence. Inputting the prosodic insertion vector into the trained acoustic sub-model can output the Mel spectrum corresponding to the Chinese sentence.
[0079] In some embodiments, the prosodic insertion vector obtained based on the phoneme sequence, phoneme position sequence, sub-word vector, trained progressively optimized recurrent network sub-model, trained language encoder, trained phoneme word embedding sub-model, and trained prosodic recognition sub-model includes:
[0080] Input the sub-character vector into the trained progressively optimized recurrent network sub-model to obtain the transformed first vector;
[0081] Obtain the positional encoding of the sub-character sequence, and input the vector obtained by adding the positional encoding of the sub-character sequence to the first vector into the trained language encoder to obtain the third vector;
[0082] The phoneme sequence is input into the trained phoneme word embedding sub-model to obtain the phoneme vector;
[0083] Based on the phoneme position sequence, the third vector is sliced to obtain the fourth vector, and the fourth vector is added to the phoneme vector to obtain the fifth vector;
[0084] The fifth vector is input into the trained prosody recognition sub-model to obtain prosody information;
[0085] The prosody information is inserted into the fifth vector to obtain the prosody insertion vector.
[0086] In this embodiment, after the sub-character vectors are input into the trained PreNet sub-model, the vector output by the trained PreNet sub-model is called the first vector. The PreNet sub-model performs spatial transformation on the sub-character vectors, which can be accomplished using only one Dense Layer (fully connected layer).
[0087] In this embodiment, the positional encoding of the sub-character sequence can be absolute positional encoding. The positional encoding of each sub-character can represent the position of the corresponding Chinese character in a Chinese sentence.
[0088] After adding the positional encoding of the sub-character sequence to the first vector, and then passing it through a trained language encoder, the hidden layer encoding of the Chinese sentence can be obtained, which is the third vector, also known as the hidden layer vector. The third vector contains the semantic information of the Chinese sentence.
[0089] The third vector is... Figure 2 H(0) in the equation. H(0), S, and S* have the same length.
[0090] H(0) = LE(PN(S*) + PE(S*)); PE(S*) represents the positional encoding of S*; PN(S*) represents the first vector obtained by PreNet transformation; LE() represents Language Encoder.
[0091] The phoneme word embedding sub-model is a phoneme embedding model that can perform word embedding processing on phoneme sequences to obtain phoneme vectors. Phoneme sequences or phoneme vectors contain explicit pronunciation information of Chinese sentences, that is, they contain the phoneme information of Chinese sentences.
[0092] Slicing operation Figure 2The Slice operation is performed on the third vector based on the phoneme position sequence to obtain the fourth vector, namely Slice(H(0),Indexw), which means slicing the corresponding vector from H(0) according to Indexw. For example, assuming H(0) is a 7*512 vector, that is, 7 words, each word has 512 features, and Indexw is (1 1 1 2 2 2 3 3 34 4 4 5 5 5 6 6 6 7 7 7), then the fourth vector is a 21*512 vector. Slice(H(0),Indexw) means that the vectors corresponding to the first character in H(0) are sliced out according to Indexw to obtain the fourth vector. That is, the first number 1 in Indexw means that the vector corresponding to the first character in H(0) is taken out (first row), the second number 1 and the third number 1 are the same, the fourth number 2 means that the vector corresponding to the second character in H(0) is taken out (second row), and so on, until all numbers in Indexw have been sliced out into corresponding vectors. Then, the vectors corresponding to each number are combined into the fourth vector according to the order in Indexw.
[0093] Adding the fourth vector to the phoneme vector yields the fifth vector. The fifth vector is... Figure 2 H(1) contains the semantic information and phoneme information of the Chinese sentence. H(1) = Slice(H(0), Indexw) + Embedding(P); Embedding(P) represents the word embedding representation of the phoneme sequence, i.e., the phoneme vector.
[0094] The prosodic recognition sub-model can achieve prosodic recognition. Any model structure capable of prosodic recognition is acceptable, without specific restrictions. Inputting the fifth vector into the trained prosodic recognition sub-model yields the prosodic recognition result, i.e., the prosodic information PP. By inserting the prosodic information PP into the fifth vector H(1) through the PI (Prosodic Insertion) module, the prosodic insertion vector H(2) can be obtained. H(2) contains both semantic and phonemic information, as well as prosodic information.
[0095] This embodiment uses a four-level prosodic structure: #1#2#3#4. The PI module can be used to embed the second, third, and fourth prosodic levels and insert them after the corresponding H(1) sequence number.
[0096] #1 The smallest unit of rhythmic words, without pause; #2 Stressed, with a very short pause; #3 Rhythmic phrases, with normal pauses; #4 Indicates the end of a complete sentence.
[0097] In some possible implementations, the language encoder can be structured like the Language Encoder in the Transformer architecture.
[0098] In some embodiments, prior to S104 above, the Chinese speech synthesis method further includes:
[0099] Obtain the training sample set;
[0100] Based on the training sample set, the recurrent network sub-model, language encoder, phoneme word embedding sub-model, prosody recognition sub-model, and acoustic sub-model are trained simultaneously to obtain a well-trained Chinese speech synthesis model.
[0101] The training sample set can include multiple training samples. Each training sample can include the training input parameters and corresponding training labels (i.e., training output parameters) of each sub-model in the Chinese speech synthesis model. A certain training parameter may be both the training label of the previous sub-model and the training input parameter of the next sub-model.
[0102] For example, the training input parameters of the training samples may include S*, Indexw, Phoneme, PP, and duration (duration, representing the duration of each phoneme in H(2)). The training labels may include PP, duration, and mel-spectrogram. S*, Indexw, and Phoneme are calculated in advance from the Chinese sentence S, and the calculation method is consistent with the inference process. PP is the prosodic information label, which serves as both the input to the PI module and the training label for the prosodic recognition sub-model during the training phase. MSE is used as the loss function for this part of the training. Duration represents the duration of each phoneme in H(2), which indicates how many mel frames correspond to this phoneme, and is used during training in the FastSpeech2 model. The MFA (Montreal forced aligner) tool used in FastSpeech2 can also be used to calculate and extract duration information.
[0103] Existing technologies train the TTS front-end and acoustic model separately, making it difficult to eliminate errors between modules. This embodiment, however, trains all models in the Chinese speech synthesis model together, effectively training the TTS front-end and acoustic model uniformly. This eliminates errors between modules and improves the quality and speed of speech synthesis.
[0104] In some embodiments, the acoustic submodel includes the FastSpeech2 model.
[0105] This embodiment uses the FastSpeech2 model as the acoustic sub-model. Of course, other feasible models can also be used as acoustic sub-models, such as FastSpeech, etc., and no specific restrictions are made here.
[0106] In some embodiments, S102 may include:
[0107] The pinyin of each character in the sequence is divided into its corresponding initial consonant, final vowel, and tone.
[0108] Based on the initials, finals, and tones of each sub-character in the sub-character sequence, as well as the punctuation marks in Chinese sentences, a phoneme sequence is obtained.
[0109] Obtain the position number of the Chinese character to which each phoneme belongs in the phoneme sequence within the Chinese sentence to obtain the phoneme position sequence.
[0110] See Figure 2 Each character in the Chinese dictionary has a unique pinyin. A simple query can determine the pinyin composition of the entire Chinese sentence, and also disambiguation of polyphonic characters within the sentence. The pinyin of each character can be segmented into its initial consonant, final vowel, and tone. These initial consonants, final vowels, and tones of each character are then used as phonemes, combined with punctuation marks in the Chinese sentence, to form a phoneme sequence (Phoneme). Simultaneously, the position index of the Chinese character to which each phoneme belongs within the Chinese sentence can be extracted and sequentially formed into a phoneme position sequence (Indexw).
[0111] For example, with Figure 2 Taking the Chinese sentence as an example, the phoneme sequence Phoneme is (m ei 3t ian 1j in 4bu 4y i 4d ian 3d ian 3), and the corresponding phoneme position sequence Indexw is (1 1 1 2 2 2 3 3 34 4 4 5 55 6 6 6 7 7 7).
[0112] Existing technologies directly use the phoneme sequence of Chinese Pinyin as input to the acoustic model, sometimes combined with prosodic information. This method requires separate models for polyphonic character disambiguation and prosodic recognition; moreover, the information input to the acoustic model does not explicitly contain the meaning of Chinese sentences, increasing the difficulty for the acoustic model to learn pronunciation rules. In addition, the TTS front-end and acoustic model are trained and inferred separately; errors between modules are not easy to eliminate, which also reduces the inference speed.
[0113] This application treats the TTS front-end and acoustic model as a whole—a Chinese speech synthesis model—for training and inference, directly using Chinese sentences as the basis, thus solving the aforementioned problems. Specifically, using Chinese sentences as the basis increases the amount of input information for the Chinese speech synthesis model, improving the speech synthesis effect and speed; training the TTS front-end and acoustic model as a whole—a unified Chinese speech synthesis model—further improves the speech synthesis effect and speed; and using sub-character recognition as the front-end task of speech synthesis improves the model's inference speed.
[0114] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0115] The following are device embodiments of this application. For details not described in detail, please refer to the corresponding method embodiments described above.
[0116] Figure 3 A schematic diagram of the Chinese speech synthesis device provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown, and are described in detail below:
[0117] like Figure 3 As shown, the Chinese speech synthesis device 30 includes: a sub-character recognition module 31, a phoneme conversion module 32, a sub-character word embedding module 33, and a speech synthesis module 34.
[0118] The sub-character recognition module 31 is used to acquire Chinese sentences and perform sub-character recognition on the Chinese sentences to obtain a sub-character sequence; wherein each sub-character in the sub-character sequence contains the meaning of the corresponding Chinese character;
[0119] Phoneme conversion module 32 is used to perform phoneme conversion on the sub-character sequence to obtain a phoneme sequence and a phoneme position sequence;
[0120] The sub-character word embedding module 33 is used to perform word embedding processing on the sub-character sequence to obtain sub-character vectors;
[0121] The speech synthesis module 34 is used to input the phoneme sequence, phoneme position sequence and sub-character vector into the trained Chinese speech synthesis model to obtain the Mel spectrum corresponding to the Chinese sentence; the Mel spectrum is used to synthesize the speech corresponding to the Chinese sentence.
[0122] In one possible implementation, the Chinese speech synthesis model in the speech synthesis module 34 includes a progressively optimized recurrent network sub-model, a language encoder, a phoneme word embedding sub-model, a prosodic recognition sub-model, and an acoustic sub-model.
[0123] In one possible implementation, the speech synthesis module 34 is specifically used for:
[0124] Based on phoneme sequences, phoneme position sequences, sub-character vectors, a trained progressively optimized recurrent network sub-model, a trained language encoder, a trained phoneme word embedding sub-model, and a trained prosodic recognition sub-model, a prosodic insertion vector is obtained; the prosodic insertion vector contains semantic information, phoneme information, and prosodic information of the Chinese sentence.
[0125] By inputting the prosody insertion vector into the trained acoustic sub-model, the Mel spectrum corresponding to the Chinese sentence is obtained.
[0126] In one possible implementation, in the speech synthesis module 34, the prosodic insertion vector is obtained based on the phoneme sequence, phoneme position sequence, sub-word vector, trained progressively optimized recurrent network sub-model, trained language encoder, trained phoneme word embedding sub-model, and trained prosodic recognition sub-model, including:
[0127] Input the sub-character vector into the trained progressively optimized recurrent network sub-model to obtain the transformed first vector;
[0128] Obtain the positional encoding of the sub-character sequence, and input the vector obtained by adding the positional encoding of the sub-character sequence to the first vector into the trained language encoder to obtain the third vector;
[0129] The phoneme sequence is input into the trained phoneme word embedding sub-model to obtain the phoneme vector;
[0130] Based on the phoneme position sequence, the third vector is sliced to obtain the fourth vector, and the fourth vector is added to the phoneme vector to obtain the fifth vector;
[0131] The fifth vector is input into the trained prosody recognition sub-model to obtain prosody information;
[0132] The prosody information is inserted into the fifth vector to obtain the prosody insertion vector.
[0133] In one possible implementation, the Chinese speech synthesis device 30 also includes a training module;
[0134] The training module is used to obtain a training sample set. Based on the training sample set, the recurrent network sub-model, language encoder, phoneme word embedding sub-model, prosody recognition sub-model and acoustic sub-model are trained simultaneously to obtain a trained Chinese speech synthesis model.
[0135] In one possible implementation, the acoustic sub-model includes the FastSpeech2 model.
[0136] In one possible implementation, the phoneme conversion module 32 is specifically used for:
[0137] The pinyin of each character in the sequence is divided into its corresponding initial consonant, final vowel, and tone.
[0138] Based on the initials, finals, and tones of each sub-character in the sub-character sequence, as well as the punctuation marks in Chinese sentences, a phoneme sequence is obtained.
[0139] Obtain the position number of the Chinese character to which each phoneme belongs in the phoneme sequence within the Chinese sentence to obtain the phoneme position sequence.
[0140] This application also provides a computer program product having program code that, when run in a corresponding processor, controller, computing device, or terminal, executes the steps in any of the above-described Chinese speech synthesis method embodiments, for example... Figure 1 S101 to S104 are shown. Those skilled in the art will understand that the methods and apparatus proposed in the embodiments of this application can be implemented in various forms, including hardware, software, firmware, dedicated processors, or combinations thereof. Dedicated processors may include application-specific integrated circuits (ASICs), reduced instruction set computers (RISCs), and / or field-programmable gate arrays (FPGAs). The proposed methods and apparatus are preferably implemented as a combination of hardware and software. The software is preferably installed as an application program on a program storage device. This is typically based on a computer platform with hardware such as one or more central processing units (CPUs), random access memory (RAM), and one or more input / output (I / O) interfaces. An operating system is also typically installed on the computer platform. The various processes and functions described herein may be part of an application program, or a portion thereof may be executed by an operating system.
[0141] Figure 4 This is a schematic diagram of the terminal provided in an embodiment of this application. For example... Figure 4 As shown, the terminal 4 in this embodiment includes a processor 40 and a memory 41. The memory 41 stores a computer program 42, and the processor 40 calls and runs the computer program 42 stored in the memory 41 to execute the steps described in the various terminal recording method embodiments above, for example... Figure 1 S101 to S104 are shown. Alternatively, the processor 40 is used to call and run the computer program 42 stored in the memory 41 to implement the functions of each module / unit in the above-described device embodiments, for example... Figure 3 The functions of modules / units 31 to 34 shown.
[0142] For example, the computer program 42 can be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to complete / implement the solution provided in this application. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 42 in the terminal 4. For example, the computer program 42 can be divided into... Figure 3 Modules / units 31 to 34 are shown.
[0143] The terminal 4 can be a computer, server, or other device; it can also be a mobile phone, tablet computer, or ECU (Electronic Control Unit). Terminal 4 may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that... Figure 4 This is merely an example of terminal 4 and does not constitute a limitation on terminal 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal may also include input / output devices, network access devices, buses, etc.
[0144] The processor 40 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0145] The memory 41 can be an internal storage unit of the terminal 4, such as a hard disk or memory of the terminal 4. The memory 41 can also be an external storage device of the terminal 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal 4. Furthermore, the memory 41 can include both internal storage units and external storage devices of the terminal 4. The memory 41 is used to store the computer program and other programs and data required by the terminal. The memory 41 can also be used to temporarily store data that has been output or will be output.
[0146] The aforementioned terminal can be applied to any device that requires speech synthesis, such as vehicles.
[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0148] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0149] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0150] In the embodiments provided in this application, it should be understood that the disclosed apparatus / control devices and methods can be implemented in other ways. For example, the apparatus / control device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0151] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0152] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0153] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various Chinese speech synthesis method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.
[0154] Furthermore, the features of the embodiments shown in the accompanying drawings or the various embodiments mentioned in this specification should not be construed as independent embodiments. Rather, each feature described in one example of an embodiment can be combined with one or more other desired features from other embodiments to produce other embodiments not described in words or with reference to the accompanying drawings.
[0155] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A Chinese speech synthesis method, characterized in that, include: A Chinese sentence is obtained, and sub-character recognition is performed on the Chinese sentence to obtain a sub-character sequence; wherein each sub-character in the sub-character sequence contains the meaning of the corresponding Chinese character; the sub-character recognition of the Chinese sentence to obtain the sub-character sequence includes: traversing and combining all sub-characters corresponding to each Chinese character in the Chinese sentence to obtain all sub-character combinations of the Chinese sentence; calculating the occurrence probability of each sub-character combination of the above Chinese sentence, and taking the sub-character combination with the highest occurrence probability as the sub-character sequence corresponding to the above Chinese sentence; The sub-character sequence is subjected to phoneme conversion to obtain a phoneme sequence and a phoneme position sequence; The sub-character sequence is subjected to word embedding processing to obtain sub-character vectors; The phoneme sequence, the phoneme position sequence, and the sub-character vector are input into a trained Chinese speech synthesis model to obtain the Mel spectrum corresponding to the Chinese sentence; the Mel spectrum is used to synthesize the speech corresponding to the Chinese sentence.
2. The Chinese speech synthesis method according to claim 1, characterized in that, The Chinese speech synthesis model includes a progressively optimized recurrent network sub-model, a language encoder, a phoneme word embedding sub-model, a prosodic recognition sub-model, and an acoustic sub-model.
3. The Chinese speech synthesis method according to claim 2, characterized in that, The step of inputting the phoneme sequence, the phoneme position sequence, and the sub-character vector into a trained Chinese speech synthesis model to obtain the Mel spectrum corresponding to the Chinese sentence includes: Based on the phoneme sequence, the phoneme position sequence, the sub-word vector, the trained progressively optimized recurrent network sub-model, the trained language encoder, the trained phoneme word embedding sub-model, and the trained prosodic recognition sub-model, a prosodic insertion vector is obtained; the prosodic insertion vector contains the semantic information, phoneme information, and prosodic information of the Chinese sentence; The prosody insertion vector is input into the trained acoustic sub-model to obtain the Mel spectrum corresponding to the Chinese sentence.
4. The Chinese speech synthesis method according to claim 3, characterized in that, The prosodic insertion vector is obtained based on the phoneme sequence, the phoneme position sequence, the sub-word vector, the trained progressively optimized recurrent network sub-model, the trained language encoder, the trained phoneme word embedding sub-model, and the trained prosodic recognition sub-model, including: The sub-character vector is input into the trained progressively optimized recurrent network sub-model to obtain the transformed first vector; The positional encoding of the sub-character sequence is obtained, and the vector obtained by adding the positional encoding of the sub-character sequence and the first vector is input into the trained language encoder to obtain the third vector; The phoneme sequence is input into the trained phoneme word embedding sub-model to obtain a phoneme vector; Based on the phoneme position sequence, the third vector is sliced to obtain the fourth vector, and the fourth vector is added to the phoneme vector to obtain the fifth vector; The fifth vector is input into the trained prosody recognition sub-model to obtain prosody information; The prosody information is inserted into the fifth vector to obtain the prosody insertion vector.
5. The Chinese speech synthesis method according to claim 2, characterized in that, Before obtaining the Mel spectrum corresponding to the Chinese sentence in the trained Chinese speech synthesis model by inputting the phoneme sequence, the phoneme position sequence, and the sub-character vector, the Chinese speech synthesis method further includes: Obtain the training sample set; Based on the training sample set, the progressively optimized recurrent network sub-model, the language encoder, the phoneme word embedding sub-model, the prosody recognition sub-model, and the acoustic sub-model are trained simultaneously to obtain the trained Chinese speech synthesis model.
6. The Chinese speech synthesis method according to claim 2, characterized in that, The acoustic sub-model includes the FastSpeech2 model.
7. The Chinese speech synthesis method according to any one of claims 1 to 6, characterized in that, The step of performing phoneme conversion on the sub-character sequence to obtain a phoneme sequence and a phoneme position sequence includes: The pinyin of each sub-character in the sub-character sequence is divided into its corresponding initial consonant, final vowel, and tone; The phoneme sequence is obtained based on the initials, finals, and tones corresponding to each sub-character in the sub-character sequence, as well as the punctuation marks in the Chinese sentence; The position number of the Chinese character to which each phoneme belongs in the phoneme sequence is obtained in the Chinese sentence, thus obtaining the phoneme position sequence.
8. A Chinese speech synthesis device, characterized in that, include: The sub-character recognition module is used to acquire Chinese sentences and perform sub-character recognition on the Chinese sentences to obtain a sub-character sequence; wherein each sub-character in the sub-character sequence contains the meaning of the corresponding Chinese character; the sub-character recognition module is specifically used to: traverse and combine all sub-characters corresponding to each Chinese character in the Chinese sentences to obtain all sub-character combinations of the Chinese sentences; calculate the occurrence probability of each sub-character combination of the above Chinese sentences, and take the sub-character combination with the highest occurrence probability as the sub-character sequence corresponding to the above Chinese sentences; A phoneme conversion module is used to perform phoneme conversion on the sub-character sequence to obtain a phoneme sequence and a phoneme position sequence; The sub-character word embedding module is used to perform word embedding processing on the sub-character sequence to obtain sub-character vectors; The speech synthesis module is used to input the phoneme sequence, the phoneme position sequence, and the sub-character vector into a trained Chinese speech synthesis model to obtain the Mel spectrum corresponding to the Chinese sentence; the Mel spectrum is used to synthesize the speech corresponding to the Chinese sentence.
9. A terminal, characterized in that, It includes a memory and a processor, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the Chinese speech synthesis method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the Chinese speech synthesis method as described in any one of claims 1 to 7.